Skip to content

Instantly share code, notes, and snippets.

@rly
Created January 16, 2024 19:28
Show Gist options
  • Save rly/7c156bf0c7d88f69acd0c2e2ad564fff to your computer and use it in GitHub Desktop.
Save rly/7c156bf0c7d88f69acd0c2e2ad564fff to your computer and use it in GitHub Desktop.
Demonstration of how to remove the Units table from an NWB file
Display the source blob
Display the rendered blob
Raw
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Removing the Units Table from an NWB file\n",
"\n",
"## Introduction\n",
"\n",
"Removing data from an NWB file is generally NOT recommended. It must be done carefully so as to maintain the validity\n",
"and internal consistency of a file. Think of it like data surgery. To remove some data from an NWB object, you may \n",
"need to create a clone of that NWB object with data removed, and you may need to disconnect NWB objects that you \n",
"do not intend to remove and then reconnect them with appropriate links and references to your cloned NWB object\n",
"with data removed. This procedure should be done with manual inspection at every step. Please consider generating\n",
"a new NWB file from scratch instead of removing data from an NWB file.\n",
"\n",
"Due to how the HDF5 storage format works, opening an NWB file in append mode and deleting or removing data from the file\n",
"will, in general, not physically delete the data from the file. It just removes the data from the index. As a result,\n",
"the file size will not be reduced.\n",
"\n",
"If you want to permanently delete data from the file and reduce the file size, you must create a new file based on the\n",
"original file. You can do this in several ways.\n",
"\n",
"1. Open the NWB file in PyNWB, modify the `NWBFile` object or its children objects in memory, and then \n",
" [export](https://pynwb.readthedocs.io/en/stable/export.html) the modified `NWBFile` object to a new file path. We will\n",
" work through that approach below.\n",
"\n",
"2. Remove (unlink) data from the file using `h5py` or other HDF5 libraries, and then use the \n",
" [h5repack](https://portal.hdfgroup.org/display/HDF5/h5repack) tool from the HDF5 library to create a new file without\n",
" the unlinked data. This method has a key disadvantage in that there are fewer checks that the modified file is \n",
" a valid NWB file. This method will not be described here.\n",
"\n",
"## Download a test NWB file\n",
"\n",
"First, download an example NWB file from DANDI. We will use a 73.2 MB NWB file from Ueli Rutishauser's Lab. \n",
"In your browser:\n",
"\n",
"1. Go to https://dandiarchive.org/dandiset/000004\n",
"2. In the top-right of the page, click Files\n",
"3. Click the first folder \"sub-P10HMH\"\n",
"4. Click the download icon for the first file \"sub-P10HMH_ses-20060901_ecephys+image.nwb\". It may take some\n",
" time to download this file, depending on your internet connection.\n",
"\n",
"## Install PyNWB\n",
"\n",
"Then, you will need to install the latest version of PyNWB from PyPI or conda-forge,\n",
"ideally in a clean virtual environment. Here, let's use `mamba` to create a new conda environment called \"nwb-remove\"\n",
"with `jupyterlab` installed so that we can run this notebook. Then let's use `pip` to install `pynwb` from PyPI.\n",
"\n",
"In a terminal or command prompt with [mambaforge](https://github.com/conda-forge/miniforge#mambaforge) installed, run:\n",
"\n",
"```bash\n",
"mamba create --name nwb-remove --yes jupyterlab\n",
"mamba activate nwb-remove\n",
"pip install pynwb\n",
"```\n",
"\n",
"PyNWB is now installed, so we can use it to open and manipulate the file.\n",
"\n",
"## Open the NWB file\n",
"\n",
"Open this notebook, change the `file_path` variable in the cell below to be the location of your downloaded\n",
"NWB file, and run the cell to read the file into an `NWBFile` object in memory."
]
},
{
"cell_type": "code",
"execution_count": 19,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"root pynwb.file.NWBFile at 0x4985526096\n",
"Fields:\n",
" acquisition: {\n",
" events <class 'pynwb.misc.AnnotationSeries'>,\n",
" experiment_ids <class 'pynwb.base.TimeSeries'>\n",
" }\n",
" data_collection: learning: 80, recognition: 81\n",
" devices: {\n",
" Neuralynx-cheetah <class 'pynwb.device.Device'>\n",
" }\n",
" electrode_groups: {\n",
" Neuralynx-cheetah-microwires-1 <class 'pynwb.ecephys.ElectrodeGroup'>,\n",
" Neuralynx-cheetah-microwires-13 <class 'pynwb.ecephys.ElectrodeGroup'>,\n",
" Neuralynx-cheetah-microwires-14 <class 'pynwb.ecephys.ElectrodeGroup'>,\n",
" Neuralynx-cheetah-microwires-15 <class 'pynwb.ecephys.ElectrodeGroup'>,\n",
" Neuralynx-cheetah-microwires-16 <class 'pynwb.ecephys.ElectrodeGroup'>,\n",
" Neuralynx-cheetah-microwires-17 <class 'pynwb.ecephys.ElectrodeGroup'>,\n",
" Neuralynx-cheetah-microwires-18 <class 'pynwb.ecephys.ElectrodeGroup'>,\n",
" Neuralynx-cheetah-microwires-19 <class 'pynwb.ecephys.ElectrodeGroup'>,\n",
" Neuralynx-cheetah-microwires-2 <class 'pynwb.ecephys.ElectrodeGroup'>,\n",
" Neuralynx-cheetah-microwires-20 <class 'pynwb.ecephys.ElectrodeGroup'>,\n",
" Neuralynx-cheetah-microwires-22 <class 'pynwb.ecephys.ElectrodeGroup'>,\n",
" Neuralynx-cheetah-microwires-23 <class 'pynwb.ecephys.ElectrodeGroup'>,\n",
" Neuralynx-cheetah-microwires-24 <class 'pynwb.ecephys.ElectrodeGroup'>,\n",
" Neuralynx-cheetah-microwires-3 <class 'pynwb.ecephys.ElectrodeGroup'>,\n",
" Neuralynx-cheetah-microwires-4 <class 'pynwb.ecephys.ElectrodeGroup'>,\n",
" Neuralynx-cheetah-microwires-5 <class 'pynwb.ecephys.ElectrodeGroup'>,\n",
" Neuralynx-cheetah-microwires-6 <class 'pynwb.ecephys.ElectrodeGroup'>,\n",
" Neuralynx-cheetah-microwires-8 <class 'pynwb.ecephys.ElectrodeGroup'>\n",
" }\n",
" electrodes: electrodes <class 'hdmf.common.table.DynamicTable'>\n",
" experiment_description: The data contained within this file describes a new/old recogntion task performed in patients with intractable epilepsy implanted with depth electrodes and Behnke-Fried microwires in the human Medical Temporal Lobe (MTL).\n",
" file_create_date: [datetime.datetime(2020, 5, 26, 19, 8, 23, 826535, tzinfo=tzoffset(None, -25200))]\n",
" identifier: H10_7\n",
" institution: Hunigton Memorial Hospital\n",
" intervals: {\n",
" trials <class 'pynwb.epoch.TimeIntervals'>\n",
" }\n",
" keywords: <StrDataset for HDF5 dataset \"keywords\": shape (7,), type \"|O\">\n",
" lab: Rutishauser\n",
" related_publications: ['Faraut et al. 2018, Scientific Data; Rutishauser et al. 2015, Nat Neurosci;']\n",
" session_description: New/Old recognition task for ID: 7. \n",
" session_start_time: 2006-09-01 00:00:00-07:00\n",
" stimulus: {\n",
" StimulusPresentation <class 'pynwb.image.OpticalSeries'>\n",
" }\n",
" subject: subject pynwb.file.Subject at 0x5245430416\n",
"Fields:\n",
" age: 37\n",
" description: Left Lateral Frontal\n",
" sex: M\n",
" species: Human\n",
" subject_id: P10HMH\n",
"\n",
" timestamps_reference_time: 2006-09-01 00:00:00-07:00\n",
" trials: trials <class 'pynwb.epoch.TimeIntervals'>\n",
" units: units <class 'pynwb.misc.Units'>\n",
"\n"
]
}
],
"source": [
"from pynwb import NWBHDF5IO\n",
"\n",
"# CHANGE THIS PATH to the location of the downloaded test NWB file\n",
"file_path = \"/Users/rly/Documents/NWB_Data/dandisets/000004/sub-P10HMH/sub-P10HMH_ses-20060901_ecephys+image.nwb\"\n",
"\n",
"read_io = NWBHDF5IO(file_path, mode=\"r\")\n",
"nwbfile = read_io.read()\n",
"print(nwbfile)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Remove the Units table\n",
"\n",
"In this example, we will remove the `Units` table from the file without replacing it. This will still result in a valid\n",
"NWB file because the `Units` table is optional.\n",
"\n",
"First, let's look at the `Units` table in the file. We can see that it has 37 rows (units)."
]
},
{
"cell_type": "code",
"execution_count": 20,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>origClusterID</th>\n",
" <th>waveform_mean_encoding</th>\n",
" <th>waveform_mean_recognition</th>\n",
" <th>IsolationDist</th>\n",
" <th>SNR</th>\n",
" <th>waveform_mean_sampling_rate</th>\n",
" <th>spike_times</th>\n",
" <th>electrodes</th>\n",
" </tr>\n",
" <tr>\n",
" <th>id</th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>970</td>\n",
" <td>[2.1764968283053565, 2.3406132503530475, 2.453...</td>\n",
" <td>[2.255677148872515, 2.438650365560623, 2.54682...</td>\n",
" <td>54.949626</td>\n",
" <td>2.074611</td>\n",
" <td>[98400.0]</td>\n",
" <td>[4790.086128, 4790.392488, 4790.808088, 4791.1...</td>\n",
" <td>x y z imp location filterin...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>2519</td>\n",
" <td>[1.9953024730819329, 2.146922925948179, 2.2750...</td>\n",
" <td>[1.8679882172094504, 2.0195327093193445, 2.149...</td>\n",
" <td>1301.944632</td>\n",
" <td>1.535590</td>\n",
" <td>[98400.0]</td>\n",
" <td>[4789.163768, 4789.171688, 4790.086128, 4790.7...</td>\n",
" <td>x y z imp location filterin...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>2569</td>\n",
" <td>[-1.0892165502550855, -1.194207382098711, -1.2...</td>\n",
" <td>[-1.2739211957333492, -1.3850635003577056, -1....</td>\n",
" <td>2700.668258</td>\n",
" <td>1.721514</td>\n",
" <td>[98400.0]</td>\n",
" <td>[4787.484848, 4787.756208, 4788.171848, 4788.3...</td>\n",
" <td>x y z imp location filterin...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>2808</td>\n",
" <td>[3.9792519048004746, 4.225281427560096, 4.4375...</td>\n",
" <td>[4.554116369553418, 4.791653736023812, 4.99612...</td>\n",
" <td>2798.328329</td>\n",
" <td>2.428861</td>\n",
" <td>[98400.0]</td>\n",
" <td>[4790.086128, 4796.592248, 4797.417168, 4798.8...</td>\n",
" <td>x y z imp location filterin...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>3048</td>\n",
" <td>[-7.368623732940135, -7.666343704498384, -7.93...</td>\n",
" <td>[-7.309179748134245, -7.610296381602412, -7.85...</td>\n",
" <td>NaN</td>\n",
" <td>3.924017</td>\n",
" <td>[98400.0]</td>\n",
" <td>[4787.674888, 4787.861608, 4788.101208, 4788.2...</td>\n",
" <td>x y z imp location filterin...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>5</th>\n",
" <td>4126</td>\n",
" <td>[2.3131939714515104, 2.49287475093624, 2.63675...</td>\n",
" <td>[2.2916029270428444, 2.4356759764683518, 2.549...</td>\n",
" <td>3209.140751</td>\n",
" <td>1.817625</td>\n",
" <td>[98400.0]</td>\n",
" <td>[4790.086088, 4794.246448, 4797.826848, 4798.8...</td>\n",
" <td>x y z imp location filterin...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>6</th>\n",
" <td>4245</td>\n",
" <td>[-5.777442084370658, -6.052566885779441, -6.26...</td>\n",
" <td>[-6.394055601868903, -6.6383642870196224, -6.8...</td>\n",
" <td>25.765571</td>\n",
" <td>3.297644</td>\n",
" <td>[98400.0]</td>\n",
" <td>[4787.630928, 4788.213568, 4788.266488, 4790.6...</td>\n",
" <td>x y z imp location filterin...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>7</th>\n",
" <td>4329</td>\n",
" <td>[-1.9242575810453464, -2.0852046175911347, -2....</td>\n",
" <td>[-1.6547120693159738, -1.782868893659467, -1.8...</td>\n",
" <td>116.662306</td>\n",
" <td>1.637649</td>\n",
" <td>[98400.0]</td>\n",
" <td>[4788.378728, 4788.789968, 4791.626488, 4791.6...</td>\n",
" <td>x y z imp location filterin...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>8</th>\n",
" <td>2498</td>\n",
" <td>[-2.4713541945868465, -2.6218310636669306, -2....</td>\n",
" <td>[-2.5624289166817404, -2.749951452957572, -2.8...</td>\n",
" <td>1016.161571</td>\n",
" <td>1.558819</td>\n",
" <td>[98400.0]</td>\n",
" <td>[4789.165328, 4789.588288, 4790.794608, 4790.8...</td>\n",
" <td>x y z imp location filterin...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>9</th>\n",
" <td>2530</td>\n",
" <td>[1.8563672622890155, 1.9997108346169516, 2.120...</td>\n",
" <td>[1.9445634547101074, 2.0918185669478353, 2.199...</td>\n",
" <td>780.642771</td>\n",
" <td>1.618448</td>\n",
" <td>[98400.0]</td>\n",
" <td>[4788.778928, 4788.966848, 4790.086088, 4790.8...</td>\n",
" <td>x y z imp location filterin...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>10</th>\n",
" <td>2088</td>\n",
" <td>[6.047871430730712, 6.375282052804018, 6.73703...</td>\n",
" <td>[5.405554112580891, 5.612333793682148, 5.96803...</td>\n",
" <td>24.444472</td>\n",
" <td>NaN</td>\n",
" <td>[98400.0]</td>\n",
" <td>[4838.037328, 4842.014248, 4843.459488, 4845.6...</td>\n",
" <td>x y z imp location filterin...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>11</th>\n",
" <td>2095</td>\n",
" <td>[-2.709146247949879, -2.925309328453019, -3.03...</td>\n",
" <td>[-1.9117917689484112, -2.0487637832506564, -2....</td>\n",
" <td>140.162756</td>\n",
" <td>1.684626</td>\n",
" <td>[98400.0]</td>\n",
" <td>[4806.942848, 4806.997728, 4812.365568, 4839.6...</td>\n",
" <td>x y z imp location filterin...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>12</th>\n",
" <td>2332</td>\n",
" <td>[2.823602517191, 2.969629137265953, 3.12642019...</td>\n",
" <td>[3.0355818059511357, 3.198584121564973, 3.3313...</td>\n",
" <td>78.762775</td>\n",
" <td>2.013060</td>\n",
" <td>[98400.0]</td>\n",
" <td>[4788.393248, 4790.086088, 4790.810608, 4796.5...</td>\n",
" <td>x y z imp location filterin...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>13</th>\n",
" <td>2357</td>\n",
" <td>[1.2274214830606864, 1.3016743035701275, 1.401...</td>\n",
" <td>[1.120015986951992, 1.2023151701844226, 1.2436...</td>\n",
" <td>36.746336</td>\n",
" <td>1.500293</td>\n",
" <td>[98400.0]</td>\n",
" <td>[4793.634568, 4795.696528, 4796.552928, 4800.8...</td>\n",
" <td>x y z imp location filterin...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>14</th>\n",
" <td>2498</td>\n",
" <td>[-2.824695043755286, -3.0307401988082607, -3.1...</td>\n",
" <td>[-3.258529426953644, -3.536159732250002, -3.68...</td>\n",
" <td>1667.376707</td>\n",
" <td>2.048690</td>\n",
" <td>[98400.0]</td>\n",
" <td>[4787.630928, 4788.266448, 4788.388648, 4788.3...</td>\n",
" <td>x y z imp location filterin...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>15</th>\n",
" <td>2585</td>\n",
" <td>[-0.327877833118743, -0.45845156443332435, -0....</td>\n",
" <td>[0.7860516060137199, 0.7340395697785383, 0.680...</td>\n",
" <td>14.371489</td>\n",
" <td>NaN</td>\n",
" <td>[98400.0]</td>\n",
" <td>[4790.892688, 4793.726568, 4803.596888, 4805.3...</td>\n",
" <td>x y z imp location filterin...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>16</th>\n",
" <td>2598</td>\n",
" <td>[2.1232642387501732, 2.270659390326574, 2.3742...</td>\n",
" <td>[1.8556815808671219, 2.0242523799238765, 2.115...</td>\n",
" <td>1279.536302</td>\n",
" <td>1.831915</td>\n",
" <td>[98400.0]</td>\n",
" <td>[4790.086088, 4790.790488, 4790.808008, 4791.7...</td>\n",
" <td>x y z imp location filterin...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>17</th>\n",
" <td>2742</td>\n",
" <td>[-1.2245068670748598, -1.4716205507666156, -1....</td>\n",
" <td>[-0.9695218513335611, -1.191220983233553, -1.3...</td>\n",
" <td>NaN</td>\n",
" <td>1.805190</td>\n",
" <td>[98400.0]</td>\n",
" <td>[4787.261848, 4787.301088, 4787.561608, 4787.5...</td>\n",
" <td>x y z imp location filte...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>18</th>\n",
" <td>1413</td>\n",
" <td>[-2.548643607790111, -2.548643607790111, -2.54...</td>\n",
" <td>[-2.5237935325968057, -2.5237935325968057, -2....</td>\n",
" <td>NaN</td>\n",
" <td>1.669595</td>\n",
" <td>[98400.0]</td>\n",
" <td>[4787.366161797764, 4787.3838466140805, 4787.3...</td>\n",
" <td>x y z imp location filte...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>19</th>\n",
" <td>1179</td>\n",
" <td>[-1.2289306492129852, -1.2289306492129852, -1....</td>\n",
" <td>[-0.8136424091326029, -0.8154535142928742, -0....</td>\n",
" <td>NaN</td>\n",
" <td>1.013530</td>\n",
" <td>[98400.0]</td>\n",
" <td>[4771.88223744342, 4787.139100252119, 4787.376...</td>\n",
" <td>x y z imp location filte...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>20</th>\n",
" <td>2647</td>\n",
" <td>[0.037290130212203614, 0.011616546233212323, 0...</td>\n",
" <td>[0.5767909527902486, 0.5489105365684017, 0.566...</td>\n",
" <td>NaN</td>\n",
" <td>1.100296</td>\n",
" <td>[98400.0]</td>\n",
" <td>[4788.837768, 4790.566488, 4790.579328, 4790.8...</td>\n",
" <td>x y z imp location filte...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>21</th>\n",
" <td>2468</td>\n",
" <td>[5.2659894782362695, 5.546721850839608, 5.8134...</td>\n",
" <td>[4.857542541911167, 5.147261569552395, 5.31181...</td>\n",
" <td>2967.764983</td>\n",
" <td>2.059100</td>\n",
" <td>[98400.0]</td>\n",
" <td>[4789.521248, 4790.402568, 4792.244968, 4792.5...</td>\n",
" <td>x y z imp location filtering...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>22</th>\n",
" <td>4603</td>\n",
" <td>[4.98627313334537, 5.284474845093591, 5.497782...</td>\n",
" <td>[4.736911863620169, 4.883046649181909, 5.13170...</td>\n",
" <td>140.995186</td>\n",
" <td>1.833047</td>\n",
" <td>[98400.0]</td>\n",
" <td>[4787.446848, 4790.402568, 4792.244968, 4792.5...</td>\n",
" <td>x y z imp location filtering...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>23</th>\n",
" <td>4648</td>\n",
" <td>[-4.734785929892845, -4.964300116472013, -5.10...</td>\n",
" <td>[-4.913822391802695, -5.129887713095916, -5.28...</td>\n",
" <td>1518.742743</td>\n",
" <td>1.750192</td>\n",
" <td>[98400.0]</td>\n",
" <td>[4788.802768, 4788.940968, 4789.501928, 4789.5...</td>\n",
" <td>x y z imp location filtering...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>24</th>\n",
" <td>4693</td>\n",
" <td>[1.323939497364863, 1.4111633753647694, 1.5467...</td>\n",
" <td>[1.8068525763401793, 2.051022355545685, 2.1776...</td>\n",
" <td>14.742652</td>\n",
" <td>1.260526</td>\n",
" <td>[98400.0]</td>\n",
" <td>[5891.409008, 5894.781968, 5904.204008, 5909.6...</td>\n",
" <td>x y z imp location filtering...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>25</th>\n",
" <td>4455</td>\n",
" <td>[-10.287331269184898, -11.208661911067304, -11...</td>\n",
" <td>[-7.315705368952566, -7.315705368952566, -7.64...</td>\n",
" <td>43.282241</td>\n",
" <td>NaN</td>\n",
" <td>[98400.0]</td>\n",
" <td>[4796.851008, 4834.599648, 4919.598208, 4922.9...</td>\n",
" <td>x y z imp location filtering...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>26</th>\n",
" <td>4961</td>\n",
" <td>[5.59877152176297, 5.841501743545079, 6.095332...</td>\n",
" <td>[5.667234047705148, 5.917575835001948, 6.11893...</td>\n",
" <td>24.832714</td>\n",
" <td>1.994933</td>\n",
" <td>[98400.0]</td>\n",
" <td>[4787.446888, 4790.111968, 4790.402568, 4792.2...</td>\n",
" <td>x y z imp location filtering...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>27</th>\n",
" <td>4528</td>\n",
" <td>[-5.16416672053587, -5.418178478304996, -5.594...</td>\n",
" <td>[-5.457686005836013, -5.7104250391377915, -5.9...</td>\n",
" <td>1841.242825</td>\n",
" <td>2.000195</td>\n",
" <td>[98400.0]</td>\n",
" <td>[4787.691448, 4787.960208, 4788.980208, 4789.8...</td>\n",
" <td>x y z imp location filtering...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>28</th>\n",
" <td>4564</td>\n",
" <td>[4.703844644646429, 4.937927746210565, 5.08580...</td>\n",
" <td>[4.6658010728239425, 4.892389336217129, 5.0866...</td>\n",
" <td>945.061969</td>\n",
" <td>1.932703</td>\n",
" <td>[98400.0]</td>\n",
" <td>[4787.446888, 4789.521288, 4790.110208, 4790.1...</td>\n",
" <td>x y z imp location filtering...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>29</th>\n",
" <td>5823</td>\n",
" <td>[5.728572791425566, 5.99580198453578, 6.261548...</td>\n",
" <td>[5.587791510078878, 5.926904930783165, 6.16167...</td>\n",
" <td>15.561102</td>\n",
" <td>2.261887</td>\n",
" <td>[98400.0]</td>\n",
" <td>[4820.443688, 4830.186608, 4908.169368, 4929.4...</td>\n",
" <td>x y z imp location filtering...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>30</th>\n",
" <td>5894</td>\n",
" <td>[-4.73077804512256, -4.898476202058618, -5.008...</td>\n",
" <td>[-4.862234492245103, -5.061440746978259, -5.18...</td>\n",
" <td>1030.742235</td>\n",
" <td>1.508859</td>\n",
" <td>[98400.0]</td>\n",
" <td>[4788.001408, 4788.331088, 4788.551088, 4789.1...</td>\n",
" <td>x y z imp location filtering...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>31</th>\n",
" <td>5909</td>\n",
" <td>[-0.5586610466677955, -0.7760079182327165, -0....</td>\n",
" <td>[-1.3264553518120181, -1.587127146104324, -1.6...</td>\n",
" <td>13.758655</td>\n",
" <td>1.093016</td>\n",
" <td>[98400.0]</td>\n",
" <td>[4792.823328, 4796.369128, 4797.807928, 4800.7...</td>\n",
" <td>x y z imp location filtering...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>32</th>\n",
" <td>5931</td>\n",
" <td>[0.14249181583647802, 0.3429984839918393, 0.44...</td>\n",
" <td>[-0.3719621633119015, -0.14229420338492527, -0...</td>\n",
" <td>101.993007</td>\n",
" <td>1.244882</td>\n",
" <td>[98400.0]</td>\n",
" <td>[4787.231848, 4787.324768, 4787.432528, 4787.5...</td>\n",
" <td>x y z imp location filtering...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>33</th>\n",
" <td>5443</td>\n",
" <td>[4.630259191385897, 4.845414865946165, 5.05708...</td>\n",
" <td>[4.907337744643092, 5.138267903502391, 5.39023...</td>\n",
" <td>977.853119</td>\n",
" <td>1.732713</td>\n",
" <td>[98400.0]</td>\n",
" <td>[4789.521288, 4790.402568, 4792.244968, 4792.5...</td>\n",
" <td>x y z imp location filtering...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>34</th>\n",
" <td>5457</td>\n",
" <td>[-4.366063858693557, -4.61496903182344, -4.753...</td>\n",
" <td>[-4.749469479795696, -4.945626544066499, -5.10...</td>\n",
" <td>1697.177504</td>\n",
" <td>1.655575</td>\n",
" <td>[98400.0]</td>\n",
" <td>[4788.338248, 4789.999368, 4790.423408, 4790.8...</td>\n",
" <td>x y z imp location filtering...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>35</th>\n",
" <td>5459</td>\n",
" <td>[-0.01842692210679583, 0.24097175324993847, 0....</td>\n",
" <td>[-0.48625057679849604, -0.2603314739988116, -0...</td>\n",
" <td>113.440252</td>\n",
" <td>1.295803</td>\n",
" <td>[98400.0]</td>\n",
" <td>[4793.288528, 4793.484768, 4797.003048, 4798.6...</td>\n",
" <td>x y z imp location filtering...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>36</th>\n",
" <td>3846</td>\n",
" <td>[-11.921466457155018, -12.357194465282975, -12...</td>\n",
" <td>[-11.558615043019364, -11.899498264858675, -12...</td>\n",
" <td>19.366741</td>\n",
" <td>3.232238</td>\n",
" <td>[98400.0]</td>\n",
" <td>[4788.976128, 4790.940448, 4792.035488, 4795.1...</td>\n",
" <td>x y z imp location filtering...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>37</th>\n",
" <td>3848</td>\n",
" <td>[-18.184675130569318, -18.917115701981917, -19...</td>\n",
" <td>[-19.78105764303847, -20.55885124442189, -21.3...</td>\n",
" <td>1306.074987</td>\n",
" <td>5.579544</td>\n",
" <td>[98400.0]</td>\n",
" <td>[4787.666088, 4788.275648, 4788.753248, 4789.2...</td>\n",
" <td>x y z imp location filtering...</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" origClusterID waveform_mean_encoding \\\n",
"id \n",
"0 970 [2.1764968283053565, 2.3406132503530475, 2.453... \n",
"1 2519 [1.9953024730819329, 2.146922925948179, 2.2750... \n",
"2 2569 [-1.0892165502550855, -1.194207382098711, -1.2... \n",
"3 2808 [3.9792519048004746, 4.225281427560096, 4.4375... \n",
"4 3048 [-7.368623732940135, -7.666343704498384, -7.93... \n",
"5 4126 [2.3131939714515104, 2.49287475093624, 2.63675... \n",
"6 4245 [-5.777442084370658, -6.052566885779441, -6.26... \n",
"7 4329 [-1.9242575810453464, -2.0852046175911347, -2.... \n",
"8 2498 [-2.4713541945868465, -2.6218310636669306, -2.... \n",
"9 2530 [1.8563672622890155, 1.9997108346169516, 2.120... \n",
"10 2088 [6.047871430730712, 6.375282052804018, 6.73703... \n",
"11 2095 [-2.709146247949879, -2.925309328453019, -3.03... \n",
"12 2332 [2.823602517191, 2.969629137265953, 3.12642019... \n",
"13 2357 [1.2274214830606864, 1.3016743035701275, 1.401... \n",
"14 2498 [-2.824695043755286, -3.0307401988082607, -3.1... \n",
"15 2585 [-0.327877833118743, -0.45845156443332435, -0.... \n",
"16 2598 [2.1232642387501732, 2.270659390326574, 2.3742... \n",
"17 2742 [-1.2245068670748598, -1.4716205507666156, -1.... \n",
"18 1413 [-2.548643607790111, -2.548643607790111, -2.54... \n",
"19 1179 [-1.2289306492129852, -1.2289306492129852, -1.... \n",
"20 2647 [0.037290130212203614, 0.011616546233212323, 0... \n",
"21 2468 [5.2659894782362695, 5.546721850839608, 5.8134... \n",
"22 4603 [4.98627313334537, 5.284474845093591, 5.497782... \n",
"23 4648 [-4.734785929892845, -4.964300116472013, -5.10... \n",
"24 4693 [1.323939497364863, 1.4111633753647694, 1.5467... \n",
"25 4455 [-10.287331269184898, -11.208661911067304, -11... \n",
"26 4961 [5.59877152176297, 5.841501743545079, 6.095332... \n",
"27 4528 [-5.16416672053587, -5.418178478304996, -5.594... \n",
"28 4564 [4.703844644646429, 4.937927746210565, 5.08580... \n",
"29 5823 [5.728572791425566, 5.99580198453578, 6.261548... \n",
"30 5894 [-4.73077804512256, -4.898476202058618, -5.008... \n",
"31 5909 [-0.5586610466677955, -0.7760079182327165, -0.... \n",
"32 5931 [0.14249181583647802, 0.3429984839918393, 0.44... \n",
"33 5443 [4.630259191385897, 4.845414865946165, 5.05708... \n",
"34 5457 [-4.366063858693557, -4.61496903182344, -4.753... \n",
"35 5459 [-0.01842692210679583, 0.24097175324993847, 0.... \n",
"36 3846 [-11.921466457155018, -12.357194465282975, -12... \n",
"37 3848 [-18.184675130569318, -18.917115701981917, -19... \n",
"\n",
" waveform_mean_recognition IsolationDist \\\n",
"id \n",
"0 [2.255677148872515, 2.438650365560623, 2.54682... 54.949626 \n",
"1 [1.8679882172094504, 2.0195327093193445, 2.149... 1301.944632 \n",
"2 [-1.2739211957333492, -1.3850635003577056, -1.... 2700.668258 \n",
"3 [4.554116369553418, 4.791653736023812, 4.99612... 2798.328329 \n",
"4 [-7.309179748134245, -7.610296381602412, -7.85... NaN \n",
"5 [2.2916029270428444, 2.4356759764683518, 2.549... 3209.140751 \n",
"6 [-6.394055601868903, -6.6383642870196224, -6.8... 25.765571 \n",
"7 [-1.6547120693159738, -1.782868893659467, -1.8... 116.662306 \n",
"8 [-2.5624289166817404, -2.749951452957572, -2.8... 1016.161571 \n",
"9 [1.9445634547101074, 2.0918185669478353, 2.199... 780.642771 \n",
"10 [5.405554112580891, 5.612333793682148, 5.96803... 24.444472 \n",
"11 [-1.9117917689484112, -2.0487637832506564, -2.... 140.162756 \n",
"12 [3.0355818059511357, 3.198584121564973, 3.3313... 78.762775 \n",
"13 [1.120015986951992, 1.2023151701844226, 1.2436... 36.746336 \n",
"14 [-3.258529426953644, -3.536159732250002, -3.68... 1667.376707 \n",
"15 [0.7860516060137199, 0.7340395697785383, 0.680... 14.371489 \n",
"16 [1.8556815808671219, 2.0242523799238765, 2.115... 1279.536302 \n",
"17 [-0.9695218513335611, -1.191220983233553, -1.3... NaN \n",
"18 [-2.5237935325968057, -2.5237935325968057, -2.... NaN \n",
"19 [-0.8136424091326029, -0.8154535142928742, -0.... NaN \n",
"20 [0.5767909527902486, 0.5489105365684017, 0.566... NaN \n",
"21 [4.857542541911167, 5.147261569552395, 5.31181... 2967.764983 \n",
"22 [4.736911863620169, 4.883046649181909, 5.13170... 140.995186 \n",
"23 [-4.913822391802695, -5.129887713095916, -5.28... 1518.742743 \n",
"24 [1.8068525763401793, 2.051022355545685, 2.1776... 14.742652 \n",
"25 [-7.315705368952566, -7.315705368952566, -7.64... 43.282241 \n",
"26 [5.667234047705148, 5.917575835001948, 6.11893... 24.832714 \n",
"27 [-5.457686005836013, -5.7104250391377915, -5.9... 1841.242825 \n",
"28 [4.6658010728239425, 4.892389336217129, 5.0866... 945.061969 \n",
"29 [5.587791510078878, 5.926904930783165, 6.16167... 15.561102 \n",
"30 [-4.862234492245103, -5.061440746978259, -5.18... 1030.742235 \n",
"31 [-1.3264553518120181, -1.587127146104324, -1.6... 13.758655 \n",
"32 [-0.3719621633119015, -0.14229420338492527, -0... 101.993007 \n",
"33 [4.907337744643092, 5.138267903502391, 5.39023... 977.853119 \n",
"34 [-4.749469479795696, -4.945626544066499, -5.10... 1697.177504 \n",
"35 [-0.48625057679849604, -0.2603314739988116, -0... 113.440252 \n",
"36 [-11.558615043019364, -11.899498264858675, -12... 19.366741 \n",
"37 [-19.78105764303847, -20.55885124442189, -21.3... 1306.074987 \n",
"\n",
" SNR waveform_mean_sampling_rate \\\n",
"id \n",
"0 2.074611 [98400.0] \n",
"1 1.535590 [98400.0] \n",
"2 1.721514 [98400.0] \n",
"3 2.428861 [98400.0] \n",
"4 3.924017 [98400.0] \n",
"5 1.817625 [98400.0] \n",
"6 3.297644 [98400.0] \n",
"7 1.637649 [98400.0] \n",
"8 1.558819 [98400.0] \n",
"9 1.618448 [98400.0] \n",
"10 NaN [98400.0] \n",
"11 1.684626 [98400.0] \n",
"12 2.013060 [98400.0] \n",
"13 1.500293 [98400.0] \n",
"14 2.048690 [98400.0] \n",
"15 NaN [98400.0] \n",
"16 1.831915 [98400.0] \n",
"17 1.805190 [98400.0] \n",
"18 1.669595 [98400.0] \n",
"19 1.013530 [98400.0] \n",
"20 1.100296 [98400.0] \n",
"21 2.059100 [98400.0] \n",
"22 1.833047 [98400.0] \n",
"23 1.750192 [98400.0] \n",
"24 1.260526 [98400.0] \n",
"25 NaN [98400.0] \n",
"26 1.994933 [98400.0] \n",
"27 2.000195 [98400.0] \n",
"28 1.932703 [98400.0] \n",
"29 2.261887 [98400.0] \n",
"30 1.508859 [98400.0] \n",
"31 1.093016 [98400.0] \n",
"32 1.244882 [98400.0] \n",
"33 1.732713 [98400.0] \n",
"34 1.655575 [98400.0] \n",
"35 1.295803 [98400.0] \n",
"36 3.232238 [98400.0] \n",
"37 5.579544 [98400.0] \n",
"\n",
" spike_times \\\n",
"id \n",
"0 [4790.086128, 4790.392488, 4790.808088, 4791.1... \n",
"1 [4789.163768, 4789.171688, 4790.086128, 4790.7... \n",
"2 [4787.484848, 4787.756208, 4788.171848, 4788.3... \n",
"3 [4790.086128, 4796.592248, 4797.417168, 4798.8... \n",
"4 [4787.674888, 4787.861608, 4788.101208, 4788.2... \n",
"5 [4790.086088, 4794.246448, 4797.826848, 4798.8... \n",
"6 [4787.630928, 4788.213568, 4788.266488, 4790.6... \n",
"7 [4788.378728, 4788.789968, 4791.626488, 4791.6... \n",
"8 [4789.165328, 4789.588288, 4790.794608, 4790.8... \n",
"9 [4788.778928, 4788.966848, 4790.086088, 4790.8... \n",
"10 [4838.037328, 4842.014248, 4843.459488, 4845.6... \n",
"11 [4806.942848, 4806.997728, 4812.365568, 4839.6... \n",
"12 [4788.393248, 4790.086088, 4790.810608, 4796.5... \n",
"13 [4793.634568, 4795.696528, 4796.552928, 4800.8... \n",
"14 [4787.630928, 4788.266448, 4788.388648, 4788.3... \n",
"15 [4790.892688, 4793.726568, 4803.596888, 4805.3... \n",
"16 [4790.086088, 4790.790488, 4790.808008, 4791.7... \n",
"17 [4787.261848, 4787.301088, 4787.561608, 4787.5... \n",
"18 [4787.366161797764, 4787.3838466140805, 4787.3... \n",
"19 [4771.88223744342, 4787.139100252119, 4787.376... \n",
"20 [4788.837768, 4790.566488, 4790.579328, 4790.8... \n",
"21 [4789.521248, 4790.402568, 4792.244968, 4792.5... \n",
"22 [4787.446848, 4790.402568, 4792.244968, 4792.5... \n",
"23 [4788.802768, 4788.940968, 4789.501928, 4789.5... \n",
"24 [5891.409008, 5894.781968, 5904.204008, 5909.6... \n",
"25 [4796.851008, 4834.599648, 4919.598208, 4922.9... \n",
"26 [4787.446888, 4790.111968, 4790.402568, 4792.2... \n",
"27 [4787.691448, 4787.960208, 4788.980208, 4789.8... \n",
"28 [4787.446888, 4789.521288, 4790.110208, 4790.1... \n",
"29 [4820.443688, 4830.186608, 4908.169368, 4929.4... \n",
"30 [4788.001408, 4788.331088, 4788.551088, 4789.1... \n",
"31 [4792.823328, 4796.369128, 4797.807928, 4800.7... \n",
"32 [4787.231848, 4787.324768, 4787.432528, 4787.5... \n",
"33 [4789.521288, 4790.402568, 4792.244968, 4792.5... \n",
"34 [4788.338248, 4789.999368, 4790.423408, 4790.8... \n",
"35 [4793.288528, 4793.484768, 4797.003048, 4798.6... \n",
"36 [4788.976128, 4790.940448, 4792.035488, 4795.1... \n",
"37 [4787.666088, 4788.275648, 4788.753248, 4789.2... \n",
"\n",
" electrodes \n",
"id \n",
"0 x y z imp location filterin... \n",
"1 x y z imp location filterin... \n",
"2 x y z imp location filterin... \n",
"3 x y z imp location filterin... \n",
"4 x y z imp location filterin... \n",
"5 x y z imp location filterin... \n",
"6 x y z imp location filterin... \n",
"7 x y z imp location filterin... \n",
"8 x y z imp location filterin... \n",
"9 x y z imp location filterin... \n",
"10 x y z imp location filterin... \n",
"11 x y z imp location filterin... \n",
"12 x y z imp location filterin... \n",
"13 x y z imp location filterin... \n",
"14 x y z imp location filterin... \n",
"15 x y z imp location filterin... \n",
"16 x y z imp location filterin... \n",
"17 x y z imp location filte... \n",
"18 x y z imp location filte... \n",
"19 x y z imp location filte... \n",
"20 x y z imp location filte... \n",
"21 x y z imp location filtering... \n",
"22 x y z imp location filtering... \n",
"23 x y z imp location filtering... \n",
"24 x y z imp location filtering... \n",
"25 x y z imp location filtering... \n",
"26 x y z imp location filtering... \n",
"27 x y z imp location filtering... \n",
"28 x y z imp location filtering... \n",
"29 x y z imp location filtering... \n",
"30 x y z imp location filtering... \n",
"31 x y z imp location filtering... \n",
"32 x y z imp location filtering... \n",
"33 x y z imp location filtering... \n",
"34 x y z imp location filtering... \n",
"35 x y z imp location filtering... \n",
"36 x y z imp location filtering... \n",
"37 x y z imp location filtering... "
]
},
"execution_count": 20,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"nwbfile.units.to_dataframe()"
]
},
{
"cell_type": "code",
"execution_count": 21,
"metadata": {},
"outputs": [],
"source": [
"# unlink the Units table from its parent container, NWBFile\n",
"nwbfile.units.reset_parent()\n",
"\n",
"# remove the Units table from the NWBFile fields\n",
"# this is a temporary hack to get around the fact that nwbfile.units is a special attribute of NWBFile\n",
"nwbfile.fields[\"units\"] = None\n",
"\n",
"# now, NWBFile.units is None\n",
"nwbfile.units"
]
},
{
"cell_type": "code",
"execution_count": 22,
"metadata": {},
"outputs": [],
"source": [
"# finally, create a new IO object for exporting the modified in-memory NWBFile object\n",
"# to a new file path\n",
"export_path = \"./trimmed_nwbfile.nwb\"\n",
"with NWBHDF5IO(export_path, mode=\"w\") as export_io:\n",
" export_io.export(src_io=read_io, nwbfile=nwbfile)\n",
"\n",
"read_io.close()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Let's verify that our new file is valid and does not have a `Units` table."
]
},
{
"cell_type": "code",
"execution_count": 23,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Validation errors: []\n",
"None\n"
]
}
],
"source": [
"from pynwb import validate\n",
"\n",
"with NWBHDF5IO(export_path, mode=\"r\") as read_export_io:\n",
" errors = validate(read_export_io)\n",
" print(\"Validation errors:\", errors) # this should be an empty list\n",
"\n",
" read_nwbfile = read_export_io.read()\n",
" print(read_nwbfile.units)\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "test",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.11.7"
}
},
"nbformat": 4,
"nbformat_minor": 2
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment