Skip to content

Instantly share code, notes, and snippets.

@kmuehlbauer

kmuehlbauer/complex.ipynb

Created January 20, 2024 18:59
Show Gist options
  • Save kmuehlbauer/1f6f8a61ad5b0a7693ae08154060b790 to your computer and use it in GitHub Desktop.
Save kmuehlbauer/1f6f8a61ad5b0a7693ae08154060b790 to your computer and use it in GitHub Desktop.
Download ZIP
Complex Data in netcdf4-python and h5netcdf
Display the source blob
Display the rendered blob
Raw
complex.ipynb
{
"cells": [
{
"cell_type": "code",
"execution_count": 1,
"id": "a7211de3-55ed-4928-b48d-f3e97d91ca0d",
"metadata": {},
"outputs": [],
"source": [
"import netCDF4 as nc\n",
"import numpy as np\n",
"import h5netcdf.legacyapi as h5nc"
]
},
{
"cell_type": "markdown",
"id": "ea0783e6-23c2-458b-8857-725ecb2e728e",
"metadata": {},
"source": [
"# 1. Complex Type in NetCDF4 files"
]
},
{
"cell_type": "markdown",
"id": "1b7bc1d0-a82b-40f3-8231-451a4d7a2872",
"metadata": {},
"source": [
"# 1.1 netcdf4-python"
]
},
{
"cell_type": "markdown",
"id": "4be979b4-1ac7-4714-83aa-db90899db82c",
"metadata": {},
"source": [
"## 1.1.1 Current approach as outline in netcdf4-python docs"
]
},
{
"cell_type": "markdown",
"id": "a854850e-7f7f-4181-a06b-807268a9cc33",
"metadata": {},
"source": [
"### create file with complex compound type"
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "46abe9f0-881c-4f0f-9d30-c56c87d628a3",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"{'names': ['r', 'i'], 'formats': ['<f8', '<f8'], 'offsets': [0, 8], 'itemsize': 16, 'aligned': True}: <class 'netCDF4._netCDF4.Variable'>\n",
"compound data(x)\n",
"compound data type: {'names': ['r', 'i'], 'formats': ['<f8', '<f8'], 'offsets': [0, 8], 'itemsize': 16, 'aligned': True}\n",
"unlimited dimensions: x\n",
"current shape = (3,)\n"
]
}
],
"source": [
"with nc.Dataset(\"complex_nc.nc\", \"w\") as ds:\n",
" size = 3 # length of 1-d complex array\n",
" # create sample complex data.\n",
" complex_array = np.array([0 + 0j, 1 + 0j, 0 + 1j])\n",
" # create complex128 compound data type.\n",
" complex128 = np.dtype([(\"r\",np.float64),(\"i\",np.float64)])\n",
" complex128_t = ds.createCompoundType(complex128, \"complex128_t\")\n",
" # create a variable with this data type, write some data to it.\n",
" x_dim = ds.createDimension(\"x\",None)\n",
" v = ds.createVariable(\"data\",complex128_t,\"x\")\n",
" data = np.empty(size,complex128) # numpy structured array\n",
" data[\"r\"] = complex_array.real; data[\"i\"] = complex_array.imag\n",
" v[:] = data # write numpy structured array to netcdf compound var\n",
" print('{}: {}'.format(v.dtype, v))"
]
},
{
"cell_type": "markdown",
"id": "502fb07a-fd23-4f7b-be64-33e2ddf4a0e7",
"metadata": {},
"source": [
"### reopen and convert to complex data again"
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "c05c32ce-1f3c-44a0-a1ab-aa7a0dab6011",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"original : complex128: [0.+0.j 1.+0.j 0.+1.j]\n",
"actual : {'names': ['r', 'i'], 'formats': ['<f8', '<f8'], 'offsets': [0, 8], 'itemsize': 16, 'aligned': True}: [(0., 0.) (1., 0.) (0., 1.)]\n",
"recreated: complex128: [0.+0.j 1.+0.j 0.+1.j]\n"
]
}
],
"source": [
"with nc.Dataset(\"complex_nc.nc\", \"r\") as ds:\n",
" v = ds.variables[\"data\"]\n",
" datain = v[:] # read in all the data into a numpy structured array\n",
" # create an empty numpy complex array\n",
" datac2 = np.empty(datain.shape,np.complex128)\n",
" # .. fill it with contents of structured array.\n",
" datac2.real = datain[\"r\"]; datac2.imag = datain[\"i\"]\n",
" print('original : {}: {}'.format(complex_array.dtype, complex_array)) # original data\n",
" print('actual : {}: {}'.format(datain.dtype, datain)) # actual data\n",
" print('recreated: {}: {}'.format(datac2.dtype, datac2)) # reconverted data"
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "9bb10f6e-a15f-4473-8f08-9ed44e514bce",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"original: complex128 - [0.+0.j 1.+0.j 0.+1.j]\n",
"actual : {'names': ['r', 'i'], 'formats': ['<f8', '<f8'], 'offsets': [0, 8], 'itemsize': 16, 'aligned': True} - [(0., 0.) (1., 0.) (0., 1.)]\n"
]
}
],
"source": [
"with nc.Dataset(\"complex_nc.nc\", \"r\") as ds:\n",
" datac2 = ds[\"data\"]\n",
" print('original: {} - {}'.format(complex_array.dtype, complex_array)) # original data\n",
" print('actual : {} - {}'.format(datac2.dtype, datac2[:])) # reread data"
]
},
{
"cell_type": "markdown",
"id": "e87b8df4-1961-4098-a4b2-11b7a7ee7439",
"metadata": {},
"source": [
"### Read with auto_complex = True"
]
},
{
"cell_type": "code",
"execution_count": 5,
"id": "7fa19b2d-6d39-4660-bb58-a477d52b0909",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"original: complex128 - [0.+0.j 1.+0.j 0.+1.j]\n",
"actual : complex128 - [0.+0.j 1.+0.j 0.+1.j]\n"
]
}
],
"source": [
"with nc.Dataset(\"complex_nc.nc\", \"r\", auto_complex=True) as ds:\n",
" datac2 = ds[\"data\"]\n",
" print('original: {} - {}'.format(complex_array.dtype, complex_array)) # original data\n",
" print('actual : {} - {}'.format(datac2.dtype, datac2[:])) # reread data"
]
},
{
"cell_type": "markdown",
"id": "af5b0fc4-8194-45fd-92f0-62cd772d71a3",
"metadata": {},
"source": [
"### Read with h5netcdf"
]
},
{
"cell_type": "code",
"execution_count": 6,
"id": "a5afae36-cf1b-478b-842a-f20bc0fd9b4a",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"original: complex128 - [0.+0.j 1.+0.j 0.+1.j]\n",
"actual : complex128 - [0.+0.j 1.+0.j 0.+1.j]\n"
]
}
],
"source": [
"with h5nc.Dataset(\"complex_nc.nc\", \"r\") as ds:\n",
" datac2 = ds[\"data\"]\n",
" print('original: {} - {}'.format(complex_array.dtype, complex_array)) # original data\n",
" print('actual : {} - {}'.format(datac2.dtype, datac2[:])) # reread data"
]
},
{
"cell_type": "code",
"execution_count": 7,
"id": "dae3c08f-63c0-4066-94ef-3f0a9c8d9647",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"netcdf complex_nc {\n",
"types:\n",
" compound complex128_t {\n",
" double r ;\n",
" double i ;\n",
" }; // complex128_t\n",
"dimensions:\n",
"\tx = UNLIMITED ; // (3 currently)\n",
"variables:\n",
"\tcomplex128_t data(x) ;\n",
"data:\n",
"\n",
" data = {0, 0}, {1, 0}, {0, 1} ;\n",
"}\n"
]
}
],
"source": [
"!ncdump complex_nc.nc"
]
},
{
"cell_type": "code",
"execution_count": 8,
"id": "066caf45-2019-4794-83a1-5bacb58f6f68",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"HDF5 \"complex_nc.nc\" {\n",
"GROUP \"/\" {\n",
" ATTRIBUTE \"_NCProperties\" {\n",
" DATATYPE H5T_STRING {\n",
" STRSIZE 34;\n",
" STRPAD H5T_STR_NULLTERM;\n",
" CSET H5T_CSET_ASCII;\n",
" CTYPE H5T_C_S1;\n",
" }\n",
" DATASPACE SCALAR\n",
" DATA {\n",
" (0): \"version=2,netcdf=4.9.2,hdf5=1.14.3\"\n",
" }\n",
" }\n",
" DATATYPE \"complex128_t\" H5T_COMPOUND {\n",
" H5T_IEEE_F64LE \"r\";\n",
" H5T_IEEE_F64LE \"i\";\n",
" }\n",
" DATASET \"data\" {\n",
" DATATYPE H5T_COMPOUND {\n",
" H5T_IEEE_F64LE \"r\";\n",
" H5T_IEEE_F64LE \"i\";\n",
" }\n",
" DATASPACE SIMPLE { ( 3 ) / ( H5S_UNLIMITED ) }\n",
" DATA {\n",
" (0): {\n",
" 0,\n",
" 0\n",
" },\n",
" (1): {\n",
" 1,\n",
" 0\n",
" },\n",
" (2): {\n",
" 0,\n",
" 1\n",
" }\n",
" }\n",
" ATTRIBUTE \"DIMENSION_LIST\" {\n",
" DATATYPE H5T_VLEN { H5T_REFERENCE { H5T_STD_REF_OBJECT } }\n",
" DATASPACE SIMPLE { ( 1 ) / ( 1 ) }\n",
" DATA {\n",
" (0): (DATASET 93888225317648 \"/x\")\n",
" }\n",
" }\n",
" ATTRIBUTE \"_Netcdf4Coordinates\" {\n",
" DATATYPE H5T_STD_I32LE\n",
" DATASPACE SIMPLE { ( 1 ) / ( 1 ) }\n",
" DATA {\n",
" (0): 0\n",
" }\n",
" }\n",
" }\n",
" DATASET \"x\" {\n",
" DATATYPE H5T_IEEE_F32BE\n",
" DATASPACE SIMPLE { ( 0 ) / ( H5S_UNLIMITED ) }\n",
" DATA {\n",
" }\n",
" ATTRIBUTE \"CLASS\" {\n",
" DATATYPE H5T_STRING {\n",
" STRSIZE 16;\n",
" STRPAD H5T_STR_NULLTERM;\n",
" CSET H5T_CSET_ASCII;\n",
" CTYPE H5T_C_S1;\n",
" }\n",
" DATASPACE SCALAR\n",
" DATA {\n",
" (0): \"DIMENSION_SCALE\"\n",
" }\n",
" }\n",
" ATTRIBUTE \"NAME\" {\n",
" DATATYPE H5T_STRING {\n",
" STRSIZE 64;\n",
" STRPAD H5T_STR_NULLTERM;\n",
" CSET H5T_CSET_ASCII;\n",
" CTYPE H5T_C_S1;\n",
" }\n",
" DATASPACE SCALAR\n",
" DATA {\n",
" (0): \"This is a netCDF dimension but not a netCDF variable. 0\"\n",
" }\n",
" }\n",
" ATTRIBUTE \"REFERENCE_LIST\" {\n",
" DATATYPE H5T_COMPOUND {\n",
" H5T_REFERENCE { H5T_STD_REF_OBJECT } \"dataset\";\n",
" H5T_STD_U32LE \"dimension\";\n",
" }\n",
" DATASPACE SIMPLE { ( 1 ) / ( 1 ) }\n",
" DATA {\n",
" (0): {\n",
" DATASET 93888225252304 \"/data\",\n",
" 0\n",
" }\n",
" }\n",
" }\n",
" ATTRIBUTE \"_Netcdf4Dimid\" {\n",
" DATATYPE H5T_STD_I32LE\n",
" DATASPACE SCALAR\n",
" DATA {\n",
" (0): 0\n",
" }\n",
" }\n",
" }\n",
"}\n",
"}\n"
]
}
],
"source": [
"!h5dump complex_nc.nc"
]
},
{
"cell_type": "markdown",
"id": "f990cdd0-19ab-4e1c-8300-e797acedee68",
"metadata": {},
"source": [
"## 1.1.2 Create with auto_complex as written in netcdf4-python docs"
]
},
{
"cell_type": "code",
"execution_count": 9,
"id": "f9fa45bf-2bcf-4f52-9a35-44e0b8fafb46",
"metadata": {},
"outputs": [],
"source": [
"with nc.Dataset(\"complex_auto.nc\", \"w\", auto_complex=True) as ds:\n",
" ds.createDimension(\"x\", size=len(complex_array))\n",
" var = ds.createVariable(\"data\", \"c16\", (\"x\",))\n",
" var[:] = complex_array"
]
},
{
"cell_type": "markdown",
"id": "463286a3-c1ec-4ea0-8afe-2d2e030905c3",
"metadata": {},
"source": [
"### reopen and convert to complex data again"
]
},
{
"cell_type": "code",
"execution_count": 10,
"id": "9402dc4e-c8cf-40ec-a994-15a054639323",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"original : complex128: [0.+0.j 1.+0.j 0.+1.j]\n",
"actual : {'names': ['r', 'i'], 'formats': ['<f8', '<f8'], 'offsets': [0, 8], 'itemsize': 16, 'aligned': True}: [(0., 0.) (1., 0.) (0., 1.)]\n",
"recreated: complex128: [0.+0.j 1.+0.j 0.+1.j]\n"
]
}
],
"source": [
"with nc.Dataset(\"complex_auto.nc\", \"r\") as ds:\n",
" v = ds.variables[\"data\"]\n",
" datain = v[:] # read in all the data into a numpy structured array\n",
" # create an empty numpy complex array\n",
" datac2 = np.empty(datain.shape,np.complex128)\n",
" # .. fill it with contents of structured array.\n",
" datac2.real = datain[\"r\"]; datac2.imag = datain[\"i\"]\n",
" print('original : {}: {}'.format(complex_array.dtype, complex_array)) # original data\n",
" print('actual : {}: {}'.format(datain.dtype, datain)) # actual data\n",
" print('recreated: {}: {}'.format(datac2.dtype, datac2)) # reconverted data"
]
},
{
"cell_type": "code",
"execution_count": 11,
"id": "e0ac1e1e-4ad3-4c7d-9447-81251705058e",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"original: complex128 - [0.+0.j 1.+0.j 0.+1.j]\n",
"actual : {'names': ['r', 'i'], 'formats': ['<f8', '<f8'], 'offsets': [0, 8], 'itemsize': 16, 'aligned': True} - [(0., 0.) (1., 0.) (0., 1.)]\n"
]
}
],
"source": [
"with nc.Dataset(\"complex_auto.nc\", \"r\") as ds:\n",
" datac2 = ds[\"data\"]\n",
" print('original: {} - {}'.format(complex_array.dtype, complex_array)) # original data\n",
" print('actual : {} - {}'.format(datac2.dtype, datac2[:])) # reread data"
]
},
{
"cell_type": "markdown",
"id": "afc003cd-138f-4d29-9586-53c25d40804e",
"metadata": {},
"source": [
"### Read with auto_complex = True"
]
},
{
"cell_type": "code",
"execution_count": 12,
"id": "a8dd615c-fdfa-4872-8a4f-8f97b5016aff",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"original: complex128 - [0.+0.j 1.+0.j 0.+1.j]\n",
"actual : complex128 - [0.+0.j 1.+0.j 0.+1.j]\n"
]
}
],
"source": [
"with nc.Dataset(\"complex_auto.nc\", \"r\", auto_complex=True) as ds:\n",
" datac2 = ds[\"data\"]\n",
" print('original: {} - {}'.format(complex_array.dtype, complex_array)) # original data\n",
" print('actual : {} - {}'.format(datac2.dtype, datac2[:])) # reread data"
]
},
{
"cell_type": "markdown",
"id": "fdfebe7c-5643-47c2-b119-209c1ab7e0ea",
"metadata": {},
"source": [
"### Read with h5netcdf"
]
},
{
"cell_type": "code",
"execution_count": 13,
"id": "b8c14b38-3d78-4ff4-b0d2-41f745246336",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"original: complex128 - [0.+0.j 1.+0.j 0.+1.j]\n",
"actual : complex128 - [0.+0.j 1.+0.j 0.+1.j]\n"
]
}
],
"source": [
"with h5nc.Dataset(\"complex_auto.nc\", \"r\") as ds:\n",
" datac2 = ds[\"data\"]\n",
" print('original: {} - {}'.format(complex_array.dtype, complex_array)) # original data\n",
" print('actual : {} - {}'.format(datac2.dtype, datac2[:])) # reread data"
]
},
{
"cell_type": "code",
"execution_count": 14,
"id": "0fe01e88-2cc3-4200-8abf-3d6538e3d302",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"netcdf complex_auto {\n",
"types:\n",
" compound _PFNC_DOUBLE_COMPLEX_TYPE {\n",
" double r ;\n",
" double i ;\n",
" }; // _PFNC_DOUBLE_COMPLEX_TYPE\n",
"dimensions:\n",
"\tx = 3 ;\n",
"variables:\n",
"\t_PFNC_DOUBLE_COMPLEX_TYPE data(x) ;\n",
"data:\n",
"\n",
" data = {0, 0}, {1, 0}, {0, 1} ;\n",
"}\n"
]
}
],
"source": [
"!ncdump complex_auto.nc"
]
},
{
"cell_type": "code",
"execution_count": 15,
"id": "8e0090b6-557b-46e7-94f2-237c37120129",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"HDF5 \"complex_auto.nc\" {\n",
"GROUP \"/\" {\n",
" ATTRIBUTE \"_NCProperties\" {\n",
" DATATYPE H5T_STRING {\n",
" STRSIZE 34;\n",
" STRPAD H5T_STR_NULLTERM;\n",
" CSET H5T_CSET_ASCII;\n",
" CTYPE H5T_C_S1;\n",
" }\n",
" DATASPACE SCALAR\n",
" DATA {\n",
" (0): \"version=2,netcdf=4.9.2,hdf5=1.14.3\"\n",
" }\n",
" }\n",
" DATATYPE \"_PFNC_DOUBLE_COMPLEX_TYPE\" H5T_COMPOUND {\n",
" H5T_IEEE_F64LE \"r\";\n",
" H5T_IEEE_F64LE \"i\";\n",
" }\n",
" DATASET \"data\" {\n",
" DATATYPE H5T_COMPOUND {\n",
" H5T_IEEE_F64LE \"r\";\n",
" H5T_IEEE_F64LE \"i\";\n",
" }\n",
" DATASPACE SIMPLE { ( 3 ) / ( 3 ) }\n",
" DATA {\n",
" (0): {\n",
" 0,\n",
" 0\n",
" },\n",
" (1): {\n",
" 1,\n",
" 0\n",
" },\n",
" (2): {\n",
" 0,\n",
" 1\n",
" }\n",
" }\n",
" ATTRIBUTE \"DIMENSION_LIST\" {\n",
" DATATYPE H5T_VLEN { H5T_REFERENCE { H5T_STD_REF_OBJECT } }\n",
" DATASPACE SIMPLE { ( 1 ) / ( 1 ) }\n",
" DATA {\n",
" (0): (DATASET 94905721664656 \"/x\")\n",
" }\n",
" }\n",
" ATTRIBUTE \"_Netcdf4Coordinates\" {\n",
" DATATYPE H5T_STD_I32LE\n",
" DATASPACE SIMPLE { ( 1 ) / ( 1 ) }\n",
" DATA {\n",
" (0): 0\n",
" }\n",
" }\n",
" }\n",
" DATASET \"x\" {\n",
" DATATYPE H5T_IEEE_F32BE\n",
" DATASPACE SIMPLE { ( 3 ) / ( 3 ) }\n",
" DATA {\n",
" (0): 0, 0, 0\n",
" }\n",
" ATTRIBUTE \"CLASS\" {\n",
" DATATYPE H5T_STRING {\n",
" STRSIZE 16;\n",
" STRPAD H5T_STR_NULLTERM;\n",
" CSET H5T_CSET_ASCII;\n",
" CTYPE H5T_C_S1;\n",
" }\n",
" DATASPACE SCALAR\n",
" DATA {\n",
" (0): \"DIMENSION_SCALE\"\n",
" }\n",
" }\n",
" ATTRIBUTE \"NAME\" {\n",
" DATATYPE H5T_STRING {\n",
" STRSIZE 64;\n",
" STRPAD H5T_STR_NULLTERM;\n",
" CSET H5T_CSET_ASCII;\n",
" CTYPE H5T_C_S1;\n",
" }\n",
" DATASPACE SCALAR\n",
" DATA {\n",
" (0): \"This is a netCDF dimension but not a netCDF variable. 3\"\n",
" }\n",
" }\n",
" ATTRIBUTE \"REFERENCE_LIST\" {\n",
" DATATYPE H5T_COMPOUND {\n",
" H5T_REFERENCE { H5T_STD_REF_OBJECT } \"dataset\";\n",
" H5T_STD_U32LE \"dimension\";\n",
" }\n",
" DATASPACE SIMPLE { ( 1 ) / ( 1 ) }\n",
" DATA {\n",
" (0): {\n",
" DATASET 94905721498880 \"/data\",\n",
" 0\n",
" }\n",
" }\n",
" }\n",
" ATTRIBUTE \"_Netcdf4Dimid\" {\n",
" DATATYPE H5T_STD_I32LE\n",
" DATASPACE SCALAR\n",
" DATA {\n",
" (0): 0\n",
" }\n",
" }\n",
" }\n",
"}\n",
"}\n"
]
}
],
"source": [
"!h5dump complex_auto.nc"
]
},
{
"cell_type": "markdown",
"id": "165a0624-50c6-4735-a42e-b1d83f10b557",
"metadata": {},
"source": [
"## 1.1.3 Summary\n",
"\n",
"As we can see from the different creation processes, the resulting files are equivalent. The advantage of using `auto_complex=True` (available with the `nc-complex`-package from netcdf4-python >=1.7.0) is the automated conversion from numpy complex data to netcdf compound data and back.\n",
"\n",
"h5netcdf can read both approaches perfectly fine as complex numpy arrays.\n"
]
},
{
"cell_type": "markdown",
"id": "0f2d6ba3-b6a8-463d-9421-3f904c6add38",
"metadata": {},
"source": [
"# 1.2 h5netcdf"
]
},
{
"cell_type": "markdown",
"id": "e3e00d28-86f8-4589-871c-54d23560ad87",
"metadata": {},
"source": [
"## 1.2.1 Current approach as outline in netcdf4-python docs"
]
},
{
"cell_type": "markdown",
"id": "d280c39d-bd6b-4233-8140-eee281ddddbb",
"metadata": {},
"source": [
"### create file with transient complex compound type"
]
},
{
"cell_type": "code",
"execution_count": 16,
"id": "1d6b03a5-bc65-4eb5-b2c4-bc7b00872e3c",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"complex128: <h5netcdf.legacyapi.Variable '/data': dimensions x, shape (3,), dtype complex128>\n",
"Attributes:\n"
]
}
],
"source": [
"with h5nc.Dataset(\"complex_h5nc.nc\", \"w\") as ds:\n",
" size = 3 # length of 1-d complex array\n",
" # create sample complex data.\n",
" complex_array = np.array([0 + 0j, 1 + 0j, 0 + 1j])\n",
" # create complex128 compound data type.\n",
" complex128 = np.dtype([(\"r\",np.float64),(\"i\",np.float64)])\n",
" # create a variable with this data type, write some data to it.\n",
" x_dim = ds.createDimension(\"x\",None)\n",
" v = ds.createVariable(\"data\", complex128, \"x\")\n",
" data = np.empty(size,complex128) # numpy structured array\n",
" data[\"r\"] = complex_array.real; data[\"i\"] = complex_array.imag\n",
" v[:] = data # write numpy structured array to netcdf compound var\n",
" print('{}: {}'.format(v.dtype, v))"
]
},
{
"cell_type": "markdown",
"id": "19cd18b1-57ab-44ac-ac33-b34f07d6ab03",
"metadata": {},
"source": [
"### reopen and convert to complex data again"
]
},
{
"cell_type": "code",
"execution_count": 17,
"id": "f57561db-97dd-44a9-af6f-e5ce6335334e",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"original : complex128: [0.+0.j 1.+0.j 0.+1.j]\n",
"actual: complex128: [0.+0.j 1.+0.j 0.+1.j]\n"
]
}
],
"source": [
"with h5nc.Dataset(\"complex_h5nc.nc\", \"r\") as ds:\n",
" datac2 = ds.variables[\"data\"]\n",
" print('original : {}: {}'.format(complex_array.dtype, complex_array)) # original data\n",
" print('actual: {}: {}'.format(datac2.dtype, datac2[:])) # reconverted data"
]
},
{
"cell_type": "markdown",
"id": "1dae0cbd-f524-41c1-9e02-f5ff285dcfbf",
"metadata": {},
"source": [
"### Read with auto_complex = True using netcdf4-python"
]
},
{
"cell_type": "code",
"execution_count": 18,
"id": "1b169771-155d-4447-b46e-7a2341c7dc22",
"metadata": {},
"outputs": [
{
"ename": "IndexError",
"evalue": "data not found in /",
"output_type": "error",
"traceback": [
"\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
"\u001b[0;31mIndexError\u001b[0m Traceback (most recent call last)",
"Cell \u001b[0;32mIn[18], line 2\u001b[0m\n\u001b[1;32m 1\u001b[0m \u001b[38;5;28;01mwith\u001b[39;00m nc\u001b[38;5;241m.\u001b[39mDataset(\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mcomplex_h5nc.nc\u001b[39m\u001b[38;5;124m\"\u001b[39m, \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mr\u001b[39m\u001b[38;5;124m\"\u001b[39m, auto_complex\u001b[38;5;241m=\u001b[39m\u001b[38;5;28;01mTrue\u001b[39;00m) \u001b[38;5;28;01mas\u001b[39;00m ds:\n\u001b[0;32m----> 2\u001b[0m datac2 \u001b[38;5;241m=\u001b[39m \u001b[43mds\u001b[49m\u001b[43m[\u001b[49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[38;5;124;43mdata\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[43m]\u001b[49m\n\u001b[1;32m 3\u001b[0m \u001b[38;5;28mprint\u001b[39m(\u001b[38;5;124m'\u001b[39m\u001b[38;5;124moriginal: \u001b[39m\u001b[38;5;132;01m{}\u001b[39;00m\u001b[38;5;124m - \u001b[39m\u001b[38;5;132;01m{}\u001b[39;00m\u001b[38;5;124m'\u001b[39m\u001b[38;5;241m.\u001b[39mformat(complex_array\u001b[38;5;241m.\u001b[39mdtype, complex_array)) \u001b[38;5;66;03m# original data\u001b[39;00m\n\u001b[1;32m 4\u001b[0m \u001b[38;5;28mprint\u001b[39m(\u001b[38;5;124m'\u001b[39m\u001b[38;5;124mactual : \u001b[39m\u001b[38;5;132;01m{}\u001b[39;00m\u001b[38;5;124m - \u001b[39m\u001b[38;5;132;01m{}\u001b[39;00m\u001b[38;5;124m'\u001b[39m\u001b[38;5;241m.\u001b[39mformat(datac2\u001b[38;5;241m.\u001b[39mdtype, datac2[:])) \u001b[38;5;66;03m# reread data\u001b[39;00m\n",
"File \u001b[0;32msrc/netCDF4/_netCDF4.pyx:2519\u001b[0m, in \u001b[0;36mnetCDF4._netCDF4.Dataset.__getitem__\u001b[0;34m()\u001b[0m\n",
"\u001b[0;31mIndexError\u001b[0m: data not found in /"
]
}
],
"source": [
"with nc.Dataset(\"complex_h5nc.nc\", \"r\", auto_complex=True) as ds:\n",
" datac2 = ds[\"data\"]\n",
" print('original: {} - {}'.format(complex_array.dtype, complex_array)) # original data\n",
" print('actual : {} - {}'.format(datac2.dtype, datac2[:])) # reread data"
]
},
{
"cell_type": "markdown",
"id": "5e1643fd-7029-4fcc-85c9-e7e4c3b528ab",
"metadata": {},
"source": [
"### reopen and convert to complex data again using netcdf4-python"
]
},
{
"cell_type": "code",
"execution_count": 19,
"id": "9cbea29f-3e8d-43b8-a353-1e876ed7af9b",
"metadata": {},
"outputs": [
{
"ename": "KeyError",
"evalue": "'data'",
"output_type": "error",
"traceback": [
"\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
"\u001b[0;31mKeyError\u001b[0m Traceback (most recent call last)",
"Cell \u001b[0;32mIn[19], line 2\u001b[0m\n\u001b[1;32m 1\u001b[0m \u001b[38;5;28;01mwith\u001b[39;00m nc\u001b[38;5;241m.\u001b[39mDataset(\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mcomplex_h5nc.nc\u001b[39m\u001b[38;5;124m\"\u001b[39m, \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mr\u001b[39m\u001b[38;5;124m\"\u001b[39m) \u001b[38;5;28;01mas\u001b[39;00m ds:\n\u001b[0;32m----> 2\u001b[0m v \u001b[38;5;241m=\u001b[39m \u001b[43mds\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mvariables\u001b[49m\u001b[43m[\u001b[49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[38;5;124;43mdata\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[43m]\u001b[49m\n\u001b[1;32m 3\u001b[0m datain \u001b[38;5;241m=\u001b[39m v[:] \u001b[38;5;66;03m# read in all the data into a numpy structured array\u001b[39;00m\n\u001b[1;32m 4\u001b[0m \u001b[38;5;66;03m# create an empty numpy complex array\u001b[39;00m\n",
"\u001b[0;31mKeyError\u001b[0m: 'data'"
]
}
],
"source": [
"with nc.Dataset(\"complex_h5nc.nc\", \"r\") as ds:\n",
" v = ds.variables[\"data\"]\n",
" datain = v[:] # read in all the data into a numpy structured array\n",
" # create an empty numpy complex array\n",
" datac2 = np.empty(datain.shape,np.complex128)\n",
" # .. fill it with contents of structured array.\n",
" datac2.real = datain[\"r\"]; datac2.imag = datain[\"i\"]\n",
" print('original : {}: {}'.format(complex_array.dtype, complex_array)) # original data\n",
" print('actual : {}: {}'.format(datain.dtype, datain)) # actual data\n",
" print('recreated: {}: {}'.format(datac2.dtype, datac2)) # reconverted data"
]
},
{
"cell_type": "code",
"execution_count": 20,
"id": "e7caf6f0-0a4e-4e83-9a01-d91e080a64ec",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"netcdf complex_h5nc {\n",
"dimensions:\n",
"\tx = UNLIMITED ; // (0 currently)\n",
"}\n"
]
}
],
"source": [
"!ncdump complex_h5nc.nc"
]
},
{
"cell_type": "code",
"execution_count": 21,
"id": "6f461711-a12b-464b-b664-647094ef947d",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"HDF5 \"complex_h5nc.nc\" {\n",
"GROUP \"/\" {\n",
" ATTRIBUTE \"_NCProperties\" {\n",
" DATATYPE H5T_STRING {\n",
" STRSIZE 73;\n",
" STRPAD H5T_STR_NULLPAD;\n",
" CSET H5T_CSET_ASCII;\n",
" CTYPE H5T_C_S1;\n",
" }\n",
" DATASPACE SCALAR\n",
" DATA {\n",
" (0): \"version=2,h5netcdf=1.4.0.dev11+gd365128.d20240120,hdf5=1.14.3,h5py=3.10.0\"\n",
" }\n",
" }\n",
" DATASET \"data\" {\n",
" DATATYPE H5T_COMPOUND {\n",
" H5T_IEEE_F64LE \"r\";\n",
" H5T_IEEE_F64LE \"i\";\n",
" }\n",
" DATASPACE SIMPLE { ( 3 ) / ( H5S_UNLIMITED ) }\n",
" DATA {\n",
" (0): {\n",
" 0,\n",
" 0\n",
" },\n",
" (1): {\n",
" 1,\n",
" 0\n",
" },\n",
" (2): {\n",
" 0,\n",
" 1\n",
" }\n",
" }\n",
" ATTRIBUTE \"DIMENSION_LIST\" {\n",
" DATATYPE H5T_VLEN { H5T_REFERENCE { H5T_STD_REF_OBJECT } }\n",
" DATASPACE SIMPLE { ( 1 ) / ( 1 ) }\n",
" DATA {\n",
" (0): (DATASET 94183731396448 \"/x\")\n",
" }\n",
" }\n",
" ATTRIBUTE \"_Netcdf4Coordinates\" {\n",
" DATATYPE H5T_STD_I32LE\n",
" DATASPACE SIMPLE { ( 1 ) / ( 1 ) }\n",
" DATA {\n",
" (0): 0\n",
" }\n",
" }\n",
" }\n",
" DATASET \"x\" {\n",
" DATATYPE H5T_IEEE_F32BE\n",
" DATASPACE SIMPLE { ( 3 ) / ( H5S_UNLIMITED ) }\n",
" DATA {\n",
" (0): 0, 0, 0\n",
" }\n",
" ATTRIBUTE \"CLASS\" {\n",
" DATATYPE H5T_STRING {\n",
" STRSIZE 16;\n",
" STRPAD H5T_STR_NULLTERM;\n",
" CSET H5T_CSET_ASCII;\n",
" CTYPE H5T_C_S1;\n",
" }\n",
" DATASPACE SCALAR\n",
" DATA {\n",
" (0): \"DIMENSION_SCALE\"\n",
" }\n",
" }\n",
" ATTRIBUTE \"NAME\" {\n",
" DATATYPE H5T_STRING {\n",
" STRSIZE 64;\n",
" STRPAD H5T_STR_NULLTERM;\n",
" CSET H5T_CSET_ASCII;\n",
" CTYPE H5T_C_S1;\n",
" }\n",
" DATASPACE SCALAR\n",
" DATA {\n",
" (0): \"This is a netCDF dimension but not a netCDF variable. 1\"\n",
" }\n",
" }\n",
" ATTRIBUTE \"REFERENCE_LIST\" {\n",
" DATATYPE H5T_COMPOUND {\n",
" H5T_REFERENCE { H5T_STD_REF_OBJECT } \"dataset\";\n",
" H5T_STD_U32LE \"dimension\";\n",
" }\n",
" DATASPACE SIMPLE { ( 1 ) / ( 1 ) }\n",
" DATA {\n",
" (0): {\n",
" DATASET 94183731329968 \"/data\",\n",
" 0\n",
" }\n",
" }\n",
" }\n",
" ATTRIBUTE \"_Netcdf4Dimid\" {\n",
" DATATYPE H5T_STD_I32LE\n",
" DATASPACE SCALAR\n",
" DATA {\n",
" (0): 0\n",
" }\n",
" }\n",
" }\n",
"}\n",
"}\n"
]
}
],
"source": [
"!h5dump complex_h5nc.nc"
]
},
{
"cell_type": "markdown",
"id": "aa523cae-8b28-4e99-8628-3abf2043ecf2",
"metadata": {},
"source": [
"## 1.2.2 Just write complex types "
]
},
{
"cell_type": "markdown",
"id": "b3ed0b2a-752e-45b5-9411-dae736e3150f",
"metadata": {},
"source": [
"### create file with transient complex compound type"
]
},
{
"cell_type": "code",
"execution_count": 25,
"id": "dcd6f587-46e7-4f53-941f-4b2d258c25ee",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"complex128: <h5netcdf.legacyapi.Variable '/data': dimensions x, shape (3,), dtype complex128>\n",
"Attributes:\n"
]
}
],
"source": [
"with h5nc.Dataset(\"complex_curr_h5nc.nc\", \"w\") as ds:\n",
" size = 3 # length of 1-d complex array\n",
" # create sample complex data.\n",
" complex_array = np.array([0 + 0j, 1 + 0j, 0 + 1j])\n",
" # create a variable with this data type, write some data to it.\n",
" x_dim = ds.createDimension(\"x\",None)\n",
" v = ds.createVariable(\"data\", complex_array.dtype, \"x\")\n",
" v[:] = complex_array # write numpy structured array to netcdf compound var\n",
" print('{}: {}'.format(v.dtype, v))"
]
},
{
"cell_type": "markdown",
"id": "e63db340-c391-49f2-bda7-7d4fbfd69046",
"metadata": {},
"source": [
"### reopen and convert to complex data again"
]
},
{
"cell_type": "code",
"execution_count": 26,
"id": "c8f804f4-9e01-4d62-94aa-d3d8e06dd367",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"original : complex128: [0.+0.j 1.+0.j 0.+1.j]\n",
"actual: complex128: [0.+0.j 1.+0.j 0.+1.j]\n"
]
}
],
"source": [
"with h5nc.Dataset(\"complex_curr_h5nc.nc\", \"r\") as ds:\n",
" datac2 = ds.variables[\"data\"]\n",
" print('original : {}: {}'.format(complex_array.dtype, complex_array)) # original data\n",
" print('actual: {}: {}'.format(datac2.dtype, datac2[:])) # reconverted data"
]
},
{
"cell_type": "markdown",
"id": "d9a0637b-6aed-4b27-a463-6f1a1d27794a",
"metadata": {},
"source": [
"### Read with auto_complex = True using netcdf4-python"
]
},
{
"cell_type": "code",
"execution_count": 27,
"id": "21b65434-b2e6-4859-964d-0b76419be7bf",
"metadata": {},
"outputs": [
{
"ename": "IndexError",
"evalue": "data not found in /",
"output_type": "error",
"traceback": [
"\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
"\u001b[0;31mIndexError\u001b[0m Traceback (most recent call last)",
"Cell \u001b[0;32mIn[27], line 2\u001b[0m\n\u001b[1;32m 1\u001b[0m \u001b[38;5;28;01mwith\u001b[39;00m nc\u001b[38;5;241m.\u001b[39mDataset(\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mcomplex_curr_h5nc.nc\u001b[39m\u001b[38;5;124m\"\u001b[39m, \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mr\u001b[39m\u001b[38;5;124m\"\u001b[39m, auto_complex\u001b[38;5;241m=\u001b[39m\u001b[38;5;28;01mTrue\u001b[39;00m) \u001b[38;5;28;01mas\u001b[39;00m ds:\n\u001b[0;32m----> 2\u001b[0m datac2 \u001b[38;5;241m=\u001b[39m \u001b[43mds\u001b[49m\u001b[43m[\u001b[49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[38;5;124;43mdata\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[43m]\u001b[49m\n\u001b[1;32m 3\u001b[0m \u001b[38;5;28mprint\u001b[39m(\u001b[38;5;124m'\u001b[39m\u001b[38;5;124moriginal: \u001b[39m\u001b[38;5;132;01m{}\u001b[39;00m\u001b[38;5;124m - \u001b[39m\u001b[38;5;132;01m{}\u001b[39;00m\u001b[38;5;124m'\u001b[39m\u001b[38;5;241m.\u001b[39mformat(complex_array\u001b[38;5;241m.\u001b[39mdtype, complex_array)) \u001b[38;5;66;03m# original data\u001b[39;00m\n\u001b[1;32m 4\u001b[0m \u001b[38;5;28mprint\u001b[39m(\u001b[38;5;124m'\u001b[39m\u001b[38;5;124mactual : \u001b[39m\u001b[38;5;132;01m{}\u001b[39;00m\u001b[38;5;124m - \u001b[39m\u001b[38;5;132;01m{}\u001b[39;00m\u001b[38;5;124m'\u001b[39m\u001b[38;5;241m.\u001b[39mformat(datac2\u001b[38;5;241m.\u001b[39mdtype, datac2[:])) \u001b[38;5;66;03m# reread data\u001b[39;00m\n",
"File \u001b[0;32msrc/netCDF4/_netCDF4.pyx:2519\u001b[0m, in \u001b[0;36mnetCDF4._netCDF4.Dataset.__getitem__\u001b[0;34m()\u001b[0m\n",
"\u001b[0;31mIndexError\u001b[0m: data not found in /"
]
}
],
"source": [
"with nc.Dataset(\"complex_curr_h5nc.nc\", \"r\", auto_complex=True) as ds:\n",
" datac2 = ds[\"data\"]\n",
" print('original: {} - {}'.format(complex_array.dtype, complex_array)) # original data\n",
" print('actual : {} - {}'.format(datac2.dtype, datac2[:])) # reread data"
]
},
{
"cell_type": "markdown",
"id": "b9fabf2f-7869-4aa6-9d37-994e0b935433",
"metadata": {},
"source": [
"### reopen and convert to complex data again using netcdf4-python"
]
},
{
"cell_type": "code",
"execution_count": 28,
"id": "70e4c896-e8f9-4c16-9690-c26ce261d26f",
"metadata": {},
"outputs": [
{
"ename": "KeyError",
"evalue": "'data'",
"output_type": "error",
"traceback": [
"\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
"\u001b[0;31mKeyError\u001b[0m Traceback (most recent call last)",
"Cell \u001b[0;32mIn[28], line 2\u001b[0m\n\u001b[1;32m 1\u001b[0m \u001b[38;5;28;01mwith\u001b[39;00m nc\u001b[38;5;241m.\u001b[39mDataset(\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mcomplex_curr_h5nc.nc\u001b[39m\u001b[38;5;124m\"\u001b[39m, \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mr\u001b[39m\u001b[38;5;124m\"\u001b[39m) \u001b[38;5;28;01mas\u001b[39;00m ds:\n\u001b[0;32m----> 2\u001b[0m v \u001b[38;5;241m=\u001b[39m \u001b[43mds\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mvariables\u001b[49m\u001b[43m[\u001b[49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[38;5;124;43mdata\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[43m]\u001b[49m\n\u001b[1;32m 3\u001b[0m datain \u001b[38;5;241m=\u001b[39m v[:] \u001b[38;5;66;03m# read in all the data into a numpy structured array\u001b[39;00m\n\u001b[1;32m 4\u001b[0m \u001b[38;5;66;03m# create an empty numpy complex array\u001b[39;00m\n",
"\u001b[0;31mKeyError\u001b[0m: 'data'"
]
}
],
"source": [
"with nc.Dataset(\"complex_curr_h5nc.nc\", \"r\") as ds:\n",
" v = ds.variables[\"data\"]\n",
" datain = v[:] # read in all the data into a numpy structured array\n",
" # create an empty numpy complex array\n",
" datac2 = np.empty(datain.shape,np.complex128)\n",
" # .. fill it with contents of structured array.\n",
" datac2.real = datain[\"r\"]; datac2.imag = datain[\"i\"]\n",
" print('original : {}: {}'.format(complex_array.dtype, complex_array)) # original data\n",
" print('actual : {}: {}'.format(datain.dtype, datain)) # actual data\n",
" print('recreated: {}: {}'.format(datac2.dtype, datac2)) # reconverted data"
]
},
{
"cell_type": "code",
"execution_count": 29,
"id": "9a061a96-4e91-4694-b344-0db87a1ce8c1",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"netcdf complex_curr_h5nc {\n",
"dimensions:\n",
"\tx = UNLIMITED ; // (0 currently)\n",
"}\n"
]
}
],
"source": [
"!ncdump complex_curr_h5nc.nc"
]
},
{
"cell_type": "code",
"execution_count": 30,
"id": "7a5be70d-2197-4d34-a4bd-7ead85584824",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"HDF5 \"complex_curr_h5nc.nc\" {\n",
"GROUP \"/\" {\n",
" ATTRIBUTE \"_NCProperties\" {\n",
" DATATYPE H5T_STRING {\n",
" STRSIZE 73;\n",
" STRPAD H5T_STR_NULLPAD;\n",
" CSET H5T_CSET_ASCII;\n",
" CTYPE H5T_C_S1;\n",
" }\n",
" DATASPACE SCALAR\n",
" DATA {\n",
" (0): \"version=2,h5netcdf=1.4.0.dev11+gd365128.d20240120,hdf5=1.14.3,h5py=3.10.0\"\n",
" }\n",
" }\n",
" DATASET \"data\" {\n",
" DATATYPE H5T_COMPOUND {\n",
" H5T_IEEE_F64LE \"r\";\n",
" H5T_IEEE_F64LE \"i\";\n",
" }\n",
" DATASPACE SIMPLE { ( 3 ) / ( H5S_UNLIMITED ) }\n",
" DATA {\n",
" (0): {\n",
" 0,\n",
" 0\n",
" },\n",
" (1): {\n",
" 1,\n",
" 0\n",
" },\n",
" (2): {\n",
" 0,\n",
" 1\n",
" }\n",
" }\n",
" ATTRIBUTE \"DIMENSION_LIST\" {\n",
" DATATYPE H5T_VLEN { H5T_REFERENCE { H5T_STD_REF_OBJECT } }\n",
" DATASPACE SIMPLE { ( 1 ) / ( 1 ) }\n",
" DATA {\n",
" (0): (DATASET 94244801568624 \"/x\")\n",
" }\n",
" }\n",
" ATTRIBUTE \"_Netcdf4Coordinates\" {\n",
" DATATYPE H5T_STD_I32LE\n",
" DATASPACE SIMPLE { ( 1 ) / ( 1 ) }\n",
" DATA {\n",
" (0): 0\n",
" }\n",
" }\n",
" }\n",
" DATASET \"x\" {\n",
" DATATYPE H5T_IEEE_F32BE\n",
" DATASPACE SIMPLE { ( 3 ) / ( H5S_UNLIMITED ) }\n",
" DATA {\n",
" (0): 0, 0, 0\n",
" }\n",
" ATTRIBUTE \"CLASS\" {\n",
" DATATYPE H5T_STRING {\n",
" STRSIZE 16;\n",
" STRPAD H5T_STR_NULLTERM;\n",
" CSET H5T_CSET_ASCII;\n",
" CTYPE H5T_C_S1;\n",
" }\n",
" DATASPACE SCALAR\n",
" DATA {\n",
" (0): \"DIMENSION_SCALE\"\n",
" }\n",
" }\n",
" ATTRIBUTE \"NAME\" {\n",
" DATATYPE H5T_STRING {\n",
" STRSIZE 64;\n",
" STRPAD H5T_STR_NULLTERM;\n",
" CSET H5T_CSET_ASCII;\n",
" CTYPE H5T_C_S1;\n",
" }\n",
" DATASPACE SCALAR\n",
" DATA {\n",
" (0): \"This is a netCDF dimension but not a netCDF variable. 1\"\n",
" }\n",
" }\n",
" ATTRIBUTE \"REFERENCE_LIST\" {\n",
" DATATYPE H5T_COMPOUND {\n",
" H5T_REFERENCE { H5T_STD_REF_OBJECT } \"dataset\";\n",
" H5T_STD_U32LE \"dimension\";\n",
" }\n",
" DATASPACE SIMPLE { ( 1 ) / ( 1 ) }\n",
" DATA {\n",
" (0): {\n",
" DATASET 94244801502144 \"/data\",\n",
" 0\n",
" }\n",
" }\n",
" }\n",
" ATTRIBUTE \"_Netcdf4Dimid\" {\n",
" DATATYPE H5T_STD_I32LE\n",
" DATASPACE SCALAR\n",
" DATA {\n",
" (0): 0\n",
" }\n",
" }\n",
" }\n",
"}\n",
"}\n"
]
}
],
"source": [
"!h5dump complex_curr_h5nc.nc"
]
},
{
"cell_type": "markdown",
"id": "728c1a2f-a429-4044-9640-a868071ff5d3",
"metadata": {},
"source": [
"## 1.2.3 Create with committing complex compound type to file "
]
},
{
"cell_type": "code",
"execution_count": 31,
"id": "9cc50707-5020-4f98-8816-9eb39b68dd60",
"metadata": {},
"outputs": [],
"source": [
"with h5nc.Dataset(\"complex_comm_h5nc.nc\", \"w\") as ds:\n",
" ds.createDimension(\"x\", size=len(complex_array))\n",
" complex128_t = ds.createCompoundType(complex_array.dtype, \"complex128_t\")\n",
" var = ds.createVariable(\"data\", complex128_t, (\"x\",))\n",
" var[:] = complex_array"
]
},
{
"cell_type": "markdown",
"id": "a9cf411b-45b8-4322-915a-841f18823ee9",
"metadata": {},
"source": [
"### reopen and convert to complex data again"
]
},
{
"cell_type": "code",
"execution_count": 32,
"id": "61d04845-66b1-4824-83b4-08c226d1a283",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"original : complex128: [0.+0.j 1.+0.j 0.+1.j]\n",
"actual: complex128: [0.+0.j 1.+0.j 0.+1.j]\n"
]
}
],
"source": [
"with h5nc.Dataset(\"complex_comm_h5nc.nc\", \"r\") as ds:\n",
" datac2 = ds.variables[\"data\"]\n",
" print('original : {}: {}'.format(complex_array.dtype, complex_array)) # original data\n",
" print('actual: {}: {}'.format(datac2.dtype, datac2[:])) # reconverted data"
]
},
{
"cell_type": "markdown",
"id": "83fa4247-5626-494a-9eec-d4980b86ec5b",
"metadata": {},
"source": [
"### Read with auto_complex = True using netcdf4-python"
]
},
{
"cell_type": "code",
"execution_count": 33,
"id": "2bef7e42-34f9-45a9-8c3b-643a045ffa75",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"original: complex128 - [0.+0.j 1.+0.j 0.+1.j]\n",
"actual : complex128 - [0.+0.j 1.+0.j 0.+1.j]\n"
]
}
],
"source": [
"with nc.Dataset(\"complex_comm_h5nc.nc\", \"r\", auto_complex=True) as ds:\n",
" datac2 = ds[\"data\"]\n",
" print('original: {} - {}'.format(complex_array.dtype, complex_array)) # original data\n",
" print('actual : {} - {}'.format(datac2.dtype, datac2[:])) # reread data"
]
},
{
"cell_type": "markdown",
"id": "846c8cf8-e70a-460e-9a47-4e4e8d61ab06",
"metadata": {},
"source": [
"### reopen and convert to complex data again using netcdf4-python"
]
},
{
"cell_type": "code",
"execution_count": 34,
"id": "b062436d-79d9-4ce0-b42d-56dea93c80bb",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"original : complex128: [0.+0.j 1.+0.j 0.+1.j]\n",
"actual : {'names': ['r', 'i'], 'formats': ['<f8', '<f8'], 'offsets': [0, 8], 'itemsize': 16, 'aligned': True}: [(0., 0.) (1., 0.) (0., 1.)]\n",
"recreated: complex128: [0.+0.j 1.+0.j 0.+1.j]\n"
]
}
],
"source": [
"with nc.Dataset(\"complex_comm_h5nc.nc\", \"r\") as ds:\n",
" v = ds.variables[\"data\"]\n",
" datain = v[:] # read in all the data into a numpy structured array\n",
" # create an empty numpy complex array\n",
" datac2 = np.empty(datain.shape,np.complex128)\n",
" # .. fill it with contents of structured array.\n",
" datac2.real = datain[\"r\"]; datac2.imag = datain[\"i\"]\n",
" print('original : {}: {}'.format(complex_array.dtype, complex_array)) # original data\n",
" print('actual : {}: {}'.format(datain.dtype, datain)) # actual data\n",
" print('recreated: {}: {}'.format(datac2.dtype, datac2)) # reconverted data"
]
},
{
"cell_type": "code",
"execution_count": 35,
"id": "0014d4c0-b62e-404b-bba6-1cfb1c845b70",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"netcdf complex_comm_h5nc {\n",
"types:\n",
" compound complex128_t {\n",
" double r ;\n",
" double i ;\n",
" }; // complex128_t\n",
"dimensions:\n",
"\tx = 3 ;\n",
"variables:\n",
"\tcomplex128_t data(x) ;\n",
"data:\n",
"\n",
" data = {0, 0}, {1, 0}, {0, 1} ;\n",
"}\n"
]
}
],
"source": [
"!ncdump complex_comm_h5nc.nc"
]
},
{
"cell_type": "code",
"execution_count": 36,
"id": "85ba4283-ab20-40c1-ae04-10343e0d9394",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"HDF5 \"complex_comm_h5nc.nc\" {\n",
"GROUP \"/\" {\n",
" ATTRIBUTE \"_NCProperties\" {\n",
" DATATYPE H5T_STRING {\n",
" STRSIZE 73;\n",
" STRPAD H5T_STR_NULLPAD;\n",
" CSET H5T_CSET_ASCII;\n",
" CTYPE H5T_C_S1;\n",
" }\n",
" DATASPACE SCALAR\n",
" DATA {\n",
" (0): \"version=2,h5netcdf=1.4.0.dev11+gd365128.d20240120,hdf5=1.14.3,h5py=3.10.0\"\n",
" }\n",
" }\n",
" DATATYPE \"complex128_t\" H5T_COMPOUND {\n",
" H5T_IEEE_F64LE \"r\";\n",
" H5T_IEEE_F64LE \"i\";\n",
" }\n",
" DATASET \"data\" {\n",
" DATATYPE H5T_COMPOUND {\n",
" H5T_IEEE_F64LE \"r\";\n",
" H5T_IEEE_F64LE \"i\";\n",
" }\n",
" DATASPACE SIMPLE { ( 3 ) / ( 3 ) }\n",
" DATA {\n",
" (0): {\n",
" 0,\n",
" 0\n",
" },\n",
" (1): {\n",
" 1,\n",
" 0\n",
" },\n",
" (2): {\n",
" 0,\n",
" 1\n",
" }\n",
" }\n",
" ATTRIBUTE \"DIMENSION_LIST\" {\n",
" DATATYPE H5T_VLEN { H5T_REFERENCE { H5T_STD_REF_OBJECT } }\n",
" DATASPACE SIMPLE { ( 1 ) / ( 1 ) }\n",
" DATA {\n",
" (0): (DATASET 93838257297024 \"/x\")\n",
" }\n",
" }\n",
" ATTRIBUTE \"_Netcdf4Coordinates\" {\n",
" DATATYPE H5T_STD_I32LE\n",
" DATASPACE SIMPLE { ( 1 ) / ( 1 ) }\n",
" DATA {\n",
" (0): 0\n",
" }\n",
" }\n",
" }\n",
" DATASET \"x\" {\n",
" DATATYPE H5T_IEEE_F32BE\n",
" DATASPACE SIMPLE { ( 3 ) / ( 3 ) }\n",
" DATA {\n",
" (0): 0, 0, 0\n",
" }\n",
" ATTRIBUTE \"CLASS\" {\n",
" DATATYPE H5T_STRING {\n",
" STRSIZE 16;\n",
" STRPAD H5T_STR_NULLTERM;\n",
" CSET H5T_CSET_ASCII;\n",
" CTYPE H5T_C_S1;\n",
" }\n",
" DATASPACE SCALAR\n",
" DATA {\n",
" (0): \"DIMENSION_SCALE\"\n",
" }\n",
" }\n",
" ATTRIBUTE \"NAME\" {\n",
" DATATYPE H5T_STRING {\n",
" STRSIZE 64;\n",
" STRPAD H5T_STR_NULLTERM;\n",
" CSET H5T_CSET_ASCII;\n",
" CTYPE H5T_C_S1;\n",
" }\n",
" DATASPACE SCALAR\n",
" DATA {\n",
" (0): \"This is a netCDF dimension but not a netCDF variable. 3\"\n",
" }\n",
" }\n",
" ATTRIBUTE \"REFERENCE_LIST\" {\n",
" DATATYPE H5T_COMPOUND {\n",
" H5T_REFERENCE { H5T_STD_REF_OBJECT } \"dataset\";\n",
" H5T_STD_U32LE \"dimension\";\n",
" }\n",
" DATASPACE SIMPLE { ( 1 ) / ( 1 ) }\n",
" DATA {\n",
" (0): {\n",
" DATASET 93838257288912 \"/data\",\n",
" 0\n",
" }\n",
" }\n",
" }\n",
" ATTRIBUTE \"_Netcdf4Dimid\" {\n",
" DATATYPE H5T_STD_I32LE\n",
" DATASPACE SCALAR\n",
" DATA {\n",
" (0): 0\n",
" }\n",
" }\n",
" }\n",
"}\n",
"}\n"
]
}
],
"source": [
"!h5dump complex_comm_h5nc.nc"
]
},
{
"cell_type": "markdown",
"id": "d031f39a-0cef-4aab-a4cb-77d194879d47",
"metadata": {},
"source": [
"## 1.2.4 Create with automatic committing complex compound type to file "
]
},
{
"cell_type": "code",
"execution_count": 37,
"id": "df15187e-b87a-4d0b-8d8b-bcad176d5927",
"metadata": {},
"outputs": [],
"source": [
"with h5nc.Dataset(\"complex_auto_h5nc.nc\", \"w\", auto_complex=True) as ds:\n",
" ds.createDimension(\"x\", size=len(complex_array))\n",
" var = ds.createVariable(\"data\", \"c16\", (\"x\",))\n",
" var[:] = complex_array"
]
},
{
"cell_type": "markdown",
"id": "99be2189-74d9-4119-ab8c-344e861d28d4",
"metadata": {},
"source": [
"### reopen and convert to complex data again"
]
},
{
"cell_type": "code",
"execution_count": 38,
"id": "15b0d892-cf60-4151-a900-9eab83c86b2b",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"original : complex128: [0.+0.j 1.+0.j 0.+1.j]\n",
"actual: complex128: [0.+0.j 1.+0.j 0.+1.j]\n"
]
}
],
"source": [
"with h5nc.Dataset(\"complex_auto_h5nc.nc\", \"r\") as ds:\n",
" datac2 = ds.variables[\"data\"]\n",
" print('original : {}: {}'.format(complex_array.dtype, complex_array)) # original data\n",
" print('actual: {}: {}'.format(datac2.dtype, datac2[:])) # reconverted data"
]
},
{
"cell_type": "markdown",
"id": "cff086e6-648e-4cc7-be5a-e3fd82224dc3",
"metadata": {},
"source": [
"### Read with auto_complex = True using netcdf4-python"
]
},
{
"cell_type": "code",
"execution_count": 39,
"id": "c32c6835-ea88-43f5-9245-a449339d8380",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"original: complex128 - [0.+0.j 1.+0.j 0.+1.j]\n",
"actual : complex128 - [0.+0.j 1.+0.j 0.+1.j]\n"
]
}
],
"source": [
"with nc.Dataset(\"complex_auto_h5nc.nc\", \"r\", auto_complex=True) as ds:\n",
" datac2 = ds[\"data\"]\n",
" print('original: {} - {}'.format(complex_array.dtype, complex_array)) # original data\n",
" print('actual : {} - {}'.format(datac2.dtype, datac2[:])) # reread data"
]
},
{
"cell_type": "markdown",
"id": "a9ab6d95-8d3c-4911-b242-4e03e72f8fee",
"metadata": {},
"source": [
"### reopen and convert to complex data again using netcdf4-python"
]
},
{
"cell_type": "code",
"execution_count": 40,
"id": "8de5e46c-c331-49db-897c-5df2b7390d9c",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"original : complex128: [0.+0.j 1.+0.j 0.+1.j]\n",
"actual : {'names': ['r', 'i'], 'formats': ['<f8', '<f8'], 'offsets': [0, 8], 'itemsize': 16, 'aligned': True}: [(0., 0.) (1., 0.) (0., 1.)]\n",
"recreated: complex128: [0.+0.j 1.+0.j 0.+1.j]\n"
]
}
],
"source": [
"with nc.Dataset(\"complex_auto_h5nc.nc\", \"r\") as ds:\n",
" v = ds.variables[\"data\"]\n",
" datain = v[:] # read in all the data into a numpy structured array\n",
" # create an empty numpy complex array\n",
" datac2 = np.empty(datain.shape,np.complex128)\n",
" # .. fill it with contents of structured array.\n",
" datac2.real = datain[\"r\"]; datac2.imag = datain[\"i\"]\n",
" print('original : {}: {}'.format(complex_array.dtype, complex_array)) # original data\n",
" print('actual : {}: {}'.format(datain.dtype, datain)) # actual data\n",
" print('recreated: {}: {}'.format(datac2.dtype, datac2)) # reconverted data"
]
},
{
"cell_type": "code",
"execution_count": 41,
"id": "38738346-195e-440d-8e74-72eb41bd5001",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"netcdf complex_auto_h5nc {\n",
"types:\n",
" compound _PFNC_DOUBLE_COMPLEX_TYPE {\n",
" double r ;\n",
" double i ;\n",
" }; // _PFNC_DOUBLE_COMPLEX_TYPE\n",
"dimensions:\n",
"\tx = 3 ;\n",
"variables:\n",
"\t_PFNC_DOUBLE_COMPLEX_TYPE data(x) ;\n",
"data:\n",
"\n",
" data = {0, 0}, {1, 0}, {0, 1} ;\n",
"}\n"
]
}
],
"source": [
"!ncdump complex_auto_h5nc.nc"
]
},
{
"cell_type": "code",
"execution_count": 42,
"id": "e731c147-2f72-41df-8153-5ad16396e6ea",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"HDF5 \"complex_auto_h5nc.nc\" {\n",
"GROUP \"/\" {\n",
" ATTRIBUTE \"_NCProperties\" {\n",
" DATATYPE H5T_STRING {\n",
" STRSIZE 73;\n",
" STRPAD H5T_STR_NULLPAD;\n",
" CSET H5T_CSET_ASCII;\n",
" CTYPE H5T_C_S1;\n",
" }\n",
" DATASPACE SCALAR\n",
" DATA {\n",
" (0): \"version=2,h5netcdf=1.4.0.dev11+gd365128.d20240120,hdf5=1.14.3,h5py=3.10.0\"\n",
" }\n",
" }\n",
" DATATYPE \"_PFNC_DOUBLE_COMPLEX_TYPE\" H5T_COMPOUND {\n",
" H5T_IEEE_F64LE \"r\";\n",
" H5T_IEEE_F64LE \"i\";\n",
" }\n",
" DATASET \"data\" {\n",
" DATATYPE H5T_COMPOUND {\n",
" H5T_IEEE_F64LE \"r\";\n",
" H5T_IEEE_F64LE \"i\";\n",
" }\n",
" DATASPACE SIMPLE { ( 3 ) / ( 3 ) }\n",
" DATA {\n",
" (0): {\n",
" 0,\n",
" 0\n",
" },\n",
" (1): {\n",
" 1,\n",
" 0\n",
" },\n",
" (2): {\n",
" 0,\n",
" 1\n",
" }\n",
" }\n",
" ATTRIBUTE \"DIMENSION_LIST\" {\n",
" DATATYPE H5T_VLEN { H5T_REFERENCE { H5T_STD_REF_OBJECT } }\n",
" DATASPACE SIMPLE { ( 1 ) / ( 1 ) }\n",
" DATA {\n",
" (0): (DATASET 94412115631488 \"/x\")\n",
" }\n",
" }\n",
" ATTRIBUTE \"_Netcdf4Coordinates\" {\n",
" DATATYPE H5T_STD_I32LE\n",
" DATASPACE SIMPLE { ( 1 ) / ( 1 ) }\n",
" DATA {\n",
" (0): 0\n",
" }\n",
" }\n",
" }\n",
" DATASET \"x\" {\n",
" DATATYPE H5T_IEEE_F32BE\n",
" DATASPACE SIMPLE { ( 3 ) / ( 3 ) }\n",
" DATA {\n",
" (0): 0, 0, 0\n",
" }\n",
" ATTRIBUTE \"CLASS\" {\n",
" DATATYPE H5T_STRING {\n",
" STRSIZE 16;\n",
" STRPAD H5T_STR_NULLTERM;\n",
" CSET H5T_CSET_ASCII;\n",
" CTYPE H5T_C_S1;\n",
" }\n",
" DATASPACE SCALAR\n",
" DATA {\n",
" (0): \"DIMENSION_SCALE\"\n",
" }\n",
" }\n",
" ATTRIBUTE \"NAME\" {\n",
" DATATYPE H5T_STRING {\n",
" STRSIZE 64;\n",
" STRPAD H5T_STR_NULLTERM;\n",
" CSET H5T_CSET_ASCII;\n",
" CTYPE H5T_C_S1;\n",
" }\n",
" DATASPACE SCALAR\n",
" DATA {\n",
" (0): \"This is a netCDF dimension but not a netCDF variable. 3\"\n",
" }\n",
" }\n",
" ATTRIBUTE \"REFERENCE_LIST\" {\n",
" DATATYPE H5T_COMPOUND {\n",
" H5T_REFERENCE { H5T_STD_REF_OBJECT } \"dataset\";\n",
" H5T_STD_U32LE \"dimension\";\n",
" }\n",
" DATASPACE SIMPLE { ( 1 ) / ( 1 ) }\n",
" DATA {\n",
" (0): {\n",
" DATASET 94412115649136 \"/data\",\n",
" 0\n",
" }\n",
" }\n",
" }\n",
" ATTRIBUTE \"_Netcdf4Dimid\" {\n",
" DATATYPE H5T_STD_I32LE\n",
" DATASPACE SCALAR\n",
" DATA {\n",
" (0): 0\n",
" }\n",
" }\n",
" }\n",
"}\n",
"}\n"
]
}
],
"source": [
"!h5dump complex_auto_h5nc.nc"
]
},
{
"cell_type": "markdown",
"id": "a70ae7c1-c590-4712-a0b7-87c5c369b801",
"metadata": {},
"source": [
"## 1.2.5 Summary\n",
"\n",
"As we can see from the different creation processes, the resulting (equivalent) files for 1.2.1 and 1.2.2 can't be read by netcdf-c/netcdf4-python. Reason is the committed (named type) is missing. If we create the committed type by hand or automatically the file can be read by netcdf-c/netcdf4-python. \n",
"\n",
"All files can be read by h5netcdf perfectly fine.\n"
]
},
{
"cell_type": "markdown",
"id": "989ee476-bf3b-4530-a0e0-9a7fb9c47a82",
"metadata": {},
"source": [
"## 1.3 Conclusion\n",
"\n",
"NetCDF4 files need a committed type (for user types like compound types) to properly connect dataset and datatype. \n",
"\n",
"This seems like a bug in h5netcdf to not write the committed type. \n",
"\n",
"The proper solution would be:\n",
"\n",
"- write the committed type in any case for complex data as it is represented as compound type in h5py/hdf5.\n",
"- if the user does not provide an existing committed type on variable creation -> h5netcdf will create one (aligne with [nc-complex](https://github.com/PlasmaFAIR/nc-complex)\n",
"- h5netcdf will read complex data as given in underlying compound type in any case into complex numpy"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.12.1"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment