A quick way to copy variables and metadata from one netcdf file to another using netcdf4-python

Converter.py

# -*- coding: utf-8 -*-
from netCDF4 import Dataset

#input file
dsin = Dataset("crop.nc")

#output file
dsout = Dataset("crop.nc3", "w", format="NETCDF3_CLASSIC")

#Copy dimensions
for dname, the_dim in dsin.dimensions.iteritems():
    print dname, len(the_dim)
    dsout.createDimension(dname, len(the_dim) if not the_dim.isunlimited() else None)


# Copy variables
for v_name, varin in dsin.variables.iteritems():
    outVar = dsout.createVariable(v_name, varin.datatype, varin.dimensions)
    print varin.datatype
    
    # Copy variable attributes
    outVar.setncatts({k: varin.getncattr(k) for k in varin.ncattrs()})
    
    outVar[:] = varin[:]
# close the output file
dsout.close()

I need my file in netCDF4 classic. It seems .setncatts won't work with netCDF4 Classic, is there a work around.

Hi @ksaha79:

I doubt the problem is with the netCDF4 library. I am guessing your python version might not support dict comprehensions.

Does this work?

outVar.setncatts({"test": "test-value"})

If yes, then you would need to replace the line with:

tmp = {}

for k in varin.ncattrs():
    tmp[k] = varin.getncattr(k)

outVar.setncatts(tmp)

Otherwise, I would need a minimal example demonstrating the problem, to be able to help you further.

Cheers

Adding the four lines again works fine with NETCDF3_CLASSIC, however, when I try to use the same for NETCDF4_CLASSIC format I get the following error:

lat 720
lon 1440
nv 2
time 1
lat float32
lon float32
time int32
lat_bnds float32
lon_bnds float32
analysed_sst int16
Traceback (most recent call last):
File "toto.py", line 51, in
outVar.setncatts(tmp)
File "netCDF4/_netCDF4.pyx", line 4153, in netCDF4._netCDF4.Variable.setncatts
File "netCDF4/_netCDF4.pyx", line 1617, in netCDF4._netCDF4._set_att
File "netCDF4/_netCDF4.pyx", line 1887, in netCDF4._netCDF4._ensure_nc_success
AttributeError: NetCDF: Attempt to define fill value when data already exists.

Strange is the code stops suddenly when the Fillvalue of "analysed_sst" is about to be filled and gives an error. Till then all the variable attributes are fine. Also rest of the two more variables also is not copied in the new file.

I have just tested a python3 version of the script on a file I have:

https://github.com/guziy/PyNotebooks/blob/master/netcdf/test_copy.ipynb

Is it possible for you to share the problematic file? Just a small part of it that causes the error.

The input files I am using can be found here: ftp://ftp.nodc.noaa.gov/pub/data.nodc/ghrsst/GDS2/L4/GLOB/NCEI/AVHRR_OI/v2/2020/085/20200325120000-NCEI-L4_GHRSST-SSTblend-AVHRR_OI-GLOB-v02.0-fv02.0.nc

I have updated the example, by adding a special treatment for the _FillValue attribute.

https://github.com/guziy/PyNotebooks/blob/master/netcdf/test_copy.ipynb

I wanted to point out that there is maybe a more elegant way of doing it now with xarray:

with xarray.open_dataset("test.nc") as ds:
       # do things with ds1 = f(ds)
       ds1.to_netcdf("copy_test.nc")

Thanks That worked for me.
Cheers