Xarray to netcdf (ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all())

General
1

Hello People,

Does anyone come across this type of problem while saving xarray dataset to netcdf file?

<xarray.Dataset>
Dimensions:  (time: 17518, y: 900, x: 900)
Coordinates:
  * time     (time) datetime64[ns] 2006-01-01T00:45:00 ... 2021-12-31T23:50:00
  * y        (y) float64 -4.658e+06 -4.657e+06 ... -3.76e+06 -3.759e+06
  * x        (x) float64 -5.23e+05 -5.22e+05 -5.21e+05 ... 3.75e+05 3.76e+05
Data variables:
    RW       (time, y, x) float32 dask.array<chunksize=(1, 900, 900), meta=np.ndarray>
Attributes:
    radarid:         10000
    formatversion:   2
    radolanversion:  01.01.00
    radarlocations:  ['bln', 'drs', 'eis', 'emd', 'ess', 'fbg', 'fld', 'fra',...

When I tried to save xarray dataset to netcdf like below. I get the following error.

ds.to_netcdf("D:/Sandeep/Thesis/Data/dsM.nc")

Error:

ds.to_netcdf("D:/Sandeep/Thesis/Data/dsM.nc", engine="h5netcdf", invalid_netcdf=True)
Traceback (most recent call last):

  File "C:\Users\Admin\AppData\Local\Temp\ipykernel_7676\1512561570.py", line 1, in <cell line: 1>
    ds.to_netcdf("D:/Sandeep/Thesis/Data/dsM.nc", engine="h5netcdf", invalid_netcdf=True)

  File "C:\Users\Admin\anaconda3\envs\wradlib\lib\site-packages\xarray\core\dataset.py", line 1899, in to_netcdf
    return to_netcdf(  # type: ignore  # mypy cannot resolve the overloads:(

  File "C:\Users\Admin\anaconda3\envs\wradlib\lib\site-packages\xarray\backends\api.py", line 1230, in to_netcdf
    dump_to_store(

  File "C:\Users\Admin\anaconda3\envs\wradlib\lib\site-packages\xarray\backends\api.py", line 1277, in dump_to_store
    store.store(variables, attrs, check_encoding, writer, unlimited_dims=unlimited_dims)

  File "C:\Users\Admin\anaconda3\envs\wradlib\lib\site-packages\xarray\backends\common.py", line 266, in store
    variables, attributes = self.encode(variables, attributes)

  File "C:\Users\Admin\anaconda3\envs\wradlib\lib\site-packages\xarray\backends\common.py", line 355, in encode
    variables, attributes = cf_encoder(variables, attributes)

  File "C:\Users\Admin\anaconda3\envs\wradlib\lib\site-packages\xarray\conventions.py", line 868, in cf_encoder
    new_vars = {k: encode_cf_variable(v, name=k) for k, v in variables.items()}

  File "C:\Users\Admin\anaconda3\envs\wradlib\lib\site-packages\xarray\conventions.py", line 868, in <dictcomp>
    new_vars = {k: encode_cf_variable(v, name=k) for k, v in variables.items()}

  File "C:\Users\Admin\anaconda3\envs\wradlib\lib\site-packages\xarray\conventions.py", line 273, in encode_cf_variable
    var = coder.encode(var, name=name)

  File "C:\Users\Admin\anaconda3\envs\wradlib\lib\site-packages\xarray\coding\variables.py", line 169, in encode
    if not pd.isnull(fill_value):

ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()


ds.RW.to_netcdf("D:/Sandeep/Thesis/Data/dsM.nc", engine="h5netcdf", invalid_netcdf=True)
Traceback (most recent call last):

  File "C:\Users\Admin\AppData\Local\Temp\ipykernel_7676\1590374310.py", line 1, in <cell line: 1>
    ds.RW.to_netcdf("D:/Sandeep/Thesis/Data/dsM.nc", engine="h5netcdf", invalid_netcdf=True)

  File "C:\Users\Admin\anaconda3\envs\wradlib\lib\site-packages\xarray\core\dataarray.py", line 3752, in to_netcdf
    return to_netcdf(  # type: ignore  # mypy cannot resolve the overloads:(

  File "C:\Users\Admin\anaconda3\envs\wradlib\lib\site-packages\xarray\backends\api.py", line 1230, in to_netcdf
    dump_to_store(

  File "C:\Users\Admin\anaconda3\envs\wradlib\lib\site-packages\xarray\backends\api.py", line 1277, in dump_to_store
    store.store(variables, attrs, check_encoding, writer, unlimited_dims=unlimited_dims)

  File "C:\Users\Admin\anaconda3\envs\wradlib\lib\site-packages\xarray\backends\common.py", line 266, in store
    variables, attributes = self.encode(variables, attributes)

  File "C:\Users\Admin\anaconda3\envs\wradlib\lib\site-packages\xarray\backends\common.py", line 355, in encode
    variables, attributes = cf_encoder(variables, attributes)

  File "C:\Users\Admin\anaconda3\envs\wradlib\lib\site-packages\xarray\conventions.py", line 868, in cf_encoder
    new_vars = {k: encode_cf_variable(v, name=k) for k, v in variables.items()}

  File "C:\Users\Admin\anaconda3\envs\wradlib\lib\site-packages\xarray\conventions.py", line 868, in <dictcomp>
    new_vars = {k: encode_cf_variable(v, name=k) for k, v in variables.items()}

  File "C:\Users\Admin\anaconda3\envs\wradlib\lib\site-packages\xarray\conventions.py", line 273, in encode_cf_variable
    var = coder.encode(var, name=name)

  File "C:\Users\Admin\anaconda3\envs\wradlib\lib\site-packages\xarray\coding\variables.py", line 169, in encode
    if not pd.isnull(fill_value):

ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()


ds['RW'][0].plot()
Out[46]: <matplotlib.collections.QuadMesh at 0x1eb4091f250>

ds
Out[47]: 
<xarray.Dataset>
Dimensions:  (time: 17518, y: 900, x: 900)
Coordinates:
  * time     (time) datetime64[ns] 2006-01-01T00:45:00 ... 2021-12-31T23:50:00
  * y        (y) float64 -4.658e+06 -4.657e+06 ... -3.76e+06 -3.759e+06
  * x        (x) float64 -5.23e+05 -5.22e+05 -5.21e+05 ... 3.75e+05 3.76e+05
Data variables:
    RW       (time, y, x) float32 dask.array<chunksize=(1, 900, 900), meta=np.ndarray>
Attributes:
    radarid:         10000
    formatversion:   2
    radolanversion:  01.01.00
    radarlocations:  ['bln', 'drs', 'eis', 'emd', 'ess', 'fbg', 'fld', 'fra',...

ds.to_netcdf("D:/Sandeep/Thesis/Data/dsM.nc")
Traceback (most recent call last):

  File "C:\Users\Admin\AppData\Local\Temp\ipykernel_7676\2740476879.py", line 1, in <cell line: 1>
    ds.to_netcdf("D:/Sandeep/Thesis/Data/dsM.nc")

  File "C:\Users\Admin\anaconda3\envs\wradlib\lib\site-packages\xarray\core\dataset.py", line 1899, in to_netcdf
    return to_netcdf(  # type: ignore  # mypy cannot resolve the overloads:(

  File "C:\Users\Admin\anaconda3\envs\wradlib\lib\site-packages\xarray\backends\api.py", line 1230, in to_netcdf
    dump_to_store(

  File "C:\Users\Admin\anaconda3\envs\wradlib\lib\site-packages\xarray\backends\api.py", line 1277, in dump_to_store
    store.store(variables, attrs, check_encoding, writer, unlimited_dims=unlimited_dims)

  File "C:\Users\Admin\anaconda3\envs\wradlib\lib\site-packages\xarray\backends\common.py", line 266, in store
    variables, attributes = self.encode(variables, attributes)

  File "C:\Users\Admin\anaconda3\envs\wradlib\lib\site-packages\xarray\backends\common.py", line 355, in encode
    variables, attributes = cf_encoder(variables, attributes)

  File "C:\Users\Admin\anaconda3\envs\wradlib\lib\site-packages\xarray\conventions.py", line 868, in cf_encoder
    new_vars = {k: encode_cf_variable(v, name=k) for k, v in variables.items()}

  File "C:\Users\Admin\anaconda3\envs\wradlib\lib\site-packages\xarray\conventions.py", line 868, in <dictcomp>
    new_vars = {k: encode_cf_variable(v, name=k) for k, v in variables.items()}

  File "C:\Users\Admin\anaconda3\envs\wradlib\lib\site-packages\xarray\conventions.py", line 273, in encode_cf_variable
    var = coder.encode(var, name=name)

  File "C:\Users\Admin\anaconda3\envs\wradlib\lib\site-packages\xarray\coding\variables.py", line 169, in encode
    if not pd.isnull(fill_value):

ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()
2

Could you please have a look at the contents of ds.RW.encoding? As the RADOLAN data has three values for _FillValue (which get mapped to NaN) this is OK for reading, but not for writing. You would need to only keep one of those _FillValue.

3 
ds.RW.encoding
Out[149]: 
{'source': 'D:\\Sandeep\\Thesis\\Data\\BIN\\Extracted\\raa01-rw_10000-0601010050-dwd---bin.gz',
 '_FillValue': array([ 2490,  2500, 65535]),
 'scale_factor': 0.1,
 'add_offset': 0.0,
 'dtype': dtype('uint16'),
 'coordinates': 'time y x'}
4

You would want to only use one of those fill values instead of all three. For example,

ds.encoding['_FillValue'] = ds.encoding._FillValue[:1]
5

I’d propably pick 65535 in that case. But that’s just my personal view.

1 Like