Pyart cloud mask hugely overestimating the cloud amount

Data
1 
import pyart
import numpy as np

radar_full = pyart.io.read(“DLI241228000230-IMD-B_corrected.nc”)
radar = radar_full.extract_sweeps(\[0\])

Check for masked values and get valid data statistics

import numpy.ma as ma
field_name = ‘corrected_reflectivity’

reflectivity_data = radar.fields\[field_name\]\[‘data’\]
valid_data = reflectivity_data\[\~ma.getmaskarray(reflectivity_data)\]

print(“Valid reflectivity data statistics:”)
print("Count of valid gates: " + str(len(valid_data)))
print("Count of masked gates: " + str(ma.getmaskarray(reflectivity_data).sum()))
print("Min: " + str(np.min(valid_data)))
print("Max: " + str(np.max(valid_data)))
print("Mean: " + str(np.mean(valid_data)))
print("Median: " + str(np.median(valid_data)))
print("Percentiles (10, 25, 50, 75, 90): " + str(np.percentile(valid_data, \[10, 25, 50, 75, 90\])))

# Any real weather echo that is weaker than the noise floor is essentially invisible to the radar

from pyart.correct.bias_and_noise import calc_noise_floor

field_name = ‘corrected_reflectivity’

# Calculate noise floor for the field

noise = calc_noise_floor(radar, field_name, height=‘gate_z’)
print(“Noise floor statistics:”)
print("Min: " + str(np.nanmin(noise)))
print("Max: " + str(np.nanmax(noise)))
print("Mean: " + str(np.nanmean(noise)))
print("Median: " + str(np.nanmedian(noise)))

calc_noise_floor gives out 360 noise floor values, one for each azimuth direction

Standaard operational weather radars usually have noise floor less than 0 dBZ.

noise
>>array(\[        inf,         inf,         inf,         inf,         inf,
               inf,         inf,         inf,         inf,         inf,
               inf,         inf,         inf,         inf,         inf,
               inf,         inf,         inf,         inf,         inf,
               inf,         inf,         inf,         inf,         inf,
               inf,         inf,         inf,         inf,         inf,
               inf,         inf,         inf,         inf,         inf,
               inf,         inf,         inf,         inf,         inf,
               inf,         inf,         inf,         inf,         inf,
               inf,         inf,         inf,         inf,         inf,
               inf,         inf,         inf,         inf,         inf,
               inf,         inf,         inf,         inf,         inf,
               inf,         inf,         inf,         inf,         inf,
               inf,         inf,         inf,         inf,         inf,
               inf,         inf,         inf,         inf,         inf,
               inf,         inf,         inf,         inf,         inf,
               inf,         inf,         inf,         inf,         inf,
               inf,         inf,         inf,         inf,         inf,
               inf,         inf,         inf,         inf,         inf,
               inf,         inf,         inf,         inf,         inf,
               inf,         inf,         inf,         inf,         inf,
               inf,         inf,         inf,         inf,         inf,
               inf,         inf,         inf,         inf,         inf,
               inf,         inf,         inf,         inf,         inf,
               inf,         inf,         inf,         inf,         inf,
               inf,         inf,         inf,         inf,         inf,
               inf,         inf,         inf,         inf,         inf,
               inf,         inf,         inf,         inf,         inf,
               inf,         inf,         inf,         inf,         inf,
               inf,         inf,         inf,         inf,         inf,
               inf,         inf,         inf,         inf,         inf,
               inf,         inf,         inf,         inf,         inf,
               inf,         inf,         inf,         inf,         inf,
               inf,         inf,         inf,         inf,         inf,
               inf,         inf,         inf,         inf,         inf,
               inf,         inf,         inf,         inf,         inf,
               inf,         inf,         inf,         inf,         inf,
               inf,         inf,         inf,         inf,         inf,
               inf,         inf,         inf,         inf,         inf,
               inf,         inf,         inf,         inf,         inf,
               inf,         inf,         inf,         inf,         inf,
               inf,         inf,         inf,         inf,         inf,
               inf,         inf,         inf,         inf,         inf,
               inf,         inf,         inf,         inf,         inf,
               inf,         inf,         inf,         inf,         inf,
               inf,         inf,         inf,         inf,         inf,
               inf,         inf,         inf,         inf,         inf,
               inf,         inf,         inf,         inf,         inf,
               inf,         inf,         inf,         inf,         inf,
               inf,         inf,         inf,         inf,         inf,
               inf,         inf,         inf,         inf,         inf,
               inf,         inf,         inf,         inf,         inf,
               inf,         inf,         inf,         inf,         inf,
               inf,         inf,         inf,         inf,         inf,
               inf,         inf,         inf,         inf,         inf,
               inf,         inf,         inf,         inf,         inf,
               inf,         inf,         inf,         inf,         inf,
               inf,         inf,         inf,         inf,         inf,
               inf,         inf,         inf,         inf,         inf,
               inf,         inf,         inf,         inf,         inf,
               inf,         inf,         inf,         inf,         inf,
               inf,         inf,         inf,         inf,         inf,
               inf,         inf,         inf,         inf,         inf,
               inf,         inf,         inf,         inf,         inf,
               inf,         inf,         inf,         inf,         inf,
               inf,         inf,         inf,         inf,         inf,
               inf,         inf,         inf,         inf,         inf,
               inf,         inf,         inf,         inf,         inf,
               inf,         inf,         inf,         inf,         inf,
               inf,         inf,         inf,         inf, 24.30768541,
       20.40761053,         inf,         inf,         inf,         inf,
               inf,         inf,         inf,         inf,         inf\])

Get range-corrected reflectivity data to compare with noise threshold

# This is done because calc_noise_floor determines the noise level from the range-corrected reflectivity.

from pyart.correct.bias_and_noise import range_correction
range_corrected_Z = range_correction(radar, field_name, height=‘gate_z’)
print(“Range-corrected reflectivity statistics:”)
print("Min: " + str(np.nanmin(range_correcetd_Z)))
print("Max: " + str(np.nanmax(range_correcetd_Z)))
print("Mean: " + str(np.nanmean(range_correcetd_Z)))
print("Median: " + str(np.nanmedian(range_correcetd_Z)))
print("Shape: " + str(range_correcetd_Z.shape))


# Add range-corrected reflectivity as a field to the radar object 

range_corrected_field = {
‘data’: range_corrected_Z,
‘units’: ‘dB’,
‘long_name’: ‘Range-corrected reflectivity’,
‘standard_name’: ‘range_corrected_reflectivity’
}

radar.add_field(‘range_corrected_Z’, range_corrected_field, replace_existing=True)
print(“Range-corrected reflectivity field added to radar”)

field_name =‘range_corrected_Z’
from pyart.correct import calc_cloud_mask

radar = calc_cloud_mask(radar, field_name, height=‘gate_z’, noise_threshold=10.0, threshold_offset=5.0, counts_threshold=12)
print(“Cloud mask calculation completed”)

# Check what fields were added after cloud mask calculation

print(“Fields in radar after cloud mask:”)
print(list(radar.fields.keys()))

Fields in radar after cloud mask:
['T', 'Z', 'V', 'W', 'ZDR', 'KDP', 'PHIDP', 'SQI', 'RHOHV', 'HCLASS', 'temperature', 'specific_attenuation', 'path_integrated_attenuation', 'corrected_reflectivity', 'specific_differential_attenuation', 'path_integrated_differential_attenuation', 'corrected_differential_reflectivity', 'range_corrected_Z', 'cloud_mask_1', 'cloud_mask_2'\]

Extract cloud mask fields and identify cloud pixels

import numpy as np

# Get the cloud mask data

cloud_mask_1 = radar.fields\[‘cloud_mask_1’\]\[‘data’\]
cloud_mask_2 = radar.fields\[‘cloud_mask_2’\]\[‘data’\]

print(“cloud mask1 min value:” + str(cloud_mask_1.min()))
print(“cloud mask1 max value:” + str(cloud_mask_1.max()))

print(“cloud mask2 min value:” + str(cloud_mask_2.min()))
print(“cloud mask2 max value:” + str(cloud_mask_2.max()))

Convert to boolean arrays (cloud pixels are where mask >= 1)

cloud_pixels_1 = np.array(cloud_mask_1) >= 1
cloud_pixels_2 = np.array(cloud_mask_2) >= 1

# Summary statistics

cloud_gates_1 = int(cloud_pixels_1.sum())
cloud_gates_2 = int(cloud_pixels_2.sum())
total_gates = 831\*360

print("Total radar gates: " + str(total_gates))
print("Cloud gates (mask_1): " + str(cloud_gates_1))
print("Cloud gates (mask_2): " + str(cloud_gates_2))
print("Cloud fraction (mask_1): " + str(round(cloud_gates_1/total_gates \* 100, 2)) + “%”)
print("Cloud fraction (mask_2): " + str(round(cloud_gates_2/total_gates \* 100, 2)) + “%”)

Total radar gates: 299160 
Cloud gates (mask_1): 296364 
Cloud gates (mask_2): 294828 
Cloud fraction (mask_1): 99.07% 
Cloud fraction (mask_2): 98.55%

Both the cloud masks are showing almost 98% or 99% of the radar volume as cloud. But that cannot be! All the radar volume cannot be cloud. Even in extreme cases, less than 30% of radar volume should correspond to cloud. Can anyone help me in debugging as to why my code is identifying almost all the radar pixels as cloud?

2

@MohitDahliya Thanks for posting your cloud mask problem. I’m sure Py-ART devs will have a look at this soon.

Some guidance for proper formatting your post. Please wrap code in three backticks “```”. You can use the “</>” button from the editor menu to pre-format it, it will look like this in the editor:

```
type or paste code here
```

If you want python highlighting just add python after the backticks:

```python
type or paste code here
```

This will render like this:

type or paste code here

I’ve tried to fix this for your post, but please have a look if this fits. Another possible take would be to link a github gist here, containing the whole notebook.

HTH,
Kai

3

Okay, I’ll take a note of it.

4

I responded in the GItHub issue tracker