How to draw the precision-recall curve for a segmentation model?

I am using an U-Net for segmenting my data of interest. The masks are grayscale and of size (256,256,1). There are 80 images in the test set. The test images (X_ts) and their respective ground-truth masks (Y_ts) are constructed, saved, and loaded like this:

from tqdm import tqdm
im_width = 256
im_height = 256
ids_test = next(os.walk("data/test/image"))[2] 
print("No. of images = ", len(ids_test))
X_ts = np.zeros((len(ids_test), im_height, im_width, 3), dtype=np.float32)
Y_ts = np.zeros((len(ids_test), im_height, im_width, 1), dtype=np.float32)
print(X_ts.shape)
print(Y_ts.shape)

for n, id_ in tqdm(enumerate(ids_test), total=len(ids_test)):
    # Load images
    img = load_img("data/test/image/"+id_, 
                   color_mode = "rgb")
    x_img = img_to_array(img)
    x_img = resize(x_img, (256,256,3), 
                    mode = 'constant', preserve_range = True)
    # Load masks
    mask = img_to_array(load_img("data/test/label/"+id_, 
                                 color_mode = "grayscale"))
    mask = resize(mask, (256,256,1), 
                  mode = 'constant', preserve_range = True)
    # Save images
    X_ts[n] = x_img/255.0
    Y_ts[n] = mask/255.0 

np.save('./data/X_ts.npy',X_ts)
np.save('./data/Y_ts.npy',Y_ts)

#load the data
X_ts = np.load('./data/X_ts.npy') 
Y_ts = np.load('./data/Y_ts.npy')

The shape of Y_ts (ground truth) is therefore (80,256,256,1) and these are of type “Array of type float32”. I predicted the masks using the trained model like this:

Y_ts_pred = model.predict(X_ts,batch_size=1)
print(Y_ts_pred.shape)
threshold=0.5
Y_ts_pred[Y_ts_pred<threshold]=0
Y_ts_pred[Y_ts_pred>=threshold]=1

The shape of Y_ts_pred is (80,256,256,1) and these are of type “Array of type float32”. Now, I compute the precision recall curves using the Y_ts (ground truth masks) and Y_ts_pred (predicted masks) like this:

from sklearn.metrics import precision_recall_curve
import numpy as np
import matplotlib.pyplot as plt

precision, recall, thresholds = precision_recall_curve(Y_ts, Y_ts_pred)
precision = np.fliplr([precision])[0]  #to avoid getting negative AUC)
recall = np.fliplr([recall])[0]  #to avoid getting negative AUC)
AUC_prec_rec = np.trapz(precision,recall)
print "nArea under Precision-Recall curve: " +str(AUC_prec_rec)
prec_rec_curve = plt.figure()
plt.plot(recall,precision,'-',label='Area Under the Curve (AUC = %0.4f)' % AUC_prec_rec)
plt.title('Precision - Recall curve')
plt.xlabel("Recall")
plt.ylabel("Precision")
plt.legend(loc="lower right")
plt.savefig("Precision_recall.png")

I get the following error:

  File "/tmp/ipykernel_2425796/385729104.py", line 1, in <module>
    precision, recall, thresholds = precision_recall_curve(Y_ts, Y_ts_pred)

  File "/home/anaconda3/envs/tf262/lib/python3.7/site-packages/sklearn/metrics/_ranking.py", line 859, in precision_recall_curve
    y_true, probas_pred, pos_label=pos_label, sample_weight=sample_weight

  File "/home/anaconda3/envs/tf262/lib/python3.7/site-packages/sklearn/metrics/_ranking.py", line 731, in _binary_clf_curve
    raise ValueError("{0} format is not supported".format(y_type))

ValueError: unknown format is not supported

Answer

I have to convert my ground truth and predicted masks into one-dimensional vectors and binarize them using a threshold value like below. Then, the above code will work.

Y_ts_pred = Y_ts_pred.reshape(Y_ts_pred.shape[0]*Y_ts_pred.shape[1]*Y_ts_pred.shape[2], 1)
print(Y_ts_pred.shape)

Y_ts = Y_ts.reshape(Y_ts.shape[0]*Y_ts.shape[1]*Y_ts.shape[2], 1)
print(Y_ts.shape)

Y_ts_pred = np.where(Y_ts_pred>0.5, 1, 0)
Y_ts  = np.where(Y_ts>0.5, 1, 0)