In an image recognition task how to deal with unexpected images [closed]

I’m trying to develop an image classifier with Keras, I followed the tutorial on the page: https://www.tensorflow.org/tutorials/images/classification?hl=en

Still using the flower dataset that the tutorial provides, I tried to test the model by making a prediction with a photo of a goat. The result was that the prediction saw that the goat photo belongs to the class of daisies at 70%. I think this is impossible considering also that there are no daisies in the photo of the goat.

This is the code I used :

import matplotlib.pyplot as plt
import numpy as np
import os
import PIL
import tensorflow as tf

from tensorflow import keras
from tensorflow.keras import layers
from tensorflow.keras.models import Sequential

import pathlib

dataset_url = "https://storage.googleapis.com/download.tensorflow.org/example_images/flower_photos.tgz"
data_dir = tf.keras.utils.get_file('flower_photos', origin=dataset_url, untar=True)
data_dir = pathlib.Path(data_dir)

image_count = len(list(data_dir.glob('*/*.jpg')))
print(image_count)

batch_size = 32
img_height = 180
img_width = 180

train_ds = tf.keras.preprocessing.image_dataset_from_directory(
    data_dir,
    validation_split=0.2,
    subset="training",
    seed=123,
    image_size=(img_height, img_width),
    batch_size=batch_size)

val_ds = tf.keras.preprocessing.image_dataset_from_directory(
    data_dir,
    validation_split=0.2,
    subset="validation",
    seed=123,
    image_size=(img_height, img_width),
    batch_size=batch_size)

class_names = train_ds.class_names
print(class_names)


AUTOTUNE = tf.data.experimental.AUTOTUNE

train_ds = train_ds.cache().shuffle(1000).prefetch(buffer_size=AUTOTUNE)
val_ds = val_ds.cache().prefetch(buffer_size=AUTOTUNE)

normalization_layer = layers.experimental.preprocessing.Rescaling(1./255)

normalized_ds = train_ds.map(lambda x, y: (normalization_layer(x), y))
image_batch, labels_batch = next(iter(normalized_ds))
first_image = image_batch[0]
# Notice the pixels values are now in `[0,1]`.
print(np.min(first_image), np.max(first_image))

# MODELLO
data_augmentation = keras.Sequential(
  [
    layers.experimental.preprocessing.RandomFlip("horizontal",
                                                 input_shape=(img_height,
                                                              img_width,
                                                              3)),
    layers.experimental.preprocessing.RandomRotation(0.1),
    layers.experimental.preprocessing.RandomZoom(0.1),
  ]
)

# MODIFICARE IN BASE A QUANTE CLASSI SI HANNO
num_classes = 5

model = Sequential([
  data_augmentation,
  layers.experimental.preprocessing.Rescaling(1./255),
  layers.Conv2D(16, 3, padding='same', activation='relu'),
  layers.MaxPooling2D(),
  layers.Conv2D(32, 3, padding='same', activation='relu'),
  layers.MaxPooling2D(),
  layers.Conv2D(64, 3, padding='same', activation='relu'),
  layers.MaxPooling2D(),
  layers.Dropout(0.2),
  layers.Flatten(),
  layers.Dense(128, activation='relu'),
  layers.Dense(num_classes)
])

model.compile(optimizer='adam',
              loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),
              metrics=['accuracy'])

model.summary()

epochs=15
history = model.fit(
  train_ds,
  validation_data=val_ds,
  epochs=epochs
)

acc = history.history['accuracy']
val_acc = history.history['val_accuracy']

loss = history.history['loss']
val_loss = history.history['val_loss']

epochs_range = range(epochs)

plt.figure(figsize=(8, 8))
plt.subplot(1, 2, 1)
plt.plot(epochs_range, acc, label='Training Accuracy')
plt.plot(epochs_range, val_acc, label='Validation Accuracy')
plt.legend(loc='lower right')
plt.title('Training and Validation Accuracy')

plt.subplot(1, 2, 2)
plt.plot(epochs_range, loss, label='Training Loss')
plt.plot(epochs_range, val_loss, label='Validation Loss')
plt.legend(loc='upper right')
plt.title('Training and Validation Loss')
plt.show()


#sunflower_url = "https://storage.googleapis.com/download.tensorflow.org/example_images/592px-#Red_sunflower.jpg"
#sunflower_path = tf.keras.utils.get_file('Red_sunflower', origin=sunflower_url)

img = keras.preprocessing.image.load_img(
    'C:\Users\matti\PycharmProjects\Tensorflow\goat.jpg', target_size=(img_height, img_width)
)
img_array = keras.preprocessing.image.img_to_array(img)
img_array = tf.expand_dims(img_array, 0) # Create a batch

predictions = model.predict(img_array)
score = tf.nn.softmax(predictions[0])

print(
    "This image most likely belongs to {} with a {:.2f} percent confidence."
    .format(class_names[np.argmax(score)], 100 * np.max(score))
)

I am using tensorflow 2.3.2 version and the other libraries at the last version.

How can I improve the model so that if it receives a photo that is totally different from the classes it used for training, such as a photo that is not a flower, it gives a prediction equal to 0% ?

I need a solution because I am trying to develop a system that having a set of images, for example three classes of a company logo: correct, wrong and acceptable, if it receives a logo it must classify it in the right class. But if it receives a photo that isn’t a company logo it must returns that it doesn’t belong in the logo set.

Answer

Since your model has been trained on a closed set of images, it will take any image of another kind to fit the schemes it learned and often it will succeed. In the end the output probability is nothing else that the probability given certain shapes combinations and colors in an image, not the abstract concept that you expect as an output (for understanding more what a neural network actually sees, you can read this article among others: https://www.kdnuggets.com/2016/06/peeking-inside-convolutional-neural-networks.html)

The solution to your problem is therefore to teach your neural network another class, a suplemental one, consisting of various generic images which are not your target. You can get the images to feed into such a class from ImageNet (http://image-net.org/), for instance. In this way you will have an adversarial output that will fire up when the images you are providing are not among the targets you expect.

Techinically speaking, in relation to the code you have provided, you just have to create a further directory in your training and validation directories, a directory containing mixed images for your extra class “other”.