I am working on a multimodal deep learning classifiers with RGB-D images. i have developed two seperate models for each case. The first one is a LSTM with CNN in the begining for the RGB images with shape (3046,200,200,3)
, and the second one is an LSTM for the depth images with shape (3046,200,200)
. I’m trying to figure out how i can do Ensemble-with-voting.
I have done the bellow code tha finds the most voted labels but seems not working well. Is there any another way to do late fusion to the models?
labels = [] loaded_model = keras.models.load_model('Saved/LSTM_RGB/LSTMRGB.h5') predictsRGB = np.argmax(loaded_model.predict(x_testRGB), axis=1) labels.append(predictsRGB) loaded_model = keras.models.load_model('Saved/LSTM_DEPTH/LSTMDEPTH.h5') predictsDepth = np.argmax(loaded_model.predict(x_testDepth), axis=1) labels.append(predictsDepth) print(labels) # Ensemble with voting labels = np.array(labels) labels = np.transpose(labels, (1, 0)) print(labels,labels.shape) labels = scipy.stats.mode(labels)[0] labels = np.squeeze(labels) print(labels) print(labels.shape)
Advertisement
Answer
You have two classifier, in that cases directly voting does not make sense because how to resolve the ties?
Since you are doing np.argmax on models’ prediction, I believe the models output probabilities as a prediction. If you can not introduce a third model, you can average out these probabilities and then take argmax.
As an example: Model 1 says class 1 with prob 0.9 and class 0 with probability 0.1 whereas model 2 says class 1 with probability 0.3 and class 0 with 0.7.
The averaged probabilities would be: Class1: 0.6 Class0: 0.4 and you would go for class1. In a way, it makes sense since model 1 was much more sure about its prediction than the model2