Why is my deep learning model predicting very similar but wrong values

So I’ve done some really basic supervised learning in the past and decided to try predictive maintenance and because I am new to this subject I decided to watch some tutorials in the world wide web. After a couple of hours into it i came across this specific tutorial (link down below) in which it is used a dataset from NASA (link down below) which has several values taken from several different engines. My main goal with predictive maintenance was to predict when certain object was about fail, so we could pinpoint it and avoid it by making a eventual repair.

The problem is that in this tutorial he doesn’t make a model nor feeds the data to it so I decided that I should give it a try, but the results are bad despite the accuracy being almost 90%. The reason they are bad is because, like the tutorial suggests, I made a column with a variable called RUL(Remaining Useful Life) which is either 1 or 0 depending on the number of “RULs” (less than 20 it is 1 else it’s 0). The RUL is calculed based of the number of cycles, so each engine starts at cycle 1 and its RUL is the max cycle it achieved minus the number of cycles, so in the next cycle(2) the RUL will be the max_cycle-2. If this doenst make sense to you I apologize in advance. Below you can find all the code that is present in the tutorial as well as my model and attempt.

import matplotlib
import numpy as np
import matplotlib.pyplot as plt
import pandas as pd
from sklearn.model_selection import train_test_split
from keras.models import Sequential
from keras.layers import Dense, Flatten


# http://ti.arc.nasa.gov/c/6/
path = r"C:UsersGinger BreadDesktoparchiveCMAPSSDatatrain_FD001.txt"
raw_data = pd.read_csv(path, sep=' ', header=None)
print(raw_data.head())

# Changing the label name
raw_data = raw_data[[f for f in range(0,26)]]
raw_data.columns = ['ID', 'Cycle', 'OpSet1', 'OpSet2', 'OpSet3', 'SensorMeasure1', 'SensorMeasure2', 'SensorMeasure3', 'SensorMeasure4', 'SensorMeasure5', 'SensorMeasure6', 'SensorMeasure7', 'SensorMeasure8', 'SensorMeasure9', 'SensorMeasure10', 'SensorMeasure11', 'SensorMeasure12', 'SensorMeasure13', 'SensorMeasure14', 'SensorMeasure15', 'SensorMeasure16', 'SensorMeasure17', 'SensorMeasure18', 'SensorMeasure19', 'SensorMeasure20', 'SensorMeasure21']
print(raw_data)

# Checking what is the max number of Cycles for each id engine
max_cycles_df = raw_data.groupby(['ID'], sort=False)['Cycle'].max().reset_index().rename(columns={'Cycle':'MaxCycleID'})
print(max_cycles_df.head())

# merge back to original dataset
FD001_df = pd.merge(raw_data, max_cycles_df, how='inner', on='ID')
FD001_df['RUL'] = FD001_df['MaxCycleID'] - FD001_df['Cycle']
print(FD001_df)
print(FD001_df[['Cycle', 'RUL']])


print('Shape:', FD001_df.shape)
print(list(FD001_df))

# create outcome variable
print(len(FD001_df[FD001_df['RUL'] == 0]))

# plot them
one_engine = []
for i, r in FD001_df.iterrows():
    rul = r['RUL']
    one_engine.append(rul)
    if rul == 0:
        plt.plot(one_engine)
        one_engine = []
plt.show()

# Write to file
FD001_df.to_csv('NASA_Full_FD001.csv', index=None)

# Create a smaller data set for classification
pred_maint_data = FD001_df[['Cycle',
                            'SensorMeasure11',
                            'SensorMeasure4',
                            'SensorMeasure9',
                            'SensorMeasure14',
                            'RUL']]
print(pred_maint_data.head())


# Create a binary outcome - we'll assume 20 days
# Percent of time RUL at 0
pt0 = len(pred_maint_data[pred_maint_data['RUL'] == 0])/len(pred_maint_data) * 100
print('Percent of time RUL at 0:', np.round(pt0,2), '%')

# create outcome variable of 1 or 0
pred_maint_data['RUL'] = [1 if out < 20 else 0 for out in pred_maint_data['RUL']]

pt0 = np.mean(pred_maint_data['RUL']) * 100
print('Percent of time RUL at 0:', np.round(pt0,2), '%')
print(pred_maint_data['RUL'].value_counts())

# save to file
pred_maint_data.to_csv('NASA_20_Cycles_Failure_FD001.csv', index=None)

# End of the tutorial Code
# Below this comment, the code was made by me

# Getting the data from a csv file and dividing it into train and test sets

data = pd.read_csv("NASA_20_Cycles_Failure_FD001.csv")

train = data.drop(['RUL', 'Cycle'], axis=1)
test = data['RUL']
X_train, X_test, y_train, y_test = train_test_split(train, test, test_size=0.2, random_state=2)
print("HERE")
print(X_train)
print("HERE2")
print(y_train)

# Model
model = Sequential()
model.add(Dense(4, init='uniform', input_shape=(4,), activation='relu'))
model.add(Dense(16, init='uniform', activation='relu'))
model.add(Dense(32, init='uniform', activation='relu'))
model.add(Dense(16, init='uniform', activation='relu'))
model.add(Dense(1, init='uniform', activation='sigmoid'))
# Compile model
model.compile(loss='mean_squared_error', optimizer='adam', metrics=['accuracy'])
# Fit the model
history = model.fit(X_train, y_train, epochs=3, verbose=1, validation_split=0.2, shuffle=True)

score = model.evaluate(X_test, y_test, verbose=2)
print(score)

print(history.history.keys())

plt.plot(history.history['val_acc'], label='train')
plt.plot(history.history['val_loss'], label='loss')
plt.legend()
plt.show()

# Seeing the predictions
predictions = model.predict(X_test)

print("Predictions")
for i in predictions:
    print(i)
    if i >= 0.5:
        print("ALELUIA")
print("Real values")
for u in y_test:
    print(u)

I’ve tried to change the number of RUL to be 1 when the value is different than 20 and the accuracy also changes but the predictions still end up being almost identical which makes the predictions round up to either all 0 or all 1. Below I’ll put the results of each prediction as well as the real value (y_test).

Note: I should probably have changed the name of the column RUL to “needs fixing” after the transformation for either 1 or 0 accordingly.

Predictions
[0.1050983]
[0.10404325]
[0.10555065]
[0.10555321]
[0.10344189]
[0.10533005]
[0.10478833]
[0.1051653]
[0.10526326]
[0.10516419]
[0.1038975]
[0.10515215]
[0.10531015]
[0.10511505]
[0.10518976]
[0.1050439]
[0.1051843]
[0.10559492]
[0.10387661]
[0.10374805]
[0.10509399]
[0.10515141]
[0.10521074]
[0.1052593]
[0.10540438]
[0.10533711]
[0.10501852]
[0.10463601]
[0.10503829]
[0.10521385]
[0.10441336]
[0.105217]
[0.10531073]
[0.10491394]
[0.10537319]
[0.10523231]
[0.10557146]
[0.1047653]
[0.10525884]
[0.10500285]
[0.10505348]
[0.10541109]
[0.10533073]
[0.10517488]
[0.1050622]
[0.10527248]
[0.1051841]
[0.10544233]
[0.10554202]
[0.10465073]
[0.10464738]
[0.10524271]
[0.10522015]
[0.10502062]
[0.10546359]
[0.10409129]
[0.10522158]
[0.1051973]
[0.10548837]
[0.10525122]
[0.10466116]
[0.10534483]
[0.10547991]
[0.10514109]
[0.10531491]
[0.10518854]
[0.10496807]
[0.10505583]
[0.10516969]
[0.10521539]
[0.10524863]
[0.10527457]
[0.10532214]
[0.10324776]
[0.10544115]
[0.10540842]
[0.10515314]
[0.10515888]
[0.10483669]
[0.10501605]
[0.104861]
[0.10495911]
[0.10525009]
[0.10529111]
[0.10487738]
[0.10509273]
[0.10518468]
[0.10503055]
[0.10417407]
[0.10479013]
[0.10534652]
[0.10522777]
[0.10319146]
[0.10549013]
[0.10530148]
[0.10484377]
[0.10440563]
[0.10472738]
[0.10502187]
[0.10525076]
[0.10510702]
[0.10532416]
[0.10381923]
[0.1054415]
[0.1054302]
[0.10558783]
[0.10514364]
[0.10523658]
[0.10515053]
[0.10526689]
[0.10520654]
[0.10515639]
[0.10531743]
[0.10403327]
[0.10531391]
[0.10477831]
[0.10548837]
[0.10522526]
[0.10505868]
[0.10418756]
[0.1053642]
[0.10524843]
[0.10497409]
[0.10368675]
[0.10495283]
[0.10515259]
[0.10517113]
[0.1049794]
[0.10523032]
[0.10518576]
[0.1050372]
[0.10507836]
[0.10503409]
[0.10540656]
[0.10431397]
[0.10471103]
[0.10531015]
[0.10523356]
[0.10539803]
[0.10527551]
[0.10513147]
[0.10511173]
[0.10527663]
[0.10504944]
[0.10528576]
[0.10536461]
[0.10493658]
[0.10516817]
[0.10531207]
[0.10546369]
[0.10419846]
[0.10411397]
[0.10465484]
[0.10431477]
[0.10501323]
[0.10528364]
[0.10537661]
[0.10482613]
[0.10522176]
[0.10495919]
[0.10519546]
[0.10520439]
[0.10508794]
[0.10462283]
[0.10532165]
[0.10358283]
[0.10522396]
[0.105164]
[0.10544351]
[0.10536326]
[0.10539526]
[0.10498454]
[0.10327268]
[0.10522733]
[0.10537484]
[0.10505765]
[0.10524735]
[0.10499822]
[0.10508626]
[0.10388456]
[0.1051572]
[0.10503815]
[0.10507942]
[0.10505755]
[0.10536408]
[0.105366]
[0.10546079]
[0.10522104]
[0.10485433]
[0.10520129]
[0.10539605]
[0.10529195]
[0.10493235]
[0.10515185]
[0.10532335]
[0.10540719]
[0.10522908]
[0.10535663]
[0.10558803]
[0.10517517]
[0.10549721]
[0.10537575]
[0.10507962]
[0.10530768]
[0.10452699]
[0.10524847]
[0.10516036]
[0.10510545]
[0.10540862]
[0.10508175]
[0.10522414]
[0.10484162]
[0.10526936]
[0.10518522]
[0.10485084]
[0.10538556]
[0.10537019]
[0.10549224]
[0.10515653]
[0.10519943]
[0.10512082]
[0.10544203]
[0.10487756]
[0.10497619]
[0.10518141]
[0.10533778]
[0.10509621]
[0.10531738]
[0.10505059]
[0.10514385]
[0.10523034]
[0.10531139]
[0.10385931]
[0.1056235]
[0.10516278]
[0.10499334]
[0.10527273]
[0.10524946]
[0.10525396]
[0.10490155]
[0.10441212]
[0.10487241]
[0.10538799]
[0.1049312]
[0.10491467]
[0.10517243]
[0.10535877]
[0.10543416]
[0.10509314]
[0.1049196]
[0.10501852]
Real values
0
0
0
0
1
0
0
0
0
0
0
0
0
0
0
0
0
0
1
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
1
1
0
0
0
0
0
0
0
0
1
0
0
1
0
0
0
0
0
0
0
0
0
0
0
0
1
0
0
0
0
0
0
0
0
0
0
0
0
0
0
1
0
1
0
0
0
0
0
0
0
0
0
1
0
0
1
0
0
0
0
0
0
0
1
0
0
0
0
0
0
0
0
0
1
0
0
0
0
0
0
1
0
0
1
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
1
0
0
0
0
0
1
0
1
0
0
0
0
0
0
1
0
0
0
0
0
1
1
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
1
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
1
0
0
0
0
0
1
0
0
0
0
0
0
0
0
0
1
0
0
0
0
0
1
0
1
0
0
0
0
0
0
0
0
0
0

As you can see it gives me always numbers near 0 which make the model seem to have a good accuracy since there is a lot of 0s in the y_test but it fails in the most important ones which are the 1s which tell me when the engine needs to be fixed! If you know what’s happening and would like to help me I would be very much appreciated!

Thanks in advance.

Note2: The amount of predictions and y_test values showed was drastically reduced in orded to fit in the body text.

Helpful links:

Youtube Video – Tutorial

Raw Code – Tutorial

Dataset – NASA

Versions:

Windows 10 Keras 2.2.0 Tensorflow 1.12.0

Answer

At the moment, you’re using 'mean_squared_error' loss. MSE is good for when you’re trying to predict an analog number. However, your ground truth here is a binary prediction: either 0 or 1. In this case, you should be trying to predict a category, not a real value. For that, you want binary cross-entropy. So, change loss='mean_squared_error' to loss='binary_crossentropy'.

Additionally, it seems that you might not be training it for long enough, and with a batch size too large. If you don’t specify batch size in model.fit, I believe that it defaults to a batch size equal to the whole dataset. This would mean that your model’s parameters get updated only once per epoch. At the moment, you have it set to only train for 3 epochs. The next thing I would try would be setting a batch size of maybe 32, and running it for many more epochs, e.g. maybe 10 to 100 depending on how big the dataset is. So you should have something like

history = model.fit(X_train, y_train, epochs=30, batch_size=32, ...