I am curious why a simple concatenation of two dataframes in pandas:
initId.shape # (66441, 1) initId.isnull().sum() # 0 ypred.shape # (66441, 1) ypred.isnull().sum() # 0
of the same shape and both without NaN values
foo = pd.concat([initId, ypred], join='outer', axis=1) foo.shape # (83384, 2) foo.isnull().sum() # 16943
can result in a lot of NaN values if joined.
How can I fix this problem and prevent NaN values being introduced? Trying to reproduce it like
aaa = pd.DataFrame([0,1,0,1,0,0], columns=['prediction']) bbb = pd.DataFrame([0,0,1,0,1,1], columns=['groundTruth']) pd.concat([aaa, bbb], axis=1)
failed e.g. worked just fine as no NaN values were introduced.
Advertisement
Answer
I think there is problem with different index values, so where concat
cannot align get NaN
:
aaa = pd.DataFrame([0,1,0,1,0,0], columns=['prediction'], index=[4,5,8,7,10,12]) print(aaa) prediction 4 0 5 1 8 0 7 1 10 0 12 0 bbb = pd.DataFrame([0,0,1,0,1,1], columns=['groundTruth']) print(bbb) groundTruth 0 0 1 0 2 1 3 0 4 1 5 1 print (pd.concat([aaa, bbb], axis=1)) prediction groundTruth 0 NaN 0.0 1 NaN 0.0 2 NaN 1.0 3 NaN 0.0 4 0.0 1.0 5 1.0 1.0 7 1.0 NaN 8 0.0 NaN 10 0.0 NaN 12 0.0 NaN
Solution is reset_index
if indexes values are not necessary:
aaa.reset_index(drop=True, inplace=True) bbb.reset_index(drop=True, inplace=True) print(aaa) prediction 0 0 1 1 2 0 3 1 4 0 5 0 print(bbb) groundTruth 0 0 1 0 2 1 3 0 4 1 5 1 print (pd.concat([aaa, bbb], axis=1)) prediction groundTruth 0 0 0 1 1 0 2 0 1 3 1 0 4 0 1 5 0 1
EDIT: If need same index like aaa
and length of DataFrames is same use:
bbb.index = aaa.index print (pd.concat([aaa, bbb], axis=1)) prediction groundTruth 4 0 0 5 1 0 8 0 1 7 1 0 10 0 1 12 0 1