Check equality with Python dataframe containing None values

I’m comparing a dataframe to a dict as follows…

if not ( (df['Column1'] == a['Column1']) & (df['Column2'] == a['Column2']) ).any():
    print("not in list")

Both my dataframe (df) and dict (a) may contain a None value, but I’ve noticed that when 2 None values are compared, they are not deemed to be equal and so I’m printing “not in list” even though both df and a both hold the same value.

Any thoughts on the best way around this issue would be really appreciated. Perhaps I have to convert the None values to a string that will return True when compared?

Answer

The issue here is that np.NAN == np.NANresolves to False. You can use pd.DataFrame.equals, which considers NaNs in the same locations as being equal. In code that reads like

if df[['Column1', 'Column2']].equals(a[['Column1', 'Column2']]):