Why is pandas broadcast formula is skipping rows

I have a dataframe and I’m doing tons (20+) of calculations creating new columns etc. All the calculations work well, including the calculation in question except for 2 rows out of roughly 1,000. The rows are not adjacent to one another and I can’t find anything remarkable about these two specific rows the calculation seems to be skipping. The data is being read from a csv and an xlsx file. The trouble rows are from apart of the data from the csv file.

The calculation is:

df['c'] =  df['b'] - df['a']

The data for the two trouble rows looks like this:

['a']       ['b']                 ['c']
  0      30.6427984591421           0
  0      9584.28792256921           0

The data for the rest of the df where the calculation works fine looks similar but is processing correctly:

['a']                ['b']                 ['c']
  102411.4521      37008.6603          -65402.7918
  202244.75895    211200.2304295         8955.4714795

Example code:

a = [0, 0, 102411.4521, 202244.75895]
b = [30.6427984591421, 9584.28792256921, 37008.6603, 211200.2304295]
df = pd.DataFrame(zip(a, b), columns=['a', 'b'])
df['c'] =  df['b'] - df['a']

Why would the calculation seemingly skip these rows?

Answer

What the issue was I needed to fillnas before my calculations:

df = df.fillna(0, inplace=False)

Advertisement

Answer