Skip to content
Advertisement

pandas shift on condition of a column isnull but it removes the first two columns

I have this problem with shifting an entire row based on a column condition if it is NaN. if column 10 is NaN, shift the entire row 3 cells to the right.

initial dataframe:

   6   7   8   9   10  11  12
0  2   1   12  3   2   12  4
1  6   2   4   7   NaN NaN NaN
2  4   1   4   3   6   3   2
3  NaN NaN NaN 4   5   3   2
4  3   1   3   2   1   2   6
5  5   2   6   1   NaN NaN NaN
6  NaN NaN NaN 4   5   3   2

I come up with this solution, but the first two columns are gone somehow.

df1_copy[df1_copy[10].isnull()] = df1_copy[df1_copy[10].isnull()].shift(3, axis=1)

Result:

   6   7   8   9   10  11  12
0  2   1   12  3   2   12  4
1  NaN NaN NaN NaN NaN 4   7
2  4   1   4   3   6   3   2
3  NaN NaN NaN 4   5   3   2
4  3   1   3   2   1   2   6
5  NaN NaN NaN NaN NaN 6   1
6  NaN NaN NaN 4   5   3   2

Expected:

   6   7   8   9   10  11  12
0  2   1   12  3   2   12  4
1  NaN NaN NaN 6   2   4   7
2  4   1   4   3   6   3   2
3  NaN NaN NaN 4   5   3   2
4  3   1   3   2   1   2   6
5  NaN NaN NaN 5   2   6   1
6  NaN NaN NaN 4   5   3   2

Although I tried a similar method for other conditions before this and it’s working, it doesn’t seem to apply to this one. any help would be so much appreciated, thank you. here is the csv

Advertisement

Answer

So turned out I was using Pandas version that is too old and it has a bug in the shift function, I come up with this workaround. Although it’s not elegant it works. Hopefully, someone can provide something better. I ended up just rearranging the order of the columns.

df_shift3 = df1_copy1.loc[df1_copy1.loc[:,10].isnull()][[10, 11, 12, 6, 7, 8, 9]]
df_shift3.columns = [6, 7, 8, 9, 10, 11, 12]
df1_copy1[df1_copy1.loc[:,10].isnull()] = df_shift3
User contributions licensed under: CC BY-SA
8 People found this is helpful
Advertisement