Without using apply (because dataframe is too big), how I can get the previous not NaN value of a specific column to use in a calc ?
For example, this dataframe:
df = pd.DataFrame([['A',1,100],['B',2,None],['C',3,None],['D',4,182],['E',5,None]], columns=['A','B','C']) A B C 0 A 1 100.0 1 B 2 NaN 2 C 3 NaN 3 D 4 182.0 4 E 5 NaN
I need to calc the difference, in the column ‘C’ of the line 3 with the line 0.
The number of NaN values between the values is variable, then
.shift() maybe is not applicable here (I think)
I need some like:
df['D'] = df.C - df.C[previous_not_nan] (in the line 3 will be 82.
df['D'] = df['C'].dropna().diff()
A B C D 0 A 1 100.0 NaN 1 B 2 NaN NaN 2 C 3 NaN NaN 3 D 4 182.0 82.0 4 E 5 NaN NaN