Below is a portion of mydataframe which has many missing values. I would like to replace the NANs in each column using a specific backward fill condition . For example, in column (A,a) missing values appear for dates 16th, 17th, 18th and 19th. The next value is ‘4’ against 20th. I want this value …
Tag: time-series
how do I fit a time-series multi head model?
I try to create a model by concatenating 2 models together. The models I want to use, shall handle time series, and I’m experimenting with Conv1D layers. As these have an 3D input shape batch_shape + (steps, input_dim) and the Keras TimeseriesGenerator is providing such, I’m happy being able to ma…
Autofill datetime in Pandas by previous increment
Given previous datetime values in a Pandas DataFrame–either as an index or as values in a column–is there a way to “autofill” remaining time increments based on the previous fixed increments? For example, given: I would like to apply a function to yield: B 2013-01-01 09:00:00 0.0 2013-…
How to select top n columns from time series data instead of using nlargest in pandas?
I have weekly based trade export time-series data that I need to make a stacked bar plot for visualizing trade activity. To do so, I aggregated my data for sum-up of each columns for all rows, then use nlargest() to select top n columns. However, doing this way might not be quite accurate because I made stack…
Python – How to clean time series data
I have a df which looks like this: I’m trying to create a new column called ‘First_Contract’: ‘First_Contract’ needs to take the third-last value of ‘Sep’ column, before ‘Sep’column reaches NaN. The subsequent values need to be filled with ‘Dec’…
How to forecast out of sample with AutoRegression from statsmodel?
I have time-series sales data. First I group-by the sales by a year. Than I want to forecast the sales for the years 2021,2022 and 2023. I have data from the year 2000. My question is similar to this one, however I want an answer on how to make forecast outside of the training index. If I do this the
How can you get rolling value count (frequency) with Pandas? (computationally efficient, no loops)
I have a list of values and I want to get their rolling frequency, so something like this: Of course I can do this with a loop but with a lot of data it can be computationally expensive so I’d much rather use a built-in or something vectorized, etc. But unfortunately, from my searching, there doesn̵…
Equidistant timeseries filling the blanks
I have the following code that generates a timeseries with 1 min steps but would like to have the time gaps filled. i.e 13:58 is missing in between. Every ip should be represented in the gap with zero values. How can this be achieved? Answer First change unstack by first level for DatetimeIndex, and add DataF…
Machine Learning Classifier use past predictions as features
I want to built a binary classifier machine learning model. I want to use the model’s previous predictions as features for the future predictions, to take into account that my training samples are not independent. Is there a framework to achieve this with scikit-learn, or any other python ML library? I …
How to forecast time series using AutoReg in python
I’m trying to build old school model using only auto regression algorithm. I found out that there’s an implementation of it in statsmodel package. I’ve read the documentation, and as I understand it should work as ARIMA. So, here’s my code: And when I want to predict new values, I̵…