Below is a portion of mydataframe which has many missing values. I would like to replace the NANs in each column using a specific backward fill condition . For example, in column (A,a) missing values appear for dates 16th, 17th, 18th and 19th. The next value is ‘4’ against 20th. I want this value (the next non missing value in
Tag: time-series
how do I fit a time-series multi head model?
I try to create a model by concatenating 2 models together. The models I want to use, shall handle time series, and I’m experimenting with Conv1D layers. As these have an 3D input shape batch_shape + (steps, input_dim) and the Keras TimeseriesGenerator is providing such, I’m happy being able to make use of it when handling single head models. This
Autofill datetime in Pandas by previous increment
Given previous datetime values in a Pandas DataFrame–either as an index or as values in a column–is there a way to “autofill” remaining time increments based on the previous fixed increments? For example, given: I would like to apply a function to yield: B 2013-01-01 09:00:00 0.0 2013-01-01 09:00:05 1.0 2013-01-01 09:00:10 2.0 2013-01-01 09:00:15 NaN 2013-01-01 09:00:20 4.0 Where
How to select top n columns from time series data instead of using nlargest in pandas?
I have weekly based trade export time-series data that I need to make a stacked bar plot for visualizing trade activity. To do so, I aggregated my data for sum-up of each columns for all rows, then use nlargest() to select top n columns. However, doing this way might not be quite accurate because I made stacked plot for different
Python – How to clean time series data
I have a df which looks like this: I’m trying to create a new column called ‘First_Contract’: ‘First_Contract’ needs to take the third-last value of ‘Sep’ column, before ‘Sep’column reaches NaN. The subsequent values need to be filled with ‘Dec’ column values. Desired output: How do I go about to achieve this? Answer Let us do it step by step
How to forecast out of sample with AutoRegression from statsmodel?
I have time-series sales data. First I group-by the sales by a year. Than I want to forecast the sales for the years 2021,2022 and 2023. I have data from the year 2000. My question is similar to this one, however I want an answer on how to make forecast outside of the training index. If I do this the
How can you get rolling value count (frequency) with Pandas? (computationally efficient, no loops)
I have a list of values and I want to get their rolling frequency, so something like this: Of course I can do this with a loop but with a lot of data it can be computationally expensive so I’d much rather use a built-in or something vectorized, etc. But unfortunately, from my searching, there doesn’t seem to be a
Equidistant timeseries filling the blanks
I have the following code that generates a timeseries with 1 min steps but would like to have the time gaps filled. i.e 13:58 is missing in between. Every ip should be represented in the gap with zero values. How can this be achieved? Answer First change unstack by first level for DatetimeIndex, and add DataFrame.asfreq for add missing minutes:
Machine Learning Classifier use past predictions as features
I want to built a binary classifier machine learning model. I want to use the model’s previous predictions as features for the future predictions, to take into account that my training samples are not independent. Is there a framework to achieve this with scikit-learn, or any other python ML library? I know this problem could be solved with memory-based Neural
How to forecast time series using AutoReg in python
I’m trying to build old school model using only auto regression algorithm. I found out that there’s an implementation of it in statsmodel package. I’ve read the documentation, and as I understand it should work as ARIMA. So, here’s my code: And when I want to predict new values, I’m trying to follow the documentation: Both returns a list of