Merge branch 'develop' into dev-merge-rl

2022-09-14 22:49:11 +02:00 · 2022-09-14 22:49:11 +02:00 · 81417cb795
commit 81417cb795
parent 69b3fcfd32 075748b21a
59 changed files with 1259 additions and 679 deletions
--- a/2
+++ b/2
@ -1,4 +1,4 @@
-FROM python:3.10.6-slim-bullseye as base
+FROM python:3.10.7-slim-bullseye as base
 # Setup env
 ENV LANG C.UTF-8
--- a/docs/assets/freqai_weight-factor.jpg
+++ b/docs/assets/freqai_weight-factor.jpg
--- a/docs/backtesting.md
+++ b/docs/backtesting.md
@ -107,7 +107,7 @@ Strategy arguments:
 ## Test your strategy with Backtesting
-Now you have good Buy and Sell strategies and some historic data, you want to test it against
+Now you have good Entry and exit strategies and some historic data, you want to test it against
 real data. This is what we call [backtesting](https://en.wikipedia.org/wiki/Backtesting).
 Backtesting will use the crypto-currencies (pairs) from your config file and load historical candle (OHLCV) data from `user_data/data/<exchange>` by default.
@ -215,7 +215,7 @@ Sometimes your account has certain fee rebates (fee reductions starting with a c
 To account for this in backtesting, you can use the `--fee` command line option to supply this value to backtesting.
 This fee must be a ratio, and will be applied twice (once for trade entry, and once for trade exit).
-For example, if the buying and selling commission fee is 0.1% (i.e., 0.001 written as ratio), then you would run backtesting as the following:
+For example, if the commission fee per order is 0.1% (i.e., 0.001 written as ratio), then you would run backtesting as the following:
 ```bash
 freqtrade backtesting --fee 0.001
@ -252,9 +252,9 @@ The most important in the backtesting is to understand the result.
 A backtesting result will look like that:
 ```
-========================================================= BACKTESTING REPORT ==========================================================
+========================================================= BACKTESTING REPORT =========================================================
-| Pair     |   Buys |   Avg Profit % |   Cum Profit % |   Tot Profit BTC |   Tot Profit % | Avg Duration |  Wins Draws Loss   Win%  |
+| Pair     | Entries |   Avg Profit % |   Cum Profit % |   Tot Profit BTC |   Tot Profit % | Avg Duration |  Wins Draws Loss   Win%  |
-|:---------|-------:|---------------:|---------------:|-----------------:|---------------:|:-------------|-------------------------:|
+|:---------|--------:|---------------:|---------------:|-----------------:|---------------:|:-------------|-------------------------:|
 | ADA/BTC  |      35 |          -0.11 |          -3.88 |      -0.00019428 |          -1.94 | 4:35:00      |    14     0    21   40.0 |
 | ARK/BTC  |      11 |          -0.41 |          -4.52 |      -0.00022647 |          -2.26 | 2:03:00      |     3     0     8   27.3 |
 | BTS/BTC  |      32 |           0.31 |           9.78 |       0.00048938 |           4.89 | 5:05:00      |    18     0    14   56.2 |
@ -275,15 +275,15 @@ A backtesting result will look like that:
 | ZEC/BTC  |      22 |          -0.46 |         -10.18 |      -0.00050971 |          -5.09 | 2:22:00      |     7     0    15   31.8 |
 | TOTAL    |     429 |           0.36 |         152.41 |       0.00762792 |          76.20 | 4:12:00      |   186     0   243   43.4 |
 ========================================================= EXIT REASON STATS ==========================================================
-| Exit Reason        |   Sells |  Wins |  Draws |  Losses |
+| Exit Reason        |   Exits |  Wins |  Draws |  Losses |
 |:-------------------|--------:|------:|-------:|--------:|
 | trailing_stop_loss |     205 |   150 |      0 |      55 |
 | stop_loss          |     166 |     0 |      0 |     166 |
 | exit_signal        |      56 |    36 |      0 |      20 |
 | force_exit         |       2 |     0 |      0 |       2 |
 ====================================================== LEFT OPEN TRADES REPORT ======================================================
-| Pair     |   Buys |   Avg Profit % |   Cum Profit % |   Tot Profit BTC |   Tot Profit % | Avg Duration   |  Win Draw Loss Win% |
+| Pair     |  Entries |   Avg Profit % |   Cum Profit % |   Tot Profit BTC |   Tot Profit % | Avg Duration   |  Win Draw Loss Win% |
-|:---------|-------:|---------------:|---------------:|-----------------:|---------------:|:---------------|--------------------:|
+|:---------|---------:|---------------:|---------------:|-----------------:|---------------:|:---------------|--------------------:|
 | ADA/BTC  |        1 |           0.89 |           0.89 |       0.00004434 |           0.44 | 6:00:00        |    1    0    0  100 |
 | LTC/BTC  |        1 |           0.68 |           0.68 |       0.00003421 |           0.34 | 2:00:00        |    1    0    0  100 |
 | TOTAL    |        2 |           0.78 |           1.57 |       0.00007855 |           0.78 | 4:00:00        |    2    0    0  100 |
@ -356,7 +356,7 @@ The column `Avg Profit %` shows the average profit for all trades made while the
 The column `Tot Profit %` shows instead the total profit % in relation to the starting balance.
 In the above results, we have a starting balance of 0.01 BTC and an absolute profit of 0.00762792 BTC - so the `Tot Profit %` will be `(0.00762792 / 0.01) * 100 ~= 76.2%`.
-Your strategy performance is influenced by your buy strategy, your exit strategy, and also by the `minimal_roi` and `stop_loss` you have set.
+Your strategy performance is influenced by your entry strategy, your exit strategy, and also by the `minimal_roi` and `stop_loss` you have set.
 For example, if your `minimal_roi` is only `"0":  0.01` you cannot expect the bot to make more profit than 1% (because it will exit every time a trade reaches 1%).
@ -515,7 +515,7 @@ You can then load the trades to perform further analysis as shown in the [data a
 Since backtesting lacks some detailed information about what happens within a candle, it needs to take a few assumptions:
 - Exchange [trading limits](#trading-limits-in-backtesting) are respected
- Buys happen at open-price
+- Entries happen at open-price
 - All orders are filled at the requested price (no slippage, no unfilled orders)
 - Exit-signal exits happen at open-price of the consecutive candle
 - Exit-signal is favored over Stoploss, because exit-signals are assumed to trigger on candle's open
@ -612,9 +612,9 @@ There will be an additional table comparing win/losses of the different strategi
 Detailed output for all strategies one after the other will be available, so make sure to scroll up to see the details per strategy.
 ```
-=========================================================== STRATEGY SUMMARY =========================================================================
+=========================================================== STRATEGY SUMMARY ===========================================================================
-| Strategy    |   Buys |   Avg Profit % |   Cum Profit % |   Tot Profit BTC |   Tot Profit % | Avg Duration   |  Wins |  Draws | Losses | Drawdown % |
+| Strategy    |  Entries |   Avg Profit % |   Cum Profit % |   Tot Profit BTC |   Tot Profit % | Avg Duration   |  Wins |  Draws | Losses | Drawdown % |
-|:------------|-------:|---------------:|---------------:|-----------------:|---------------:|:---------------|------:|-------:|-------:|-----------:|
+|:------------|---------:|---------------:|---------------:|-----------------:|---------------:|:---------------|------:|-------:|-------:|-----------:|
 | Strategy1   |      429 |           0.36 |         152.41 |       0.00762792 |          76.20 | 4:12:00        |   186 |      0 |    243 |       45.2 |
 | Strategy2   |     1487 |          -0.13 |        -197.58 |      -0.00988917 |         -98.79 | 4:43:00        |   662 |      0 |    825 |     241.68 |
 ```
--- a/docs/faq.md
+++ b/docs/faq.md
@ -4,7 +4,7 @@
 Freqtrade supports spot trading only.
-### Can I open short positions?
+### Can my bot open short positions?
 Freqtrade can open short positions in futures markets.
 This requires the strategy to be made for this - and `"trading_mode": "futures"` in the configuration.
@ -12,9 +12,9 @@ Please make sure to read the [relevant documentation page](leverage.md) first.
 In spot markets, you can in some cases use leveraged spot tokens, which reflect an inverted pair (eg. BTCUP/USD, BTCDOWN/USD, ETHBULL/USD, ETHBEAR/USD,...) which can be traded with Freqtrade.
-### Can I trade options or futures?
+### Can my bot trade options or futures?
-Futures trading is supported for selected exchanges.
+Futures trading is supported for selected exchanges. Please refer to the [documentation start page](index.md#supported-futures-exchanges-experimental) for an uptodate list of supported exchanges.
 ## Beginner Tips & Tricks
@ -22,6 +22,13 @@ Futures trading is supported for selected exchanges.
 ## Freqtrade common issues
 ### Can freqtrade open multiple positions on the same pair in parallel?
 No. Freqtrade will only open one position per pair at a time.
 You can however use the [`adjust_trade_position()` callback](strategy-callbacks.md#adjust-trade-position) to adjust an open position.
 Backtesting provides an option for this in `--eps` - however this is only there to highlight "hidden" signals, and will not work in live.
 ### The bot does not start
 Running the bot with `freqtrade trade --config config.json` shows the output `freqtrade: command not found`.
@ -30,7 +37,7 @@ This could be caused by the following reasons:
 * The virtual environment is not active.
  * Run `source .env/bin/activate` to activate the virtual environment.
-* The installation did not work correctly.
+* The installation did not complete successfully.
  * Please check the [Installation documentation](installation.md).
 ### I have waited 5 minutes, why hasn't the bot made any trades yet?
--- a/docs/freqai.md
+++ b/docs/freqai.md
@ -98,6 +98,7 @@ Mandatory parameters are marked as **Required**, which means that they are requi
 | `expiration_hours` | Avoid making predictions if a model is more than `expiration_hours` old. <br> Defaults set to 0, which means models never expire. <br> **Datatype:** Positive integer.
 | `fit_live_predictions_candles` | Number of historical candles to use for computing target (label) statistics from prediction data, instead of from the training data set. <br> **Datatype:** Positive integer.
 | `follow_mode` | If true, this instance of FreqAI will look for models associated with `identifier` and load those for inferencing. A `follower` will **not** train new models. <br> **Datatype:** Boolean. Default: `False`.
 | `continual_learning` | If true, FreqAI will start training new models from the final state of the most recently trained model. <br> **Datatype:** Boolean. Default: `False`.
 |  |  **Feature parameters**
 | `feature_parameters` | A dictionary containing the parameters used to engineer the feature set. Details and examples are shown [here](#feature-engineering). <br> **Datatype:** Dictionary.
 | `include_timeframes` | A list of timeframes that all indicators in `populate_any_indicators` will be created for. The list is added as features to the base asset feature set. <br> **Datatype:** List of timeframes (strings).
@ -113,14 +114,16 @@ Mandatory parameters are marked as **Required**, which means that they are requi
 | `use_SVM_to_remove_outliers` | Train a support vector machine to detect and remove outliers from the training data set, as well as from incoming data points. See details about how it works [here](#removing-outliers-using-a-support-vector-machine-svm). <br> **Datatype:** Boolean.
 | `svm_params` | All parameters available in Sklearn's `SGDOneClassSVM()`. See details about some select parameters [here](#removing-outliers-using-a-support-vector-machine-svm). <br> **Datatype:** Dictionary.
 | `use_DBSCAN_to_remove_outliers` | Cluster data using DBSCAN to identify and remove outliers from training and prediction data. See details about how it works [here](#removing-outliers-with-dbscan). <br> **Datatype:** Boolean. 
-| `outlier_protection_percentage` | If more than `outlier_protection_percentage` fraction of points are removed as outliers, FreqAI will log a warning message and ignore outlier detection while keeping the original dataset intact. <br> **Datatype:** float. Default: `30`
+| `inlier_metric_window` | If set, FreqAI will add the `inlier_metric` to the training feature set and set the lookback to be the `inlier_metric_window`. Details of how the `inlier_metric` is computed can be found [here](#using-the-inliermetric) <br> **Datatype:** int. Default: 0
-| `reverse_train_test_order` | If true, FreqAI will train on the latest data split and test on historical split of the data. This allows the model to be trained up to the most recent data point, while avoiding overfitting. However, users should be careful to understand unorthodox nature of this parameter before employing it. <br> **Datatype:** bool. Default: False
+| `noise_standard_deviation` | If > 0, FreqAI adds noise to the training features. FreqAI generates random deviates from a gaussian distribution with a standard deviation of `noise_standard_deviation` and adds them to all data points. Value should be kept relative to the normalized space between -1 and 1). In other words, since data is always normalized between -1 and 1 in FreqAI, the user can expect a `noise_standard_deviation: 0.05` to see 32% of data randomly increased/decreased by more than 2.5% (i.e. the percent of data falling within the first standard deviation). Good for preventing overfitting. <br> **Datatype:** int. Default: 0
 | `outlier_protection_percentage` | If more than `outlier_protection_percentage` % of points are detected as outliers by the SVM or DBSCAN, FreqAI will log a warning message and ignore outlier detection while keeping the original dataset intact. If the outlier protection is triggered, no predictions will be made based on the training data. <br> **Datatype:** Float. Default: `30`
 | `reverse_train_test_order` | If true, FreqAI will train on the latest data split and test on historical split of the data. This allows the model to be trained up to the most recent data point, while avoiding overfitting. However, users should be careful to understand unorthodox nature of this parameter before employing it. <br> **Datatype:** Boolean. Default: False
 |  |  **Data split parameters**
 | `data_split_parameters` | Include any additional parameters available from Scikit-learn `test_train_split()`, which are shown [here](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.train_test_split.html) (external website). <br> **Datatype:** Dictionary.
 | `test_size` | Fraction of data that should be used for testing instead of training. <br> **Datatype:** Positive float < 1.
-| `shuffle` | Shuffle the training data points during training. Typically, for time-series forecasting, this is set to `False`. <br>
+| `shuffle` | Shuffle the training data points during training. Typically, for time-series forecasting, this is set to `False`. <br> **Datatype:** Boolean.
 |  |  **Model training parameters**
-| `model_training_parameters` | A flexible dictionary that includes all parameters available by the user selected model library. For example, if the user uses `LightGBMRegressor`, this dictionary can contain any parameter available by the `LightGBMRegressor` [here](https://lightgbm.readthedocs.io/en/latest/pythonapi/lightgbm.LGBMRegressor.html) (external website). If the user selects a different model, such as `PPO` from stable_baselines3, this dictionary can contain any parameter from that model.  <br> **Datatype:** Dictionary
+| `model_training_parameters` | A flexible dictionary that includes all parameters available by the user selected model library. For example, if the user uses `LightGBMRegressor`, this dictionary can contain any parameter available by the `LightGBMRegressor` [here](https://lightgbm.readthedocs.io/en/latest/pythonapi/lightgbm.LGBMRegressor.html) (external website). If the user selects a different model, this dictionary can contain any parameter from that model.  <br> **Datatype:** Dictionary.
 | `n_estimators` | The number of boosted trees to fit in regression. <br> **Datatype:** Integer.
 | `learning_rate` | Boosting learning rate during regression. <br> **Datatype:** Float.
 | `n_jobs`, `thread_count`, `task_type` | Set the number of threads for parallel processing and the `task_type` (`gpu` or `cpu`). Different model libraries use different parameter names. <br> **Datatype:** Float.
@ -289,8 +292,10 @@ The FreqAI strategy requires the user to include the following lines of code in
 Notice how the `populate_any_indicators()` is where the user adds their own features ([more information](#feature-engineering)) and labels ([more information](#setting-classifier-targets)). See a full example at `templates/FreqaiExampleStrategy.py`.
 *Important*: The `self.freqai.start()` function cannot be called outside the `populate_indicators()`.
 ### Setting the `startup_candle_count`
-Users need to take care to set the `startup_candle_count` in their strategy the same way they would for any normal Freqtrade strategy (see details [here](strategy-customization.md/#strategy-startup-period)). This value is used by Freqtrade to ensure that a sufficient amount of data is provided when calling on the `dataprovider` to avoid any NaNs at the beginning of the first training. Users can easily set this value by identifying the longest period (in candle units) that they pass to their indicator creation functions (e.g. talib functions). In the present example, the user would pass 20 to as this value (since it is the maximum value in their `indicators_periods_candles`).
+Users need to take care to set the `startup_candle_count` in their strategy the same way they would for any normal Freqtrade strategy (see details [here](strategy-customization.md#strategy-startup-period)). This value is used by Freqtrade to ensure that a sufficient amount of data is provided when calling on the `dataprovider` to avoid any NaNs at the beginning of the first training. Users can easily set this value by identifying the longest period (in candle units) that they pass to their indicator creation functions (e.g. talib functions). In the present example, the user would pass 20 to as this value (since it is the maximum value in their `indicators_periods_candles`).
 !!! Note
    Typically it is best for users to be safe and multiply their expected `startup_candle_count` by 2. There are instances where the talib functions actually require more data than just the passed `period`. Anecdotally, multiplying the `startup_candle_count` by 2 always leads to a fully NaN free training dataset. Look out for this log message to confirm that your data is clean:
@ -525,10 +530,10 @@ and if a full `live_retrain_hours` has elapsed since the end of the loaded model
 The FreqAI backtesting module can be executed with the following command:
 ```bash
-freqtrade backtesting --strategy FreqaiExampleStrategy --config config_examples/config_freqai.example.json --freqaimodel LightGBMRegressor --timerange 20210501-20210701
+freqtrade backtesting --strategy FreqaiExampleStrategy --strategy-path freqtrade/templates --config config_examples/config_freqai.example.json --freqaimodel LightGBMRegressor --timerange 20210501-20210701
 ```
-Backtesting mode requires the user to have the data pre-downloaded (unlike in dry/live mode where FreqAI automatically downloads the necessary data). The user should be careful to consider that the time range of the downloaded data is more than the backtesting time range. This is because FreqAI needs data prior to the desired backtesting time range in order to train a model to be ready to make predictions on the first candle of the user-set backtesting time range. More details on how to calculate the data to download can be found [here](#deciding-the-sliding-training-window-and-backtesting-duration).
+Backtesting mode requires the user to have the data [pre-downloaded](#downloading-data-for-backtesting) (unlike in dry/live mode where FreqAI automatically downloads the necessary data). The user should be careful to consider that the time range of the downloaded data is more than the backtesting time range. This is because FreqAI needs data prior to the desired backtesting time range in order to train a model to be ready to make predictions on the first candle of the user-set backtesting time range. More details on how to calculate the data to download can be found [here](#deciding-the-sliding-training-window-and-backtesting-duration). 
 If this command has never been executed with the existing config file, it will train a new model
 for each pair, for each backtesting window within the expanded `--timerange`.
@ -542,6 +547,31 @@ for each pair, for each backtesting window within the expanded `--timerange`.
 ---
 ### Hyperopt
 Users can hyperopt using the same command as typical [hyperopt](hyperopt.md):
 ```bash
 freqtrade hyperopt --hyperopt-loss SharpeHyperOptLoss --strategy FreqaiExampleStrategy --freqaimodel LightGBMRegressor --strategy-path freqtrade/templates --config config_examples/config_freqai.example.json --timerange 20220428-20220507
 ```
 Users need to have the data pre-downloaded in the same fashion as if they were doing a FreqAI [backtest](#backtesting). In addition, users must consider some restrictions when trying to [Hyperopt](hyperopt.md)  FreqAI strategies:
 - The `--analyze-per-epoch` hyperopt parameter is not compatible with FreqAI.
 - It's not possible to hyperopt indicators in `populate_any_indicators()` function. This means that the user cannot optimize model parameters using hyperopt. Apart from this exception, it is possible to optimize all other [spaces](hyperopt.md#running-hyperopt-with-smaller-search-space).
 - The [Backtesting](#backtesting) instructions also apply to Hyperopt.
 The best method for combining hyperopt and FreqAI is to focus on hyperopting entry/exit thresholds/criteria. Users need to focus on hyperopting parameters that are not used in their FreqAI features. For example, users should not try to hyperopt rolling window lengths in their feature creation, or any of their FreqAI config which changes predictions. In order to efficiently hyperopt the FreqAI strategy, FreqAI stores predictions as dataframes and reuses them. Hence the requirement to hyperopt entry/exit thresholds/criteria only. 
 A good example of a hyperoptable parameter in FreqAI is a value for `DI_values` beyond which we consider outliers and below which we consider inliers:
 ```python
 di_max = IntParameter(low=1, high=20, default=10, space='buy', optimize=True, load=True)
 dataframe['outlier'] = np.where(dataframe['DI_values'] > self.di_max.value/10, 1, 0)
 ```
 Which would help the user understand the appropriate Dissimilarity Index values for their particular parameter space.
 ### Deciding the size of the sliding training window and backtesting duration
 The user defines the backtesting timerange with the typical `--timerange` parameter in the
@ -556,7 +586,7 @@ FreqAI will train have trained 8 separate models at the end of `--timerange` (be
    Although fractional `backtest_period_days` is allowed, the user should be aware that the `--timerange` is divided by this value to determine the number of models that FreqAI will need to train in order to backtest the full range. For example, if the user wants to set a `--timerange` of 10 days, and asks for a `backtest_period_days` of 0.1, FreqAI will need to train 100 models per pair to complete the full backtest. Because of this, a true backtest of FreqAI adaptive training would take a *very* long time. The best way to fully test a model is to run it dry and let it constantly train. In this case, backtesting would take the exact same amount of time as a dry run.
 ### Downloading data for backtesting
-Live/dry instances will download the data automatically for the user, but users who wish to use backtesting functionality still need to download the necessary data using `download-data` (details [here](data-download/#data-downloading)). FreqAI users need to pay careful attention to understanding how much *additional* data needs to be downloaded to ensure that they have a sufficient amount of training data *before* the start of their backtesting timerange. The amount of additional data can be roughly estimated by taking subtracting `train_period_days` and the `startup_candle_count` ([details](#setting-the-startupcandlecount)) from the beginning of the desired backtesting timerange. 
+Live/dry instances will download the data automatically for the user, but users who wish to use backtesting functionality still need to download the necessary data using `download-data` (details [here](data-download.md#data-downloading)). FreqAI users need to pay careful attention to understanding how much *additional* data needs to be downloaded to ensure that they have a sufficient amount of training data *before* the start of their backtesting timerange. The amount of additional data can be roughly estimated by moving the start date of the timerange backwards by `train_period_days` and the `startup_candle_count` ([details](#setting-the-startupcandlecount)) from the beginning of the desired backtesting timerange. 
 As an example, if we wish to backtest the `--timerange` above of `20210501-20210701`, and we use the example config which sets `train_period_days` to 15. The startup candle count is 40 on a maximum `include_timeframes` of 1h. We would need 20210501 - 15 days - 40 * 1h / 24 hours = 20210414 (16.7 days earlier than the start of the desired training timerange).
@ -653,6 +683,18 @@ testing; the other points are used for training.
 The test data is used to evaluate the performance of the model after training. If the test score is high, the model is able to capture the behavior of the data well. If the test score is low, either the model either does not capture the complexity of the data, the test data is significantly different from the train data, or a different model should be used.
 ### Using the `inlier_metric`
 The `inlier_metric` is a metric aimed at quantifying how different a prediction data point is from the most recent historic data points. 
 User can set `inlier_metric_window` to set the look back window. FreqAI will compute the distance between the present prediction point and each of the previous data points (total of `inlier_metric_window` points). 
 This function goes one step further - during training, it computes the `inlier_metric` for all training data points and builds weibull distributions for each each lookback point. The cumulative distribution function for the weibull distribution is used to produce a quantile for each of the data points. The quantiles for each lookback point are averaged to create the `inlier_metric`. 
 FreqAI adds this `inlier_metric` score to the training features! In other words, your model is trained to recognize how this temporal inlier metric is related to the user set labels. 
 This function does **not** remove outliers from the data set.
 ### Controlling the model learning process
 Model training parameters are unique to the machine learning library selected by the user. FreqAI allows the user to set any parameter for any library using the `model_training_parameters` dictionary in the user configuration file. The example configuration file (found in `config_examples/config_freqai.example.json`) show some of the example parameters associated with `Catboost` and `LightGBM`, but the user can add any parameters available in those libraries.
@ -750,93 +792,6 @@ Given a number of data points $N$, and a distance $\varepsilon$, DBSCAN clusters
 FreqAI uses `sklearn.cluster.DBSCAN` (details are available on scikit-learn's webpage [here](#https://scikit-learn.org/stable/modules/generated/sklearn.cluster.DBSCAN.html)) with `min_samples` ($N$) taken as double the no. of user-defined features, and `eps` ($\varepsilon$) taken as the longest distance in the *k-distance graph* computed from the nearest neighbors in the pairwise distances of all data points in the feature set.
 ## Reinforcement Learning
 Setting up and running a Reinforcement Learning model is as quick and simple as running a Regressor. Users can start training and trading live from example files using:
 ```bash
 freqtrade trade --freqaimodel ReinforcementLearner --strategy ReinforcementLearningExample5ac --strategy-path freqtrade/freqai/example_strats --config config_examples/config_freqai-rl.example.json
 ```
 As users begin to modify the strategy and the prediction model, they will quickly realize some important differences between the Reinforcement Learner and the Regressors/Classifiers. Firstly, the strategy does not set a target value (no labels!). Instead, the user sets a `calculate_reward()` function inside their custom `ReinforcementLearner.py` file. A default `calculate_reward()` is provided inside `prediction_models/ReinforcementLearner.py` to give users the necessary building blocks to start their own models. It is inside the `calculate_reward()` where users express their creative theories about the market. For example, the user wants to reward their agent when it makes a winning trade, and penalize the agent when it makes a losing trade. Or perhaps, the user wishes to reward the agnet for entering trades, and penalize the agent for sitting in trades too long. Below we show examples of how these rewards are all calculated:
 ```python
    class MyRLEnv(Base5ActionRLEnv):
        """
        User made custom environment. This class inherits from BaseEnvironment and gym.env.
        Users can override any functions from those parent classes. Here is an example
        of a user customized `calculate_reward()` function.
        """
        def calculate_reward(self, action):
            # first, penalize if the action is not valid
            if not self._is_valid(action):
                return -2
            pnl = self.get_unrealized_profit()
            rew = np.sign(pnl) * (pnl + 1)
            factor = 100
            # reward agent for entering trades
            if action in (Actions.Long_enter.value, Actions.Short_enter.value) \
                    and self._position == Positions.Neutral:
                return 25
            # discourage agent from not entering trades
            if action == Actions.Neutral.value and self._position == Positions.Neutral:
                return -1
            max_trade_duration = self.rl_config.get('max_trade_duration_candles', 300)
            trade_duration = self._current_tick - self._last_trade_tick
            if trade_duration <= max_trade_duration:
                factor *= 1.5
            elif trade_duration > max_trade_duration:
                factor *= 0.5
            # discourage sitting in position
            if self._position in (Positions.Short, Positions.Long) and \
               action == Actions.Neutral.value:
                return -1 * trade_duration / max_trade_duration
            # close long
            if action == Actions.Long_exit.value and self._position == Positions.Long:
                if pnl > self.profit_aim * self.rr:
                    factor *= self.rl_config['model_reward_parameters'].get('win_reward_factor', 2)
                return float(rew * factor)
            # close short
            if action == Actions.Short_exit.value and self._position == Positions.Short:
                if pnl > self.profit_aim * self.rr:
                    factor *= self.rl_config['model_reward_parameters'].get('win_reward_factor', 2)
                return float(rew * factor)
            return 0.
 ```
 After users realize there are no labels to set, they will soon understand that the agent is making its "own" entry and exit decisions. This makes strategy construction rather simple (as shown in `example_strats/ReinforcementLearningExample5ac.py`). The entry and exit signals come from the agent in the form of an integer - which are used directly to decide entries and exits in the strategy. 
 ### Using Tensorboard
 Reinforcement Learning models benefit from tracking training metrics. FreqAI has integrated Tensorboard to allow users to track training and evaluation performance across all coins and across all retrainings. To start, the user should ensure Tensorboard is installed on their computer:
 ```bash
 pip3 install tensorboard
 ```
 Next, the user can activate Tensorboard with the following command:
 ```bash
 cd freqtrade
 tensorboard --logdir user_data/models/unique-id
 ```
 where `unique-id` is the `identifier` set in the `freqai` configuration file. 
 ![tensorboard](assets/tensorboard.jpg)
 ## Additional information
 ### Common pitfalls
@ -860,5 +815,5 @@ Code review, software architecture brainstorming:
@xmatthias
 Beta testing and bug reporting:
-@bloodhunter4rc, Salah Lamkadem @ikonx, @ken11o2, @longyu, @paranoidandy, @smidelis, @smarm
+@bloodhunter4rc, Salah Lamkadem @ikonx, @ken11o2, @longyu, @paranoidandy, @smidelis, @smarm,
 Juha Nykänen @suikula, Wagner Costa @wagnercosta
--- a/docs/strategy-customization.md
+++ b/docs/strategy-customization.md
@ -824,6 +824,8 @@ Options:
 - Merge the dataframe without lookahead bias
 - Forward-fill (optional)
 For a full sample, please refer to the [complete data provider example](#complete-data-provider-sample) below.
 All columns of the informative dataframe will be available on the returning dataframe in a renamed fashion:
 !!! Example "Column renaming"
--- a/docs/telegram-usage.md
+++ b/docs/telegram-usage.md
@ -90,7 +90,8 @@ Example configuration showing the different settings:
            "trailing_stop_loss": "on",
            "stop_loss": "on",
            "stoploss_on_exchange": "on",
-            "custom_exit": "silent"
+            "custom_exit": "silent",
            "partial_exit": "on"
        },
        "entry_cancel": "silent",
        "exit_cancel": "on",
--- a/freqtrade/commands/cli_options.py
+++ b/freqtrade/commands/cli_options.py
@ -455,8 +455,6 @@ AVAILABLE_CLI_OPTIONS = {
        '-t', '--timeframes',
        help='Specify which tickers to download. Space-separated list. '
        'Default: `1m 5m`.',
        choices=['1m', '3m', '5m', '15m', '30m', '1h', '2h', '4h',
                 '6h', '8h', '12h', '1d', '3d', '1w', '2w', '1M', '1y'],
        default=['1m', '5m'],
        nargs='+',
    ),
--- a/freqtrade/commands/db_commands.py
+++ b/freqtrade/commands/db_commands.py
@ -4,7 +4,7 @@ from typing import Any, Dict
 from sqlalchemy import func
 from freqtrade.configuration.config_setup import setup_utils_configuration
-from freqtrade.enums.runmode import RunMode
+from freqtrade.enums import RunMode
 logger = logging.getLogger(__name__)
--- a/freqtrade/configuration/config_validation.py
+++ b/freqtrade/configuration/config_validation.py
@ -84,6 +84,7 @@ def validate_config_consistency(conf: Dict[str, Any], preliminary: bool = False)
    _validate_protections(conf)
    _validate_unlimited_amount(conf)
    _validate_ask_orderbook(conf)
    _validate_freqai_hyperopt(conf)
    validate_migrated_strategy_settings(conf)
    # validate configuration before returning
@ -323,6 +324,14 @@ def _validate_pricing_rules(conf: Dict[str, Any]) -> None:
            del conf['ask_strategy']
 def _validate_freqai_hyperopt(conf: Dict[str, Any]) -> None:
    freqai_enabled = conf.get('freqai', {}).get('enabled', False)
    analyze_per_epoch = conf.get('analyze_per_epoch', False)
    if analyze_per_epoch and freqai_enabled:
        raise OperationalException(
            'Using analyze-per-epoch parameter is not supported with a FreqAI strategy.')
 def _strategy_settings(conf: Dict[str, Any]) -> None:
    process_deprecated_setting(conf, None, 'use_sell_signal', None, 'use_exit_signal')
--- a/freqtrade/data/history/history_utils.py
+++ b/freqtrade/data/history/history_utils.py
@ -228,9 +228,9 @@ def _download_pair_history(pair: str, *,
                    )
        logger.debug("Current Start: %s",
-                     f"{data.iloc[0]['date']:%Y-%m-%d %H:%M:%S}" if not data.empty else 'None')
+                     f"{data.iloc[0]['date']:DATETIME_PRINT_FORMAT}" if not data.empty else 'None')
        logger.debug("Current End: %s",
-                     f"{data.iloc[-1]['date']:%Y-%m-%d %H:%M:%S}" if not data.empty else 'None')
+                     f"{data.iloc[-1]['date']:DATETIME_PRINT_FORMAT}" if not data.empty else 'None')
        # Default since_ms to 30 days if nothing is given
        new_data = exchange.get_historic_ohlcv(pair=pair,
@ -254,9 +254,9 @@ def _download_pair_history(pair: str, *,
                                         fill_missing=False, drop_incomplete=False)
        logger.debug("New  Start: %s",
-                     f"{data.iloc[0]['date']:%Y-%m-%d %H:%M:%S}" if not data.empty else 'None')
+                     f"{data.iloc[0]['date']:DATETIME_PRINT_FORMAT}" if not data.empty else 'None')
        logger.debug("New End: %s",
-                     f"{data.iloc[-1]['date']:%Y-%m-%d %H:%M:%S}" if not data.empty else 'None')
+                     f"{data.iloc[-1]['date']:DATETIME_PRINT_FORMAT}" if not data.empty else 'None')
        data_handler.ohlcv_store(pair, timeframe, data=data, candle_type=candle_type)
        return True
--- a/freqtrade/exchange/exchange.py
+++ b/freqtrade/exchange/exchange.py
@ -205,7 +205,7 @@ class Exchange:
        logger.debug("Exchange object destroyed, closing async loop")
        if (self._api_async and inspect.iscoroutinefunction(self._api_async.close)
                and self._api_async.session):
-            logger.info("Closing async ccxt session.")
+            logger.debug("Closing async ccxt session.")
            self.loop.run_until_complete(self._api_async.close())
    def validate_config(self, config):
@ -446,6 +446,15 @@ class Exchange:
        contract_size = self.get_contract_size(pair)
        return contracts_to_amount(num_contracts, contract_size)
    def amount_to_contract_precision(self, pair: str, amount: float) -> float:
        """
        Helper wrapper around amount_to_contract_precision
        """
        contract_size = self.get_contract_size(pair)
        return amount_to_contract_precision(amount, self.get_precision_amount(pair),
                                            self.precisionMode, contract_size)
    def set_sandbox(self, api: ccxt.Exchange, exchange_config: dict, name: str) -> None:
        if exchange_config.get('sandbox'):
            if api.urls.get('test'):
@ -2500,8 +2509,13 @@ class Exchange:
            cache=False,
            drop_incomplete=False,
        )
        try:
            # we can't assume we always get histories - for example during exchange downtimes
            funding_rates = candle_histories[funding_comb]
            mark_rates = candle_histories[mark_comb]
        except KeyError:
            raise ExchangeError("Could not find funding rates.") from None
        funding_mark_rates = self.combine_funding_and_mark(
            funding_rates=funding_rates, mark_rates=mark_rates)
@ -2581,6 +2595,8 @@ class Exchange:
        :param is_short: trade direction
        :param amount: Trade amount
        :param open_date: Open date of the trade
        :return: funding fee since open_date
        :raies: ExchangeError if something goes wrong.
        """
        if self.trading_mode == TradingMode.FUTURES:
            if self._config['dry_run']:
--- a/freqtrade/exchange/okx.py
+++ b/freqtrade/exchange/okx.py
@ -4,8 +4,7 @@ from typing import Dict, List, Optional, Tuple
 import ccxt
 from freqtrade.constants import BuySell
-from freqtrade.enums import MarginMode, TradingMode
+from freqtrade.enums import CandleType, MarginMode, TradingMode
 from freqtrade.enums.candletype import CandleType
 from freqtrade.exceptions import DDosProtection, OperationalException, TemporaryError
 from freqtrade.exchange import Exchange, date_minus_candles
 from freqtrade.exchange.common import retrier
--- a/freqtrade/freqai/prediction_models/BaseClassifierModel.py
+++ b/freqtrade/freqai/prediction_models/BaseClassifierModel.py
@ -21,12 +21,12 @@ class BaseClassifierModel(IFreqaiModel):
    """
    def train(
-        self, unfiltered_dataframe: DataFrame, pair: str, dk: FreqaiDataKitchen
+        self, unfiltered_df: DataFrame, pair: str, dk: FreqaiDataKitchen, **kwargs
    ) -> Any:
        """
        Filter the training data and train a model to it. Train makes heavy use of the datakitchen
        for storing, saving, loading, and analyzing the data.
-        :param unfiltered_dataframe: Full dataframe for the current training period
+        :param unfiltered_df: Full dataframe for the current training period
        :param metadata: pair metadata from strategy.
        :return:
        :model: Trained model which can be used to inference (self.predict)
@ -36,14 +36,14 @@ class BaseClassifierModel(IFreqaiModel):
        # filter the features requested by user in the configuration file and elegantly handle NaNs
        features_filtered, labels_filtered = dk.filter_features(
-            unfiltered_dataframe,
+            unfiltered_df,
            dk.training_features_list,
            dk.label_list,
            training_filter=True,
        )
-        start_date = unfiltered_dataframe["date"].iloc[0].strftime("%Y-%m-%d")
+        start_date = unfiltered_df["date"].iloc[0].strftime("%Y-%m-%d")
-        end_date = unfiltered_dataframe["date"].iloc[-1].strftime("%Y-%m-%d")
+        end_date = unfiltered_df["date"].iloc[-1].strftime("%Y-%m-%d")
        logger.info(f"-------------------- Training on data from {start_date} to "
                    f"{end_date}--------------------")
        # split data into train/test data.
@ -61,32 +61,32 @@ class BaseClassifierModel(IFreqaiModel):
            f' features and {len(data_dictionary["train_features"])} data points'
        )
-        model = self.fit(data_dictionary)
+        model = self.fit(data_dictionary, dk)
        logger.info(f"--------------------done training {pair}--------------------")
        return model
    def predict(
-        self, unfiltered_dataframe: DataFrame, dk: FreqaiDataKitchen, first: bool = False
+        self, unfiltered_df: DataFrame, dk: FreqaiDataKitchen, **kwargs
    ) -> Tuple[DataFrame, npt.NDArray[np.int_]]:
        """
        Filter the prediction features data and predict with it.
-        :param: unfiltered_dataframe: Full dataframe for the current backtest period.
+        :param: unfiltered_df: Full dataframe for the current backtest period.
        :return:
        :pred_df: dataframe containing the predictions
        :do_predict: np.array of 1s and 0s to indicate places where freqai needed to remove
        data (NaNs) or felt uncertain about data (PCA and DI index)
        """
-        dk.find_features(unfiltered_dataframe)
+        dk.find_features(unfiltered_df)
-        filtered_dataframe, _ = dk.filter_features(
+        filtered_df, _ = dk.filter_features(
-            unfiltered_dataframe, dk.training_features_list, training_filter=False
+            unfiltered_df, dk.training_features_list, training_filter=False
        )
-        filtered_dataframe = dk.normalize_data_from_metadata(filtered_dataframe)
+        filtered_df = dk.normalize_data_from_metadata(filtered_df)
-        dk.data_dictionary["prediction_features"] = filtered_dataframe
+        dk.data_dictionary["prediction_features"] = filtered_df
-        self.data_cleaning_predict(dk, filtered_dataframe)
+        self.data_cleaning_predict(dk, filtered_df)
        predictions = self.model.predict(dk.data_dictionary["prediction_features"])
        pred_df = DataFrame(predictions, columns=dk.label_list)
--- a/freqtrade/freqai/prediction_models/BaseRegressionModel.py
+++ b/freqtrade/freqai/prediction_models/BaseRegressionModel.py
@ -20,12 +20,12 @@ class BaseRegressionModel(IFreqaiModel):
    """
    def train(
-        self, unfiltered_dataframe: DataFrame, pair: str, dk: FreqaiDataKitchen
+        self, unfiltered_df: DataFrame, pair: str, dk: FreqaiDataKitchen, **kwargs
    ) -> Any:
        """
        Filter the training data and train a model to it. Train makes heavy use of the datakitchen
        for storing, saving, loading, and analyzing the data.
-        :param unfiltered_dataframe: Full dataframe for the current training period
+        :param unfiltered_df: Full dataframe for the current training period
        :param metadata: pair metadata from strategy.
        :return:
        :model: Trained model which can be used to inference (self.predict)
@ -35,14 +35,14 @@ class BaseRegressionModel(IFreqaiModel):
        # filter the features requested by user in the configuration file and elegantly handle NaNs
        features_filtered, labels_filtered = dk.filter_features(
-            unfiltered_dataframe,
+            unfiltered_df,
            dk.training_features_list,
            dk.label_list,
            training_filter=True,
        )
-        start_date = unfiltered_dataframe["date"].iloc[0].strftime("%Y-%m-%d")
+        start_date = unfiltered_df["date"].iloc[0].strftime("%Y-%m-%d")
-        end_date = unfiltered_dataframe["date"].iloc[-1].strftime("%Y-%m-%d")
+        end_date = unfiltered_df["date"].iloc[-1].strftime("%Y-%m-%d")
        logger.info(f"-------------------- Training on data from {start_date} to "
                    f"{end_date}--------------------")
        # split data into train/test data.
@ -60,33 +60,33 @@ class BaseRegressionModel(IFreqaiModel):
            f' features and {len(data_dictionary["train_features"])} data points'
        )
-        model = self.fit(data_dictionary)
+        model = self.fit(data_dictionary, dk)
        logger.info(f"--------------------done training {pair}--------------------")
        return model
    def predict(
-        self, unfiltered_dataframe: DataFrame, dk: FreqaiDataKitchen, first: bool = False
+        self, unfiltered_df: DataFrame, dk: FreqaiDataKitchen, **kwargs
    ) -> Tuple[DataFrame, npt.NDArray[np.int_]]:
        """
        Filter the prediction features data and predict with it.
-        :param: unfiltered_dataframe: Full dataframe for the current backtest period.
+        :param: unfiltered_df: Full dataframe for the current backtest period.
        :return:
        :pred_df: dataframe containing the predictions
        :do_predict: np.array of 1s and 0s to indicate places where freqai needed to remove
        data (NaNs) or felt uncertain about data (PCA and DI index)
        """
-        dk.find_features(unfiltered_dataframe)
+        dk.find_features(unfiltered_df)
-        filtered_dataframe, _ = dk.filter_features(
+        filtered_df, _ = dk.filter_features(
-            unfiltered_dataframe, dk.training_features_list, training_filter=False
+            unfiltered_df, dk.training_features_list, training_filter=False
        )
-        filtered_dataframe = dk.normalize_data_from_metadata(filtered_dataframe)
+        filtered_df = dk.normalize_data_from_metadata(filtered_df)
-        dk.data_dictionary["prediction_features"] = filtered_dataframe
+        dk.data_dictionary["prediction_features"] = filtered_df
        # optional additional data cleaning/analysis
-        self.data_cleaning_predict(dk, filtered_dataframe)
+        self.data_cleaning_predict(dk, filtered_df)
        predictions = self.model.predict(dk.data_dictionary["prediction_features"])
        pred_df = DataFrame(predictions, columns=dk.label_list)
--- a/freqtrade/freqai/prediction_models/BaseTensorFlowModel.py
+++ b/freqtrade/freqai/prediction_models/BaseTensorFlowModel.py
@ -17,12 +17,12 @@ class BaseTensorFlowModel(IFreqaiModel):
    """
    def train(
-        self, unfiltered_dataframe: DataFrame, pair: str, dk: FreqaiDataKitchen
+        self, unfiltered_df: DataFrame, pair: str, dk: FreqaiDataKitchen, **kwargs
    ) -> Any:
        """
        Filter the training data and train a model to it. Train makes heavy use of the datakitchen
        for storing, saving, loading, and analyzing the data.
-        :param unfiltered_dataframe: Full dataframe for the current training period
+        :param unfiltered_df: Full dataframe for the current training period
        :param metadata: pair metadata from strategy.
        :return:
        :model: Trained model which can be used to inference (self.predict)
@ -32,14 +32,14 @@ class BaseTensorFlowModel(IFreqaiModel):
        # filter the features requested by user in the configuration file and elegantly handle NaNs
        features_filtered, labels_filtered = dk.filter_features(
-            unfiltered_dataframe,
+            unfiltered_df,
            dk.training_features_list,
            dk.label_list,
            training_filter=True,
        )
-        start_date = unfiltered_dataframe["date"].iloc[0].strftime("%Y-%m-%d")
+        start_date = unfiltered_df["date"].iloc[0].strftime("%Y-%m-%d")
-        end_date = unfiltered_dataframe["date"].iloc[-1].strftime("%Y-%m-%d")
+        end_date = unfiltered_df["date"].iloc[-1].strftime("%Y-%m-%d")
        logger.info(f"-------------------- Training on data from {start_date} to "
                    f"{end_date}--------------------")
        # split data into train/test data.
@ -57,7 +57,7 @@ class BaseTensorFlowModel(IFreqaiModel):
            f' features and {len(data_dictionary["train_features"])} data points'
        )
-        model = self.fit(data_dictionary)
+        model = self.fit(data_dictionary, dk)
        logger.info(f"--------------------done training {pair}--------------------")
--- a/freqtrade/freqai/base_models/FreqaiMultiOutputRegressor.py
+++ b/freqtrade/freqai/base_models/FreqaiMultiOutputRegressor.py
@ -0,0 +1,65 @@
 from joblib import Parallel
 from sklearn.multioutput import MultiOutputRegressor, _fit_estimator
 from sklearn.utils.fixes import delayed
 from sklearn.utils.validation import has_fit_parameter
 class FreqaiMultiOutputRegressor(MultiOutputRegressor):
    def fit(self, X, y, sample_weight=None, fit_params=None):
        """Fit the model to data, separately for each output variable.
        Parameters
        ----------
        X : {array-like, sparse matrix} of shape (n_samples, n_features)
            The input data.
        y : {array-like, sparse matrix} of shape (n_samples, n_outputs)
            Multi-output targets. An indicator matrix turns on multilabel
            estimation.
        sample_weight : array-like of shape (n_samples,), default=None
            Sample weights. If `None`, then samples are equally weighted.
            Only supported if the underlying regressor supports sample
            weights.
        fit_params : A list of dicts for the fit_params
            Parameters passed to the ``estimator.fit`` method of each step.
            Each dict may contain same or different values (e.g. different
            eval_sets or init_models)
            .. versionadded:: 0.23
        Returns
        -------
        self : object
            Returns a fitted instance.
        """
        if not hasattr(self.estimator, "fit"):
            raise ValueError("The base estimator should implement a fit method")
        y = self._validate_data(X="no_validation", y=y, multi_output=True)
        if y.ndim == 1:
            raise ValueError(
                "y must have at least two dimensions for "
                "multi-output regression but has only one."
            )
        if sample_weight is not None and not has_fit_parameter(
            self.estimator, "sample_weight"
        ):
            raise ValueError("Underlying estimator does not support sample weights.")
        if not fit_params:
            fit_params = [None] * y.shape[1]
        self.estimators_ = Parallel(n_jobs=self.n_jobs)(
            delayed(_fit_estimator)(
                self.estimator, X, y[:, i], sample_weight, **fit_params[i]
            )
            for i in range(y.shape[1])
        )
        if hasattr(self.estimators_[0], "n_features_in_"):
            self.n_features_in_ = self.estimators_[0].n_features_in_
        if hasattr(self.estimators_[0], "feature_names_in_"):
            self.feature_names_in_ = self.estimators_[0].feature_names_in_
        return
--- a/freqtrade/freqai/data_drawer.py
+++ b/freqtrade/freqai/data_drawer.py
@ -76,6 +76,8 @@ class FreqaiDataDrawer:
            self.full_path / f"follower_dictionary-{self.follower_name}.json"
        )
        self.historic_predictions_path = Path(self.full_path / "historic_predictions.pkl")
        self.historic_predictions_bkp_path = Path(
            self.full_path / "historic_predictions.backup.pkl")
        self.pair_dictionary_path = Path(self.full_path / "pair_dictionary.json")
        self.follow_mode = follow_mode
        if follow_mode:
@ -119,6 +121,7 @@ class FreqaiDataDrawer:
        """
        exists = self.historic_predictions_path.is_file()
        if exists:
            try:
                with open(self.historic_predictions_path, "rb") as fp:
                    self.historic_predictions = cloudpickle.load(fp)
                logger.info(
@ -126,6 +129,13 @@ class FreqaiDataDrawer:
                    "that statistics may be inaccurate if the bot has been offline for "
                    "an extended period of time."
                )
            except EOFError:
                logger.warning(
                    'Historical prediction file was corrupted. Trying to load backup file.')
                with open(self.historic_predictions_bkp_path, "rb") as fp:
                    self.historic_predictions = cloudpickle.load(fp)
                logger.warning('FreqAI successfully loaded the backup historical predictions file.')
        elif not self.follow_mode:
            logger.info("Could not find existing historic_predictions, starting from scratch")
        else:
@ -143,6 +153,9 @@ class FreqaiDataDrawer:
        with open(self.historic_predictions_path, "wb") as fp:
            cloudpickle.dump(self.historic_predictions, fp, protocol=cloudpickle.DEFAULT_PROTOCOL)
        # create a backup
        shutil.copy(self.historic_predictions_path, self.historic_predictions_bkp_path)
    def save_drawer_to_disk(self):
        """
        Save data drawer full of all pair model metadata in present model folder.
--- a/freqtrade/freqai/data_kitchen.py
+++ b/freqtrade/freqai/data_kitchen.py
@ -1,7 +1,8 @@
 import copy
 import datetime
 import logging
 import shutil
 from datetime import datetime, timezone
 from math import cos, sin
 from pathlib import Path
 from typing import Any, Dict, List, Tuple
@ -9,6 +10,7 @@ import numpy as np
 import numpy.typing as npt
 import pandas as pd
 from pandas import DataFrame
 from scipy import stats
 from sklearn import linear_model
 from sklearn.cluster import DBSCAN
 from sklearn.metrics.pairwise import pairwise_distances
@ -182,7 +184,7 @@ class FreqaiDataKitchen:
    def filter_features(
        self,
-        unfiltered_dataframe: DataFrame,
+        unfiltered_df: DataFrame,
        training_feature_list: List,
        label_list: List = list(),
        training_filter: bool = True,
@ -193,31 +195,35 @@ class FreqaiDataKitchen:
        0s in the prediction dataset. However, prediction dataset do_predict will reflect any
        row that had a NaN and will shield user from that prediction.
        :params:
-        :unfiltered_dataframe: the full dataframe for the present training period
+        :unfiltered_df: the full dataframe for the present training period
        :training_feature_list: list, the training feature list constructed by
        self.build_feature_list() according to user specified parameters in the configuration file.
        :labels: the labels for the dataset
        :training_filter: boolean which lets the function know if it is training data or
        prediction data to be filtered.
        :returns:
-        :filtered_dataframe: dataframe cleaned of NaNs and only containing the user
+        :filtered_df: dataframe cleaned of NaNs and only containing the user
        requested feature set.
        :labels: labels cleaned of NaNs.
        """
-        filtered_dataframe = unfiltered_dataframe.filter(training_feature_list, axis=1)
+        filtered_df = unfiltered_df.filter(training_feature_list, axis=1)
-        filtered_dataframe = filtered_dataframe.replace([np.inf, -np.inf], np.nan)
+        filtered_df = filtered_df.replace([np.inf, -np.inf], np.nan)
-        drop_index = pd.isnull(filtered_dataframe).any(1)  # get the rows that have NaNs,
+        drop_index = pd.isnull(filtered_df).any(1)  # get the rows that have NaNs,
        drop_index = drop_index.replace(True, 1).replace(False, 0)  # pep8 requirement.
        if (training_filter):
            const_cols = list((filtered_df.nunique() == 1).loc[lambda x: x].index)
            if const_cols:
                filtered_df = filtered_df.filter(filtered_df.columns.difference(const_cols))
                logger.warning(f"Removed features {const_cols} with constant values.")
            # we don't care about total row number (total no. datapoints) in training, we only care
            # about removing any row with NaNs
            # if labels has multiple columns (user wants to train multiple modelEs), we detect here
-            labels = unfiltered_dataframe.filter(label_list, axis=1)
+            labels = unfiltered_df.filter(label_list, axis=1)
            drop_index_labels = pd.isnull(labels).any(1)
            drop_index_labels = drop_index_labels.replace(True, 1).replace(False, 0)
-            dates = unfiltered_dataframe['date']
+            dates = unfiltered_df['date']
-            filtered_dataframe = filtered_dataframe[
+            filtered_df = filtered_df[
                (drop_index == 0) & (drop_index_labels == 0)
            ]  # dropping values
            labels = labels[
@ -227,13 +233,13 @@ class FreqaiDataKitchen:
                (drop_index == 0) & (drop_index_labels == 0)
            ]
            logger.info(
-                f"dropped {len(unfiltered_dataframe) - len(filtered_dataframe)} training points"
+                f"dropped {len(unfiltered_df) - len(filtered_df)} training points"
-                f" due to NaNs in populated dataset {len(unfiltered_dataframe)}."
+                f" due to NaNs in populated dataset {len(unfiltered_df)}."
            )
-            if (1 - len(filtered_dataframe) / len(unfiltered_dataframe)) > 0.1 and self.live:
+            if (1 - len(filtered_df) / len(unfiltered_df)) > 0.1 and self.live:
-                worst_indicator = str(unfiltered_dataframe.count().idxmin())
+                worst_indicator = str(unfiltered_df.count().idxmin())
                logger.warning(
-                    f" {(1 - len(filtered_dataframe)/len(unfiltered_dataframe)) * 100:.0f} percent "
+                    f" {(1 - len(filtered_df)/len(unfiltered_df)) * 100:.0f} percent "
                    " of training data dropped due to NaNs, model may perform inconsistent "
                    f"with expectations. Verify {worst_indicator}"
                )
@ -242,9 +248,9 @@ class FreqaiDataKitchen:
        else:
            # we are backtesting so we need to preserve row number to send back to strategy,
            # so now we use do_predict to avoid any prediction based on a NaN
-            drop_index = pd.isnull(filtered_dataframe).any(1)
+            drop_index = pd.isnull(filtered_df).any(1)
            self.data["filter_drop_index_prediction"] = drop_index
-            filtered_dataframe.fillna(0, inplace=True)
+            filtered_df.fillna(0, inplace=True)
            # replacing all NaNs with zeros to avoid issues in 'prediction', but any prediction
            # that was based on a single NaN is ultimately protected from buys with do_predict
            drop_index = ~drop_index
@ -253,11 +259,11 @@ class FreqaiDataKitchen:
                logger.info(
                    "dropped %s of %s prediction data points due to NaNs.",
                    len(self.do_predict) - self.do_predict.sum(),
-                    len(filtered_dataframe),
+                    len(filtered_df),
                )
            labels = []
-        return filtered_dataframe, labels
+        return filtered_df, labels
    def build_data_dictionary(
        self,
@ -360,7 +366,7 @@ class FreqaiDataKitchen:
    def denormalize_labels_from_metadata(self, df: DataFrame) -> DataFrame:
        """
-        Normalize a set of data using the mean and standard deviation from
+        Denormalize a set of data using the mean and standard deviation from
        the associated training data.
        :param df: Dataframe of predictions to be denormalized
        """
@ -399,7 +405,7 @@ class FreqaiDataKitchen:
        config_timerange = TimeRange.parse_timerange(self.config["timerange"])
        if config_timerange.stopts == 0:
            config_timerange.stopts = int(
-                datetime.datetime.now(tz=datetime.timezone.utc).timestamp()
+                datetime.now(tz=timezone.utc).timestamp()
            )
        timerange_train = copy.deepcopy(full_timerange)
        timerange_backtest = copy.deepcopy(full_timerange)
@ -416,8 +422,8 @@ class FreqaiDataKitchen:
            timerange_train.stopts = timerange_train.startts + train_period_days
            first = False
-            start = datetime.datetime.utcfromtimestamp(timerange_train.startts)
+            start = datetime.fromtimestamp(timerange_train.startts, tz=timezone.utc)
-            stop = datetime.datetime.utcfromtimestamp(timerange_train.stopts)
+            stop = datetime.fromtimestamp(timerange_train.stopts, tz=timezone.utc)
            tr_training_list.append(start.strftime("%Y%m%d") + "-" + stop.strftime("%Y%m%d"))
            tr_training_list_timerange.append(copy.deepcopy(timerange_train))
@ -430,8 +436,8 @@ class FreqaiDataKitchen:
            if timerange_backtest.stopts > config_timerange.stopts:
                timerange_backtest.stopts = config_timerange.stopts
-            start = datetime.datetime.utcfromtimestamp(timerange_backtest.startts)
+            start = datetime.fromtimestamp(timerange_backtest.startts, tz=timezone.utc)
-            stop = datetime.datetime.utcfromtimestamp(timerange_backtest.stopts)
+            stop = datetime.fromtimestamp(timerange_backtest.stopts, tz=timezone.utc)
            tr_backtesting_list.append(start.strftime("%Y%m%d") + "-" + stop.strftime("%Y%m%d"))
            tr_backtesting_list_timerange.append(copy.deepcopy(timerange_backtest))
@ -451,13 +457,35 @@ class FreqaiDataKitchen:
                   it is sliced down to just the present training period.
        """
-        start = datetime.datetime.fromtimestamp(timerange.startts, tz=datetime.timezone.utc)
+        start = datetime.fromtimestamp(timerange.startts, tz=timezone.utc)
-        stop = datetime.datetime.fromtimestamp(timerange.stopts, tz=datetime.timezone.utc)
+        stop = datetime.fromtimestamp(timerange.stopts, tz=timezone.utc)
        df = df.loc[df["date"] >= start, :]
-        df = df.loc[df["date"] <= stop, :]
+        if not self.live:
            df = df.loc[df["date"] < stop, :]
        return df
    def remove_training_from_backtesting(
        self
    ) -> DataFrame:
        """
        Function which takes the backtesting time range and
        remove training data from dataframe, keeping only the
        startup_candle_count candles
        """
        startup_candle_count = self.config.get('startup_candle_count', 0)
        tf = self.config['timeframe']
        tr = self.config["timerange"]
        backtesting_timerange = TimeRange.parse_timerange(tr)
        if startup_candle_count > 0 and backtesting_timerange:
            backtesting_timerange.subtract_start(timeframe_to_seconds(tf) * startup_candle_count)
        start = datetime.fromtimestamp(backtesting_timerange.startts, tz=timezone.utc)
        df = self.return_dataframe
        df = df.loc[df["date"] >= start, :]
        return df
    def principal_component_analysis(self) -> None:
        """
        Performs Principal Component Analysis on the data for dimensionality reduction
@ -652,8 +680,6 @@ class FreqaiDataKitchen:
        is an outlier.
        """
        from math import cos, sin
        if predict:
            if not self.data['DBSCAN_eps']:
                return
@ -746,6 +772,111 @@ class FreqaiDataKitchen:
        return
    def compute_inlier_metric(self, set_='train') -> None:
        """
        Compute inlier metric from backwards distance distributions.
        This metric defines how well features from a timepoint fit
        into previous timepoints.
        """
        no_prev_pts = self.freqai_config["feature_parameters"]["inlier_metric_window"]
        if set_ == 'train':
            compute_df = copy.deepcopy(self.data_dictionary['train_features'])
        elif set_ == 'test':
            compute_df = copy.deepcopy(self.data_dictionary['test_features'])
        else:
            compute_df = copy.deepcopy(self.data_dictionary['prediction_features'])
        compute_df_reindexed = compute_df.reindex(
            index=np.flip(compute_df.index)
        )
        pairwise = pd.DataFrame(
            np.triu(
                pairwise_distances(compute_df_reindexed, n_jobs=self.thread_count)
            ),
            columns=compute_df_reindexed.index,
            index=compute_df_reindexed.index
        )
        pairwise = pairwise.round(5)
        column_labels = [
            '{}{}'.format('d', i) for i in range(1, no_prev_pts + 1)
        ]
        distances = pd.DataFrame(
            columns=column_labels, index=compute_df.index
        )
        for index in compute_df.index[no_prev_pts:]:
            current_row = pairwise.loc[[index]]
            current_row_no_zeros = current_row.loc[
                :, (current_row != 0).any(axis=0)
            ]
            distances.loc[[index]] = current_row_no_zeros.iloc[
                :, :no_prev_pts
            ]
        distances = distances.replace([np.inf, -np.inf], np.nan)
        drop_index = pd.isnull(distances).any(1)
        distances = distances[drop_index == 0]
        inliers = pd.DataFrame(index=distances.index)
        for key in distances.keys():
            current_distances = distances[key].dropna()
            fit_params = stats.weibull_min.fit(current_distances)
            quantiles = stats.weibull_min.cdf(current_distances, *fit_params)
            df_inlier = pd.DataFrame(
                {key: quantiles}, index=distances.index
            )
            inliers = pd.concat(
                [inliers, df_inlier], axis=1
            )
        inlier_metric = pd.DataFrame(
            data=inliers.sum(axis=1) / no_prev_pts,
            columns=['inlier_metric'],
            index=compute_df.index
        )
        inlier_metric = (2 * (inlier_metric - inlier_metric.min()) /
                         (inlier_metric.max() - inlier_metric.min()) - 1)
        if set_ in ('train', 'test'):
            inlier_metric = inlier_metric.iloc[no_prev_pts:]
            compute_df = compute_df.iloc[no_prev_pts:]
            self.remove_beginning_points_from_data_dict(set_, no_prev_pts)
            self.data_dictionary[f'{set_}_features'] = pd.concat(
                [compute_df, inlier_metric], axis=1)
        else:
            self.data_dictionary['prediction_features'] = pd.concat(
                [compute_df, inlier_metric], axis=1)
            self.data_dictionary['prediction_features'].fillna(0, inplace=True)
        logger.info('Inlier metric computed and added to features.')
        return None
    def remove_beginning_points_from_data_dict(self, set_='train', no_prev_pts: int = 10):
        features = self.data_dictionary[f'{set_}_features']
        weights = self.data_dictionary[f'{set_}_weights']
        labels = self.data_dictionary[f'{set_}_labels']
        self.data_dictionary[f'{set_}_weights'] = weights[no_prev_pts:]
        self.data_dictionary[f'{set_}_features'] = features.iloc[no_prev_pts:]
        self.data_dictionary[f'{set_}_labels'] = labels.iloc[no_prev_pts:]
    def add_noise_to_training_features(self) -> None:
        """
        Add noise to train features to reduce the risk of overfitting.
        """
        mu = 0  # no shift
        sigma = self.freqai_config["feature_parameters"]["noise_standard_deviation"]
        compute_df = self.data_dictionary['train_features']
        noise = np.random.normal(mu, sigma, [compute_df.shape[0], compute_df.shape[1]])
        self.data_dictionary['train_features'] += noise
        return
    def find_features(self, dataframe: DataFrame) -> None:
        """
        Find features in the strategy provided dataframe
@ -848,6 +979,7 @@ class FreqaiDataKitchen:
        to_keep = [col for col in dataframe.columns if not col.startswith("&")]
        self.return_dataframe = pd.concat([dataframe[to_keep], self.full_df], axis=1)
        self.return_dataframe = self.remove_training_from_backtesting()
        self.full_df = DataFrame()
        return
@ -871,14 +1003,14 @@ class FreqaiDataKitchen:
                                       "Please indicate the end date of your desired backtesting. "
                                       "timerange.")
            # backtest_timerange.stopts = int(
-            #     datetime.datetime.now(tz=datetime.timezone.utc).timestamp()
+            #     datetime.now(tz=timezone.utc).timestamp()
            # )
        backtest_timerange.startts = (
            backtest_timerange.startts - backtest_period_days * SECONDS_IN_DAY
        )
-        start = datetime.datetime.utcfromtimestamp(backtest_timerange.startts)
+        start = datetime.fromtimestamp(backtest_timerange.startts, tz=timezone.utc)
-        stop = datetime.datetime.utcfromtimestamp(backtest_timerange.stopts)
+        stop = datetime.fromtimestamp(backtest_timerange.stopts, tz=timezone.utc)
        full_timerange = start.strftime("%Y%m%d") + "-" + stop.strftime("%Y%m%d")
        self.full_path = Path(
@ -904,7 +1036,7 @@ class FreqaiDataKitchen:
        :return:
            bool = If the model is expired or not.
        """
-        time = datetime.datetime.now(tz=datetime.timezone.utc).timestamp()
+        time = datetime.now(tz=timezone.utc).timestamp()
        elapsed_time = (time - trained_timestamp) / 3600  # hours
        max_time = self.freqai_config.get("expiration_hours", 0)
        if max_time > 0:
@ -916,7 +1048,7 @@ class FreqaiDataKitchen:
        self, trained_timestamp: int
    ) -> Tuple[bool, TimeRange, TimeRange]:
-        time = datetime.datetime.now(tz=datetime.timezone.utc).timestamp()
+        time = datetime.now(tz=timezone.utc).timestamp()
        trained_timerange = TimeRange()
        data_load_timerange = TimeRange()
@ -1094,7 +1226,6 @@ class FreqaiDataKitchen:
    def save_backtesting_prediction(
        self, append_df: DataFrame
    ) -> None:
        """
        Save prediction dataframe from backtesting to h5 file format
        :param append_df: dataframe for backtesting period
@ -1108,7 +1239,6 @@ class FreqaiDataKitchen:
    def get_backtesting_prediction(
        self
    ) -> DataFrame:
        """
        Get prediction dataframe from h5 file format
        """
--- a/freqtrade/freqai/freqai_interface.py
+++ b/freqtrade/freqai/freqai_interface.py
@ -1,13 +1,12 @@
 # import contextlib
 import datetime
 import logging
 import shutil
 import threading
 import time
 from abc import ABC, abstractmethod
 from datetime import datetime, timezone
 from pathlib import Path
 from threading import Lock
-from typing import Any, Dict, Optional, Tuple
+from typing import Any, Dict, List, Optional, Tuple
 import numpy as np
 import pandas as pd
@ -15,6 +14,7 @@ from numpy.typing import NDArray
 from pandas import DataFrame
 from freqtrade.configuration import TimeRange
 from freqtrade.constants import DATETIME_PRINT_FORMAT
 from freqtrade.enums import RunMode
 from freqtrade.exceptions import OperationalException
 from freqtrade.exchange import timeframe_to_seconds
@ -27,13 +27,6 @@ pd.options.mode.chained_assignment = None
 logger = logging.getLogger(__name__)
 def threaded(fn):
    def wrapper(*args, **kwargs):
        threading.Thread(target=fn, args=args, kwargs=kwargs).start()
    return wrapper
 class IFreqaiModel(ABC):
    """
    Class containing all tools for training and prediction in the strategy.
@ -66,7 +59,6 @@ class IFreqaiModel(ABC):
            "data_split_parameters", {})
        self.model_training_parameters: Dict[str, Any] = config.get("freqai", {}).get(
            "model_training_parameters", {})
        self.feature_parameters = config.get("freqai", {}).get("feature_parameters")
        self.retrain = False
        self.first = True
        self.set_full_path()
@ -77,11 +69,14 @@ class IFreqaiModel(ABC):
        self.dd = FreqaiDataDrawer(Path(self.full_path), self.config, self.follow_mode)
        self.identifier: str = self.freqai_info.get("identifier", "no_id_provided")
        self.scanning = False
        self.ft_params = self.freqai_info["feature_parameters"]
        self.keras: bool = self.freqai_info.get("keras", False)
-        if self.keras and self.freqai_info.get("feature_parameters", {}).get("DI_threshold", 0):
+        if self.keras and self.ft_params.get("DI_threshold", 0):
-            self.freqai_info["feature_parameters"]["DI_threshold"] = 0
+            self.ft_params["DI_threshold"] = 0
            logger.warning("DI threshold is not configured for Keras models yet. Deactivating.")
        self.CONV_WIDTH = self.freqai_info.get("conv_width", 2)
        if self.ft_params.get("inlier_metric_window", 0):
            self.CONV_WIDTH = self.ft_params.get("inlier_metric_window", 0) * 2
        self.pair_it = 0
        self.pair_it_train = 0
        self.total_pairs = len(self.config.get("exchange", {}).get("pair_whitelist"))
@ -93,6 +88,16 @@ class IFreqaiModel(ABC):
        self.begin_time: float = 0
        self.begin_time_train: float = 0
        self.base_tf_seconds = timeframe_to_seconds(self.config['timeframe'])
        self.continual_learning = self.freqai_info.get('continual_learning', False)
        self._threads: List[threading.Thread] = []
        self._stop_event = threading.Event()
    def __getstate__(self):
        """
        Return an empty state to be pickled in hyperopt
        """
        return ({})
        self.strategy: Optional[IStrategy] = None
    def assert_config(self, config: Dict[str, Any]) -> None:
@ -148,15 +153,34 @@ class IFreqaiModel(ABC):
        self.model = None
        self.dk = None
-    @threaded
+    def shutdown(self):
-    def start_scanning(self, strategy: IStrategy) -> None:
+        """
        Cleans up threads on Shutdown, set stop event. Join threads to wait
        for current training iteration.
        """
        logger.info("Stopping FreqAI")
        self._stop_event.set()
        logger.info("Waiting on Training iteration")
        for _thread in self._threads:
            _thread.join()
    def start_scanning(self, *args, **kwargs) -> None:
        """
        Start `self._start_scanning` in a separate thread
        """
        _thread = threading.Thread(target=self._start_scanning, args=args, kwargs=kwargs)
        self._threads.append(_thread)
        _thread.start()
    def _start_scanning(self, strategy: IStrategy) -> None:
        """
        Function designed to constantly scan pairs for retraining on a separate thread (intracandle)
        to improve model youth. This function is agnostic to data preparation/collection/storage,
        it simply trains on what ever data is available in the self.dd.
        :param strategy: IStrategy = The user defined strategy class
        """
-        while 1:
+        while not self._stop_event.is_set():
            time.sleep(1)
            for pair in self.config.get("exchange", {}).get("pair_whitelist"):
@ -175,7 +199,7 @@ class IFreqaiModel(ABC):
                if retrain:
                    self.train_timer('start')
-                    self.train_model_in_series(
+                    self.extract_data_and_train_model(
                        new_trained_timerange, pair, strategy, dk, data_load_timerange
                    )
                    self.train_timer('stop')
@ -215,12 +239,12 @@ class IFreqaiModel(ABC):
            dataframe_backtest = dk.slice_dataframe(tr_backtest, dataframe)
            trained_timestamp = tr_train
-            tr_train_startts_str = datetime.datetime.utcfromtimestamp(tr_train.startts).strftime(
+            tr_train_startts_str = datetime.fromtimestamp(
-                "%Y-%m-%d %H:%M:%S"
+                                                tr_train.startts,
-            )
+                                                tz=timezone.utc).strftime(DATETIME_PRINT_FORMAT)
-            tr_train_stopts_str = datetime.datetime.utcfromtimestamp(tr_train.stopts).strftime(
+            tr_train_stopts_str = datetime.fromtimestamp(
-                "%Y-%m-%d %H:%M:%S"
+                                                tr_train.stopts,
-            )
+                                                tz=timezone.utc).strftime(DATETIME_PRINT_FORMAT)
            logger.info(
                f"Training {metadata['pair']}, {self.pair_it}/{self.total_pairs} pairs"
                f" from {tr_train_startts_str} to {tr_train_stopts_str}, {train_it}/{total_trains} "
@ -405,24 +429,30 @@ class IFreqaiModel(ABC):
    def data_cleaning_train(self, dk: FreqaiDataKitchen) -> None:
        """
-        Base data cleaning method for train
+        Base data cleaning method for train.
-        Any function inside this method should drop training data points from the filtered_dataframe
+        Functions here improve/modify the input data by identifying outliers,
-        based on user decided logic. See FreqaiDataKitchen::use_SVM_to_remove_outliers() for an
+        computing additional metrics, adding noise, reducing dimensionality etc.
        example of how outlier data points are dropped from the dataframe used for training.
        """
-        if self.freqai_info["feature_parameters"].get(
+        ft_params = self.freqai_info["feature_parameters"]
        if ft_params.get('inlier_metric_window', 0):
            dk.compute_inlier_metric(set_='train')
            if self.freqai_info["data_split_parameters"]["test_size"] > 0:
                dk.compute_inlier_metric(set_='test')
        if ft_params.get(
            "principal_component_analysis", False
        ):
            dk.principal_component_analysis()
-        if self.freqai_info["feature_parameters"].get("use_SVM_to_remove_outliers", False):
+        if ft_params.get("use_SVM_to_remove_outliers", False):
            dk.use_SVM_to_remove_outliers(predict=False)
-        if self.freqai_info["feature_parameters"].get("DI_threshold", 0):
+        if ft_params.get("DI_threshold", 0):
            dk.data["avg_mean_dist"] = dk.compute_distances()
-        if self.freqai_info["feature_parameters"].get("use_DBSCAN_to_remove_outliers", False):
+        if ft_params.get("use_DBSCAN_to_remove_outliers", False):
            if dk.pair in self.dd.old_DBSCAN_eps:
                eps = self.dd.old_DBSCAN_eps[dk.pair]
            else:
@ -430,29 +460,31 @@ class IFreqaiModel(ABC):
            dk.use_DBSCAN_to_remove_outliers(predict=False, eps=eps)
            self.dd.old_DBSCAN_eps[dk.pair] = dk.data['DBSCAN_eps']
        if self.freqai_info["feature_parameters"].get('noise_standard_deviation', 0):
            dk.add_noise_to_training_features()
    def data_cleaning_predict(self, dk: FreqaiDataKitchen, dataframe: DataFrame) -> None:
        """
        Base data cleaning method for predict.
-        These functions each modify dk.do_predict, which is a dataframe with equal length
+        Functions here are complementary to the functions of data_cleaning_train.
        to the number of candles coming from and returning to the strategy. Inside do_predict,
         1 allows prediction and < 0 signals to the strategy that the model is not confident in
         the prediction.
         See FreqaiDataKitchen::remove_outliers() for an example
        of how the do_predict vector is modified. do_predict is ultimately passed back to strategy
        for buy signals.
        """
-        if self.freqai_info["feature_parameters"].get(
+        ft_params = self.freqai_info["feature_parameters"]
        if ft_params.get('inlier_metric_window', 0):
            dk.compute_inlier_metric(set_='predict')
        if ft_params.get(
            "principal_component_analysis", False
        ):
-            dk.pca_transform(dataframe)
+            dk.pca_transform(self.dk.data_dictionary['prediction_features'])
-        if self.freqai_info["feature_parameters"].get("use_SVM_to_remove_outliers", False):
+        if ft_params.get("use_SVM_to_remove_outliers", False):
            dk.use_SVM_to_remove_outliers(predict=True)
-        if self.freqai_info["feature_parameters"].get("DI_threshold", 0):
+        if ft_params.get("DI_threshold", 0):
            dk.check_if_pred_in_training_spaces()
-        if self.freqai_info["feature_parameters"].get("use_DBSCAN_to_remove_outliers", False):
+        if ft_params.get("use_DBSCAN_to_remove_outliers", False):
            dk.use_DBSCAN_to_remove_outliers(predict=True)
    def model_exists(
@ -488,7 +520,7 @@ class IFreqaiModel(ABC):
            Path(self.full_path, Path(self.config["config_files"][0]).name),
        )
-    def train_model_in_series(
+    def extract_data_and_train_model(
        self,
        new_trained_timerange: TimeRange,
        pair: str,
@ -580,7 +612,7 @@ class IFreqaiModel(ABC):
        # # for keras type models, the conv_window needs to be prepended so
        # # viewing is correct in frequi
-        if self.freqai_info.get('keras', False):
+        if self.freqai_info.get('keras', False) or self.ft_params.get('inlier_metric_window', 0):
            n_lost_points = self.freqai_info.get('conv_width', 2)
            zeros_df = DataFrame(np.zeros((n_lost_points, len(hist_preds_df.columns))),
                                 columns=hist_preds_df.columns)
@ -646,21 +678,30 @@ class IFreqaiModel(ABC):
                self.train_time = 0
        return
    def get_init_model(self, pair: str) -> Any:
        if pair not in self.dd.model_dictionary or not self.continual_learning:
            init_model = None
        else:
            init_model = self.dd.model_dictionary[pair]
        return init_model
    # Following methods which are overridden by user made prediction models.
    # See freqai/prediction_models/CatboostPredictionModel.py for an example.
    @abstractmethod
-    def train(self, unfiltered_dataframe: DataFrame, pair: str, dk: FreqaiDataKitchen) -> Any:
+    def train(self, unfiltered_df: DataFrame, pair: str,
              dk: FreqaiDataKitchen, **kwargs) -> Any:
        """
        Filter the training data and train a model to it. Train makes heavy use of the datahandler
        for storing, saving, loading, and analyzing the data.
-        :param unfiltered_dataframe: Full dataframe for the current training period
+        :param unfiltered_df: Full dataframe for the current training period
        :param metadata: pair metadata from strategy.
        :return: Trained model which can be used to inference (self.predict)
        """
    @abstractmethod
-    def fit(self, data_dictionary: Dict[str, Any], pair: str = '') -> Any:
+    def fit(self, data_dictionary: Dict[str, Any], dk: FreqaiDataKitchen, **kwargs) -> Any:
        """
        Most regressors use the same function names and arguments e.g. user
        can drop in LGBMRegressor in place of CatBoostRegressor and all data
@ -673,11 +714,11 @@ class IFreqaiModel(ABC):
    @abstractmethod
    def predict(
-        self, dataframe: DataFrame, dk: FreqaiDataKitchen, first: bool = True
+        self, unfiltered_df: DataFrame, dk: FreqaiDataKitchen, **kwargs
    ) -> Tuple[DataFrame, NDArray[np.int_]]:
        """
        Filter the prediction features data and predict with it.
-        :param unfiltered_dataframe: Full dataframe for the current backtest period.
+        :param unfiltered_df: Full dataframe for the current backtest period.
        :param dk: FreqaiDataKitchen = Data management/analysis tool associated to present pair only
        :param first: boolean = whether this is the first prediction or not.
        :return:
--- a/freqtrade/freqai/prediction_models/CatboostClassifier.py
+++ b/freqtrade/freqai/prediction_models/CatboostClassifier.py
@ -3,7 +3,8 @@ from typing import Any, Dict
 from catboost import CatBoostClassifier, Pool
-from freqtrade.freqai.prediction_models.BaseClassifierModel import BaseClassifierModel
+from freqtrade.freqai.base_models.BaseClassifierModel import BaseClassifierModel
 from freqtrade.freqai.data_kitchen import FreqaiDataKitchen
 logger = logging.getLogger(__name__)
@ -16,7 +17,7 @@ class CatboostClassifier(BaseClassifierModel):
    has its own DataHandler where data is held, saved, loaded, and managed.
    """
-    def fit(self, data_dictionary: Dict[str, Any], pair: str = '') -> Any:
+    def fit(self, data_dictionary: Dict, dk: FreqaiDataKitchen, **kwargs) -> Any:
        """
        User sets up the training and test data to fit their desired model here
        :params:
@ -36,6 +37,8 @@ class CatboostClassifier(BaseClassifierModel):
            **self.model_training_parameters,
        )
-        cbr.fit(train_data)
+        init_model = self.get_init_model(dk.pair)
        cbr.fit(train_data, init_model=init_model)
        return cbr
--- a/freqtrade/freqai/prediction_models/CatboostRegressor.py
+++ b/freqtrade/freqai/prediction_models/CatboostRegressor.py
@ -1,10 +1,10 @@
 import gc
 import logging
 from typing import Any, Dict
 from catboost import CatBoostRegressor, Pool
-from freqtrade.freqai.prediction_models.BaseRegressionModel import BaseRegressionModel
+from freqtrade.freqai.base_models.BaseRegressionModel import BaseRegressionModel
 from freqtrade.freqai.data_kitchen import FreqaiDataKitchen
 logger = logging.getLogger(__name__)
@ -17,7 +17,7 @@ class CatboostRegressor(BaseRegressionModel):
    has its own DataHandler where data is held, saved, loaded, and managed.
    """
-    def fit(self, data_dictionary: Dict[str, Any], pair: str = '') -> Any:
+    def fit(self, data_dictionary: Dict, dk: FreqaiDataKitchen, **kwargs) -> Any:
        """
        User sets up the training and test data to fit their desired model here
        :param data_dictionary: the dictionary constructed by DataHandler to hold
@ -38,16 +38,13 @@ class CatboostRegressor(BaseRegressionModel):
                weight=data_dictionary["test_weights"],
            )
        init_model = self.get_init_model(dk.pair)
        model = CatBoostRegressor(
            allow_writing_files=False,
            **self.model_training_parameters,
        )
-        model.fit(X=train_data, eval_set=test_data)
+        model.fit(X=train_data, eval_set=test_data, init_model=init_model)
        # some evidence that catboost pools have memory leaks:
        # https://github.com/catboost/catboost/issues/1835
        del train_data, test_data
        gc.collect()
        return model
--- a/freqtrade/freqai/prediction_models/CatboostRegressorMultiTarget.py
+++ b/freqtrade/freqai/prediction_models/CatboostRegressorMultiTarget.py
@ -1,10 +1,11 @@
 import logging
 from typing import Any, Dict
-from catboost import CatBoostRegressor  # , Pool
+from catboost import CatBoostRegressor, Pool
 from sklearn.multioutput import MultiOutputRegressor
-from freqtrade.freqai.prediction_models.BaseRegressionModel import BaseRegressionModel
+from freqtrade.freqai.base_models.BaseRegressionModel import BaseRegressionModel
 from freqtrade.freqai.base_models.FreqaiMultiOutputRegressor import FreqaiMultiOutputRegressor
 from freqtrade.freqai.data_kitchen import FreqaiDataKitchen
 logger = logging.getLogger(__name__)
@ -17,7 +18,7 @@ class CatboostRegressorMultiTarget(BaseRegressionModel):
    has its own DataHandler where data is held, saved, loaded, and managed.
    """
-    def fit(self, data_dictionary: Dict) -> Any:
+    def fit(self, data_dictionary: Dict, dk: FreqaiDataKitchen, **kwargs) -> Any:
        """
        User sets up the training and test data to fit their desired model here
        :param data_dictionary: the dictionary constructed by DataHandler to hold
@ -31,14 +32,37 @@ class CatboostRegressorMultiTarget(BaseRegressionModel):
        X = data_dictionary["train_features"]
        y = data_dictionary["train_labels"]
-        eval_set = (data_dictionary["test_features"], data_dictionary["test_labels"])
+
        sample_weight = data_dictionary["train_weights"]
-        model = MultiOutputRegressor(estimator=cbr)
+        eval_sets = [None] * y.shape[1]
        model.fit(X=X, y=y, sample_weight=sample_weight)  # , eval_set=eval_set)
        if self.freqai_info.get('data_split_parameters', {}).get('test_size', 0.1) != 0:
-            train_score = model.score(X, y)
+            eval_sets = [None] * data_dictionary['test_labels'].shape[1]
-            test_score = model.score(*eval_set)
+
-            logger.info(f"Train score {train_score}, Test score {test_score}")
+            for i in range(data_dictionary['test_labels'].shape[1]):
                eval_sets[i] = Pool(
                    data=data_dictionary["test_features"],
                    label=data_dictionary["test_labels"].iloc[:, i],
                    weight=data_dictionary["test_weights"],
                )
        init_model = self.get_init_model(dk.pair)
        if init_model:
            init_models = init_model.estimators_
        else:
            init_models = [None] * y.shape[1]
        fit_params = []
        for i in range(len(eval_sets)):
            fit_params.append(
                {'eval_set': eval_sets[i],  'init_model': init_models[i]})
        model = FreqaiMultiOutputRegressor(estimator=cbr)
        thread_training = self.freqai_info.get('multitarget_parallel_training', False)
        if thread_training:
            model.n_jobs = y.shape[1]
        model.fit(X=X, y=y, sample_weight=sample_weight, fit_params=fit_params)
        return model
--- a/freqtrade/freqai/prediction_models/LightGBMClassifier.py
+++ b/freqtrade/freqai/prediction_models/LightGBMClassifier.py
@ -3,7 +3,8 @@ from typing import Any, Dict
 from lightgbm import LGBMClassifier
-from freqtrade.freqai.prediction_models.BaseClassifierModel import BaseClassifierModel
+from freqtrade.freqai.base_models.BaseClassifierModel import BaseClassifierModel
 from freqtrade.freqai.data_kitchen import FreqaiDataKitchen
 logger = logging.getLogger(__name__)
@ -16,7 +17,7 @@ class LightGBMClassifier(BaseClassifierModel):
    has its own DataHandler where data is held, saved, loaded, and managed.
    """
-    def fit(self, data_dictionary: Dict) -> Any:
+    def fit(self, data_dictionary: Dict, dk: FreqaiDataKitchen, **kwargs) -> Any:
        """
        User sets up the training and test data to fit their desired model here
        :params:
@ -35,9 +36,11 @@ class LightGBMClassifier(BaseClassifierModel):
        y = data_dictionary["train_labels"].to_numpy()[:, 0]
        train_weights = data_dictionary["train_weights"]
        init_model = self.get_init_model(dk.pair)
        model = LGBMClassifier(**self.model_training_parameters)
        model.fit(X=X, y=y, eval_set=eval_set, sample_weight=train_weights,
-                  eval_sample_weight=[test_weights])
+                  eval_sample_weight=[test_weights], init_model=init_model)
        return model
--- a/freqtrade/freqai/prediction_models/LightGBMRegressor.py
+++ b/freqtrade/freqai/prediction_models/LightGBMRegressor.py
@ -3,7 +3,8 @@ from typing import Any, Dict
 from lightgbm import LGBMRegressor
-from freqtrade.freqai.prediction_models.BaseRegressionModel import BaseRegressionModel
+from freqtrade.freqai.base_models.BaseRegressionModel import BaseRegressionModel
 from freqtrade.freqai.data_kitchen import FreqaiDataKitchen
 logger = logging.getLogger(__name__)
@ -16,7 +17,7 @@ class LightGBMRegressor(BaseRegressionModel):
    has its own DataHandler where data is held, saved, loaded, and managed.
    """
-    def fit(self, data_dictionary: Dict) -> Any:
+    def fit(self, data_dictionary: Dict, dk: FreqaiDataKitchen, **kwargs) -> Any:
        """
        Most regressors use the same function names and arguments e.g. user
        can drop in LGBMRegressor in place of CatBoostRegressor and all data
@ -35,9 +36,11 @@ class LightGBMRegressor(BaseRegressionModel):
        y = data_dictionary["train_labels"]
        train_weights = data_dictionary["train_weights"]
        init_model = self.get_init_model(dk.pair)
        model = LGBMRegressor(**self.model_training_parameters)
        model.fit(X=X, y=y, eval_set=eval_set, sample_weight=train_weights,
-                  eval_sample_weight=[eval_weights])
+                  eval_sample_weight=[eval_weights], init_model=init_model)
        return model
--- a/freqtrade/freqai/prediction_models/LightGBMRegressorMultiTarget.py
+++ b/freqtrade/freqai/prediction_models/LightGBMRegressorMultiTarget.py
@ -2,9 +2,10 @@ import logging
 from typing import Any, Dict
 from lightgbm import LGBMRegressor
 from sklearn.multioutput import MultiOutputRegressor
-from freqtrade.freqai.prediction_models.BaseRegressionModel import BaseRegressionModel
+from freqtrade.freqai.base_models.BaseRegressionModel import BaseRegressionModel
 from freqtrade.freqai.base_models.FreqaiMultiOutputRegressor import FreqaiMultiOutputRegressor
 from freqtrade.freqai.data_kitchen import FreqaiDataKitchen
 logger = logging.getLogger(__name__)
@ -17,7 +18,7 @@ class LightGBMRegressorMultiTarget(BaseRegressionModel):
    has its own DataHandler where data is held, saved, loaded, and managed.
    """
-    def fit(self, data_dictionary: Dict) -> Any:
+    def fit(self, data_dictionary: Dict, dk: FreqaiDataKitchen, **kwargs) -> Any:
        """
        User sets up the training and test data to fit their desired model here
        :param data_dictionary: the dictionary constructed by DataHandler to hold
@ -28,12 +29,36 @@ class LightGBMRegressorMultiTarget(BaseRegressionModel):
        X = data_dictionary["train_features"]
        y = data_dictionary["train_labels"]
        eval_set = (data_dictionary["test_features"], data_dictionary["test_labels"])
        sample_weight = data_dictionary["train_weights"]
-        model = MultiOutputRegressor(estimator=lgb)
+        eval_weights = None
-        model.fit(X=X, y=y, sample_weight=sample_weight)  # , eval_set=eval_set)
+        eval_sets = [None] * y.shape[1]
-        train_score = model.score(X, y)
+
-        test_score = model.score(*eval_set)
+        if self.freqai_info.get('data_split_parameters', {}).get('test_size', 0.1) != 0:
-        logger.info(f"Train score {train_score}, Test score {test_score}")
+            eval_weights = [data_dictionary["test_weights"]]
            eval_sets = [(None, None)] * data_dictionary['test_labels'].shape[1]  # type: ignore
            for i in range(data_dictionary['test_labels'].shape[1]):
                eval_sets[i] = (  # type: ignore
                    data_dictionary["test_features"],
                    data_dictionary["test_labels"].iloc[:, i]
                )
        init_model = self.get_init_model(dk.pair)
        if init_model:
            init_models = init_model.estimators_
        else:
            init_models = [None] * y.shape[1]
        fit_params = []
        for i in range(len(eval_sets)):
            fit_params.append(
                {'eval_set': eval_sets[i], 'eval_sample_weight': eval_weights,
                 'init_model': init_models[i]})
        model = FreqaiMultiOutputRegressor(estimator=lgb)
        thread_training = self.freqai_info.get('multitarget_parallel_training', False)
        if thread_training:
            model.n_jobs = y.shape[1]
        model.fit(X=X, y=y, sample_weight=sample_weight, fit_params=fit_params)
        return model
--- a/freqtrade/freqai/prediction_models/XGBoostRegressor.py
+++ b/freqtrade/freqai/prediction_models/XGBoostRegressor.py
@ -0,0 +1,45 @@
 import logging
 from typing import Any, Dict
 from xgboost import XGBRegressor
 from freqtrade.freqai.base_models.BaseRegressionModel import BaseRegressionModel
 from freqtrade.freqai.data_kitchen import FreqaiDataKitchen
 logger = logging.getLogger(__name__)
 class XGBoostRegressor(BaseRegressionModel):
    """
    User created prediction model. The class needs to override three necessary
    functions, predict(), train(), fit(). The class inherits ModelHandler which
    has its own DataHandler where data is held, saved, loaded, and managed.
    """
    def fit(self, data_dictionary: Dict, dk: FreqaiDataKitchen, **kwargs) -> Any:
        """
        User sets up the training and test data to fit their desired model here
        :param data_dictionary: the dictionary constructed by DataHandler to hold
                                all the training and test data/labels.
        """
        X = data_dictionary["train_features"]
        y = data_dictionary["train_labels"]
        if self.freqai_info.get("data_split_parameters", {}).get("test_size", 0.1) == 0:
            eval_set = None
        else:
            eval_set = [(data_dictionary["test_features"], data_dictionary["test_labels"])]
            eval_weights = [data_dictionary['test_weights']]
        sample_weight = data_dictionary["train_weights"]
        xgb_model = self.get_init_model(dk.pair)
        model = XGBRegressor(**self.model_training_parameters)
        model.fit(X=X, y=y, sample_weight=sample_weight, eval_set=eval_set,
                  sample_weight_eval_set=eval_weights, xgb_model=xgb_model)
        return model
--- a/freqtrade/freqai/prediction_models/XGBoostRegressorMultiTarget.py
+++ b/freqtrade/freqai/prediction_models/XGBoostRegressorMultiTarget.py
@ -0,0 +1,63 @@
 import logging
 from typing import Any, Dict
 from xgboost import XGBRegressor
 from freqtrade.freqai.base_models.BaseRegressionModel import BaseRegressionModel
 from freqtrade.freqai.base_models.FreqaiMultiOutputRegressor import FreqaiMultiOutputRegressor
 from freqtrade.freqai.data_kitchen import FreqaiDataKitchen
 logger = logging.getLogger(__name__)
 class XGBoostRegressorMultiTarget(BaseRegressionModel):
    """
    User created prediction model. The class needs to override three necessary
    functions, predict(), train(), fit(). The class inherits ModelHandler which
    has its own DataHandler where data is held, saved, loaded, and managed.
    """
    def fit(self, data_dictionary: Dict, dk: FreqaiDataKitchen, **kwargs) -> Any:
        """
        User sets up the training and test data to fit their desired model here
        :param data_dictionary: the dictionary constructed by DataHandler to hold
                                all the training and test data/labels.
        """
        xgb = XGBRegressor(**self.model_training_parameters)
        X = data_dictionary["train_features"]
        y = data_dictionary["train_labels"]
        sample_weight = data_dictionary["train_weights"]
        eval_weights = None
        eval_sets = [None] * y.shape[1]
        if self.freqai_info.get('data_split_parameters', {}).get('test_size', 0.1) != 0:
            eval_weights = [data_dictionary["test_weights"]]
            for i in range(data_dictionary['test_labels'].shape[1]):
                eval_sets[i] = [(  # type: ignore
                    data_dictionary["test_features"],
                    data_dictionary["test_labels"].iloc[:, i]
                )]
        init_model = self.get_init_model(dk.pair)
        if init_model:
            init_models = init_model.estimators_
        else:
            init_models = [None] * y.shape[1]
        fit_params = []
        for i in range(len(eval_sets)):
            fit_params.append(
                {'eval_set': eval_sets[i], 'sample_weight_eval_set': eval_weights,
                 'xgb_model': init_models[i]})
        model = FreqaiMultiOutputRegressor(estimator=xgb)
        thread_training = self.freqai_info.get('multitarget_parallel_training', False)
        if thread_training:
            model.n_jobs = y.shape[1]
        model.fit(X=X, y=y, sample_weight=sample_weight, fit_params=fit_params)
        return model
--- a/freqtrade/freqtradebot.py
+++ b/freqtrade/freqtradebot.py
@ -142,12 +142,17 @@ class FreqtradeBot(LoggingMixin):
        :return: None
        """
        logger.info('Cleaning up modules ...')
-
+        try:
            # Wrap db activities in shutdown to avoid problems if database is gone,
            # and raises further exceptions.
            if self.config['cancel_open_orders_on_exit']:
                self.cancel_all_open_orders()
            self.check_for_open_trades()
        finally:
            self.strategy.ft_bot_cleanup()
            self.rpc.cleanup()
            Trade.commit()
            self.exchange.close()
@ -276,16 +281,17 @@ class FreqtradeBot(LoggingMixin):
    def update_funding_fees(self):
        if self.trading_mode == TradingMode.FUTURES:
            trades = Trade.get_open_trades()
            try:
                for trade in trades:
                    funding_fees = self.exchange.get_funding_fees(
                        pair=trade.pair,
                        amount=trade.amount,
                        is_short=trade.is_short,
-                    open_date=trade.open_date_utc
+                        open_date=trade.date_last_filled_utc
                    )
                    trade.funding_fees = funding_fees
-        else:
+            except ExchangeError:
-            return 0.0
+                logger.warning("Could not update funding fees for open trades.")
    def startup_backpopulate_precision(self):
@ -578,7 +584,9 @@ class FreqtradeBot(LoggingMixin):
        if stake_amount is not None and stake_amount < 0.0:
            # We should decrease our position
-            amount = abs(float(FtPrecise(stake_amount) / FtPrecise(current_exit_rate)))
+            amount = self.exchange.amount_to_contract_precision(
                trade.pair,
                abs(float(FtPrecise(stake_amount) / FtPrecise(current_exit_rate))))
            if amount > trade.amount:
                # This is currently ineffective as remaining would become < min tradable
                # Fixing this would require checking for 0.0 there -
@ -587,9 +595,14 @@ class FreqtradeBot(LoggingMixin):
                    f"Adjusting amount to trade.amount as it is higher. {amount} > {trade.amount}")
                amount = trade.amount
            if amount == 0.0:
                logger.info("Amount to sell is 0.0 due to exchange limits - not selling.")
                return
            remaining = (trade.amount - amount) * current_exit_rate
            if remaining < min_exit_stake:
-                logger.info(f'Remaining amount of {remaining} would be too small.')
+                logger.info(f"Remaining amount of {remaining} would be smaller "
                            f"than the minimum of {min_exit_stake}.")
                return
            self.execute_trade_exit(trade, current_exit_rate, exit_check=ExitCheckTuple(
@ -659,14 +672,12 @@ class FreqtradeBot(LoggingMixin):
        if not stake_amount:
            return False
-        if pos_adjust:
+        msg = (f"Position adjust: about to create a new order for {pair} with stake: "
-            logger.info(f"Position adjust: about to create a new order for {pair} with stake: "
+               f"{stake_amount} for {trade}" if pos_adjust
-                        f"{stake_amount} for {trade}")
+               else
        else:
            logger.info(
               f"{name} signal found: about create a new trade for {pair} with stake_amount: "
               f"{stake_amount} ...")
-
+        logger.info(msg)
        amount = (stake_amount / enter_limit_requested) * leverage
        order_type = ordertype or self.strategy.order_types['entry']
@ -726,10 +737,16 @@ class FreqtradeBot(LoggingMixin):
        fee = self.exchange.get_fee(symbol=pair, taker_or_maker='maker')
        base_currency = self.exchange.get_pair_base_currency(pair)
        open_date = datetime.now(timezone.utc)
-        funding_fees = self.exchange.get_funding_fees(
+
            pair=pair, amount=amount, is_short=is_short, open_date=open_date)
        # This is a new trade
        if trade is None:
            funding_fees = 0.0
            try:
                funding_fees = self.exchange.get_funding_fees(
                    pair=pair, amount=amount, is_short=is_short, open_date=open_date)
            except ExchangeError:
                logger.warning("Could not find funding fee.")
            trade = Trade(
                pair=pair,
                base_currency=base_currency,
@ -906,7 +923,7 @@ class FreqtradeBot(LoggingMixin):
            'stake_amount': trade.stake_amount,
            'stake_currency': self.config['stake_currency'],
            'fiat_currency': self.config.get('fiat_display_currency', None),
-            'amount': order.safe_amount_after_fee,
+            'amount': order.safe_amount_after_fee if fill else order.amount,
            'open_date': trade.open_date or datetime.utcnow(),
            'current_rate': current_rate,
            'sub_trade': sub_trade,
@ -1480,12 +1497,16 @@ class FreqtradeBot(LoggingMixin):
        :param exit_check: CheckTuple with signal and reason
        :return: True if it succeeds False
        """
        try:
            trade.funding_fees = self.exchange.get_funding_fees(
                pair=trade.pair,
                amount=trade.amount,
                is_short=trade.is_short,
-            open_date=trade.open_date_utc,
+                open_date=trade.date_last_filled_utc,
            )
        except ExchangeError:
            logger.warning("Could not update funding fee.")
        exit_type = 'exit'
        exit_reason = exit_tag or exit_check.exit_reason
        if exit_check.exit_type in (
--- a/freqtrade/optimize/backtesting.py
+++ b/freqtrade/optimize/backtesting.py
@ -537,7 +537,11 @@ class Backtesting:
                    return pos_trade
        if stake_amount is not None and stake_amount < 0.0:
-            amount = abs(stake_amount) / current_rate
+            amount = amount_to_contract_precision(
                abs(stake_amount) / current_rate, trade.amount_precision,
                self.precision_mode, trade.contract_size)
            if amount == 0.0:
                return trade
            if amount > trade.amount:
                # This is currently ineffective as remaining would become < min tradable
                amount = trade.amount
@ -686,7 +690,7 @@ class Backtesting:
                self.futures_data[trade.pair],
                amount=trade.amount,
                is_short=trade.is_short,
-                open_date=trade.open_date_utc,
+                open_date=trade.date_last_filled_utc,
                close_date=exit_candle_time,
            )
--- a/freqtrade/optimize/hyperopt.py
+++ b/freqtrade/optimize/hyperopt.py
@ -421,9 +421,10 @@ class Hyperopt:
        preprocessed = self.backtesting.strategy.advise_all_indicators(data)
        # Trim startup period from analyzed dataframe to get correct dates for output.
-        processed = trim_dataframes(preprocessed, self.timerange, self.backtesting.required_startup)
+        trimmed = trim_dataframes(preprocessed, self.timerange, self.backtesting.required_startup)
-        self.min_date, self.max_date = get_timerange(processed)
+        self.min_date, self.max_date = get_timerange(trimmed)
-        return processed
+        # Real trimming will happen as part of backtesting.
        return preprocessed
    def prepare_hyperopt_data(self) -> None:
        HyperoptStateContainer.set_state(HyperoptState.DATALOAD)
--- a/freqtrade/optimize/optimize_reports.py
+++ b/freqtrade/optimize/optimize_reports.py
@ -75,7 +75,8 @@ def _get_line_floatfmt(stake_currency: str) -> List[str]:
            '.2f', 'd', 's', 's']
-def _get_line_header(first_column: str, stake_currency: str, direction: str = 'Buys') -> List[str]:
+def _get_line_header(first_column: str, stake_currency: str,
                     direction: str = 'Entries') -> List[str]:
    """
    Generate header lines (goes in line with _generate_result_line())
    """
@ -642,7 +643,7 @@ def text_table_tags(tag_type: str, tag_results: List[Dict[str, Any]], stake_curr
    if (tag_type == "enter_tag"):
        headers = _get_line_header("TAG", stake_currency)
    else:
-        headers = _get_line_header("TAG", stake_currency, 'Sells')
+        headers = _get_line_header("TAG", stake_currency, 'Exits')
    floatfmt = _get_line_floatfmt(stake_currency)
    output = [
        [
--- a/freqtrade/persistence/migrations.py
+++ b/freqtrade/persistence/migrations.py
@ -212,17 +212,18 @@ def migrate_orders_table(engine, table_back_name: str, cols_order: List):
    ft_fee_base = get_column_def(cols_order, 'ft_fee_base', 'null')
    average = get_column_def(cols_order, 'average', 'null')
    stop_price = get_column_def(cols_order, 'stop_price', 'null')
    funding_fee = get_column_def(cols_order, 'funding_fee', '0.0')
    # sqlite does not support literals for booleans
    with engine.begin() as connection:
        connection.execute(text(f"""
            insert into orders (id, ft_trade_id, ft_order_side, ft_pair, ft_is_open, order_id,
            status, symbol, order_type, side, price, amount, filled, average, remaining, cost,
-            stop_price, order_date, order_filled_date, order_update_date, ft_fee_base)
+            stop_price, order_date, order_filled_date, order_update_date, ft_fee_base, funding_fee)
            select id, ft_trade_id, ft_order_side, ft_pair, ft_is_open, order_id,
            status, symbol, order_type, side, price, amount, filled, {average} average, remaining,
            cost, {stop_price} stop_price, order_date, order_filled_date,
-            order_update_date, {ft_fee_base} ft_fee_base
+            order_update_date, {ft_fee_base} ft_fee_base, {funding_fee} funding_fee
            from {table_back_name}
            """))
@ -307,9 +308,10 @@ def check_migrate(engine, decl_base, previous_tables) -> None:
    # Check if migration necessary
    # Migrates both trades and orders table!
    # if ('orders' not in previous_tables
-    # or not has_column(cols_orders, 'stop_price')):
+    # or not has_column(cols_orders, 'funding_fee')):
    migrating = False
-    if not has_column(cols_trades, 'contract_size'):
+    # if not has_column(cols_trades, 'contract_size'):
    if not has_column(cols_orders, 'funding_fee'):
        migrating = True
        logger.info(f"Running database migration for trades - "
                    f"backup: {table_back_name}, {order_table_bak_name}")
--- a/freqtrade/persistence/trade_model.py
+++ b/freqtrade/persistence/trade_model.py
@ -65,6 +65,8 @@ class Order(_DECL_BASE):
    order_filled_date = Column(DateTime, nullable=True)
    order_update_date = Column(DateTime, nullable=True)
    funding_fee = Column(Float, nullable=True)
    ft_fee_base = Column(Float, nullable=True)
    @property
@ -72,6 +74,13 @@ class Order(_DECL_BASE):
        """ Order-date with UTC timezoneinfo"""
        return self.order_date.replace(tzinfo=timezone.utc)
    @property
    def order_filled_utc(self) -> Optional[datetime]:
        """ last order-date with UTC timezoneinfo"""
        return (
            self.order_filled_date.replace(tzinfo=timezone.utc) if self.order_filled_date else None
        )
    @property
    def safe_price(self) -> float:
        return self.average or self.price
@ -119,6 +128,10 @@ class Order(_DECL_BASE):
        self.ft_is_open = True
        if self.status in NON_OPEN_EXCHANGE_STATES:
            self.ft_is_open = False
            if self.trade:
                # Assign funding fee up to this point
                # (represents the funding fee since the last order)
                self.funding_fee = self.trade.funding_fees
            if (order.get('filled', 0.0) or 0.0) > 0:
                self.order_filled_date = datetime.now(timezone.utc)
        self.order_update_date = datetime.now(timezone.utc)
@ -179,6 +192,10 @@ class Order(_DECL_BASE):
        self.remaining = 0
        self.status = 'closed'
        self.ft_is_open = False
        # Assign funding fees to Order.
        # Assumes backtesting will use date_last_filled_utc to calculate future funding fees.
        self.funding_fee = trade.funding_fees
        if (self.ft_order_side == trade.entry_side):
            trade.open_rate = self.price
            trade.recalc_trade_from_orders()
@ -346,6 +363,15 @@ class LocalTrade():
        else:
            return self.amount
    @property
    def date_last_filled_utc(self) -> datetime:
        """ Date of the last filled order"""
        orders = self.select_filled_orders()
        if not orders:
            return self.open_date_utc
        return max([self.open_date_utc,
                    max(o.order_filled_utc for o in orders if o.order_filled_utc)])
    @property
    def open_date_utc(self):
        return self.open_date.replace(tzinfo=timezone.utc)
@ -843,10 +869,14 @@ class LocalTrade():
        close_profit = 0.0
        close_profit_abs = 0.0
        profit = None
-        for o in self.orders:
+        # Reset funding fees
        self.funding_fees = 0.0
        funding_fees = 0.0
        ordercount = len(self.orders) - 1
        for i, o in enumerate(self.orders):
            if o.ft_is_open or not o.filled:
                continue
-
+            funding_fees += (o.funding_fee or 0.0)
            tmp_amount = FtPrecise(o.safe_amount_after_fee)
            tmp_price = FtPrecise(o.safe_price)
@ -861,7 +891,11 @@ class LocalTrade():
                    avg_price = current_stake / current_amount
            if is_exit:
-                # Process partial exits
+                # Process exits
                if i == ordercount and is_closing:
                    # Apply funding fees only to the last closing order
                    self.funding_fees = funding_fees
                exit_rate = o.safe_price
                exit_amount = o.safe_amount_after_fee
                profit = self.calc_profit(rate=exit_rate, amount=exit_amount,
@ -871,6 +905,7 @@ class LocalTrade():
                    exit_rate, amount=exit_amount, open_rate=avg_price)
            else:
                total_stake = total_stake + self._calc_open_trade_value(tmp_amount, price)
        self.funding_fees = funding_fees
        if close_profit:
            self.close_profit = close_profit
--- a/freqtrade/rpc/discord.py
+++ b/freqtrade/rpc/discord.py
@ -1,7 +1,7 @@
 import logging
 from typing import Any, Dict
-from freqtrade.enums.rpcmessagetype import RPCMessageType
+from freqtrade.enums import RPCMessageType
 from freqtrade.rpc import RPC
 from freqtrade.rpc.webhook import Webhook
--- a/freqtrade/rpc/rpc.py
+++ b/freqtrade/rpc/rpc.py
@ -261,11 +261,15 @@ class RPC:
                        profit_str += f" ({fiat_profit:.2f})"
                        fiat_profit_sum = fiat_profit if isnan(fiat_profit_sum) \
                            else fiat_profit_sum + fiat_profit
                open_order = (trade.select_order_by_order_id(
                    trade.open_order_id) if trade.open_order_id else None)
                detail_trade = [
                    f'{trade.id} {direction_str}',
-                    trade.pair + ('*' if (trade.open_order_id is not None
+                    trade.pair + ('*' if (open_order
-                                          and trade.close_rate_requested is None) else '')
+                                  and open_order.ft_order_side == trade.entry_side) else '')
-                    + ('**' if (trade.close_rate_requested is not None) else ''),
+                    + ('**' if (open_order and
                                open_order.ft_order_side == trade.exit_side is not None) else ''),
                    shorten_date(arrow.get(trade.open_date).humanize(only_distance=True)),
                    profit_str
                ]
--- a/freqtrade/rpc/telegram.py
+++ b/freqtrade/rpc/telegram.py
@ -6,6 +6,7 @@ This module manage Telegram communication
 import json
 import logging
 import re
 from copy import deepcopy
 from dataclasses import dataclass
 from datetime import date, datetime, timedelta
 from functools import partial
@ -374,7 +375,7 @@ class Telegram(RPCHandler):
            message += f"\n*Duration:* `{msg['duration']} ({msg['duration_min']:.1f} min)`"
        return message
-    def compose_message(self, msg: Dict[str, Any], msg_type: RPCMessageType) -> str:
+    def compose_message(self, msg: Dict[str, Any], msg_type: RPCMessageType) -> Optional[str]:
        if msg_type in [RPCMessageType.ENTRY, RPCMessageType.ENTRY_FILL]:
            message = self._format_entry_msg(msg)
@ -411,7 +412,8 @@ class Telegram(RPCHandler):
        elif msg_type == RPCMessageType.STRATEGY_MSG:
            message = f"{msg['msg']}"
        else:
-            raise NotImplementedError(f"Unknown message type: {msg_type}")
+            logger.debug("Unknown message type: %s", msg_type)
            return None
        return message
    def send_msg(self, msg: Dict[str, Any]) -> None:
@ -438,8 +440,8 @@ class Telegram(RPCHandler):
            # Notification disabled
            return
-        message = self.compose_message(msg, msg_type)
+        message = self.compose_message(deepcopy(msg), msg_type)
-
+        if message:
            self._send_msg(message, disable_notification=(noti == 'silent'))
    def _get_sell_emoji(self, msg):
--- a/freqtrade/strategy/interface.py
+++ b/freqtrade/strategy/interface.py
@ -12,9 +12,8 @@ from pandas import DataFrame
 from freqtrade.constants import ListPairsWithTimeframes
 from freqtrade.data.dataprovider import DataProvider
-from freqtrade.enums import (CandleType, ExitCheckTuple, ExitType, SignalDirection, SignalTagType,
+from freqtrade.enums import (CandleType, ExitCheckTuple, ExitType, RunMode, SignalDirection,
-                             SignalType, TradingMode)
+                             SignalTagType, SignalType, TradingMode)
 from freqtrade.enums.runmode import RunMode
 from freqtrade.exceptions import OperationalException, StrategyError
 from freqtrade.exchange import timeframe_to_minutes, timeframe_to_next_date, timeframe_to_seconds
 from freqtrade.persistence import Order, PairLocks, Trade
@ -169,6 +168,10 @@ class IStrategy(ABC, HyperStrategyMixin):
                    raise OperationalException(
                        'freqAI is not enabled. '
                        'Please enable it in your config to use this strategy.')
                def shutdown(self, *args, **kwargs):
                    pass
            self.freqai = DummyClass()  # type: ignore
    def ft_bot_start(self, **kwargs) -> None:
@ -182,6 +185,12 @@ class IStrategy(ABC, HyperStrategyMixin):
        self.ft_load_hyper_params(self.config.get('runmode') == RunMode.HYPEROPT)
    def ft_bot_cleanup(self) -> None:
        """
        Clean up FreqAI and child threads
        """
        self.freqai.shutdown()
    @abstractmethod
    def populate_indicators(self, dataframe: DataFrame, metadata: dict) -> DataFrame:
        """
--- a/freqtrade/strategy/parameters.py
+++ b/freqtrade/strategy/parameters.py
@ -7,7 +7,7 @@ from abc import ABC, abstractmethod
 from contextlib import suppress
 from typing import Any, Optional, Sequence, Union
-from freqtrade.enums.hyperoptstate import HyperoptState
+from freqtrade.enums import HyperoptState
 from freqtrade.optimize.hyperopt_tools import HyperoptStateContainer
--- a/freqtrade/strategy/strategy_helper.py
+++ b/freqtrade/strategy/strategy_helper.py
@ -1,3 +1,5 @@
 from typing import Optional
 import pandas as pd
 from freqtrade.exchange import timeframe_to_minutes
@ -6,7 +8,8 @@ from freqtrade.exchange import timeframe_to_minutes
 def merge_informative_pair(dataframe: pd.DataFrame, informative: pd.DataFrame,
                           timeframe: str, timeframe_inf: str, ffill: bool = True,
                           append_timeframe: bool = True,
-                           date_column: str = 'date') -> pd.DataFrame:
+                           date_column: str = 'date',
                           suffix: Optional[str] = None) -> pd.DataFrame:
    """
    Correctly merge informative samples to the original dataframe, avoiding lookahead bias.
@ -28,6 +31,8 @@ def merge_informative_pair(dataframe: pd.DataFrame, informative: pd.DataFrame,
    :param ffill: Forwardfill missing values - optional but usually required
    :param append_timeframe: Rename columns by appending timeframe.
    :param date_column: A custom date column name.
    :param suffix: A string suffix to add at the end of the informative columns. If specified,
                   append_timeframe must be false.
    :return: Merged dataframe
    :raise: ValueError if the secondary timeframe is shorter than the dataframe timeframe
    """
@ -50,10 +55,16 @@ def merge_informative_pair(dataframe: pd.DataFrame, informative: pd.DataFrame,
    # Rename columns to be unique
    date_merge = 'date_merge'
-    if append_timeframe:
+    if suffix and append_timeframe:
        raise ValueError("You can not specify `append_timeframe` as True and a `suffix`.")
    elif append_timeframe:
        date_merge = f'date_merge_{timeframe_inf}'
        informative.columns = [f"{col}_{timeframe_inf}" for col in informative.columns]
    elif suffix:
        date_merge = f'date_merge_{suffix}'
        informative.columns = [f"{col}_{suffix}" for col in informative.columns]
    # Combine the 2 dataframes
    # all indicators on the informative sample MUST be calculated before this point
    if ffill:
--- a/freqtrade/templates/FreqaiExampleStrategy.py
+++ b/freqtrade/templates/FreqaiExampleStrategy.py
@ -6,9 +6,7 @@ import talib.abstract as ta
 from pandas import DataFrame
 from technical import qtpylib
-from freqtrade.exchange import timeframe_to_prev_date
+from freqtrade.strategy import CategoricalParameter, IStrategy, merge_informative_pair
 from freqtrade.persistence import Trade
 from freqtrade.strategy import DecimalParameter, IntParameter, IStrategy, merge_informative_pair
 logger = logging.getLogger(__name__)
@ -31,9 +29,6 @@ class FreqaiExampleStrategy(IStrategy):
        "main_plot": {},
        "subplots": {
            "prediction": {"prediction": {"color": "blue"}},
            "target_roi": {
                "target_roi": {"color": "brown"},
            },
            "do_predict": {
                "do_predict": {"color": "brown"},
            },
@ -47,10 +42,10 @@ class FreqaiExampleStrategy(IStrategy):
    startup_candle_count: int = 40
    can_short = False
-    linear_roi_offset = DecimalParameter(
+    std_dev_multiplier_buy = CategoricalParameter(
-        0.00, 0.02, default=0.005, space="sell", optimize=False, load=True
+        [0.75, 1, 1.25, 1.5, 1.75], default=1.25, space="buy", optimize=True)
-    )
+    std_dev_multiplier_sell = CategoricalParameter(
-    max_roi_time_long = IntParameter(0, 800, default=400, space="sell", optimize=False, load=True)
+        [0.1, 0.25, 0.4], space="sell", default=0.2, optimize=True)
    def informative_pairs(self):
        whitelist_pairs = self.dp.current_whitelist()
@ -92,12 +87,10 @@ class FreqaiExampleStrategy(IStrategy):
            t = int(t)
            informative[f"%-{coin}rsi-period_{t}"] = ta.RSI(informative, timeperiod=t)
            informative[f"%-{coin}mfi-period_{t}"] = ta.MFI(informative, timeperiod=t)
-            informative[f"%-{coin}adx-period_{t}"] = ta.ADX(informative, window=t)
+            informative[f"%-{coin}adx-period_{t}"] = ta.ADX(informative, timeperiod=t)
            informative[f"%-{coin}sma-period_{t}"] = ta.SMA(informative, timeperiod=t)
            informative[f"%-{coin}ema-period_{t}"] = ta.EMA(informative, timeperiod=t)
            informative[f"%-{coin}mfi-period_{t}"] = ta.MFI(informative, timeperiod=t)
            bollinger = qtpylib.bollinger_bands(
                qtpylib.typical_price(informative), window=t, stds=2.2
            )
@ -189,21 +182,26 @@ class FreqaiExampleStrategy(IStrategy):
        # `populate_any_indicators()` for each training period.
        dataframe = self.freqai.start(dataframe, metadata, self)
-
+        for val in self.std_dev_multiplier_buy.range:
-        dataframe["target_roi"] = dataframe["&-s_close_mean"] + dataframe["&-s_close_std"] * 1.25
+            dataframe[f'target_roi_{val}'] = dataframe["&-s_close_mean"] + \
-        dataframe["sell_roi"] = dataframe["&-s_close_mean"] - dataframe["&-s_close_std"] * 1.25
+                dataframe["&-s_close_std"] * val
        for val in self.std_dev_multiplier_sell.range:
            dataframe[f'sell_roi_{val}'] = dataframe["&-s_close_mean"] - \
                dataframe["&-s_close_std"] * val
        return dataframe
    def populate_entry_trend(self, df: DataFrame, metadata: dict) -> DataFrame:
-        enter_long_conditions = [df["do_predict"] == 1, df["&-s_close"] > df["target_roi"]]
+        enter_long_conditions = [df["do_predict"] == 1, df["&-s_close"]
                                 > df[f"target_roi_{self.std_dev_multiplier_buy.value}"]]
        if enter_long_conditions:
            df.loc[
                reduce(lambda x, y: x & y, enter_long_conditions), ["enter_long", "enter_tag"]
            ] = (1, "long")
-        enter_short_conditions = [df["do_predict"] == 1, df["&-s_close"] < df["sell_roi"]]
+        enter_short_conditions = [df["do_predict"] == 1, df["&-s_close"]
                                  < df[f"sell_roi_{self.std_dev_multiplier_sell.value}"]]
        if enter_short_conditions:
            df.loc[
@ -213,11 +211,13 @@ class FreqaiExampleStrategy(IStrategy):
        return df
    def populate_exit_trend(self, df: DataFrame, metadata: dict) -> DataFrame:
-        exit_long_conditions = [df["do_predict"] == 1, df["&-s_close"] < df["sell_roi"] * 0.25]
+        exit_long_conditions = [df["do_predict"] == 1, df["&-s_close"] <
                                df[f"sell_roi_{self.std_dev_multiplier_sell.value}"] * 0.25]
        if exit_long_conditions:
            df.loc[reduce(lambda x, y: x & y, exit_long_conditions), "exit_long"] = 1
-        exit_short_conditions = [df["do_predict"] == 1, df["&-s_close"] > df["target_roi"] * 0.25]
+        exit_short_conditions = [df["do_predict"] == 1, df["&-s_close"] >
                                 df[f"target_roi_{self.std_dev_multiplier_buy.value}"] * 0.25]
        if exit_short_conditions:
            df.loc[reduce(lambda x, y: x & y, exit_short_conditions), "exit_short"] = 1
@ -226,83 +226,6 @@ class FreqaiExampleStrategy(IStrategy):
    def get_ticker_indicator(self):
        return int(self.config["timeframe"][:-1])
    def custom_exit(
        self, pair: str, trade: Trade, current_time, current_rate, current_profit, **kwargs
    ):
        dataframe, _ = self.dp.get_analyzed_dataframe(pair=pair, timeframe=self.timeframe)
        trade_date = timeframe_to_prev_date(self.config["timeframe"], trade.open_date_utc)
        trade_candle = dataframe.loc[(dataframe["date"] == trade_date)]
        if trade_candle.empty:
            return None
        trade_candle = trade_candle.squeeze()
        follow_mode = self.config.get("freqai", {}).get("follow_mode", False)
        if not follow_mode:
            pair_dict = self.freqai.dd.pair_dict
        else:
            pair_dict = self.freqai.dd.follower_dict
        entry_tag = trade.enter_tag
        if (
            "prediction" + entry_tag not in pair_dict[pair]
            or pair_dict[pair]['extras']["prediction" + entry_tag] == 0
        ):
            pair_dict[pair]['extras']["prediction" + entry_tag] = abs(trade_candle["&-s_close"])
            if not follow_mode:
                self.freqai.dd.save_drawer_to_disk()
            else:
                self.freqai.dd.save_follower_dict_to_disk()
        roi_price = pair_dict[pair]['extras']["prediction" + entry_tag]
        roi_time = self.max_roi_time_long.value
        roi_decay = roi_price * (
            1 - ((current_time - trade.open_date_utc).seconds) / (roi_time * 60)
        )
        if roi_decay < 0:
            roi_decay = self.linear_roi_offset.value
        else:
            roi_decay += self.linear_roi_offset.value
        if current_profit > roi_decay:
            return "roi_custom_win"
        if current_profit < -roi_decay:
            return "roi_custom_loss"
    def confirm_trade_exit(
        self,
        pair: str,
        trade: Trade,
        order_type: str,
        amount: float,
        rate: float,
        time_in_force: str,
        exit_reason: str,
        current_time,
        **kwargs,
    ) -> bool:
        entry_tag = trade.enter_tag
        follow_mode = self.config.get("freqai", {}).get("follow_mode", False)
        if not follow_mode:
            pair_dict = self.freqai.dd.pair_dict
        else:
            pair_dict = self.freqai.dd.follower_dict
        pair_dict[pair]['extras']["prediction" + entry_tag] = 0
        if not follow_mode:
            self.freqai.dd.save_drawer_to_disk()
        else:
            self.freqai.dd.save_follower_dict_to_disk()
        return True
    def confirm_trade_entry(
        self,
        pair: str,
--- a/freqtrade/templates/FreqaiHybridExampleStrategy.py
+++ b/freqtrade/templates/FreqaiHybridExampleStrategy.py
@ -135,7 +135,7 @@ class FreqaiExampleHybridStrategy(IStrategy):
            t = int(t)
            informative[f"%-{coin}rsi-period_{t}"] = ta.RSI(informative, timeperiod=t)
            informative[f"%-{coin}mfi-period_{t}"] = ta.MFI(informative, timeperiod=t)
-            informative[f"%-{coin}adx-period_{t}"] = ta.ADX(informative, window=t)
+            informative[f"%-{coin}adx-period_{t}"] = ta.ADX(informative, timeperiod=t)
            informative[f"%-{coin}sma-period_{t}"] = ta.SMA(informative, timeperiod=t)
            informative[f"%-{coin}ema-period_{t}"] = ta.EMA(informative, timeperiod=t)
            informative[f"%-{coin}roc-period_{t}"] = ta.ROC(informative, timeperiod=t)
--- a/requirements-dev.txt
+++ b/requirements-dev.txt
@ -10,7 +10,7 @@ flake8==5.0.4
 flake8-tidy-imports==4.8.0
 mypy==0.971
 pre-commit==2.20.0
-pytest==7.1.2
+pytest==7.1.3
 pytest-asyncio==0.19.0
 pytest-cov==3.0.0
 pytest-mock==3.8.2
--- a/requirements-freqai.txt
+++ b/requirements-freqai.txt
@ -6,6 +6,7 @@ scikit-learn==1.1.2
 joblib==1.1.0
 catboost==1.0.6; platform_machine != 'aarch64'
 lightgbm==3.3.2
 xgboost==1.6.2
 torch==1.12.1
 stable-baselines3==1.6.0
 gym==0.21.0
--- a/requirements.txt
+++ b/requirements.txt
@ -1,22 +1,22 @@
 numpy==1.23.2
-pandas==1.4.3
+pandas==1.4.4
 pandas-ta==0.3.14b
-ccxt==1.92.84
+ccxt==1.93.3
 # Pin cryptography for now due to rust build errors with piwheels
 cryptography==37.0.4
 aiohttp==3.8.1
 SQLAlchemy==1.4.40
-python-telegram-bot==13.13
+python-telegram-bot==13.14
-arrow==1.2.2
+arrow==1.2.3
 cachetools==4.2.2
 requests==2.28.1
 urllib3==1.26.12
-jsonschema==4.14.0
+jsonschema==4.15.0
 TA-Lib==0.4.24
 technical==1.3.0
 tabulate==0.8.10
-pycoingecko==2.2.0
+pycoingecko==3.0.0
 jinja2==3.1.2
 tables==3.7.0
 blosc==1.10.6
@ -34,17 +34,17 @@ orjson==3.8.0
 sdnotify==0.3.2
 # API Server
-fastapi==0.81.0
+fastapi==0.82.0
 uvicorn==0.18.3
 pyjwt==2.4.0
 aiofiles==0.8.0
-psutil==5.9.1
+psutil==5.9.2
 # Support for colorized terminal output
 colorama==0.4.5
 # Building config files interactively
 questionary==1.10.0
-prompt-toolkit==3.0.30
+prompt-toolkit==3.0.31
 # Extensions to datetime library
 python-dateutil==2.8.2
--- a/tests/data/test_history.py
+++ b/tests/data/test_history.py
@ -13,7 +13,7 @@ from pandas import DataFrame
 from pandas.testing import assert_frame_equal
 from freqtrade.configuration import TimeRange
-from freqtrade.constants import AVAILABLE_DATAHANDLERS
+from freqtrade.constants import AVAILABLE_DATAHANDLERS, DATETIME_PRINT_FORMAT
 from freqtrade.data.converter import ohlcv_to_dataframe
 from freqtrade.data.history.hdf5datahandler import HDF5DataHandler
 from freqtrade.data.history.history_utils import (_download_pair_history, _download_trades_history,
@ -386,7 +386,7 @@ def test_load_partial_missing(testdatadir, caplog) -> None:
    assert td != len(data['UNITTEST/BTC'])
    start_real = data['UNITTEST/BTC'].iloc[0, 0]
    assert log_has(f'UNITTEST/BTC, spot, 5m, '
-                   f'data starts at {start_real.strftime("%Y-%m-%d %H:%M:%S")}',
+                   f'data starts at {start_real.strftime(DATETIME_PRINT_FORMAT)}',
                   caplog)
    # Make sure we start fresh - test missing data at end
    caplog.clear()
@ -401,7 +401,7 @@ def test_load_partial_missing(testdatadir, caplog) -> None:
    # Shift endtime with +5 - as last candle is dropped (partial candle)
    end_real = arrow.get(data['UNITTEST/BTC'].iloc[-1, 0]).shift(minutes=5)
    assert log_has(f'UNITTEST/BTC, spot, 5m, '
-                   f'data ends at {end_real.strftime("%Y-%m-%d %H:%M:%S")}',
+                   f'data ends at {end_real.strftime(DATETIME_PRINT_FORMAT)}',
                   caplog)
--- a/tests/exchange/test_ccxt_compat.py
+++ b/tests/exchange/test_ccxt_compat.py
@ -267,13 +267,8 @@ class TestCCXTExchange():
        now = datetime.now(timezone.utc) - timedelta(minutes=(timeframe_to_minutes(timeframe) * 2))
        assert exchange.klines(pair_tf).iloc[-1]['date'] >= timeframe_to_prev_date(timeframe, now)
-    def test_ccxt__async_get_candle_history(self, exchange):
+    def ccxt__async_get_candle_history(self, exchange, exchangename, pair, timeframe):
-        exchange, exchangename = exchange
+
        # For some weired reason, this test returns random lengths for bittrex.
        if not exchange._ft_has['ohlcv_has_history'] or exchangename == 'bittrex':
            return
        pair = EXCHANGES[exchangename]['pair']
        timeframe = EXCHANGES[exchangename]['timeframe']
        candle_type = CandleType.SPOT
        timeframe_ms = timeframe_to_msecs(timeframe)
        now = timeframe_to_prev_date(
@ -299,6 +294,24 @@ class TestCCXTExchange():
            assert len(candles) >= min(candle_count, candle_count1)
            assert candles[0][0] == since_ms or (since_ms + timeframe_ms)
    def test_ccxt__async_get_candle_history(self, exchange):
        exchange, exchangename = exchange
        # For some weired reason, this test returns random lengths for bittrex.
        if not exchange._ft_has['ohlcv_has_history'] or exchangename in ('bittrex', 'gateio'):
            return
        pair = EXCHANGES[exchangename]['pair']
        timeframe = EXCHANGES[exchangename]['timeframe']
        self.ccxt__async_get_candle_history(exchange, exchangename, pair, timeframe)
    def test_ccxt__async_get_candle_history_futures(self, exchange_futures):
        exchange, exchangename = exchange_futures
        if not exchange:
            # exchange_futures only returns values for supported exchanges
            return
        pair = EXCHANGES[exchangename].get('futures_pair', EXCHANGES[exchangename]['pair'])
        timeframe = EXCHANGES[exchangename]['timeframe']
        self.ccxt__async_get_candle_history(exchange, exchangename, pair, timeframe)
    def test_ccxt_fetch_funding_rate_history(self, exchange_futures):
        exchange, exchangename = exchange_futures
        if not exchange:
--- a/tests/exchange/test_exchange.py
+++ b/tests/exchange/test_exchange.py
@ -11,8 +11,9 @@ import pytest
 from pandas import DataFrame
 from freqtrade.enums import CandleType, MarginMode, TradingMode
-from freqtrade.exceptions import (DDosProtection, DependencyException, InvalidOrderException,
+from freqtrade.exceptions import (DDosProtection, DependencyException, ExchangeError,
-                                  OperationalException, PricingError, TemporaryError)
+                                  InvalidOrderException, OperationalException, PricingError,
                                  TemporaryError)
 from freqtrade.exchange import (Binance, Bittrex, Exchange, Kraken, amount_to_precision,
                                date_minus_candles, market_is_active, price_to_precision,
                                timeframe_to_minutes, timeframe_to_msecs, timeframe_to_next_date,
@ -4179,17 +4180,24 @@ def test__fetch_and_calculate_funding_fees(
    type(api_mock).has = PropertyMock(return_value={'fetchOHLCV': True})
    type(api_mock).has = PropertyMock(return_value={'fetchFundingRateHistory': True})
-    exchange = get_patched_exchange(mocker, default_conf, api_mock, id=exchange)
+    ex = get_patched_exchange(mocker, default_conf, api_mock, id=exchange)
    mocker.patch('freqtrade.exchange.Exchange.timeframes', PropertyMock(
        return_value=['1h', '4h', '8h']))
-    funding_fees = exchange._fetch_and_calculate_funding_fees(
+    funding_fees = ex._fetch_and_calculate_funding_fees(
        pair='ADA/USDT', amount=amount, is_short=True, open_date=d1, close_date=d2)
    assert pytest.approx(funding_fees) == expected_fees
    # Fees for Longs are inverted
-    funding_fees = exchange._fetch_and_calculate_funding_fees(
+    funding_fees = ex._fetch_and_calculate_funding_fees(
        pair='ADA/USDT', amount=amount, is_short=False, open_date=d1, close_date=d2)
    assert pytest.approx(funding_fees) == -expected_fees
    # Return empty "refresh_latest"
    mocker.patch("freqtrade.exchange.Exchange.refresh_latest_ohlcv", return_value={})
    ex = get_patched_exchange(mocker, default_conf, api_mock, id=exchange)
    with pytest.raises(ExchangeError, match="Could not find funding rates."):
        ex._fetch_and_calculate_funding_fees(
            pair='ADA/USDT', amount=amount, is_short=False, open_date=d1, close_date=d2)
@pytest.mark.parametrize('exchange,expected_fees', [
    ('binance', -0.0009140999999999999),
@ -4456,6 +4464,39 @@ def test__amount_to_contracts(
    assert result_amount == param_amount
@pytest.mark.parametrize('pair,amount,expected_spot,expected_fut', [
    # Contract size of 0.01
    ('ADA/USDT:USDT', 40, 40, 40),
    ('ADA/USDT:USDT', 10.4445555, 10.4, 10.444),
    ('LTC/ETH', 30, 30, 30),
    ('LTC/USD', 30, 30, 30),
    # contract size of 10
    ('ETH/USDT:USDT', 10.111, 10.1, 10),
    ('ETH/USDT:USDT', 10.188, 10.1, 10),
    ('ETH/USDT:USDT', 10.988, 10.9, 10),
 ])
 def test_amount_to_contract_precision(
    mocker,
    default_conf,
    pair,
    amount,
    expected_spot,
    expected_fut,
 ):
    api_mock = MagicMock()
    default_conf['trading_mode'] = 'spot'
    default_conf['margin_mode'] = 'isolated'
    exchange = get_patched_exchange(mocker, default_conf, api_mock)
    result_size = exchange.amount_to_contract_precision(pair, amount)
    assert result_size == expected_spot
    default_conf['trading_mode'] = 'futures'
    exchange = get_patched_exchange(mocker, default_conf, api_mock)
    result_size = exchange.amount_to_contract_precision(pair, amount)
    assert result_size == expected_fut
@pytest.mark.parametrize('exchange_name,open_rate,is_short,trading_mode,margin_mode', [
    # Bittrex
    ('bittrex', 2.0, False, 'spot', None),
--- a/tests/exchange/test_okx.py
+++ b/tests/exchange/test_okx.py
@ -4,8 +4,7 @@ from unittest.mock import MagicMock, PropertyMock
 import pytest
-from freqtrade.enums import MarginMode, TradingMode
+from freqtrade.enums import CandleType, MarginMode, TradingMode
 from freqtrade.enums.candletype import CandleType
 from freqtrade.exchange.exchange import timeframe_to_minutes
 from tests.conftest import get_mock_coro, get_patched_exchange, log_has
 from tests.exchange.test_exchange import ccxt_exceptionhandlers
--- a/tests/freqai/conftest.py
+++ b/tests/freqai/conftest.py
@ -81,6 +81,37 @@ def get_patched_freqaimodel(mocker, freqaiconf):
    return freqaimodel
 def make_unfiltered_dataframe(mocker, freqai_conf):
    freqai_conf.update({"timerange": "20180110-20180130"})
    strategy = get_patched_freqai_strategy(mocker, freqai_conf)
    exchange = get_patched_exchange(mocker, freqai_conf)
    strategy.dp = DataProvider(freqai_conf, exchange)
    strategy.freqai_info = freqai_conf.get("freqai", {})
    freqai = strategy.freqai
    freqai.live = True
    freqai.dk = FreqaiDataKitchen(freqai_conf)
    freqai.dk.pair = "ADA/BTC"
    data_load_timerange = TimeRange.parse_timerange("20180110-20180130")
    freqai.dd.load_all_pair_histories(data_load_timerange, freqai.dk)
    freqai.dd.pair_dict = MagicMock()
    new_timerange = TimeRange.parse_timerange("20180120-20180130")
    corr_dataframes, base_dataframes = freqai.dd.get_base_and_corr_dataframes(
            data_load_timerange, freqai.dk.pair, freqai.dk
        )
    unfiltered_dataframe = freqai.dk.use_strategy_to_populate_indicators(
                strategy, corr_dataframes, base_dataframes, freqai.dk.pair
            )
    unfiltered_dataframe = freqai.dk.slice_dataframe(new_timerange, unfiltered_dataframe)
    return freqai, unfiltered_dataframe
 def make_data_dictionary(mocker, freqai_conf):
    freqai_conf.update({"timerange": "20180110-20180130"})
@ -92,12 +123,11 @@ def make_data_dictionary(mocker, freqai_conf):
    freqai.live = True
    freqai.dk = FreqaiDataKitchen(freqai_conf)
    freqai.dk.pair = "ADA/BTC"
-    timerange = TimeRange.parse_timerange("20180110-20180130")
+    data_load_timerange = TimeRange.parse_timerange("20180110-20180130")
-    freqai.dd.load_all_pair_histories(timerange, freqai.dk)
+    freqai.dd.load_all_pair_histories(data_load_timerange, freqai.dk)
    freqai.dd.pair_dict = MagicMock()
    data_load_timerange = TimeRange.parse_timerange("20180110-20180130")
    new_timerange = TimeRange.parse_timerange("20180120-20180130")
    corr_dataframes, base_dataframes = freqai.dd.get_base_and_corr_dataframes(
--- a/tests/freqai/test_freqai_datakitchen.py
+++ b/tests/freqai/test_freqai_datakitchen.py
@ -1,12 +1,13 @@
 import datetime
 import shutil
 from datetime import datetime, timedelta, timezone
 from pathlib import Path
 import pytest
 from freqtrade.exceptions import OperationalException
 from tests.conftest import log_has_re
-from tests.freqai.conftest import get_patched_data_kitchen, make_data_dictionary
+from tests.freqai.conftest import (get_patched_data_kitchen, make_data_dictionary,
                                   make_unfiltered_dataframe)
@pytest.mark.parametrize(
@ -56,16 +57,13 @@ def test_split_timerange(
    shutil.rmtree(Path(dk.full_path))
-@pytest.mark.parametrize(
+def test_check_if_model_expired(mocker, freqai_conf):
-    "timestamp, expected",
+
    [
        (datetime.datetime.now(tz=datetime.timezone.utc).timestamp() - 7200, True),
        (datetime.datetime.now(tz=datetime.timezone.utc).timestamp(), False),
    ],
 )
 def test_check_if_model_expired(mocker, freqai_conf, timestamp, expected):
    dk = get_patched_data_kitchen(mocker, freqai_conf)
-    assert dk.check_if_model_expired(timestamp) == expected
+    now = datetime.now(tz=timezone.utc).timestamp()
    assert dk.check_if_model_expired(now) is False
    now = (datetime.now(tz=timezone.utc) - timedelta(hours=2)).timestamp()
    assert dk.check_if_model_expired(now) is True
    shutil.rmtree(Path(dk.full_path))
@ -74,7 +72,7 @@ def test_use_DBSCAN_to_remove_outliers(mocker, freqai_conf, caplog):
    # freqai_conf['freqai']['feature_parameters'].update({"outlier_protection_percentage": 1})
    freqai.dk.use_DBSCAN_to_remove_outliers(predict=False)
    assert log_has_re(
-        "DBSCAN found eps of 2.42.",
+        "DBSCAN found eps of 2.36.",
        caplog,
    )
@ -83,7 +81,7 @@ def test_compute_distances(mocker, freqai_conf):
    freqai = make_data_dictionary(mocker, freqai_conf)
    freqai_conf['freqai']['feature_parameters'].update({"DI_threshold": 1})
    avg_mean_dist = freqai.dk.compute_distances()
-    assert round(avg_mean_dist, 2) == 2.56
+    assert round(avg_mean_dist, 2) == 2.54
 def test_use_SVM_to_remove_outliers_and_outlier_protection(mocker, freqai_conf, caplog):
@ -91,6 +89,75 @@ def test_use_SVM_to_remove_outliers_and_outlier_protection(mocker, freqai_conf,
    freqai_conf['freqai']['feature_parameters'].update({"outlier_protection_percentage": 0.1})
    freqai.dk.use_SVM_to_remove_outliers(predict=False)
    assert log_has_re(
-        "SVM detected 8.46%",
+        "SVM detected 8.09%",
        caplog,
    )
 def test_compute_inlier_metric(mocker, freqai_conf, caplog):
    freqai = make_data_dictionary(mocker, freqai_conf)
    freqai_conf['freqai']['feature_parameters'].update({"inlier_metric_window": 10})
    freqai.dk.compute_inlier_metric(set_='train')
    assert log_has_re(
        "Inlier metric computed and added to features.",
        caplog,
    )
 def test_add_noise_to_training_features(mocker, freqai_conf):
    freqai = make_data_dictionary(mocker, freqai_conf)
    freqai_conf['freqai']['feature_parameters'].update({"noise_standard_deviation": 0.1})
    freqai.dk.add_noise_to_training_features()
 def test_remove_beginning_points_from_data_dict(mocker, freqai_conf):
    freqai = make_data_dictionary(mocker, freqai_conf)
    freqai.dk.remove_beginning_points_from_data_dict(set_='train')
 def test_principal_component_analysis(mocker, freqai_conf, caplog):
    freqai = make_data_dictionary(mocker, freqai_conf)
    freqai.dk.principal_component_analysis()
    assert log_has_re(
        "reduced feature dimension by",
        caplog,
    )
 def test_normalize_data(mocker, freqai_conf):
    freqai = make_data_dictionary(mocker, freqai_conf)
    data_dict = freqai.dk.data_dictionary
    freqai.dk.normalize_data(data_dict)
    assert len(freqai.dk.data) == 56
 def test_filter_features(mocker, freqai_conf):
    freqai, unfiltered_dataframe = make_unfiltered_dataframe(mocker, freqai_conf)
    freqai.dk.find_features(unfiltered_dataframe)
    filtered_df, labels = freqai.dk.filter_features(
            unfiltered_dataframe,
            freqai.dk.training_features_list,
            freqai.dk.label_list,
            training_filter=True,
    )
    assert len(filtered_df.columns) == 26
 def test_make_train_test_datasets(mocker, freqai_conf):
    freqai, unfiltered_dataframe = make_unfiltered_dataframe(mocker, freqai_conf)
    freqai.dk.find_features(unfiltered_dataframe)
    features_filtered, labels_filtered = freqai.dk.filter_features(
            unfiltered_dataframe,
            freqai.dk.training_features_list,
            freqai.dk.label_list,
            training_filter=True,
        )
    data_dictionary = freqai.dk.make_train_test_datasets(features_filtered, labels_filtered)
    assert data_dictionary
    assert len(data_dictionary) == 7
    assert len(data_dictionary['train_features'].index) == 1916
--- a/tests/freqai/test_freqai_interface.py
+++ b/tests/freqai/test_freqai_interface.py
@ -17,8 +17,18 @@ def is_arm() -> bool:
    return "arm" in machine or "aarch64" in machine
-def test_train_model_in_series_LightGBM(mocker, freqai_conf):
+@pytest.mark.parametrize('model', [
    'LightGBMRegressor',
    'XGBoostRegressor',
    'CatboostRegressor',
    ])
 def test_extract_data_and_train_model_Regressors(mocker, freqai_conf, model):
    if is_arm() and model == 'CatboostRegressor':
        pytest.skip("CatBoost is not supported on ARM")
    freqai_conf.update({"freqaimodel": model})
    freqai_conf.update({"timerange": "20180110-20180130"})
    freqai_conf.update({"strategy": "freqai_test_strat"})
    strategy = get_patched_freqai_strategy(mocker, freqai_conf)
    exchange = get_patched_exchange(mocker, freqai_conf)
@ -35,7 +45,8 @@ def test_train_model_in_series_LightGBM(mocker, freqai_conf):
    data_load_timerange = TimeRange.parse_timerange("20180110-20180130")
    new_timerange = TimeRange.parse_timerange("20180120-20180130")
-    freqai.train_model_in_series(new_timerange, "ADA/BTC", strategy, freqai.dk, data_load_timerange)
+    freqai.extract_data_and_train_model(
        new_timerange, "ADA/BTC", strategy, freqai.dk, data_load_timerange)
    assert Path(freqai.dk.data_path / f"{freqai.dk.model_filename}_model.joblib").is_file()
    assert Path(freqai.dk.data_path / f"{freqai.dk.model_filename}_metadata.json").is_file()
@ -45,10 +56,18 @@ def test_train_model_in_series_LightGBM(mocker, freqai_conf):
    shutil.rmtree(Path(freqai.dk.full_path))
-def test_train_model_in_series_LightGBMMultiModel(mocker, freqai_conf):
+@pytest.mark.parametrize('model', [
    'LightGBMRegressorMultiTarget',
    'XGBoostRegressorMultiTarget',
    'CatboostRegressorMultiTarget',
    ])
 def test_extract_data_and_train_model_MultiTargets(mocker, freqai_conf, model):
    if is_arm() and model == 'CatboostRegressorMultiTarget':
        pytest.skip("CatBoost is not supported on ARM")
    freqai_conf.update({"timerange": "20180110-20180130"})
    freqai_conf.update({"strategy": "freqai_test_multimodel_strat"})
-    freqai_conf.update({"freqaimodel": "LightGBMRegressorMultiTarget"})
+    freqai_conf.update({"freqaimodel": model})
    strategy = get_patched_freqai_strategy(mocker, freqai_conf)
    exchange = get_patched_exchange(mocker, freqai_conf)
    strategy.dp = DataProvider(freqai_conf, exchange)
@ -64,7 +83,8 @@ def test_train_model_in_series_LightGBMMultiModel(mocker, freqai_conf):
    data_load_timerange = TimeRange.parse_timerange("20180110-20180130")
    new_timerange = TimeRange.parse_timerange("20180120-20180130")
-    freqai.train_model_in_series(new_timerange, "ADA/BTC", strategy, freqai.dk, data_load_timerange)
+    freqai.extract_data_and_train_model(
        new_timerange, "ADA/BTC", strategy, freqai.dk, data_load_timerange)
    assert len(freqai.dk.label_list) == 2
    assert Path(freqai.dk.data_path / f"{freqai.dk.model_filename}_model.joblib").is_file()
@ -76,75 +96,17 @@ def test_train_model_in_series_LightGBMMultiModel(mocker, freqai_conf):
    shutil.rmtree(Path(freqai.dk.full_path))
-@pytest.mark.skipif(is_arm(), reason="no ARM for Catboost ...")
+@pytest.mark.parametrize('model', [
-def test_train_model_in_series_Catboost(mocker, freqai_conf):
+    'LightGBMClassifier',
-    freqai_conf.update({"timerange": "20180110-20180130"})
+    'CatboostClassifier',
-    freqai_conf.update({"freqaimodel": "CatboostRegressor"})
+    ])
-    # freqai_conf.get('freqai', {}).update(
+def test_extract_data_and_train_model_Classifiers(mocker, freqai_conf, model):
-    #     {'model_training_parameters': {"n_estimators": 100, "verbose": 0}})
+    if is_arm() and model == 'CatboostClassifier':
-    strategy = get_patched_freqai_strategy(mocker, freqai_conf)
+        pytest.skip("CatBoost is not supported on ARM")
    exchange = get_patched_exchange(mocker, freqai_conf)
    strategy.dp = DataProvider(freqai_conf, exchange)
-    strategy.freqai_info = freqai_conf.get("freqai", {})
+    freqai_conf.update({"freqaimodel": model})
    freqai = strategy.freqai
    freqai.live = True
    freqai.dk = FreqaiDataKitchen(freqai_conf)
    timerange = TimeRange.parse_timerange("20180110-20180130")
    freqai.dd.load_all_pair_histories(timerange, freqai.dk)
    freqai.dd.pair_dict = MagicMock()
    data_load_timerange = TimeRange.parse_timerange("20180110-20180130")
    new_timerange = TimeRange.parse_timerange("20180120-20180130")
    freqai.train_model_in_series(new_timerange, "ADA/BTC",
                                 strategy, freqai.dk, data_load_timerange)
    assert Path(freqai.dk.data_path / f"{freqai.dk.model_filename}_model.joblib").exists()
    assert Path(freqai.dk.data_path / f"{freqai.dk.model_filename}_metadata.json").exists()
    assert Path(freqai.dk.data_path / f"{freqai.dk.model_filename}_trained_df.pkl").exists()
    assert Path(freqai.dk.data_path / f"{freqai.dk.model_filename}_svm_model.joblib").exists()
    shutil.rmtree(Path(freqai.dk.full_path))
@pytest.mark.skipif(is_arm(), reason="no ARM for Catboost ...")
 def test_train_model_in_series_CatboostClassifier(mocker, freqai_conf):
    freqai_conf.update({"timerange": "20180110-20180130"})
    freqai_conf.update({"freqaimodel": "CatboostClassifier"})
    freqai_conf.update({"strategy": "freqai_test_classifier"})
    strategy = get_patched_freqai_strategy(mocker, freqai_conf)
    exchange = get_patched_exchange(mocker, freqai_conf)
    strategy.dp = DataProvider(freqai_conf, exchange)
    strategy.freqai_info = freqai_conf.get("freqai", {})
    freqai = strategy.freqai
    freqai.live = True
    freqai.dk = FreqaiDataKitchen(freqai_conf)
    timerange = TimeRange.parse_timerange("20180110-20180130")
    freqai.dd.load_all_pair_histories(timerange, freqai.dk)
    freqai.dd.pair_dict = MagicMock()
    data_load_timerange = TimeRange.parse_timerange("20180110-20180130")
    new_timerange = TimeRange.parse_timerange("20180120-20180130")
    freqai.train_model_in_series(new_timerange, "ADA/BTC",
                                 strategy, freqai.dk, data_load_timerange)
    assert Path(freqai.dk.data_path / f"{freqai.dk.model_filename}_model.joblib").exists()
    assert Path(freqai.dk.data_path / f"{freqai.dk.model_filename}_metadata.json").exists()
    assert Path(freqai.dk.data_path / f"{freqai.dk.model_filename}_trained_df.pkl").exists()
    assert Path(freqai.dk.data_path / f"{freqai.dk.model_filename}_svm_model.joblib").exists()
    shutil.rmtree(Path(freqai.dk.full_path))
 def test_train_model_in_series_LightGBMClassifier(mocker, freqai_conf):
    freqai_conf.update({"timerange": "20180110-20180130"})
    freqai_conf.update({"freqaimodel": "LightGBMClassifier"})
    freqai_conf.update({"strategy": "freqai_test_classifier"})
    strategy = get_patched_freqai_strategy(mocker, freqai_conf)
    exchange = get_patched_exchange(mocker, freqai_conf)
    strategy.dp = DataProvider(freqai_conf, exchange)
@ -161,7 +123,7 @@ def test_train_model_in_series_LightGBMClassifier(mocker, freqai_conf):
    data_load_timerange = TimeRange.parse_timerange("20180110-20180130")
    new_timerange = TimeRange.parse_timerange("20180120-20180130")
-    freqai.train_model_in_series(new_timerange, "ADA/BTC",
+    freqai.extract_data_and_train_model(new_timerange, "ADA/BTC",
                                        strategy, freqai.dk, data_load_timerange)
    assert Path(freqai.dk.data_path / f"{freqai.dk.model_filename}_model.joblib").exists()
@ -296,7 +258,8 @@ def test_follow_mode(mocker, freqai_conf):
    data_load_timerange = TimeRange.parse_timerange("20180110-20180130")
    new_timerange = TimeRange.parse_timerange("20180120-20180130")
-    freqai.train_model_in_series(new_timerange, "ADA/BTC", strategy, freqai.dk, data_load_timerange)
+    freqai.extract_data_and_train_model(
        new_timerange, "ADA/BTC", strategy, freqai.dk, data_load_timerange)
    assert Path(freqai.dk.data_path / f"{freqai.dk.model_filename}_model.joblib").is_file()
    assert Path(freqai.dk.data_path / f"{freqai.dk.model_filename}_metadata.json").is_file()
@ -345,7 +308,8 @@ def test_principal_component_analysis(mocker, freqai_conf):
    data_load_timerange = TimeRange.parse_timerange("20180110-20180130")
    new_timerange = TimeRange.parse_timerange("20180120-20180130")
-    freqai.train_model_in_series(new_timerange, "ADA/BTC", strategy, freqai.dk, data_load_timerange)
+    freqai.extract_data_and_train_model(
        new_timerange, "ADA/BTC", strategy, freqai.dk, data_load_timerange)
    assert Path(freqai.dk.data_path / f"{freqai.dk.model_filename}_pca_object.pkl")
--- a/tests/optimize/test_optimize_reports.py
+++ b/tests/optimize/test_optimize_reports.py
@ -40,14 +40,14 @@ def test_text_table_bt_results():
    )
    result_str = (
-        '|    Pair |   Buys |   Avg Profit % |   Cum Profit % |   Tot Profit BTC |   Tot Profit % |'
+        '|    Pair |   Entries |   Avg Profit % |   Cum Profit % |   Tot Profit BTC |   '
-        '   Avg Duration |   Win  Draw  Loss  Win% |\n'
+        'Tot Profit % |   Avg Duration |   Win  Draw  Loss  Win% |\n'
-        '|---------+--------+----------------+----------------+------------------+----------------+'
+        '|---------+-----------+----------------+----------------+------------------+'
-        '----------------+-------------------------|\n'
+        '----------------+----------------+-------------------------|\n'
-        '| ETH/BTC |      3 |           8.33 |          25.00 |       0.50000000 |          12.50 |'
+        '| ETH/BTC |         3 |           8.33 |          25.00 |       0.50000000 |          '
-        '        0:20:00 |     2     0     1  66.7 |\n'
+        '12.50 |        0:20:00 |     2     0     1  66.7 |\n'
-        '|   TOTAL |      3 |           8.33 |          25.00 |       0.50000000 |          12.50 |'
+        '|   TOTAL |         3 |           8.33 |          25.00 |       0.50000000 |          '
-        '        0:20:00 |     2     0     1  66.7 |'
+        '12.50 |        0:20:00 |     2     0     1  66.7 |'
    )
    pair_results = generate_pair_metrics(['ETH/BTC'], stake_currency='BTC',
@ -402,9 +402,9 @@ def test_text_table_strategy(testdatadir):
    bt_res_data_comparison = bt_res_data.pop('strategy_comparison')
    result_str = (
-        '|       Strategy |   Buys |   Avg Profit % |   Cum Profit % |   Tot Profit BTC |'
+        '|       Strategy |   Entries |   Avg Profit % |   Cum Profit % |   Tot Profit BTC |'
        '   Tot Profit % |   Avg Duration |   Win  Draw  Loss  Win% |              Drawdown |\n'
-        '|----------------+--------+----------------+----------------+------------------+'
+        '|----------------+-----------+----------------+----------------+------------------+'
        '----------------+----------------+-------------------------+-----------------------|\n'
        '| StrategyTestV2 |       179 |           0.08 |          14.39 |       0.02608550 |'
        '         260.85 |        3:40:00 |   170     0     9  95.0 | 0.00308222 BTC  8.67% |\n'
--- a/tests/rpc/test_rpc_telegram.py
+++ b/tests/rpc/test_rpc_telegram.py
@ -2138,11 +2138,11 @@ def test_send_msg_strategy_msg_notification(default_conf, mocker) -> None:
 def test_send_msg_unknown_type(default_conf, mocker) -> None:
-    telegram, _, _ = get_telegram_testobject(mocker, default_conf)
+    telegram, _, msg_mock = get_telegram_testobject(mocker, default_conf)
    with pytest.raises(NotImplementedError, match=r'Unknown message type: None'):
    telegram.send_msg({
        'type': None,
    })
    msg_mock.call_count == 0
@pytest.mark.parametrize('message_type,enter,enter_signal,leverage', [
--- a/tests/strategy/test_interface.py
+++ b/tests/strategy/test_interface.py
@ -11,8 +11,7 @@ from pandas import DataFrame
 from freqtrade.configuration import TimeRange
 from freqtrade.data.dataprovider import DataProvider
 from freqtrade.data.history import load_data
-from freqtrade.enums import ExitCheckTuple, ExitType, SignalDirection
+from freqtrade.enums import ExitCheckTuple, ExitType, HyperoptState, SignalDirection
 from freqtrade.enums.hyperoptstate import HyperoptState
 from freqtrade.exceptions import OperationalException, StrategyError
 from freqtrade.optimize.hyperopt_tools import HyperoptStateContainer
 from freqtrade.optimize.space import SKDecimal
--- a/tests/strategy/test_strategy_helpers.py
+++ b/tests/strategy/test_strategy_helpers.py
@ -117,6 +117,29 @@ def test_merge_informative_pair_lower():
        merge_informative_pair(data, informative, '1h', '15m', ffill=True)
 def test_merge_informative_pair_suffix():
    data = generate_test_data('15m', 20)
    informative = generate_test_data('1h', 20)
    result = merge_informative_pair(data, informative, '15m', '1h',
                                    append_timeframe=False, suffix="suf")
    assert 'date' in result.columns
    assert result['date'].equals(data['date'])
    assert 'date_suf' in result.columns
    assert 'open_suf' in result.columns
    assert 'open_1h' not in result.columns
 def test_merge_informative_pair_suffix_append_timeframe():
    data = generate_test_data('15m', 20)
    informative = generate_test_data('1h', 20)
    with pytest.raises(ValueError, match=r"You can not specify `append_timeframe` .*"):
        merge_informative_pair(data, informative, '15m', '1h', suffix="suf")
 def test_stoploss_from_open():
    open_price_ranges = [
        [0.01, 1.00, 30],
--- a/tests/test_freqtradebot.py
+++ b/tests/test_freqtradebot.py
@ -506,7 +506,7 @@ def test_create_trades_multiple_trades(
 def test_create_trades_preopen(default_conf_usdt, ticker_usdt, fee, mocker,
-                               limit_buy_order_usdt_open) -> None:
+                               limit_buy_order_usdt_open, caplog) -> None:
    patch_RPCManager(mocker)
    patch_exchange(mocker)
    default_conf_usdt['max_open_trades'] = 4
@ -515,6 +515,7 @@ def test_create_trades_preopen(default_conf_usdt, ticker_usdt, fee, mocker,
        fetch_ticker=ticker_usdt,
        create_order=MagicMock(return_value=limit_buy_order_usdt_open),
        get_fee=fee,
        get_funding_fees=MagicMock(side_effect=ExchangeError()),
    )
    freqtrade = FreqtradeBot(default_conf_usdt)
    patch_get_signal(freqtrade)
@ -522,6 +523,7 @@ def test_create_trades_preopen(default_conf_usdt, ticker_usdt, fee, mocker,
    # Create 2 existing trades
    freqtrade.execute_entry('ETH/USDT', default_conf_usdt['stake_amount'])
    freqtrade.execute_entry('NEO/BTC', default_conf_usdt['stake_amount'])
    assert log_has("Could not find funding fee.", caplog)
    assert len(Trade.get_open_trades()) == 2
    # Change order_id for new orders
@ -3655,6 +3657,7 @@ def test_may_execute_trade_exit_after_stoploss_on_exchange_hit(
    assert trade.exit_reason == ExitType.STOPLOSS_ON_EXCHANGE.value
    assert rpc_mock.call_count == 3
    assert rpc_mock.call_args_list[0][0][0]['type'] == RPCMessageType.ENTRY
    assert rpc_mock.call_args_list[0][0][0]['amount'] > 20
    assert rpc_mock.call_args_list[1][0][0]['type'] == RPCMessageType.ENTRY_FILL
    assert rpc_mock.call_args_list[2][0][0]['type'] == RPCMessageType.EXIT_FILL
@ -3665,7 +3668,7 @@ def test_may_execute_trade_exit_after_stoploss_on_exchange_hit(
        (True, 29.70297029, 2.2, 2.3, -8.63762376, -0.1443212, 'loss'),
    ])
 def test_execute_trade_exit_market_order(
-    default_conf_usdt, ticker_usdt, fee, is_short, current_rate, amount,
+    default_conf_usdt, ticker_usdt, fee, is_short, current_rate, amount, caplog,
    limit, profit_amount, profit_ratio, profit_or_loss, ticker_usdt_sell_up, mocker
 ) -> None:
    """
@ -3693,6 +3696,7 @@ def test_execute_trade_exit_market_order(
        fetch_ticker=ticker_usdt,
        get_fee=fee,
        _is_dry_limit_order_filled=MagicMock(return_value=True),
        get_funding_fees=MagicMock(side_effect=ExchangeError()),
    )
    patch_whitelist(mocker, default_conf_usdt)
    freqtrade = FreqtradeBot(default_conf_usdt)
@ -3718,6 +3722,7 @@ def test_execute_trade_exit_market_order(
        limit=ticker_usdt_sell_up()['ask' if is_short else 'bid'],
        exit_check=ExitCheckTuple(exit_type=ExitType.ROI)
    )
    assert log_has("Could not update funding fee.", caplog)
    assert not trade.is_open
    assert pytest.approx(trade.close_profit) == profit_ratio
@ -5429,6 +5434,16 @@ def test_update_funding_fees(
        ))
 def test_update_funding_fees_error(mocker, default_conf, caplog):
    mocker.patch('freqtrade.exchange.Exchange.get_funding_fees', side_effect=ExchangeError())
    default_conf['trading_mode'] = 'futures'
    default_conf['margin_mode'] = 'isolated'
    freqtrade = get_patched_freqtradebot(mocker, default_conf)
    freqtrade.update_funding_fees()
    log_has("Could not update funding fees for open trades.", caplog)
 def test_position_adjust(mocker, default_conf_usdt, fee) -> None:
    patch_RPCManager(mocker)
    patch_exchange(mocker)
--- a/tests/test_integration.py
+++ b/tests/test_integration.py
@ -485,7 +485,7 @@ def test_dca_exiting(default_conf_usdt, ticker_usdt, fee, mocker, caplog) -> Non
    assert len(trade.orders) == 1
    assert pytest.approx(trade.stake_amount) == 60
    assert pytest.approx(trade.amount) == 30.0
-    assert log_has_re("Remaining amount of 1.6.* would be too small.", caplog)
+    assert log_has_re("Remaining amount of 1.6.* would be smaller than the minimum of 10.", caplog)
    freqtrade.strategy.adjust_trade_position = MagicMock(return_value=-20)
@ -504,9 +504,21 @@ def test_dca_exiting(default_conf_usdt, ticker_usdt, fee, mocker, caplog) -> Non
    freqtrade.strategy.adjust_trade_position = MagicMock(return_value=-50)
    freqtrade.process()
    assert log_has_re("Adjusting amount to trade.amount as it is higher.*", caplog)
-    assert log_has_re("Remaining amount of 0.0 would be too small.", caplog)
+    assert log_has_re("Remaining amount of 0.0 would be smaller than the minimum of 10.", caplog)
    trade = Trade.get_trades().first()
    assert len(trade.orders) == 2
    assert trade.orders[-1].ft_order_side == 'sell'
    assert pytest.approx(trade.stake_amount) == 40.198
    assert trade.is_open
    # use amount that would trunc to 0.0 once selling
    mocker.patch("freqtrade.exchange.Exchange.amount_to_contract_precision",
                 lambda s, p, v: round(v, 1))
    freqtrade.strategy.adjust_trade_position = MagicMock(return_value=-0.01)
    freqtrade.process()
    trade = Trade.get_trades().first()
    assert len(trade.orders) == 2
    assert trade.orders[-1].ft_order_side == 'sell'
    assert pytest.approx(trade.stake_amount) == 40.198
    assert trade.is_open
    assert log_has_re('Amount to sell is 0.0 due to exchange limits - not selling.', caplog)
--- a/tests/test_persistence.py
+++ b/tests/test_persistence.py
@ -9,7 +9,7 @@ import arrow
 import pytest
 from sqlalchemy import create_engine, text
-from freqtrade import constants
+from freqtrade.constants import DATETIME_PRINT_FORMAT, DEFAULT_DB_PROD_URL
 from freqtrade.enums import TradingMode
 from freqtrade.exceptions import DependencyException, OperationalException
 from freqtrade.persistence import LocalTrade, Order, Trade, init_db
@ -52,7 +52,7 @@ def test_init_invalid_db_url():
 def test_init_prod_db(default_conf, mocker):
    default_conf.update({'dry_run': False})
-    default_conf.update({'db_url': constants.DEFAULT_DB_PROD_URL})
+    default_conf.update({'db_url': DEFAULT_DB_PROD_URL})
    create_engine_mock = mocker.patch('freqtrade.persistence.models.create_engine', MagicMock())
@ -615,21 +615,25 @@ def test_calc_open_close_trade_price(
        is_short=is_short,
        leverage=lev,
        trading_mode=trading_mode,
        funding_fees=funding_fees
    )
    entry_order = limit_order[trade.entry_side]
    exit_order = limit_order[trade.exit_side]
    trade.open_order_id = f'something-{is_short}-{lev}-{exchange}'
    oobj = Order.parse_from_ccxt_object(entry_order, 'ADA/USDT', trade.entry_side)
-    trade.orders.append(oobj)
+    oobj.trade = trade
    oobj.update_from_ccxt_object(entry_order)
    trade.update_trade(oobj)
    trade.funding_fees = funding_fees
    oobj = Order.parse_from_ccxt_object(exit_order, 'ADA/USDT', trade.exit_side)
-    trade.orders.append(oobj)
+    oobj.trade = trade
    oobj.update_from_ccxt_object(exit_order)
    trade.update_trade(oobj)
    assert trade.is_open is False
    assert trade.funding_fees == funding_fees
    assert pytest.approx(trade._calc_open_trade_value(trade.amount, trade.open_rate)) == open_value
    assert pytest.approx(trade.calc_close_trade_value(trade.close_rate)) == close_value
@ -1735,7 +1739,7 @@ def test_to_json(fee):
                      'base_currency': 'ADA',
                      'quote_currency': 'USDT',
                      'is_open': None,
-                      'open_date': trade.open_date.strftime("%Y-%m-%d %H:%M:%S"),
+                      'open_date': trade.open_date.strftime(DATETIME_PRINT_FORMAT),
                      'open_timestamp': int(trade.open_date.timestamp() * 1000),
                      'open_order_id': 'dry_run_buy_12345',
                      'close_date': None,
@ -1813,9 +1817,9 @@ def test_to_json(fee):
                      'pair': 'XRP/BTC',
                      'base_currency': 'XRP',
                      'quote_currency': 'BTC',
-                      'open_date': trade.open_date.strftime("%Y-%m-%d %H:%M:%S"),
+                      'open_date': trade.open_date.strftime(DATETIME_PRINT_FORMAT),
                      'open_timestamp': int(trade.open_date.timestamp() * 1000),
-                      'close_date': trade.close_date.strftime("%Y-%m-%d %H:%M:%S"),
+                      'close_date': trade.close_date.strftime(DATETIME_PRINT_FORMAT),
                      'close_timestamp': int(trade.close_date.timestamp() * 1000),
                      'open_rate': 0.123,
                      'close_rate': 0.125,