Merge pull request #7954 from freqtrade/new_release

New release 2022.12

.github/workflows/ci.yml

@@ -148,6 +148,19 @@ jobs:
       if: runner.os == 'macOS'
       run: |
         brew update
+        # homebrew fails to update python due to unlinking failures
+        # https://github.com/actions/runner-images/issues/6817
+        rm /usr/local/bin/2to3 || true
+        rm /usr/local/bin/2to3-3.11 || true
+        rm /usr/local/bin/idle3 || true
+        rm /usr/local/bin/idle3.11 || true
+        rm /usr/local/bin/pydoc3 || true
+        rm /usr/local/bin/pydoc3.11 || true
+        rm /usr/local/bin/python3 || true
+        rm /usr/local/bin/python3.11 || true
+        rm /usr/local/bin/python3-config || true
+        rm /usr/local/bin/python3.11-config || true
+
         brew install hdf5 c-blosc
         python -m pip install --upgrade pip wheel
         export LD_LIBRARY_PATH=${HOME}/dependencies/lib:$LD_LIBRARY_PATH

.pre-commit-config.yaml

@@ -15,9 +15,9 @@ repos:
         additional_dependencies:
           - types-cachetools==5.2.1
           - types-filelock==3.2.7
-          - types-requests==2.28.11.5
+          - types-requests==2.28.11.7
           - types-tabulate==0.9.0.0
-          - types-python-dateutil==2.8.19.4
+          - types-python-dateutil==2.8.19.5
         # stages: [push]
 
   - repo: https://github.com/pycqa/isort

README.md

@@ -1,6 +1,7 @@
 # ![freqtrade](https://raw.githubusercontent.com/freqtrade/freqtrade/develop/docs/assets/freqtrade_poweredby.svg)
 
 [![Freqtrade CI](https://github.com/freqtrade/freqtrade/workflows/Freqtrade%20CI/badge.svg)](https://github.com/freqtrade/freqtrade/actions/)
+[![DOI](https://joss.theoj.org/papers/10.21105/joss.04864/status.svg)](https://doi.org/10.21105/joss.04864)
 [![Coverage Status](https://coveralls.io/repos/github/freqtrade/freqtrade/badge.svg?branch=develop&service=github)](https://coveralls.io/github/freqtrade/freqtrade?branch=develop)
 [![Documentation](https://readthedocs.org/projects/freqtrade/badge/)](https://www.freqtrade.io)
 [![Maintainability](https://api.codeclimate.com/v1/badges/5737e6d668200b7518ff/maintainability)](https://codeclimate.com/github/freqtrade/freqtrade/maintainability)

docs/freqai-parameter-table.md

@@ -15,7 +15,7 @@ Mandatory parameters are marked as **Required** and have to be set in one of the
 | `identifier` | **Required.** <br> A unique ID for the current model. If models are saved to disk, the `identifier` allows for reloading specific pre-trained models/data. <br> **Datatype:** String.
 | `live_retrain_hours` | Frequency of retraining during dry/live runs. <br> **Datatype:** Float > 0. <br> Default: `0` (models retrain as often as possible).
 | `expiration_hours` | Avoid making predictions if a model is more than `expiration_hours` old. <br> **Datatype:** Positive integer. <br> Default: `0` (models never expire).
-| `purge_old_models` | Delete obsolete models. <br> **Datatype:** Boolean. <br> Default: `False` (all historic models remain on disk).
+| `purge_old_models` | Delete all unused models during live runs (not relevant to backtesting). If set to false (not default), dry/live runs will accumulate all unused models to disk. If <br> **Datatype:** Boolean. <br> Default: `True`.
 | `save_backtest_models` | Save models to disk when running backtesting. Backtesting operates most efficiently by saving the prediction data and reusing them directly for subsequent runs (when you wish to tune entry/exit parameters). Saving backtesting models to disk also allows to use the same model files for starting a dry/live instance with the same model `identifier`. <br> **Datatype:** Boolean. <br> Default: `False` (no models are saved).
 | `fit_live_predictions_candles` | Number of historical candles to use for computing target (label) statistics from prediction data, instead of from the training dataset (more information can be found [here](freqai-configuration.md#creating-a-dynamic-target-threshold)). <br> **Datatype:** Positive integer.
 | `follow_mode` | Use a `follower` that will look for models associated with a specific `identifier` and load those for inferencing. A `follower` will **not** train new models. <br> **Datatype:** Boolean. <br> Default: `False`.

docs/freqai-reinforcement-learning.md

@@ -275,12 +275,12 @@ FreqAI also provides a built in episodic summary logger called `self.tensorboard
 
 ### Choosing a base environment
 
-FreqAI provides two base environments, `Base4ActionEnvironment` and `Base5ActionEnvironment`. As the names imply, the environments are customized for agents that can select from 4 or 5 actions. In the `Base4ActionEnvironment`, the agent can enter long, enter short, hold neutral, or exit position. Meanwhile, in the `Base5ActionEnvironment`, the agent has the same actions as Base4, but instead of a single exit action, it separates exit long and exit short. The main changes stemming from the environment selection include:
+FreqAI provides three base environments, `Base3ActionRLEnvironment`, `Base4ActionEnvironment` and `Base5ActionEnvironment`. As the names imply, the environments are customized for agents that can select from 3, 4 or 5 actions. The `Base3ActionEnvironment` is the simplest, the agent can select from hold, long, or short. This environment can also be used for long-only bots (it automatically follows the `can_short` flag from the strategy), where long is the enter condition and short is the exit condition. Meanwhile, in the `Base4ActionEnvironment`, the agent can enter long, enter short, hold neutral, or exit position. Finally, in the `Base5ActionEnvironment`, the agent has the same actions as Base4, but instead of a single exit action, it separates exit long and exit short. The main changes stemming from the environment selection include:
 
 * the actions available in the `calculate_reward`
 * the actions consumed by the user strategy
 
-Both of the FreqAI provided environments inherit from an action/position agnostic environment object called the `BaseEnvironment`, which contains all shared logic. The architecture is designed to be easily customized. The simplest customization is the `calculate_reward()` (see details [here](#creating-a-custom-reward-function)). However, the customizations can be further extended into any of the functions inside the environment. You can do this by simply overriding those functions inside your `MyRLEnv` in the prediction model file. Or for more advanced customizations, it is encouraged to create an entirely new environment inherited from `BaseEnvironment`.
+All of the FreqAI provided environments inherit from an action/position agnostic environment object called the `BaseEnvironment`, which contains all shared logic. The architecture is designed to be easily customized. The simplest customization is the `calculate_reward()` (see details [here](#creating-a-custom-reward-function)). However, the customizations can be further extended into any of the functions inside the environment. You can do this by simply overriding those functions inside your `MyRLEnv` in the prediction model file. Or for more advanced customizations, it is encouraged to create an entirely new environment inherited from `BaseEnvironment`.
 
 !!! Note
-    FreqAI does not provide by default, a long-only training environment. However, creating one should be as simple as copy-pasting one of the built in environments and removing the `short` actions (and all associated references to those).
+    Only the `Base3ActionRLEnv` can do long-only training/trading (set the user strategy attribute `can_short = False`).

docs/freqai.md

@@ -72,11 +72,25 @@ pip install -r requirements-freqai.txt
 
 If you are using docker, a dedicated tag with FreqAI dependencies is available as `:freqai`. As such - you can replace the image line in your docker-compose file with `image: freqtradeorg/freqtrade:develop_freqai`. This image contains the regular FreqAI dependencies. Similar to native installs, Catboost will not be available on ARM based devices.
 
-
 ### FreqAI position in open-source machine learning landscape
 
 Forecasting chaotic time-series based systems, such as equity/cryptocurrency markets, requires a broad set of tools geared toward testing a wide range of hypotheses. Fortunately, a recent maturation of robust machine learning libraries (e.g. `scikit-learn`) has opened up a wide range of research possibilities. Scientists from a diverse range of fields can now easily prototype their studies on an abundance of established machine learning algorithms. Similarly, these user-friendly libraries enable "citzen scientists" to use their basic Python skills for data-exploration. However, leveraging these machine learning libraries on historical and live chaotic data sources can be logistically difficult and expensive. Additionally, robust data-collection, storage, and handling presents a disparate challenge. [`FreqAI`](#freqai) aims to provide a generalized and extensible open-sourced framework geared toward live deployments of adaptive modeling for market forecasting. The `FreqAI` framework is effectively a sandbox for the rich world of open-source machine learning libraries. Inside the `FreqAI` sandbox, users find they can combine a wide variety of third-party libraries to test creative hypotheses on a free live 24/7 chaotic data source - cryptocurrency exchange data. 
 
+### Citing FreqAI
+
+FreqAI is [published in the Journal of Open Source Software](https://joss.theoj.org/papers/10.21105/joss.04864). If you find FreqAI useful in your research, please use the following citation:
+
+```bibtex
+@article{Caulk2022, 
+    doi = {10.21105/joss.04864},
+    url = {https://doi.org/10.21105/joss.04864},
+    year = {2022}, publisher = {The Open Journal},
+    volume = {7}, number = {80}, pages = {4864},
+    author = {Robert A. Caulk and Elin Törnquist and Matthias Voppichler and Andrew R. Lawless and Ryan McMullan and Wagner Costa Santos and Timothy C. Pogue and Johan van der Vlugt and Stefan P. Gehring and Pascal Schmidt},
+    title = {FreqAI: generalizing adaptive modeling for chaotic time-series market forecasts},
+    journal = {Journal of Open Source Software} } 
+```
+
 ## Common pitfalls
 
 FreqAI cannot be combined with dynamic `VolumePairlists` (or any pairlist filter that adds and removes pairs dynamically).
@@ -99,6 +113,8 @@ Code review and software architecture brainstorming:
 
 Software development:
 Wagner Costa @wagnercosta
+Emre Suzen @aemr3
+Timothy Pogue @wizrds
 
 Beta testing and bug reporting:
-Stefan Gehring @bloodhunter4rc, @longyu, Andrew Lawless @paranoidandy, Pascal Schmidt @smidelis, Ryan McMullan @smarmau, Juha Nykänen @suikula, Johan van der Vlugt @jooopiert, Richárd Józsa @richardjosza, Timothy Pogue @wizrds
+Stefan Gehring @bloodhunter4rc, @longyu, Andrew Lawless @paranoidandy, Pascal Schmidt @smidelis, Ryan McMullan @smarmau, Juha Nykänen @suikula, Johan van der Vlugt @jooopiert, Richárd Józsa @richardjosza

docs/includes/pairlists.md

@@ -23,6 +23,7 @@ You may also use something like `.*DOWN/BTC` or `.*UP/BTC` to exclude leveraged
 * [`StaticPairList`](#static-pair-list) (default, if not configured differently)
 * [`VolumePairList`](#volume-pair-list)
 * [`ProducerPairList`](#producerpairlist)
+* [`RemotePairList`](#remotepairlist)
 * [`AgeFilter`](#agefilter)
 * [`OffsetFilter`](#offsetfilter)
 * [`PerformanceFilter`](#performancefilter)
@@ -173,6 +174,48 @@ You can limit the length of the pairlist with the optional parameter `number_ass
     `ProducerPairList` can also be used multiple times in sequence, combining the pairs from multiple producers.
     Obviously in complex such configurations, the Producer may not provide data for all pairs, so the strategy must be fit for this.
 
+#### RemotePairList
+
+It allows the user to fetch a pairlist from a remote server or a locally stored json file within the freqtrade directory, enabling dynamic updates and customization of the trading pairlist.
+
+The RemotePairList is defined in the pairlists section of the configuration settings. It uses the following configuration options:
+
+```json
+"pairlists": [
+    {
+        "method": "RemotePairList",
+        "pairlist_url": "https://example.com/pairlist",
+        "number_assets": 10,
+        "refresh_period": 1800,
+        "keep_pairlist_on_failure": true,
+        "read_timeout": 60,
+        "bearer_token": "my-bearer-token"
+    }
+]
+```
+
+The `pairlist_url` option specifies the URL of the remote server where the pairlist is located, or the path to a local file (if file:/// is prepended). This allows the user to use either a remote server or a local file as the source for the pairlist.
+
+The user is responsible for providing a server or local file that returns a JSON object with the following structure:
+
+```json
+{
+    "pairs": ["XRP/USDT", "ETH/USDT", "LTC/USDT"],
+    "refresh_period": 1800,
+}
+```
+
+The `pairs` property should contain a list of strings with the trading pairs to be used by the bot. The `refresh_period` property is optional and specifies the number of seconds that the pairlist should be cached before being refreshed.
+
+The optional `keep_pairlist_on_failure` specifies whether the previous received pairlist should be used if the remote server is not reachable or returns an error. The default value is true.
+
+The optional `read_timeout` specifies the maximum amount of time (in seconds) to wait for a response from the remote source, The default value is 60.
+
+The optional `bearer_token` will be included in the requests Authorization Header.
+
+!!! Note
+    In case of a server error the last received pairlist will be kept if `keep_pairlist_on_failure` is set to true, when set to false a empty pairlist is returned.
+
 #### AgeFilter
 
 Removes pairs that have been listed on the exchange for less than `min_days_listed` days (defaults to `10`) or more than `max_days_listed` days (defaults `None` mean infinity).

docs/index.md

@@ -1,6 +1,7 @@
 ![freqtrade](assets/freqtrade_poweredby.svg)
 
 [![Freqtrade CI](https://github.com/freqtrade/freqtrade/workflows/Freqtrade%20CI/badge.svg)](https://github.com/freqtrade/freqtrade/actions/)
+[![DOI](https://joss.theoj.org/papers/10.21105/joss.04864/status.svg)](https://doi.org/10.21105/joss.04864)
 [![Coverage Status](https://coveralls.io/repos/github/freqtrade/freqtrade/badge.svg?branch=develop&service=github)](https://coveralls.io/github/freqtrade/freqtrade?branch=develop)
 [![Maintainability](https://api.codeclimate.com/v1/badges/5737e6d668200b7518ff/maintainability)](https://codeclimate.com/github/freqtrade/freqtrade/maintainability)
 

docs/overrides/main.html

@@ -11,9 +11,6 @@
   {% endif %}
   <div class="md-sidebar md-sidebar--primary" data-md-component="sidebar" data-md-type="navigation" {{ hidden }}>
     <div class="md-sidebar__scrollwrap">
-      <div id="widget-wrapper">
-
-      </div>
       <div class="md-sidebar__inner">
         {% include "partials/nav.html" %}
       </div>
@@ -44,25 +41,4 @@
     <script src="https://code.jquery.com/jquery-3.4.1.min.js"
       integrity="sha256-CSXorXvZcTkaix6Yvo6HppcZGetbYMGWSFlBw8HfCJo=" crossorigin="anonymous"></script>
 
-    <!-- Load binance SDK -->
-    <script async defer src="https://public.bnbstatic.com/static/js/broker-sdk/broker-sdk@1.0.0.min.js"></script>
-
-    <script>
-      window.onload = function () {
-        var sidebar = document.getElementById('widget-wrapper')
-        var newDiv = document.createElement("div");
-        newDiv.id = "widget";
-        try {
-          sidebar.prepend(newDiv);
-
-          window.binanceBrokerPortalSdk.initBrokerSDK('#widget', {
-            apiHost: 'https://www.binance.com',
-            brokerId: 'R4BD3S82',
-            slideTime: 4e4,
-          });
-        } catch(err) {
-          console.log(err)
-        }
-      }
-    </script>
 {% endblock %}

docs/strategy-customization.md

@@ -989,38 +989,18 @@ from freqtrade.persistence import Trade
 The following example queries for the current pair and trades from today, however other filters can easily be added.
 
 ``` python
-if self.config['runmode'].value in ('live', 'dry_run'):
-    trades = Trade.get_trades([Trade.pair == metadata['pair'],
-                               Trade.open_date > datetime.utcnow() - timedelta(days=1),
-                               Trade.is_open.is_(False),
-                ]).order_by(Trade.close_date).all()
-    # Summarize profit for this pair.
-    curdayprofit = sum(trade.close_profit for trade in trades)
+trades = Trade.get_trades_proxy(pair=metadata['pair'],
+                                open_date=datetime.now(timezone.utc) - timedelta(days=1),
+                                is_open=False,
+            ]).order_by(Trade.close_date).all()
+# Summarize profit for this pair.
+curdayprofit = sum(trade.close_profit for trade in trades)
 ```
 
-Get amount of stake_currency currently invested in Trades:
-
-``` python
-if self.config['runmode'].value in ('live', 'dry_run'):
-    total_stakes = Trade.total_open_trades_stakes()
-```
-
-Retrieve performance per pair.
-Returns a List of dicts per pair.
-
-``` python
-if self.config['runmode'].value in ('live', 'dry_run'):
-    performance = Trade.get_overall_performance()
-```
-
-Sample return value: ETH/BTC had 5 trades, with a total profit of 1.5% (ratio of 0.015).
-
-``` json
-{"pair": "ETH/BTC", "profit": 0.015, "count": 5}
-```
+For a full list of available methods, please consult the [Trade object](trade-object.md) documentation.
 
 !!! Warning
-    Trade history is not available during backtesting or hyperopt.
+    Trade history is not available in `populate_*` methods during backtesting or hyperopt, and will result in empty results.
 
 ## Prevent trades from happening for a specific pair
 

docs/stylesheets/ft.extra.css

@@ -11,18 +11,3 @@
 .rst-versions .rst-other-versions {
     color: white;
 }
-
-
-#widget-wrapper {
-    height: calc(220px * 0.5625 + 18px);
-    width: 220px;
-    margin: 0 auto 16px auto;
-    border-style: solid;
-    border-color: var(--md-code-bg-color);
-    border-width: 1px;
-    border-radius: 5px;
-}
-
-@media screen and (max-width: calc(76.25em - 1px)) {
-    #widget-wrapper { display: none; }
-}

docs/trade-object.md

@@ -0,0 +1,148 @@
+# Trade Object
+
+## Trade
+
+A position freqtrade enters is stored in a `Trade` object - which is persisted to the database.
+It's a core concept of freqtrade - and something you'll come across in many sections of the documentation, which will most likely point you to this location.
+
+It will be passed to the strategy in many [strategy callbacks](strategy-callbacks.md). The object passed to the strategy cannot be modified directly. Indirect modifications may occur based on callback results.
+
+## Trade - Available attributes
+
+The following attributes / properties are available for each individual trade - and can be used with `trade.<property>` (e.g. `trade.pair`).
+
+|  Attribute | DataType | Description |
+|------------|-------------|-------------|
+`pair`| string | Pair of this trade
+`is_open`| boolean | Is the trade currently open, or has it been concluded
+`open_rate`| float | Rate this trade was entered at (Avg. entry rate in case of trade-adjustments)
+`close_rate`| float | Close rate - only set when is_open = False
+`stake_amount`| float | Amount in Stake (or Quote) currency.
+`amount`| float | Amount in Asset / Base currency that is currently owned.
+`open_date`| datetime | Timestamp when trade was opened **use `open_date_utc` instead**
+`open_date_utc`| datetime | Timestamp when trade was opened - in UTC
+`close_date`| datetime | Timestamp when trade was closed **use `close_date_utc` instead**
+`close_date_utc`| datetime | Timestamp when trade was closed - in UTC
+`close_profit`| float | Relative profit at the time of trade closure. `0.01` == 1%
+`close_profit_abs`| float | Absolute profit (in stake currency) at the time of trade closure.
+`leverage` | float | Leverage used for this trade - defaults to 1.0 in spot markets.
+`enter_tag`| string | Tag provided on entry via the `enter_tag` column in the dataframe
+`is_short` | boolean | True for short trades, False otherwise
+`orders` | Order[] | List of order objects attached to this trade (includes both filled and cancelled orders)
+`date_last_filled_utc` | datetime | Time of the last filled order
+`entry_side` | "buy" / "sell" | Order Side the trade was entered
+`exit_side` | "buy" / "sell" | Order Side that will result in a trade exit / position reduction.
+`trade_direction` | "long" / "short" | Trade direction in text - long or short.
+`nr_of_successful_entries` | int | Number of successful (filled) entry orders
+`nr_of_successful_exits` | int | Number of successful (filled) exit orders
+
+## Class methods
+
+The following are class methods - which return generic information, and usually result in an explicit query against the database.
+They can be used as `Trade.<method>` - e.g. `open_trades = Trade.get_open_trade_count()`
+
+!!! Warning "Backtesting/hyperopt"
+    Most methods will work in both backtesting / hyperopt and live/dry modes.
+    During backtesting, it's limited to usage in [strategy callbacks](strategy-callbacks.md). Usage in `populate_*()` methods is not supported and will result in wrong results.
+
+### get_trades_proxy
+
+When your strategy needs some information on existing (open or close) trades - it's best to use `Trade.get_trades_proxy()`.
+
+Usage:
+
+``` python
+from freqtrade.persistence import Trade
+from datetime import timedelta
+
+# ...
+trade_hist = Trade.get_trades_proxy(pair='ETH/USDT', is_open=False, open_date=current_date - timedelta(days=2))
+
+```
+
+`get_trades_proxy()` supports the following keyword arguments. All arguments are optional - calling `get_trades_proxy()` without arguments will return a list of all trades in the database.
+
+* `pair` e.g. `pair='ETH/USDT'`
+* `is_open` e.g. `is_open=False`
+* `open_date` e.g. `open_date=current_date - timedelta(days=2)`
+* `close_date` e.g. `close_date=current_date - timedelta(days=5)`
+
+### get_open_trade_count
+
+Get the number of currently open trades
+
+``` python
+from freqtrade.persistence import Trade
+# ...
+open_trades = Trade.get_open_trade_count()
+```
+
+### get_total_closed_profit
+
+Retrieve the total profit the bot has generated so far.
+Aggregates `close_profit_abs` for all closed trades.
+
+``` python
+from freqtrade.persistence import Trade
+
+# ...
+profit = Trade.get_total_closed_profit()
+```
+
+### total_open_trades_stakes
+
+Retrieve the total stake_amount that's currently in trades.
+
+``` python
+from freqtrade.persistence import Trade
+
+# ...
+profit = Trade.total_open_trades_stakes()
+```
+
+### get_overall_performance
+
+Retrieve the overall performance - similar to the `/performance` telegram command.
+
+``` python
+from freqtrade.persistence import Trade
+
+# ...
+if self.config['runmode'].value in ('live', 'dry_run'):
+    performance = Trade.get_overall_performance()
+```
+
+Sample return value: ETH/BTC had 5 trades, with a total profit of 1.5% (ratio of 0.015).
+
+``` json
+{"pair": "ETH/BTC", "profit": 0.015, "count": 5}
+```
+
+## Order Object
+
+An `Order` object represents an order on the exchange (or a simulated order in dry-run mode).
+An `Order` object will always be tied to it's corresponding [`Trade`](#trade-object), and only really makes sense in the context of a trade.
+
+### Order - Available attributes
+
+an Order object is typically attached to a trade.
+Most properties here can be None as they are dependant on the exchange response.
+
+|  Attribute | DataType | Description |
+|------------|-------------|-------------|
+`trade` | Trade | Trade object this order is attached to
+`ft_pair` | string | Pair this order is for
+`ft_is_open` | boolean | is the order filled?
+`order_type` | string | Order type as defined on the exchange - usually market, limit or stoploss
+`status` | string | Status as defined by ccxt. Usually open, closed, expired or canceled
+`side` | string | Buy or Sell
+`price` | float | Price the order was placed at
+`average` | float | Average price the order filled at
+`amount` | float | Amount in base currency
+`filled` | float | Filled amount (in base currency)
+`remaining` | float | Remaining amount
+`cost` | float | Cost of the order - usually average * filled
+`order_date` | datetime | Order creation date **use `order_date_utc` instead**
+`order_date_utc` | datetime | Order creation date (in UTC)
+`order_fill_date` | datetime |  Order fill date **use `order_fill_utc` instead**
+`order_fill_date_utc` | datetime | Order fill date

freqtrade/__init__.py

@@ -1,5 +1,5 @@
 """ Freqtrade bot """
-__version__ = '2022.12.dev'
+__version__ = '2022.12'
 
 if 'dev' in __version__:
     try:

freqtrade/constants.py

@@ -31,7 +31,7 @@ HYPEROPT_LOSS_BUILTIN = ['ShortTradeDurHyperOptLoss', 'OnlyProfitHyperOptLoss',
                          'CalmarHyperOptLoss',
                          'MaxDrawDownHyperOptLoss', 'MaxDrawDownRelativeHyperOptLoss',
                          'ProfitDrawDownHyperOptLoss']
-AVAILABLE_PAIRLISTS = ['StaticPairList', 'VolumePairList', 'ProducerPairList',
+AVAILABLE_PAIRLISTS = ['StaticPairList', 'VolumePairList', 'ProducerPairList', 'RemotePairList',
                        'AgeFilter', 'OffsetFilter', 'PerformanceFilter',
                        'PrecisionFilter', 'PriceFilter', 'RangeStabilityFilter',
                        'ShuffleFilter', 'SpreadFilter', 'VolatilityFilter']

freqtrade/data/btanalysis.py

@@ -20,8 +20,8 @@ from freqtrade.persistence import LocalTrade, Trade, init_db
 logger = logging.getLogger(__name__)
 
 # Newest format
-BT_DATA_COLUMNS = ['pair', 'stake_amount', 'amount', 'open_date', 'close_date',
-                   'open_rate', 'close_rate',
+BT_DATA_COLUMNS = ['pair', 'stake_amount', 'max_stake_amount', 'amount',
+                   'open_date', 'close_date', 'open_rate', 'close_rate',
                    'fee_open', 'fee_close', 'trade_duration',
                    'profit_ratio', 'profit_abs', 'exit_reason',
                    'initial_stop_loss_abs', 'initial_stop_loss_ratio', 'stop_loss_abs',
@@ -241,6 +241,33 @@ def find_existing_backtest_stats(dirname: Union[Path, str], run_ids: Dict[str, s
     return results
 
 
+def _load_backtest_data_df_compatibility(df: pd.DataFrame) -> pd.DataFrame:
+    """
+    Compatibility support for older backtest data.
+    """
+    df['open_date'] = pd.to_datetime(df['open_date'],
+                                     utc=True,
+                                     infer_datetime_format=True
+                                     )
+    df['close_date'] = pd.to_datetime(df['close_date'],
+                                      utc=True,
+                                      infer_datetime_format=True
+                                      )
+    # Compatibility support for pre short Columns
+    if 'is_short' not in df.columns:
+        df['is_short'] = False
+    if 'leverage' not in df.columns:
+        df['leverage'] = 1.0
+    if 'enter_tag' not in df.columns:
+        df['enter_tag'] = df['buy_tag']
+        df = df.drop(['buy_tag'], axis=1)
+    if 'max_stake_amount' not in df.columns:
+        df['max_stake_amount'] = df['stake_amount']
+    if 'orders' not in df.columns:
+        df['orders'] = None
+    return df
+
+
 def load_backtest_data(filename: Union[Path, str], strategy: Optional[str] = None) -> pd.DataFrame:
     """
     Load backtest data file.
@@ -269,24 +296,7 @@ def load_backtest_data(filename: Union[Path, str], strategy: Optional[str] = Non
         data = data['strategy'][strategy]['trades']
         df = pd.DataFrame(data)
         if not df.empty:
-            df['open_date'] = pd.to_datetime(df['open_date'],
-                                             utc=True,
-                                             infer_datetime_format=True
-                                             )
-            df['close_date'] = pd.to_datetime(df['close_date'],
-                                              utc=True,
-                                              infer_datetime_format=True
-                                              )
-            # Compatibility support for pre short Columns
-            if 'is_short' not in df.columns:
-                df['is_short'] = 0
-            if 'leverage' not in df.columns:
-                df['leverage'] = 1.0
-            if 'enter_tag' not in df.columns:
-                df['enter_tag'] = df['buy_tag']
-                df = df.drop(['buy_tag'], axis=1)
-            if 'orders' not in df.columns:
-                df['orders'] = None
+            df = _load_backtest_data_df_compatibility(df)
 
     else:
         # old format - only with lists.

freqtrade/exchange/binance.py

@@ -31,7 +31,7 @@ class Binance(Exchange):
         "ccxt_futures_name": "future"
     }
     _ft_has_futures: Dict = {
-        "stoploss_order_types": {"limit": "limit", "market": "market"},
+        "stoploss_order_types": {"limit": "stop", "market": "stop_market"},
         "tickers_have_price": False,
     }
 

freqtrade/freqai/RL/Base3ActionRLEnv.py

@@ -0,0 +1,125 @@
+import logging
+from enum import Enum
+
+from gym import spaces
+
+from freqtrade.freqai.RL.BaseEnvironment import BaseEnvironment, Positions
+
+
+logger = logging.getLogger(__name__)
+
+
+class Actions(Enum):
+    Neutral = 0
+    Buy = 1
+    Sell = 2
+
+
+class Base3ActionRLEnv(BaseEnvironment):
+    """
+    Base class for a 3 action environment
+    """
+    def __init__(self, **kwargs):
+        super().__init__(**kwargs)
+        self.actions = Actions
+
+    def set_action_space(self):
+        self.action_space = spaces.Discrete(len(Actions))
+
+    def step(self, action: int):
+        """
+        Logic for a single step (incrementing one candle in time)
+        by the agent
+        :param: action: int = the action type that the agent plans
+            to take for the current step.
+        :returns:
+            observation = current state of environment
+            step_reward = the reward from `calculate_reward()`
+            _done = if the agent "died" or if the candles finished
+            info = dict passed back to openai gym lib
+        """
+        self._done = False
+        self._current_tick += 1
+
+        if self._current_tick == self._end_tick:
+            self._done = True
+
+        self._update_unrealized_total_profit()
+        step_reward = self.calculate_reward(action)
+        self.total_reward += step_reward
+        self.tensorboard_log(self.actions._member_names_[action])
+
+        trade_type = None
+        if self.is_tradesignal(action):
+            if action == Actions.Buy.value:
+                if self._position == Positions.Short:
+                    self._update_total_profit()
+                self._position = Positions.Long
+                trade_type = "long"
+                self._last_trade_tick = self._current_tick
+            elif action == Actions.Sell.value and self.can_short:
+                if self._position == Positions.Long:
+                    self._update_total_profit()
+                self._position = Positions.Short
+                trade_type = "short"
+                self._last_trade_tick = self._current_tick
+            elif action == Actions.Sell.value and not self.can_short:
+                self._update_total_profit()
+                self._position = Positions.Neutral
+                trade_type = "neutral"
+                self._last_trade_tick = None
+            else:
+                print("case not defined")
+
+            if trade_type is not None:
+                self.trade_history.append(
+                    {'price': self.current_price(), 'index': self._current_tick,
+                     'type': trade_type})
+
+        if (self._total_profit < self.max_drawdown or
+                self._total_unrealized_profit < self.max_drawdown):
+            self._done = True
+
+        self._position_history.append(self._position)
+
+        info = dict(
+            tick=self._current_tick,
+            action=action,
+            total_reward=self.total_reward,
+            total_profit=self._total_profit,
+            position=self._position.value,
+            trade_duration=self.get_trade_duration(),
+            current_profit_pct=self.get_unrealized_profit()
+        )
+
+        observation = self._get_observation()
+
+        self._update_history(info)
+
+        return observation, step_reward, self._done, info
+
+    def is_tradesignal(self, action: int) -> bool:
+        """
+        Determine if the signal is a trade signal
+        e.g.: agent wants a Actions.Buy while it is in a Positions.short
+        """
+        return (
+            (action == Actions.Buy.value and self._position == Positions.Neutral)
+            or (action == Actions.Sell.value and self._position == Positions.Long)
+            or (action == Actions.Sell.value and self._position == Positions.Neutral
+                and self.can_short)
+            or (action == Actions.Buy.value and self._position == Positions.Short
+                and self.can_short)
+        )
+
+    def _is_valid(self, action: int) -> bool:
+        """
+        Determine if the signal is valid.
+        e.g.: agent wants a Actions.Sell while it is in a Positions.Long
+        """
+        if self.can_short:
+            return action in [Actions.Buy.value, Actions.Sell.value, Actions.Neutral.value]
+        else:
+            if action == Actions.Sell.value and self._position != Positions.Long:
+                return False
+            return True

freqtrade/freqai/RL/Base4ActionRLEnv.py

@@ -88,7 +88,8 @@ class Base4ActionRLEnv(BaseEnvironment):
                     {'price': self.current_price(), 'index': self._current_tick,
                      'type': trade_type})
 
-        if self._total_profit < 1 - self.rl_config.get('max_training_drawdown_pct', 0.8):
+        if (self._total_profit < self.max_drawdown or
+                self._total_unrealized_profit < self.max_drawdown):
             self._done = True
 
         self._position_history.append(self._position)

freqtrade/freqai/RL/BaseEnvironment.py

@@ -45,7 +45,7 @@ class BaseEnvironment(gym.Env):
     def __init__(self, df: DataFrame = DataFrame(), prices: DataFrame = DataFrame(),
                  reward_kwargs: dict = {}, window_size=10, starting_point=True,
                  id: str = 'baseenv-1', seed: int = 1, config: dict = {}, live: bool = False,
-                 fee: float = 0.0015):
+                 fee: float = 0.0015, can_short: bool = False):
         """
         Initializes the training/eval environment.
         :param df: dataframe of features
@@ -58,6 +58,7 @@ class BaseEnvironment(gym.Env):
         :param config: Typical user configuration file
         :param live: Whether or not this environment is active in dry/live/backtesting
         :param fee: The fee to use for environmental interactions.
+        :param can_short: Whether or not the environment can short
         """
         self.config = config
         self.rl_config = config['freqai']['rl_config']
@@ -73,6 +74,7 @@ class BaseEnvironment(gym.Env):
         # set here to default 5Ac, but all children envs can override this
         self.actions: Type[Enum] = BaseActions
         self.tensorboard_metrics: dict = {}
+        self.can_short = can_short
         self.live = live
         if not self.live and self.add_state_info:
             self.add_state_info = False

freqtrade/freqai/RL/BaseReinforcementLearningModel.py

@@ -165,7 +165,8 @@ class BaseReinforcementLearningModel(IFreqaiModel):
         env_info = {"window_size": self.CONV_WIDTH,
                     "reward_kwargs": self.reward_params,
                     "config": self.config,
-                    "live": self.live}
+                    "live": self.live,
+                    "can_short": self.can_short}
         if self.data_provider:
             env_info["fee"] = self.data_provider._exchange \
                 .get_fee(symbol=self.data_provider.current_whitelist()[0])  # type: ignore

freqtrade/freqai/freqai_interface.py

@@ -104,6 +104,7 @@ class IFreqaiModel(ABC):
         self.metadata: Dict[str, Any] = self.dd.load_global_metadata_from_disk()
         self.data_provider: Optional[DataProvider] = None
         self.max_system_threads = max(int(psutil.cpu_count() * 2 - 2), 1)
+        self.can_short = True  # overridden in start() with strategy.can_short
 
         record_params(config, self.full_path)
 
@@ -133,6 +134,7 @@ class IFreqaiModel(ABC):
         self.live = strategy.dp.runmode in (RunMode.DRY_RUN, RunMode.LIVE)
         self.dd.set_pair_dict_info(metadata)
         self.data_provider = strategy.dp
+        self.can_short = strategy.can_short
 
         if self.live:
             self.inference_timer('start')

freqtrade/freqtradebot.py

@@ -912,6 +912,7 @@ class FreqtradeBot(LoggingMixin):
             stake_amount=stake_amount,
             min_stake_amount=min_stake_amount,
             max_stake_amount=max_stake_amount,
+            trade_amount=trade.stake_amount if trade else None,
         )
 
         return enter_limit_requested, stake_amount, leverage

freqtrade/optimize/backtesting.py

@@ -769,6 +769,7 @@ class Backtesting:
             stake_amount=stake_amount,
             min_stake_amount=min_stake_amount,
             max_stake_amount=max_stake_amount,
+            trade_amount=trade.stake_amount if trade else None
         )
 
         return propose_rate, stake_amount_val, leverage, min_stake_amount

freqtrade/persistence/migrations.py

@@ -109,11 +109,10 @@ def migrate_trades_and_orders_table(
     else:
         is_short = get_column_def(cols, 'is_short', '0')
 
-    # Margin Properties
+    # Futures Properties
     interest_rate = get_column_def(cols, 'interest_rate', '0.0')
-
-    # Futures properties
     funding_fees = get_column_def(cols, 'funding_fees', '0.0')
+    max_stake_amount = get_column_def(cols, 'max_stake_amount', 'stake_amount')
 
     # If ticker-interval existed use that, else null.
     if has_column(cols, 'ticker_interval'):
@@ -162,7 +161,8 @@ def migrate_trades_and_orders_table(
             timeframe, open_trade_value, close_profit_abs,
             trading_mode, leverage, liquidation_price, is_short,
             interest_rate, funding_fees, realized_profit,
-            amount_precision, price_precision, precision_mode, contract_size
+            amount_precision, price_precision, precision_mode, contract_size,
+            max_stake_amount
             )
         select id, lower(exchange), pair, {base_currency} base_currency,
             {stake_currency} stake_currency,
@@ -190,7 +190,8 @@ def migrate_trades_and_orders_table(
             {is_short} is_short, {interest_rate} interest_rate,
             {funding_fees} funding_fees, {realized_profit} realized_profit,
             {amount_precision} amount_precision, {price_precision} price_precision,
-            {precision_mode} precision_mode, {contract_size} contract_size
+            {precision_mode} precision_mode, {contract_size} contract_size,
+            {max_stake_amount} max_stake_amount
             from {trade_back_name}
             """))
 
@@ -310,8 +311,8 @@ def check_migrate(engine, decl_base, previous_tables) -> None:
     # if ('orders' not in previous_tables
     # or not has_column(cols_orders, 'funding_fee')):
     migrating = False
-    # if not has_column(cols_trades, 'contract_size'):
-    if not has_column(cols_orders, 'funding_fee'):
+    # if not has_column(cols_orders, 'funding_fee'):
+    if not has_column(cols_trades, 'max_stake_amount'):
         migrating = True
         logger.info(f"Running database migration for trades - "
                     f"backup: {table_back_name}, {order_table_bak_name}")

freqtrade/persistence/trade_model.py

@@ -293,6 +293,7 @@ class LocalTrade():
     close_profit: Optional[float] = None
     close_profit_abs: Optional[float] = None
     stake_amount: float = 0.0
+    max_stake_amount: float = 0.0
     amount: float = 0.0
     amount_requested: Optional[float] = None
     open_date: datetime
@@ -397,12 +398,6 @@ class LocalTrade():
     def close_date_utc(self):
         return self.close_date.replace(tzinfo=timezone.utc)
 
-    @property
-    def enter_side(self) -> str:
-        """ DEPRECATED, please use entry_side instead"""
-        # TODO: Please remove me after 2022.5
-        return self.entry_side
-
     @property
     def entry_side(self) -> str:
         if self.is_short:
@@ -475,8 +470,8 @@ class LocalTrade():
             'amount': round(self.amount, 8),
             'amount_requested': round(self.amount_requested, 8) if self.amount_requested else None,
             'stake_amount': round(self.stake_amount, 8),
+            'max_stake_amount': round(self.max_stake_amount, 8) if self.max_stake_amount else None,
             'strategy': self.strategy,
-            'buy_tag': self.enter_tag,
             'enter_tag': self.enter_tag,
             'timeframe': self.timeframe,
 
@@ -513,7 +508,6 @@ class LocalTrade():
             'profit_pct': round(self.close_profit * 100, 2) if self.close_profit else None,
             'profit_abs': self.close_profit_abs,
 
-            'sell_reason': self.exit_reason,  # Deprecated
             'exit_reason': self.exit_reason,
             'exit_order_status': self.exit_order_status,
             'stop_loss_abs': self.stop_loss,
@@ -882,6 +876,7 @@ class LocalTrade():
         ZERO = FtPrecise(0.0)
         current_amount = FtPrecise(0.0)
         current_stake = FtPrecise(0.0)
+        max_stake_amount = FtPrecise(0.0)
         total_stake = 0.0  # Total stake after all buy orders (does not subtract!)
         avg_price = FtPrecise(0.0)
         close_profit = 0.0
@@ -923,7 +918,9 @@ class LocalTrade():
                     exit_rate, amount=exit_amount, open_rate=avg_price)
             else:
                 total_stake = total_stake + self._calc_open_trade_value(tmp_amount, price)
+                max_stake_amount += (tmp_amount * price)
         self.funding_fees = funding_fees
+        self.max_stake_amount = float(max_stake_amount)
 
         if close_profit:
             self.close_profit = close_profit
@@ -1175,6 +1172,7 @@ class Trade(_DECL_BASE, LocalTrade):
     close_profit = Column(Float)
     close_profit_abs = Column(Float)
     stake_amount = Column(Float, nullable=False)
+    max_stake_amount = Column(Float)
     amount = Column(Float)
     amount_requested = Column(Float)
     open_date = Column(DateTime, nullable=False, default=datetime.utcnow)

freqtrade/plugins/pairlist/RemotePairList.py

@@ -0,0 +1,206 @@
+"""
+Remote PairList provider
+
+Provides pair list fetched from a remote source
+"""
+import json
+import logging
+from pathlib import Path
+from typing import Any, Dict, List, Tuple
+
+import requests
+from cachetools import TTLCache
+
+from freqtrade import __version__
+from freqtrade.constants import Config
+from freqtrade.exceptions import OperationalException
+from freqtrade.exchange.types import Tickers
+from freqtrade.plugins.pairlist.IPairList import IPairList
+
+
+logger = logging.getLogger(__name__)
+
+
+class RemotePairList(IPairList):
+
+    def __init__(self, exchange, pairlistmanager,
+                 config: Config, pairlistconfig: Dict[str, Any],
+                 pairlist_pos: int) -> None:
+        super().__init__(exchange, pairlistmanager, config, pairlistconfig, pairlist_pos)
+
+        if 'number_assets' not in self._pairlistconfig:
+            raise OperationalException(
+                '`number_assets` not specified. Please check your configuration '
+                'for "pairlist.config.number_assets"')
+
+        if 'pairlist_url' not in self._pairlistconfig:
+            raise OperationalException(
+                '`pairlist_url` not specified. Please check your configuration '
+                'for "pairlist.config.pairlist_url"')
+
+        self._number_pairs = self._pairlistconfig['number_assets']
+        self._refresh_period: int = self._pairlistconfig.get('refresh_period', 1800)
+        self._keep_pairlist_on_failure = self._pairlistconfig.get('keep_pairlist_on_failure', True)
+        self._pair_cache: TTLCache = TTLCache(maxsize=1, ttl=self._refresh_period)
+        self._pairlist_url = self._pairlistconfig.get('pairlist_url', '')
+        self._read_timeout = self._pairlistconfig.get('read_timeout', 60)
+        self._bearer_token = self._pairlistconfig.get('bearer_token', '')
+        self._init_done = False
+        self._last_pairlist: List[Any] = list()
+
+    @property
+    def needstickers(self) -> bool:
+        """
+        Boolean property defining if tickers are necessary.
+        If no Pairlist requires tickers, an empty Dict is passed
+        as tickers argument to filter_pairlist
+        """
+        return False
+
+    def short_desc(self) -> str:
+        """
+        Short whitelist method description - used for startup-messages
+        """
+        return f"{self.name} - {self._pairlistconfig['number_assets']} pairs from RemotePairlist."
+
+    def process_json(self, jsonparse) -> List[str]:
+
+        pairlist = jsonparse.get('pairs', [])
+        remote_refresh_period = int(jsonparse.get('refresh_period', self._refresh_period))
+
+        if self._refresh_period < remote_refresh_period:
+            self.log_once(f'Refresh Period has been increased from {self._refresh_period}'
+                          f' to minimum allowed: {remote_refresh_period} from Remote.', logger.info)
+
+            self._refresh_period = remote_refresh_period
+            self._pair_cache = TTLCache(maxsize=1, ttl=remote_refresh_period)
+
+        self._init_done = True
+
+        return pairlist
+
+    def return_last_pairlist(self) -> List[str]:
+        if self._keep_pairlist_on_failure:
+            pairlist = self._last_pairlist
+            self.log_once('Keeping last fetched pairlist', logger.info)
+        else:
+            pairlist = []
+
+        return pairlist
+
+    def fetch_pairlist(self) -> Tuple[List[str], float]:
+
+        headers = {
+            'User-Agent': 'Freqtrade/' + __version__ + ' Remotepairlist'
+        }
+
+        if self._bearer_token:
+            headers['Authorization'] = f'Bearer {self._bearer_token}'
+
+        try:
+            response = requests.get(self._pairlist_url, headers=headers,
+                                    timeout=self._read_timeout)
+            content_type = response.headers.get('content-type')
+            time_elapsed = response.elapsed.total_seconds()
+
+            if "application/json" in str(content_type):
+                jsonparse = response.json()
+
+                try:
+                    pairlist = self.process_json(jsonparse)
+                except Exception as e:
+
+                    if self._init_done:
+                        pairlist = self.return_last_pairlist()
+                        logger.warning(f'Error while processing JSON data: {type(e)}')
+                    else:
+                        raise OperationalException(f'Error while processing JSON data: {type(e)}')
+
+            else:
+                if self._init_done:
+                    self.log_once(f'Error: RemotePairList is not of type JSON: '
+                                  f' {self._pairlist_url}', logger.info)
+                    pairlist = self.return_last_pairlist()
+                else:
+                    raise OperationalException('RemotePairList is not of type JSON, abort.')
+
+        except requests.exceptions.RequestException:
+            self.log_once(f'Was not able to fetch pairlist from:'
+                          f' {self._pairlist_url}', logger.info)
+
+            pairlist = self.return_last_pairlist()
+
+            time_elapsed = 0
+
+        return pairlist, time_elapsed
+
+    def gen_pairlist(self, tickers: Tickers) -> List[str]:
+        """
+        Generate the pairlist
+        :param tickers: Tickers (from exchange.get_tickers). May be cached.
+        :return: List of pairs
+        """
+
+        if self._init_done:
+            pairlist = self._pair_cache.get('pairlist')
+        else:
+            pairlist = []
+
+        time_elapsed = 0.0
+
+        if pairlist:
+            # Item found - no refresh necessary
+            return pairlist.copy()
+        else:
+            if self._pairlist_url.startswith("file:///"):
+                filename = self._pairlist_url.split("file:///", 1)[1]
+                file_path = Path(filename)
+
+                if file_path.exists():
+                    with open(filename) as json_file:
+                        # Load the JSON data into a dictionary
+                        jsonparse = json.load(json_file)
+
+                        try:
+                            pairlist = self.process_json(jsonparse)
+                        except Exception as e:
+                            if self._init_done:
+                                pairlist = self.return_last_pairlist()
+                                logger.warning(f'Error while processing JSON data: {type(e)}')
+                            else:
+                                raise OperationalException('Error while processing'
+                                                           f'JSON data: {type(e)}')
+                else:
+                    raise ValueError(f"{self._pairlist_url} does not exist.")
+            else:
+                # Fetch Pairlist from Remote URL
+                pairlist, time_elapsed = self.fetch_pairlist()
+
+        self.log_once(f"Fetched pairs: {pairlist}", logger.debug)
+
+        pairlist = self._whitelist_for_active_markets(pairlist)
+        pairlist = pairlist[:self._number_pairs]
+
+        self._pair_cache['pairlist'] = pairlist.copy()
+
+        if time_elapsed != 0.0:
+            self.log_once(f'Pairlist Fetched in {time_elapsed} seconds.', logger.info)
+        else:
+            self.log_once('Fetched Pairlist.', logger.info)
+
+        self._last_pairlist = list(pairlist)
+
+        return pairlist
+
+    def filter_pairlist(self, pairlist: List[str], tickers: Dict) -> List[str]:
+        """
+        Filters and sorts pairlist and returns the whitelist again.
+        Called on each bot iteration - please use internal caching if necessary
+        :param pairlist: pairlist to filter or sort
+        :param tickers: Tickers (from exchange.get_tickers). May be cached.
+        :return: new whitelist
+        """
+        rpl_pairlist = self.gen_pairlist(tickers)
+        merged_list = pairlist + rpl_pairlist
+        merged_list = sorted(set(merged_list), key=merged_list.index)
+        return merged_list

freqtrade/plugins/pairlist/VolumePairList.py

@@ -135,7 +135,7 @@ class VolumePairList(IPairList):
                 filtered_tickers = [
                     v for k, v in tickers.items()
                     if (self._exchange.get_pair_quote_currency(k) == self._stake_currency
-                        and (self._use_range or v[self._sort_key] is not None)
+                        and (self._use_range or v.get(self._sort_key) is not None)
                         and v['symbol'] in _pairlist)]
                 pairlist = [s['symbol'] for s in filtered_tickers]
             else:

freqtrade/rpc/api_server/api_backtest.py

@@ -11,6 +11,7 @@ from freqtrade.configuration.config_validation import validate_config_consistenc
 from freqtrade.data.btanalysis import get_backtest_resultlist, load_and_merge_backtest_result
 from freqtrade.enums import BacktestState
 from freqtrade.exceptions import DependencyException
+from freqtrade.misc import deep_merge_dicts
 from freqtrade.rpc.api_server.api_schemas import (BacktestHistoryEntry, BacktestRequest,
                                                   BacktestResponse)
 from freqtrade.rpc.api_server.deps import get_config, is_webserver_mode
@@ -37,10 +38,11 @@ async def api_start_backtest(bt_settings: BacktestRequest, background_tasks: Bac
 
     btconfig = deepcopy(config)
     settings = dict(bt_settings)
+    if settings.get('freqai', None) is not None:
+        settings['freqai'] = dict(settings['freqai'])
     # Pydantic models will contain all keys, but non-provided ones are None
-    for setting in settings.keys():
-        if settings[setting] is not None:
-            btconfig[setting] = settings[setting]
+
+    btconfig = deep_merge_dicts(settings, btconfig, allow_null_overrides=False)
     try:
         btconfig['stake_amount'] = float(btconfig['stake_amount'])
     except ValueError:

freqtrade/rpc/api_server/api_schemas.py

@@ -217,8 +217,8 @@ class TradeSchema(BaseModel):
     amount: float
     amount_requested: float
     stake_amount: float
+    max_stake_amount: Optional[float]
     strategy: str
-    buy_tag: Optional[str]  # Deprecated
     enter_tag: Optional[str]
     timeframe: int
     fee_open: Optional[float]
@@ -243,7 +243,6 @@ class TradeSchema(BaseModel):
     profit_pct: Optional[float]
     profit_abs: Optional[float]
     profit_fiat: Optional[float]
-    sell_reason: Optional[str]  # Deprecated
     exit_reason: Optional[str]
     exit_order_status: Optional[str]
     stop_loss_abs: Optional[float]
@@ -372,6 +371,10 @@ class StrategyListResponse(BaseModel):
     strategies: List[str]
 
 
+class FreqAIModelListResponse(BaseModel):
+    freqaimodels: List[str]
+
+
 class StrategyResponse(BaseModel):
     strategy: str
     code: str
@@ -410,6 +413,10 @@ class PairHistory(BaseModel):
         }
 
 
+class BacktestFreqAIInputs(BaseModel):
+    identifier: str
+
+
 class BacktestRequest(BaseModel):
     strategy: str
     timeframe: Optional[str]
@@ -419,6 +426,9 @@ class BacktestRequest(BaseModel):
     stake_amount: Optional[str]
     enable_protections: bool
     dry_run_wallet: Optional[float]
+    backtest_cache: Optional[str]
+    freqaimodel: Optional[str]
+    freqai: Optional[BacktestFreqAIInputs]
 
 
 class BacktestResponse(BaseModel):

freqtrade/rpc/api_server/api_v1.py

@@ -13,12 +13,13 @@ from freqtrade.rpc import RPC
 from freqtrade.rpc.api_server.api_schemas import (AvailablePairs, Balances, BlacklistPayload,
                                                   BlacklistResponse, Count, Daily,
                                                   DeleteLockRequest, DeleteTrade, ForceEnterPayload,
-                                                  ForceEnterResponse, ForceExitPayload, Health,
-                                                  Locks, Logs, OpenTradeSchema, PairHistory,
-                                                  PerformanceEntry, Ping, PlotConfig, Profit,
-                                                  ResultMsg, ShowConfig, Stats, StatusMsg,
-                                                  StrategyListResponse, StrategyResponse, SysInfo,
-                                                  Version, WhitelistResponse)
+                                                  ForceEnterResponse, ForceExitPayload,
+                                                  FreqAIModelListResponse, Health, Locks, Logs,
+                                                  OpenTradeSchema, PairHistory, PerformanceEntry,
+                                                  Ping, PlotConfig, Profit, ResultMsg, ShowConfig,
+                                                  Stats, StatusMsg, StrategyListResponse,
+                                                  StrategyResponse, SysInfo, Version,
+                                                  WhitelistResponse)
 from freqtrade.rpc.api_server.deps import get_config, get_exchange, get_rpc, get_rpc_optional
 from freqtrade.rpc.rpc import RPCException
 
@@ -38,7 +39,8 @@ logger = logging.getLogger(__name__)
 # 2.17: Forceentry - leverage, partial force_exit
 # 2.20: Add websocket endpoints
 # 2.21: Add new_candle messagetype
-API_VERSION = 2.21
+# 2.22: Add FreqAI to backtesting
+API_VERSION = 2.22
 
 # Public API, requires no auth.
 router_public = APIRouter()
@@ -279,6 +281,16 @@ def get_strategy(strategy: str, config=Depends(get_config)):
     }
 
 
+@router.get('/freqaimodels', response_model=FreqAIModelListResponse, tags=['freqai'])
+def list_freqaimodels(config=Depends(get_config)):
+    from freqtrade.resolvers.freqaimodel_resolver import FreqaiModelResolver
+    strategies = FreqaiModelResolver.search_all_objects(
+        config, False)
+    strategies = sorted(strategies, key=lambda x: x['name'])
+
+    return {'freqaimodels': [x['name'] for x in strategies]}
+
+
 @router.get('/available_pairs', response_model=AvailablePairs, tags=['candle data'])
 def list_available_pairs(timeframe: Optional[str] = None, stake_currency: Optional[str] = None,
                          candletype: Optional[CandleType] = None, config=Depends(get_config)):

freqtrade/wallets.py

@@ -291,12 +291,17 @@ class Wallets:
         return self._check_available_stake_amount(stake_amount, available_amount)
 
     def validate_stake_amount(self, pair: str, stake_amount: Optional[float],
-                              min_stake_amount: Optional[float], max_stake_amount: float):
+                              min_stake_amount: Optional[float], max_stake_amount: float,
+                              trade_amount: Optional[float]):
         if not stake_amount:
             logger.debug(f"Stake amount is {stake_amount}, ignoring possible trade for {pair}.")
             return 0
 
         max_stake_amount = min(max_stake_amount, self.get_available_stake_amount())
+        if trade_amount:
+            # if in a trade, then the resulting trade size cannot go beyond the max stake
+            # Otherwise we could no longer exit.
+            max_stake_amount = min(max_stake_amount, max_stake_amount - trade_amount)
 
         if min_stake_amount is not None and min_stake_amount > max_stake_amount:
             if self._log:

mkdocs.yml

@@ -41,6 +41,7 @@ nav:
         - Backtest analysis: advanced-backtesting.md
     - Advanced Topics:
         - Advanced Post-installation Tasks: advanced-setup.md
+        - Trade Object: trade-object.md
         - Advanced Strategy: strategy-advanced.md
         - Advanced Hyperopt: advanced-hyperopt.md
         - Producer/Consumer mode: producer-consumer.md

requirements-dev.txt

@@ -10,24 +10,24 @@ coveralls==3.3.1
 flake8==6.0.0
 flake8-tidy-imports==4.8.0
 mypy==0.991
-pre-commit==2.20.0
+pre-commit==2.21.0
 pytest==7.2.0
 pytest-asyncio==0.20.3
 pytest-cov==4.0.0
 pytest-mock==3.10.0
 pytest-random-order==1.1.0
-isort==5.10.1
+isort==5.11.4
 # For datetime mocking
 time-machine==2.8.2
 # fastapi testing
 httpx==0.23.1
 
 # Convert jupyter notebooks to markdown documents
-nbconvert==7.2.6
+nbconvert==7.2.7
 
 # mypy types
 types-cachetools==5.2.1
 types-filelock==3.2.7
-types-requests==2.28.11.5
+types-requests==2.28.11.7
 types-tabulate==0.9.0.0
-types-python-dateutil==2.8.19.4
+types-python-dateutil==2.8.19.5

requirements-freqai-rl.txt

@@ -2,7 +2,7 @@
 -r requirements-freqai.txt
 
 # Required for freqai-rl
-torch==1.13.0
+torch==1.13.1
 stable-baselines3==1.6.2
 sb3-contrib==1.6.2
 # Gym is forced to this version by stable-baselines3.

requirements.txt

@@ -1,8 +1,8 @@
-numpy==1.23.5
+numpy==1.24.1
 pandas==1.5.2
 pandas-ta==0.3.14b
 
-ccxt==2.2.92
+ccxt==2.4.60
 # Pin cryptography for now due to rust build errors with piwheels
 cryptography==38.0.1; platform_machine == 'armv7l'
 cryptography==38.0.4; platform_machine != 'armv7l'
@@ -20,8 +20,7 @@ tabulate==0.9.0
 pycoingecko==3.1.0
 jinja2==3.1.2
 tables==3.7.0
-blosc==1.10.6; platform_machine == 'arm64'
-blosc==1.11.0; platform_machine != 'arm64'
+blosc==1.11.1
 joblib==1.2.0
 pyarrow==10.0.1; platform_machine != 'armv7l'
 

tests/commands/test_commands.py

@@ -1529,7 +1529,7 @@ def test_backtesting_show(mocker, testdatadir, capsys):
     args = [
         "backtesting-show",
         "--export-filename",
-        f"{testdatadir / 'backtest_results/backtest-result_new.json'}",
+        f"{testdatadir / 'backtest_results/backtest-result.json'}",
         "--show-pair-list"
     ]
     pargs = get_args(args)

tests/data/test_btanalysis.py

@@ -30,10 +30,10 @@ def test_get_latest_backtest_filename(testdatadir, mocker):
 
     testdir_bt = testdatadir / "backtest_results"
     res = get_latest_backtest_filename(testdir_bt)
-    assert res == 'backtest-result_new.json'
+    assert res == 'backtest-result.json'
 
     res = get_latest_backtest_filename(str(testdir_bt))
-    assert res == 'backtest-result_new.json'
+    assert res == 'backtest-result.json'
 
     mocker.patch("freqtrade.data.btanalysis.json_load", return_value={})
 
@@ -81,7 +81,7 @@ def test_load_backtest_data_old_format(testdatadir, mocker):
 
 def test_load_backtest_data_new_format(testdatadir):
 
-    filename = testdatadir / "backtest_results/backtest-result_new.json"
+    filename = testdatadir / "backtest_results/backtest-result.json"
     bt_data = load_backtest_data(filename)
     assert isinstance(bt_data, DataFrame)
     assert set(bt_data.columns) == set(BT_DATA_COLUMNS)
@@ -182,7 +182,7 @@ def test_extract_trades_of_period(testdatadir):
 
 
 def test_analyze_trade_parallelism(testdatadir):
-    filename = testdatadir / "backtest_results/backtest-result_new.json"
+    filename = testdatadir / "backtest_results/backtest-result.json"
     bt_data = load_backtest_data(filename)
 
     res = analyze_trade_parallelism(bt_data, "5m")
@@ -256,7 +256,7 @@ def test_combine_dataframes_with_mean_no_data(testdatadir):
 
 
 def test_create_cum_profit(testdatadir):
-    filename = testdatadir / "backtest_results/backtest-result_new.json"
+    filename = testdatadir / "backtest_results/backtest-result.json"
     bt_data = load_backtest_data(filename)
     timerange = TimeRange.parse_timerange("20180110-20180112")
 
@@ -268,11 +268,11 @@ def test_create_cum_profit(testdatadir):
                                     "cum_profits", timeframe="5m")
     assert "cum_profits" in cum_profits.columns
     assert cum_profits.iloc[0]['cum_profits'] == 0
-    assert pytest.approx(cum_profits.iloc[-1]['cum_profits']) == 8.723007518796964e-06
+    assert pytest.approx(cum_profits.iloc[-1]['cum_profits']) == 9.0225563e-05
 
 
 def test_create_cum_profit1(testdatadir):
-    filename = testdatadir / "backtest_results/backtest-result_new.json"
+    filename = testdatadir / "backtest_results/backtest-result.json"
     bt_data = load_backtest_data(filename)
     # Move close-time to "off" the candle, to make sure the logic still works
     bt_data['close_date'] = bt_data.loc[:, 'close_date'] + DateOffset(seconds=20)
@@ -286,7 +286,7 @@ def test_create_cum_profit1(testdatadir):
                                     "cum_profits", timeframe="5m")
     assert "cum_profits" in cum_profits.columns
     assert cum_profits.iloc[0]['cum_profits'] == 0
-    assert pytest.approx(cum_profits.iloc[-1]['cum_profits']) == 8.723007518796964e-06
+    assert pytest.approx(cum_profits.iloc[-1]['cum_profits']) == 9.0225563e-05
 
     with pytest.raises(ValueError, match='Trade dataframe empty.'):
         create_cum_profit(df.set_index('date'), bt_data[bt_data["pair"] == 'NOTAPAIR'],
@@ -294,18 +294,18 @@ def test_create_cum_profit1(testdatadir):
 
 
 def test_calculate_max_drawdown(testdatadir):
-    filename = testdatadir / "backtest_results/backtest-result_new.json"
+    filename = testdatadir / "backtest_results/backtest-result.json"
     bt_data = load_backtest_data(filename)
     _, hdate, lowdate, hval, lval, drawdown = calculate_max_drawdown(
         bt_data, value_col="profit_abs")
     assert isinstance(drawdown, float)
-    assert pytest.approx(drawdown) == 0.12071099
+    assert pytest.approx(drawdown) == 0.29753914
     assert isinstance(hdate, Timestamp)
     assert isinstance(lowdate, Timestamp)
     assert isinstance(hval, float)
     assert isinstance(lval, float)
-    assert hdate == Timestamp('2018-01-25 01:30:00', tz='UTC')
-    assert lowdate == Timestamp('2018-01-25 03:50:00', tz='UTC')
+    assert hdate == Timestamp('2018-01-16 19:30:00', tz='UTC')
+    assert lowdate == Timestamp('2018-01-16 22:25:00', tz='UTC')
 
     underwater = calculate_underwater(bt_data)
     assert isinstance(underwater, DataFrame)
@@ -318,14 +318,15 @@ def test_calculate_max_drawdown(testdatadir):
 
 
 def test_calculate_csum(testdatadir):
-    filename = testdatadir / "backtest_results/backtest-result_new.json"
+    filename = testdatadir / "backtest_results/backtest-result.json"
     bt_data = load_backtest_data(filename)
     csum_min, csum_max = calculate_csum(bt_data)
 
     assert isinstance(csum_min, float)
     assert isinstance(csum_max, float)
-    assert csum_min < 0.01
-    assert csum_max > 0.02
+    assert csum_min < csum_max
+    assert csum_min < 0.0001
+    assert csum_max > 0.0002
     csum_min1, csum_max1 = calculate_csum(bt_data, 5)
 
     assert csum_min1 == csum_min + 5

tests/exchange/test_binance.py

@@ -23,7 +23,7 @@ from tests.exchange.test_exchange import ccxt_exceptionhandlers
 def test_stoploss_order_binance(default_conf, mocker, limitratio, expected, side, trademode):
     api_mock = MagicMock()
     order_id = 'test_prod_buy_{}'.format(randint(0, 10 ** 6))
-    order_type = 'stop_loss_limit' if trademode == TradingMode.SPOT else 'limit'
+    order_type = 'stop_loss_limit' if trademode == TradingMode.SPOT else 'stop'
 
     api_mock.create_order = MagicMock(return_value={
         'id': order_id,

tests/freqai/test_freqai_interface.py

@@ -27,20 +27,23 @@ def is_mac() -> bool:
     return "Darwin" in machine
 
 
-@pytest.mark.parametrize('model, pca, dbscan, float32', [
-    ('LightGBMRegressor', True, False, True),
-    ('XGBoostRegressor', False, True, False),
-    ('XGBoostRFRegressor', False, False, False),
-    ('CatboostRegressor', False, False, False),
-    ('ReinforcementLearner', False, True, False),
-    ('ReinforcementLearner_multiproc', False, False, False),
-    ('ReinforcementLearner_test_4ac', False, False, False)
+@pytest.mark.parametrize('model, pca, dbscan, float32, can_short', [
+    ('LightGBMRegressor', True, False, True, True),
+    ('XGBoostRegressor', False, True, False, True),
+    ('XGBoostRFRegressor', False, False, False, True),
+    ('CatboostRegressor', False, False, False, True),
+    ('ReinforcementLearner', False, True, False, True),
+    ('ReinforcementLearner_multiproc', False, False, False, True),
+    ('ReinforcementLearner_test_3ac', False, False, False, False),
+    ('ReinforcementLearner_test_3ac', False, False, False, True),
+    ('ReinforcementLearner_test_4ac', False, False, False, True)
     ])
-def test_extract_data_and_train_model_Standard(mocker, freqai_conf, model, pca, dbscan, float32):
+def test_extract_data_and_train_model_Standard(mocker, freqai_conf, model, pca,
+                                               dbscan, float32, can_short):
     if is_arm() and model == 'CatboostRegressor':
         pytest.skip("CatBoost is not supported on ARM")
 
-    if is_mac() and 'Reinforcement' in model:
+    if is_mac() and not is_arm() and 'Reinforcement' in model:
         pytest.skip("Reinforcement learning module not available on intel based Mac OS")
 
     model_save_ext = 'joblib'
@@ -58,9 +61,6 @@ def test_extract_data_and_train_model_Standard(mocker, freqai_conf, model, pca,
         freqai_conf['freqai']['feature_parameters'].update({"use_SVM_to_remove_outliers": True})
         freqai_conf['freqai']['data_split_parameters'].update({'shuffle': True})
 
-    if 'test_4ac' in model:
-        freqai_conf["freqaimodel_path"] = str(Path(__file__).parents[1] / "freqai" / "test_models")
-
     if 'ReinforcementLearner' in model:
         model_save_ext = 'zip'
         freqai_conf = make_rl_config(freqai_conf)
@@ -68,7 +68,7 @@ def test_extract_data_and_train_model_Standard(mocker, freqai_conf, model, pca,
         freqai_conf['freqai']['feature_parameters'].update({"use_SVM_to_remove_outliers": True})
         freqai_conf['freqai']['data_split_parameters'].update({'shuffle': True})
 
-    if 'test_4ac' in model:
+    if 'test_3ac' in model or 'test_4ac' in model:
         freqai_conf["freqaimodel_path"] = str(Path(__file__).parents[1] / "freqai" / "test_models")
 
     strategy = get_patched_freqai_strategy(mocker, freqai_conf)
@@ -77,6 +77,7 @@ def test_extract_data_and_train_model_Standard(mocker, freqai_conf, model, pca,
     strategy.freqai_info = freqai_conf.get("freqai", {})
     freqai = strategy.freqai
     freqai.live = True
+    freqai.can_short = can_short
     freqai.dk = FreqaiDataKitchen(freqai_conf)
     freqai.dk.set_paths('ADA/BTC', 10000)
     timerange = TimeRange.parse_timerange("20180110-20180130")

tests/freqai/test_models/ReinforcementLearner_test_3ac.py

@@ -0,0 +1,65 @@
+import logging
+
+import numpy as np
+
+from freqtrade.freqai.prediction_models.ReinforcementLearner import ReinforcementLearner
+from freqtrade.freqai.RL.Base3ActionRLEnv import Actions, Base3ActionRLEnv, Positions
+
+
+logger = logging.getLogger(__name__)
+
+
+class ReinforcementLearner_test_3ac(ReinforcementLearner):
+    """
+    User created Reinforcement Learning Model prediction model.
+    """
+
+    class MyRLEnv(Base3ActionRLEnv):
+        """
+        User can override any function in BaseRLEnv and gym.Env. Here the user
+        sets a custom reward based on profit and trade duration.
+        """
+
+        def calculate_reward(self, action: int) -> float:
+
+            # first, penalize if the action is not valid
+            if not self._is_valid(action):
+                return -2
+
+            pnl = self.get_unrealized_profit()
+            rew = np.sign(pnl) * (pnl + 1)
+            factor = 100.
+
+            # reward agent for entering trades
+            if (action in (Actions.Buy.value, Actions.Sell.value)
+                    and self._position == Positions.Neutral):
+                return 25
+            # discourage agent from not entering trades
+            if action == Actions.Neutral.value and self._position == Positions.Neutral:
+                return -1
+
+            max_trade_duration = self.rl_config.get('max_trade_duration_candles', 300)
+            trade_duration = self._current_tick - self._last_trade_tick  # type: ignore
+
+            if trade_duration <= max_trade_duration:
+                factor *= 1.5
+            elif trade_duration > max_trade_duration:
+                factor *= 0.5
+
+            # discourage sitting in position
+            if self._position in (Positions.Short, Positions.Long) and (
+                action == Actions.Neutral.value
+                or (action == Actions.Sell.value and self._position == Positions.Short)
+                or (action == Actions.Buy.value and self._position == Positions.Long)
+            ):
+                return -1 * trade_duration / max_trade_duration
+
+            # close position
+            if (action == Actions.Buy.value and self._position == Positions.Short) or (
+                action == Actions.Sell.value and self._position == Positions.Long
+            ):
+                if pnl > self.profit_aim * self.rr:
+                    factor *= self.rl_config["model_reward_parameters"].get("win_reward_factor", 2)
+                return float(rew * factor)
+
+            return 0.

tests/optimize/test_backtesting.py

@@ -710,6 +710,7 @@ def test_backtest_one(default_conf, fee, mocker, testdatadir) -> None:
     expected = pd.DataFrame(
         {'pair': [pair, pair],
          'stake_amount': [0.001, 0.001],
+         'max_stake_amount': [0.001, 0.001],
          'amount': [0.00957442, 0.0097064],
          'open_date': pd.to_datetime([Arrow(2018, 1, 29, 18, 40, 0).datetime,
                                       Arrow(2018, 1, 30, 3, 30, 0).datetime], utc=True

tests/optimize/test_backtesting_adjust_position.py

@@ -50,6 +50,7 @@ def test_backtest_position_adjustment(default_conf, fee, mocker, testdatadir) ->
     expected = pd.DataFrame(
         {'pair': [pair, pair],
          'stake_amount': [500.0, 100.0],
+         'max_stake_amount': [500.0, 100],
          'amount': [4806.87657523, 970.63960782],
          'open_date': pd.to_datetime([Arrow(2018, 1, 29, 18, 40, 0).datetime,
                                       Arrow(2018, 1, 30, 3, 30, 0).datetime], utc=True

tests/optimize/test_optimize_reports.py

@@ -308,7 +308,7 @@ def test_generate_pair_metrics():
 
 def test_generate_daily_stats(testdatadir):
 
-    filename = testdatadir / "backtest_results/backtest-result_new.json"
+    filename = testdatadir / "backtest_results/backtest-result.json"
     bt_data = load_backtest_data(filename)
     res = generate_daily_stats(bt_data)
     assert isinstance(res, dict)
@@ -328,7 +328,7 @@ def test_generate_daily_stats(testdatadir):
 
 
 def test_generate_trading_stats(testdatadir):
-    filename = testdatadir / "backtest_results/backtest-result_new.json"
+    filename = testdatadir / "backtest_results/backtest-result.json"
     bt_data = load_backtest_data(filename)
     res = generate_trading_stats(bt_data)
     assert isinstance(res, dict)
@@ -444,7 +444,7 @@ def test_generate_edge_table():
 
 
 def test_generate_periodic_breakdown_stats(testdatadir):
-    filename = testdatadir / "backtest_results/backtest-result_new.json"
+    filename = testdatadir / "backtest_results/backtest-result.json"
     bt_data = load_backtest_data(filename).to_dict(orient='records')
 
     res = generate_periodic_breakdown_stats(bt_data, 'day')
@@ -472,7 +472,7 @@ def test__get_resample_from_period():
 
 
 def test_show_sorted_pairlist(testdatadir, default_conf, capsys):
-    filename = testdatadir / "backtest_results/backtest-result_new.json"
+    filename = testdatadir / "backtest_results/backtest-result.json"
     bt_data = load_backtest_stats(filename)
     default_conf['backtest_show_pair_list'] = True
 

tests/persistence/test_migrations.py

@@ -0,0 +1,412 @@
+# pragma pylint: disable=missing-docstring, C0103
+import logging
+from pathlib import Path
+from unittest.mock import MagicMock
+
+import pytest
+from sqlalchemy import create_engine, text
+
+from freqtrade.constants import DEFAULT_DB_PROD_URL
+from freqtrade.enums import TradingMode
+from freqtrade.exceptions import OperationalException
+from freqtrade.persistence import Trade, init_db
+from freqtrade.persistence.migrations import get_last_sequence_ids, set_sequence_ids
+from freqtrade.persistence.models import PairLock
+from tests.conftest import log_has
+
+
+spot, margin, futures = TradingMode.SPOT, TradingMode.MARGIN, TradingMode.FUTURES
+
+
+def test_init_create_session(default_conf):
+    # Check if init create a session
+    init_db(default_conf['db_url'])
+    assert hasattr(Trade, '_session')
+    assert 'scoped_session' in type(Trade._session).__name__
+
+
+def test_init_custom_db_url(default_conf, tmpdir):
+    # Update path to a value other than default, but still in-memory
+    filename = f"{tmpdir}/freqtrade2_test.sqlite"
+    assert not Path(filename).is_file()
+
+    default_conf.update({'db_url': f'sqlite:///{filename}'})
+
+    init_db(default_conf['db_url'])
+    assert Path(filename).is_file()
+    r = Trade._session.execute(text("PRAGMA journal_mode"))
+    assert r.first() == ('wal',)
+
+
+def test_init_invalid_db_url():
+    # Update path to a value other than default, but still in-memory
+    with pytest.raises(OperationalException, match=r'.*no valid database URL*'):
+        init_db('unknown:///some.url')
+
+    with pytest.raises(OperationalException, match=r'Bad db-url.*For in-memory database, pl.*'):
+        init_db('sqlite:///')
+
+
+def test_init_prod_db(default_conf, mocker):
+    default_conf.update({'dry_run': False})
+    default_conf.update({'db_url': DEFAULT_DB_PROD_URL})
+
+    create_engine_mock = mocker.patch('freqtrade.persistence.models.create_engine', MagicMock())
+
+    init_db(default_conf['db_url'])
+    assert create_engine_mock.call_count == 1
+    assert create_engine_mock.mock_calls[0][1][0] == 'sqlite:///tradesv3.sqlite'
+
+
+def test_init_dryrun_db(default_conf, tmpdir):
+    filename = f"{tmpdir}/freqtrade2_prod.sqlite"
+    assert not Path(filename).is_file()
+    default_conf.update({
+        'dry_run': True,
+        'db_url': f'sqlite:///{filename}'
+    })
+
+    init_db(default_conf['db_url'])
+    assert Path(filename).is_file()
+
+
+def test_migrate_new(mocker, default_conf, fee, caplog):
+    """
+    Test Database migration (starting with new pairformat)
+    """
+    caplog.set_level(logging.DEBUG)
+    amount = 103.223
+    # Always create all columns apart from the last!
+    create_table_old = """CREATE TABLE IF NOT EXISTS "trades" (
+                                id INTEGER NOT NULL,
+                                exchange VARCHAR NOT NULL,
+                                pair VARCHAR NOT NULL,
+                                is_open BOOLEAN NOT NULL,
+                                fee FLOAT NOT NULL,
+                                open_rate FLOAT,
+                                close_rate FLOAT,
+                                close_profit FLOAT,
+                                stake_amount FLOAT NOT NULL,
+                                amount FLOAT,
+                                open_date DATETIME NOT NULL,
+                                close_date DATETIME,
+                                open_order_id VARCHAR,
+                                stop_loss FLOAT,
+                                initial_stop_loss FLOAT,
+                                max_rate FLOAT,
+                                sell_reason VARCHAR,
+                                strategy VARCHAR,
+                                ticker_interval INTEGER,
+                                stoploss_order_id VARCHAR,
+                                PRIMARY KEY (id),
+                                CHECK (is_open IN (0, 1))
+                                );"""
+    create_table_order = """CREATE TABLE orders (
+                                id INTEGER NOT NULL,
+                                ft_trade_id INTEGER,
+                                ft_order_side VARCHAR(25) NOT NULL,
+                                ft_pair VARCHAR(25) NOT NULL,
+                                ft_is_open BOOLEAN NOT NULL,
+                                order_id VARCHAR(255) NOT NULL,
+                                status VARCHAR(255),
+                                symbol VARCHAR(25),
+                                order_type VARCHAR(50),
+                                side VARCHAR(25),
+                                price FLOAT,
+                                amount FLOAT,
+                                filled FLOAT,
+                                remaining FLOAT,
+                                cost FLOAT,
+                                order_date DATETIME,
+                                order_filled_date DATETIME,
+                                order_update_date DATETIME,
+                                PRIMARY KEY (id)
+                            );"""
+    insert_table_old = """INSERT INTO trades (exchange, pair, is_open, fee,
+                          open_rate, stake_amount, amount, open_date,
+                          stop_loss, initial_stop_loss, max_rate, ticker_interval,
+                          open_order_id, stoploss_order_id)
+                          VALUES ('binance', 'ETC/BTC', 1, {fee},
+                          0.00258580, {stake}, {amount},
+                          '2019-11-28 12:44:24.000000',
+                          0.0, 0.0, 0.0, '5m',
+                          'buy_order', 'dry_stop_order_id222')
+                          """.format(fee=fee.return_value,
+                                     stake=default_conf.get("stake_amount"),
+                                     amount=amount
+                                     )
+    insert_orders = f"""
+        insert into orders (
+            ft_trade_id,
+            ft_order_side,
+            ft_pair,
+            ft_is_open,
+            order_id,
+            status,
+            symbol,
+            order_type,
+            side,
+            price,
+            amount,
+            filled,
+            remaining,
+            cost)
+        values (
+            1,
+            'buy',
+            'ETC/BTC',
+            0,
+            'dry_buy_order',
+            'closed',
+            'ETC/BTC',
+            'limit',
+            'buy',
+            0.00258580,
+            {amount},
+            {amount},
+            0,
+            {amount * 0.00258580}
+        ),
+        (
+            1,
+            'buy',
+            'ETC/BTC',
+            1,
+            'dry_buy_order22',
+            'canceled',
+            'ETC/BTC',
+            'limit',
+            'buy',
+            0.00258580,
+            {amount},
+            {amount},
+            0,
+            {amount * 0.00258580}
+        ),
+         (
+            1,
+            'stoploss',
+            'ETC/BTC',
+            1,
+            'dry_stop_order_id11X',
+            'canceled',
+            'ETC/BTC',
+            'limit',
+            'sell',
+            0.00258580,
+            {amount},
+            {amount},
+            0,
+            {amount * 0.00258580}
+        ),
+        (
+            1,
+            'stoploss',
+            'ETC/BTC',
+            1,
+            'dry_stop_order_id222',
+            'open',
+            'ETC/BTC',
+            'limit',
+            'sell',
+            0.00258580,
+            {amount},
+            {amount},
+            0,
+            {amount * 0.00258580}
+        )
+    """
+    engine = create_engine('sqlite://')
+    mocker.patch('freqtrade.persistence.models.create_engine', lambda *args, **kwargs: engine)
+
+    # Create table using the old format
+    with engine.begin() as connection:
+        connection.execute(text(create_table_old))
+        connection.execute(text(create_table_order))
+        connection.execute(text("create index ix_trades_is_open on trades(is_open)"))
+        connection.execute(text("create index ix_trades_pair on trades(pair)"))
+        connection.execute(text(insert_table_old))
+        connection.execute(text(insert_orders))
+
+        # fake previous backup
+        connection.execute(text("create table trades_bak as select * from trades"))
+
+        connection.execute(text("create table trades_bak1 as select * from trades"))
+    # Run init to test migration
+    init_db(default_conf['db_url'])
+
+    assert len(Trade.query.filter(Trade.id == 1).all()) == 1
+    trade = Trade.query.filter(Trade.id == 1).first()
+    assert trade.fee_open == fee.return_value
+    assert trade.fee_close == fee.return_value
+    assert trade.open_rate_requested is None
+    assert trade.close_rate_requested is None
+    assert trade.is_open == 1
+    assert trade.amount == amount
+    assert trade.amount_requested == amount
+    assert trade.stake_amount == default_conf.get("stake_amount")
+    assert trade.pair == "ETC/BTC"
+    assert trade.exchange == "binance"
+    assert trade.max_rate == 0.0
+    assert trade.min_rate is None
+    assert trade.stop_loss == 0.0
+    assert trade.initial_stop_loss == 0.0
+    assert trade.exit_reason is None
+    assert trade.strategy is None
+    assert trade.timeframe == '5m'
+    assert trade.stoploss_order_id == 'dry_stop_order_id222'
+    assert trade.stoploss_last_update is None
+    assert log_has("trying trades_bak1", caplog)
+    assert log_has("trying trades_bak2", caplog)
+    assert log_has("Running database migration for trades - backup: trades_bak2, orders_bak0",
+                   caplog)
+    assert log_has("Database migration finished.", caplog)
+    assert pytest.approx(trade.open_trade_value) == trade._calc_open_trade_value(
+        trade.amount, trade.open_rate)
+    assert trade.close_profit_abs is None
+    assert trade.stake_amount == trade.max_stake_amount
+
+    orders = trade.orders
+    assert len(orders) == 4
+    assert orders[0].order_id == 'dry_buy_order'
+    assert orders[0].ft_order_side == 'buy'
+
+    assert orders[-1].order_id == 'dry_stop_order_id222'
+    assert orders[-1].ft_order_side == 'stoploss'
+    assert orders[-1].ft_is_open is True
+
+    assert orders[1].order_id == 'dry_buy_order22'
+    assert orders[1].ft_order_side == 'buy'
+    assert orders[1].ft_is_open is False
+
+    assert orders[2].order_id == 'dry_stop_order_id11X'
+    assert orders[2].ft_order_side == 'stoploss'
+    assert orders[2].ft_is_open is False
+
+
+def test_migrate_too_old(mocker, default_conf, fee, caplog):
+    """
+    Test Database migration (starting with new pairformat)
+    """
+    caplog.set_level(logging.DEBUG)
+    amount = 103.223
+    create_table_old = """CREATE TABLE IF NOT EXISTS "trades" (
+                                id INTEGER NOT NULL,
+                                exchange VARCHAR NOT NULL,
+                                pair VARCHAR NOT NULL,
+                                is_open BOOLEAN NOT NULL,
+                                fee_open FLOAT NOT NULL,
+                                fee_close FLOAT NOT NULL,
+                                open_rate FLOAT,
+                                close_rate FLOAT,
+                                close_profit FLOAT,
+                                stake_amount FLOAT NOT NULL,
+                                amount FLOAT,
+                                open_date DATETIME NOT NULL,
+                                close_date DATETIME,
+                                open_order_id VARCHAR,
+                                PRIMARY KEY (id),
+                                CHECK (is_open IN (0, 1))
+                                );"""
+
+    insert_table_old = """INSERT INTO trades (exchange, pair, is_open, fee_open, fee_close,
+                          open_rate, stake_amount, amount, open_date)
+                          VALUES ('binance', 'ETC/BTC', 1, {fee}, {fee},
+                          0.00258580, {stake}, {amount},
+                          '2019-11-28 12:44:24.000000')
+                          """.format(fee=fee.return_value,
+                                     stake=default_conf.get("stake_amount"),
+                                     amount=amount
+                                     )
+    engine = create_engine('sqlite://')
+    mocker.patch('freqtrade.persistence.models.create_engine', lambda *args, **kwargs: engine)
+
+    # Create table using the old format
+    with engine.begin() as connection:
+        connection.execute(text(create_table_old))
+        connection.execute(text(insert_table_old))
+
+    # Run init to test migration
+    with pytest.raises(OperationalException, match=r'Your database seems to be very old'):
+        init_db(default_conf['db_url'])
+
+
+def test_migrate_get_last_sequence_ids():
+    engine = MagicMock()
+    engine.begin = MagicMock()
+    engine.name = 'postgresql'
+    get_last_sequence_ids(engine, 'trades_bak', 'orders_bak')
+
+    assert engine.begin.call_count == 2
+    engine.reset_mock()
+    engine.begin.reset_mock()
+
+    engine.name = 'somethingelse'
+    get_last_sequence_ids(engine, 'trades_bak', 'orders_bak')
+
+    assert engine.begin.call_count == 0
+
+
+def test_migrate_set_sequence_ids():
+    engine = MagicMock()
+    engine.begin = MagicMock()
+    engine.name = 'postgresql'
+    set_sequence_ids(engine, 22, 55, 5)
+
+    assert engine.begin.call_count == 1
+    engine.reset_mock()
+    engine.begin.reset_mock()
+
+    engine.name = 'somethingelse'
+    set_sequence_ids(engine, 22, 55, 6)
+
+    assert engine.begin.call_count == 0
+
+
+def test_migrate_pairlocks(mocker, default_conf, fee, caplog):
+    """
+    Test Database migration (starting with new pairformat)
+    """
+    caplog.set_level(logging.DEBUG)
+    # Always create all columns apart from the last!
+    create_table_old = """CREATE TABLE pairlocks (
+                            id INTEGER NOT NULL,
+                            pair VARCHAR(25) NOT NULL,
+                            reason VARCHAR(255),
+                            lock_time DATETIME NOT NULL,
+                            lock_end_time DATETIME NOT NULL,
+                            active BOOLEAN NOT NULL,
+                            PRIMARY KEY (id)
+                        )
+                                """
+    create_index1 = "CREATE INDEX ix_pairlocks_pair ON pairlocks (pair)"
+    create_index2 = "CREATE INDEX ix_pairlocks_lock_end_time ON pairlocks (lock_end_time)"
+    create_index3 = "CREATE INDEX ix_pairlocks_active ON pairlocks (active)"
+    insert_table_old = """INSERT INTO pairlocks (
+        id, pair, reason, lock_time, lock_end_time, active)
+        VALUES (1, 'ETH/BTC', 'Auto lock', '2021-07-12 18:41:03', '2021-07-11 18:45:00', 1)
+                          """
+    insert_table_old2 = """INSERT INTO pairlocks (
+        id, pair, reason, lock_time, lock_end_time, active)
+        VALUES (2, '*', 'Lock all', '2021-07-12 18:41:03', '2021-07-12 19:00:00', 1)
+                          """
+    engine = create_engine('sqlite://')
+    mocker.patch('freqtrade.persistence.models.create_engine', lambda *args, **kwargs: engine)
+    # Create table using the old format
+    with engine.begin() as connection:
+        connection.execute(text(create_table_old))
+
+        connection.execute(text(insert_table_old))
+        connection.execute(text(insert_table_old2))
+        connection.execute(text(create_index1))
+        connection.execute(text(create_index2))
+        connection.execute(text(create_index3))
+
+    init_db(default_conf['db_url'])
+
+    assert len(PairLock.query.all()) == 2
+    assert len(PairLock.query.filter(PairLock.pair == '*').all()) == 1
+    pairlocks = PairLock.query.filter(PairLock.pair == 'ETH/BTC').all()
+    assert len(pairlocks) == 1
+    pairlocks[0].pair == 'ETH/BTC'
+    pairlocks[0].side == '*'

tests/persistence/test_persistence.py

@@ -1,78 +1,20 @@
 # pragma pylint: disable=missing-docstring, C0103
-import logging
 from datetime import datetime, timedelta, timezone
-from pathlib import Path
 from types import FunctionType
-from unittest.mock import MagicMock
 
 import arrow
 import pytest
-from sqlalchemy import create_engine, text
 
-from freqtrade.constants import DATETIME_PRINT_FORMAT, DEFAULT_DB_PROD_URL
+from freqtrade.constants import DATETIME_PRINT_FORMAT
 from freqtrade.enums import TradingMode
-from freqtrade.exceptions import DependencyException, OperationalException
+from freqtrade.exceptions import DependencyException
 from freqtrade.persistence import LocalTrade, Order, Trade, init_db
-from freqtrade.persistence.migrations import get_last_sequence_ids, set_sequence_ids
-from freqtrade.persistence.models import PairLock
 from tests.conftest import create_mock_trades, create_mock_trades_with_leverage, log_has, log_has_re
 
 
 spot, margin, futures = TradingMode.SPOT, TradingMode.MARGIN, TradingMode.FUTURES
 
 
-def test_init_create_session(default_conf):
-    # Check if init create a session
-    init_db(default_conf['db_url'])
-    assert hasattr(Trade, '_session')
-    assert 'scoped_session' in type(Trade._session).__name__
-
-
-def test_init_custom_db_url(default_conf, tmpdir):
-    # Update path to a value other than default, but still in-memory
-    filename = f"{tmpdir}/freqtrade2_test.sqlite"
-    assert not Path(filename).is_file()
-
-    default_conf.update({'db_url': f'sqlite:///{filename}'})
-
-    init_db(default_conf['db_url'])
-    assert Path(filename).is_file()
-    r = Trade._session.execute(text("PRAGMA journal_mode"))
-    assert r.first() == ('wal',)
-
-
-def test_init_invalid_db_url():
-    # Update path to a value other than default, but still in-memory
-    with pytest.raises(OperationalException, match=r'.*no valid database URL*'):
-        init_db('unknown:///some.url')
-
-    with pytest.raises(OperationalException, match=r'Bad db-url.*For in-memory database, pl.*'):
-        init_db('sqlite:///')
-
-
-def test_init_prod_db(default_conf, mocker):
-    default_conf.update({'dry_run': False})
-    default_conf.update({'db_url': DEFAULT_DB_PROD_URL})
-
-    create_engine_mock = mocker.patch('freqtrade.persistence.models.create_engine', MagicMock())
-
-    init_db(default_conf['db_url'])
-    assert create_engine_mock.call_count == 1
-    assert create_engine_mock.mock_calls[0][1][0] == 'sqlite:///tradesv3.sqlite'
-
-
-def test_init_dryrun_db(default_conf, tmpdir):
-    filename = f"{tmpdir}/freqtrade2_prod.sqlite"
-    assert not Path(filename).is_file()
-    default_conf.update({
-        'dry_run': True,
-        'db_url': f'sqlite:///{filename}'
-    })
-
-    init_db(default_conf['db_url'])
-    assert Path(filename).is_file()
-
-
 @pytest.mark.parametrize('is_short', [False, True])
 @pytest.mark.usefixtures("init_persistence")
 def test_enter_exit_side(fee, is_short):
@@ -316,8 +258,7 @@ def test_interest(fee, exchange, is_short, lev, minutes, rate, interest,
     (True, 3.0, 30.0, margin),
 ])
 @pytest.mark.usefixtures("init_persistence")
-def test_borrowed(limit_buy_order_usdt, limit_sell_order_usdt, fee,
-                  caplog, is_short, lev, borrowed, trading_mode):
+def test_borrowed(fee, is_short, lev, borrowed, trading_mode):
     """
         10 minute limit trade on Binance/Kraken at 1x, 3x leverage
         fee: 0.25% quote
@@ -1204,347 +1145,6 @@ def test_calc_profit(
                          trade.open_rate)) == round(profit_ratio, 8)
 
 
-def test_migrate_new(mocker, default_conf, fee, caplog):
-    """
-    Test Database migration (starting with new pairformat)
-    """
-    caplog.set_level(logging.DEBUG)
-    amount = 103.223
-    # Always create all columns apart from the last!
-    create_table_old = """CREATE TABLE IF NOT EXISTS "trades" (
-                                id INTEGER NOT NULL,
-                                exchange VARCHAR NOT NULL,
-                                pair VARCHAR NOT NULL,
-                                is_open BOOLEAN NOT NULL,
-                                fee FLOAT NOT NULL,
-                                open_rate FLOAT,
-                                close_rate FLOAT,
-                                close_profit FLOAT,
-                                stake_amount FLOAT NOT NULL,
-                                amount FLOAT,
-                                open_date DATETIME NOT NULL,
-                                close_date DATETIME,
-                                open_order_id VARCHAR,
-                                stop_loss FLOAT,
-                                initial_stop_loss FLOAT,
-                                max_rate FLOAT,
-                                sell_reason VARCHAR,
-                                strategy VARCHAR,
-                                ticker_interval INTEGER,
-                                stoploss_order_id VARCHAR,
-                                PRIMARY KEY (id),
-                                CHECK (is_open IN (0, 1))
-                                );"""
-    create_table_order = """CREATE TABLE orders (
-                                id INTEGER NOT NULL,
-                                ft_trade_id INTEGER,
-                                ft_order_side VARCHAR(25) NOT NULL,
-                                ft_pair VARCHAR(25) NOT NULL,
-                                ft_is_open BOOLEAN NOT NULL,
-                                order_id VARCHAR(255) NOT NULL,
-                                status VARCHAR(255),
-                                symbol VARCHAR(25),
-                                order_type VARCHAR(50),
-                                side VARCHAR(25),
-                                price FLOAT,
-                                amount FLOAT,
-                                filled FLOAT,
-                                remaining FLOAT,
-                                cost FLOAT,
-                                order_date DATETIME,
-                                order_filled_date DATETIME,
-                                order_update_date DATETIME,
-                                PRIMARY KEY (id)
-                            );"""
-    insert_table_old = """INSERT INTO trades (exchange, pair, is_open, fee,
-                          open_rate, stake_amount, amount, open_date,
-                          stop_loss, initial_stop_loss, max_rate, ticker_interval,
-                          open_order_id, stoploss_order_id)
-                          VALUES ('binance', 'ETC/BTC', 1, {fee},
-                          0.00258580, {stake}, {amount},
-                          '2019-11-28 12:44:24.000000',
-                          0.0, 0.0, 0.0, '5m',
-                          'buy_order', 'dry_stop_order_id222')
-                          """.format(fee=fee.return_value,
-                                     stake=default_conf.get("stake_amount"),
-                                     amount=amount
-                                     )
-    insert_orders = f"""
-        insert into orders (
-            ft_trade_id,
-            ft_order_side,
-            ft_pair,
-            ft_is_open,
-            order_id,
-            status,
-            symbol,
-            order_type,
-            side,
-            price,
-            amount,
-            filled,
-            remaining,
-            cost)
-        values (
-            1,
-            'buy',
-            'ETC/BTC',
-            0,
-            'dry_buy_order',
-            'closed',
-            'ETC/BTC',
-            'limit',
-            'buy',
-            0.00258580,
-            {amount},
-            {amount},
-            0,
-            {amount * 0.00258580}
-        ),
-        (
-            1,
-            'buy',
-            'ETC/BTC',
-            1,
-            'dry_buy_order22',
-            'canceled',
-            'ETC/BTC',
-            'limit',
-            'buy',
-            0.00258580,
-            {amount},
-            {amount},
-            0,
-            {amount * 0.00258580}
-        ),
-         (
-            1,
-            'stoploss',
-            'ETC/BTC',
-            1,
-            'dry_stop_order_id11X',
-            'canceled',
-            'ETC/BTC',
-            'limit',
-            'sell',
-            0.00258580,
-            {amount},
-            {amount},
-            0,
-            {amount * 0.00258580}
-        ),
-        (
-            1,
-            'stoploss',
-            'ETC/BTC',
-            1,
-            'dry_stop_order_id222',
-            'open',
-            'ETC/BTC',
-            'limit',
-            'sell',
-            0.00258580,
-            {amount},
-            {amount},
-            0,
-            {amount * 0.00258580}
-        )
-    """
-    engine = create_engine('sqlite://')
-    mocker.patch('freqtrade.persistence.models.create_engine', lambda *args, **kwargs: engine)
-
-    # Create table using the old format
-    with engine.begin() as connection:
-        connection.execute(text(create_table_old))
-        connection.execute(text(create_table_order))
-        connection.execute(text("create index ix_trades_is_open on trades(is_open)"))
-        connection.execute(text("create index ix_trades_pair on trades(pair)"))
-        connection.execute(text(insert_table_old))
-        connection.execute(text(insert_orders))
-
-        # fake previous backup
-        connection.execute(text("create table trades_bak as select * from trades"))
-
-        connection.execute(text("create table trades_bak1 as select * from trades"))
-    # Run init to test migration
-    init_db(default_conf['db_url'])
-
-    assert len(Trade.query.filter(Trade.id == 1).all()) == 1
-    trade = Trade.query.filter(Trade.id == 1).first()
-    assert trade.fee_open == fee.return_value
-    assert trade.fee_close == fee.return_value
-    assert trade.open_rate_requested is None
-    assert trade.close_rate_requested is None
-    assert trade.is_open == 1
-    assert trade.amount == amount
-    assert trade.amount_requested == amount
-    assert trade.stake_amount == default_conf.get("stake_amount")
-    assert trade.pair == "ETC/BTC"
-    assert trade.exchange == "binance"
-    assert trade.max_rate == 0.0
-    assert trade.min_rate is None
-    assert trade.stop_loss == 0.0
-    assert trade.initial_stop_loss == 0.0
-    assert trade.exit_reason is None
-    assert trade.strategy is None
-    assert trade.timeframe == '5m'
-    assert trade.stoploss_order_id == 'dry_stop_order_id222'
-    assert trade.stoploss_last_update is None
-    assert log_has("trying trades_bak1", caplog)
-    assert log_has("trying trades_bak2", caplog)
-    assert log_has("Running database migration for trades - backup: trades_bak2, orders_bak0",
-                   caplog)
-    assert log_has("Database migration finished.", caplog)
-    assert pytest.approx(trade.open_trade_value) == trade._calc_open_trade_value(
-        trade.amount, trade.open_rate)
-    assert trade.close_profit_abs is None
-
-    orders = trade.orders
-    assert len(orders) == 4
-    assert orders[0].order_id == 'dry_buy_order'
-    assert orders[0].ft_order_side == 'buy'
-
-    assert orders[-1].order_id == 'dry_stop_order_id222'
-    assert orders[-1].ft_order_side == 'stoploss'
-    assert orders[-1].ft_is_open is True
-
-    assert orders[1].order_id == 'dry_buy_order22'
-    assert orders[1].ft_order_side == 'buy'
-    assert orders[1].ft_is_open is False
-
-    assert orders[2].order_id == 'dry_stop_order_id11X'
-    assert orders[2].ft_order_side == 'stoploss'
-    assert orders[2].ft_is_open is False
-
-
-def test_migrate_too_old(mocker, default_conf, fee, caplog):
-    """
-    Test Database migration (starting with new pairformat)
-    """
-    caplog.set_level(logging.DEBUG)
-    amount = 103.223
-    create_table_old = """CREATE TABLE IF NOT EXISTS "trades" (
-                                id INTEGER NOT NULL,
-                                exchange VARCHAR NOT NULL,
-                                pair VARCHAR NOT NULL,
-                                is_open BOOLEAN NOT NULL,
-                                fee_open FLOAT NOT NULL,
-                                fee_close FLOAT NOT NULL,
-                                open_rate FLOAT,
-                                close_rate FLOAT,
-                                close_profit FLOAT,
-                                stake_amount FLOAT NOT NULL,
-                                amount FLOAT,
-                                open_date DATETIME NOT NULL,
-                                close_date DATETIME,
-                                open_order_id VARCHAR,
-                                PRIMARY KEY (id),
-                                CHECK (is_open IN (0, 1))
-                                );"""
-
-    insert_table_old = """INSERT INTO trades (exchange, pair, is_open, fee_open, fee_close,
-                          open_rate, stake_amount, amount, open_date)
-                          VALUES ('binance', 'ETC/BTC', 1, {fee}, {fee},
-                          0.00258580, {stake}, {amount},
-                          '2019-11-28 12:44:24.000000')
-                          """.format(fee=fee.return_value,
-                                     stake=default_conf.get("stake_amount"),
-                                     amount=amount
-                                     )
-    engine = create_engine('sqlite://')
-    mocker.patch('freqtrade.persistence.models.create_engine', lambda *args, **kwargs: engine)
-
-    # Create table using the old format
-    with engine.begin() as connection:
-        connection.execute(text(create_table_old))
-        connection.execute(text(insert_table_old))
-
-    # Run init to test migration
-    with pytest.raises(OperationalException, match=r'Your database seems to be very old'):
-        init_db(default_conf['db_url'])
-
-
-def test_migrate_get_last_sequence_ids():
-    engine = MagicMock()
-    engine.begin = MagicMock()
-    engine.name = 'postgresql'
-    get_last_sequence_ids(engine, 'trades_bak', 'orders_bak')
-
-    assert engine.begin.call_count == 2
-    engine.reset_mock()
-    engine.begin.reset_mock()
-
-    engine.name = 'somethingelse'
-    get_last_sequence_ids(engine, 'trades_bak', 'orders_bak')
-
-    assert engine.begin.call_count == 0
-
-
-def test_migrate_set_sequence_ids():
-    engine = MagicMock()
-    engine.begin = MagicMock()
-    engine.name = 'postgresql'
-    set_sequence_ids(engine, 22, 55, 5)
-
-    assert engine.begin.call_count == 1
-    engine.reset_mock()
-    engine.begin.reset_mock()
-
-    engine.name = 'somethingelse'
-    set_sequence_ids(engine, 22, 55, 6)
-
-    assert engine.begin.call_count == 0
-
-
-def test_migrate_pairlocks(mocker, default_conf, fee, caplog):
-    """
-    Test Database migration (starting with new pairformat)
-    """
-    caplog.set_level(logging.DEBUG)
-    # Always create all columns apart from the last!
-    create_table_old = """CREATE TABLE pairlocks (
-                            id INTEGER NOT NULL,
-                            pair VARCHAR(25) NOT NULL,
-                            reason VARCHAR(255),
-                            lock_time DATETIME NOT NULL,
-                            lock_end_time DATETIME NOT NULL,
-                            active BOOLEAN NOT NULL,
-                            PRIMARY KEY (id)
-                        )
-                                """
-    create_index1 = "CREATE INDEX ix_pairlocks_pair ON pairlocks (pair)"
-    create_index2 = "CREATE INDEX ix_pairlocks_lock_end_time ON pairlocks (lock_end_time)"
-    create_index3 = "CREATE INDEX ix_pairlocks_active ON pairlocks (active)"
-    insert_table_old = """INSERT INTO pairlocks (
-        id, pair, reason, lock_time, lock_end_time, active)
-        VALUES (1, 'ETH/BTC', 'Auto lock', '2021-07-12 18:41:03', '2021-07-11 18:45:00', 1)
-                          """
-    insert_table_old2 = """INSERT INTO pairlocks (
-        id, pair, reason, lock_time, lock_end_time, active)
-        VALUES (2, '*', 'Lock all', '2021-07-12 18:41:03', '2021-07-12 19:00:00', 1)
-                          """
-    engine = create_engine('sqlite://')
-    mocker.patch('freqtrade.persistence.models.create_engine', lambda *args, **kwargs: engine)
-    # Create table using the old format
-    with engine.begin() as connection:
-        connection.execute(text(create_table_old))
-
-        connection.execute(text(insert_table_old))
-        connection.execute(text(insert_table_old2))
-        connection.execute(text(create_index1))
-        connection.execute(text(create_index2))
-        connection.execute(text(create_index3))
-
-    init_db(default_conf['db_url'])
-
-    assert len(PairLock.query.all()) == 2
-    assert len(PairLock.query.filter(PairLock.pair == '*').all()) == 1
-    pairlocks = PairLock.query.filter(PairLock.pair == 'ETH/BTC').all()
-    assert len(pairlocks) == 1
-    pairlocks[0].pair == 'ETH/BTC'
-    pairlocks[0].side == '*'
-
-
 def test_adjust_stop_loss(fee):
     trade = Trade(
         pair='ADA/USDT',
@@ -1758,6 +1358,7 @@ def test_to_json(fee):
                       'amount': 123.0,
                       'amount_requested': 123.0,
                       'stake_amount': 0.001,
+                      'max_stake_amount': None,
                       'trade_duration': None,
                       'trade_duration_s': None,
                       'realized_profit': 0.0,
@@ -1767,7 +1368,6 @@ def test_to_json(fee):
                       'profit_ratio': None,
                       'profit_pct': None,
                       'profit_abs': None,
-                      'sell_reason': None,
                       'exit_reason': None,
                       'exit_order_status': None,
                       'stop_loss_abs': None,
@@ -1782,7 +1382,6 @@ def test_to_json(fee):
                       'min_rate': None,
                       'max_rate': None,
                       'strategy': None,
-                      'buy_tag': None,
                       'enter_tag': None,
                       'timeframe': None,
                       'exchange': 'binance',
@@ -1826,6 +1425,7 @@ def test_to_json(fee):
                       'amount': 100.0,
                       'amount_requested': 101.0,
                       'stake_amount': 0.001,
+                      'max_stake_amount': None,
                       'trade_duration': 60,
                       'trade_duration_s': 3600,
                       'stop_loss_abs': None,
@@ -1857,11 +1457,9 @@ def test_to_json(fee):
                       'open_order_id': None,
                       'open_rate_requested': None,
                       'open_trade_value': 12.33075,
-                      'sell_reason': None,
                       'exit_reason': None,
                       'exit_order_status': None,
                       'strategy': None,
-                      'buy_tag': 'buys_signal_001',
                       'enter_tag': 'buys_signal_001',
                       'timeframe': None,
                       'exchange': 'binance',

tests/plugins/test_pairlist.py

@@ -22,6 +22,11 @@ from tests.conftest import (create_mock_trades_usdt, get_patched_exchange, get_p
                             log_has, log_has_re, num_log_has)
 
 
+# Exclude RemotePairList from tests.
+# It has a mandatory parameter, and requires special handling, which happens in test_remotepairlist.
+TESTABLE_PAIRLISTS = [p for p in AVAILABLE_PAIRLISTS if p not in ['RemotePairList']]
+
+
 @pytest.fixture(scope="function")
 def whitelist_conf(default_conf):
     default_conf['stake_currency'] = 'BTC'
@@ -824,7 +829,7 @@ def test_pair_whitelist_not_supported_Spread(mocker, default_conf, tickers) -> N
         get_patched_freqtradebot(mocker, default_conf)
 
 
-@pytest.mark.parametrize("pairlist", AVAILABLE_PAIRLISTS)
+@pytest.mark.parametrize("pairlist", TESTABLE_PAIRLISTS)
 def test_pairlist_class(mocker, whitelist_conf, markets, pairlist):
     whitelist_conf['pairlists'][0]['method'] = pairlist
     mocker.patch.multiple('freqtrade.exchange.Exchange',
@@ -839,7 +844,7 @@ def test_pairlist_class(mocker, whitelist_conf, markets, pairlist):
     assert isinstance(freqtrade.pairlists.blacklist, list)
 
 
-@pytest.mark.parametrize("pairlist", AVAILABLE_PAIRLISTS)
+@pytest.mark.parametrize("pairlist", TESTABLE_PAIRLISTS)
 @pytest.mark.parametrize("whitelist,log_message", [
     (['ETH/BTC', 'TKN/BTC'], ""),
     # TRX/ETH not in markets
@@ -872,7 +877,7 @@ def test__whitelist_for_active_markets(mocker, whitelist_conf, markets, pairlist
     assert log_message in caplog.text
 
 
-@pytest.mark.parametrize("pairlist", AVAILABLE_PAIRLISTS)
+@pytest.mark.parametrize("pairlist", TESTABLE_PAIRLISTS)
 def test__whitelist_for_active_markets_empty(mocker, whitelist_conf, pairlist, tickers):
     whitelist_conf['pairlists'][0]['method'] = pairlist
 

tests/plugins/test_remotepairlist.py

@@ -0,0 +1,185 @@
+import json
+from unittest.mock import MagicMock
+
+import pytest
+import requests
+
+from freqtrade.exceptions import OperationalException
+from freqtrade.plugins.pairlist.RemotePairList import RemotePairList
+from freqtrade.plugins.pairlistmanager import PairListManager
+from tests.conftest import get_patched_exchange, get_patched_freqtradebot, log_has
+
+
+@pytest.fixture(scope="function")
+def rpl_config(default_conf):
+    default_conf['stake_currency'] = 'USDT'
+
+    default_conf['exchange']['pair_whitelist'] = [
+        'ETH/USDT',
+        'BTC/USDT',
+    ]
+    default_conf['exchange']['pair_blacklist'] = [
+        'BLK/USDT'
+    ]
+    return default_conf
+
+
+def test_gen_pairlist_with_local_file(mocker, rpl_config):
+
+    mock_file = MagicMock()
+    mock_file.read.return_value = '{"pairs": ["TKN/USDT","ETH/USDT","NANO/USDT"]}'
+    mocker.patch('freqtrade.plugins.pairlist.RemotePairList.open', return_value=mock_file)
+
+    mock_file_path = mocker.patch('freqtrade.plugins.pairlist.RemotePairList.Path')
+    mock_file_path.exists.return_value = True
+
+    jsonparse = json.loads(mock_file.read.return_value)
+    mocker.patch('freqtrade.plugins.pairlist.RemotePairList.json.load', return_value=jsonparse)
+
+    rpl_config['pairlists'] = [
+        {
+            "method": "RemotePairList",
+            'number_assets': 2,
+            'refresh_period': 1800,
+            'keep_pairlist_on_failure': True,
+            'pairlist_url': 'file:///pairlist.json',
+            'bearer_token': '',
+            'read_timeout': 60
+        }
+    ]
+
+    exchange = get_patched_exchange(mocker, rpl_config)
+    pairlistmanager = PairListManager(exchange, rpl_config)
+
+    remote_pairlist = RemotePairList(exchange, pairlistmanager, rpl_config,
+                                     rpl_config['pairlists'][0], 0)
+
+    result = remote_pairlist.gen_pairlist([])
+
+    assert result == ['TKN/USDT', 'ETH/USDT']
+
+
+def test_fetch_pairlist_mock_response_html(mocker, rpl_config):
+    mock_response = MagicMock()
+    mock_response.headers = {'content-type': 'text/html'}
+
+    rpl_config['pairlists'] = [
+        {
+            "method": "RemotePairList",
+            "pairlist_url": "http://example.com/pairlist",
+            "number_assets": 10,
+            "read_timeout": 10,
+            "keep_pairlist_on_failure": True,
+        }
+    ]
+
+    exchange = get_patched_exchange(mocker, rpl_config)
+    pairlistmanager = PairListManager(exchange, rpl_config)
+
+    mocker.patch("freqtrade.plugins.pairlist.RemotePairList.requests.get",
+                 return_value=mock_response)
+    remote_pairlist = RemotePairList(exchange, pairlistmanager, rpl_config,
+                                     rpl_config['pairlists'][0], 0)
+
+    with pytest.raises(OperationalException, match='RemotePairList is not of type JSON, abort.'):
+        remote_pairlist.fetch_pairlist()
+
+
+def test_fetch_pairlist_timeout_keep_last_pairlist(mocker, rpl_config, caplog):
+    rpl_config['pairlists'] = [
+        {
+            "method": "RemotePairList",
+            "pairlist_url": "http://example.com/pairlist",
+            "number_assets": 10,
+            "read_timeout": 10,
+            "keep_pairlist_on_failure": True,
+        }
+    ]
+
+    exchange = get_patched_exchange(mocker, rpl_config)
+    pairlistmanager = PairListManager(exchange, rpl_config)
+
+    mocker.patch("freqtrade.plugins.pairlist.RemotePairList.requests.get",
+                 side_effect=requests.exceptions.RequestException)
+
+    remote_pairlist = RemotePairList(exchange, pairlistmanager, rpl_config,
+                                     rpl_config['pairlists'][0], 0)
+
+    remote_pairlist._last_pairlist = ["BTC/USDT", "ETH/USDT", "LTC/USDT"]
+
+    pairs, time_elapsed = remote_pairlist.fetch_pairlist()
+    assert log_has(f"Was not able to fetch pairlist from: {remote_pairlist._pairlist_url}", caplog)
+    assert log_has("Keeping last fetched pairlist", caplog)
+    assert pairs == ["BTC/USDT", "ETH/USDT", "LTC/USDT"]
+
+
+def test_remote_pairlist_init_no_pairlist_url(mocker, rpl_config):
+
+    rpl_config['pairlists'] = [
+        {
+            "method": "RemotePairList",
+            "number_assets": 10,
+            "keep_pairlist_on_failure": True,
+        }
+    ]
+
+    get_patched_exchange(mocker, rpl_config)
+    with pytest.raises(OperationalException, match=r'`pairlist_url` not specified.'
+                       r' Please check your configuration for "pairlist.config.pairlist_url"'):
+        get_patched_freqtradebot(mocker, rpl_config)
+
+
+def test_remote_pairlist_init_no_number_assets(mocker, rpl_config):
+
+    rpl_config['pairlists'] = [
+        {
+            "method": "RemotePairList",
+            "pairlist_url": "http://example.com/pairlist",
+            "keep_pairlist_on_failure": True,
+        }
+    ]
+
+    get_patched_exchange(mocker, rpl_config)
+
+    with pytest.raises(OperationalException, match=r'`number_assets` not specified. '
+                       'Please check your configuration for "pairlist.config.number_assets"'):
+        get_patched_freqtradebot(mocker, rpl_config)
+
+
+def test_fetch_pairlist_mock_response_valid(mocker, rpl_config):
+
+    rpl_config['pairlists'] = [
+        {
+            "method": "RemotePairList",
+            "pairlist_url": "http://example.com/pairlist",
+            "number_assets": 10,
+            "refresh_period": 10,
+            "read_timeout": 10,
+            "keep_pairlist_on_failure": True,
+        }
+    ]
+
+    mock_response = MagicMock()
+
+    mock_response.json.return_value = {
+        "pairs": ["ETH/USDT", "XRP/USDT", "LTC/USDT", "EOS/USDT"],
+        "refresh_period": 60
+    }
+
+    mock_response.headers = {
+        "content-type": "application/json"
+    }
+
+    mock_response.elapsed.total_seconds.return_value = 0.4
+    mocker.patch("freqtrade.plugins.pairlist.RemotePairList.requests.get",
+                 return_value=mock_response)
+
+    exchange = get_patched_exchange(mocker, rpl_config)
+    pairlistmanager = PairListManager(exchange, rpl_config)
+    remote_pairlist = RemotePairList(exchange, pairlistmanager, rpl_config,
+                                     rpl_config['pairlists'][0], 0)
+    pairs, time_elapsed = remote_pairlist.fetch_pairlist()
+
+    assert pairs == ["ETH/USDT", "XRP/USDT", "LTC/USDT", "EOS/USDT"]
+    assert time_elapsed == 0.4
+    assert remote_pairlist._refresh_period == 60

tests/rpc/test_rpc.py

@@ -46,13 +46,11 @@ def test_rpc_trade_status(default_conf, ticker, fee, mocker) -> None:
         'open_rate_requested': ANY,
         'open_trade_value': 0.0010025,
         'close_rate_requested': ANY,
-        'sell_reason': ANY,
         'exit_reason': ANY,
         'exit_order_status': ANY,
         'min_rate': ANY,
         'max_rate': ANY,
         'strategy': ANY,
-        'buy_tag': ANY,
         'enter_tag': ANY,
         'timeframe': 5,
         'open_order_id': ANY,
@@ -64,6 +62,7 @@ def test_rpc_trade_status(default_conf, ticker, fee, mocker) -> None:
         'amount': 91.07468123,
         'amount_requested': 91.07468124,
         'stake_amount': 0.001,
+        'max_stake_amount': ANY,
         'trade_duration': None,
         'trade_duration_s': None,
         'close_profit': None,

tests/rpc/test_rpc_apiserver.py

@@ -985,6 +985,7 @@ def test_api_status(botclient, mocker, ticker, fee, markets, is_short,
         'base_currency': 'ETH',
         'quote_currency': 'BTC',
         'stake_amount': 0.001,
+        'max_stake_amount': ANY,
         'stop_loss_abs': ANY,
         'stop_loss_pct': ANY,
         'stop_loss_ratio': ANY,
@@ -1014,11 +1015,9 @@ def test_api_status(botclient, mocker, ticker, fee, markets, is_short,
         'open_order_id': open_order_id,
         'open_rate_requested': ANY,
         'open_trade_value': open_trade_value,
-        'sell_reason': None,
         'exit_reason': None,
         'exit_order_status': None,
         'strategy': CURRENT_TEST_STRATEGY,
-        'buy_tag': None,
         'enter_tag': None,
         'timeframe': 5,
         'exchange': 'binance',
@@ -1188,6 +1187,7 @@ def test_api_force_entry(botclient, mocker, fee, endpoint):
         'base_currency': 'ETH',
         'quote_currency': 'BTC',
         'stake_amount': 1,
+        'max_stake_amount': ANY,
         'stop_loss_abs': None,
         'stop_loss_pct': None,
         'stop_loss_ratio': None,
@@ -1218,11 +1218,9 @@ def test_api_force_entry(botclient, mocker, fee, endpoint):
         'open_order_id': '123456',
         'open_rate_requested': None,
         'open_trade_value': 0.24605460,
-        'sell_reason': None,
         'exit_reason': None,
         'exit_order_status': None,
         'strategy': CURRENT_TEST_STRATEGY,
-        'buy_tag': None,
         'enter_tag': None,
         'timeframe': 5,
         'exchange': 'binance',
@@ -1488,6 +1486,44 @@ def test_api_strategy(botclient):
     assert_response(rc, 500)
 
 
+def test_api_freqaimodels(botclient, tmpdir, mocker):
+    ftbot, client = botclient
+    ftbot.config['user_data_dir'] = Path(tmpdir)
+    mocker.patch(
+        "freqtrade.resolvers.freqaimodel_resolver.FreqaiModelResolver.search_all_objects",
+        return_value=[
+            {'name': 'LightGBMClassifier'},
+            {'name': 'LightGBMClassifierMultiTarget'},
+            {'name': 'LightGBMRegressor'},
+            {'name': 'LightGBMRegressorMultiTarget'},
+            {'name': 'ReinforcementLearner'},
+            {'name': 'ReinforcementLearner_multiproc'},
+            {'name': 'XGBoostClassifier'},
+            {'name': 'XGBoostRFClassifier'},
+            {'name': 'XGBoostRFRegressor'},
+            {'name': 'XGBoostRegressor'},
+            {'name': 'XGBoostRegressorMultiTarget'},
+        ])
+
+    rc = client_get(client, f"{BASE_URI}/freqaimodels")
+
+    assert_response(rc)
+
+    assert rc.json() == {'freqaimodels': [
+        'LightGBMClassifier',
+        'LightGBMClassifierMultiTarget',
+        'LightGBMRegressor',
+        'LightGBMRegressorMultiTarget',
+        'ReinforcementLearner',
+        'ReinforcementLearner_multiproc',
+        'XGBoostClassifier',
+        'XGBoostRFClassifier',
+        'XGBoostRFRegressor',
+        'XGBoostRegressor',
+        'XGBoostRegressorMultiTarget'
+    ]}
+
+
 def test_list_available_pairs(botclient):
     ftbot, client = botclient
 
@@ -1671,7 +1707,7 @@ def test_api_backtest_history(botclient, mocker, testdatadir):
     mocker.patch('freqtrade.data.btanalysis._get_backtest_files',
                  return_value=[
                      testdatadir / 'backtest_results/backtest-result_multistrat.json',
-                     testdatadir / 'backtest_results/backtest-result_new.json'
+                     testdatadir / 'backtest_results/backtest-result.json'
                      ])
 
     rc = client_get(client, f"{BASE_URI}/backtest/history")

tests/rpc/test_rpc_emc.py

@@ -267,7 +267,7 @@ async def test_emc_create_connection_error(default_conf, caplog, mocker):
     emc = ExternalMessageConsumer(default_conf, dp)
 
     try:
-        await asyncio.sleep(0.01)
+        await asyncio.sleep(0.05)
         assert log_has("Unexpected error has occurred:", caplog)
     finally:
         emc.shutdown()

tests/rpc/test_rpc_webhook.py

@@ -356,6 +356,14 @@ def test_exception_send_msg(default_conf, mocker, caplog):
             }
         webhook.send_msg(msg)
 
+    # Test no failure for not implemented but known messagetypes
+    for e in RPCMessageType:
+        msg = {
+            'type': e,
+            'status': 'whatever'
+            }
+        webhook.send_msg(msg)
+
 
 def test__send_msg(default_conf, mocker, caplog):
     default_conf["webhook"] = get_webhook_dict()

tests/test_freqtradebot.py

@@ -2378,7 +2378,7 @@ def test_close_trade(
     trade.is_short = is_short
     assert trade
 
-    oobj = Order.parse_from_ccxt_object(enter_order, enter_order['symbol'], trade.enter_side)
+    oobj = Order.parse_from_ccxt_object(enter_order, enter_order['symbol'], trade.entry_side)
     trade.update_trade(oobj)
     oobj = Order.parse_from_ccxt_object(exit_order, exit_order['symbol'], trade.exit_side)
     trade.update_trade(oobj)

tests/test_plotting.py

@@ -46,7 +46,7 @@ def test_init_plotscript(default_conf, mocker, testdatadir):
     default_conf['trade_source'] = "file"
     default_conf['timeframe'] = "5m"
     default_conf["datadir"] = testdatadir
-    default_conf['exportfilename'] = testdatadir / "backtest-result_new.json"
+    default_conf['exportfilename'] = testdatadir / "backtest-result.json"
     supported_markets = ["TRX/BTC", "ADA/BTC"]
     ret = init_plotscript(default_conf, supported_markets)
     assert "ohlcv" in ret
@@ -158,7 +158,7 @@ def test_plot_trades(testdatadir, caplog):
     assert fig == fig1
     assert log_has("No trades found.", caplog)
     pair = "ADA/BTC"
-    filename = testdatadir / "backtest_results/backtest-result_new.json"
+    filename = testdatadir / "backtest_results/backtest-result.json"
     trades = load_backtest_data(filename)
     trades = trades.loc[trades['pair'] == pair]
 
@@ -299,7 +299,7 @@ def test_generate_plot_file(mocker, caplog):
 
 
 def test_add_profit(testdatadir):
-    filename = testdatadir / "backtest_results/backtest-result_new.json"
+    filename = testdatadir / "backtest_results/backtest-result.json"
     bt_data = load_backtest_data(filename)
     timerange = TimeRange.parse_timerange("20180110-20180112")
 
@@ -319,7 +319,7 @@ def test_add_profit(testdatadir):
 
 
 def test_generate_profit_graph(testdatadir):
-    filename = testdatadir / "backtest_results/backtest-result_new.json"
+    filename = testdatadir / "backtest_results/backtest-result.json"
     trades = load_backtest_data(filename)
     timerange = TimeRange.parse_timerange("20180110-20180112")
     pairs = ["TRX/BTC", "XLM/BTC"]
@@ -354,7 +354,7 @@ def test_generate_profit_graph(testdatadir):
 
     profit = find_trace_in_fig_data(figure.data, "Profit")
     assert isinstance(profit, go.Scatter)
-    drawdown = find_trace_in_fig_data(figure.data, "Max drawdown 35.69%")
+    drawdown = find_trace_in_fig_data(figure.data, "Max drawdown 73.89%")
     assert isinstance(drawdown, go.Scatter)
     parallel = find_trace_in_fig_data(figure.data, "Parallel trades")
     assert isinstance(parallel, go.Scatter)
@@ -395,7 +395,7 @@ def test_load_and_plot_trades(default_conf, mocker, caplog, testdatadir):
 
     default_conf['trade_source'] = 'file'
     default_conf["datadir"] = testdatadir
-    default_conf['exportfilename'] = testdatadir / "backtest-result_new.json"
+    default_conf['exportfilename'] = testdatadir / "backtest-result.json"
     default_conf['indicators1'] = ["sma5", "ema10"]
     default_conf['indicators2'] = ["macd"]
     default_conf['pairs'] = ["ETH/BTC", "LTC/BTC"]
@@ -466,7 +466,7 @@ def test_plot_profit(default_conf, mocker, testdatadir):
                        match=r"No trades found, cannot generate Profit-plot.*"):
         plot_profit(default_conf)
 
-    default_conf['exportfilename'] = testdatadir / "backtest_results/backtest-result_new.json"
+    default_conf['exportfilename'] = testdatadir / "backtest_results/backtest-result.json"
 
     plot_profit(default_conf)
 

tests/test_wallets.py

@@ -180,17 +180,17 @@ def test_get_trade_stake_amount_unlimited_amount(default_conf, ticker, balance_r
     assert result == 0
 
 
-@pytest.mark.parametrize('stake_amount,min_stake,stake_available,max_stake,expected', [
-    (22, 11, 50, 10000, 22),
-    (100, 11, 500, 10000, 100),
-    (1000, 11, 500, 10000, 500),  # Above stake_available
-    (700, 11, 1000, 400, 400),  # Above max_stake, below stake available
-    (20, 15, 10, 10000, 0),  # Minimum stake > stake_available
-    (9, 11, 100, 10000, 11),  # Below min stake
-    (1, 15, 10, 10000, 0),  # Below min stake and min_stake > stake_available
-    (20, 50, 100, 10000, 0),  # Below min stake and stake * 1.3 > min_stake
-    (1000, None, 1000, 10000, 1000),  # No min-stake-amount could be determined
-
+@pytest.mark.parametrize('stake_amount,min_stake,stake_available,max_stake,trade_amount,expected', [
+    (22, 11, 50, 10000, None, 22),
+    (100, 11, 500, 10000, None, 100),
+    (1000, 11, 500, 10000, None, 500),  # Above stake_available
+    (700, 11, 1000, 400, None, 400),  # Above max_stake, below stake available
+    (20, 15, 10, 10000, None, 0),  # Minimum stake > stake_available
+    (9, 11, 100, 10000, None, 11),  # Below min stake
+    (1, 15, 10, 10000, None, 0),  # Below min stake and min_stake > stake_available
+    (20, 50, 100, 10000, None, 0),  # Below min stake and stake * 1.3 > min_stake
+    (1000, None, 1000, 10000, None, 1000),  # No min-stake-amount could be determined
+    (2000, 15, 2000, 3000, 1500, 500),  # Rebuy - resulting in too high stake amount. Adjusting.
 ])
 def test_validate_stake_amount(
     mocker,
@@ -199,13 +199,15 @@ def test_validate_stake_amount(
     min_stake,
     stake_available,
     max_stake,
+    trade_amount,
     expected,
 ):
     freqtrade = get_patched_freqtradebot(mocker, default_conf)
 
     mocker.patch("freqtrade.wallets.Wallets.get_available_stake_amount",
                  return_value=stake_available)
-    res = freqtrade.wallets.validate_stake_amount('XRP/USDT', stake_amount, min_stake, max_stake)
+    res = freqtrade.wallets.validate_stake_amount(
+        'XRP/USDT', stake_amount, min_stake, max_stake, trade_amount)
     assert res == expected
 
 

tests/testdata/backtest_results/.last_result.json

@@ -1 +1 @@
-{"latest_backtest":"backtest-result_new.json"}
+{"latest_backtest":"backtest-result.json"}