Merge branch 'feat/short' into pr/samgermain/5378

README.md

@@ -37,7 +37,7 @@ Please read the [exchange specific notes](docs/exchanges.md) to learn about even
 Exchanges confirmed working by the community:
 
 - [X] [Bitvavo](https://bitvavo.com/)
-- [X] [Kukoin](https://www.kucoin.com/)
+- [X] [Kucoin](https://www.kucoin.com/)
 
 ## Documentation
 

build_helpers/publish_docker_arm64.sh

@@ -74,7 +74,5 @@ fi
 
 docker images
 
-if [ $? -ne 0 ]; then
-    echo "failed building image"
-    return 1
-fi
+# Cleanup old images from arm64 node.
+docker image prune -a --force --filter "until=24h"

docs/bot-basics.md

@@ -35,12 +35,13 @@ By default, loop runs every few seconds (`internals.process_throttle_secs`) and
   * Calls `check_buy_timeout()` strategy callback for open buy orders.
   * Calls `check_sell_timeout()` strategy callback for open sell orders.
 * Verifies existing positions and eventually places sell orders.
-  * Considers stoploss, ROI and sell-signal.
-  * Determine sell-price based on `ask_strategy` configuration setting.
+  * Considers stoploss, ROI and sell-signal, `custom_sell()` and `custom_stoploss()`.
+  * Determine sell-price based on `ask_strategy` configuration setting or by using the `custom_exit_price()` callback.
   * Before a sell order is placed, `confirm_trade_exit()` strategy callback is called.
 * Check if trade-slots are still available (if `max_open_trades` is reached).
 * Verifies buy signal trying to enter new positions.
-  * Determine buy-price based on `bid_strategy` configuration setting.
+  * Determine buy-price based on `bid_strategy` configuration setting, or by using the `custom_entry_price()` callback.
+  * Determine stake size by calling the `custom_stake_amount()` callback.
   * Before a buy order is placed, `confirm_trade_entry()` strategy callback is called.
 
 This loop will be repeated again and again until the bot is stopped.
@@ -52,9 +53,10 @@ This loop will be repeated again and again until the bot is stopped.
 * Load historic data for configured pairlist.
 * Calls `bot_loop_start()` once.
 * Calculate indicators (calls `populate_indicators()` once per pair).
-* Calculate buy / sell signals (calls `populate_buy_trend()` and `populate_sell_trend()` once per pair)
-* Confirm trade buy / sell (calls `confirm_trade_entry()` and `confirm_trade_exit()` if implemented in the strategy)
+* Calculate buy / sell signals (calls `populate_buy_trend()` and `populate_sell_trend()` once per pair).
 * Loops per candle simulating entry and exit points.
+  * Confirm trade buy / sell (calls `confirm_trade_entry()` and `confirm_trade_exit()` if implemented in the strategy).
+  * Call `custom_stoploss()` and `custom_sell()` to find custom exit points.
 * Generate backtest report output
 
 !!! Note

docs/configuration.md

@@ -105,11 +105,12 @@ Mandatory parameters are marked as **Required**, which means that they are requi
 | `ask_strategy.order_book_top` | Bot will use the top N rate in Order Book "price_side" to sell. I.e. a value of 2 will allow the bot to pick the 2nd ask rate in [Order Book Asks](#sell-price-with-orderbook-enabled)<br>*Defaults to `1`.* <br> **Datatype:** Positive Integer
 | `use_sell_signal` | Use sell signals produced by the strategy in addition to the `minimal_roi`. [Strategy Override](#parameters-in-the-strategy). <br>*Defaults to `true`.* <br> **Datatype:** Boolean
 | `sell_profit_only` | Wait until the bot reaches `sell_profit_offset` before taking a sell decision. [Strategy Override](#parameters-in-the-strategy). <br>*Defaults to `false`.* <br> **Datatype:** Boolean
-| `sell_profit_offset` | Sell-signal is only active above this value. [Strategy Override](#parameters-in-the-strategy). <br>*Defaults to `0.0`.* <br> **Datatype:** Float (as ratio)
+| `sell_profit_offset` | Sell-signal is only active above this value. Only active in combination with `sell_profit_only=True`. [Strategy Override](#parameters-in-the-strategy). <br>*Defaults to `0.0`.* <br> **Datatype:** Float (as ratio)
 | `ignore_roi_if_buy_signal` | Do not sell if the buy signal is still active. This setting takes preference over `minimal_roi` and `use_sell_signal`. [Strategy Override](#parameters-in-the-strategy). <br>*Defaults to `false`.* <br> **Datatype:** Boolean
 | `ignore_buying_expired_candle_after` | Specifies the number of seconds until a buy signal is no longer used. <br> **Datatype:** Integer
 | `order_types` | Configure order-types depending on the action (`"buy"`, `"sell"`, `"stoploss"`, `"stoploss_on_exchange"`). [More information below](#understand-order_types). [Strategy Override](#parameters-in-the-strategy).<br> **Datatype:** Dict
 | `order_time_in_force` | Configure time in force for buy and sell orders. [More information below](#understand-order_time_in_force). [Strategy Override](#parameters-in-the-strategy). <br> **Datatype:** Dict
+| `custom_price_max_distance_ratio` | Configure maximum distance ratio between current and custom entry or exit price. <br>*Defaults to `0.02` 2%).*<br> **Datatype:** Positive float
 | `exchange.name` | **Required.** Name of the exchange class to use. [List below](#user-content-what-values-for-exchangename). <br> **Datatype:** String
 | `exchange.sandbox` | Use the 'sandbox' version of the exchange, where the exchange provides a sandbox for risk-free integration. See [here](sandbox-testing.md) in more details.<br> **Datatype:** Boolean
 | `exchange.key` | API key to use for the exchange. Only required when you are in production mode.<br>**Keep it in secret, do not disclose publicly.** <br> **Datatype:** String

docs/developer.md

@@ -240,11 +240,18 @@ The `IProtection` parent class provides a helper method for this in `calculate_l
 !!! Note
     This section is a Work in Progress and is not a complete guide on how to test a new exchange with Freqtrade.
 
+!!! Note
+    Make sure to use an up-to-date version of CCXT before running any of the below tests.
+    You can get the latest version of ccxt by running `pip install -U ccxt` with activated virtual environment.
+    Native docker is not supported for these tests, however the available dev-container will support all required actions and eventually necessary changes.
+
 Most exchanges supported by CCXT should work out of the box.
 
 To quickly test the public endpoints of an exchange, add a configuration for your exchange to `test_ccxt_compat.py` and run these tests with `pytest --longrun tests/exchange/test_ccxt_compat.py`.
 Completing these tests successfully a good basis point (it's a requirement, actually), however these won't guarantee correct exchange functioning, as this only tests public endpoints, but no private endpoint (like generate order or similar).
 
+Also try to use `freqtrade download-data` for an extended timerange and verify that the data downloaded correctly (no holes, the specified timerange was actually downloaded).
+
 ### Stoploss On Exchange
 
 Check if the new exchange supports Stoploss on Exchange orders through their API.

docs/exchanges.md

@@ -105,7 +105,7 @@ To use subaccounts with FTX, you need to edit the configuration and add the foll
 
 ## Kucoin
 
-Kucoin requries a passphrase for each api key, you will therefore need to add this key into the configuration so your exchange section looks as follows:
+Kucoin requires a passphrase for each api key, you will therefore need to add this key into the configuration so your exchange section looks as follows:
 
 ```json
 "exchange": {

docs/includes/pairlists.md

@@ -58,7 +58,7 @@ This option must be configured along with `exchange.skip_pair_validation` in the
 
 When used in the chain of Pairlist Handlers in a non-leading position (after StaticPairList and other Pairlist Filters), `VolumePairList` considers outputs of previous Pairlist Handlers, adding its sorting/selection of the pairs by the trading volume.
 
-When used on the leading position of the chain of Pairlist Handlers, it does not consider `pair_whitelist` configuration setting, but selects the top assets from all available markets (with matching stake-currency) on the exchange.
+When used in the leading position of the chain of Pairlist Handlers, the `pair_whitelist` configuration setting is ignored. Instead, `VolumePairList` selects the top assets from all available markets with matching stake-currency on the exchange.
 
 The `refresh_period` setting allows to define the period (in seconds), at which the pairlist will be refreshed. Defaults to 1800s (30 minutes).
 The pairlist cache (`refresh_period`) on `VolumePairList` is only applicable to generating pairlists.
@@ -74,11 +74,14 @@ Filtering instances (not the first position in the list) will not apply any cach
         "method": "VolumePairList",
         "number_assets": 20,
         "sort_key": "quoteVolume",
+        "min_value": 0,
         "refresh_period": 1800
     }
 ],
 ```
 
+You can define a minimum volume with `min_value` - which will filter out pairs with a volume lower than the specified value in the specified timerange.
+
 `VolumePairList` can also operate in an advanced mode to build volume over a given timerange of specified candle size. It utilizes exchange historical candle data, builds a typical price (calculated by (open+high+low)/3) and multiplies the typical price with every candle's volume. The sum is the `quoteVolume` over the given range. This allows different scenarios, for a  more smoothened volume, when using longer ranges with larger candle sizes, or the opposite when using a short range with small candles.
 
 For convenience `lookback_days` can be specified, which will imply that 1d candles will be used for the lookback. In the example below the pairlist would be created based on the last 7 days:
@@ -89,6 +92,7 @@ For convenience `lookback_days` can be specified, which will imply that 1d candl
         "method": "VolumePairList",
         "number_assets": 20,
         "sort_key": "quoteVolume",
+        "min_value": 0,
         "refresh_period": 86400,
         "lookback_days": 7
     }
@@ -109,6 +113,7 @@ More sophisticated approach can be used, by using `lookback_timeframe` for candl
         "method": "VolumePairList",
         "number_assets": 20,
         "sort_key": "quoteVolume",
+        "min_value": 0,
         "refresh_period": 3600,
         "lookback_timeframe": "1h",
         "lookback_period": 72

docs/index.md

@@ -47,7 +47,7 @@ Please read the [exchange specific notes](exchanges.md) to learn about eventual,
 Exchanges confirmed working by the community:
 
 - [X] [Bitvavo](https://bitvavo.com/)
-- [X] [Kukoin](https://www.kucoin.com/)
+- [X] [Kucoin](https://www.kucoin.com/)
 
 ## Requirements
 

docs/requirements-docs.txt

@@ -1,4 +1,4 @@
 mkdocs==1.2.2
-mkdocs-material==7.2.2
+mkdocs-material==7.2.4
 mdx_truly_sane_lists==1.2
 pymdown-extensions==8.2

docs/strategy-advanced.md

@@ -357,6 +357,55 @@ See [Dataframe access](#dataframe-access) for more information about dataframe u
 
 ---
 
+## Custom order price rules
+
+By default, freqtrade use the orderbook to automatically set an order price([Relevant documentation](configuration.md#prices-used-for-orders)), you also have the option to create custom order prices based on your strategy.
+
+You can use this feature by creating a `custom_entry_price()` function in your strategy file to customize entry prices and `custom_exit_price()` for exits.
+
+!!! Note
+    If your custom pricing function return None or an invalid value, price will fall back to `proposed_rate`, which is based on the regular pricing configuration.
+
+### Custom order entry and exit price example
+
+``` python
+from datetime import datetime, timedelta, timezone
+from freqtrade.persistence import Trade
+
+class AwesomeStrategy(IStrategy):
+
+    # ... populate_* methods
+
+    def custom_entry_price(self, pair: str, current_time: datetime,
+                           proposed_rate, **kwargs) -> float:
+
+        dataframe, last_updated = self.dp.get_analyzed_dataframe(pair=pair,
+                                                                timeframe=self.timeframe)
+        new_entryprice = dataframe['bollinger_10_lowerband'].iat[-1]
+        
+        return new_entryprice
+
+    def custom_exit_price(self, pair: str, trade: Trade,
+                          current_time: datetime, proposed_rate: float,
+                          current_profit: float, **kwargs) -> float:
+
+        dataframe, last_updated = self.dp.get_analyzed_dataframe(pair=pair,
+                                                                timeframe=self.timeframe)
+        new_exitprice = dataframe['bollinger_10_upperband'].iat[-1]
+        
+        return new_exitprice
+
+```
+
+!!! Warning
+    Modifying entry and exit prices will only work for limit orders. Depending on the price chosen, this can result in a lot of unfilled orders. By default the maximum allowed distance between the current price and the custom price is 2%, this value can be changed in config with the `custom_price_max_distance_ratio` parameter.
+
+!!! Example
+    If the new_entryprice is 97, the proposed_rate is 100 and the `custom_price_max_distance_ratio` is set to 2%, The retained valid custom entry price will be 98.
+
+!!! Warning "No backtesting support"
+    Custom entry-prices are currently not supported during backtesting.
+
 ## Custom order timeout rules
 
 Simple, time-based order-timeouts can be configured either via strategy or in the configuration in the `unfilledtimeout` section.

docs/strategy_analysis_example.md

@@ -228,7 +228,7 @@ graph = generate_candlestick_graph(pair=pair,
 # Show graph inline
 # graph.show()
 
-# Render graph in a seperate window
+# Render graph in a separate window
 graph.show(renderer="browser")
 
 ```

freqtrade/commands/hyperopt_commands.py

@@ -1,6 +1,6 @@
 import logging
 from operator import itemgetter
-from typing import Any, Dict, List
+from typing import Any, Dict
 
 from colorama import init as colorama_init
 
@@ -28,30 +28,12 @@ def start_hyperopt_list(args: Dict[str, Any]) -> None:
     no_details = config.get('hyperopt_list_no_details', False)
     no_header = False
 
-    filteroptions = {
-        'only_best': config.get('hyperopt_list_best', False),
-        'only_profitable': config.get('hyperopt_list_profitable', False),
-        'filter_min_trades': config.get('hyperopt_list_min_trades', 0),
-        'filter_max_trades': config.get('hyperopt_list_max_trades', 0),
-        'filter_min_avg_time': config.get('hyperopt_list_min_avg_time', None),
-        'filter_max_avg_time': config.get('hyperopt_list_max_avg_time', None),
-        'filter_min_avg_profit': config.get('hyperopt_list_min_avg_profit', None),
-        'filter_max_avg_profit': config.get('hyperopt_list_max_avg_profit', None),
-        'filter_min_total_profit': config.get('hyperopt_list_min_total_profit', None),
-        'filter_max_total_profit': config.get('hyperopt_list_max_total_profit', None),
-        'filter_min_objective': config.get('hyperopt_list_min_objective', None),
-        'filter_max_objective': config.get('hyperopt_list_max_objective', None),
-    }
-
     results_file = get_latest_hyperopt_file(
         config['user_data_dir'] / 'hyperopt_results',
         config.get('hyperoptexportfilename'))
 
     # Previous evaluations
-    epochs = HyperoptTools.load_previous_results(results_file)
-    total_epochs = len(epochs)
-
-    epochs = hyperopt_filter_epochs(epochs, filteroptions)
+    epochs, total_epochs = HyperoptTools.load_filtered_results(results_file, config)
 
     if print_colorized:
         colorama_init(autoreset=True)
@@ -59,7 +41,7 @@ def start_hyperopt_list(args: Dict[str, Any]) -> None:
     if not export_csv:
         try:
             print(HyperoptTools.get_result_table(config, epochs, total_epochs,
-                                                 not filteroptions['only_best'],
+                                                 not config.get('hyperopt_list_best', False),
                                                  print_colorized, 0))
         except KeyboardInterrupt:
             print('User interrupted..')
@@ -71,7 +53,7 @@ def start_hyperopt_list(args: Dict[str, Any]) -> None:
 
     if epochs and export_csv:
         HyperoptTools.export_csv_file(
-            config, epochs, total_epochs, not filteroptions['only_best'], export_csv
+            config, epochs, total_epochs, not config.get('hyperopt_list_best', False), export_csv
         )
 
 
@@ -91,26 +73,9 @@ def start_hyperopt_show(args: Dict[str, Any]) -> None:
 
     n = config.get('hyperopt_show_index', -1)
 
-    filteroptions = {
-        'only_best': config.get('hyperopt_list_best', False),
-        'only_profitable': config.get('hyperopt_list_profitable', False),
-        'filter_min_trades': config.get('hyperopt_list_min_trades', 0),
-        'filter_max_trades': config.get('hyperopt_list_max_trades', 0),
-        'filter_min_avg_time': config.get('hyperopt_list_min_avg_time', None),
-        'filter_max_avg_time': config.get('hyperopt_list_max_avg_time', None),
-        'filter_min_avg_profit': config.get('hyperopt_list_min_avg_profit', None),
-        'filter_max_avg_profit': config.get('hyperopt_list_max_avg_profit', None),
-        'filter_min_total_profit': config.get('hyperopt_list_min_total_profit', None),
-        'filter_max_total_profit': config.get('hyperopt_list_max_total_profit', None),
-        'filter_min_objective': config.get('hyperopt_list_min_objective', None),
-        'filter_max_objective': config.get('hyperopt_list_max_objective', None)
-    }
-
     # Previous evaluations
-    epochs = HyperoptTools.load_previous_results(results_file)
-    total_epochs = len(epochs)
+    epochs, total_epochs = HyperoptTools.load_filtered_results(results_file, config)
 
-    epochs = hyperopt_filter_epochs(epochs, filteroptions)
     filtered_epochs = len(epochs)
 
     if n > filtered_epochs:
@@ -137,138 +102,3 @@ def start_hyperopt_show(args: Dict[str, Any]) -> None:
 
         HyperoptTools.show_epoch_details(val, total_epochs, print_json, no_header,
                                          header_str="Epoch details")
-
-
-def hyperopt_filter_epochs(epochs: List, filteroptions: dict) -> List:
-    """
-    Filter our items from the list of hyperopt results
-    TODO: after 2021.5 remove all "legacy" mode queries.
-    """
-    if filteroptions['only_best']:
-        epochs = [x for x in epochs if x['is_best']]
-    if filteroptions['only_profitable']:
-        epochs = [x for x in epochs if x['results_metrics'].get(
-            'profit', x['results_metrics'].get('profit_total', 0)) > 0]
-
-    epochs = _hyperopt_filter_epochs_trade_count(epochs, filteroptions)
-
-    epochs = _hyperopt_filter_epochs_duration(epochs, filteroptions)
-
-    epochs = _hyperopt_filter_epochs_profit(epochs, filteroptions)
-
-    epochs = _hyperopt_filter_epochs_objective(epochs, filteroptions)
-
-    logger.info(f"{len(epochs)} " +
-                ("best " if filteroptions['only_best'] else "") +
-                ("profitable " if filteroptions['only_profitable'] else "") +
-                "epochs found.")
-    return epochs
-
-
-def _hyperopt_filter_epochs_trade(epochs: List, trade_count: int):
-    """
-    Filter epochs with trade-counts > trades
-    """
-    return [
-        x for x in epochs
-        if x['results_metrics'].get(
-            'trade_count', x['results_metrics'].get('total_trades', 0)
-        ) > trade_count
-    ]
-
-
-def _hyperopt_filter_epochs_trade_count(epochs: List, filteroptions: dict) -> List:
-
-    if filteroptions['filter_min_trades'] > 0:
-        epochs = _hyperopt_filter_epochs_trade(epochs, filteroptions['filter_min_trades'])
-
-    if filteroptions['filter_max_trades'] > 0:
-        epochs = [
-            x for x in epochs
-            if x['results_metrics'].get(
-                'trade_count', x['results_metrics'].get('total_trades')
-            ) < filteroptions['filter_max_trades']
-        ]
-    return epochs
-
-
-def _hyperopt_filter_epochs_duration(epochs: List, filteroptions: dict) -> List:
-
-    def get_duration_value(x):
-        # Duration in minutes ...
-        if 'duration' in x['results_metrics']:
-            return x['results_metrics']['duration']
-        else:
-            # New mode
-            if 'holding_avg_s' in x['results_metrics']:
-                avg = x['results_metrics']['holding_avg_s']
-                return avg // 60
-            raise OperationalException(
-                "Holding-average not available. Please omit the filter on average time, "
-                "or rerun hyperopt with this version")
-
-    if filteroptions['filter_min_avg_time'] is not None:
-        epochs = _hyperopt_filter_epochs_trade(epochs, 0)
-        epochs = [
-            x for x in epochs
-            if get_duration_value(x) > filteroptions['filter_min_avg_time']
-        ]
-    if filteroptions['filter_max_avg_time'] is not None:
-        epochs = _hyperopt_filter_epochs_trade(epochs, 0)
-        epochs = [
-            x for x in epochs
-            if get_duration_value(x) < filteroptions['filter_max_avg_time']
-        ]
-
-    return epochs
-
-
-def _hyperopt_filter_epochs_profit(epochs: List, filteroptions: dict) -> List:
-
-    if filteroptions['filter_min_avg_profit'] is not None:
-        epochs = _hyperopt_filter_epochs_trade(epochs, 0)
-        epochs = [
-            x for x in epochs
-            if x['results_metrics'].get(
-                'avg_profit', x['results_metrics'].get('profit_mean', 0) * 100
-            ) > filteroptions['filter_min_avg_profit']
-        ]
-    if filteroptions['filter_max_avg_profit'] is not None:
-        epochs = _hyperopt_filter_epochs_trade(epochs, 0)
-        epochs = [
-            x for x in epochs
-            if x['results_metrics'].get(
-                'avg_profit', x['results_metrics'].get('profit_mean', 0) * 100
-            ) < filteroptions['filter_max_avg_profit']
-        ]
-    if filteroptions['filter_min_total_profit'] is not None:
-        epochs = _hyperopt_filter_epochs_trade(epochs, 0)
-        epochs = [
-            x for x in epochs
-            if x['results_metrics'].get(
-                'profit', x['results_metrics'].get('profit_total_abs', 0)
-            ) > filteroptions['filter_min_total_profit']
-        ]
-    if filteroptions['filter_max_total_profit'] is not None:
-        epochs = _hyperopt_filter_epochs_trade(epochs, 0)
-        epochs = [
-            x for x in epochs
-            if x['results_metrics'].get(
-                'profit', x['results_metrics'].get('profit_total_abs', 0)
-            ) < filteroptions['filter_max_total_profit']
-        ]
-    return epochs
-
-
-def _hyperopt_filter_epochs_objective(epochs: List, filteroptions: dict) -> List:
-
-    if filteroptions['filter_min_objective'] is not None:
-        epochs = _hyperopt_filter_epochs_trade(epochs, 0)
-
-        epochs = [x for x in epochs if x['loss'] < filteroptions['filter_min_objective']]
-    if filteroptions['filter_max_objective'] is not None:
-        epochs = _hyperopt_filter_epochs_trade(epochs, 0)
-
-        epochs = [x for x in epochs if x['loss'] > filteroptions['filter_max_objective']]
-
-    return epochs

freqtrade/constants.py

@@ -191,6 +191,9 @@ CONF_SCHEMA = {
             },
             'required': ['price_side']
         },
+        'custom_price_max_distance_ratio': {
+           'type': 'number', 'minimum': 0.0
+        },
         'order_types': {
             'type': 'object',
             'properties': {

freqtrade/data/btanalysis.py

@@ -19,7 +19,7 @@ logger = logging.getLogger(__name__)
 BT_DATA_COLUMNS_OLD = ["pair", "profit_percent", "open_date", "close_date", "index",
                        "trade_duration", "open_rate", "close_rate", "open_at_end", "sell_reason"]
 
-# Mid-term format, crated by BacktestResult Named Tuple
+# Mid-term format, created by BacktestResult Named Tuple
 BT_DATA_COLUMNS_MID = ['pair', 'profit_percent', 'open_date', 'close_date', 'trade_duration',
                        'open_rate', 'close_rate', 'open_at_end', 'sell_reason', 'fee_open',
                        'fee_close', 'amount', 'profit_abs', 'profit_ratio']

freqtrade/data/converter.py

@@ -242,7 +242,7 @@ def convert_trades_format(config: Dict[str, Any], convert_from: str, convert_to:
     :param config: Config dictionary
     :param convert_from: Source format
     :param convert_to: Target format
-    :param erase: Erase souce data (does not apply if source and target format are identical)
+    :param erase: Erase source data (does not apply if source and target format are identical)
     """
     from freqtrade.data.history.idatahandler import get_datahandler
     src = get_datahandler(config['datadir'], convert_from)
@@ -267,7 +267,7 @@ def convert_ohlcv_format(config: Dict[str, Any], convert_from: str, convert_to:
     :param config: Config dictionary
     :param convert_from: Source format
     :param convert_to: Target format
-    :param erase: Erase souce data (does not apply if source and target format are identical)
+    :param erase: Erase source data (does not apply if source and target format are identical)
     """
     from freqtrade.data.history.idatahandler import get_datahandler
     src = get_datahandler(config['datadir'], convert_from)

freqtrade/data/history/history_utils.py

@@ -117,10 +117,11 @@ def refresh_data(datadir: Path,
     :param timerange: Limit data to be loaded to this timerange
     """
     data_handler = get_datahandler(datadir, data_format)
-    for pair in pairs:
-        _download_pair_history(pair=pair, timeframe=timeframe,
-                               datadir=datadir, timerange=timerange,
-                               exchange=exchange, data_handler=data_handler)
+    for idx, pair in enumerate(pairs):
+        process = f'{idx}/{len(pairs)}'
+        _download_pair_history(pair=pair, process=process,
+                               timeframe=timeframe, datadir=datadir,
+                               timerange=timerange, exchange=exchange, data_handler=data_handler)
 
 
 def _load_cached_data_for_updating(pair: str, timeframe: str, timerange: Optional[TimeRange],
@@ -153,13 +154,14 @@ def _load_cached_data_for_updating(pair: str, timeframe: str, timerange: Optiona
     return data, start_ms
 
 
-def _download_pair_history(datadir: Path,
+def _download_pair_history(pair: str, *,
+                           datadir: Path,
                            exchange: Exchange,
-                           pair: str, *,
-                           new_pairs_days: int = 30,
                            timeframe: str = '5m',
-                           timerange: Optional[TimeRange] = None,
-                           data_handler: IDataHandler = None) -> bool:
+                           process: str = '',
+                           new_pairs_days: int = 30,
+                           data_handler: IDataHandler = None,
+                           timerange: Optional[TimeRange] = None) -> bool:
     """
     Download latest candles from the exchange for the pair and timeframe passed in parameters
     The data is downloaded starting from the last correct data that
@@ -177,7 +179,7 @@ def _download_pair_history(datadir: Path,
 
     try:
         logger.info(
-            f'Download history data for pair: "{pair}", timeframe: {timeframe} '
+            f'Download history data for pair: "{pair}" ({process}), timeframe: {timeframe} '
             f'and store in {datadir}.'
         )
 
@@ -234,7 +236,7 @@ def refresh_backtest_ohlcv_data(exchange: Exchange, pairs: List[str], timeframes
     """
     pairs_not_available = []
     data_handler = get_datahandler(datadir, data_format)
-    for pair in pairs:
+    for idx, pair in enumerate(pairs, start=1):
         if pair not in exchange.markets:
             pairs_not_available.append(pair)
             logger.info(f"Skipping pair {pair}...")
@@ -247,10 +249,11 @@ def refresh_backtest_ohlcv_data(exchange: Exchange, pairs: List[str], timeframes
                         f'Deleting existing data for pair {pair}, interval {timeframe}.')
 
             logger.info(f'Downloading pair {pair}, interval {timeframe}.')
-            _download_pair_history(datadir=datadir, exchange=exchange,
-                                   pair=pair, timeframe=str(timeframe),
-                                   new_pairs_days=new_pairs_days,
-                                   timerange=timerange, data_handler=data_handler)
+            process = f'{idx}/{len(pairs)}'
+            _download_pair_history(pair=pair, process=process,
+                                   datadir=datadir, exchange=exchange,
+                                   timerange=timerange, data_handler=data_handler,
+                                   timeframe=str(timeframe), new_pairs_days=new_pairs_days)
     return pairs_not_available
 
 

freqtrade/edge/edge_positioning.py

@@ -151,7 +151,7 @@ class Edge:
         # Fake run-mode to Edge
         prior_rm = self.config['runmode']
         self.config['runmode'] = RunMode.EDGE
-        preprocessed = self.strategy.ohlcvdata_to_dataframe(data)
+        preprocessed = self.strategy.advise_all_indicators(data)
         self.config['runmode'] = prior_rm
 
         # Print timeframe

freqtrade/exchange/__init__.py

@@ -15,6 +15,7 @@ from freqtrade.exchange.exchange import (available_exchanges, ccxt_exchanges,
                                          timeframe_to_seconds, validate_exchange,
                                          validate_exchanges)
 from freqtrade.exchange.ftx import Ftx
+from freqtrade.exchange.gateio import Gateio
 from freqtrade.exchange.hitbtc import Hitbtc
 from freqtrade.exchange.kraken import Kraken
 from freqtrade.exchange.kucoin import Kucoin

freqtrade/exchange/exchange.py

@@ -618,6 +618,8 @@ class Exchange:
         if self.exchange_has('fetchL2OrderBook'):
             ob = self.fetch_l2_order_book(pair, 20)
             ob_type = 'asks' if side == 'buy' else 'bids'
+            slippage = 0.05
+            max_slippage_val = rate * ((1 + slippage) if side == 'buy' else (1 - slippage))
 
             remaining_amount = amount
             filled_amount = 0
@@ -626,7 +628,9 @@ class Exchange:
                 book_entry_coin_volume = book_entry[1]
                 if remaining_amount > 0:
                     if remaining_amount < book_entry_coin_volume:
+                        # Orderbook at this slot bigger than remaining amount
                         filled_amount += remaining_amount * book_entry_price
+                        break
                     else:
                         filled_amount += book_entry_coin_volume * book_entry_price
                     remaining_amount -= book_entry_coin_volume
@@ -635,7 +639,14 @@ class Exchange:
             else:
                 # If remaining_amount wasn't consumed completely (break was not called)
                 filled_amount += remaining_amount * book_entry_price
-            forecast_avg_filled_price = filled_amount / amount
+            forecast_avg_filled_price = max(filled_amount, 0) / amount
+            # Limit max. slippage to specified value
+            if side == 'buy':
+                forecast_avg_filled_price = min(forecast_avg_filled_price, max_slippage_val)
+
+            else:
+                forecast_avg_filled_price = max(forecast_avg_filled_price, max_slippage_val)
+
             return self.price_to_precision(pair, forecast_avg_filled_price)
 
         return rate
@@ -1242,7 +1253,7 @@ class Exchange:
         logger.debug("Refreshing candle (OHLCV) data for %d pairs", len(pair_list))
 
         input_coroutines = []
-
+        cached_pairs = []
         # Gather coroutines to run
         for pair, timeframe in set(pair_list):
             if (((pair, timeframe) not in self._klines)
@@ -1254,6 +1265,7 @@ class Exchange:
                     "Using cached candle (OHLCV) data for pair %s, timeframe %s ...",
                     pair, timeframe
                 )
+                cached_pairs.append((pair, timeframe))
 
         results = asyncio.get_event_loop().run_until_complete(
             asyncio.gather(*input_coroutines, return_exceptions=True))
@@ -1276,6 +1288,10 @@ class Exchange:
             results_df[(pair, timeframe)] = ohlcv_df
             if cache:
                 self._klines[(pair, timeframe)] = ohlcv_df
+        # Return cached klines
+        for pair, timeframe in cached_pairs:
+            results_df[(pair, timeframe)] = self.klines((pair, timeframe), copy=False)
+
         return results_df
 
     def _now_is_time_to_refresh(self, pair: str, timeframe: str) -> bool:
@@ -1486,7 +1502,7 @@ class Exchange:
         :returns List of trade data
         """
         if not self.exchange_has("fetchTrades"):
-            raise OperationalException("This exchange does not suport downloading Trades.")
+            raise OperationalException("This exchange does not support downloading Trades.")
 
         return asyncio.get_event_loop().run_until_complete(
             self._async_get_trade_history(pair=pair, since=since,

freqtrade/exchange/gateio.py

@@ -0,0 +1,23 @@
+""" Gate.io exchange subclass """
+import logging
+from typing import Dict
+
+from freqtrade.exchange import Exchange
+
+
+logger = logging.getLogger(__name__)
+
+
+class Gateio(Exchange):
+    """
+    Gate.io exchange class. Contains adjustments needed for Freqtrade to work
+    with this exchange.
+
+    Please note that this exchange is not included in the list of exchanges
+    officially supported by the Freqtrade development team. So some features
+    may still not work as expected.
+    """
+
+    _ft_has: Dict = {
+        "ohlcv_candle_limit": 1000,
+    }

freqtrade/freqtradebot.py

@@ -479,7 +479,13 @@ class FreqtradeBot(LoggingMixin):
             buy_limit_requested = price
         else:
             # Calculate price
-            buy_limit_requested = self.exchange.get_rate(pair, refresh=True, side="buy")
+            proposed_buy_rate = self.exchange.get_rate(pair, refresh=True, side="buy")
+            custom_entry_price = strategy_safe_wrapper(self.strategy.custom_entry_price,
+                                                       default_retval=proposed_buy_rate)(
+                pair=pair, current_time=datetime.now(timezone.utc),
+                proposed_rate=proposed_buy_rate)
+
+            buy_limit_requested = self.get_valid_price(custom_entry_price, proposed_buy_rate)
 
         if not buy_limit_requested:
             raise PricingError('Could not determine buy price.')
@@ -977,7 +983,7 @@ class FreqtradeBot(LoggingMixin):
             # if trade is partially complete, edit the stake details for the trade
             # and close the order
             # cancel_order may not contain the full order dict, so we need to fallback
-            # to the order dict aquired before cancelling.
+            # to the order dict acquired before cancelling.
             # we need to fall back to the values from order if corder does not contain these keys.
             trade.amount = filled_amount
             trade.stake_amount = trade.amount * trade.open_rate
@@ -1076,6 +1082,17 @@ class FreqtradeBot(LoggingMixin):
            and self.strategy.order_types['stoploss_on_exchange']:
             limit = trade.stop_loss
 
+        # set custom_exit_price if available
+        proposed_limit_rate = limit
+        current_profit = trade.calc_profit_ratio(limit)
+        custom_exit_price = strategy_safe_wrapper(self.strategy.custom_exit_price,
+                                                  default_retval=proposed_limit_rate)(
+            pair=trade.pair, trade=trade,
+            current_time=datetime.now(timezone.utc),
+            proposed_rate=proposed_limit_rate, current_profit=current_profit)
+
+        limit = self.get_valid_price(custom_exit_price, proposed_limit_rate)
+
         # First cancelling stoploss on exchange ...
         if self.strategy.order_types.get('stoploss_on_exchange') and trade.stoploss_order_id:
             try:
@@ -1364,6 +1381,8 @@ class FreqtradeBot(LoggingMixin):
         if fee_currency:
             # fee_rate should use mean
             fee_rate = sum(fee_rate_array) / float(len(fee_rate_array)) if fee_rate_array else None
+            if fee_rate is not None and fee_rate < 0.02:
+                # Only update if fee-rate is < 2%
                 trade.update_fee(fee_cost, fee_currency, fee_rate, order.get('side', ''))
 
         if not isclose(amount, order_amount, abs_tol=constants.MATH_CLOSE_PREC):
@@ -1375,3 +1394,26 @@ class FreqtradeBot(LoggingMixin):
                                               amount=amount, fee_abs=fee_abs)
         else:
             return amount
+
+    def get_valid_price(self, custom_price: float, proposed_price: float) -> float:
+        """
+        Return the valid price.
+        Check if the custom price is of the good type if not return proposed_price
+        :return: valid price for the order
+        """
+        if custom_price:
+            try:
+                valid_custom_price = float(custom_price)
+            except ValueError:
+                valid_custom_price = proposed_price
+        else:
+            valid_custom_price = proposed_price
+
+        cust_p_max_dist_r = self.config.get('custom_price_max_distance_ratio', 0.02)
+        min_custom_price_allowed = proposed_price - (proposed_price * cust_p_max_dist_r)
+        max_custom_price_allowed = proposed_price + (proposed_price * cust_p_max_dist_r)
+
+        # Bracket between min_custom_price_allowed and max_custom_price_allowed
+        return max(
+            min(valid_custom_price, max_custom_price_allowed),
+            min_custom_price_allowed)

freqtrade/optimize/backtesting.py

@@ -133,6 +133,9 @@ class Backtesting:
         self.abort = False
 
     def __del__(self):
+        self.cleanup()
+
+    def cleanup(self):
         LoggingMixin.show_output = True
         PairLocks.use_db = True
         Trade.use_db = True
@@ -219,7 +222,7 @@ class Backtesting:
         # Every change to this headers list must evaluate further usages of the resulting tuple
         # and eventually change the constants for indexes at the top
         headers = ['date', 'open', 'high', 'low', 'close', 'enter_long', 'exit_long',
-                   'enter_short', 'exit_short']
+                   'enter_short', 'exit_short', 'long_tag', 'short_tag']
         data: Dict = {}
         self.progress.init_step(BacktestState.CONVERT, len(processed))
 
@@ -227,20 +230,14 @@ class Backtesting:
         for pair, pair_data in processed.items():
             self.check_abort()
             self.progress.increment()
-            has_buy_tag = 'long_tag' in pair_data
-            has_short_tag = 'short_tag' in pair_data
-            headers = headers + ['long_tag'] if has_buy_tag else headers
-            headers = headers + ['short_tag'] if has_short_tag else headers
+
             if not pair_data.empty:
                 # Cleanup from prior runs
                 pair_data.loc[:, 'buy'] = 0  # TODO: Should be renamed to enter_long
                 pair_data.loc[:, 'enter_short'] = 0
                 pair_data.loc[:, 'sell'] = 0  # TODO: should be renamed to exit_long
                 pair_data.loc[:, 'exit_short'] = 0
-                # pair_data.loc[:, 'sell'] = 0
-                if has_buy_tag:
                 pair_data.loc[:, 'long_tag'] = None  # cleanup if buy_tag is exist
-                if has_short_tag:
                 pair_data.loc[:, 'short_tag'] = None  # cleanup if short_tag is exist
 
             df_analyzed = self.strategy.advise_sell(
@@ -256,7 +253,6 @@ class Backtesting:
             df_analyzed.loc[:, 'enter_short'] = df_analyzed.loc[:, 'enter_short'].shift(1)
             df_analyzed.loc[:, 'exit_long'] = df_analyzed.loc[:, 'exit_long'].shift(1)
             df_analyzed.loc[:, 'exit_short'] = df_analyzed.loc[:, 'exit_short'].shift(1)
-            if has_buy_tag:
             df_analyzed.loc[:, 'long_tag'] = df_analyzed.loc[:, 'long_tag'].shift(1)
 
             df_analyzed.drop(df_analyzed.head(1).index, inplace=True)
@@ -264,6 +260,8 @@ class Backtesting:
             # Update dataprovider cache
             self.dataprovider._set_cached_df(pair, self.timeframe, df_analyzed)
 
+            df_analyzed = df_analyzed.drop(df_analyzed.head(1).index)
+
             # Convert from Pandas to list for performance reasons
             # (Looping Pandas is slow.)
             data[pair] = df_analyzed[headers].values.tolist()
@@ -337,13 +335,14 @@ class Backtesting:
 
     def _get_sell_trade_entry(self, trade: LocalTrade, sell_row: Tuple) -> Optional[LocalTrade]:
         # TODO: short exits
+        sell_candle_time = sell_row[DATE_IDX].to_pydatetime()
         sell = self.strategy.should_sell(trade, sell_row[OPEN_IDX],  # type: ignore
-                                         sell_row[DATE_IDX].to_pydatetime(), sell_row[BUY_IDX],
+                                         sell_candle_time, sell_row[BUY_IDX],
                                          sell_row[SELL_IDX],
                                          low=sell_row[LOW_IDX], high=sell_row[HIGH_IDX])
 
         if sell.sell_flag:
-            trade.close_date = sell_row[DATE_IDX].to_pydatetime()
+            trade.close_date = sell_candle_time
             trade.sell_reason = sell.sell_reason
             trade_dur = int((trade.close_date_utc - trade.open_date_utc).total_seconds() // 60)
             closerate = self._get_close_rate(sell_row, trade, sell, trade_dur)
@@ -355,7 +354,7 @@ class Backtesting:
                     rate=closerate,
                     time_in_force=time_in_force,
                     sell_reason=sell.sell_reason,
-                    current_time=sell_row[DATE_IDX].to_pydatetime()):
+                    current_time=sell_candle_time):
                 return None
 
             trade.close(closerate, show_msg=False)
@@ -494,6 +493,8 @@ class Backtesting:
             for i, pair in enumerate(data):
                 row_index = indexes[pair]
                 try:
+                    # Row is treated as "current incomplete candle".
+                    # Buy / sell signals are shifted by 1 to compensate for this.
                     row = data[pair][row_index]
                 except IndexError:
                     # missing Data for one pair at the end.
@@ -505,8 +506,8 @@ class Backtesting:
                     continue
 
                 row_index += 1
-                self.dataprovider._set_dataframe_max_index(row_index)
                 indexes[pair] = row_index
+                self.dataprovider._set_dataframe_max_index(row_index)
 
                 # without positionstacking, we can only have one open trade per pair.
                 # max_open_trades must be respected
@@ -530,7 +531,7 @@ class Backtesting:
                         open_trades[pair].append(trade)
                         LocalTrade.add_bt_trade(trade)
 
-                for trade in open_trades[pair]:
+                for trade in list(open_trades[pair]):
                     # also check the buying candle for sell conditions.
                     trade_entry = self._get_sell_trade_entry(trade, row)
                     # Sell occurred
@@ -561,7 +562,8 @@ class Backtesting:
             'final_balance': self.wallets.get_total(self.strategy.config['stake_currency']),
         }
 
-    def backtest_one_strategy(self, strat: IStrategy, data: Dict[str, Any], timerange: TimeRange):
+    def backtest_one_strategy(self, strat: IStrategy, data: Dict[str, DataFrame],
+                              timerange: TimeRange):
         self.progress.init_step(BacktestState.ANALYZE, 0)
 
         logger.info("Running backtesting for Strategy %s", strat.get_strategy_name())
@@ -580,7 +582,7 @@ class Backtesting:
             max_open_trades = 0
 
         # need to reprocess data every time to populate signals
-        preprocessed = self.strategy.ohlcvdata_to_dataframe(data)
+        preprocessed = self.strategy.advise_all_indicators(data)
 
         # Trim startup period from analyzed dataframe
         preprocessed_tmp = trim_dataframes(preprocessed, timerange, self.required_startup)

freqtrade/optimize/hyperopt.py

@@ -394,7 +394,7 @@ class Hyperopt:
         data, timerange = self.backtesting.load_bt_data()
         logger.info("Dataload complete. Calculating indicators")
 
-        preprocessed = self.backtesting.strategy.ohlcvdata_to_dataframe(data)
+        preprocessed = self.backtesting.strategy.advise_all_indicators(data)
 
         # Trim startup period from analyzed dataframe to get correct dates for output.
         processed = trim_dataframes(preprocessed, timerange, self.backtesting.required_startup)

freqtrade/optimize/hyperopt_epoch_filters.py

@@ -0,0 +1,128 @@
+import logging
+from typing import List
+
+from freqtrade.exceptions import OperationalException
+
+
+logger = logging.getLogger(__name__)
+
+
+def hyperopt_filter_epochs(epochs: List, filteroptions: dict, log: bool = True) -> List:
+    """
+    Filter our items from the list of hyperopt results
+    """
+    if filteroptions['only_best']:
+        epochs = [x for x in epochs if x['is_best']]
+    if filteroptions['only_profitable']:
+        epochs = [x for x in epochs
+                  if x['results_metrics'].get('profit_total', 0) > 0]
+
+    epochs = _hyperopt_filter_epochs_trade_count(epochs, filteroptions)
+
+    epochs = _hyperopt_filter_epochs_duration(epochs, filteroptions)
+
+    epochs = _hyperopt_filter_epochs_profit(epochs, filteroptions)
+
+    epochs = _hyperopt_filter_epochs_objective(epochs, filteroptions)
+    if log:
+        logger.info(f"{len(epochs)} " +
+                    ("best " if filteroptions['only_best'] else "") +
+                    ("profitable " if filteroptions['only_profitable'] else "") +
+                    "epochs found.")
+    return epochs
+
+
+def _hyperopt_filter_epochs_trade(epochs: List, trade_count: int):
+    """
+    Filter epochs with trade-counts > trades
+    """
+    return [
+        x for x in epochs if x['results_metrics'].get('total_trades', 0) > trade_count
+    ]
+
+
+def _hyperopt_filter_epochs_trade_count(epochs: List, filteroptions: dict) -> List:
+
+    if filteroptions['filter_min_trades'] > 0:
+        epochs = _hyperopt_filter_epochs_trade(epochs, filteroptions['filter_min_trades'])
+
+    if filteroptions['filter_max_trades'] > 0:
+        epochs = [
+            x for x in epochs
+            if x['results_metrics'].get('total_trades') < filteroptions['filter_max_trades']
+        ]
+    return epochs
+
+
+def _hyperopt_filter_epochs_duration(epochs: List, filteroptions: dict) -> List:
+
+    def get_duration_value(x):
+        # Duration in minutes ...
+        if 'holding_avg_s' in x['results_metrics']:
+            avg = x['results_metrics']['holding_avg_s']
+            return avg // 60
+        raise OperationalException(
+            "Holding-average not available. Please omit the filter on average time, "
+            "or rerun hyperopt with this version")
+
+    if filteroptions['filter_min_avg_time'] is not None:
+        epochs = _hyperopt_filter_epochs_trade(epochs, 0)
+        epochs = [
+            x for x in epochs
+            if get_duration_value(x) > filteroptions['filter_min_avg_time']
+        ]
+    if filteroptions['filter_max_avg_time'] is not None:
+        epochs = _hyperopt_filter_epochs_trade(epochs, 0)
+        epochs = [
+            x for x in epochs
+            if get_duration_value(x) < filteroptions['filter_max_avg_time']
+        ]
+
+    return epochs
+
+
+def _hyperopt_filter_epochs_profit(epochs: List, filteroptions: dict) -> List:
+
+    if filteroptions['filter_min_avg_profit'] is not None:
+        epochs = _hyperopt_filter_epochs_trade(epochs, 0)
+        epochs = [
+            x for x in epochs
+            if x['results_metrics'].get('profit_mean', 0) * 100
+            > filteroptions['filter_min_avg_profit']
+        ]
+    if filteroptions['filter_max_avg_profit'] is not None:
+        epochs = _hyperopt_filter_epochs_trade(epochs, 0)
+        epochs = [
+            x for x in epochs
+            if x['results_metrics'].get('profit_mean', 0) * 100
+            < filteroptions['filter_max_avg_profit']
+        ]
+    if filteroptions['filter_min_total_profit'] is not None:
+        epochs = _hyperopt_filter_epochs_trade(epochs, 0)
+        epochs = [
+            x for x in epochs
+            if x['results_metrics'].get('profit_total_abs', 0)
+            > filteroptions['filter_min_total_profit']
+        ]
+    if filteroptions['filter_max_total_profit'] is not None:
+        epochs = _hyperopt_filter_epochs_trade(epochs, 0)
+        epochs = [
+            x for x in epochs
+            if x['results_metrics'].get('profit_total_abs', 0)
+            < filteroptions['filter_max_total_profit']
+        ]
+    return epochs
+
+
+def _hyperopt_filter_epochs_objective(epochs: List, filteroptions: dict) -> List:
+
+    if filteroptions['filter_min_objective'] is not None:
+        epochs = _hyperopt_filter_epochs_trade(epochs, 0)
+
+        epochs = [x for x in epochs if x['loss'] < filteroptions['filter_min_objective']]
+    if filteroptions['filter_max_objective'] is not None:
+        epochs = _hyperopt_filter_epochs_trade(epochs, 0)
+
+        epochs = [x for x in epochs if x['loss'] > filteroptions['filter_max_objective']]
+
+    return epochs

freqtrade/optimize/hyperopt_tools.py

@@ -4,7 +4,7 @@ import logging
 from copy import deepcopy
 from datetime import datetime, timezone
 from pathlib import Path
-from typing import Any, Dict, List, Optional
+from typing import Any, Dict, Iterator, List, Optional, Tuple
 
 import numpy as np
 import rapidjson
@@ -15,6 +15,7 @@ from pandas import isna, json_normalize
 from freqtrade.constants import FTHYPT_FILEVERSION, USERPATH_STRATEGIES
 from freqtrade.exceptions import OperationalException
 from freqtrade.misc import deep_merge_dicts, round_coin_value, round_dict, safe_value_fallback2
+from freqtrade.optimize.hyperopt_epoch_filters import hyperopt_filter_epochs
 
 
 logger = logging.getLogger(__name__)
@@ -89,46 +90,70 @@ class HyperoptTools():
             return any(s in config['spaces'] for s in [space, 'all', 'default'])
 
     @staticmethod
-    def _read_results_pickle(results_file: Path) -> List:
+    def _read_results(results_file: Path, batch_size: int = 10) -> Iterator[List[Any]]:
         """
-        Read hyperopt results from pickle file
-        LEGACY method - new files are written as json and cannot be read with this method.
-        """
-        from joblib import load
-
-        logger.info(f"Reading pickled epochs from '{results_file}'")
-        data = load(results_file)
-        return data
-
-    @staticmethod
-    def _read_results(results_file: Path) -> List:
-        """
-        Read hyperopt results from file
+        Stream hyperopt results from file
         """
         import rapidjson
         logger.info(f"Reading epochs from '{results_file}'")
         with results_file.open('r') as f:
-            data = [rapidjson.loads(line) for line in f]
-        return data
+            data = []
+            for line in f:
+                data += [rapidjson.loads(line)]
+                if len(data) >= batch_size:
+                    yield data
+                    data = []
+        yield data
 
     @staticmethod
-    def load_previous_results(results_file: Path) -> List:
-        """
-        Load data for epochs from the file if we have one
-        """
-        epochs: List = []
+    def _test_hyperopt_results_exist(results_file) -> bool:
         if results_file.is_file() and results_file.stat().st_size > 0:
             if results_file.suffix == '.pickle':
-                epochs = HyperoptTools._read_results_pickle(results_file)
+                raise OperationalException(
+                    "Legacy hyperopt results are no longer supported."
+                    "Please rerun hyperopt or use an older version to load this file."
+                )
+            return True
         else:
-                epochs = HyperoptTools._read_results(results_file)
-            # Detection of some old format, without 'is_best' field saved
-            if epochs[0].get('is_best') is None:
+            # No file found.
+            return False
+
+    @staticmethod
+    def load_filtered_results(results_file: Path, config: Dict[str, Any]) -> Tuple[List, int]:
+        filteroptions = {
+            'only_best': config.get('hyperopt_list_best', False),
+            'only_profitable': config.get('hyperopt_list_profitable', False),
+            'filter_min_trades': config.get('hyperopt_list_min_trades', 0),
+            'filter_max_trades': config.get('hyperopt_list_max_trades', 0),
+            'filter_min_avg_time': config.get('hyperopt_list_min_avg_time', None),
+            'filter_max_avg_time': config.get('hyperopt_list_max_avg_time', None),
+            'filter_min_avg_profit': config.get('hyperopt_list_min_avg_profit', None),
+            'filter_max_avg_profit': config.get('hyperopt_list_max_avg_profit', None),
+            'filter_min_total_profit': config.get('hyperopt_list_min_total_profit', None),
+            'filter_max_total_profit': config.get('hyperopt_list_max_total_profit', None),
+            'filter_min_objective': config.get('hyperopt_list_min_objective', None),
+            'filter_max_objective': config.get('hyperopt_list_max_objective', None),
+        }
+        if not HyperoptTools._test_hyperopt_results_exist(results_file):
+            # No file found.
+            return [], 0
+
+        epochs = []
+        total_epochs = 0
+        for epochs_tmp in HyperoptTools._read_results(results_file):
+            if total_epochs == 0 and epochs_tmp[0].get('is_best') is None:
                 raise OperationalException(
                     "The file with HyperoptTools results is incompatible with this version "
                     "of Freqtrade and cannot be loaded.")
-            logger.info(f"Loaded {len(epochs)} previous evaluations from disk.")
-        return epochs
+            total_epochs += len(epochs_tmp)
+            epochs += hyperopt_filter_epochs(epochs_tmp, filteroptions, log=False)
+
+        logger.info(f"Loaded {total_epochs} previous evaluations from disk.")
+
+        # Final filter run ...
+        epochs = hyperopt_filter_epochs(epochs, filteroptions, log=True)
+
+        return epochs, total_epochs
 
     @staticmethod
     def show_epoch_details(results, total_epochs: int, print_json: bool,
@@ -433,7 +458,6 @@ class HyperoptTools():
         trials['Best'] = ''
         trials['Stake currency'] = config['stake_currency']
 
-        if 'results_metrics.total_trades' in trials:
         base_metrics = ['Best', 'current_epoch', 'results_metrics.total_trades',
                         'results_metrics.profit_mean', 'results_metrics.profit_median',
                         'results_metrics.profit_total',
@@ -441,13 +465,7 @@ class HyperoptTools():
                         'results_metrics.profit_total_abs', 'results_metrics.holding_avg',
                         'loss', 'is_initial_point', 'is_best']
         perc_multi = 100
-        else:
-            perc_multi = 1
-            base_metrics = ['Best', 'current_epoch', 'results_metrics.trade_count',
-                            'results_metrics.avg_profit', 'results_metrics.median_profit',
-                            'results_metrics.total_profit',
-                            'Stake currency', 'results_metrics.profit', 'results_metrics.duration',
-                            'loss', 'is_initial_point', 'is_best']
+
         param_metrics = [("params_dict."+param) for param in results[0]['params_dict'].keys()]
         trials = trials[base_metrics + param_metrics]
 
@@ -475,11 +493,6 @@ class HyperoptTools():
         trials['Avg profit'] = trials['Avg profit'].apply(
             lambda x: f'{x * perc_multi:,.2f}%' if not isna(x) else ""
         )
-        if perc_multi == 1:
-            trials['Avg duration'] = trials['Avg duration'].apply(
-                lambda x: f'{x:,.1f} m' if isinstance(
-                    x, float) else f"{x.total_seconds() // 60:,.1f} m" if not isna(x) else ""
-            )
         trials['Objective'] = trials['Objective'].apply(
             lambda x: f'{x:,.5f}' if x != 100000 else ""
         )

freqtrade/persistence/models.py

@@ -166,7 +166,7 @@ class Order(_DECL_BASE):
         self.ft_is_open = True
         if self.status in ('closed', 'canceled', 'cancelled'):
             self.ft_is_open = False
-            if order.get('filled', 0) > 0:
+            if (order.get('filled', 0.0) or 0.0) > 0:
                 self.order_filled_date = datetime.now(timezone.utc)
         self.order_update_date = datetime.now(timezone.utc)
 
@@ -451,12 +451,12 @@ class LocalTrade():
         LocalTrade.trades_open = []
         LocalTrade.total_profit = 0
 
-    def adjust_min_max_rates(self, current_price: float) -> None:
+    def adjust_min_max_rates(self, current_price: float, current_price_low: float) -> None:
         """
         Adjust the max_rate and min_rate.
         """
         self.max_rate = max(current_price, self.max_rate or self.open_rate)
-        self.min_rate = min(current_price, self.min_rate or self.open_rate)
+        self.min_rate = min(current_price_low, self.min_rate or self.open_rate)
 
     def adjust_stop_loss(self, current_price: float, stoploss: float,
                          initial: bool = False) -> None:

freqtrade/plot/plotting.py

@@ -538,7 +538,7 @@ def load_and_plot_trades(config: Dict[str, Any]):
     - Initializes plot-script
     - Get candle (OHLCV) data
     - Generate Dafaframes populated with indicators and signals based on configured strategy
-    - Load trades excecuted during the selected period
+    - Load trades executed during the selected period
     - Generate Plotly plot objects
     - Generate plot files
     :return: None

freqtrade/plugins/pairlist/IPairList.py

@@ -150,18 +150,20 @@ class IPairList(LoggingMixin, ABC):
         for pair in pairlist:
             # pair is not in the generated dynamic market or has the wrong stake currency
             if pair not in markets:
-                logger.warning(f"Pair {pair} is not compatible with exchange "
-                               f"{self._exchange.name}. Removing it from whitelist..")
+                self.log_once(f"Pair {pair} is not compatible with exchange "
+                              f"{self._exchange.name}. Removing it from whitelist..",
+                              logger.warning)
                 continue
 
             if not self._exchange.market_is_tradable(markets[pair]):
-                logger.warning(f"Pair {pair} is not tradable with Freqtrade."
-                               "Removing it from whitelist..")
+                self.log_once(f"Pair {pair} is not tradable with Freqtrade."
+                              "Removing it from whitelist..", logger.warning)
                 continue
 
             if self._exchange.get_pair_quote_currency(pair) != self._config['stake_currency']:
-                logger.warning(f"Pair {pair} is not compatible with your stake currency "
-                               f"{self._config['stake_currency']}. Removing it from whitelist..")
+                self.log_once(f"Pair {pair} is not compatible with your stake currency "
+                              f"{self._config['stake_currency']}. Removing it from whitelist..",
+                              logger.warning)
                 continue
 
             # Check if market is active

freqtrade/plugins/pairlist/VolumePairList.py

@@ -4,6 +4,7 @@ Volume PairList provider
 Provides dynamic pair list based on trade volumes
 """
 import logging
+from functools import partial
 from typing import Any, Dict, List
 
 import arrow
@@ -115,7 +116,7 @@ class VolumePairList(IPairList):
         pairlist = self._pair_cache.get('pairlist')
         if pairlist:
             # Item found - no refresh necessary
-            return pairlist
+            return pairlist.copy()
         else:
             # Use fresh pairlist
             # Check if pair quote currency equals to the stake currency.
@@ -126,7 +127,7 @@ class VolumePairList(IPairList):
             pairlist = [s['symbol'] for s in filtered_tickers]
 
             pairlist = self.filter_pairlist(pairlist, tickers)
-            self._pair_cache['pairlist'] = pairlist
+            self._pair_cache['pairlist'] = pairlist.copy()
 
         return pairlist
 
@@ -203,7 +204,7 @@ class VolumePairList(IPairList):
 
         # Validate whitelist to only have active market pairs
         pairs = self._whitelist_for_active_markets([s['symbol'] for s in sorted_tickers])
-        pairs = self.verify_blacklist(pairs, logger.info)
+        pairs = self.verify_blacklist(pairs, partial(self.log_once, logmethod=logger.info))
         # Limit pairlist to the requested number of pairs
         pairs = pairs[:self._number_pairs]
 

freqtrade/plugins/pairlist/rangestabilityfilter.py

@@ -120,5 +120,6 @@ class RangeStabilityFilter(IPairList):
                         logger.info)
                     result = False
             self._pair_cache[pair] = result
-
+        else:
+            self.log_once(f"Removed {pair} from whitelist, no candles found.", logger.info)
         return result

freqtrade/rpc/api_server/api_v1.py

@@ -223,11 +223,11 @@ def list_strategies(config=Depends(get_config)):
 @router.get('/strategy/{strategy}', response_model=StrategyResponse, tags=['strategy'])
 def get_strategy(strategy: str, config=Depends(get_config)):
 
-    config = deepcopy(config)
+    config_ = deepcopy(config)
     from freqtrade.resolvers.strategy_resolver import StrategyResolver
     try:
-        strategy_obj = StrategyResolver._load_strategy(strategy, config,
-                                                       extra_dir=config.get('strategy_path'))
+        strategy_obj = StrategyResolver._load_strategy(strategy, config_,
+                                                       extra_dir=config_.get('strategy_path'))
     except OperationalException:
         raise HTTPException(status_code=404, detail='Strategy not found')
 

freqtrade/rpc/api_server/uvicorn_threaded.py

@@ -32,8 +32,11 @@ class UvicornServer(uvicorn.Server):
             asyncio_setup()
         else:
             asyncio.set_event_loop(uvloop.new_event_loop())
-
+        try:
             loop = asyncio.get_event_loop()
+        except RuntimeError:
+            # When running in a thread, we'll not have an eventloop yet.
+            loop = asyncio.new_event_loop()
         loop.run_until_complete(self.serve(sockets=sockets))
 
     @contextlib.contextmanager

freqtrade/rpc/api_server/web_ui.py

@@ -29,6 +29,16 @@ async def ui_version():
     }
 
 
+def is_relative_to(path, base) -> bool:
+    # Helper function simulating behaviour of is_relative_to, which was only added in python 3.9
+    try:
+        path.relative_to(base)
+        return True
+    except ValueError:
+        pass
+    return False
+
+
 @router_ui.get('/{rest_of_path:path}', include_in_schema=False)
 async def index_html(rest_of_path: str):
     """
@@ -37,8 +47,11 @@ async def index_html(rest_of_path: str):
     if rest_of_path.startswith('api') or rest_of_path.startswith('.'):
         raise HTTPException(status_code=404, detail="Not Found")
     uibase = Path(__file__).parent / 'ui/installed/'
-    if (uibase / rest_of_path).is_file():
-        return FileResponse(str(uibase / rest_of_path))
+    filename = uibase / rest_of_path
+    # It's security relevant to check "relative_to".
+    # Without this, Directory-traversal is possible.
+    if filename.is_file() and is_relative_to(filename, uibase):
+        return FileResponse(str(filename))
 
     index_file = uibase / 'index.html'
     if not index_file.is_file():

freqtrade/rpc/fiat_convert.py

@@ -5,7 +5,7 @@ e.g BTC to USD
 
 import datetime
 import logging
-from typing import Dict
+from typing import Dict, List
 
 from cachetools.ttl import TTLCache
 from pycoingecko import CoinGeckoAPI
@@ -25,8 +25,7 @@ class CryptoToFiatConverter:
     """
     __instance = None
     _coingekko: CoinGeckoAPI = None
-
-    _cryptomap: Dict = {}
+    _coinlistings: List[Dict] = []
     _backoff: float = 0.0
 
     def __new__(cls):
@@ -49,9 +48,8 @@ class CryptoToFiatConverter:
 
     def _load_cryptomap(self) -> None:
         try:
-            coinlistings = self._coingekko.get_coins_list()
-            # Create mapping table from symbol to coingekko_id
-            self._cryptomap = {x['symbol']: x['id'] for x in coinlistings}
+            # Use list-comprehension to ensure we get a list.
+            self._coinlistings = [x for x in self._coingekko.get_coins_list()]
         except RequestException as request_exception:
             if "429" in str(request_exception):
                 logger.warning(
@@ -69,6 +67,24 @@ class CryptoToFiatConverter:
             logger.error(
                 f"Could not load FIAT Cryptocurrency map for the following problem: {exception}")
 
+    def _get_gekko_id(self, crypto_symbol):
+        if not self._coinlistings:
+            if self._backoff <= datetime.datetime.now().timestamp():
+                self._load_cryptomap()
+                # Still not loaded.
+                if not self._coinlistings:
+                    return None
+            else:
+                return None
+        found = [x for x in self._coinlistings if x['symbol'] == crypto_symbol]
+        if len(found) == 1:
+            return found[0]['id']
+
+        if len(found) > 0:
+            # Wrong!
+            logger.warning(f"Found multiple mappings in goingekko for {crypto_symbol}.")
+            return None
+
     def convert_amount(self, crypto_amount: float, crypto_symbol: str, fiat_symbol: str) -> float:
         """
         Convert an amount of crypto-currency to fiat
@@ -143,22 +159,14 @@ class CryptoToFiatConverter:
         if crypto_symbol == fiat_symbol:
             return 1.0
 
-        if self._cryptomap == {}:
-            if self._backoff <= datetime.datetime.now().timestamp():
-                self._load_cryptomap()
-                # return 0.0 if we still don't have data to check, no reason to proceed
-                if self._cryptomap == {}:
-                    return 0.0
-            else:
-                return 0.0
+        _gekko_id = self._get_gekko_id(crypto_symbol)
 
-        if crypto_symbol not in self._cryptomap:
+        if not _gekko_id:
             # return 0 for unsupported stake currencies (fiat-convert should not break the bot)
             logger.warning("unsupported crypto-symbol %s - returning 0.0", crypto_symbol)
             return 0.0
 
         try:
-            _gekko_id = self._cryptomap[crypto_symbol]
             return float(
                 self._coingekko.get_price(
                     ids=_gekko_id,

freqtrade/rpc/rpc.py

@@ -776,7 +776,7 @@ class RPC:
         if has_content:
 
             dataframe.loc[:, '__date_ts'] = dataframe.loc[:, 'date'].view(int64) // 1000 // 1000
-            # Move open to seperate column when signal for easy plotting
+            # Move open to separate column when signal for easy plotting
             if 'buy' in dataframe.columns:
                 buy_mask = (dataframe['buy'] == 1)
                 buy_signals = int(buy_mask.sum())

freqtrade/strategy/interface.py

@@ -281,6 +281,43 @@ class IStrategy(ABC, HyperStrategyMixin):
         """
         return self.stoploss
 
+    def custom_entry_price(self, pair: str, current_time: datetime, proposed_rate: float,
+                           **kwargs) -> float:
+        """
+        Custom entry price logic, returning the new entry price.
+
+        For full documentation please go to https://www.freqtrade.io/en/latest/strategy-advanced/
+
+        When not implemented by a strategy, returns None, orderbook is used to set entry price
+
+        :param pair: Pair that's currently analyzed
+        :param current_time: datetime object, containing the current datetime
+        :param proposed_rate: Rate, calculated based on pricing settings in ask_strategy.
+        :param **kwargs: Ensure to keep this here so updates to this won't break your strategy.
+        :return float: New entry price value if provided
+        """
+        return proposed_rate
+
+    def custom_exit_price(self, pair: str, trade: Trade,
+                          current_time: datetime, proposed_rate: float,
+                          current_profit: float, **kwargs) -> float:
+        """
+        Custom exit price logic, returning the new exit price.
+
+        For full documentation please go to https://www.freqtrade.io/en/latest/strategy-advanced/
+
+        When not implemented by a strategy, returns None, orderbook is used to set exit price
+
+        :param pair: Pair that's currently analyzed
+        :param trade: trade object.
+        :param current_time: datetime object, containing the current datetime
+        :param proposed_rate: Rate, calculated based on pricing settings in ask_strategy.
+        :param current_profit: Current profit (as ratio), calculated based on current_rate.
+        :param **kwargs: Ensure to keep this here so updates to this won't break your strategy.
+        :return float: New exit price value if provided
+        """
+        return proposed_rate
+
     def custom_sell(self, pair: str, trade: Trade, current_time: datetime, current_rate: float,
                     current_profit: float, **kwargs) -> Optional[Union[str, bool]]:
         """
@@ -591,7 +628,7 @@ class IStrategy(ABC, HyperStrategyMixin):
         current_rate = rate
         current_profit = trade.calc_profit_ratio(current_rate)
 
-        trade.adjust_min_max_rates(high or current_rate)
+        trade.adjust_min_max_rates(high or current_rate, low or current_rate)
 
         stoplossflag = self.stop_loss_reached(current_rate=current_rate, trade=trade,
                                               current_time=date, current_profit=current_profit,
@@ -761,7 +798,7 @@ class IStrategy(ABC, HyperStrategyMixin):
         else:
             return current_profit > roi
 
-    def ohlcvdata_to_dataframe(self, data: Dict[str, DataFrame]) -> Dict[str, DataFrame]:
+    def advise_all_indicators(self, data: Dict[str, DataFrame]) -> Dict[str, DataFrame]:
         """
         Populates indicators for given candle (OHLCV) data (for multiple pairs)
         Does not run advise_buy or advise_sell!

freqtrade/templates/base_config.json.j2

@@ -25,7 +25,7 @@
     "ask_strategy": {
         "price_side": "ask",
         "use_order_book": true,
-        "order_book_top": 1,
+        "order_book_top": 1
     },
     {{ exchange | indent(4) }},
     "pairlists": [

requirements-dev.txt

@@ -6,7 +6,7 @@
 coveralls==3.2.0
 flake8==3.9.2
 flake8-type-annotations==0.1.0
-flake8-tidy-imports==4.3.0
+flake8-tidy-imports==4.4.1
 mypy==0.910
 pytest==6.2.4
 pytest-asyncio==0.15.1
@@ -19,7 +19,7 @@ isort==5.9.3
 nbconvert==6.1.0
 
 # mypy types
-types-cachetools==0.1.9
-types-filelock==0.1.4
-types-requests==2.25.1
-types-tabulate==0.1.1
+types-cachetools==4.2.0
+types-filelock==0.1.5
+types-requests==2.25.6
+types-tabulate==0.8.2

requirements-plot.txt

@@ -1,5 +1,5 @@
 # Include all requirements to run the bot.
 -r requirements.txt
 
-plotly==5.1.0
+plotly==5.2.1
 

requirements.txt

@@ -1,11 +1,11 @@
-numpy==1.21.1
-pandas==1.3.1
+numpy==1.21.2
+pandas==1.3.2
 
-ccxt==1.54.24
+ccxt==1.55.28
 # Pin cryptography for now due to rust build errors with piwheels
 cryptography==3.4.7
 aiohttp==3.7.4.post0
-SQLAlchemy==1.4.22
+SQLAlchemy==1.4.23
 python-telegram-bot==13.7
 arrow==1.1.1
 cachetools==4.2.2
@@ -32,7 +32,7 @@ sdnotify==0.3.2
 
 # API Server
 fastapi==0.68.0
-uvicorn==0.14.0
+uvicorn==0.15.0
 pyjwt==2.1.0
 aiofiles==0.7.0
 
@@ -40,4 +40,4 @@ aiofiles==0.7.0
 colorama==0.4.4
 # Building config files interactively
 questionary==1.10.0
-prompt-toolkit==3.0.19
+prompt-toolkit==3.0.20

setup.sh

@@ -163,7 +163,7 @@ function update() {
 # Reset Develop or Stable branch
 function reset() {
     echo "----------------------------"
-    echo "Reseting branch and virtual env"
+    echo "Resetting branch and virtual env"
     echo "----------------------------"
 
     if [ "1" == $(git branch -vv |grep -cE "\* develop|\* stable") ]

tests/commands/test_commands.py

@@ -938,13 +938,19 @@ def test_start_test_pairlist(mocker, caplog, tickers, default_conf, capsys):
         pytest.fail(f'Expected well formed JSON, but failed to parse: {captured.out}')
 
 
-def test_hyperopt_list(mocker, capsys, caplog, saved_hyperopt_results,
-                       saved_hyperopt_results_legacy, tmpdir):
+def test_hyperopt_list(mocker, capsys, caplog, saved_hyperopt_results, tmpdir):
     csv_file = Path(tmpdir) / "test.csv"
-    for res in (saved_hyperopt_results, saved_hyperopt_results_legacy):
     mocker.patch(
-            'freqtrade.optimize.hyperopt_tools.HyperoptTools.load_previous_results',
-            MagicMock(return_value=res)
+        'freqtrade.optimize.hyperopt_tools.HyperoptTools._test_hyperopt_results_exist',
+        return_value=True
+        )
+
+    def fake_iterator(*args, **kwargs):
+        yield from [saved_hyperopt_results]
+
+    mocker.patch(
+        'freqtrade.optimize.hyperopt_tools.HyperoptTools._read_results',
+        side_effect=fake_iterator
     )
 
     args = [
@@ -1177,8 +1183,16 @@ def test_hyperopt_list(mocker, capsys, caplog, saved_hyperopt_results,
 
 def test_hyperopt_show(mocker, capsys, saved_hyperopt_results):
     mocker.patch(
-        'freqtrade.optimize.hyperopt_tools.HyperoptTools.load_previous_results',
-        MagicMock(return_value=saved_hyperopt_results)
+        'freqtrade.optimize.hyperopt_tools.HyperoptTools._test_hyperopt_results_exist',
+        return_value=True
+    )
+
+    def fake_iterator(*args, **kwargs):
+        yield from [saved_hyperopt_results]
+
+    mocker.patch(
+        'freqtrade.optimize.hyperopt_tools.HyperoptTools._read_results',
+        side_effect=fake_iterator
     )
     mocker.patch('freqtrade.commands.hyperopt_commands.show_backtest_result')
 

tests/conftest.py

@@ -1851,138 +1851,6 @@ def open_trade():
     )
 
 
-@pytest.fixture
-def saved_hyperopt_results_legacy():
-    return [
-        {
-            'loss': 0.4366182531161519,
-            'params_dict': {
-                'mfi-value': 15, 'fastd-value': 20, 'adx-value': 25, 'rsi-value': 28, 'mfi-enabled': False, 'fastd-enabled': True, 'adx-enabled': True, 'rsi-enabled': True, 'trigger': 'macd_cross_signal', 'sell-mfi-value': 88, 'sell-fastd-value': 97, 'sell-adx-value': 51, 'sell-rsi-value': 67, 'sell-mfi-enabled': False, 'sell-fastd-enabled': False, 'sell-adx-enabled': True, 'sell-rsi-enabled': True, 'sell-trigger': 'sell-bb_upper', 'roi_t1': 1190, 'roi_t2': 541, 'roi_t3': 408, 'roi_p1': 0.026035863879169705, 'roi_p2': 0.12508730043628782, 'roi_p3': 0.27766427921605896, 'stoploss': -0.2562930402099556},  # noqa: E501
-            'params_details': {'buy': {'mfi-value': 15, 'fastd-value': 20, 'adx-value': 25, 'rsi-value': 28, 'mfi-enabled': False, 'fastd-enabled': True, 'adx-enabled': True, 'rsi-enabled': True, 'trigger': 'macd_cross_signal'}, 'sell': {'sell-mfi-value': 88, 'sell-fastd-value': 97, 'sell-adx-value': 51, 'sell-rsi-value': 67, 'sell-mfi-enabled': False, 'sell-fastd-enabled': False, 'sell-adx-enabled': True, 'sell-rsi-enabled': True, 'sell-trigger': 'sell-bb_upper'}, 'roi': {0: 0.4287874435315165, 408: 0.15112316431545753, 949: 0.026035863879169705, 2139: 0}, 'stoploss': {'stoploss': -0.2562930402099556}},  # noqa: E501
-            'results_metrics': {'trade_count': 2, 'avg_profit': -1.254995, 'median_profit': -1.2222, 'total_profit': -0.00125625, 'profit': -2.50999, 'duration': 3930.0},  # noqa: E501
-            'results_explanation': '     2 trades. Avg profit  -1.25%. Total profit -0.00125625 BTC (  -2.51Σ%). Avg duration 3930.0 min.',  # noqa: E501
-            'total_profit': -0.00125625,
-            'current_epoch': 1,
-            'is_initial_point': True,
-            'is_best': True
-        }, {
-            'loss': 20.0,
-            'params_dict': {
-                'mfi-value': 17, 'fastd-value': 38, 'adx-value': 48, 'rsi-value': 22, 'mfi-enabled': True, 'fastd-enabled': False, 'adx-enabled': True, 'rsi-enabled': True, 'trigger': 'macd_cross_signal', 'sell-mfi-value': 96, 'sell-fastd-value': 68, 'sell-adx-value': 63, 'sell-rsi-value': 81, 'sell-mfi-enabled': False, 'sell-fastd-enabled': True, 'sell-adx-enabled': True, 'sell-rsi-enabled': True, 'sell-trigger': 'sell-sar_reversal', 'roi_t1': 334, 'roi_t2': 683, 'roi_t3': 140, 'roi_p1': 0.06403981740598495, 'roi_p2': 0.055519840060645045, 'roi_p3': 0.3253712811342459, 'stoploss': -0.338070047333259},  # noqa: E501
-            'params_details': {
-                'buy': {'mfi-value': 17, 'fastd-value': 38, 'adx-value': 48, 'rsi-value': 22, 'mfi-enabled': True, 'fastd-enabled': False, 'adx-enabled': True, 'rsi-enabled': True, 'trigger': 'macd_cross_signal'},  # noqa: E501
-                'sell': {'sell-mfi-value': 96, 'sell-fastd-value': 68, 'sell-adx-value': 63, 'sell-rsi-value': 81, 'sell-mfi-enabled': False, 'sell-fastd-enabled': True, 'sell-adx-enabled': True, 'sell-rsi-enabled': True, 'sell-trigger': 'sell-sar_reversal'},  # noqa: E501
-                'roi': {0: 0.4449309386008759, 140: 0.11955965746663, 823: 0.06403981740598495, 1157: 0},  # noqa: E501
-                'stoploss': {'stoploss': -0.338070047333259}},
-            'results_metrics': {'trade_count': 1, 'avg_profit': 0.12357, 'median_profit': -1.2222, 'total_profit': 6.185e-05, 'profit': 0.12357, 'duration': 1200.0},  # noqa: E501
-            'results_explanation': '     1 trades. Avg profit   0.12%. Total profit  0.00006185 BTC (   0.12Σ%). Avg duration 1200.0 min.',  # noqa: E501
-            'total_profit': 6.185e-05,
-            'current_epoch': 2,
-            'is_initial_point': True,
-            'is_best': False
-        }, {
-            'loss': 14.241196856510731,
-            'params_dict': {'mfi-value': 25, 'fastd-value': 16, 'adx-value': 29, 'rsi-value': 20, 'mfi-enabled': False, 'fastd-enabled': False, 'adx-enabled': False, 'rsi-enabled': False, 'trigger': 'macd_cross_signal', 'sell-mfi-value': 98, 'sell-fastd-value': 72, 'sell-adx-value': 51, 'sell-rsi-value': 82, 'sell-mfi-enabled': True, 'sell-fastd-enabled': True, 'sell-adx-enabled': True, 'sell-rsi-enabled': True, 'sell-trigger': 'sell-macd_cross_signal', 'roi_t1': 889, 'roi_t2': 533, 'roi_t3': 263, 'roi_p1': 0.04759065393663096, 'roi_p2': 0.1488819964638463, 'roi_p3': 0.4102801822104605, 'stoploss': -0.05394588767607611},  # noqa: E501
-            'params_details': {'buy': {'mfi-value': 25, 'fastd-value': 16, 'adx-value': 29, 'rsi-value': 20, 'mfi-enabled': False, 'fastd-enabled': False, 'adx-enabled': False, 'rsi-enabled': False, 'trigger': 'macd_cross_signal'}, 'sell': {'sell-mfi-value': 98, 'sell-fastd-value': 72, 'sell-adx-value': 51, 'sell-rsi-value': 82, 'sell-mfi-enabled': True, 'sell-fastd-enabled': True, 'sell-adx-enabled': True, 'sell-rsi-enabled': True, 'sell-trigger': 'sell-macd_cross_signal'}, 'roi': {0: 0.6067528326109377, 263: 0.19647265040047726, 796: 0.04759065393663096, 1685: 0}, 'stoploss': {'stoploss': -0.05394588767607611}},  # noqa: E501
-            'results_metrics': {'trade_count': 621, 'avg_profit': -0.43883302093397747, 'median_profit': -1.2222, 'total_profit': -0.13639474, 'profit': -272.515306, 'duration': 1691.207729468599},  # noqa: E501
-            'results_explanation': '   621 trades. Avg profit  -0.44%. Total profit -0.13639474 BTC (-272.52Σ%). Avg duration 1691.2 min.',  # noqa: E501
-            'total_profit': -0.13639474,
-            'current_epoch': 3,
-            'is_initial_point': True,
-            'is_best': False
-        }, {
-            'loss': 100000,
-            'params_dict': {'mfi-value': 13, 'fastd-value': 35, 'adx-value': 39, 'rsi-value': 29, 'mfi-enabled': True, 'fastd-enabled': False, 'adx-enabled': False, 'rsi-enabled': True, 'trigger': 'macd_cross_signal', 'sell-mfi-value': 87, 'sell-fastd-value': 54, 'sell-adx-value': 63, 'sell-rsi-value': 93, 'sell-mfi-enabled': False, 'sell-fastd-enabled': True, 'sell-adx-enabled': True, 'sell-rsi-enabled': True, 'sell-trigger': 'sell-bb_upper', 'roi_t1': 1402, 'roi_t2': 676, 'roi_t3': 215, 'roi_p1': 0.06264755784937427, 'roi_p2': 0.14258587851894644, 'roi_p3': 0.20671291201040828, 'stoploss': -0.11818343570194478},  # noqa: E501
-            'params_details': {'buy': {'mfi-value': 13, 'fastd-value': 35, 'adx-value': 39, 'rsi-value': 29, 'mfi-enabled': True, 'fastd-enabled': False, 'adx-enabled': False, 'rsi-enabled': True, 'trigger': 'macd_cross_signal'}, 'sell': {'sell-mfi-value': 87, 'sell-fastd-value': 54, 'sell-adx-value': 63, 'sell-rsi-value': 93, 'sell-mfi-enabled': False, 'sell-fastd-enabled': True, 'sell-adx-enabled': True, 'sell-rsi-enabled': True, 'sell-trigger': 'sell-bb_upper'}, 'roi': {0: 0.411946348378729, 215: 0.2052334363683207, 891: 0.06264755784937427, 2293: 0}, 'stoploss': {'stoploss': -0.11818343570194478}},  # noqa: E501
-            'results_metrics': {'trade_count': 0, 'avg_profit': None, 'median_profit': None, 'total_profit': 0, 'profit': 0.0, 'duration': None},  # noqa: E501
-            'results_explanation': '     0 trades. Avg profit    nan%. Total profit  0.00000000 BTC (   0.00Σ%). Avg duration   nan min.',  # noqa: E501
-            'total_profit': 0, 'current_epoch': 4, 'is_initial_point': True, 'is_best': False
-        }, {
-            'loss': 0.22195522184191518,
-            'params_dict': {'mfi-value': 17, 'fastd-value': 21, 'adx-value': 38, 'rsi-value': 33, 'mfi-enabled': True, 'fastd-enabled': False, 'adx-enabled': True, 'rsi-enabled': False, 'trigger': 'macd_cross_signal', 'sell-mfi-value': 87, 'sell-fastd-value': 82, 'sell-adx-value': 78, 'sell-rsi-value': 69, 'sell-mfi-enabled': True, 'sell-fastd-enabled': False, 'sell-adx-enabled': True, 'sell-rsi-enabled': False, 'sell-trigger': 'sell-macd_cross_signal', 'roi_t1': 1269, 'roi_t2': 601, 'roi_t3': 444, 'roi_p1': 0.07280999507931168, 'roi_p2': 0.08946698095898986, 'roi_p3': 0.1454876733325284, 'stoploss': -0.18181041180901014},   # noqa: E501
-            'params_details': {'buy': {'mfi-value': 17, 'fastd-value': 21, 'adx-value': 38, 'rsi-value': 33, 'mfi-enabled': True, 'fastd-enabled': False, 'adx-enabled': True, 'rsi-enabled': False, 'trigger': 'macd_cross_signal'}, 'sell': {'sell-mfi-value': 87, 'sell-fastd-value': 82, 'sell-adx-value': 78, 'sell-rsi-value': 69, 'sell-mfi-enabled': True, 'sell-fastd-enabled': False, 'sell-adx-enabled': True, 'sell-rsi-enabled': False, 'sell-trigger': 'sell-macd_cross_signal'}, 'roi': {0: 0.3077646493708299, 444: 0.16227697603830155, 1045: 0.07280999507931168, 2314: 0}, 'stoploss': {'stoploss': -0.18181041180901014}},  # noqa: E501
-            'results_metrics': {'trade_count': 14, 'avg_profit': -0.3539515, 'median_profit': -1.2222, 'total_profit': -0.002480140000000001, 'profit': -4.955321, 'duration': 3402.8571428571427},  # noqa: E501
-            'results_explanation': '    14 trades. Avg profit  -0.35%. Total profit -0.00248014 BTC (  -4.96Σ%). Avg duration 3402.9 min.',  # noqa: E501
-            'total_profit': -0.002480140000000001,
-            'current_epoch': 5,
-            'is_initial_point': True,
-            'is_best': True
-        }, {
-            'loss': 0.545315889154162,
-            'params_dict': {'mfi-value': 22, 'fastd-value': 43, 'adx-value': 46, 'rsi-value': 20, 'mfi-enabled': False, 'fastd-enabled': False, 'adx-enabled': True, 'rsi-enabled': True, 'trigger': 'bb_lower', 'sell-mfi-value': 87, 'sell-fastd-value': 65, 'sell-adx-value': 94, 'sell-rsi-value': 63, 'sell-mfi-enabled': False, 'sell-fastd-enabled': True, 'sell-adx-enabled': True, 'sell-rsi-enabled': True, 'sell-trigger': 'sell-macd_cross_signal', 'roi_t1': 319, 'roi_t2': 556, 'roi_t3': 216, 'roi_p1': 0.06251955472249589, 'roi_p2': 0.11659519602202795, 'roi_p3': 0.0953744132197762, 'stoploss': -0.024551752215582423},  # noqa: E501
-            'params_details': {'buy': {'mfi-value': 22, 'fastd-value': 43, 'adx-value': 46, 'rsi-value': 20, 'mfi-enabled': False, 'fastd-enabled': False, 'adx-enabled': True, 'rsi-enabled': True, 'trigger': 'bb_lower'}, 'sell': {'sell-mfi-value': 87, 'sell-fastd-value': 65, 'sell-adx-value': 94, 'sell-rsi-value': 63, 'sell-mfi-enabled': False, 'sell-fastd-enabled': True, 'sell-adx-enabled': True, 'sell-rsi-enabled': True, 'sell-trigger': 'sell-macd_cross_signal'}, 'roi': {0: 0.2744891639643, 216: 0.17911475074452382, 772: 0.06251955472249589, 1091: 0}, 'stoploss': {'stoploss': -0.024551752215582423}},  # noqa: E501
-            'results_metrics': {'trade_count': 39, 'avg_profit': -0.21400679487179478, 'median_profit': -1.2222, 'total_profit': -0.0041773, 'profit': -8.346264999999997, 'duration': 636.9230769230769},  # noqa: E501
-            'results_explanation': '    39 trades. Avg profit  -0.21%. Total profit -0.00417730 BTC (  -8.35Σ%). Avg duration 636.9 min.',  # noqa: E501
-            'total_profit': -0.0041773,
-            'current_epoch': 6,
-            'is_initial_point': True,
-            'is_best': False
-        }, {
-            'loss': 4.713497421432944,
-            'params_dict': {'mfi-value': 13, 'fastd-value': 41, 'adx-value': 21, 'rsi-value': 29, 'mfi-enabled': False, 'fastd-enabled': True, 'adx-enabled': False, 'rsi-enabled': False, 'trigger': 'bb_lower', 'sell-mfi-value': 99, 'sell-fastd-value': 60, 'sell-adx-value': 81, 'sell-rsi-value': 69, 'sell-mfi-enabled': True, 'sell-fastd-enabled': True, 'sell-adx-enabled': True, 'sell-rsi-enabled': False, 'sell-trigger': 'sell-macd_cross_signal', 'roi_t1': 771, 'roi_t2': 620, 'roi_t3': 145, 'roi_p1': 0.0586919200378493, 'roi_p2': 0.04984118697312542, 'roi_p3': 0.37521058680247044, 'stoploss': -0.14613268022709905},  # noqa: E501
-            'params_details': {
-                'buy': {'mfi-value': 13, 'fastd-value': 41, 'adx-value': 21, 'rsi-value': 29, 'mfi-enabled': False, 'fastd-enabled': True, 'adx-enabled': False, 'rsi-enabled': False, 'trigger': 'bb_lower'}, 'sell': {'sell-mfi-value': 99, 'sell-fastd-value': 60, 'sell-adx-value': 81, 'sell-rsi-value': 69, 'sell-mfi-enabled': True, 'sell-fastd-enabled': True, 'sell-adx-enabled': True, 'sell-rsi-enabled': False, 'sell-trigger': 'sell-macd_cross_signal'}, 'roi': {0: 0.4837436938134452, 145: 0.10853310701097472, 765: 0.0586919200378493, 1536: 0},  # noqa: E501
-                'stoploss': {'stoploss': -0.14613268022709905}},  # noqa: E501
-            'results_metrics': {'trade_count': 318, 'avg_profit': -0.39833954716981146, 'median_profit': -1.2222, 'total_profit': -0.06339929, 'profit': -126.67197600000004, 'duration': 3140.377358490566},  # noqa: E501
-            'results_explanation': '   318 trades. Avg profit  -0.40%. Total profit -0.06339929 BTC (-126.67Σ%). Avg duration 3140.4 min.',  # noqa: E501
-            'total_profit': -0.06339929,
-            'current_epoch': 7,
-            'is_initial_point': True,
-            'is_best': False
-        }, {
-            'loss': 20.0,  # noqa: E501
-            'params_dict': {'mfi-value': 24, 'fastd-value': 43, 'adx-value': 33, 'rsi-value': 20, 'mfi-enabled': False, 'fastd-enabled': True, 'adx-enabled': True, 'rsi-enabled': True, 'trigger': 'sar_reversal', 'sell-mfi-value': 89, 'sell-fastd-value': 74, 'sell-adx-value': 70, 'sell-rsi-value': 70, 'sell-mfi-enabled': False, 'sell-fastd-enabled': False, 'sell-adx-enabled': False, 'sell-rsi-enabled': True, 'sell-trigger': 'sell-sar_reversal', 'roi_t1': 1149, 'roi_t2': 375, 'roi_t3': 289, 'roi_p1': 0.05571820757172588, 'roi_p2': 0.0606240398618907, 'roi_p3': 0.1729012220156157, 'stoploss': -0.1588514289110401},  # noqa: E501
-            'params_details': {'buy': {'mfi-value': 24, 'fastd-value': 43, 'adx-value': 33, 'rsi-value': 20, 'mfi-enabled': False, 'fastd-enabled': True, 'adx-enabled': True, 'rsi-enabled': True, 'trigger': 'sar_reversal'}, 'sell': {'sell-mfi-value': 89, 'sell-fastd-value': 74, 'sell-adx-value': 70, 'sell-rsi-value': 70, 'sell-mfi-enabled': False, 'sell-fastd-enabled': False, 'sell-adx-enabled': False, 'sell-rsi-enabled': True, 'sell-trigger': 'sell-sar_reversal'}, 'roi': {0: 0.2892434694492323, 289: 0.11634224743361658, 664: 0.05571820757172588, 1813: 0}, 'stoploss': {'stoploss': -0.1588514289110401}},  # noqa: E501
-            'results_metrics': {'trade_count': 1, 'avg_profit': 0.0, 'median_profit': 0.0, 'total_profit': 0.0, 'profit': 0.0, 'duration': 5340.0},  # noqa: E501
-            'results_explanation': '     1 trades. Avg profit   0.00%. Total profit  0.00000000 BTC (   0.00Σ%). Avg duration 5340.0 min.',  # noqa: E501
-            'total_profit': 0.0,
-            'current_epoch': 8,
-            'is_initial_point': True,
-            'is_best': False
-        }, {
-            'loss': 2.4731817780991223,
-            'params_dict': {'mfi-value': 22, 'fastd-value': 20, 'adx-value': 29, 'rsi-value': 40, 'mfi-enabled': False, 'fastd-enabled': False, 'adx-enabled': False, 'rsi-enabled': False, 'trigger': 'sar_reversal', 'sell-mfi-value': 97, 'sell-fastd-value': 65, 'sell-adx-value': 81, 'sell-rsi-value': 64, 'sell-mfi-enabled': True, 'sell-fastd-enabled': True, 'sell-adx-enabled': True, 'sell-rsi-enabled': True, 'sell-trigger': 'sell-bb_upper', 'roi_t1': 1012, 'roi_t2': 584, 'roi_t3': 422, 'roi_p1': 0.036764323603472565, 'roi_p2': 0.10335480573205287, 'roi_p3': 0.10322347377503042, 'stoploss': -0.2780610808108503},  # noqa: E501
-            'params_details': {'buy': {'mfi-value': 22, 'fastd-value': 20, 'adx-value': 29, 'rsi-value': 40, 'mfi-enabled': False, 'fastd-enabled': False, 'adx-enabled': False, 'rsi-enabled': False, 'trigger': 'sar_reversal'}, 'sell': {'sell-mfi-value': 97, 'sell-fastd-value': 65, 'sell-adx-value': 81, 'sell-rsi-value': 64, 'sell-mfi-enabled': True, 'sell-fastd-enabled': True, 'sell-adx-enabled': True, 'sell-rsi-enabled': True, 'sell-trigger': 'sell-bb_upper'}, 'roi': {0: 0.2433426031105559, 422: 0.14011912933552545, 1006: 0.036764323603472565, 2018: 0}, 'stoploss': {'stoploss': -0.2780610808108503}},  # noqa: E501
-            'results_metrics': {'trade_count': 229, 'avg_profit': -0.38433433624454144, 'median_profit': -1.2222, 'total_profit': -0.044050070000000004, 'profit': -88.01256299999999, 'duration': 6505.676855895196},  # noqa: E501
-            'results_explanation': '   229 trades. Avg profit  -0.38%. Total profit -0.04405007 BTC ( -88.01Σ%). Avg duration 6505.7 min.',  # noqa: E501
-            'total_profit': -0.044050070000000004,  # noqa: E501
-            'current_epoch': 9,
-            'is_initial_point': True,
-            'is_best': False
-        }, {
-            'loss': -0.2604606005845212,  # noqa: E501
-            'params_dict': {'mfi-value': 23, 'fastd-value': 24, 'adx-value': 22, 'rsi-value': 24, 'mfi-enabled': False, 'fastd-enabled': False, 'adx-enabled': False, 'rsi-enabled': True, 'trigger': 'macd_cross_signal', 'sell-mfi-value': 97, 'sell-fastd-value': 70, 'sell-adx-value': 64, 'sell-rsi-value': 80, 'sell-mfi-enabled': False, 'sell-fastd-enabled': True, 'sell-adx-enabled': True, 'sell-rsi-enabled': True, 'sell-trigger': 'sell-sar_reversal', 'roi_t1': 792, 'roi_t2': 464, 'roi_t3': 215, 'roi_p1': 0.04594053535385903, 'roi_p2': 0.09623192684243963, 'roi_p3': 0.04428219070850663, 'stoploss': -0.16992287161634415},  # noqa: E501
-            'params_details': {'buy': {'mfi-value': 23, 'fastd-value': 24, 'adx-value': 22, 'rsi-value': 24, 'mfi-enabled': False, 'fastd-enabled': False, 'adx-enabled': False, 'rsi-enabled': True, 'trigger': 'macd_cross_signal'}, 'sell': {'sell-mfi-value': 97, 'sell-fastd-value': 70, 'sell-adx-value': 64, 'sell-rsi-value': 80, 'sell-mfi-enabled': False, 'sell-fastd-enabled': True, 'sell-adx-enabled': True, 'sell-rsi-enabled': True, 'sell-trigger': 'sell-sar_reversal'}, 'roi': {0: 0.18645465290480528, 215: 0.14217246219629864, 679: 0.04594053535385903, 1471: 0}, 'stoploss': {'stoploss': -0.16992287161634415}},  # noqa: E501
-            'results_metrics': {'trade_count': 4, 'avg_profit': 0.1080385, 'median_profit': -1.2222, 'total_profit': 0.00021629, 'profit': 0.432154, 'duration': 2850.0},  # noqa: E501
-            'results_explanation': '     4 trades. Avg profit   0.11%. Total profit  0.00021629 BTC (   0.43Σ%). Avg duration 2850.0 min.',  # noqa: E501
-            'total_profit': 0.00021629,
-            'current_epoch': 10,
-            'is_initial_point': True,
-            'is_best': True
-        }, {
-            'loss': 4.876465945994304,  # noqa: E501
-            'params_dict': {'mfi-value': 20, 'fastd-value': 32, 'adx-value': 49, 'rsi-value': 23, 'mfi-enabled': True, 'fastd-enabled': True, 'adx-enabled': False, 'rsi-enabled': False, 'trigger': 'bb_lower', 'sell-mfi-value': 75, 'sell-fastd-value': 56, 'sell-adx-value': 61, 'sell-rsi-value': 62, 'sell-mfi-enabled': False, 'sell-fastd-enabled': False, 'sell-adx-enabled': True, 'sell-rsi-enabled': True, 'sell-trigger': 'sell-macd_cross_signal', 'roi_t1': 579, 'roi_t2': 614, 'roi_t3': 273, 'roi_p1': 0.05307643172744114, 'roi_p2': 0.1352282078262871, 'roi_p3': 0.1913307406325751, 'stoploss': -0.25728526022513887},  # noqa: E501
-            'params_details': {'buy': {'mfi-value': 20, 'fastd-value': 32, 'adx-value': 49, 'rsi-value': 23, 'mfi-enabled': True, 'fastd-enabled': True, 'adx-enabled': False, 'rsi-enabled': False, 'trigger': 'bb_lower'}, 'sell': {'sell-mfi-value': 75, 'sell-fastd-value': 56, 'sell-adx-value': 61, 'sell-rsi-value': 62, 'sell-mfi-enabled': False, 'sell-fastd-enabled': False, 'sell-adx-enabled': True, 'sell-rsi-enabled': True, 'sell-trigger': 'sell-macd_cross_signal'}, 'roi': {0: 0.3796353801863034, 273: 0.18830463955372825, 887: 0.05307643172744114, 1466: 0}, 'stoploss': {'stoploss': -0.25728526022513887}},  # noqa: E501
-            'results_metrics': {'trade_count': 117, 'avg_profit': -1.2698609145299145, 'median_profit': -1.2222, 'total_profit': -0.07436117, 'profit': -148.573727, 'duration': 4282.5641025641025},  # noqa: E501
-            'results_explanation': '   117 trades. Avg profit  -1.27%. Total profit -0.07436117 BTC (-148.57Σ%). Avg duration 4282.6 min.',  # noqa: E501
-            'total_profit': -0.07436117,
-            'current_epoch': 11,
-            'is_initial_point': True,
-            'is_best': False
-        }, {
-            'loss': 100000,
-            'params_dict': {'mfi-value': 10, 'fastd-value': 36, 'adx-value': 31, 'rsi-value': 22, 'mfi-enabled': True, 'fastd-enabled': True, 'adx-enabled': True, 'rsi-enabled': False, 'trigger': 'sar_reversal', 'sell-mfi-value': 80, 'sell-fastd-value': 71, 'sell-adx-value': 60, 'sell-rsi-value': 85, 'sell-mfi-enabled': False, 'sell-fastd-enabled': False, 'sell-adx-enabled': True, 'sell-rsi-enabled': True, 'sell-trigger': 'sell-bb_upper', 'roi_t1': 1156, 'roi_t2': 581, 'roi_t3': 408, 'roi_p1': 0.06860454019988212, 'roi_p2': 0.12473718444931989, 'roi_p3': 0.2896360635226823, 'stoploss': -0.30889015124682806},  # noqa: E501
-            'params_details': {'buy': {'mfi-value': 10, 'fastd-value': 36, 'adx-value': 31, 'rsi-value': 22, 'mfi-enabled': True, 'fastd-enabled': True, 'adx-enabled': True, 'rsi-enabled': False, 'trigger': 'sar_reversal'}, 'sell': {'sell-mfi-value': 80, 'sell-fastd-value': 71, 'sell-adx-value': 60, 'sell-rsi-value': 85, 'sell-mfi-enabled': False, 'sell-fastd-enabled': False, 'sell-adx-enabled': True, 'sell-rsi-enabled': True, 'sell-trigger': 'sell-bb_upper'}, 'roi': {0: 0.4829777881718843, 408: 0.19334172464920202, 989: 0.06860454019988212, 2145: 0}, 'stoploss': {'stoploss': -0.30889015124682806}},  # noqa: E501
-            'results_metrics': {'trade_count': 0, 'avg_profit': None, 'median_profit': None, 'total_profit': 0, 'profit': 0.0, 'duration': None},  # noqa: E501
-            'results_explanation': '     0 trades. Avg profit    nan%. Total profit  0.00000000 BTC (   0.00Σ%). Avg duration   nan min.',  # noqa: E501
-            'total_profit': 0,
-            'current_epoch': 12,
-            'is_initial_point': True,
-            'is_best': False
-            }
-    ]
-
-
 @pytest.fixture
 def saved_hyperopt_results():
     hyperopt_res = [

tests/data/test_converter.py

@@ -119,7 +119,7 @@ def test_ohlcv_fill_up_missing_data2(caplog):
     # 3rd candle has been filled
     row = data2.loc[2, :]
     assert row['volume'] == 0
-    # close shoult match close of previous candle
+    # close should match close of previous candle
     assert row['close'] == data.loc[1, 'close']
     assert row['open'] == row['close']
     assert row['high'] == row['close']

tests/data/test_dataprovider.py

@@ -66,7 +66,7 @@ def test_historic_ohlcv_dataformat(mocker, default_conf, ohlcv_history):
     hdf5loadmock.assert_not_called()
     jsonloadmock.assert_called_once()
 
-    # Swiching to dataformat hdf5
+    # Switching to dataformat hdf5
     hdf5loadmock.reset_mock()
     jsonloadmock.reset_mock()
     default_conf["dataformat_ohlcv"] = "hdf5"

tests/data/test_history.py

@@ -133,8 +133,8 @@ def test_load_data_with_new_pair_1min(ohlcv_history_list, mocker, caplog,
     load_pair_history(datadir=tmpdir1, timeframe='1m', pair='MEME/BTC')
     assert file.is_file()
     assert log_has_re(
-        'Download history data for pair: "MEME/BTC", timeframe: 1m '
-        'and store in .*', caplog
+        r'Download history data for pair: "MEME/BTC" \(0/1\), timeframe: 1m '
+        r'and store in .*', caplog
     )
 
 
@@ -200,15 +200,15 @@ def test_load_cached_data_for_updating(mocker, testdatadir) -> None:
     assert start_ts == test_data[0][0] - 1000
 
     # timeframe starts in the center of the cached data
-    # should return the chached data w/o the last item
+    # should return the cached data w/o the last item
     timerange = TimeRange('date', None, test_data[0][0] / 1000 + 1, 0)
     data, start_ts = _load_cached_data_for_updating('UNITTEST/BTC', '1m', timerange, data_handler)
 
     assert_frame_equal(data, test_data_df.iloc[:-1])
     assert test_data[-2][0] <= start_ts < test_data[-1][0]
 
-    # timeframe starts after the chached data
-    # should return the chached data w/o the last item
+    # timeframe starts after the cached data
+    # should return the cached data w/o the last item
     timerange = TimeRange('date', None, test_data[-1][0] / 1000 + 100, 0)
     data, start_ts = _load_cached_data_for_updating('UNITTEST/BTC', '1m', timerange, data_handler)
     assert_frame_equal(data, test_data_df.iloc[:-1])
@@ -278,8 +278,10 @@ def test_download_pair_history2(mocker, default_conf, testdatadir) -> None:
         return_value=None)
     mocker.patch('freqtrade.exchange.Exchange.get_historic_ohlcv', return_value=tick)
     exchange = get_patched_exchange(mocker, default_conf)
-    _download_pair_history(testdatadir, exchange, pair="UNITTEST/BTC", timeframe='1m')
-    _download_pair_history(testdatadir, exchange, pair="UNITTEST/BTC", timeframe='3m')
+    _download_pair_history(datadir=testdatadir, exchange=exchange, pair="UNITTEST/BTC",
+                           timeframe='1m')
+    _download_pair_history(datadir=testdatadir, exchange=exchange, pair="UNITTEST/BTC",
+                           timeframe='3m')
     assert json_dump_mock.call_count == 2
 
 
@@ -381,7 +383,7 @@ def test_get_timerange(default_conf, mocker, testdatadir) -> None:
     default_conf.update({'strategy': 'DefaultStrategy'})
     strategy = StrategyResolver.load_strategy(default_conf)
 
-    data = strategy.ohlcvdata_to_dataframe(
+    data = strategy.advise_all_indicators(
         load_data(
             datadir=testdatadir,
             timeframe='1m',
@@ -399,7 +401,7 @@ def test_validate_backtest_data_warn(default_conf, mocker, caplog, testdatadir)
     default_conf.update({'strategy': 'DefaultStrategy'})
     strategy = StrategyResolver.load_strategy(default_conf)
 
-    data = strategy.ohlcvdata_to_dataframe(
+    data = strategy.advise_all_indicators(
         load_data(
             datadir=testdatadir,
             timeframe='1m',
@@ -424,7 +426,7 @@ def test_validate_backtest_data(default_conf, mocker, caplog, testdatadir) -> No
     strategy = StrategyResolver.load_strategy(default_conf)
 
     timerange = TimeRange('index', 'index', 200, 250)
-    data = strategy.ohlcvdata_to_dataframe(
+    data = strategy.advise_all_indicators(
         load_data(
             datadir=testdatadir,
             timeframe='5m',

tests/exchange/test_ccxt_compat.py

@@ -42,6 +42,11 @@ EXCHANGES = {
         'hasQuoteVolume': True,
         'timeframe': '5m',
     },
+    'gateio': {
+        'pair': 'BTC/USDT',
+        'hasQuoteVolume': True,
+        'timeframe': '5m',
+    },
 }
 
 
@@ -142,8 +147,8 @@ class TestCCXTExchange():
     def test_ccxt_get_fee(self, exchange):
         exchange, exchangename = exchange
         pair = EXCHANGES[exchangename]['pair']
-
-        assert 0 < exchange.get_fee(pair, 'limit', 'buy') < 1
-        assert 0 < exchange.get_fee(pair, 'limit', 'sell') < 1
-        assert 0 < exchange.get_fee(pair, 'market', 'buy') < 1
-        assert 0 < exchange.get_fee(pair, 'market', 'sell') < 1
+        threshold = 0.01
+        assert 0 < exchange.get_fee(pair, 'limit', 'buy') < threshold
+        assert 0 < exchange.get_fee(pair, 'limit', 'sell') < threshold
+        assert 0 < exchange.get_fee(pair, 'market', 'buy') < threshold
+        assert 0 < exchange.get_fee(pair, 'market', 'sell') < threshold

tests/exchange/test_exchange.py

@@ -984,16 +984,21 @@ def test_create_dry_run_order_limit_fill(default_conf, mocker, side, startprice,
     assert order['fee']
 
 
-@pytest.mark.parametrize("side,amount,endprice", [
-    ("buy", 1, 25.566),
-    ("buy", 100, 25.5672),  # Requires interpolation
-    ("buy", 1000, 25.575),  # More than orderbook return
-    ("sell", 1, 25.563),
-    ("sell", 100, 25.5625),  # Requires interpolation
-    ("sell", 1000, 25.5555),  # More than orderbook return
+@pytest.mark.parametrize("side,rate,amount,endprice", [
+    # spread is 25.263-25.266
+    ("buy", 25.564, 1, 25.566),
+    ("buy", 25.564, 100, 25.5672),  # Requires interpolation
+    ("buy", 25.590, 100, 25.5672),  # Price above spread ... average is lower
+    ("buy", 25.564, 1000, 25.575),  # More than orderbook return
+    ("buy", 24.000, 100000, 25.200),  # Run into max_slippage of 5%
+    ("sell", 25.564, 1, 25.563),
+    ("sell", 25.564, 100, 25.5625),  # Requires interpolation
+    ("sell", 25.510, 100, 25.5625),  # price below spread - average is higher
+    ("sell", 25.564, 1000, 25.5555),  # More than orderbook return
+    ("sell", 27, 10000, 25.65),  # max-slippage 5%
 ])
 @pytest.mark.parametrize("exchange_name", EXCHANGES)
-def test_create_dry_run_order_market_fill(default_conf, mocker, side, amount, endprice,
+def test_create_dry_run_order_market_fill(default_conf, mocker, side, rate, amount, endprice,
                                           exchange_name, order_book_l2_usd):
     default_conf['dry_run'] = True
     exchange = get_patched_exchange(mocker, default_conf, id=exchange_name)
@@ -1003,7 +1008,7 @@ def test_create_dry_run_order_market_fill(default_conf, mocker, side, amount, en
                           )
 
     order = exchange.create_dry_run_order(
-        pair='LTC/USDT', ordertype='market', side=side, amount=amount, rate=25.5)
+        pair='LTC/USDT', ordertype='market', side=side, amount=amount, rate=rate)
     assert 'id' in order
     assert f'dry_run_{side}_' in order["id"]
     assert order["side"] == side
@@ -1559,13 +1564,16 @@ def test_refresh_latest_ohlcv(mocker, default_conf, caplog) -> None:
     pairs = [('IOTA/ETH', '5m'), ('XRP/ETH', '5m')]
     # empty dicts
     assert not exchange._klines
-    exchange.refresh_latest_ohlcv(pairs, cache=False)
+    res = exchange.refresh_latest_ohlcv(pairs, cache=False)
     # No caching
     assert not exchange._klines
+
+    assert len(res) == len(pairs)
     assert exchange._api_async.fetch_ohlcv.call_count == 2
     exchange._api_async.fetch_ohlcv.reset_mock()
 
-    exchange.refresh_latest_ohlcv(pairs)
+    res = exchange.refresh_latest_ohlcv(pairs)
+    assert len(res) == len(pairs)
 
     assert log_has(f'Refreshing candle (OHLCV) data for {len(pairs)} pairs', caplog)
     assert exchange._klines
@@ -1582,12 +1590,16 @@ def test_refresh_latest_ohlcv(mocker, default_conf, caplog) -> None:
         assert exchange.klines(pair, copy=False) is exchange.klines(pair, copy=False)
 
     # test caching
-    exchange.refresh_latest_ohlcv([('IOTA/ETH', '5m'), ('XRP/ETH', '5m')])
+    res = exchange.refresh_latest_ohlcv([('IOTA/ETH', '5m'), ('XRP/ETH', '5m')])
+    assert len(res) == len(pairs)
 
     assert exchange._api_async.fetch_ohlcv.call_count == 2
     assert log_has(f"Using cached candle (OHLCV) data for pair {pairs[0][0]}, "
                    f"timeframe {pairs[0][1]} ...",
                    caplog)
+    res = exchange.refresh_latest_ohlcv([('IOTA/ETH', '5m'), ('XRP/ETH', '5m'), ('XRP/ETH', '1d')],
+                                        cache=False)
+    assert len(res) == 3
 
 
 @pytest.mark.asyncio
@@ -2177,7 +2189,7 @@ def test_get_historic_trades_notsupported(default_conf, mocker, caplog, exchange
     pair = 'ETH/BTC'
 
     with pytest.raises(OperationalException,
-                       match="This exchange does not suport downloading Trades."):
+                       match="This exchange does not support downloading Trades."):
         exchange.get_historic_trades(pair, since=trades_history[0][0],
                                      until=trades_history[-1][0])
 

tests/optimize/__init__.py

@@ -56,4 +56,6 @@ def _build_backtest_dataframe(data):
     # Ensure floats are in place
     for column in ['open', 'high', 'low', 'close', 'volume']:
         frame[column] = frame[column].astype('float64')
+    if 'buy_tag' not in columns:
+        frame['buy_tag'] = None
     return frame

tests/optimize/test_backtesting.py

@@ -1,6 +1,7 @@
 # pragma pylint: disable=missing-docstring, W0212, line-too-long, C0103, unused-argument
 
 import random
+from datetime import timedelta
 from pathlib import Path
 from unittest.mock import MagicMock, PropertyMock
 
@@ -85,7 +86,7 @@ def simple_backtest(config, contour, mocker, testdatadir) -> None:
     backtesting._set_strategy(backtesting.strategylist[0])
 
     data = load_data_test(contour, testdatadir)
-    processed = backtesting.strategy.ohlcvdata_to_dataframe(data)
+    processed = backtesting.strategy.advise_all_indicators(data)
     min_date, max_date = get_timerange(processed)
     assert isinstance(processed, dict)
     results = backtesting.backtest(
@@ -107,7 +108,7 @@ def _make_backtest_conf(mocker, datadir, conf=None, pair='UNITTEST/BTC'):
     patch_exchange(mocker)
     backtesting = Backtesting(conf)
     backtesting._set_strategy(backtesting.strategylist[0])
-    processed = backtesting.strategy.ohlcvdata_to_dataframe(data)
+    processed = backtesting.strategy.advise_all_indicators(data)
     min_date, max_date = get_timerange(processed)
     return {
         'processed': processed,
@@ -289,7 +290,7 @@ def test_backtesting_init(mocker, default_conf, order_types) -> None:
     backtesting._set_strategy(backtesting.strategylist[0])
     assert backtesting.config == default_conf
     assert backtesting.timeframe == '5m'
-    assert callable(backtesting.strategy.ohlcvdata_to_dataframe)
+    assert callable(backtesting.strategy.advise_all_indicators)
     assert callable(backtesting.strategy.advise_buy)
     assert callable(backtesting.strategy.advise_sell)
     assert isinstance(backtesting.strategy.dp, DataProvider)
@@ -335,14 +336,14 @@ def test_data_to_dataframe_bt(default_conf, mocker, testdatadir) -> None:
                              fill_up_missing=True)
     backtesting = Backtesting(default_conf)
     backtesting._set_strategy(backtesting.strategylist[0])
-    processed = backtesting.strategy.ohlcvdata_to_dataframe(data)
+    processed = backtesting.strategy.advise_all_indicators(data)
     assert len(processed['UNITTEST/BTC']) == 102
 
     # Load strategy to compare the result between Backtesting function and strategy are the same
     default_conf.update({'strategy': 'DefaultStrategy'})
     strategy = StrategyResolver.load_strategy(default_conf)
 
-    processed2 = strategy.ohlcvdata_to_dataframe(data)
+    processed2 = strategy.advise_all_indicators(data)
     assert processed['UNITTEST/BTC'].equals(processed2['UNITTEST/BTC'])
 
 
@@ -535,6 +536,8 @@ def test_backtest__enter_trade(default_conf, fee, mocker) -> None:
     trade = backtesting._enter_trade(pair, row=row)
     assert trade is None
 
+    backtesting.cleanup()
+
 
 def test_backtest_one(default_conf, fee, mocker, testdatadir) -> None:
     default_conf['use_sell_signal'] = False
@@ -547,7 +550,7 @@ def test_backtest_one(default_conf, fee, mocker, testdatadir) -> None:
     timerange = TimeRange('date', None, 1517227800, 0)
     data = history.load_data(datadir=testdatadir, timeframe='5m', pairs=['UNITTEST/BTC'],
                              timerange=timerange)
-    processed = backtesting.strategy.ohlcvdata_to_dataframe(data)
+    processed = backtesting.strategy.advise_all_indicators(data)
     min_date, max_date = get_timerange(processed)
     result = backtesting.backtest(
         processed=processed,
@@ -581,7 +584,7 @@ def test_backtest_one(default_conf, fee, mocker, testdatadir) -> None:
          'initial_stop_loss_ratio': [-0.1, -0.1],
          'stop_loss_abs': [0.0940005, 0.09272236],
          'stop_loss_ratio': [-0.1, -0.1],
-         'min_rate': [0.1038, 0.10302485],
+         'min_rate': [0.10370188, 0.10300000000000001],
          'max_rate': [0.10501, 0.1038888],
          'is_open': [False, False],
          'buy_tag': [None, None],
@@ -612,7 +615,7 @@ def test_backtest_1min_timeframe(default_conf, fee, mocker, testdatadir) -> None
     timerange = TimeRange.parse_timerange('1510688220-1510700340')
     data = history.load_data(datadir=testdatadir, timeframe='1m', pairs=['UNITTEST/BTC'],
                              timerange=timerange)
-    processed = backtesting.strategy.ohlcvdata_to_dataframe(data)
+    processed = backtesting.strategy.advise_all_indicators(data)
     min_date, max_date = get_timerange(processed)
     results = backtesting.backtest(
         processed=processed,
@@ -631,7 +634,7 @@ def test_processed(default_conf, mocker, testdatadir) -> None:
     backtesting._set_strategy(backtesting.strategylist[0])
 
     dict_of_tickerrows = load_data_test('raise', testdatadir)
-    dataframes = backtesting.strategy.ohlcvdata_to_dataframe(dict_of_tickerrows)
+    dataframes = backtesting.strategy.advise_all_indicators(dict_of_tickerrows)
     dataframe = dataframes['UNITTEST/BTC']
     cols = dataframe.columns
     # assert the dataframe got some of the indicator columns
@@ -739,8 +742,13 @@ def test_backtest_alternate_buy_sell(default_conf, fee, mocker, testdatadir):
     # 100 buys signals
     results = result['results']
     assert len(results) == 100
-    # Cached data should be 200 (no change since required_startup is 0)
-    assert len(backtesting.dataprovider.get_analyzed_dataframe('UNITTEST/BTC', '1m')[0]) == 200
+    # Cached data should be 200
+    analyzed_df = backtesting.dataprovider.get_analyzed_dataframe('UNITTEST/BTC', '1m')[0]
+    assert len(analyzed_df) == 200
+    # Expect last candle to be 1 below end date (as the last candle is assumed as "incomplete"
+    # during backtesting)
+    expected_last_candle_date = backtest_conf['end_date'] - timedelta(minutes=1)
+    assert analyzed_df.iloc[-1]['date'].to_pydatetime() == expected_last_candle_date
 
     # One trade was force-closed at the end
     assert len(results.loc[results['is_open']]) == 0
@@ -772,6 +780,7 @@ def test_backtest_multi_pair(default_conf, fee, mocker, tres, pair, testdatadir)
     data = trim_dictlist(data, -500)
 
     # Remove data for one pair from the beginning of the data
+    if tres > 0:
         data[pair] = data[pair][tres:].reset_index()
     default_conf['timeframe'] = '5m'
 
@@ -780,7 +789,7 @@ def test_backtest_multi_pair(default_conf, fee, mocker, tres, pair, testdatadir)
     backtesting.strategy.advise_buy = _trend_alternate_hold  # Override
     backtesting.strategy.advise_sell = _trend_alternate_hold  # Override
 
-    processed = backtesting.strategy.ohlcvdata_to_dataframe(data)
+    processed = backtesting.strategy.advise_all_indicators(data)
     min_date, max_date = get_timerange(processed)
     backtest_conf = {
         'processed': processed,
@@ -798,8 +807,11 @@ def test_backtest_multi_pair(default_conf, fee, mocker, tres, pair, testdatadir)
     assert len(evaluate_result_multi(results['results'], '5m', 3)) == 0
 
     # Cached data correctly removed amounts
-    removed_candles = len(data[pair]) - 1 - backtesting.strategy.startup_candle_count
+    offset = 1 if tres == 0 else 0
+    removed_candles = len(data[pair]) - offset - backtesting.strategy.startup_candle_count
     assert len(backtesting.dataprovider.get_analyzed_dataframe(pair, '5m')[0]) == removed_candles
+    assert len(backtesting.dataprovider.get_analyzed_dataframe(
+        'NXT/BTC', '5m')[0]) == len(data['NXT/BTC']) - 1 - backtesting.strategy.startup_candle_count
 
     backtest_conf = {
         'processed': processed,

tests/optimize/test_hyperopt.py

@@ -354,7 +354,7 @@ def test_start_calls_optimizer(mocker, hyperopt_conf, capsys) -> None:
     del hyperopt_conf['timeframe']
 
     hyperopt = Hyperopt(hyperopt_conf)
-    hyperopt.backtesting.strategy.ohlcvdata_to_dataframe = MagicMock()
+    hyperopt.backtesting.strategy.advise_all_indicators = MagicMock()
     hyperopt.custom_hyperopt.generate_roi_table = MagicMock(return_value={})
 
     hyperopt.start()
@@ -429,7 +429,7 @@ def test_hyperopt_format_results(hyperopt):
 
 def test_populate_indicators(hyperopt, testdatadir) -> None:
     data = load_data(testdatadir, '1m', ['UNITTEST/BTC'], fill_up_missing=True)
-    dataframes = hyperopt.backtesting.strategy.ohlcvdata_to_dataframe(data)
+    dataframes = hyperopt.backtesting.strategy.advise_all_indicators(data)
     dataframe = hyperopt.custom_hyperopt.populate_indicators(dataframes['UNITTEST/BTC'],
                                                              {'pair': 'UNITTEST/BTC'})
 
@@ -441,7 +441,7 @@ def test_populate_indicators(hyperopt, testdatadir) -> None:
 
 def test_buy_strategy_generator(hyperopt, testdatadir) -> None:
     data = load_data(testdatadir, '1m', ['UNITTEST/BTC'], fill_up_missing=True)
-    dataframes = hyperopt.backtesting.strategy.ohlcvdata_to_dataframe(data)
+    dataframes = hyperopt.backtesting.strategy.advise_all_indicators(data)
     dataframe = hyperopt.custom_hyperopt.populate_indicators(dataframes['UNITTEST/BTC'],
                                                              {'pair': 'UNITTEST/BTC'})
 
@@ -470,7 +470,7 @@ def test_buy_strategy_generator(hyperopt, testdatadir) -> None:
 
 def test_sell_strategy_generator(hyperopt, testdatadir) -> None:
     data = load_data(testdatadir, '1m', ['UNITTEST/BTC'], fill_up_missing=True)
-    dataframes = hyperopt.backtesting.strategy.ohlcvdata_to_dataframe(data)
+    dataframes = hyperopt.backtesting.strategy.advise_all_indicators(data)
     dataframe = hyperopt.custom_hyperopt.populate_indicators(dataframes['UNITTEST/BTC'],
                                                              {'pair': 'UNITTEST/BTC'})
 
@@ -671,7 +671,7 @@ def test_print_json_spaces_all(mocker, hyperopt_conf, capsys) -> None:
                           })
 
     hyperopt = Hyperopt(hyperopt_conf)
-    hyperopt.backtesting.strategy.ohlcvdata_to_dataframe = MagicMock()
+    hyperopt.backtesting.strategy.advise_all_indicators = MagicMock()
     hyperopt.custom_hyperopt.generate_roi_table = MagicMock(return_value={})
 
     hyperopt.start()
@@ -724,7 +724,7 @@ def test_print_json_spaces_default(mocker, hyperopt_conf, capsys) -> None:
     hyperopt_conf.update({'print_json': True})
 
     hyperopt = Hyperopt(hyperopt_conf)
-    hyperopt.backtesting.strategy.ohlcvdata_to_dataframe = MagicMock()
+    hyperopt.backtesting.strategy.advise_all_indicators = MagicMock()
     hyperopt.custom_hyperopt.generate_roi_table = MagicMock(return_value={})
 
     hyperopt.start()
@@ -772,7 +772,7 @@ def test_print_json_spaces_roi_stoploss(mocker, hyperopt_conf, capsys) -> None:
                           })
 
     hyperopt = Hyperopt(hyperopt_conf)
-    hyperopt.backtesting.strategy.ohlcvdata_to_dataframe = MagicMock()
+    hyperopt.backtesting.strategy.advise_all_indicators = MagicMock()
     hyperopt.custom_hyperopt.generate_roi_table = MagicMock(return_value={})
 
     hyperopt.start()
@@ -816,7 +816,7 @@ def test_simplified_interface_roi_stoploss(mocker, hyperopt_conf, capsys) -> Non
     hyperopt_conf.update({'spaces': 'roi stoploss'})
 
     hyperopt = Hyperopt(hyperopt_conf)
-    hyperopt.backtesting.strategy.ohlcvdata_to_dataframe = MagicMock()
+    hyperopt.backtesting.strategy.advise_all_indicators = MagicMock()
     hyperopt.custom_hyperopt.generate_roi_table = MagicMock(return_value={})
 
     del hyperopt.custom_hyperopt.__class__.buy_strategy_generator
@@ -855,7 +855,7 @@ def test_simplified_interface_all_failed(mocker, hyperopt_conf) -> None:
     hyperopt_conf.update({'spaces': 'all', })
 
     hyperopt = Hyperopt(hyperopt_conf)
-    hyperopt.backtesting.strategy.ohlcvdata_to_dataframe = MagicMock()
+    hyperopt.backtesting.strategy.advise_all_indicators = MagicMock()
     hyperopt.custom_hyperopt.generate_roi_table = MagicMock(return_value={})
 
     del hyperopt.custom_hyperopt.__class__.buy_strategy_generator
@@ -897,7 +897,7 @@ def test_simplified_interface_buy(mocker, hyperopt_conf, capsys) -> None:
     hyperopt_conf.update({'spaces': 'buy'})
 
     hyperopt = Hyperopt(hyperopt_conf)
-    hyperopt.backtesting.strategy.ohlcvdata_to_dataframe = MagicMock()
+    hyperopt.backtesting.strategy.advise_all_indicators = MagicMock()
     hyperopt.custom_hyperopt.generate_roi_table = MagicMock(return_value={})
 
     # TODO: sell_strategy_generator() is actually not called because
@@ -951,7 +951,7 @@ def test_simplified_interface_sell(mocker, hyperopt_conf, capsys) -> None:
     hyperopt_conf.update({'spaces': 'sell', })
 
     hyperopt = Hyperopt(hyperopt_conf)
-    hyperopt.backtesting.strategy.ohlcvdata_to_dataframe = MagicMock()
+    hyperopt.backtesting.strategy.advise_all_indicators = MagicMock()
     hyperopt.custom_hyperopt.generate_roi_table = MagicMock(return_value={})
 
     # TODO: buy_strategy_generator() is actually not called because
@@ -996,7 +996,7 @@ def test_simplified_interface_failed(mocker, hyperopt_conf, method, space) -> No
     hyperopt_conf.update({'spaces': space})
 
     hyperopt = Hyperopt(hyperopt_conf)
-    hyperopt.backtesting.strategy.ohlcvdata_to_dataframe = MagicMock()
+    hyperopt.backtesting.strategy.advise_all_indicators = MagicMock()
     hyperopt.custom_hyperopt.generate_roi_table = MagicMock(return_value={})
 
     delattr(hyperopt.custom_hyperopt.__class__, method)

tests/optimize/test_hyperopt_tools.py

@@ -10,7 +10,7 @@ import rapidjson
 from freqtrade.constants import FTHYPT_FILEVERSION
 from freqtrade.exceptions import OperationalException
 from freqtrade.optimize.hyperopt_tools import HyperoptTools, hyperopt_serializer
-from tests.conftest import log_has, log_has_re
+from tests.conftest import log_has
 
 
 # Functions for recurrent object patching
@@ -20,9 +20,14 @@ def create_results() -> List[Dict]:
 
 
 def test_save_results_saves_epochs(hyperopt, tmpdir, caplog) -> None:
+
+    hyperopt.results_file = Path(tmpdir / 'ut_results.fthypt')
+
+    hyperopt_epochs = HyperoptTools.load_filtered_results(hyperopt.results_file, {})
+    assert hyperopt_epochs == ([], 0)
+
     # Test writing to temp dir and reading again
     epochs = create_results()
-    hyperopt.results_file = Path(tmpdir / 'ut_results.fthypt')
 
     caplog.set_level(logging.DEBUG)
 
@@ -33,36 +38,28 @@ def test_save_results_saves_epochs(hyperopt, tmpdir, caplog) -> None:
     hyperopt._save_result(epochs[0])
     assert log_has(f"2 epochs saved to '{hyperopt.results_file}'.", caplog)
 
-    hyperopt_epochs = HyperoptTools.load_previous_results(hyperopt.results_file)
+    hyperopt_epochs = HyperoptTools.load_filtered_results(hyperopt.results_file, {})
     assert len(hyperopt_epochs) == 2
+    assert hyperopt_epochs[1] == 2
+    assert len(hyperopt_epochs[0]) == 2
 
-
-def test_load_previous_results(testdatadir, caplog) -> None:
-
-    results_file = testdatadir / 'hyperopt_results_SampleStrategy.pickle'
-
-    hyperopt_epochs = HyperoptTools.load_previous_results(results_file)
-
-    assert len(hyperopt_epochs) == 5
-    assert log_has_re(r"Reading pickled epochs from .*", caplog)
-
-    caplog.clear()
-
-    # Modern version
-    results_file = testdatadir / 'strategy_SampleStrategy.fthypt'
-
-    hyperopt_epochs = HyperoptTools.load_previous_results(results_file)
-
-    assert len(hyperopt_epochs) == 5
-    assert log_has_re(r"Reading epochs from .*", caplog)
+    result_gen = HyperoptTools._read_results(hyperopt.results_file, 1)
+    epoch = next(result_gen)
+    assert len(epoch) == 1
+    assert epoch[0] == epochs[0]
+    epoch = next(result_gen)
+    assert len(epoch) == 1
+    epoch = next(result_gen)
+    assert len(epoch) == 0
+    with pytest.raises(StopIteration):
+        next(result_gen)
 
 
 def test_load_previous_results2(mocker, testdatadir, caplog) -> None:
-    mocker.patch('freqtrade.optimize.hyperopt_tools.HyperoptTools._read_results_pickle',
-                 return_value=[{'asdf': '222'}])
     results_file = testdatadir / 'hyperopt_results_SampleStrategy.pickle'
-    with pytest.raises(OperationalException, match=r"The file .* incompatible.*"):
-        HyperoptTools.load_previous_results(results_file)
+    with pytest.raises(OperationalException,
+                       match=r"Legacy hyperopt results are no longer supported.*"):
+        HyperoptTools.load_filtered_results(results_file, {})
 
 
 @pytest.mark.parametrize("spaces, expected_results", [

tests/rpc/test_fiat_convert.py

@@ -22,7 +22,7 @@ def test_fiat_convert_is_supported(mocker):
 def test_fiat_convert_find_price(mocker):
     fiat_convert = CryptoToFiatConverter()
 
-    fiat_convert._cryptomap = {}
+    fiat_convert._coinlistings = {}
     fiat_convert._backoff = 0
     mocker.patch('freqtrade.rpc.fiat_convert.CryptoToFiatConverter._load_cryptomap',
                  return_value=None)
@@ -44,7 +44,7 @@ def test_fiat_convert_find_price(mocker):
 
 
 def test_fiat_convert_unsupported_crypto(mocker, caplog):
-    mocker.patch('freqtrade.rpc.fiat_convert.CryptoToFiatConverter._cryptomap', return_value=[])
+    mocker.patch('freqtrade.rpc.fiat_convert.CryptoToFiatConverter._coinlistings', return_value=[])
     fiat_convert = CryptoToFiatConverter()
     assert fiat_convert._find_price(crypto_symbol='CRYPTO_123', fiat_symbol='EUR') == 0.0
     assert log_has('unsupported crypto-symbol CRYPTO_123 - returning 0.0', caplog)
@@ -88,9 +88,9 @@ def test_fiat_convert_two_FIAT(mocker):
 def test_loadcryptomap(mocker):
 
     fiat_convert = CryptoToFiatConverter()
-    assert len(fiat_convert._cryptomap) == 2
+    assert len(fiat_convert._coinlistings) == 2
 
-    assert fiat_convert._cryptomap["btc"] == "bitcoin"
+    assert fiat_convert._get_gekko_id("btc") == "bitcoin"
 
 
 def test_fiat_init_network_exception(mocker):
@@ -102,11 +102,10 @@ def test_fiat_init_network_exception(mocker):
     )
     # with pytest.raises(RequestEsxception):
     fiat_convert = CryptoToFiatConverter()
-    fiat_convert._cryptomap = {}
+    fiat_convert._coinlistings = {}
     fiat_convert._load_cryptomap()
 
-    length_cryptomap = len(fiat_convert._cryptomap)
-    assert length_cryptomap == 0
+    assert len(fiat_convert._coinlistings) == 0
 
 
 def test_fiat_convert_without_network(mocker):
@@ -132,11 +131,10 @@ def test_fiat_too_many_requests_response(mocker, caplog):
     )
     # with pytest.raises(RequestEsxception):
     fiat_convert = CryptoToFiatConverter()
-    fiat_convert._cryptomap = {}
+    fiat_convert._coinlistings = {}
     fiat_convert._load_cryptomap()
 
-    length_cryptomap = len(fiat_convert._cryptomap)
-    assert length_cryptomap == 0
+    assert len(fiat_convert._coinlistings) == 0
     assert fiat_convert._backoff > datetime.datetime.now().timestamp()
     assert log_has(
         'Too many requests for Coingecko API, backing off and trying again later.',
@@ -144,20 +142,33 @@ def test_fiat_too_many_requests_response(mocker, caplog):
     )
 
 
+def test_fiat_multiple_coins(mocker, caplog):
+    fiat_convert = CryptoToFiatConverter()
+    fiat_convert._coinlistings = [
+        {'id': 'helium', 'symbol': 'hnt', 'name': 'Helium'},
+        {'id': 'hymnode', 'symbol': 'hnt', 'name': 'Hymnode'},
+        {'id': 'bitcoin', 'symbol': 'btc', 'name': 'Bitcoin'},
+    ]
+
+    assert fiat_convert._get_gekko_id('btc') == 'bitcoin'
+    assert fiat_convert._get_gekko_id('hnt') is None
+
+    assert log_has('Found multiple mappings in goingekko for hnt.', caplog)
+
+
 def test_fiat_invalid_response(mocker, caplog):
     # Because CryptoToFiatConverter is a Singleton we reset the listings
-    listmock = MagicMock(return_value="{'novalidjson':DEADBEEFf}")
+    listmock = MagicMock(return_value=None)
     mocker.patch.multiple(
         'freqtrade.rpc.fiat_convert.CoinGeckoAPI',
         get_coins_list=listmock,
     )
     # with pytest.raises(RequestEsxception):
     fiat_convert = CryptoToFiatConverter()
-    fiat_convert._cryptomap = {}
+    fiat_convert._coinlistings = []
     fiat_convert._load_cryptomap()
 
-    length_cryptomap = len(fiat_convert._cryptomap)
-    assert length_cryptomap == 0
+    assert len(fiat_convert._coinlistings) == 0
     assert log_has_re('Could not load FIAT Cryptocurrency map for the following problem: .*',
                       caplog)
 

tests/rpc/test_rpc_apiserver.py

@@ -109,6 +109,11 @@ def test_api_ui_fallback(botclient):
     rc = client_get(client, "/something")
     assert rc.status_code == 200
 
+    # Test directory traversal
+    rc = client_get(client, '%2F%2F%2Fetc/passwd')
+    assert rc.status_code == 200
+    assert '`freqtrade install-ui`' in rc.text
+
 
 def test_api_ui_version(botclient, mocker):
     ftbot, client = botclient

tests/strategy/test_interface.py

@@ -232,25 +232,25 @@ def test_assert_df(ohlcv_history, caplog):
     _STRATEGY.disable_dataframe_checks = False
 
 
-def test_ohlcvdata_to_dataframe(default_conf, testdatadir) -> None:
+def test_advise_all_indicators(default_conf, testdatadir) -> None:
     default_conf.update({'strategy': 'DefaultStrategy'})
     strategy = StrategyResolver.load_strategy(default_conf)
 
     timerange = TimeRange.parse_timerange('1510694220-1510700340')
     data = load_data(testdatadir, '1m', ['UNITTEST/BTC'], timerange=timerange,
                      fill_up_missing=True)
-    processed = strategy.ohlcvdata_to_dataframe(data)
+    processed = strategy.advise_all_indicators(data)
     assert len(processed['UNITTEST/BTC']) == 102  # partial candle was removed
 
 
-def test_ohlcvdata_to_dataframe_copy(mocker, default_conf, testdatadir) -> None:
+def test_advise_all_indicators_copy(mocker, default_conf, testdatadir) -> None:
     default_conf.update({'strategy': 'DefaultStrategy'})
     strategy = StrategyResolver.load_strategy(default_conf)
     aimock = mocker.patch('freqtrade.strategy.interface.IStrategy.advise_indicators')
     timerange = TimeRange.parse_timerange('1510694220-1510700340')
     data = load_data(testdatadir, '1m', ['UNITTEST/BTC'], timerange=timerange,
                      fill_up_missing=True)
-    strategy.ohlcvdata_to_dataframe(data)
+    strategy.advise_all_indicators(data)
     assert aimock.call_count == 1
     # Ensure that a copy of the dataframe is passed to advice_indicators
     assert aimock.call_args_list[0][0][0] is not data
@@ -402,7 +402,7 @@ def test_stop_loss_reached(default_conf, fee, profit, adjusted, expected, traili
         exchange='binance',
         open_rate=1,
     )
-    trade.adjust_min_max_rates(trade.open_rate)
+    trade.adjust_min_max_rates(trade.open_rate, trade.open_rate)
     strategy.trailing_stop = trailing
     strategy.trailing_stop_positive = -0.05
     strategy.use_custom_stoploss = custom
@@ -556,6 +556,7 @@ def test__analyze_ticker_internal_skip_analyze(ohlcv_history, mocker, caplog) ->
 def test_is_pair_locked(default_conf):
     default_conf.update({'strategy': 'DefaultStrategy'})
     PairLocks.timeframe = default_conf['timeframe']
+    PairLocks.use_db = True
     strategy = StrategyResolver.load_strategy(default_conf)
     # No lock should be present
     assert len(PairLocks.get_pair_locks(None)) == 0
@@ -633,7 +634,7 @@ def test_strategy_safe_wrapper_error(caplog, error):
     assert ret
 
     caplog.clear()
-    # Test supressing error
+    # Test suppressing error
     ret = strategy_safe_wrapper(failing_method, message='DeadBeef', supress_error=True)()
     assert log_has_re(r'DeadBeef.*', caplog)
 

tests/test_freqtradebot.py

@@ -904,6 +904,40 @@ def test_execute_buy(mocker, default_conf, fee, limit_buy_order, limit_buy_order
     with pytest.raises(PricingError, match="Could not determine buy price."):
         freqtrade.execute_buy(pair, stake_amount)
 
+    # In case of custom entry price
+    mocker.patch('freqtrade.exchange.Exchange.get_rate', return_value=0.50)
+    limit_buy_order['status'] = 'open'
+    limit_buy_order['id'] = '5566'
+    freqtrade.strategy.custom_entry_price = lambda **kwargs: 0.508
+    assert freqtrade.execute_buy(pair, stake_amount)
+    trade = Trade.query.all()[6]
+    assert trade
+    assert trade.open_rate_requested == 0.508
+
+    # In case of custom entry price set to None
+    limit_buy_order['status'] = 'open'
+    limit_buy_order['id'] = '5567'
+    freqtrade.strategy.custom_entry_price = lambda **kwargs: None
+
+    mocker.patch.multiple(
+        'freqtrade.exchange.Exchange',
+        get_rate=MagicMock(return_value=10),
+    )
+
+    assert freqtrade.execute_buy(pair, stake_amount)
+    trade = Trade.query.all()[7]
+    assert trade
+    assert trade.open_rate_requested == 10
+
+    # In case of custom entry price not float type
+    limit_buy_order['status'] = 'open'
+    limit_buy_order['id'] = '5568'
+    freqtrade.strategy.custom_entry_price = lambda **kwargs: "string price"
+    assert freqtrade.execute_buy(pair, stake_amount)
+    trade = Trade.query.all()[8]
+    assert trade
+    assert trade.open_rate_requested == 10
+
 
 def test_execute_buy_confirm_error(mocker, default_conf, fee, limit_buy_order) -> None:
     freqtrade = get_patched_freqtradebot(mocker, default_conf)
@@ -2716,6 +2750,70 @@ def test_execute_sell_down(default_conf, ticker, fee, ticker_sell_down, mocker)
     } == last_msg
 
 
+def test_execute_sell_custom_exit_price(default_conf, ticker, fee, ticker_sell_up, mocker) -> None:
+    rpc_mock = patch_RPCManager(mocker)
+    patch_exchange(mocker)
+    mocker.patch.multiple(
+        'freqtrade.exchange.Exchange',
+        fetch_ticker=ticker,
+        get_fee=fee,
+        _is_dry_limit_order_filled=MagicMock(return_value=False),
+    )
+    patch_whitelist(mocker, default_conf)
+    freqtrade = FreqtradeBot(default_conf)
+    patch_get_signal(freqtrade)
+    freqtrade.strategy.confirm_trade_exit = MagicMock(return_value=False)
+
+    # Create some test data
+    freqtrade.enter_positions()
+    rpc_mock.reset_mock()
+
+    trade = Trade.query.first()
+    assert trade
+    assert freqtrade.strategy.confirm_trade_exit.call_count == 0
+
+    # Increase the price and sell it
+    mocker.patch.multiple(
+        'freqtrade.exchange.Exchange',
+        fetch_ticker=ticker_sell_up
+    )
+
+    freqtrade.strategy.confirm_trade_exit = MagicMock(return_value=True)
+
+    # Set a custom exit price
+    freqtrade.strategy.custom_exit_price = lambda **kwargs: 1.170e-05
+
+    freqtrade.execute_sell(trade=trade, limit=ticker_sell_up()['bid'],
+                           sell_reason=SellCheckTuple(sell_type=SellType.SELL_SIGNAL))
+
+    # Sell price must be different to default bid price
+
+    assert freqtrade.strategy.confirm_trade_exit.call_count == 1
+
+    assert rpc_mock.call_count == 1
+    last_msg = rpc_mock.call_args_list[-1][0][0]
+    assert {
+        'trade_id': 1,
+        'type': RPCMessageType.SELL,
+        'exchange': 'Binance',
+        'pair': 'ETH/BTC',
+        'gain': 'profit',
+        'limit': 1.170e-05,
+        'amount': 91.07468123,
+        'order_type': 'limit',
+        'open_rate': 1.098e-05,
+        'current_rate': 1.173e-05,
+        'profit_amount': 6.041e-05,
+        'profit_ratio': 0.06025919,
+        'stake_currency': 'BTC',
+        'fiat_currency': 'USD',
+        'sell_reason': SellType.SELL_SIGNAL.value,
+        'open_date': ANY,
+        'close_date': ANY,
+        'close_rate': ANY,
+    } == last_msg
+
+
 def test_execute_sell_down_stoploss_on_exchange_dry_run(default_conf, ticker, fee,
                                                         ticker_sell_down, mocker) -> None:
     rpc_mock = patch_RPCManager(mocker)
@@ -4503,3 +4601,43 @@ def test_refind_lost_order(mocker, default_conf, fee, caplog):
 
     freqtrade.refind_lost_order(trades[4])
     assert log_has(f"Error updating {order['id']}.", caplog)
+
+
+def test_get_valid_price(mocker, default_conf) -> None:
+    patch_RPCManager(mocker)
+    patch_exchange(mocker)
+    freqtrade = FreqtradeBot(default_conf)
+    freqtrade.config['custom_price_max_distance_ratio'] = 0.02
+
+    custom_price_string = "10"
+    custom_price_badstring = "10abc"
+    custom_price_float = 10.0
+    custom_price_int = 10
+
+    custom_price_over_max_alwd = 11.0
+    custom_price_under_min_alwd = 9.0
+    proposed_price = 10.1
+
+    valid_price_from_string = freqtrade.get_valid_price(custom_price_string, proposed_price)
+    valid_price_from_badstring = freqtrade.get_valid_price(custom_price_badstring, proposed_price)
+    valid_price_from_int = freqtrade.get_valid_price(custom_price_int, proposed_price)
+    valid_price_from_float = freqtrade.get_valid_price(custom_price_float, proposed_price)
+
+    valid_price_at_max_alwd = freqtrade.get_valid_price(custom_price_over_max_alwd, proposed_price)
+    valid_price_at_min_alwd = freqtrade.get_valid_price(custom_price_under_min_alwd, proposed_price)
+
+    assert isinstance(valid_price_from_string, float)
+    assert isinstance(valid_price_from_badstring, float)
+    assert isinstance(valid_price_from_int, float)
+    assert isinstance(valid_price_from_float, float)
+
+    assert valid_price_from_string == custom_price_float
+    assert valid_price_from_badstring == proposed_price
+    assert valid_price_from_int == custom_price_int
+    assert valid_price_from_float == custom_price_float
+
+    assert valid_price_at_max_alwd < custom_price_over_max_alwd
+    assert valid_price_at_max_alwd > proposed_price
+
+    assert valid_price_at_min_alwd > custom_price_under_min_alwd
+    assert valid_price_at_min_alwd < proposed_price

tests/test_persistence.py

@@ -1587,25 +1587,30 @@ def test_adjust_min_max_rates(fee):
         open_rate=1,
     )
 
-    trade.adjust_min_max_rates(trade.open_rate)
+    trade.adjust_min_max_rates(trade.open_rate, trade.open_rate)
     assert trade.max_rate == 1
     assert trade.min_rate == 1
 
     # check min adjusted, max remained
-    trade.adjust_min_max_rates(0.96)
+    trade.adjust_min_max_rates(0.96, 0.96)
     assert trade.max_rate == 1
     assert trade.min_rate == 0.96
 
     # check max adjusted, min remains
-    trade.adjust_min_max_rates(1.05)
+    trade.adjust_min_max_rates(1.05, 1.05)
     assert trade.max_rate == 1.05
     assert trade.min_rate == 0.96
 
     # current rate "in the middle" - no adjustment
-    trade.adjust_min_max_rates(1.03)
+    trade.adjust_min_max_rates(1.03, 1.03)
     assert trade.max_rate == 1.05
     assert trade.min_rate == 0.96
 
+    # current rate "in the middle" - no adjustment
+    trade.adjust_min_max_rates(1.10, 0.91)
+    assert trade.max_rate == 1.10
+    assert trade.min_rate == 0.91
+
 
 @pytest.mark.usefixtures("init_persistence")
 @pytest.mark.parametrize('use_db', [True, False])
@@ -2099,6 +2104,11 @@ def test_update_order_from_ccxt(caplog):
     assert o.ft_is_open
     assert o.order_filled_date is None
 
+    # Order is unfilled, "filled" not set
+    # https://github.com/freqtrade/freqtrade/issues/5404
+    ccxt_order.update({'filled': None, 'remaining': 20.0, 'status': 'canceled'})
+    o.update_from_ccxt_object(ccxt_order)
+
     # Order has been closed
     ccxt_order.update({'filled': 20.0, 'remaining': 0.0, 'status': 'closed'})
     o.update_from_ccxt_object(ccxt_order)