Merge pull request #2094 from hroff-1902/hyperopt-roi-stoploss

Simplify custom hyperopts -- no need to copy ugly methods in every custom implementation

docs/hyperopt.md

@@ -18,19 +18,24 @@ Configuring hyperopt is similar to writing your own strategy, and many tasks wil
 
 ### Checklist on all tasks / possibilities in hyperopt
 
-Depending on the space you want to optimize, only some of the below are required.
+Depending on the space you want to optimize, only some of the below are required:
 
 * fill `populate_indicators` - probably a copy from your strategy
 * fill `buy_strategy_generator` - for buy signal optimization
 * fill `indicator_space` - for buy signal optimzation
 * fill `sell_strategy_generator` - for sell signal optimization
 * fill `sell_indicator_space` - for sell signal optimzation
-* fill `roi_space` - for ROI optimization
+
-* fill `generate_roi_table` - for ROI optimization (if you need more than 3 entries)
+Optional, but recommended:
-* fill `stoploss_space` - stoploss optimization
+
-* Optional but recommended
+* copy `populate_buy_trend` from your strategy - otherwise default-strategy will be used
-  * copy `populate_buy_trend` from your strategy - otherwise default-strategy will be used
+* copy `populate_sell_trend` from your strategy - otherwise default-strategy will be used
-  * copy `populate_sell_trend` from your strategy - otherwise default-strategy will be used
+
+Rarely you may also need to override:
+
+* `roi_space` - for custom ROI optimization (if you need the ranges for the ROI parameters in the optimization hyperspace that differ from default)
+* `generate_roi_table` - for custom ROI optimization (if you need more than 4 entries in the ROI table)
+* `stoploss_space` - for custom stoploss optimization (if you need the range for the stoploss parameter in the optimization hyperspace that differs from default)
 
 ### 1. Install a Custom Hyperopt File
 
@@ -345,7 +350,7 @@ def populate_buy_trend(self, dataframe: DataFrame) -> DataFrame:
 
 ### Understand Hyperopt ROI results
 
-If you are optimizing ROI, you're result will look as follows and include a ROI table.
+If you are optimizing ROI (i.e. if optimization search-space contains 'all' or 'roi'), your result will look as follows and include a ROI table:
 
 ```
 Best result:
@@ -376,6 +381,41 @@ minimal_roi = {
     }
 ```
 
+If you are optimizing ROI, Freqtrade creates the 'roi' optimization hyperspace for you -- it's the hyperspace of components for the ROI tables. By default, each ROI table generated by the Freqtrade consists of 4 rows (steps) with the values that can vary in the following ranges:
+
+| # | minutes | ROI percentage |
+|---|---|---|
+| 1 | always 0 | 0.03...0.31 |
+| 2 | 10...40 | 0.02...0.11 |
+| 3 | 20...100 | 0.01...0.04 |
+| 4 | 30...220 | always 0 |
+
+This structure of the ROI table is sufficient in most cases. Override the `roi_space()` method defining the ranges desired if you need components of the ROI tables to vary in other ranges.
+
+Override the `generate_roi_table()` and `roi_space()` methods and implement your own custom approach for generation of the ROI tables during hyperoptimization in these methods if you need a different structure of the ROI table or other amount of rows (steps) in the ROI tables.
+
+### Understand Hyperopt Stoploss results
+
+If you are optimizing stoploss values (i.e. if optimization search-space contains 'all' or 'stoploss'), your result will look as follows and include stoploss:
+
+```
+Best result:
+
+    44/100:    135 trades. Avg profit  0.57%. Total profit  0.03871918 BTC (0.7722Σ%). Avg duration 180.4 mins. Objective: 1.94367
+
+Buy hyperspace params:
+{   'adx-value': 44,
+    'rsi-value': 29,
+    'adx-enabled': False,
+    'rsi-enabled': True,
+    'trigger': 'bb_lower'}
+Stoploss: -0.37996664668703606
+```
+
+If you are optimizing stoploss values, Freqtrade creates the 'stoploss' optimization hyperspace for you. By default, the stoploss values in that hyperspace can vary in the range -0.5...-0.02, which is sufficient in most cases.
+
+Override the `stoploss_space()` method and define the desired range in it if you need stoploss values to vary in other range during hyperoptimization.
+
 ### Validate backtesting results
 
 Once the optimized strategy has been implemented into your strategy, you should backtest this strategy to make sure everything is working as expected.

freqtrade/optimize/default_hyperopt.py

@@ -5,7 +5,7 @@ from typing import Any, Callable, Dict, List
 
 import talib.abstract as ta
 from pandas import DataFrame
-from skopt.space import Categorical, Dimension, Integer, Real
+from skopt.space import Categorical, Dimension, Integer
 
 import freqtrade.vendor.qtpylib.indicators as qtpylib
 from freqtrade.optimize.hyperopt_interface import IHyperOpt
@@ -13,10 +13,9 @@ from freqtrade.optimize.hyperopt_interface import IHyperOpt
 
 class DefaultHyperOpts(IHyperOpt):
     """
-    Default hyperopt provided by freqtrade bot.
+    Default hyperopt provided by the Freqtrade bot.
     You can override it with your own hyperopt
     """
-
     @staticmethod
     def populate_indicators(dataframe: DataFrame, metadata: dict) -> DataFrame:
         dataframe['adx'] = ta.ADX(dataframe)
@@ -156,42 +155,6 @@ class DefaultHyperOpts(IHyperOpt):
                          'sell-sar_reversal'], name='sell-trigger')
         ]
 
-    @staticmethod
-    def generate_roi_table(params: Dict) -> Dict[int, float]:
-        """
-        Generate the ROI table that will be used by Hyperopt
-        """
-        roi_table = {}
-        roi_table[0] = params['roi_p1'] + params['roi_p2'] + params['roi_p3']
-        roi_table[params['roi_t3']] = params['roi_p1'] + params['roi_p2']
-        roi_table[params['roi_t3'] + params['roi_t2']] = params['roi_p1']
-        roi_table[params['roi_t3'] + params['roi_t2'] + params['roi_t1']] = 0
-
-        return roi_table
-
-    @staticmethod
-    def stoploss_space() -> List[Dimension]:
-        """
-        Stoploss Value to search
-        """
-        return [
-            Real(-0.5, -0.02, name='stoploss'),
-        ]
-
-    @staticmethod
-    def roi_space() -> List[Dimension]:
-        """
-        Values to search for each ROI steps
-        """
-        return [
-            Integer(10, 120, name='roi_t1'),
-            Integer(10, 60, name='roi_t2'),
-            Integer(10, 40, name='roi_t3'),
-            Real(0.01, 0.04, name='roi_p1'),
-            Real(0.01, 0.07, name='roi_p2'),
-            Real(0.01, 0.20, name='roi_p3'),
-        ]
-
     def populate_buy_trend(self, dataframe: DataFrame, metadata: dict) -> DataFrame:
         """
         Based on TA indicators. Should be a copy of from strategy

freqtrade/optimize/hyperopt_interface.py

@@ -7,7 +7,7 @@ from abc import ABC, abstractmethod
 from typing import Dict, Any, Callable, List
 
 from pandas import DataFrame
-from skopt.space import Dimension
+from skopt.space import Dimension, Integer, Real
 
 
 class IHyperOpt(ABC):
@@ -26,56 +26,80 @@ class IHyperOpt(ABC):
     @abstractmethod
     def populate_indicators(dataframe: DataFrame, metadata: dict) -> DataFrame:
         """
-        Populate indicators that will be used in the Buy and Sell strategy
+        Populate indicators that will be used in the Buy and Sell strategy.
-        :param dataframe: Raw data from the exchange and parsed by parse_ticker_dataframe()
+        :param dataframe: Raw data from the exchange and parsed by parse_ticker_dataframe().
-        :return: a Dataframe with all mandatory indicators for the strategies
+        :return: A Dataframe with all mandatory indicators for the strategies.
         """
 
     @staticmethod
     @abstractmethod
     def buy_strategy_generator(params: Dict[str, Any]) -> Callable:
         """
-        Create a buy strategy generator
+        Create a buy strategy generator.
         """
 
     @staticmethod
     @abstractmethod
     def sell_strategy_generator(params: Dict[str, Any]) -> Callable:
         """
-        Create a sell strategy generator
+        Create a sell strategy generator.
         """
 
     @staticmethod
     @abstractmethod
     def indicator_space() -> List[Dimension]:
         """
-        Create an indicator space
+        Create an indicator space.
         """
 
     @staticmethod
     @abstractmethod
     def sell_indicator_space() -> List[Dimension]:
         """
-        Create a sell indicator space
+        Create a sell indicator space.
         """
 
     @staticmethod
-    @abstractmethod
     def generate_roi_table(params: Dict) -> Dict[int, float]:
         """
-        Create an roi table
+        Create a ROI table.
+
+        Generates the ROI table that will be used by Hyperopt.
+        You may override it in your custom Hyperopt class.
         """
+        roi_table = {}
+        roi_table[0] = params['roi_p1'] + params['roi_p2'] + params['roi_p3']
+        roi_table[params['roi_t3']] = params['roi_p1'] + params['roi_p2']
+        roi_table[params['roi_t3'] + params['roi_t2']] = params['roi_p1']
+        roi_table[params['roi_t3'] + params['roi_t2'] + params['roi_t1']] = 0
+
+        return roi_table
 
     @staticmethod
-    @abstractmethod
     def stoploss_space() -> List[Dimension]:
         """
-        Create a stoploss space
+        Create a stoploss space.
+
+        Defines range of stoploss values to search.
+        You may override it in your custom Hyperopt class.
         """
+        return [
+            Real(-0.5, -0.02, name='stoploss'),
+        ]
 
     @staticmethod
-    @abstractmethod
     def roi_space() -> List[Dimension]:
         """
-        Create a roi space
+        Create a ROI space.
+
+        Defines values to search for each ROI steps.
+        You may override it in your custom Hyperopt class.
         """
+        return [
+            Integer(10, 120, name='roi_t1'),
+            Integer(10, 60, name='roi_t2'),
+            Integer(10, 40, name='roi_t3'),
+            Real(0.01, 0.04, name='roi_p1'),
+            Real(0.01, 0.07, name='roi_p2'),
+            Real(0.01, 0.20, name='roi_p3'),
+        ]

user_data/hyperopts/sample_hyperopt.py

@@ -14,20 +14,27 @@ import freqtrade.vendor.qtpylib.indicators as qtpylib
 from freqtrade.optimize.hyperopt_interface import IHyperOpt
 
 
-# This class is a sample. Feel free to customize it.
 class SampleHyperOpts(IHyperOpt):
     """
-    This is a test hyperopt to inspire you.
+    This is a sample hyperopt to inspire you.
-    More information in https://github.com/freqtrade/freqtrade/blob/develop/docs/hyperopt.md
+    Feel free to customize it.
-     You can:
-    - Rename the class name (Do not forget to update class_name)
-    - Add any methods you want to build your hyperopt
-    - Add any lib you need to build your hyperopt
-     You must keep:
-    - the prototype for the methods: populate_indicators, indicator_space, buy_strategy_generator,
-    roi_space, generate_roi_table, stoploss_space
-    """
 
+    More information in https://github.com/freqtrade/freqtrade/blob/develop/docs/hyperopt.md
+
+    You should:
+    - Rename the class name to some unique name.
+    - Add any methods you want to build your hyperopt.
+    - Add any lib you need to build your hyperopt.
+
+    You must keep:
+    - The prototypes for the methods: populate_indicators, indicator_space, buy_strategy_generator.
+
+    The roi_space, generate_roi_table, stoploss_space methods are no longer required to be
+    copied in every custom hyperopt. However, you may override them if you need the
+    'roi' and the 'stoploss' spaces that differ from the defaults offered by Freqtrade.
+    Sample implementation of these methods can be found in
+    https://github.com/freqtrade/freqtrade/blob/develop/user_data/hyperopts/sample_hyperopt_advanced.py
+    """
     @staticmethod
     def populate_indicators(dataframe: DataFrame, metadata: dict) -> DataFrame:
         dataframe['adx'] = ta.ADX(dataframe)
@@ -167,42 +174,6 @@ class SampleHyperOpts(IHyperOpt):
                          'sell-sar_reversal'], name='sell-trigger')
         ]
 
-    @staticmethod
-    def generate_roi_table(params: Dict) -> Dict[int, float]:
-        """
-        Generate the ROI table that will be used by Hyperopt
-        """
-        roi_table = {}
-        roi_table[0] = params['roi_p1'] + params['roi_p2'] + params['roi_p3']
-        roi_table[params['roi_t3']] = params['roi_p1'] + params['roi_p2']
-        roi_table[params['roi_t3'] + params['roi_t2']] = params['roi_p1']
-        roi_table[params['roi_t3'] + params['roi_t2'] + params['roi_t1']] = 0
-
-        return roi_table
-
-    @staticmethod
-    def stoploss_space() -> List[Dimension]:
-        """
-        Stoploss Value to search
-        """
-        return [
-            Real(-0.5, -0.02, name='stoploss'),
-        ]
-
-    @staticmethod
-    def roi_space() -> List[Dimension]:
-        """
-        Values to search for each ROI steps
-        """
-        return [
-            Integer(10, 120, name='roi_t1'),
-            Integer(10, 60, name='roi_t2'),
-            Integer(10, 40, name='roi_t3'),
-            Real(0.01, 0.04, name='roi_p1'),
-            Real(0.01, 0.07, name='roi_p2'),
-            Real(0.01, 0.20, name='roi_p3'),
-        ]
-
     def populate_buy_trend(self, dataframe: DataFrame, metadata: dict) -> DataFrame:
         """
         Based on TA indicators. Should be a copy of from strategy

user_data/hyperopts/sample_hyperopt_advanced.py

@@ -0,0 +1,261 @@
+# pragma pylint: disable=missing-docstring, invalid-name, pointless-string-statement
+
+from functools import reduce
+from math import exp
+from typing import Any, Callable, Dict, List
+from datetime import datetime
+
+import numpy as np# noqa F401
+import talib.abstract as ta
+from pandas import DataFrame
+from skopt.space import Categorical, Dimension, Integer, Real
+
+import freqtrade.vendor.qtpylib.indicators as qtpylib
+from freqtrade.optimize.hyperopt_interface import IHyperOpt
+
+
+class AdvancedSampleHyperOpts(IHyperOpt):
+    """
+    This is a sample hyperopt to inspire you.
+    Feel free to customize it.
+
+    More information in https://github.com/freqtrade/freqtrade/blob/develop/docs/hyperopt.md
+
+    You should:
+    - Rename the class name to some unique name.
+    - Add any methods you want to build your hyperopt.
+    - Add any lib you need to build your hyperopt.
+
+    You must keep:
+    - The prototypes for the methods: populate_indicators, indicator_space, buy_strategy_generator.
+
+    The roi_space, generate_roi_table, stoploss_space methods are no longer required to be
+    copied in every custom hyperopt. However, you may override them if you need the
+    'roi' and the 'stoploss' spaces that differ from the defaults offered by Freqtrade.
+
+    This sample illustrates how to override these methods.
+    """
+    @staticmethod
+    def populate_indicators(dataframe: DataFrame, metadata: dict) -> DataFrame:
+        dataframe['adx'] = ta.ADX(dataframe)
+        macd = ta.MACD(dataframe)
+        dataframe['macd'] = macd['macd']
+        dataframe['macdsignal'] = macd['macdsignal']
+        dataframe['mfi'] = ta.MFI(dataframe)
+        dataframe['rsi'] = ta.RSI(dataframe)
+        stoch_fast = ta.STOCHF(dataframe)
+        dataframe['fastd'] = stoch_fast['fastd']
+        dataframe['minus_di'] = ta.MINUS_DI(dataframe)
+        # Bollinger bands
+        bollinger = qtpylib.bollinger_bands(qtpylib.typical_price(dataframe), window=20, stds=2)
+        dataframe['bb_lowerband'] = bollinger['lower']
+        dataframe['bb_upperband'] = bollinger['upper']
+        dataframe['sar'] = ta.SAR(dataframe)
+        return dataframe
+
+    @staticmethod
+    def buy_strategy_generator(params: Dict[str, Any]) -> Callable:
+        """
+        Define the buy strategy parameters to be used by hyperopt
+        """
+        def populate_buy_trend(dataframe: DataFrame, metadata: dict) -> DataFrame:
+            """
+            Buy strategy Hyperopt will build and use
+            """
+            conditions = []
+            # GUARDS AND TRENDS
+            if 'mfi-enabled' in params and params['mfi-enabled']:
+                conditions.append(dataframe['mfi'] < params['mfi-value'])
+            if 'fastd-enabled' in params and params['fastd-enabled']:
+                conditions.append(dataframe['fastd'] < params['fastd-value'])
+            if 'adx-enabled' in params and params['adx-enabled']:
+                conditions.append(dataframe['adx'] > params['adx-value'])
+            if 'rsi-enabled' in params and params['rsi-enabled']:
+                conditions.append(dataframe['rsi'] < params['rsi-value'])
+
+            # TRIGGERS
+            if 'trigger' in params:
+                if params['trigger'] == 'bb_lower':
+                    conditions.append(dataframe['close'] < dataframe['bb_lowerband'])
+                if params['trigger'] == 'macd_cross_signal':
+                    conditions.append(qtpylib.crossed_above(
+                        dataframe['macd'], dataframe['macdsignal']
+                    ))
+                if params['trigger'] == 'sar_reversal':
+                    conditions.append(qtpylib.crossed_above(
+                        dataframe['close'], dataframe['sar']
+                    ))
+
+            if conditions:
+                dataframe.loc[
+                    reduce(lambda x, y: x & y, conditions),
+                    'buy'] = 1
+
+            return dataframe
+
+        return populate_buy_trend
+
+    @staticmethod
+    def indicator_space() -> List[Dimension]:
+        """
+        Define your Hyperopt space for searching strategy parameters
+        """
+        return [
+            Integer(10, 25, name='mfi-value'),
+            Integer(15, 45, name='fastd-value'),
+            Integer(20, 50, name='adx-value'),
+            Integer(20, 40, name='rsi-value'),
+            Categorical([True, False], name='mfi-enabled'),
+            Categorical([True, False], name='fastd-enabled'),
+            Categorical([True, False], name='adx-enabled'),
+            Categorical([True, False], name='rsi-enabled'),
+            Categorical(['bb_lower', 'macd_cross_signal', 'sar_reversal'], name='trigger')
+        ]
+
+    @staticmethod
+    def sell_strategy_generator(params: Dict[str, Any]) -> Callable:
+        """
+        Define the sell strategy parameters to be used by hyperopt
+        """
+        def populate_sell_trend(dataframe: DataFrame, metadata: dict) -> DataFrame:
+            """
+            Sell strategy Hyperopt will build and use
+            """
+            # print(params)
+            conditions = []
+            # GUARDS AND TRENDS
+            if 'sell-mfi-enabled' in params and params['sell-mfi-enabled']:
+                conditions.append(dataframe['mfi'] > params['sell-mfi-value'])
+            if 'sell-fastd-enabled' in params and params['sell-fastd-enabled']:
+                conditions.append(dataframe['fastd'] > params['sell-fastd-value'])
+            if 'sell-adx-enabled' in params and params['sell-adx-enabled']:
+                conditions.append(dataframe['adx'] < params['sell-adx-value'])
+            if 'sell-rsi-enabled' in params and params['sell-rsi-enabled']:
+                conditions.append(dataframe['rsi'] > params['sell-rsi-value'])
+
+            # TRIGGERS
+            if 'sell-trigger' in params:
+                if params['sell-trigger'] == 'sell-bb_upper':
+                    conditions.append(dataframe['close'] > dataframe['bb_upperband'])
+                if params['sell-trigger'] == 'sell-macd_cross_signal':
+                    conditions.append(qtpylib.crossed_above(
+                        dataframe['macdsignal'], dataframe['macd']
+                    ))
+                if params['sell-trigger'] == 'sell-sar_reversal':
+                    conditions.append(qtpylib.crossed_above(
+                        dataframe['sar'], dataframe['close']
+                    ))
+
+            if conditions:
+                dataframe.loc[
+                    reduce(lambda x, y: x & y, conditions),
+                    'sell'] = 1
+
+            return dataframe
+
+        return populate_sell_trend
+
+    @staticmethod
+    def sell_indicator_space() -> List[Dimension]:
+        """
+        Define your Hyperopt space for searching sell strategy parameters
+        """
+        return [
+            Integer(75, 100, name='sell-mfi-value'),
+            Integer(50, 100, name='sell-fastd-value'),
+            Integer(50, 100, name='sell-adx-value'),
+            Integer(60, 100, name='sell-rsi-value'),
+            Categorical([True, False], name='sell-mfi-enabled'),
+            Categorical([True, False], name='sell-fastd-enabled'),
+            Categorical([True, False], name='sell-adx-enabled'),
+            Categorical([True, False], name='sell-rsi-enabled'),
+            Categorical(['sell-bb_upper',
+                         'sell-macd_cross_signal',
+                         'sell-sar_reversal'], name='sell-trigger')
+        ]
+
+    @staticmethod
+    def generate_roi_table(params: Dict) -> Dict[int, float]:
+        """
+        Generate the ROI table that will be used by Hyperopt
+
+        This implementation generates the default legacy Freqtrade ROI tables.
+
+        Change it if you need different number of steps in the generated
+        ROI tables or other structure of the ROI tables.
+
+        Please keep it aligned with parameters in the 'roi' optimization
+        hyperspace defined by the roi_space method.
+        """
+        roi_table = {}
+        roi_table[0] = params['roi_p1'] + params['roi_p2'] + params['roi_p3']
+        roi_table[params['roi_t3']] = params['roi_p1'] + params['roi_p2']
+        roi_table[params['roi_t3'] + params['roi_t2']] = params['roi_p1']
+        roi_table[params['roi_t3'] + params['roi_t2'] + params['roi_t1']] = 0
+
+        return roi_table
+
+    @staticmethod
+    def roi_space() -> List[Dimension]:
+        """
+        Values to search for each ROI steps
+
+        Override it if you need some different ranges for the parameters in the
+        'roi' optimization hyperspace.
+
+        Please keep it aligned with the implementation of the
+        generate_roi_table method.
+        """
+        return [
+            Integer(10, 120, name='roi_t1'),
+            Integer(10, 60, name='roi_t2'),
+            Integer(10, 40, name='roi_t3'),
+            Real(0.01, 0.04, name='roi_p1'),
+            Real(0.01, 0.07, name='roi_p2'),
+            Real(0.01, 0.20, name='roi_p3'),
+        ]
+
+    @staticmethod
+    def stoploss_space() -> List[Dimension]:
+        """
+        Stoploss Value to search
+
+        Override it if you need some different range for the parameter in the
+        'stoploss' optimization hyperspace.
+        """
+        return [
+            Real(-0.5, -0.02, name='stoploss'),
+        ]
+
+    def populate_buy_trend(self, dataframe: DataFrame, metadata: dict) -> DataFrame:
+        """
+        Based on TA indicators. Should be a copy of from strategy
+        must align to populate_indicators in this file
+        Only used when --spaces does not include buy
+        """
+        dataframe.loc[
+            (
+                (dataframe['close'] < dataframe['bb_lowerband']) &
+                (dataframe['mfi'] < 16) &
+                (dataframe['adx'] > 25) &
+                (dataframe['rsi'] < 21)
+            ),
+            'buy'] = 1
+
+        return dataframe
+
+    def populate_sell_trend(self, dataframe: DataFrame, metadata: dict) -> DataFrame:
+        """
+        Based on TA indicators. Should be a copy of from strategy
+        must align to populate_indicators in this file
+        Only used when --spaces does not include sell
+        """
+        dataframe.loc[
+            (
+                (qtpylib.crossed_above(
+                    dataframe['macdsignal'], dataframe['macd']
+                )) &
+                (dataframe['fastd'] > 54)
+            ),
+            'sell'] = 1
+        return dataframe