Merge pull request #4790 from freqtrade/hyper_range

add IntParameter.range

docs/hyperopt.md

@@ -165,11 +165,22 @@ Rarely you may also need to create a [nested class](advanced-hyperopt.md#overrid
 !!! Tip "Quickly optimize ROI, stoploss and trailing stoploss"
     You can quickly optimize the spaces `roi`, `stoploss` and `trailing` without changing anything in your strategy.
 
-    ```python
+    ``` bash
     # Have a working strategy at hand.
     freqtrade hyperopt --hyperopt-loss SharpeHyperOptLossDaily --spaces roi stoploss trailing --strategy MyWorkingStrategy --config config.json -e 100
     ```
 
+### Hyperopt execution logic
+
+Hyperopt will first load your data into memory and will then run `populate_indicators()` once per Pair to generate all indicators.
+
+Hyperopt will then spawn into different processes (number of processors, or `-j <n>`), and run backtesting over and over again, changing the parameters that are part of the `--spaces` defined.
+
+For every new set of parameters, freqtrade will run first `populate_buy_trend()` followed by `populate_sell_trend()`, and then run the regular backtesting process to simulate trades.
+
+After backtesting, the results are passed into the [loss function](#loss-functions), which will evaluate if this result was better or worse than previous results.  
+Based on the loss function result, hyperopt will determine the next set of parameters to try in the next round of backtesting.
+
 ### Configure your Guards and Triggers
 
 There are two places you need to change in your strategy file to add a new buy hyperopt for testing:
@@ -187,60 +198,54 @@ There you have two different types of indicators: 1. `guards` and 2. `triggers`.
     However, this guide will make this distinction to make it clear that signals should not be "sticking".
     Sticking signals are signals that are active for multiple candles. This can lead into buying a signal late (right before the signal disappears - which means that the chance of success is a lot lower than right at the beginning).
 
-Hyper-optimization will, for each epoch round, pick one trigger and possibly
+Hyper-optimization will, for each epoch round, pick one trigger and possibly multiple guards.
-multiple guards. The constructed strategy will be something like "*buy exactly when close price touches lower Bollinger band, BUT only if
-ADX > 10*".
-
-```python
-from freqtrade.strategy import IntParameter, IStrategy
-
-class MyAwesomeStrategy(IStrategy):
-    # If parameter is prefixed with `buy_` or `sell_` then specifying `space` parameter is optional
-    # and space is inferred from parameter name.
-    buy_adx_min = IntParameter(0, 100, default=10)
-
-    def populate_buy_trend(self, dataframe: 'DataFrame', metadata: dict) -> 'DataFrame':
-        dataframe.loc[
-            (
-                (dataframe['adx'] > self.buy_adx_min.value)
-            ), 'buy'] = 1
-        return dataframe
-```
 
 #### Sell optimization
 
 Similar to the buy-signal above, sell-signals can also be optimized.
 Place the corresponding settings into the following methods
 
-* Define the parameters at the class level hyperopt shall be optimizing.
+* Define the parameters at the class level hyperopt shall be optimizing, either naming them `sell_*`, or by explicitly defining `space='sell'`.
 * Within `populate_sell_trend()` - use defined parameter values instead of raw constants.
 
 The configuration and rules are the same than for buy signals.
 
-```python
+## Solving a Mystery
-class MyAwesomeStrategy(IStrategy):
-    # There is no strict parameter naming scheme. If you do not use `buy_` or `sell_` prefixes - 
-    # please specify to which space parameter belongs using `space` parameter. Possible values: 
-    # 'buy' or 'sell'.
-    adx_max = IntParameter(0, 100, default=50, space='sell')
 
-    def populate_sell_trend(self, dataframe: DataFrame, metadata: dict) -> DataFrame:
+Let's say you are curious: should you use MACD crossings or lower Bollinger Bands to trigger your buys. 
-        dataframe.loc[
+And you also wonder should you use RSI or ADX to help with those buy decisions. 
-            (
+If you decide to use RSI or ADX, which values should I use for them? 
-                (dataframe['adx'] < self.adx_max.value)
+
-            ), 'buy'] = 1
+So let's use hyperparameter optimization to solve this mystery.
+
+### Defining indicators to be used
+
+We start by calculating the indicators our strategy is going to use.
+
+``` python
+class MyAwesomeStrategy(IStrategy):
+
+    def populate_indicators(self, dataframe: DataFrame, metadata: dict) -> DataFrame:
+        """
+        Generate all indicators used by the strategy
+        """
+        dataframe['adx'] = ta.ADX(dataframe)
+        dataframe['rsi'] = ta.RSI(dataframe)
+        macd = ta.MACD(dataframe)
+        dataframe['macd'] = macd['macd']
+        dataframe['macdsignal'] = macd['macdsignal']
+        dataframe['macdhist'] = macd['macdhist']
+
+        bollinger = ta.BBANDS(dataframe, timeperiod=20, nbdevup=2.0, nbdevdn=2.0)
+        dataframe['bb_lowerband'] = boll['lowerband']
+        dataframe['bb_middleband'] = boll['middleband']
+        dataframe['bb_upperband'] = boll['upperband']
         return dataframe
 ```
 
-## Solving a Mystery
+### Hyperoptable parameters
 
-Let's say you are curious: should you use MACD crossings or lower Bollinger
+We continue to define hyperoptable parameters:
-Bands to trigger your buys. And you also wonder should you use RSI or ADX to
-help with those buy decisions. If you decide to use RSI or ADX, which values
-should I use for them? So let's use hyperparameter optimization to solve this
-mystery.
-
-We will start by defining hyperoptable parameters:
 
 ```python
 class MyAwesomeStrategy(IStrategy):
@@ -288,7 +293,7 @@ So let's write the buy strategy using these values:
 ```
 
 Hyperopt will now call `populate_buy_trend()` many times (`epochs`) with different value combinations.  
-It will use the given historical data and make buys based on the buy signals generated with the above function.  
+It will use the given historical data and simulate buys based on the buy signals generated with the above function.  
 Based on the results, hyperopt will tell you which parameter combination produced the best results (based on the configured [loss function](#loss-functions)).
 
 !!! Note
@@ -314,6 +319,90 @@ There are four parameter types each suited for different purposes.
 !!! Warning
     Hyperoptable parameters cannot be used in `populate_indicators` - as hyperopt does not recalculate indicators for each epoch, so the starting value would be used in this case.
 
+### Optimizing an indicator parameter
+
+Assuming you have a simple strategy in mind - a EMA cross strategy (2 Moving averages crossing) - and you'd like to find the ideal parameters for this strategy.
+
+``` python
+from pandas import DataFrame
+from functools import reduce
+
+import talib.abstract as ta
+
+from freqtrade.strategy import IStrategy
+from freqtrade.strategy import CategoricalParameter, DecimalParameter, IntParameter
+import freqtrade.vendor.qtpylib.indicators as qtpylib
+
+class MyAwesomeStrategy(IStrategy):
+    stoploss = -0.05
+    timeframe = '15m'
+    # Define the parameter spaces
+    buy_ema_short = IntParameter(3, 50, default=5)
+    buy_ema_long = IntParameter(15, 200, default=50)
+
+
+    def populate_indicators(self, dataframe: DataFrame, metadata: dict) -> DataFrame:
+        """Generate all indicators used by the strategy"""
+        
+        # Calculate all ema_short values
+        for val in self.buy_ema_short.range:
+            dataframe[f'ema_short_{val}'] = ta.EMA(dataframe, timeperiod=val)
+        
+        # Calculate all ema_long values
+        for val in self.buy_ema_long.range:
+            dataframe[f'ema_long_{val}'] = ta.EMA(dataframe, timeperiod=val)
+        
+        return dataframe
+
+    def populate_buy_trend(self, dataframe: DataFrame, metadata: dict) -> DataFrame:
+        conditions = []
+        conditions.append(qtpylib.crossed_above(
+                dataframe[f'ema_short_{self.buy_ema_short.value}'], dataframe[f'ema_long_{self.buy_ema_long.value}']
+            ))
+
+        # Check that volume is not 0
+        conditions.append(dataframe['volume'] > 0)
+
+        if conditions:
+            dataframe.loc[
+                reduce(lambda x, y: x & y, conditions),
+                'buy'] = 1
+        return dataframe
+
+    def populate_sell_trend(self, dataframe: DataFrame, metadata: dict) -> DataFrame:
+        conditions = []
+        conditions.append(qtpylib.crossed_above(
+                dataframe[f'ema_long_{self.buy_ema_long.value}'], dataframe[f'ema_short_{self.buy_ema_short.value}']
+            ))
+
+        # Check that volume is not 0
+        conditions.append(dataframe['volume'] > 0)
+
+        if conditions:
+            dataframe.loc[
+                reduce(lambda x, y: x & y, conditions),
+                'sell'] = 1
+        return dataframe
+```
+
+Breaking it down:
+
+Using `self.buy_ema_short.range` will return a range object containing all entries between the Parameters low and high value.
+In this case (`IntParameter(3, 50, default=5)`), the loop would run for all numbers between 3 and 50 (`[3, 4, 5, ... 49, 50]`).
+By using this in a loop, hyperopt will generate 48 new columns (`['buy_ema_3', 'buy_ema_4', ... , 'buy_ema_50']`).
+
+Hyperopt itself will then use the selected value to create the buy and sell signals
+
+While this strategy is most likely too simple to provide consistent profit, it should serve as an example how optimize indicator parameters.
+
+!!! Note
+    `self.buy_ema_short.range` will act differently between hyperopt and other modes. For hyperopt, the above example may generate 48 new columns, however for all other modes (backtesting, dry/live), it will only generate the column for the selected value. You should therefore avoid using the resulting column with explicit values (values other than `self.buy_ema_short.value`).
+
+??? Hint "Performance tip"
+    By doing the calculation of all possible indicators in `populate_indicators()`, the calculation of the indicator happens only once for every parameter.  
+    While this may slow down the hyperopt startup speed, the overall performance will increase as the Hyperopt execution itself may pick the same value for multiple epochs (changing other values).
+    You should however try to use space ranges as small as possible. Every new column will require more memory, and every possibility hyperopt can try will increase the search space.
+
 ## Loss-functions
 
 Each hyperparameter tuning requires a target. This is usually defined as a loss function (sometimes also called objective function), which should decrease for more desirable results, and increase for bad results.
@@ -606,6 +695,16 @@ number).
 You can also enable position stacking in the configuration file by explicitly setting
 `"position_stacking"=true`.
 
+## Out of Memory errors
+
+As hyperopt consumes a lot of memory (the complete data needs to be in memory once per parallel backtesting process), it's likely that you run into "out of memory" errors.
+To combat these, you have multiple options:
+
+* reduce the amount of pairs
+* reduce the timerange used (`--timerange <timerange>`)
+* reduce the number of parallel processes (`-j <n>`)
+* Increase the memory of your machine
+
 ## Show details of Hyperopt results
 
 After you run Hyperopt for the desired amount of epochs, you can later list all results for analysis, select only best or profitable once, and show the details for any of the epochs previously evaluated. This can be done with the `hyperopt-list` and `hyperopt-show` sub-commands. The usage of these sub-commands is described in the [Utils](utils.md#list-hyperopt-results) chapter.

freqtrade/strategy/hyper.py

@@ -5,7 +5,7 @@ This module defines a base class for auto-hyperoptable strategies.
 import logging
 from abc import ABC, abstractmethod
 from contextlib import suppress
-from typing import Any, Iterator, Optional, Sequence, Tuple, Union
+from typing import Any, Dict, Iterator, Optional, Sequence, Tuple, Union
 
 
 with suppress(ImportError):
@@ -13,6 +13,7 @@ with suppress(ImportError):
     from freqtrade.optimize.space import SKDecimal
 
 from freqtrade.exceptions import OperationalException
+from freqtrade.state import RunMode
 
 
 logger = logging.getLogger(__name__)
@@ -25,6 +26,7 @@ class BaseParameter(ABC):
     category: Optional[str]
     default: Any
     value: Any
+    hyperopt: bool = False
 
     def __init__(self, *, default: Any, space: Optional[str] = None,
                  optimize: bool = True, load: bool = True, **kwargs):
@@ -121,6 +123,20 @@ class IntParameter(NumericParameter):
         """
         return Integer(low=self.low, high=self.high, name=name, **self._space_params)
 
+    @property
+    def range(self):
+        """
+        Get each value in this space as list.
+        Returns a List from low to high (inclusive) in Hyperopt mode.
+        Returns a List with 1 item (`value`) in "non-hyperopt" mode, to avoid
+        calculating 100ds of indicators.
+        """
+        if self.hyperopt:
+            # Scikit-optimize ranges are "inclusive", while python's "range" is exclusive
+            return range(self.low, self.high + 1)
+        else:
+            return range(self.value, self.value + 1)
+
 
 class RealParameter(NumericParameter):
     default: float
@@ -227,12 +243,11 @@ class HyperStrategyMixin(object):
      strategy logic.
     """
 
-    def __init__(self, *args, **kwargs):
+    def __init__(self, config: Dict[str, Any], *args, **kwargs):
         """
         Initialize hyperoptable strategy mixin.
         """
-        self._load_params(getattr(self, 'buy_params', None))
+        self._load_hyper_params(config.get('runmode') == RunMode.HYPEROPT)
-        self._load_params(getattr(self, 'sell_params', None))
 
     def enumerate_parameters(self, category: str = None) -> Iterator[Tuple[str, BaseParameter]]:
         """
@@ -254,18 +269,29 @@ class HyperStrategyMixin(object):
                             (attr_name.startswith(category + '_') and attr.category is None)):
                         yield attr_name, attr
 
-    def _load_params(self, params: dict) -> None:
+    def _load_hyper_params(self, hyperopt: bool = False) -> None:
+        """
+        Load Hyperoptable parameters
+        """
+        self._load_params(getattr(self, 'buy_params', None), 'buy', hyperopt)
+        self._load_params(getattr(self, 'sell_params', None), 'sell', hyperopt)
+
+    def _load_params(self, params: dict, space: str, hyperopt: bool = False) -> None:
         """
         Set optimizeable parameter values.
         :param params: Dictionary with new parameter values.
         """
         if not params:
-            return
+            logger.info(f"No params for {space} found, using default values.")
+
         for attr_name, attr in self.enumerate_parameters():
-            if attr_name in params:
+            attr.hyperopt = hyperopt
+            if params and attr_name in params:
                 if attr.load:
                     attr.value = params[attr_name]
                     logger.info(f'Strategy Parameter: {attr_name} = {attr.value}')
                 else:
                     logger.warning(f'Parameter "{attr_name}" exists, but is disabled. '
                                    f'Default value "{attr.value}" used.')
+            else:
+                logger.info(f'Strategy Parameter(default): {attr_name} = {attr.value}')

tests/optimize/conftest.py

@@ -6,6 +6,7 @@ import pandas as pd
 import pytest
 
 from freqtrade.optimize.hyperopt import Hyperopt
+from freqtrade.state import RunMode
 from freqtrade.strategy.interface import SellType
 from tests.conftest import patch_exchange
 
@@ -15,6 +16,7 @@ def hyperopt_conf(default_conf):
     hyperconf = deepcopy(default_conf)
     hyperconf.update({
         'datadir': Path(default_conf['datadir']),
+        'runmode': RunMode.HYPEROPT,
         'hyperopt': 'DefaultHyperOpt',
         'hyperopt_loss': 'ShortTradeDurHyperOptLoss',
                          'hyperopt_path': str(Path(__file__).parent / 'hyperopts'),

tests/optimize/test_hyperopt.py

@@ -21,6 +21,7 @@ from freqtrade.optimize.hyperopt_tools import HyperoptTools
 from freqtrade.optimize.space import SKDecimal
 from freqtrade.resolvers.hyperopt_resolver import HyperOptResolver
 from freqtrade.state import RunMode
+from freqtrade.strategy.hyper import IntParameter
 from tests.conftest import (get_args, log_has, log_has_re, patch_exchange,
                             patched_configuration_load_config_file)
 
@@ -1103,6 +1104,14 @@ def test_in_strategy_auto_hyperopt(mocker, hyperopt_conf, tmpdir) -> None:
     })
     hyperopt = Hyperopt(hyperopt_conf)
     assert isinstance(hyperopt.custom_hyperopt, HyperOptAuto)
+    assert isinstance(hyperopt.backtesting.strategy.buy_rsi, IntParameter)
+
+    assert hyperopt.backtesting.strategy.buy_rsi.hyperopt is True
+    assert hyperopt.backtesting.strategy.buy_rsi.value == 35
+    buy_rsi_range = hyperopt.backtesting.strategy.buy_rsi.range
+    assert isinstance(buy_rsi_range, range)
+    # Range from 0 - 50 (inclusive)
+    assert len(list(buy_rsi_range)) == 51
 
     hyperopt.start()
 

tests/strategy/test_interface.py

@@ -588,6 +588,14 @@ def test_hyperopt_parameters():
     intpar = IntParameter(low=0, high=5, default=1, space='buy')
     assert intpar.value == 1
     assert isinstance(intpar.get_space(''), Integer)
+    assert isinstance(intpar.range, range)
+    assert len(list(intpar.range)) == 1
+    # Range contains ONLY the default / value.
+    assert list(intpar.range) == [intpar.value]
+    intpar.hyperopt = True
+
+    assert len(list(intpar.range)) == 6
+    assert list(intpar.range) == [0, 1, 2, 3, 4, 5]
 
     fltpar = RealParameter(low=0.0, high=5.5, default=1.0, space='buy')
     assert isinstance(fltpar.get_space(''), Real)