adelorenzo/stable
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Update documentation regarding Legacy Hyperopt

Browse Source
This commit is contained in:
Matthias 2021-09-11 17:52:47 +02:00
parent fd6bf591f8
commit 3675df8344
15 changed files with 55 additions and 527 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

8
README.md
View File

@ -79,22 +79,22 @@ For any other type of installation please refer to [Installation doc](https://ww
``` ```
usage: freqtrade [-h] [-V] usage: freqtrade [-h] [-V]
{trade,create-userdir,new-config,new-hyperopt,new-strategy,download-data,convert-data,convert-trade-data,backtesting,edge,hyperopt,hyperopt-list,hyperopt-show,list-exchanges,list-hyperopts,list-markets,list-pairs,list-strategies,list-timeframes,show-trades,test-pairlist,plot-dataframe,plot-profit} {trade,create-userdir,new-config,new-strategy,download-data,convert-data,convert-trade-data,list-data,backtesting,edge,hyperopt,hyperopt-list,hyperopt-show,list-exchanges,list-hyperopts,list-markets,list-pairs,list-strategies,list-timeframes,show-trades,test-pairlist,install-ui,plot-dataframe,plot-profit,webserver}
... ...
Free, open source crypto trading bot Free, open source crypto trading bot
positional arguments: positional arguments:
{trade,create-userdir,new-config,new-hyperopt,new-strategy,download-data,convert-data,convert-trade-data,backtesting,edge,hyperopt,hyperopt-list,hyperopt-show,list-exchanges,list-hyperopts,list-markets,list-pairs,list-strategies,list-timeframes,show-trades,test-pairlist,plot-dataframe,plot-profit} {trade,create-userdir,new-config,new-strategy,download-data,convert-data,convert-trade-data,list-data,backtesting,edge,hyperopt,hyperopt-list,hyperopt-show,list-exchanges,list-hyperopts,list-markets,list-pairs,list-strategies,list-timeframes,show-trades,test-pairlist,install-ui,plot-dataframe,plot-profit,webserver}
trade Trade module. trade Trade module.
create-userdir Create user-data directory. create-userdir Create user-data directory.
new-config Create new config new-config Create new config
new-hyperopt Create new hyperopt
new-strategy Create new strategy new-strategy Create new strategy
download-data Download backtesting data. download-data Download backtesting data.
convert-data Convert candle (OHLCV) data from one format to convert-data Convert candle (OHLCV) data from one format to
another. another.
convert-trade-data Convert trade data from one format to another. convert-trade-data Convert trade data from one format to another.
list-data List downloaded data.
backtesting Backtesting module. backtesting Backtesting module.
edge Edge module. edge Edge module.
hyperopt Hyperopt module. hyperopt Hyperopt module.
@ -108,8 +108,10 @@ positional arguments:
list-timeframes Print available timeframes for the exchange. list-timeframes Print available timeframes for the exchange.
show-trades Show trades. show-trades Show trades.
test-pairlist Test your pairlist configuration. test-pairlist Test your pairlist configuration.
install-ui Install FreqUI
plot-dataframe Plot candles with indicators. plot-dataframe Plot candles with indicators.
plot-profit Generate plot showing profits. plot-profit Generate plot showing profits.
webserver Webserver module.
optional arguments: optional arguments:
-h, --help show this help message and exit -h, --help show this help message and exit

298
docs/advanced-hyperopt.md
View File

@ -67,10 +67,10 @@ Currently, the arguments are:
This function needs to return a floating point number (`float`). Smaller numbers will be interpreted as better results. The parameters and balancing for this is up to you. This function needs to return a floating point number (`float`). Smaller numbers will be interpreted as better results. The parameters and balancing for this is up to you.
!!! Note !!! Note
This function is called once per iteration - so please make sure to have this as optimized as possible to not slow hyperopt down unnecessarily. This function is called once per epoch - so please make sure to have this as optimized as possible to not slow hyperopt down unnecessarily.
!!! Note !!! Note "`*args` and `**kwargs`"
Please keep the arguments `*args` and `**kwargs` in the interface to allow us to extend this interface later. Please keep the arguments `*args` and `**kwargs` in the interface to allow us to extend this interface in the future.
## Overriding pre-defined spaces ## Overriding pre-defined spaces
@ -82,8 +82,22 @@ class MyAwesomeStrategy(IStrategy):
# Define a custom stoploss space. # Define a custom stoploss space.
def stoploss_space(): def stoploss_space():
return [SKDecimal(-0.05, -0.01, decimals=3, name='stoploss')] return [SKDecimal(-0.05, -0.01, decimals=3, name='stoploss')]
# Define custom ROI space
def roi_space() -> List[Dimension]:
return [
Integer(10, 120, name='roi_t1'),
Integer(10, 60, name='roi_t2'),
Integer(10, 40, name='roi_t3'),
SKDecimal(0.01, 0.04, decimals=3, name='roi_p1'),
SKDecimal(0.01, 0.07, decimals=3, name='roi_p2'),
SKDecimal(0.01, 0.20, decimals=3, name='roi_p3'),
]
``` ```
!!! Note
All overrides are optional and can be mixed/matched as necessary.
## Space options ## Space options
For the additional spaces, scikit-optimize (in combination with Freqtrade) provides the following space types: For the additional spaces, scikit-optimize (in combination with Freqtrade) provides the following space types:
@ -105,281 +119,3 @@ from freqtrade.optimize.space import Categorical, Dimension, Integer, SKDecimal,
Assuming the definition of a rather small space (`SKDecimal(0.10, 0.15, decimals=2, name='xxx')`) - SKDecimal will have 5 possibilities (`[0.10, 0.11, 0.12, 0.13, 0.14, 0.15]`). Assuming the definition of a rather small space (`SKDecimal(0.10, 0.15, decimals=2, name='xxx')`) - SKDecimal will have 5 possibilities (`[0.10, 0.11, 0.12, 0.13, 0.14, 0.15]`).
A corresponding real space `Real(0.10, 0.15 name='xxx')` on the other hand has an almost unlimited number of possibilities (`[0.10, 0.010000000001, 0.010000000002, ... 0.014999999999, 0.01500000000]`). A corresponding real space `Real(0.10, 0.15 name='xxx')` on the other hand has an almost unlimited number of possibilities (`[0.10, 0.010000000001, 0.010000000002, ... 0.014999999999, 0.01500000000]`).
---
## Legacy Hyperopt
This Section explains the configuration of an explicit Hyperopt file (separate to the strategy).
!!! Warning "Deprecated / legacy mode"
Since the 2021.4 release you no longer have to write a separate hyperopt class, but all strategies can be hyperopted.
Please read the [main hyperopt page](hyperopt.md) for more details.
### Prepare hyperopt file
Configuring an explicit hyperopt file is similar to writing your own strategy, and many tasks will be similar.
!!! Tip "About this page"
For this page, we will be using a fictional strategy called `AwesomeStrategy` - which will be optimized using the `AwesomeHyperopt` class.
#### Create a Custom Hyperopt File
The simplest way to get started is to use the following command, which will create a new hyperopt file from a template, which will be located under `user_data/hyperopts/AwesomeHyperopt.py`.
Let assume you want a hyperopt file `AwesomeHyperopt.py`:
``` bash
freqtrade new-hyperopt --hyperopt AwesomeHyperopt
```
#### Legacy Hyperopt checklist
Checklist on all tasks / possibilities in hyperopt
Depending on the space you want to optimize, only some of the below are required:
* fill `buy_strategy_generator` - for buy signal optimization
* fill `indicator_space` - for buy signal optimization
* fill `sell_strategy_generator` - for sell signal optimization
* fill `sell_indicator_space` - for sell signal optimization
!!! Note
`populate_indicators` needs to create all indicators any of thee spaces may use, otherwise hyperopt will not work.
Optional in hyperopt - can also be loaded from a strategy (recommended):
* `populate_indicators` - fallback to create indicators
* `populate_buy_trend` - fallback if not optimizing for buy space. should come from strategy
* `populate_sell_trend` - fallback if not optimizing for sell space. should come from strategy
!!! Note
You always have to provide a strategy to Hyperopt, even if your custom Hyperopt class contains all methods.
Assuming the optional methods are not in your hyperopt file, please use `--strategy AweSomeStrategy` which contains these methods so hyperopt can use these methods instead.
Rarely you may also need to override:
* `roi_space` - for custom ROI optimization (if you need the ranges for the ROI parameters in the optimization hyperspace that differ from default)
* `generate_roi_table` - for custom ROI optimization (if you need the ranges for the values in the ROI table that differ from default or the number of entries (steps) in the ROI table which differs from the default 4 steps)
* `stoploss_space` - for custom stoploss optimization (if you need the range for the stoploss parameter in the optimization hyperspace that differs from default)
* `trailing_space` - for custom trailing stop optimization (if you need the ranges for the trailing stop parameters in the optimization hyperspace that differ from default)
#### Defining a buy signal optimization
Let's say you are curious: should you use MACD crossings or lower Bollinger
Bands to trigger your buys. And you also wonder should you use RSI or ADX to
help with those buy decisions. If you decide to use RSI or ADX, which values
should I use for them? So let's use hyperparameter optimization to solve this
mystery.
We will start by defining a search space:
```python
def indicator_space() -> List[Dimension]:
"""
Define your Hyperopt space for searching strategy parameters
"""
return [
Integer(20, 40, name='adx-value'),
Integer(20, 40, name='rsi-value'),
Categorical([True, False], name='adx-enabled'),
Categorical([True, False], name='rsi-enabled'),
Categorical(['bb_lower', 'macd_cross_signal'], name='trigger')
]
```
Above definition says: I have five parameters I want you to randomly combine
to find the best combination. Two of them are integer values (`adx-value` and `rsi-value`) and I want you test in the range of values 20 to 40.
Then we have three category variables. First two are either `True` or `False`.
We use these to either enable or disable the ADX and RSI guards.
The last one we call `trigger` and use it to decide which buy trigger we want to use.
So let's write the buy strategy generator using these values:
```python
@staticmethod
def buy_strategy_generator(params: Dict[str, Any]) -> Callable:
"""
Define the buy strategy parameters to be used by Hyperopt.
"""
def populate_buy_trend(dataframe: DataFrame, metadata: dict) -> DataFrame:
conditions = []
# GUARDS AND TRENDS
if 'adx-enabled' in params and params['adx-enabled']:
conditions.append(dataframe['adx'] > params['adx-value'])
if 'rsi-enabled' in params and params['rsi-enabled']:
conditions.append(dataframe['rsi'] < params['rsi-value'])
# TRIGGERS
if 'trigger' in params:
if params['trigger'] == 'bb_lower':
conditions.append(dataframe['close'] < dataframe['bb_lowerband'])
if params['trigger'] == 'macd_cross_signal':
conditions.append(qtpylib.crossed_above(
dataframe['macd'], dataframe['macdsignal']
))
# Check that volume is not 0
conditions.append(dataframe['volume'] > 0)
if conditions:
dataframe.loc[
reduce(lambda x, y: x & y, conditions),
'buy'] = 1
return dataframe
return populate_buy_trend
```
Hyperopt will now call `populate_buy_trend()` many times (`epochs`) with different value combinations.
It will use the given historical data and make buys based on the buy signals generated with the above function.
Based on the results, hyperopt will tell you which parameter combination produced the best results (based on the configured [loss function](#loss-functions)).
!!! Note
The above setup expects to find ADX, RSI and Bollinger Bands in the populated indicators.
When you want to test an indicator that isn't used by the bot currently, remember to
add it to the `populate_indicators()` method in your strategy or hyperopt file.
#### Sell optimization
Similar to the buy-signal above, sell-signals can also be optimized.
Place the corresponding settings into the following methods
* Inside `sell_indicator_space()` - the parameters hyperopt shall be optimizing.
* Within `sell_strategy_generator()` - populate the nested method `populate_sell_trend()` to apply the parameters.
The configuration and rules are the same than for buy signals.
To avoid naming collisions in the search-space, please prefix all sell-spaces with `sell-`.
### Execute Hyperopt
Once you have updated your hyperopt configuration you can run it.
Because hyperopt tries a lot of combinations to find the best parameters it will take time to get a good result. More time usually results in better results.
We strongly recommend to use `screen` or `tmux` to prevent any connection loss.
```bash
freqtrade hyperopt --config config.json --hyperopt <hyperoptname> --hyperopt-loss <hyperoptlossname> --strategy <strategyname> -e 500 --spaces all
```
Use `<hyperoptname>` as the name of the custom hyperopt used.
The `-e` option will set how many evaluations hyperopt will do. Since hyperopt uses Bayesian search, running too many epochs at once may not produce greater results. Experience has shown that best results are usually not improving much after 500-1000 epochs.
Doing multiple runs (executions) with a few 1000 epochs and different random state will most likely produce different results.
The `--spaces all` option determines that all possible parameters should be optimized. Possibilities are listed below.
!!! Note
Hyperopt will store hyperopt results with the timestamp of the hyperopt start time.
Reading commands (`hyperopt-list`, `hyperopt-show`) can use `--hyperopt-filename <filename>` to read and display older hyperopt results.
You can find a list of filenames with `ls -l user_data/hyperopt_results/`.
#### Running Hyperopt using methods from a strategy
Hyperopt can reuse `populate_indicators`, `populate_buy_trend`, `populate_sell_trend` from your strategy, assuming these methods are **not** in your custom hyperopt file, and a strategy is provided.
```bash
freqtrade hyperopt --hyperopt AwesomeHyperopt --hyperopt-loss SharpeHyperOptLossDaily --strategy AwesomeStrategy
```
### Understand the Hyperopt Result
Once Hyperopt is completed you can use the result to create a new strategy.
Given the following result from hyperopt:
```
Best result:
44/100: 135 trades. Avg profit 0.57%. Total profit 0.03871918 BTC (0.7722%). Avg duration 180.4 mins. Objective: 1.94367
Buy hyperspace params:
{ 'adx-value': 44,
'rsi-value': 29,
'adx-enabled': False,
'rsi-enabled': True,
'trigger': 'bb_lower'}
```
You should understand this result like:
* The buy trigger that worked best was `bb_lower`.
* You should not use ADX because `adx-enabled: False`)
* You should **consider** using the RSI indicator (`rsi-enabled: True` and the best value is `29.0` (`rsi-value: 29.0`)
You have to look inside your strategy file into `buy_strategy_generator()`
method, what those values match to.
So for example you had `rsi-value: 29.0` so we would look at `rsi`-block, that translates to the following code block:
```python
(dataframe['rsi'] < 29.0)
```
Translating your whole hyperopt result as the new buy-signal would then look like:
```python
def populate_buy_trend(self, dataframe: DataFrame) -> DataFrame:
dataframe.loc[
(
(dataframe['rsi'] < 29.0) & # rsi-value
dataframe['close'] < dataframe['bb_lowerband'] # trigger
),
'buy'] = 1
return dataframe
```
### Validate backtesting results
Once the optimized parameters and conditions have been implemented into your strategy, you should backtest the strategy to make sure everything is working as expected.
To achieve same results (number of trades, their durations, profit, etc.) than during Hyperopt, please use same configuration and parameters (timerange, timeframe, ...) used for hyperopt `--dmmp`/`--disable-max-market-positions` and `--eps`/`--enable-position-stacking` for Backtesting.
Should results not match, please double-check to make sure you transferred all conditions correctly.
Pay special care to the stoploss (and trailing stoploss) parameters, as these are often set in configuration files, which override changes to the strategy.
You should also carefully review the log of your backtest to ensure that there were no parameters inadvertently set by the configuration (like `stoploss` or `trailing_stop`).
### Sharing methods with your strategy
Hyperopt classes provide access to the Strategy via the `strategy` class attribute.
This can be a great way to reduce code duplication if used correctly, but will also complicate usage for inexperienced users.
``` python
from pandas import DataFrame
from freqtrade.strategy.interface import IStrategy
import freqtrade.vendor.qtpylib.indicators as qtpylib
class MyAwesomeStrategy(IStrategy):
buy_params = {
'rsi-value': 30,
'adx-value': 35,
}
def populate_buy_trend(self, dataframe: DataFrame, metadata: dict) -> DataFrame:
return self.buy_strategy_generator(self.buy_params, dataframe, metadata)
@staticmethod
def buy_strategy_generator(params, dataframe: DataFrame, metadata: dict) -> DataFrame:
dataframe.loc[
(
qtpylib.crossed_above(dataframe['rsi'], params['rsi-value']) &
dataframe['adx'] > params['adx-value']) &
dataframe['volume'] > 0
)
, 'buy'] = 1
return dataframe
class MyAwesomeHyperOpt(IHyperOpt):
...
@staticmethod
def buy_strategy_generator(params: Dict[str, Any]) -> Callable:
"""
Define the buy strategy parameters to be used by Hyperopt.
"""
def populate_buy_trend(dataframe: DataFrame, metadata: dict) -> DataFrame:
# Call strategy's buy strategy generator
return self.StrategyClass.buy_strategy_generator(params, dataframe, metadata)
return populate_buy_trend
```

8
docs/bot-usage.md
View File

@ -12,22 +12,22 @@ This page explains the different parameters of the bot and how to run it.
``` ```
usage: freqtrade [-h] [-V] usage: freqtrade [-h] [-V]
{trade,create-userdir,new-config,new-hyperopt,new-strategy,download-data,convert-data,convert-trade-data,backtesting,edge,hyperopt,hyperopt-list,hyperopt-show,list-exchanges,list-hyperopts,list-markets,list-pairs,list-strategies,list-timeframes,show-trades,test-pairlist,plot-dataframe,plot-profit} {trade,create-userdir,new-config,new-strategy,download-data,convert-data,convert-trade-data,list-data,backtesting,edge,hyperopt,hyperopt-list,hyperopt-show,list-exchanges,list-hyperopts,list-markets,list-pairs,list-strategies,list-timeframes,show-trades,test-pairlist,install-ui,plot-dataframe,plot-profit,webserver}
... ...
Free, open source crypto trading bot Free, open source crypto trading bot
positional arguments: positional arguments:
{trade,create-userdir,new-config,new-hyperopt,new-strategy,download-data,convert-data,convert-trade-data,backtesting,edge,hyperopt,hyperopt-list,hyperopt-show,list-exchanges,list-hyperopts,list-markets,list-pairs,list-strategies,list-timeframes,show-trades,test-pairlist,plot-dataframe,plot-profit} {trade,create-userdir,new-config,new-strategy,download-data,convert-data,convert-trade-data,list-data,backtesting,edge,hyperopt,hyperopt-list,hyperopt-show,list-exchanges,list-hyperopts,list-markets,list-pairs,list-strategies,list-timeframes,show-trades,test-pairlist,install-ui,plot-dataframe,plot-profit,webserver}
trade Trade module. trade Trade module.
create-userdir Create user-data directory. create-userdir Create user-data directory.
new-config Create new config new-config Create new config
new-hyperopt Create new hyperopt
new-strategy Create new strategy new-strategy Create new strategy
download-data Download backtesting data. download-data Download backtesting data.
convert-data Convert candle (OHLCV) data from one format to convert-data Convert candle (OHLCV) data from one format to
another. another.
convert-trade-data Convert trade data from one format to another. convert-trade-data Convert trade data from one format to another.
list-data List downloaded data.
backtesting Backtesting module. backtesting Backtesting module.
edge Edge module. edge Edge module.
hyperopt Hyperopt module. hyperopt Hyperopt module.
@ -41,8 +41,10 @@ positional arguments:
list-timeframes Print available timeframes for the exchange. list-timeframes Print available timeframes for the exchange.
show-trades Show trades. show-trades Show trades.
test-pairlist Test your pairlist configuration. test-pairlist Test your pairlist configuration.
install-ui Install FreqUI
plot-dataframe Plot candles with indicators. plot-dataframe Plot candles with indicators.
plot-profit Generate plot showing profits. plot-profit Generate plot showing profits.
webserver Webserver module.
optional arguments: optional arguments:
-h, --help show this help message and exit -h, --help show this help message and exit

5
docs/deprecated.md
View File

@ -38,3 +38,8 @@ Since only quoteVolume can be compared between assets, the other options (bidVol
Using `order_book_min` and `order_book_max` used to allow stepping the orderbook and trying to find the next ROI slot - trying to place sell-orders early. Using `order_book_min` and `order_book_max` used to allow stepping the orderbook and trying to find the next ROI slot - trying to place sell-orders early.
As this does however increase risk and provides no benefit, it's been removed for maintainability purposes in 2021.7. As this does however increase risk and provides no benefit, it's been removed for maintainability purposes in 2021.7.
### Legacy Hyperopt mode
Using separate hyperopt files was deprecated in 2021.4 and was removed in 2021.9.
Please switch to the new [Parametrized Strategies](hyperopt.md) to benefit from the new hyperopt interface.

2
docs/faq.md
View File

@ -167,7 +167,7 @@ Since hyperopt uses Bayesian search, running for too many epochs may not produce
It's therefore recommended to run between 500-1000 epochs over and over until you hit at least 10.000 epochs in total (or are satisfied with the result). You can best judge by looking at the results - if the bot keeps discovering better strategies, it's best to keep on going. It's therefore recommended to run between 500-1000 epochs over and over until you hit at least 10.000 epochs in total (or are satisfied with the result). You can best judge by looking at the results - if the bot keeps discovering better strategies, it's best to keep on going.
```bash ```bash
freqtrade hyperopt --hyperopt SampleHyperopt --hyperopt-loss SharpeHyperOptLossDaily --strategy SampleStrategy -e 1000 freqtrade hyperopt --hyperopt-loss SharpeHyperOptLossDaily --strategy SampleStrategy -e 1000
``` ```
### Why does it take a long time to run hyperopt? ### Why does it take a long time to run hyperopt?

21
docs/hyperopt.md
View File

@ -44,9 +44,8 @@ usage: freqtrade hyperopt [-h] [-v] [--logfile FILE] [-V] [-c PATH] [-d PATH]
[--data-format-ohlcv {json,jsongz,hdf5}] [--data-format-ohlcv {json,jsongz,hdf5}]
[--max-open-trades INT] [--max-open-trades INT]
[--stake-amount STAKE_AMOUNT] [--fee FLOAT] [--stake-amount STAKE_AMOUNT] [--fee FLOAT]
[-p PAIRS [PAIRS ...]] [--hyperopt NAME] [-p PAIRS [PAIRS ...]] [--hyperopt-path PATH]
[--hyperopt-path PATH] [--eps] [--dmmp] [--eps] [--dmmp] [--enable-protections]
[--enable-protections]
[--dry-run-wallet DRY_RUN_WALLET] [-e INT] [--dry-run-wallet DRY_RUN_WALLET] [-e INT]
[--spaces {all,buy,sell,roi,stoploss,trailing,protection,default} [{all,buy,sell,roi,stoploss,trailing,protection,default} ...]] [--spaces {all,buy,sell,roi,stoploss,trailing,protection,default} [{all,buy,sell,roi,stoploss,trailing,protection,default} ...]]
[--print-all] [--no-color] [--print-json] [-j JOBS] [--print-all] [--no-color] [--print-json] [-j JOBS]
@ -73,10 +72,8 @@ optional arguments:
-p PAIRS [PAIRS ...], --pairs PAIRS [PAIRS ...] -p PAIRS [PAIRS ...], --pairs PAIRS [PAIRS ...]
Limit command to these pairs. Pairs are space- Limit command to these pairs. Pairs are space-
separated. separated.
--hyperopt NAME Specify hyperopt class name which will be used by the --hyperopt-path PATH Specify additional lookup path for Hyperopt Loss
bot. functions.
--hyperopt-path PATH Specify additional lookup path for Hyperopt and
Hyperopt Loss functions.
--eps, --enable-position-stacking --eps, --enable-position-stacking
Allow buying the same pair multiple times (position Allow buying the same pair multiple times (position
stacking). stacking).
@ -558,7 +555,7 @@ For example, to use one month of data, pass `--timerange 20210101-20210201` (fro
Full command: Full command:
```bash ```bash
freqtrade hyperopt --hyperopt <hyperoptname> --strategy <strategyname> --timerange 20210101-20210201 freqtrade hyperopt --strategy <strategyname> --timerange 20210101-20210201
``` ```
### Running Hyperopt with Smaller Search Space ### Running Hyperopt with Smaller Search Space
@ -684,7 +681,7 @@ If you have the `generate_roi_table()` and `roi_space()` methods in your custom
Override the `roi_space()` method if you need components of the ROI tables to vary in other ranges. Override the `generate_roi_table()` and `roi_space()` methods and implement your own custom approach for generation of the ROI tables during hyperoptimization if you need a different structure of the ROI tables or other amount of rows (steps). Override the `roi_space()` method if you need components of the ROI tables to vary in other ranges. Override the `generate_roi_table()` and `roi_space()` methods and implement your own custom approach for generation of the ROI tables during hyperoptimization if you need a different structure of the ROI tables or other amount of rows (steps).
A sample for these methods can be found in [sample_hyperopt_advanced.py](https://github.com/freqtrade/freqtrade/blob/develop/freqtrade/templates/sample_hyperopt_advanced.py). A sample for these methods can be found in the [overriding pre-defined spaces section](advanced-hyperopt.md#overriding-pre-defined-spaces).
!!! Note "Reduced search space" !!! Note "Reduced search space"
To limit the search space further, Decimals are limited to 3 decimal places (a precision of 0.001). This is usually sufficient, every value more precise than this will usually result in overfitted results. You can however [overriding pre-defined spaces](advanced-hyperopt.md#pverriding-pre-defined-spaces) to change this to your needs. To limit the search space further, Decimals are limited to 3 decimal places (a precision of 0.001). This is usually sufficient, every value more precise than this will usually result in overfitted results. You can however [overriding pre-defined spaces](advanced-hyperopt.md#pverriding-pre-defined-spaces) to change this to your needs.
@ -726,7 +723,7 @@ If you are optimizing stoploss values, Freqtrade creates the 'stoploss' optimiza
If you have the `stoploss_space()` method in your custom hyperopt file, remove it in order to utilize Stoploss hyperoptimization space generated by Freqtrade by default. If you have the `stoploss_space()` method in your custom hyperopt file, remove it in order to utilize Stoploss hyperoptimization space generated by Freqtrade by default.
Override the `stoploss_space()` method and define the desired range in it if you need stoploss values to vary in other range during hyperoptimization. A sample for this method can be found in [user_data/hyperopts/sample_hyperopt_advanced.py](https://github.com/freqtrade/freqtrade/blob/develop/freqtrade/templates/sample_hyperopt_advanced.py). Override the `stoploss_space()` method and define the desired range in it if you need stoploss values to vary in other range during hyperoptimization. A sample for this method can be found in the [overriding pre-defined spaces section](advanced-hyperopt.md#overriding-pre-defined-spaces).
!!! Note "Reduced search space" !!! Note "Reduced search space"
To limit the search space further, Decimals are limited to 3 decimal places (a precision of 0.001). This is usually sufficient, every value more precise than this will usually result in overfitted results. You can however [overriding pre-defined spaces](advanced-hyperopt.md#pverriding-pre-defined-spaces) to change this to your needs. To limit the search space further, Decimals are limited to 3 decimal places (a precision of 0.001). This is usually sufficient, every value more precise than this will usually result in overfitted results. You can however [overriding pre-defined spaces](advanced-hyperopt.md#pverriding-pre-defined-spaces) to change this to your needs.
@ -764,10 +761,10 @@ As stated in the comment, you can also use it as the values of the corresponding
If you are optimizing trailing stop values, Freqtrade creates the 'trailing' optimization hyperspace for you. By default, the `trailing_stop` parameter is always set to True in that hyperspace, the value of the `trailing_only_offset_is_reached` vary between True and False, the values of the `trailing_stop_positive` and `trailing_stop_positive_offset` parameters vary in the ranges 0.02...0.35 and 0.01...0.1 correspondingly, which is sufficient in most cases. If you are optimizing trailing stop values, Freqtrade creates the 'trailing' optimization hyperspace for you. By default, the `trailing_stop` parameter is always set to True in that hyperspace, the value of the `trailing_only_offset_is_reached` vary between True and False, the values of the `trailing_stop_positive` and `trailing_stop_positive_offset` parameters vary in the ranges 0.02...0.35 and 0.01...0.1 correspondingly, which is sufficient in most cases.
Override the `trailing_space()` method and define the desired range in it if you need values of the trailing stop parameters to vary in other ranges during hyperoptimization. A sample for this method can be found in [user_data/hyperopts/sample_hyperopt_advanced.py](https://github.com/freqtrade/freqtrade/blob/develop/freqtrade/templates/sample_hyperopt_advanced.py). Override the `trailing_space()` method and define the desired range in it if you need values of the trailing stop parameters to vary in other ranges during hyperoptimization. A sample for this method can be found in the [overriding pre-defined spaces section](advanced-hyperopt.md#overriding-pre-defined-spaces).
!!! Note "Reduced search space" !!! Note "Reduced search space"
To limit the search space further, Decimals are limited to 3 decimal places (a precision of 0.001). This is usually sufficient, every value more precise than this will usually result in overfitted results. You can however [overriding pre-defined spaces](advanced-hyperopt.md#pverriding-pre-defined-spaces) to change this to your needs. To limit the search space further, Decimals are limited to 3 decimal places (a precision of 0.001). This is usually sufficient, every value more precise than this will usually result in overfitted results. You can however [overriding pre-defined spaces](advanced-hyperopt.md#overriding-pre-defined-spaces) to change this to your needs.
### Reproducible results ### Reproducible results

2
docs/utils.md
View File

@ -26,9 +26,7 @@ optional arguments:
├── data ├── data
├── hyperopt_results ├── hyperopt_results
├── hyperopts ├── hyperopts
│   ├── sample_hyperopt_advanced.py
│   ├── sample_hyperopt_loss.py │   ├── sample_hyperopt_loss.py
│   └── sample_hyperopt.py
├── notebooks ├── notebooks
│   └── strategy_analysis_example.ipynb │   └── strategy_analysis_example.ipynb
├── plot ├── plot

2
freqtrade/commands/arguments.py
View File

@ -90,7 +90,7 @@ ARGS_HYPEROPT_SHOW = ["hyperopt_list_best", "hyperopt_list_profitable", "hyperop
NO_CONF_REQURIED = ["convert-data", "convert-trade-data", "download-data", "list-timeframes", NO_CONF_REQURIED = ["convert-data", "convert-trade-data", "download-data", "list-timeframes",
"list-markets", "list-pairs", "list-strategies", "list-data", "list-markets", "list-pairs", "list-strategies", "list-data",
"list-hyperopts", "hyperopt-list", "hyperopt-show", "hyperopt-list", "hyperopt-show",
"plot-dataframe", "plot-profit", "show-trades"] "plot-dataframe", "plot-profit", "show-trades"]
NO_CONF_ALLOWED = ["create-userdir", "list-exchanges", "new-strategy"] NO_CONF_ALLOWED = ["create-userdir", "list-exchanges", "new-strategy"]

4
freqtrade/commands/cli_options.py
View File

@ -1,7 +1,7 @@
""" """
Definition of cli arguments used in arguments.py Definition of cli arguments used in arguments.py
""" """
from argparse import ArgumentTypeError from argparse import SUPPRESS, ArgumentTypeError
from freqtrade import __version__, constants from freqtrade import __version__, constants
from freqtrade.constants import HYPEROPT_LOSS_BUILTIN from freqtrade.constants import HYPEROPT_LOSS_BUILTIN
@ -203,7 +203,7 @@ AVAILABLE_CLI_OPTIONS = {
# Hyperopt # Hyperopt
"hyperopt": Arg( "hyperopt": Arg(
'--hyperopt', '--hyperopt',
help='Specify hyperopt class name which will be used by the bot.', help=SUPPRESS,
metavar='NAME', metavar='NAME',
required=False, required=False,
), ),

2
freqtrade/constants.py
View File

@ -69,9 +69,7 @@ DUST_PER_COIN = {
# Source files with destination directories within user-directory # Source files with destination directories within user-directory
USER_DATA_FILES = { USER_DATA_FILES = {
'sample_strategy.py': USERPATH_STRATEGIES, 'sample_strategy.py': USERPATH_STRATEGIES,
'sample_hyperopt_advanced.py': USERPATH_HYPEROPTS,
'sample_hyperopt_loss.py': USERPATH_HYPEROPTS, 'sample_hyperopt_loss.py': USERPATH_HYPEROPTS,
'sample_hyperopt.py': USERPATH_HYPEROPTS,
'strategy_analysis_example.ipynb': USERPATH_NOTEBOOKS, 'strategy_analysis_example.ipynb': USERPATH_NOTEBOOKS,
} }

4
tests/exchange/test_exchange.py
View File

@ -185,7 +185,7 @@ def test_exchange_resolver(default_conf, mocker, caplog):
def test_validate_order_time_in_force(default_conf, mocker, caplog): def test_validate_order_time_in_force(default_conf, mocker, caplog):
caplog.set_level(logging.INFO) caplog.set_level(logging.INFO)
# explicitly test bittrex, exchanges implementing other policies need seperate tests # explicitly test bittrex, exchanges implementing other policies need separate tests
ex = get_patched_exchange(mocker, default_conf, id="bittrex") ex = get_patched_exchange(mocker, default_conf, id="bittrex")
tif = { tif = {
"buy": "gtc", "buy": "gtc",
@ -2464,7 +2464,7 @@ def test_fetch_order(default_conf, mocker, exchange_name, caplog):
@pytest.mark.parametrize("exchange_name", EXCHANGES) @pytest.mark.parametrize("exchange_name", EXCHANGES)
def test_fetch_stoploss_order(default_conf, mocker, exchange_name): def test_fetch_stoploss_order(default_conf, mocker, exchange_name):
# Don't test FTX here - that needs a seperate test # Don't test FTX here - that needs a separate test
if exchange_name == 'ftx': if exchange_name == 'ftx':
return return
default_conf['dry_run'] = True default_conf['dry_run'] = True

202
tests/optimize/hyperopts/hyperopt_test_sep_file.py
View File

@ -1,202 +0,0 @@
# pragma pylint: disable=missing-docstring, invalid-name, pointless-string-statement
from functools import reduce
from typing import Any, Callable, Dict, List
import talib.abstract as ta
from pandas import DataFrame
from skopt.space import Categorical, Dimension, Integer
import freqtrade.vendor.qtpylib.indicators as qtpylib
from freqtrade.optimize.hyperopt_interface import IHyperOpt
class HyperoptTestSepFile(IHyperOpt):
"""
Default hyperopt provided by the Freqtrade bot.
You can override it with your own Hyperopt
"""
@staticmethod
def populate_indicators(dataframe: DataFrame, metadata: dict) -> DataFrame:
"""
Add several indicators needed for buy and sell strategies defined below.
"""
# ADX
dataframe['adx'] = ta.ADX(dataframe)
# MACD
macd = ta.MACD(dataframe)
dataframe['macd'] = macd['macd']
dataframe['macdsignal'] = macd['macdsignal']
# MFI
dataframe['mfi'] = ta.MFI(dataframe)
# RSI
dataframe['rsi'] = ta.RSI(dataframe)
# Stochastic Fast
stoch_fast = ta.STOCHF(dataframe)
dataframe['fastd'] = stoch_fast['fastd']
# Minus-DI
dataframe['minus_di'] = ta.MINUS_DI(dataframe)
# Bollinger bands
bollinger = qtpylib.bollinger_bands(qtpylib.typical_price(dataframe), window=20, stds=2)
dataframe['bb_lowerband'] = bollinger['lower']
dataframe['bb_upperband'] = bollinger['upper']
# SAR
dataframe['sar'] = ta.SAR(dataframe)
return dataframe
@staticmethod
def buy_strategy_generator(params: Dict[str, Any]) -> Callable:
"""
Define the buy strategy parameters to be used by Hyperopt.
"""
def populate_buy_trend(dataframe: DataFrame, metadata: dict) -> DataFrame:
"""
Buy strategy Hyperopt will build and use.
"""
conditions = []
# GUARDS AND TRENDS
if 'mfi-enabled' in params and params['mfi-enabled']:
conditions.append(dataframe['mfi'] < params['mfi-value'])
if 'fastd-enabled' in params and params['fastd-enabled']:
conditions.append(dataframe['fastd'] < params['fastd-value'])
if 'adx-enabled' in params and params['adx-enabled']:
conditions.append(dataframe['adx'] > params['adx-value'])
if 'rsi-enabled' in params and params['rsi-enabled']:
conditions.append(dataframe['rsi'] < params['rsi-value'])
# TRIGGERS
if 'trigger' in params:
if params['trigger'] == 'bb_lower':
conditions.append(dataframe['close'] < dataframe['bb_lowerband'])
if params['trigger'] == 'macd_cross_signal':
conditions.append(qtpylib.crossed_above(
dataframe['macd'], dataframe['macdsignal']
))
if params['trigger'] == 'sar_reversal':
conditions.append(qtpylib.crossed_above(
dataframe['close'], dataframe['sar']
))
if conditions:
dataframe.loc[
reduce(lambda x, y: x & y, conditions),
'buy'] = 1
return dataframe
return populate_buy_trend
@staticmethod
def indicator_space() -> List[Dimension]:
"""
Define your Hyperopt space for searching buy strategy parameters.
"""
return [
Integer(10, 25, name='mfi-value'),
Integer(15, 45, name='fastd-value'),
Integer(20, 50, name='adx-value'),
Integer(20, 40, name='rsi-value'),
Categorical([True, False], name='mfi-enabled'),
Categorical([True, False], name='fastd-enabled'),
Categorical([True, False], name='adx-enabled'),
Categorical([True, False], name='rsi-enabled'),
Categorical(['bb_lower', 'macd_cross_signal', 'sar_reversal'], name='trigger')
]
@staticmethod
def sell_strategy_generator(params: Dict[str, Any]) -> Callable:
"""
Define the sell strategy parameters to be used by Hyperopt.
"""
def populate_sell_trend(dataframe: DataFrame, metadata: dict) -> DataFrame:
"""
Sell strategy Hyperopt will build and use.
"""
conditions = []
# GUARDS AND TRENDS
if 'sell-mfi-enabled' in params and params['sell-mfi-enabled']:
conditions.append(dataframe['mfi'] > params['sell-mfi-value'])
if 'sell-fastd-enabled' in params and params['sell-fastd-enabled']:
conditions.append(dataframe['fastd'] > params['sell-fastd-value'])
if 'sell-adx-enabled' in params and params['sell-adx-enabled']:
conditions.append(dataframe['adx'] < params['sell-adx-value'])
if 'sell-rsi-enabled' in params and params['sell-rsi-enabled']:
conditions.append(dataframe['rsi'] > params['sell-rsi-value'])
# TRIGGERS
if 'sell-trigger' in params:
if params['sell-trigger'] == 'sell-bb_upper':
conditions.append(dataframe['close'] > dataframe['bb_upperband'])
if params['sell-trigger'] == 'sell-macd_cross_signal':
conditions.append(qtpylib.crossed_above(
dataframe['macdsignal'], dataframe['macd']
))
if params['sell-trigger'] == 'sell-sar_reversal':
conditions.append(qtpylib.crossed_above(
dataframe['sar'], dataframe['close']
))
if conditions:
dataframe.loc[
reduce(lambda x, y: x & y, conditions),
'sell'] = 1
return dataframe
return populate_sell_trend
@staticmethod
def sell_indicator_space() -> List[Dimension]:
"""
Define your Hyperopt space for searching sell strategy parameters.
"""
return [
Integer(75, 100, name='sell-mfi-value'),
Integer(50, 100, name='sell-fastd-value'),
Integer(50, 100, name='sell-adx-value'),
Integer(60, 100, name='sell-rsi-value'),
Categorical([True, False], name='sell-mfi-enabled'),
Categorical([True, False], name='sell-fastd-enabled'),
Categorical([True, False], name='sell-adx-enabled'),
Categorical([True, False], name='sell-rsi-enabled'),
Categorical(['sell-bb_upper',
'sell-macd_cross_signal',
'sell-sar_reversal'], name='sell-trigger')
]
def populate_buy_trend(self, dataframe: DataFrame, metadata: dict) -> DataFrame:
"""
Based on TA indicators. Should be a copy of same method from strategy.
Must align to populate_indicators in this file.
Only used when --spaces does not include buy space.
"""
dataframe.loc[
(
(dataframe['close'] < dataframe['bb_lowerband']) &
(dataframe['mfi'] < 16) &
(dataframe['adx'] > 25) &
(dataframe['rsi'] < 21)
),
'buy'] = 1
return dataframe
def populate_sell_trend(self, dataframe: DataFrame, metadata: dict) -> DataFrame:
"""
Based on TA indicators. Should be a copy of same method from strategy.
Must align to populate_indicators in this file.
Only used when --spaces does not include sell space.
"""
dataframe.loc[
(
(qtpylib.crossed_above(
dataframe['macdsignal'], dataframe['macd']
)) &
(dataframe['fastd'] > 54)
),
'sell'] = 1
return dataframe

14
tests/optimize/test_hyperopt.py
View File

@ -21,8 +21,6 @@ from freqtrade.strategy.hyper import IntParameter
from tests.conftest import (get_args, log_has, log_has_re, patch_exchange, from tests.conftest import (get_args, log_has, log_has_re, patch_exchange,
patched_configuration_load_config_file) patched_configuration_load_config_file)
from .hyperopts.hyperopt_test_sep_file import HyperoptTestSepFile
def test_setup_hyperopt_configuration_without_arguments(mocker, default_conf, caplog) -> None: def test_setup_hyperopt_configuration_without_arguments(mocker, default_conf, caplog) -> None:
patched_configuration_load_config_file(mocker, default_conf) patched_configuration_load_config_file(mocker, default_conf)
@ -30,7 +28,7 @@ def test_setup_hyperopt_configuration_without_arguments(mocker, default_conf, ca
args = [ args = [
'hyperopt', 'hyperopt',
'--config', 'config.json', '--config', 'config.json',
'--hyperopt', 'HyperoptTestSepFile', '--strategy', 'HyperoptableStrategy',
] ]
config = setup_optimize_configuration(get_args(args), RunMode.HYPEROPT) config = setup_optimize_configuration(get_args(args), RunMode.HYPEROPT)
@ -62,7 +60,7 @@ def test_setup_hyperopt_configuration_with_arguments(mocker, default_conf, caplo
args = [ args = [
'hyperopt', 'hyperopt',
'--config', 'config.json', '--config', 'config.json',
'--hyperopt', 'HyperoptTestSepFile', '--strategy', 'HyperoptableStrategy',
'--datadir', '/foo/bar', '--datadir', '/foo/bar',
'--timeframe', '1m', '--timeframe', '1m',
'--timerange', ':100', '--timerange', ':100',
@ -114,7 +112,7 @@ def test_setup_hyperopt_configuration_stake_amount(mocker, default_conf) -> None
args = [ args = [
'hyperopt', 'hyperopt',
'--config', 'config.json', '--config', 'config.json',
'--hyperopt', 'HyperoptTestSepFile', '--strategy', 'HyperoptableStrategy',
'--stake-amount', '1', '--stake-amount', '1',
'--starting-balance', '2' '--starting-balance', '2'
] ]
@ -142,9 +140,7 @@ def test_start_not_installed(mocker, default_conf, import_fails) -> None:
args = [ args = [
'hyperopt', 'hyperopt',
'--config', 'config.json', '--config', 'config.json',
'--hyperopt', 'HyperoptTestSepFile', '--strategy', 'HyperoptableStrategy',
'--hyperopt-path',
str(Path(__file__).parent / "hyperopts"),
'--epochs', '5', '--epochs', '5',
'--hyperopt-loss', 'SharpeHyperOptLossDaily', '--hyperopt-loss', 'SharpeHyperOptLossDaily',
] ]
@ -203,7 +199,7 @@ def test_start_filelock(mocker, hyperopt_conf, caplog) -> None:
args = [ args = [
'hyperopt', 'hyperopt',
'--config', 'config.json', '--config', 'config.json',
'--hyperopt', 'HyperoptTestSepFile', '--strategy', 'HyperoptableStrategy',
'--hyperopt-loss', 'SharpeHyperOptLossDaily', '--hyperopt-loss', 'SharpeHyperOptLossDaily',
'--epochs', '5' '--epochs', '5'
] ]

2
tests/plugins/test_pairlocks.py
View File

@ -68,7 +68,7 @@ def test_PairLocks(use_db):
# Global lock # Global lock
PairLocks.lock_pair('*', lock_time) PairLocks.lock_pair('*', lock_time)
assert PairLocks.is_global_lock(lock_time + timedelta(minutes=-50)) assert PairLocks.is_global_lock(lock_time + timedelta(minutes=-50))
# Global lock also locks every pair seperately # Global lock also locks every pair separately
assert PairLocks.is_pair_locked(pair, lock_time + timedelta(minutes=-50)) assert PairLocks.is_pair_locked(pair, lock_time + timedelta(minutes=-50))
assert PairLocks.is_pair_locked('XRP/USDT', lock_time + timedelta(minutes=-50)) assert PairLocks.is_pair_locked('XRP/USDT', lock_time + timedelta(minutes=-50))

8
tests/test_directory_operations.py
View File

@ -74,16 +74,12 @@ def test_copy_sample_files(mocker, default_conf, caplog) -> None:
copymock = mocker.patch('shutil.copy', MagicMock()) copymock = mocker.patch('shutil.copy', MagicMock())
copy_sample_files(Path('/tmp/bar')) copy_sample_files(Path('/tmp/bar'))
assert copymock.call_count == 5 assert copymock.call_count == 3
assert copymock.call_args_list[0][0][1] == str( assert copymock.call_args_list[0][0][1] == str(
Path('/tmp/bar') / 'strategies/sample_strategy.py') Path('/tmp/bar') / 'strategies/sample_strategy.py')
assert copymock.call_args_list[1][0][1] == str( assert copymock.call_args_list[1][0][1] == str(
Path('/tmp/bar') / 'hyperopts/sample_hyperopt_advanced.py')
assert copymock.call_args_list[2][0][1] == str(
Path('/tmp/bar') / 'hyperopts/sample_hyperopt_loss.py') Path('/tmp/bar') / 'hyperopts/sample_hyperopt_loss.py')
assert copymock.call_args_list[3][0][1] == str( assert copymock.call_args_list[2][0][1] == str(
Path('/tmp/bar') / 'hyperopts/sample_hyperopt.py')
assert copymock.call_args_list[4][0][1] == str(
Path('/tmp/bar') / 'notebooks/strategy_analysis_example.ipynb') Path('/tmp/bar') / 'notebooks/strategy_analysis_example.ipynb')