adelorenzo
/
stable
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Merge pull request #6543 from froggleston/v3_fixes 

Add support for storing buy candle indicator rows in backtesting results
This commit is contained in:
Matthias 2022-04-23 09:45:11 +02:00 committed by GitHub
parent 5a90d5ece6 580a2c6545
commit 89f16ad3a5
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
12 changed files with 228 additions and 22 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

73
docs/advanced-backtesting.md Normal file
View File

@ -0,0 +1,73 @@
# Advanced Backtesting Analysis
## Analyze the buy/entry and sell/exit tags
It can be helpful to understand how a strategy behaves according to the buy/entry tags used to
mark up different buy conditions. You might want to see more complex statistics about each buy and
sell condition above those provided by the default backtesting output. You may also want to
determine indicator values on the signal candle that resulted in a trade opening.
!!! Note
The following buy reason analysis is only available for backtesting, *not hyperopt*.
We need to run backtesting with the `--export` option set to `signals` to enable the exporting of
signals **and** trades:
``` bash
freqtrade backtesting -c <config.json> --timeframe <tf> --strategy <strategy_name> --timerange=<timerange> --export=signals
```
To analyze the buy tags, we need to use the `buy_reasons.py` script from
[froggleston's repo](https://github.com/froggleston/freqtrade-buyreasons). Follow the instructions
in their README to copy the script into your `freqtrade/scripts/` folder.
This will tell freqtrade to output a pickled dictionary of strategy, pairs and corresponding
DataFrame of the candles that resulted in buy signals. Depending on how many buys your strategy
makes, this file may get quite large, so periodically check your `user_data/backtest_results`
folder to delete old exports.
Before running your next backtest, make sure you either delete your old backtest results or run
backtesting with the `--cache none` option to make sure no cached results are used.
If all goes well, you should now see a `backtest-result-{timestamp}_signals.pkl` file in the
`user_data/backtest_results` folder.
Now run the `buy_reasons.py` script, supplying a few options:
``` bash
python3 scripts/buy_reasons.py -c <config.json> -s <strategy_name> -t <timerange> -g0,1,2,3,4
```
The `-g` option is used to specify the various tabular outputs, ranging from the simplest (0)
to the most detailed per pair, per buy and per sell tag (4). More options are available by
running with the `-h` option.
### Tuning the buy tags and sell tags to display
To show only certain buy and sell tags in the displayed output, use the following two options:
```
--buy_reason_list : Comma separated list of buy signals to analyse. Default: "all"
--sell_reason_list : Comma separated list of sell signals to analyse. Default: "stop_loss,trailing_stop_loss"
```
For example:
```bash
python3 scripts/buy_reasons.py -c <config.json> -s <strategy_name> -t <timerange> -g0,1,2,3,4 --buy_reason_list "buy_tag_a,buy_tag_b" --sell_reason_list "roi,custom_sell_tag_a,stop_loss"
```
### Outputting signal candle indicators
The real power of the buy_reasons.py script comes from the ability to print out the indicator
values present on signal candles to allow fine-grained investigation and tuning of buy signal
indicators. To print out a column for a given set of indicators, use the `--indicator-list`
option:
```bash
python3 scripts/buy_reasons.py -c <config.json> -s <strategy_name> -t <timerange> -g0,1,2,3,4 --buy_reason_list "buy_tag_a,buy_tag_b" --sell_reason_list "roi,custom_sell_tag_a,stop_loss" --indicator_list "rsi,rsi_1h,bb_lowerband,ema_9,macd,macdsignal"
```
The indicators have to be present in your strategy's main DataFrame (either for your main
timeframe or for informative timeframes) otherwise they will simply be ignored in the script
output.

24
docs/backtesting.md
View File

@ -20,7 +20,8 @@ usage: freqtrade backtesting [-h] [-v] [--logfile FILE] [-V] [-c PATH]
[--dry-run-wallet DRY_RUN_WALLET]
[--timeframe-detail TIMEFRAME_DETAIL]
[--strategy-list STRATEGY_LIST [STRATEGY_LIST ...]]
[--export {none,trades}] [--export-filename PATH]
[--export {none,trades,signals}]
[--export-filename PATH]
[--breakdown {day,week,month} [{day,week,month} ...]]
[--cache {none,day,week,month}]
@ -63,18 +64,17 @@ optional arguments:
`30m`, `1h`, `1d`).
--strategy-list STRATEGY_LIST [STRATEGY_LIST ...]
Provide a space-separated list of strategies to
backtest. Please note that timeframe needs to be
set either in config or via command line. When using
this together with `--export trades`, the strategy-
name is injected into the filename (so `backtest-
data.json` becomes `backtest-data-SampleStrategy.json`
--export {none,trades}
backtest. Please note that timeframe needs to be set
either in config or via command line. When using this
together with `--export trades`, the strategy-name is
injected into the filename (so `backtest-data.json`
becomes `backtest-data-SampleStrategy.json`
--export {none,trades,signals}
Export backtest results (default: trades).
--export-filename PATH
Save backtest results to the file with this filename.
Requires `--export` to be set as well. Example:
`--export-filename=user_data/backtest_results/backtest
_today.json`
--export-filename PATH, --backtest-filename PATH
Use this filename for backtest results.Requires
`--export` to be set as well. Example: `--export-filen
ame=user_data/backtest_results/backtest_today.json`
--breakdown {day,week,month} [{day,week,month} ...]
Show backtesting breakdown per [day, week, month].
--cache {none,day,week,month}

6
docs/configuration.md
View File

@ -11,7 +11,7 @@ Per default, the bot loads the configuration from the `config.json` file, locate
You can specify a different configuration file used by the bot with the `-c/--config` command-line option.
If you used the [Quick start](installation.md/#quick-start) method for installing
If you used the [Quick start](installation.md/#quick-start) method for installing
the bot, the installation script should have already created the default configuration file (`config.json`) for you.
If the default configuration file is not created we recommend to use `freqtrade new-config --config config.json` to generate a basic configuration file.
@ -64,7 +64,7 @@ This is similar to using multiple `--config` parameters, but simpler in usage as
"config-private.json"
]
```
``` bash
freqtrade trade --config user_data/config.json <...>
```
@ -100,7 +100,7 @@ This is similar to using multiple `--config` parameters, but simpler in usage as
"stake_amount": "unlimited",
}
```
Resulting combined configuration:
``` json title="Result"

1
docs/data-analysis.md
View File

@ -122,5 +122,6 @@ Best avoid relative paths, since this starts at the storage location of the jupy
* [Strategy debugging](strategy_analysis_example.md) - also available as Jupyter notebook (`user_data/notebooks/strategy_analysis_example.ipynb`)
* [Plotting](plotting.md)
* [Tag Analysis](advanced-backtesting.md)
Feel free to submit an issue or Pull Request enhancing this document if you would like to share ideas on how to best analyze the data.

2
docs/strategy_analysis_example.md
View File

@ -93,7 +93,7 @@ from freqtrade.data.btanalysis import load_backtest_data, load_backtest_stats
# if backtest_dir points to a directory, it'll automatically load the last backtest file.
backtest_dir = config["user_data_dir"] / "backtest_results"
# backtest_dir can also point to a specific file
# backtest_dir can also point to a specific file
# backtest_dir = config["user_data_dir"] / "backtest_results/backtest-result-2020-07-01_20-04-22.json"
```

2
freqtrade/constants.py
View File

@ -14,7 +14,7 @@ PROCESS_THROTTLE_SECS = 5 # sec
HYPEROPT_EPOCH = 100 # epochs
RETRY_TIMEOUT = 30 # sec
TIMEOUT_UNITS = ['minutes', 'seconds']
EXPORT_OPTIONS = ['none', 'trades']
EXPORT_OPTIONS = ['none', 'trades', 'signals']
DEFAULT_DB_PROD_URL = 'sqlite:///tradesv3.sqlite'
DEFAULT_DB_DRYRUN_URL = 'sqlite:///tradesv3.dryrun.sqlite'
UNLIMITED_STAKE_AMOUNT = 'unlimited'

16
freqtrade/misc.py
View File

@ -12,6 +12,7 @@ from typing import Any, Iterator, List, Union
from typing.io import IO
from urllib.parse import urlparse
import joblib
import rapidjson
from freqtrade.constants import DECIMAL_PER_COIN_FALLBACK, DECIMALS_PER_COIN
@ -86,6 +87,21 @@ def file_dump_json(filename: Path, data: Any, is_zip: bool = False, log: bool =
logger.debug(f'done json to "{filename}"')
def file_dump_joblib(filename: Path, data: Any, log: bool = True) -> None:
"""
Dump object data into a file
:param filename: file to create
:param data: Object data to save
:return:
"""
if log:
logger.info(f'dumping joblib to "{filename}"')
with open(filename, 'wb') as fp:
joblib.dump(data, fp)
logger.debug(f'done joblib dump to "{filename}"')
def json_load(datafile: IO) -> Any:
"""
load data with rapidjson

35
freqtrade/optimize/backtesting.py
View File

@ -19,13 +19,15 @@ from freqtrade.data import history
from freqtrade.data.btanalysis import find_existing_backtest_stats, trade_list_to_dataframe
from freqtrade.data.converter import trim_dataframe, trim_dataframes
from freqtrade.data.dataprovider import DataProvider
from freqtrade.enums import BacktestState, CandleType, ExitCheckTuple, ExitType, TradingMode
from freqtrade.enums import (BacktestState, CandleType, ExitCheckTuple, ExitType, RunMode,
TradingMode)
from freqtrade.exceptions import DependencyException, OperationalException
from freqtrade.exchange import timeframe_to_minutes, timeframe_to_seconds
from freqtrade.misc import get_strategy_run_id
from freqtrade.mixins import LoggingMixin
from freqtrade.optimize.bt_progress import BTProgress
from freqtrade.optimize.optimize_reports import (generate_backtest_stats, show_backtest_results,
store_backtest_signal_candles,
store_backtest_stats)
from freqtrade.persistence import LocalTrade, Order, PairLocks, Trade
from freqtrade.plugins.pairlistmanager import PairListManager
@ -73,6 +75,8 @@ class Backtesting:
self.run_ids: Dict[str, str] = {}
self.strategylist: List[IStrategy] = []
self.all_results: Dict[str, Dict] = {}
self.processed_dfs: Dict[str, Dict] = {}
self._exchange_name = self.config['exchange']['name']
self.exchange = ExchangeResolver.load_exchange(self._exchange_name, self.config)
self.dataprovider = DataProvider(self.config, self.exchange)
@ -1070,8 +1074,31 @@ class Backtesting:
})
self.all_results[self.strategy.get_strategy_name()] = results
if (self.config.get('export', 'none') == 'signals' and
self.dataprovider.runmode == RunMode.BACKTEST):
self._generate_trade_signal_candles(preprocessed_tmp, results)
return min_date, max_date
def _generate_trade_signal_candles(self, preprocessed_df, bt_results):
signal_candles_only = {}
for pair in preprocessed_df.keys():
signal_candles_only_df = DataFrame()
pairdf = preprocessed_df[pair]
resdf = bt_results['results']
pairresults = resdf.loc[(resdf["pair"] == pair)]
if pairdf.shape[0] > 0:
for t, v in pairresults.open_date.items():
allinds = pairdf.loc[(pairdf['date'] < v)]
signal_inds = allinds.iloc[[-1]]
signal_candles_only_df = signal_candles_only_df.append(signal_inds)
signal_candles_only[pair] = signal_candles_only_df
self.processed_dfs[self.strategy.get_strategy_name()] = signal_candles_only
def _get_min_cached_backtest_date(self):
min_backtest_date = None
backtest_cache_age = self.config.get('backtest_cache', constants.BACKTEST_CACHE_DEFAULT)
@ -1130,9 +1157,13 @@ class Backtesting:
else:
self.results = results
if self.config.get('export', 'none') == 'trades':
if self.config.get('export', 'none') in ('trades', 'signals'):
store_backtest_stats(self.config['exportfilename'], self.results)
if (self.config.get('export', 'none') == 'signals' and
self.dataprovider.runmode == RunMode.BACKTEST):
store_backtest_signal_candles(self.config['exportfilename'], self.processed_dfs)
# Results may be mixed up now. Sort them so they follow --strategy-list order.
if 'strategy_list' in self.config and len(self.results) > 0:
self.results['strategy_comparison'] = sorted(

27
freqtrade/optimize/optimize_reports.py
View File

@ -11,8 +11,8 @@ from tabulate import tabulate
from freqtrade.constants import DATETIME_PRINT_FORMAT, LAST_BT_RESULT_FN, UNLIMITED_STAKE_AMOUNT
from freqtrade.data.btanalysis import (calculate_csum, calculate_market_change,
calculate_max_drawdown)
from freqtrade.misc import (decimals_per_coin, file_dump_json, get_backtest_metadata_filename,
round_coin_value)
from freqtrade.misc import (decimals_per_coin, file_dump_joblib, file_dump_json,
get_backtest_metadata_filename, round_coin_value)
logger = logging.getLogger(__name__)
@ -45,6 +45,29 @@ def store_backtest_stats(recordfilename: Path, stats: Dict[str, DataFrame]) -> N
file_dump_json(latest_filename, {'latest_backtest': str(filename.name)})
def store_backtest_signal_candles(recordfilename: Path, candles: Dict[str, Dict]) -> Path:
"""
Stores backtest trade signal candles
:param recordfilename: Path object, which can either be a filename or a directory.
Filenames will be appended with a timestamp right before the suffix
while for directories, <directory>/backtest-result-<datetime>_signals.pkl will be used
as filename
:param stats: Dict containing the backtesting signal candles
"""
if recordfilename.is_dir():
filename = (recordfilename /
f'backtest-result-{datetime.now().strftime("%Y-%m-%d_%H-%M-%S")}_signals.pkl')
else:
filename = Path.joinpath(
recordfilename.parent,
f'{recordfilename.stem}-{datetime.now().strftime("%Y-%m-%d_%H-%M-%S")}_signals.pkl'
)
file_dump_joblib(filename, candles)
return filename
def _get_line_floatfmt(stake_currency: str) -> List[str]:
"""
Generate floatformat (goes in line with _generate_result_line())

1
mkdocs.yml
View File

@ -29,6 +29,7 @@ nav:
- Data Analysis:
- Jupyter Notebooks: data-analysis.md
- Strategy analysis: strategy_analysis_example.md
- Backtest analysis: advanced-backtesting.md
- Advanced Topics:
- Advanced Post-installation Tasks: advanced-setup.md
- Edge Positioning: edge.md

5
tests/optimize/test_backtesting.py
View File

@ -384,14 +384,16 @@ def test_backtesting_start(default_conf, mocker, testdatadir, caplog) -> None:
mocker.patch('freqtrade.optimize.backtesting.generate_backtest_stats')
mocker.patch('freqtrade.optimize.backtesting.show_backtest_results')
sbs = mocker.patch('freqtrade.optimize.backtesting.store_backtest_stats')
sbc = mocker.patch('freqtrade.optimize.backtesting.store_backtest_signal_candles')
mocker.patch('freqtrade.plugins.pairlistmanager.PairListManager.whitelist',
PropertyMock(return_value=['UNITTEST/BTC']))
default_conf['timeframe'] = '1m'
default_conf['datadir'] = testdatadir
default_conf['export'] = 'trades'
default_conf['export'] = 'signals'
default_conf['exportfilename'] = 'export.txt'
default_conf['timerange'] = '-1510694220'
default_conf['runmode'] = RunMode.BACKTEST
backtesting = Backtesting(default_conf)
backtesting._set_strategy(backtesting.strategylist[0])
@ -407,6 +409,7 @@ def test_backtesting_start(default_conf, mocker, testdatadir, caplog) -> None:
assert backtesting.strategy.dp._pairlists is not None
assert backtesting.strategy.bot_loop_start.call_count == 1
assert sbs.call_count == 1
assert sbc.call_count == 1
def test_backtesting_start_no_data(default_conf, mocker, caplog, testdatadir) -> None:

58
tests/optimize/test_optimize_reports.py
View File

@ -2,6 +2,7 @@ import re
from datetime import timedelta
from pathlib import Path
import joblib
import pandas as pd
import pytest
from arrow import Arrow
@ -19,6 +20,7 @@ from freqtrade.optimize.optimize_reports import (_get_resample_from_period, gene
generate_periodic_breakdown_stats,
generate_strategy_comparison,
generate_trading_stats, show_sorted_pairlist,
store_backtest_signal_candles,
store_backtest_stats, text_table_bt_results,
text_table_exit_reason, text_table_strategy)
from freqtrade.resolvers.strategy_resolver import StrategyResolver
@ -201,6 +203,62 @@ def test_store_backtest_stats(testdatadir, mocker):
assert str(dump_mock.call_args_list[0][0][0]).startswith(str(testdatadir / 'testresult'))
def test_store_backtest_candles(testdatadir, mocker):
dump_mock = mocker.patch('freqtrade.optimize.optimize_reports.file_dump_joblib')
candle_dict = {'DefStrat': {'UNITTEST/BTC': pd.DataFrame()}}
# mock directory exporting
store_backtest_signal_candles(testdatadir, candle_dict)
assert dump_mock.call_count == 1
assert isinstance(dump_mock.call_args_list[0][0][0], Path)
assert str(dump_mock.call_args_list[0][0][0]).endswith(str('_signals.pkl'))
dump_mock.reset_mock()
# mock file exporting
filename = Path(testdatadir / 'testresult')
store_backtest_signal_candles(filename, candle_dict)
assert dump_mock.call_count == 1
assert isinstance(dump_mock.call_args_list[0][0][0], Path)
# result will be testdatadir / testresult-<timestamp>_signals.pkl
assert str(dump_mock.call_args_list[0][0][0]).endswith(str('_signals.pkl'))
dump_mock.reset_mock()
def test_write_read_backtest_candles(tmpdir):
candle_dict = {'DefStrat': {'UNITTEST/BTC': pd.DataFrame()}}
# test directory exporting
stored_file = store_backtest_signal_candles(Path(tmpdir), candle_dict)
scp = open(stored_file, "rb")
pickled_signal_candles = joblib.load(scp)
scp.close()
assert pickled_signal_candles.keys() == candle_dict.keys()
assert pickled_signal_candles['DefStrat'].keys() == pickled_signal_candles['DefStrat'].keys()
assert pickled_signal_candles['DefStrat']['UNITTEST/BTC'] \
.equals(pickled_signal_candles['DefStrat']['UNITTEST/BTC'])
_clean_test_file(stored_file)
# test file exporting
filename = Path(tmpdir / 'testresult')
stored_file = store_backtest_signal_candles(filename, candle_dict)
scp = open(stored_file, "rb")
pickled_signal_candles = joblib.load(scp)
scp.close()
assert pickled_signal_candles.keys() == candle_dict.keys()
assert pickled_signal_candles['DefStrat'].keys() == pickled_signal_candles['DefStrat'].keys()
assert pickled_signal_candles['DefStrat']['UNITTEST/BTC'] \
.equals(pickled_signal_candles['DefStrat']['UNITTEST/BTC'])
_clean_test_file(stored_file)
def test_generate_pair_metrics():
results = pd.DataFrame(