Merge pull request #6543 from froggleston/v3_fixes
Add support for storing buy candle indicator rows in backtesting results
This commit is contained in:
Matthias
GitHub
commit
89f16ad3a5
73
docs/advanced-backtesting.md
Normal file
73
docs/advanced-backtesting.md
Normal file
@ -0,0 +1,73 @@
|
||||
# Advanced Backtesting Analysis
|
||||
|
||||
## Analyze the buy/entry and sell/exit tags
|
||||
|
||||
It can be helpful to understand how a strategy behaves according to the buy/entry tags used to
|
||||
mark up different buy conditions. You might want to see more complex statistics about each buy and
|
||||
sell condition above those provided by the default backtesting output. You may also want to
|
||||
determine indicator values on the signal candle that resulted in a trade opening.
|
||||
|
||||
!!! Note
|
||||
The following buy reason analysis is only available for backtesting, *not hyperopt*.
|
||||
|
||||
We need to run backtesting with the `--export` option set to `signals` to enable the exporting of
|
||||
signals **and** trades:
|
||||
|
||||
``` bash
|
||||
freqtrade backtesting -c <config.json> --timeframe <tf> --strategy <strategy_name> --timerange=<timerange> --export=signals
|
||||
```
|
||||
|
||||
To analyze the buy tags, we need to use the `buy_reasons.py` script from
|
||||
[froggleston's repo](https://github.com/froggleston/freqtrade-buyreasons). Follow the instructions
|
||||
in their README to copy the script into your `freqtrade/scripts/` folder.
|
||||
|
||||
This will tell freqtrade to output a pickled dictionary of strategy, pairs and corresponding
|
||||
DataFrame of the candles that resulted in buy signals. Depending on how many buys your strategy
|
||||
makes, this file may get quite large, so periodically check your `user_data/backtest_results`
|
||||
folder to delete old exports.
|
||||
|
||||
Before running your next backtest, make sure you either delete your old backtest results or run
|
||||
backtesting with the `--cache none` option to make sure no cached results are used.
|
||||
|
||||
If all goes well, you should now see a `backtest-result-{timestamp}_signals.pkl` file in the
|
||||
`user_data/backtest_results` folder.
|
||||
|
||||
Now run the `buy_reasons.py` script, supplying a few options:
|
||||
|
||||
``` bash
|
||||
python3 scripts/buy_reasons.py -c <config.json> -s <strategy_name> -t <timerange> -g0,1,2,3,4
|
||||
```
|
||||
|
||||
The `-g` option is used to specify the various tabular outputs, ranging from the simplest (0)
|
||||
to the most detailed per pair, per buy and per sell tag (4). More options are available by
|
||||
running with the `-h` option.
|
||||
|
||||
### Tuning the buy tags and sell tags to display
|
||||
|
||||
To show only certain buy and sell tags in the displayed output, use the following two options:
|
||||
|
||||
```
|
||||
--buy_reason_list : Comma separated list of buy signals to analyse. Default: "all"
|
||||
--sell_reason_list : Comma separated list of sell signals to analyse. Default: "stop_loss,trailing_stop_loss"
|
||||
```
|
||||
|
||||
For example:
|
||||
|
||||
```bash
|
||||
python3 scripts/buy_reasons.py -c <config.json> -s <strategy_name> -t <timerange> -g0,1,2,3,4 --buy_reason_list "buy_tag_a,buy_tag_b" --sell_reason_list "roi,custom_sell_tag_a,stop_loss"
|
||||
```
|
||||
|
||||
### Outputting signal candle indicators
|
||||
|
||||
The real power of the buy_reasons.py script comes from the ability to print out the indicator
|
||||
values present on signal candles to allow fine-grained investigation and tuning of buy signal
|
||||
indicators. To print out a column for a given set of indicators, use the `--indicator-list`
|
||||
option:
|
||||
|
||||
```bash
|
||||
python3 scripts/buy_reasons.py -c <config.json> -s <strategy_name> -t <timerange> -g0,1,2,3,4 --buy_reason_list "buy_tag_a,buy_tag_b" --sell_reason_list "roi,custom_sell_tag_a,stop_loss" --indicator_list "rsi,rsi_1h,bb_lowerband,ema_9,macd,macdsignal"
|
||||
```
|
||||
|
||||
The indicators have to be present in your strategy's main DataFrame (either for your main
|
||||
timeframe or for informative timeframes) otherwise they will simply be ignored in the script
|
||||
output.
|
||||
@ -20,7 +20,8 @@ usage: freqtrade backtesting [-h] [-v] [--logfile FILE] [-V] [-c PATH]
|
||||
[--dry-run-wallet DRY_RUN_WALLET]
|
||||
[--timeframe-detail TIMEFRAME_DETAIL]
|
||||
[--strategy-list STRATEGY_LIST [STRATEGY_LIST ...]]
|
||||
[--export {none,trades}] [--export-filename PATH]
|
||||
[--export {none,trades,signals}]
|
||||
[--export-filename PATH]
|
||||
[--breakdown {day,week,month} [{day,week,month} ...]]
|
||||
[--cache {none,day,week,month}]
|
||||
|
||||
@ -63,18 +64,17 @@ optional arguments:
|
||||
`30m`, `1h`, `1d`).
|
||||
--strategy-list STRATEGY_LIST [STRATEGY_LIST ...]
|
||||
Provide a space-separated list of strategies to
|
||||
backtest. Please note that timeframe needs to be
|
||||
set either in config or via command line. When using
|
||||
this together with `--export trades`, the strategy-
|
||||
name is injected into the filename (so `backtest-
|
||||
data.json` becomes `backtest-data-SampleStrategy.json`
|
||||
--export {none,trades}
|
||||
backtest. Please note that timeframe needs to be set
|
||||
either in config or via command line. When using this
|
||||
together with `--export trades`, the strategy-name is
|
||||
injected into the filename (so `backtest-data.json`
|
||||
becomes `backtest-data-SampleStrategy.json`
|
||||
--export {none,trades,signals}
|
||||
Export backtest results (default: trades).
|
||||
--export-filename PATH
|
||||
Save backtest results to the file with this filename.
|
||||
Requires `--export` to be set as well. Example:
|
||||
`--export-filename=user_data/backtest_results/backtest
|
||||
_today.json`
|
||||
--export-filename PATH, --backtest-filename PATH
|
||||
Use this filename for backtest results.Requires
|
||||
`--export` to be set as well. Example: `--export-filen
|
||||
ame=user_data/backtest_results/backtest_today.json`
|
||||
--breakdown {day,week,month} [{day,week,month} ...]
|
||||
Show backtesting breakdown per [day, week, month].
|
||||
--cache {none,day,week,month}
|
||||
|
||||
@ -122,5 +122,6 @@ Best avoid relative paths, since this starts at the storage location of the jupy
|
||||
|
||||
* [Strategy debugging](strategy_analysis_example.md) - also available as Jupyter notebook (`user_data/notebooks/strategy_analysis_example.ipynb`)
|
||||
* [Plotting](plotting.md)
|
||||
* [Tag Analysis](advanced-backtesting.md)
|
||||
|
||||
Feel free to submit an issue or Pull Request enhancing this document if you would like to share ideas on how to best analyze the data.
|
||||
|
||||
@ -14,7 +14,7 @@ PROCESS_THROTTLE_SECS = 5 # sec
|
||||
HYPEROPT_EPOCH = 100 # epochs
|
||||
RETRY_TIMEOUT = 30 # sec
|
||||
TIMEOUT_UNITS = ['minutes', 'seconds']
|
||||
EXPORT_OPTIONS = ['none', 'trades']
|
||||
EXPORT_OPTIONS = ['none', 'trades', 'signals']
|
||||
DEFAULT_DB_PROD_URL = 'sqlite:///tradesv3.sqlite'
|
||||
DEFAULT_DB_DRYRUN_URL = 'sqlite:///tradesv3.dryrun.sqlite'
|
||||
UNLIMITED_STAKE_AMOUNT = 'unlimited'
|
||||
|
||||
@ -12,6 +12,7 @@ from typing import Any, Iterator, List, Union
|
||||
from typing.io import IO
|
||||
from urllib.parse import urlparse
|
||||
|
||||
import joblib
|
||||
import rapidjson
|
||||
|
||||
from freqtrade.constants import DECIMAL_PER_COIN_FALLBACK, DECIMALS_PER_COIN
|
||||
@ -86,6 +87,21 @@ def file_dump_json(filename: Path, data: Any, is_zip: bool = False, log: bool =
|
||||
logger.debug(f'done json to "{filename}"')
|
||||
|
||||
|
||||
def file_dump_joblib(filename: Path, data: Any, log: bool = True) -> None:
|
||||
"""
|
||||
Dump object data into a file
|
||||
:param filename: file to create
|
||||
:param data: Object data to save
|
||||
:return:
|
||||
"""
|
||||
|
||||
if log:
|
||||
logger.info(f'dumping joblib to "{filename}"')
|
||||
with open(filename, 'wb') as fp:
|
||||
joblib.dump(data, fp)
|
||||
logger.debug(f'done joblib dump to "{filename}"')
|
||||
|
||||
|
||||
def json_load(datafile: IO) -> Any:
|
||||
"""
|
||||
load data with rapidjson
|
||||
|
||||
@ -19,13 +19,15 @@ from freqtrade.data import history
|
||||
from freqtrade.data.btanalysis import find_existing_backtest_stats, trade_list_to_dataframe
|
||||
from freqtrade.data.converter import trim_dataframe, trim_dataframes
|
||||
from freqtrade.data.dataprovider import DataProvider
|
||||
from freqtrade.enums import BacktestState, CandleType, ExitCheckTuple, ExitType, TradingMode
|
||||
from freqtrade.enums import (BacktestState, CandleType, ExitCheckTuple, ExitType, RunMode,
|
||||
TradingMode)
|
||||
from freqtrade.exceptions import DependencyException, OperationalException
|
||||
from freqtrade.exchange import timeframe_to_minutes, timeframe_to_seconds
|
||||
from freqtrade.misc import get_strategy_run_id
|
||||
from freqtrade.mixins import LoggingMixin
|
||||
from freqtrade.optimize.bt_progress import BTProgress
|
||||
from freqtrade.optimize.optimize_reports import (generate_backtest_stats, show_backtest_results,
|
||||
store_backtest_signal_candles,
|
||||
store_backtest_stats)
|
||||
from freqtrade.persistence import LocalTrade, Order, PairLocks, Trade
|
||||
from freqtrade.plugins.pairlistmanager import PairListManager
|
||||
@ -73,6 +75,8 @@ class Backtesting:
|
||||
self.run_ids: Dict[str, str] = {}
|
||||
self.strategylist: List[IStrategy] = []
|
||||
self.all_results: Dict[str, Dict] = {}
|
||||
self.processed_dfs: Dict[str, Dict] = {}
|
||||
|
||||
self._exchange_name = self.config['exchange']['name']
|
||||
self.exchange = ExchangeResolver.load_exchange(self._exchange_name, self.config)
|
||||
self.dataprovider = DataProvider(self.config, self.exchange)
|
||||
@ -1070,8 +1074,31 @@ class Backtesting:
|
||||
})
|
||||
self.all_results[self.strategy.get_strategy_name()] = results
|
||||
|
||||
if (self.config.get('export', 'none') == 'signals' and
|
||||
self.dataprovider.runmode == RunMode.BACKTEST):
|
||||
self._generate_trade_signal_candles(preprocessed_tmp, results)
|
||||
|
||||
return min_date, max_date
|
||||
|
||||
def _generate_trade_signal_candles(self, preprocessed_df, bt_results):
|
||||
signal_candles_only = {}
|
||||
for pair in preprocessed_df.keys():
|
||||
signal_candles_only_df = DataFrame()
|
||||
|
||||
pairdf = preprocessed_df[pair]
|
||||
resdf = bt_results['results']
|
||||
pairresults = resdf.loc[(resdf["pair"] == pair)]
|
||||
|
||||
if pairdf.shape[0] > 0:
|
||||
for t, v in pairresults.open_date.items():
|
||||
allinds = pairdf.loc[(pairdf['date'] < v)]
|
||||
signal_inds = allinds.iloc[[-1]]
|
||||
signal_candles_only_df = signal_candles_only_df.append(signal_inds)
|
||||
|
||||
signal_candles_only[pair] = signal_candles_only_df
|
||||
|
||||
self.processed_dfs[self.strategy.get_strategy_name()] = signal_candles_only
|
||||
|
||||
def _get_min_cached_backtest_date(self):
|
||||
min_backtest_date = None
|
||||
backtest_cache_age = self.config.get('backtest_cache', constants.BACKTEST_CACHE_DEFAULT)
|
||||
@ -1130,9 +1157,13 @@ class Backtesting:
|
||||
else:
|
||||
self.results = results
|
||||
|
||||
if self.config.get('export', 'none') == 'trades':
|
||||
if self.config.get('export', 'none') in ('trades', 'signals'):
|
||||
store_backtest_stats(self.config['exportfilename'], self.results)
|
||||
|
||||
if (self.config.get('export', 'none') == 'signals' and
|
||||
self.dataprovider.runmode == RunMode.BACKTEST):
|
||||
store_backtest_signal_candles(self.config['exportfilename'], self.processed_dfs)
|
||||
|
||||
# Results may be mixed up now. Sort them so they follow --strategy-list order.
|
||||
if 'strategy_list' in self.config and len(self.results) > 0:
|
||||
self.results['strategy_comparison'] = sorted(
|
||||
|
||||
@ -11,8 +11,8 @@ from tabulate import tabulate
|
||||
from freqtrade.constants import DATETIME_PRINT_FORMAT, LAST_BT_RESULT_FN, UNLIMITED_STAKE_AMOUNT
|
||||
from freqtrade.data.btanalysis import (calculate_csum, calculate_market_change,
|
||||
calculate_max_drawdown)
|
||||
from freqtrade.misc import (decimals_per_coin, file_dump_json, get_backtest_metadata_filename,
|
||||
round_coin_value)
|
||||
from freqtrade.misc import (decimals_per_coin, file_dump_joblib, file_dump_json,
|
||||
get_backtest_metadata_filename, round_coin_value)
|
||||
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
@ -45,6 +45,29 @@ def store_backtest_stats(recordfilename: Path, stats: Dict[str, DataFrame]) -> N
|
||||
file_dump_json(latest_filename, {'latest_backtest': str(filename.name)})
|
||||
|
||||
|
||||
def store_backtest_signal_candles(recordfilename: Path, candles: Dict[str, Dict]) -> Path:
|
||||
"""
|
||||
Stores backtest trade signal candles
|
||||
:param recordfilename: Path object, which can either be a filename or a directory.
|
||||
Filenames will be appended with a timestamp right before the suffix
|
||||
while for directories, <directory>/backtest-result-<datetime>_signals.pkl will be used
|
||||
as filename
|
||||
:param stats: Dict containing the backtesting signal candles
|
||||
"""
|
||||
if recordfilename.is_dir():
|
||||
filename = (recordfilename /
|
||||
f'backtest-result-{datetime.now().strftime("%Y-%m-%d_%H-%M-%S")}_signals.pkl')
|
||||
else:
|
||||
filename = Path.joinpath(
|
||||
recordfilename.parent,
|
||||
f'{recordfilename.stem}-{datetime.now().strftime("%Y-%m-%d_%H-%M-%S")}_signals.pkl'
|
||||
)
|
||||
|
||||
file_dump_joblib(filename, candles)
|
||||
|
||||
return filename
|
||||
|
||||
|
||||
def _get_line_floatfmt(stake_currency: str) -> List[str]:
|
||||
"""
|
||||
Generate floatformat (goes in line with _generate_result_line())
|
||||
|
||||
@ -29,6 +29,7 @@ nav:
|
||||
- Data Analysis:
|
||||
- Jupyter Notebooks: data-analysis.md
|
||||
- Strategy analysis: strategy_analysis_example.md
|
||||
- Backtest analysis: advanced-backtesting.md
|
||||
- Advanced Topics:
|
||||
- Advanced Post-installation Tasks: advanced-setup.md
|
||||
- Edge Positioning: edge.md
|
||||
|
||||
@ -384,14 +384,16 @@ def test_backtesting_start(default_conf, mocker, testdatadir, caplog) -> None:
|
||||
mocker.patch('freqtrade.optimize.backtesting.generate_backtest_stats')
|
||||
mocker.patch('freqtrade.optimize.backtesting.show_backtest_results')
|
||||
sbs = mocker.patch('freqtrade.optimize.backtesting.store_backtest_stats')
|
||||
sbc = mocker.patch('freqtrade.optimize.backtesting.store_backtest_signal_candles')
|
||||
mocker.patch('freqtrade.plugins.pairlistmanager.PairListManager.whitelist',
|
||||
PropertyMock(return_value=['UNITTEST/BTC']))
|
||||
|
||||
default_conf['timeframe'] = '1m'
|
||||
default_conf['datadir'] = testdatadir
|
||||
default_conf['export'] = 'trades'
|
||||
default_conf['export'] = 'signals'
|
||||
default_conf['exportfilename'] = 'export.txt'
|
||||
default_conf['timerange'] = '-1510694220'
|
||||
default_conf['runmode'] = RunMode.BACKTEST
|
||||
|
||||
backtesting = Backtesting(default_conf)
|
||||
backtesting._set_strategy(backtesting.strategylist[0])
|
||||
@ -407,6 +409,7 @@ def test_backtesting_start(default_conf, mocker, testdatadir, caplog) -> None:
|
||||
assert backtesting.strategy.dp._pairlists is not None
|
||||
assert backtesting.strategy.bot_loop_start.call_count == 1
|
||||
assert sbs.call_count == 1
|
||||
assert sbc.call_count == 1
|
||||
|
||||
|
||||
def test_backtesting_start_no_data(default_conf, mocker, caplog, testdatadir) -> None:
|
||||
|
||||
@ -2,6 +2,7 @@ import re
|
||||
from datetime import timedelta
|
||||
from pathlib import Path
|
||||
|
||||
import joblib
|
||||
import pandas as pd
|
||||
import pytest
|
||||
from arrow import Arrow
|
||||
@ -19,6 +20,7 @@ from freqtrade.optimize.optimize_reports import (_get_resample_from_period, gene
|
||||
generate_periodic_breakdown_stats,
|
||||
generate_strategy_comparison,
|
||||
generate_trading_stats, show_sorted_pairlist,
|
||||
store_backtest_signal_candles,
|
||||
store_backtest_stats, text_table_bt_results,
|
||||
text_table_exit_reason, text_table_strategy)
|
||||
from freqtrade.resolvers.strategy_resolver import StrategyResolver
|
||||
@ -201,6 +203,62 @@ def test_store_backtest_stats(testdatadir, mocker):
|
||||
assert str(dump_mock.call_args_list[0][0][0]).startswith(str(testdatadir / 'testresult'))
|
||||
|
||||
|
||||
def test_store_backtest_candles(testdatadir, mocker):
|
||||
|
||||
dump_mock = mocker.patch('freqtrade.optimize.optimize_reports.file_dump_joblib')
|
||||
|
||||
candle_dict = {'DefStrat': {'UNITTEST/BTC': pd.DataFrame()}}
|
||||
|
||||
# mock directory exporting
|
||||
store_backtest_signal_candles(testdatadir, candle_dict)
|
||||
|
||||
assert dump_mock.call_count == 1
|
||||
assert isinstance(dump_mock.call_args_list[0][0][0], Path)
|
||||
assert str(dump_mock.call_args_list[0][0][0]).endswith(str('_signals.pkl'))
|
||||
|
||||
dump_mock.reset_mock()
|
||||
# mock file exporting
|
||||
filename = Path(testdatadir / 'testresult')
|
||||
store_backtest_signal_candles(filename, candle_dict)
|
||||
assert dump_mock.call_count == 1
|
||||
assert isinstance(dump_mock.call_args_list[0][0][0], Path)
|
||||
# result will be testdatadir / testresult-<timestamp>_signals.pkl
|
||||
assert str(dump_mock.call_args_list[0][0][0]).endswith(str('_signals.pkl'))
|
||||
dump_mock.reset_mock()
|
||||
|
||||
|
||||
def test_write_read_backtest_candles(tmpdir):
|
||||
|
||||
candle_dict = {'DefStrat': {'UNITTEST/BTC': pd.DataFrame()}}
|
||||
|
||||
# test directory exporting
|
||||
stored_file = store_backtest_signal_candles(Path(tmpdir), candle_dict)
|
||||
scp = open(stored_file, "rb")
|
||||
pickled_signal_candles = joblib.load(scp)
|
||||
scp.close()
|
||||
|
||||
assert pickled_signal_candles.keys() == candle_dict.keys()
|
||||
assert pickled_signal_candles['DefStrat'].keys() == pickled_signal_candles['DefStrat'].keys()
|
||||
assert pickled_signal_candles['DefStrat']['UNITTEST/BTC'] \
|
||||
.equals(pickled_signal_candles['DefStrat']['UNITTEST/BTC'])
|
||||
|
||||
_clean_test_file(stored_file)
|
||||
|
||||
# test file exporting
|
||||
filename = Path(tmpdir / 'testresult')
|
||||
stored_file = store_backtest_signal_candles(filename, candle_dict)
|
||||
scp = open(stored_file, "rb")
|
||||
pickled_signal_candles = joblib.load(scp)
|
||||
scp.close()
|
||||
|
||||
assert pickled_signal_candles.keys() == candle_dict.keys()
|
||||
assert pickled_signal_candles['DefStrat'].keys() == pickled_signal_candles['DefStrat'].keys()
|
||||
assert pickled_signal_candles['DefStrat']['UNITTEST/BTC'] \
|
||||
.equals(pickled_signal_candles['DefStrat']['UNITTEST/BTC'])
|
||||
|
||||
_clean_test_file(stored_file)
|
||||
|
||||
|
||||
def test_generate_pair_metrics():
|
||||
|
||||
results = pd.DataFrame(
|
||||
|
||||
Loading…
Reference in New Issue
Block a user