diff --git a/docs/advanced-backtesting.md b/docs/advanced-backtesting.md new file mode 100644 index 000000000..8a8c1af77 --- /dev/null +++ b/docs/advanced-backtesting.md @@ -0,0 +1,73 @@ +# Advanced Backtesting Analysis + +## Analyze the buy/entry and sell/exit tags + +It can be helpful to understand how a strategy behaves according to the buy/entry tags used to +mark up different buy conditions. You might want to see more complex statistics about each buy and +sell condition above those provided by the default backtesting output. You may also want to +determine indicator values on the signal candle that resulted in a trade opening. + +!!! Note + The following buy reason analysis is only available for backtesting, *not hyperopt*. + +We need to run backtesting with the `--export` option set to `signals` to enable the exporting of +signals **and** trades: + +``` bash +freqtrade backtesting -c --timeframe --strategy --timerange= --export=signals +``` + +To analyze the buy tags, we need to use the `buy_reasons.py` script from +[froggleston's repo](https://github.com/froggleston/freqtrade-buyreasons). Follow the instructions +in their README to copy the script into your `freqtrade/scripts/` folder. + +This will tell freqtrade to output a pickled dictionary of strategy, pairs and corresponding +DataFrame of the candles that resulted in buy signals. Depending on how many buys your strategy +makes, this file may get quite large, so periodically check your `user_data/backtest_results` +folder to delete old exports. + +Before running your next backtest, make sure you either delete your old backtest results or run +backtesting with the `--cache none` option to make sure no cached results are used. + +If all goes well, you should now see a `backtest-result-{timestamp}_signals.pkl` file in the +`user_data/backtest_results` folder. + +Now run the `buy_reasons.py` script, supplying a few options: + +``` bash +python3 scripts/buy_reasons.py -c -s -t -g0,1,2,3,4 +``` + +The `-g` option is used to specify the various tabular outputs, ranging from the simplest (0) +to the most detailed per pair, per buy and per sell tag (4). More options are available by +running with the `-h` option. + +### Tuning the buy tags and sell tags to display + +To show only certain buy and sell tags in the displayed output, use the following two options: + +``` +--buy_reason_list : Comma separated list of buy signals to analyse. Default: "all" +--sell_reason_list : Comma separated list of sell signals to analyse. Default: "stop_loss,trailing_stop_loss" +``` + +For example: + +```bash +python3 scripts/buy_reasons.py -c -s -t -g0,1,2,3,4 --buy_reason_list "buy_tag_a,buy_tag_b" --sell_reason_list "roi,custom_sell_tag_a,stop_loss" +``` + +### Outputting signal candle indicators + +The real power of the buy_reasons.py script comes from the ability to print out the indicator +values present on signal candles to allow fine-grained investigation and tuning of buy signal +indicators. To print out a column for a given set of indicators, use the `--indicator-list` +option: + +```bash +python3 scripts/buy_reasons.py -c -s -t -g0,1,2,3,4 --buy_reason_list "buy_tag_a,buy_tag_b" --sell_reason_list "roi,custom_sell_tag_a,stop_loss" --indicator_list "rsi,rsi_1h,bb_lowerband,ema_9,macd,macdsignal" +``` + +The indicators have to be present in your strategy's main DataFrame (either for your main +timeframe or for informative timeframes) otherwise they will simply be ignored in the script +output. diff --git a/docs/backtesting.md b/docs/backtesting.md index 5d836d01b..f732068f1 100644 --- a/docs/backtesting.md +++ b/docs/backtesting.md @@ -20,7 +20,8 @@ usage: freqtrade backtesting [-h] [-v] [--logfile FILE] [-V] [-c PATH] [--dry-run-wallet DRY_RUN_WALLET] [--timeframe-detail TIMEFRAME_DETAIL] [--strategy-list STRATEGY_LIST [STRATEGY_LIST ...]] - [--export {none,trades}] [--export-filename PATH] + [--export {none,trades,signals}] + [--export-filename PATH] [--breakdown {day,week,month} [{day,week,month} ...]] [--cache {none,day,week,month}] @@ -63,18 +64,17 @@ optional arguments: `30m`, `1h`, `1d`). --strategy-list STRATEGY_LIST [STRATEGY_LIST ...] Provide a space-separated list of strategies to - backtest. Please note that timeframe needs to be - set either in config or via command line. When using - this together with `--export trades`, the strategy- - name is injected into the filename (so `backtest- - data.json` becomes `backtest-data-SampleStrategy.json` - --export {none,trades} + backtest. Please note that timeframe needs to be set + either in config or via command line. When using this + together with `--export trades`, the strategy-name is + injected into the filename (so `backtest-data.json` + becomes `backtest-data-SampleStrategy.json` + --export {none,trades,signals} Export backtest results (default: trades). - --export-filename PATH - Save backtest results to the file with this filename. - Requires `--export` to be set as well. Example: - `--export-filename=user_data/backtest_results/backtest - _today.json` + --export-filename PATH, --backtest-filename PATH + Use this filename for backtest results.Requires + `--export` to be set as well. Example: `--export-filen + ame=user_data/backtest_results/backtest_today.json` --breakdown {day,week,month} [{day,week,month} ...] Show backtesting breakdown per [day, week, month]. --cache {none,day,week,month} diff --git a/docs/configuration.md b/docs/configuration.md index 369c4e2dd..5770450a6 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -11,7 +11,7 @@ Per default, the bot loads the configuration from the `config.json` file, locate You can specify a different configuration file used by the bot with the `-c/--config` command-line option. -If you used the [Quick start](installation.md/#quick-start) method for installing +If you used the [Quick start](installation.md/#quick-start) method for installing the bot, the installation script should have already created the default configuration file (`config.json`) for you. If the default configuration file is not created we recommend to use `freqtrade new-config --config config.json` to generate a basic configuration file. @@ -64,7 +64,7 @@ This is similar to using multiple `--config` parameters, but simpler in usage as "config-private.json" ] ``` - + ``` bash freqtrade trade --config user_data/config.json <...> ``` @@ -100,7 +100,7 @@ This is similar to using multiple `--config` parameters, but simpler in usage as "stake_amount": "unlimited", } ``` - + Resulting combined configuration: ``` json title="Result" diff --git a/docs/data-analysis.md b/docs/data-analysis.md index 9a79ee5ed..926ed3eae 100644 --- a/docs/data-analysis.md +++ b/docs/data-analysis.md @@ -122,5 +122,6 @@ Best avoid relative paths, since this starts at the storage location of the jupy * [Strategy debugging](strategy_analysis_example.md) - also available as Jupyter notebook (`user_data/notebooks/strategy_analysis_example.ipynb`) * [Plotting](plotting.md) +* [Tag Analysis](advanced-backtesting.md) Feel free to submit an issue or Pull Request enhancing this document if you would like to share ideas on how to best analyze the data. diff --git a/docs/strategy_analysis_example.md b/docs/strategy_analysis_example.md index 2fa84a6df..ae0c6a6a3 100644 --- a/docs/strategy_analysis_example.md +++ b/docs/strategy_analysis_example.md @@ -93,7 +93,7 @@ from freqtrade.data.btanalysis import load_backtest_data, load_backtest_stats # if backtest_dir points to a directory, it'll automatically load the last backtest file. backtest_dir = config["user_data_dir"] / "backtest_results" -# backtest_dir can also point to a specific file +# backtest_dir can also point to a specific file # backtest_dir = config["user_data_dir"] / "backtest_results/backtest-result-2020-07-01_20-04-22.json" ``` diff --git a/freqtrade/constants.py b/freqtrade/constants.py index c6a2ab5d3..1a21ec77f 100644 --- a/freqtrade/constants.py +++ b/freqtrade/constants.py @@ -14,7 +14,7 @@ PROCESS_THROTTLE_SECS = 5 # sec HYPEROPT_EPOCH = 100 # epochs RETRY_TIMEOUT = 30 # sec TIMEOUT_UNITS = ['minutes', 'seconds'] -EXPORT_OPTIONS = ['none', 'trades'] +EXPORT_OPTIONS = ['none', 'trades', 'signals'] DEFAULT_DB_PROD_URL = 'sqlite:///tradesv3.sqlite' DEFAULT_DB_DRYRUN_URL = 'sqlite:///tradesv3.dryrun.sqlite' UNLIMITED_STAKE_AMOUNT = 'unlimited' diff --git a/freqtrade/misc.py b/freqtrade/misc.py index d5572ea0b..be12d8224 100644 --- a/freqtrade/misc.py +++ b/freqtrade/misc.py @@ -12,6 +12,7 @@ from typing import Any, Iterator, List, Union from typing.io import IO from urllib.parse import urlparse +import joblib import rapidjson from freqtrade.constants import DECIMAL_PER_COIN_FALLBACK, DECIMALS_PER_COIN @@ -86,6 +87,21 @@ def file_dump_json(filename: Path, data: Any, is_zip: bool = False, log: bool = logger.debug(f'done json to "{filename}"') +def file_dump_joblib(filename: Path, data: Any, log: bool = True) -> None: + """ + Dump object data into a file + :param filename: file to create + :param data: Object data to save + :return: + """ + + if log: + logger.info(f'dumping joblib to "{filename}"') + with open(filename, 'wb') as fp: + joblib.dump(data, fp) + logger.debug(f'done joblib dump to "{filename}"') + + def json_load(datafile: IO) -> Any: """ load data with rapidjson diff --git a/freqtrade/optimize/backtesting.py b/freqtrade/optimize/backtesting.py index cbb220e45..f5571c4e2 100644 --- a/freqtrade/optimize/backtesting.py +++ b/freqtrade/optimize/backtesting.py @@ -19,13 +19,15 @@ from freqtrade.data import history from freqtrade.data.btanalysis import find_existing_backtest_stats, trade_list_to_dataframe from freqtrade.data.converter import trim_dataframe, trim_dataframes from freqtrade.data.dataprovider import DataProvider -from freqtrade.enums import BacktestState, CandleType, ExitCheckTuple, ExitType, TradingMode +from freqtrade.enums import (BacktestState, CandleType, ExitCheckTuple, ExitType, RunMode, + TradingMode) from freqtrade.exceptions import DependencyException, OperationalException from freqtrade.exchange import timeframe_to_minutes, timeframe_to_seconds from freqtrade.misc import get_strategy_run_id from freqtrade.mixins import LoggingMixin from freqtrade.optimize.bt_progress import BTProgress from freqtrade.optimize.optimize_reports import (generate_backtest_stats, show_backtest_results, + store_backtest_signal_candles, store_backtest_stats) from freqtrade.persistence import LocalTrade, Order, PairLocks, Trade from freqtrade.plugins.pairlistmanager import PairListManager @@ -73,6 +75,8 @@ class Backtesting: self.run_ids: Dict[str, str] = {} self.strategylist: List[IStrategy] = [] self.all_results: Dict[str, Dict] = {} + self.processed_dfs: Dict[str, Dict] = {} + self._exchange_name = self.config['exchange']['name'] self.exchange = ExchangeResolver.load_exchange(self._exchange_name, self.config) self.dataprovider = DataProvider(self.config, self.exchange) @@ -1070,8 +1074,31 @@ class Backtesting: }) self.all_results[self.strategy.get_strategy_name()] = results + if (self.config.get('export', 'none') == 'signals' and + self.dataprovider.runmode == RunMode.BACKTEST): + self._generate_trade_signal_candles(preprocessed_tmp, results) + return min_date, max_date + def _generate_trade_signal_candles(self, preprocessed_df, bt_results): + signal_candles_only = {} + for pair in preprocessed_df.keys(): + signal_candles_only_df = DataFrame() + + pairdf = preprocessed_df[pair] + resdf = bt_results['results'] + pairresults = resdf.loc[(resdf["pair"] == pair)] + + if pairdf.shape[0] > 0: + for t, v in pairresults.open_date.items(): + allinds = pairdf.loc[(pairdf['date'] < v)] + signal_inds = allinds.iloc[[-1]] + signal_candles_only_df = signal_candles_only_df.append(signal_inds) + + signal_candles_only[pair] = signal_candles_only_df + + self.processed_dfs[self.strategy.get_strategy_name()] = signal_candles_only + def _get_min_cached_backtest_date(self): min_backtest_date = None backtest_cache_age = self.config.get('backtest_cache', constants.BACKTEST_CACHE_DEFAULT) @@ -1130,9 +1157,13 @@ class Backtesting: else: self.results = results - if self.config.get('export', 'none') == 'trades': + if self.config.get('export', 'none') in ('trades', 'signals'): store_backtest_stats(self.config['exportfilename'], self.results) + if (self.config.get('export', 'none') == 'signals' and + self.dataprovider.runmode == RunMode.BACKTEST): + store_backtest_signal_candles(self.config['exportfilename'], self.processed_dfs) + # Results may be mixed up now. Sort them so they follow --strategy-list order. if 'strategy_list' in self.config and len(self.results) > 0: self.results['strategy_comparison'] = sorted( diff --git a/freqtrade/optimize/optimize_reports.py b/freqtrade/optimize/optimize_reports.py index c08fa07a1..6288ee16a 100644 --- a/freqtrade/optimize/optimize_reports.py +++ b/freqtrade/optimize/optimize_reports.py @@ -11,8 +11,8 @@ from tabulate import tabulate from freqtrade.constants import DATETIME_PRINT_FORMAT, LAST_BT_RESULT_FN, UNLIMITED_STAKE_AMOUNT from freqtrade.data.btanalysis import (calculate_csum, calculate_market_change, calculate_max_drawdown) -from freqtrade.misc import (decimals_per_coin, file_dump_json, get_backtest_metadata_filename, - round_coin_value) +from freqtrade.misc import (decimals_per_coin, file_dump_joblib, file_dump_json, + get_backtest_metadata_filename, round_coin_value) logger = logging.getLogger(__name__) @@ -45,6 +45,29 @@ def store_backtest_stats(recordfilename: Path, stats: Dict[str, DataFrame]) -> N file_dump_json(latest_filename, {'latest_backtest': str(filename.name)}) +def store_backtest_signal_candles(recordfilename: Path, candles: Dict[str, Dict]) -> Path: + """ + Stores backtest trade signal candles + :param recordfilename: Path object, which can either be a filename or a directory. + Filenames will be appended with a timestamp right before the suffix + while for directories, /backtest-result-_signals.pkl will be used + as filename + :param stats: Dict containing the backtesting signal candles + """ + if recordfilename.is_dir(): + filename = (recordfilename / + f'backtest-result-{datetime.now().strftime("%Y-%m-%d_%H-%M-%S")}_signals.pkl') + else: + filename = Path.joinpath( + recordfilename.parent, + f'{recordfilename.stem}-{datetime.now().strftime("%Y-%m-%d_%H-%M-%S")}_signals.pkl' + ) + + file_dump_joblib(filename, candles) + + return filename + + def _get_line_floatfmt(stake_currency: str) -> List[str]: """ Generate floatformat (goes in line with _generate_result_line()) diff --git a/mkdocs.yml b/mkdocs.yml index 8b36ba699..a43322f78 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -29,6 +29,7 @@ nav: - Data Analysis: - Jupyter Notebooks: data-analysis.md - Strategy analysis: strategy_analysis_example.md + - Backtest analysis: advanced-backtesting.md - Advanced Topics: - Advanced Post-installation Tasks: advanced-setup.md - Edge Positioning: edge.md diff --git a/tests/optimize/test_backtesting.py b/tests/optimize/test_backtesting.py index 08957acf9..797d3bafa 100644 --- a/tests/optimize/test_backtesting.py +++ b/tests/optimize/test_backtesting.py @@ -384,14 +384,16 @@ def test_backtesting_start(default_conf, mocker, testdatadir, caplog) -> None: mocker.patch('freqtrade.optimize.backtesting.generate_backtest_stats') mocker.patch('freqtrade.optimize.backtesting.show_backtest_results') sbs = mocker.patch('freqtrade.optimize.backtesting.store_backtest_stats') + sbc = mocker.patch('freqtrade.optimize.backtesting.store_backtest_signal_candles') mocker.patch('freqtrade.plugins.pairlistmanager.PairListManager.whitelist', PropertyMock(return_value=['UNITTEST/BTC'])) default_conf['timeframe'] = '1m' default_conf['datadir'] = testdatadir - default_conf['export'] = 'trades' + default_conf['export'] = 'signals' default_conf['exportfilename'] = 'export.txt' default_conf['timerange'] = '-1510694220' + default_conf['runmode'] = RunMode.BACKTEST backtesting = Backtesting(default_conf) backtesting._set_strategy(backtesting.strategylist[0]) @@ -407,6 +409,7 @@ def test_backtesting_start(default_conf, mocker, testdatadir, caplog) -> None: assert backtesting.strategy.dp._pairlists is not None assert backtesting.strategy.bot_loop_start.call_count == 1 assert sbs.call_count == 1 + assert sbc.call_count == 1 def test_backtesting_start_no_data(default_conf, mocker, caplog, testdatadir) -> None: diff --git a/tests/optimize/test_optimize_reports.py b/tests/optimize/test_optimize_reports.py index 05c0bf575..ff8d420b3 100644 --- a/tests/optimize/test_optimize_reports.py +++ b/tests/optimize/test_optimize_reports.py @@ -2,6 +2,7 @@ import re from datetime import timedelta from pathlib import Path +import joblib import pandas as pd import pytest from arrow import Arrow @@ -19,6 +20,7 @@ from freqtrade.optimize.optimize_reports import (_get_resample_from_period, gene generate_periodic_breakdown_stats, generate_strategy_comparison, generate_trading_stats, show_sorted_pairlist, + store_backtest_signal_candles, store_backtest_stats, text_table_bt_results, text_table_exit_reason, text_table_strategy) from freqtrade.resolvers.strategy_resolver import StrategyResolver @@ -201,6 +203,62 @@ def test_store_backtest_stats(testdatadir, mocker): assert str(dump_mock.call_args_list[0][0][0]).startswith(str(testdatadir / 'testresult')) +def test_store_backtest_candles(testdatadir, mocker): + + dump_mock = mocker.patch('freqtrade.optimize.optimize_reports.file_dump_joblib') + + candle_dict = {'DefStrat': {'UNITTEST/BTC': pd.DataFrame()}} + + # mock directory exporting + store_backtest_signal_candles(testdatadir, candle_dict) + + assert dump_mock.call_count == 1 + assert isinstance(dump_mock.call_args_list[0][0][0], Path) + assert str(dump_mock.call_args_list[0][0][0]).endswith(str('_signals.pkl')) + + dump_mock.reset_mock() + # mock file exporting + filename = Path(testdatadir / 'testresult') + store_backtest_signal_candles(filename, candle_dict) + assert dump_mock.call_count == 1 + assert isinstance(dump_mock.call_args_list[0][0][0], Path) + # result will be testdatadir / testresult-_signals.pkl + assert str(dump_mock.call_args_list[0][0][0]).endswith(str('_signals.pkl')) + dump_mock.reset_mock() + + +def test_write_read_backtest_candles(tmpdir): + + candle_dict = {'DefStrat': {'UNITTEST/BTC': pd.DataFrame()}} + + # test directory exporting + stored_file = store_backtest_signal_candles(Path(tmpdir), candle_dict) + scp = open(stored_file, "rb") + pickled_signal_candles = joblib.load(scp) + scp.close() + + assert pickled_signal_candles.keys() == candle_dict.keys() + assert pickled_signal_candles['DefStrat'].keys() == pickled_signal_candles['DefStrat'].keys() + assert pickled_signal_candles['DefStrat']['UNITTEST/BTC'] \ + .equals(pickled_signal_candles['DefStrat']['UNITTEST/BTC']) + + _clean_test_file(stored_file) + + # test file exporting + filename = Path(tmpdir / 'testresult') + stored_file = store_backtest_signal_candles(filename, candle_dict) + scp = open(stored_file, "rb") + pickled_signal_candles = joblib.load(scp) + scp.close() + + assert pickled_signal_candles.keys() == candle_dict.keys() + assert pickled_signal_candles['DefStrat'].keys() == pickled_signal_candles['DefStrat'].keys() + assert pickled_signal_candles['DefStrat']['UNITTEST/BTC'] \ + .equals(pickled_signal_candles['DefStrat']['UNITTEST/BTC']) + + _clean_test_file(stored_file) + + def test_generate_pair_metrics(): results = pd.DataFrame(