Merge pull request #6543 from froggleston/v3_fixes

Add support for storing buy candle indicator rows in backtesting results

docs/advanced-backtesting.md

@@ -0,0 +1,73 @@
+# Advanced Backtesting Analysis
+
+## Analyze the buy/entry and sell/exit tags
+
+It can be helpful to understand how a strategy behaves according to the buy/entry tags used to
+mark up different buy conditions. You might want to see more complex statistics about each buy and
+sell condition above those provided by the default backtesting output. You may also want to
+determine indicator values on the signal candle that resulted in a trade opening.
+
+!!! Note
+    The following buy reason analysis is only available for backtesting, *not hyperopt*.
+
+We need to run backtesting with the `--export` option set to `signals` to enable the exporting of
+signals **and** trades:
+
+``` bash
+freqtrade backtesting -c <config.json> --timeframe <tf> --strategy <strategy_name> --timerange=<timerange> --export=signals
+```
+
+To analyze the buy tags, we need to use the `buy_reasons.py` script from
+[froggleston's repo](https://github.com/froggleston/freqtrade-buyreasons). Follow the instructions
+in their README to copy the script into your `freqtrade/scripts/` folder.
+
+This will tell freqtrade to output a pickled dictionary of strategy, pairs and corresponding
+DataFrame of the candles that resulted in buy signals. Depending on how many buys your strategy
+makes, this file may get quite large, so periodically check your `user_data/backtest_results`
+folder to delete old exports.
+
+Before running your next backtest, make sure you either delete your old backtest results or run
+backtesting with the `--cache none` option to make sure no cached results are used.
+
+If all goes well, you should now see a `backtest-result-{timestamp}_signals.pkl` file in the
+`user_data/backtest_results` folder.
+
+Now run the `buy_reasons.py` script, supplying a few options:
+
+``` bash
+python3 scripts/buy_reasons.py -c <config.json> -s <strategy_name> -t <timerange> -g0,1,2,3,4
+```
+
+The `-g` option is used to specify the various tabular outputs, ranging from the simplest (0)
+to the most detailed per pair, per buy and per sell tag (4). More options are available by
+running with the `-h` option.
+
+### Tuning the buy tags and sell tags to display
+
+To show only certain buy and sell tags in the displayed output, use the following two options:
+
+```
+--buy_reason_list : Comma separated list of buy signals to analyse. Default: "all"
+--sell_reason_list : Comma separated list of sell signals to analyse. Default: "stop_loss,trailing_stop_loss"
+```
+
+For example:
+
+```bash
+python3 scripts/buy_reasons.py -c <config.json> -s <strategy_name> -t <timerange> -g0,1,2,3,4 --buy_reason_list "buy_tag_a,buy_tag_b" --sell_reason_list "roi,custom_sell_tag_a,stop_loss"
+```
+
+### Outputting signal candle indicators
+
+The real power of the buy_reasons.py script comes from the ability to print out the indicator
+values present on signal candles to allow fine-grained investigation and tuning of buy signal
+indicators. To print out a column for a given set of indicators, use the `--indicator-list`
+option:
+
+```bash
+python3 scripts/buy_reasons.py -c <config.json> -s <strategy_name> -t <timerange> -g0,1,2,3,4 --buy_reason_list "buy_tag_a,buy_tag_b" --sell_reason_list "roi,custom_sell_tag_a,stop_loss" --indicator_list "rsi,rsi_1h,bb_lowerband,ema_9,macd,macdsignal"
+```
+
+The indicators have to be present in your strategy's main DataFrame (either for your main
+timeframe or for informative timeframes) otherwise they will simply be ignored in the script
+output.

docs/backtesting.md

@@ -20,7 +20,8 @@ usage: freqtrade backtesting [-h] [-v] [--logfile FILE] [-V] [-c PATH]
                              [--dry-run-wallet DRY_RUN_WALLET]
                              [--timeframe-detail TIMEFRAME_DETAIL]
                              [--strategy-list STRATEGY_LIST [STRATEGY_LIST ...]]
-                             [--export {none,trades}] [--export-filename PATH]
+                             [--export {none,trades,signals}]
+                             [--export-filename PATH]
                              [--breakdown {day,week,month} [{day,week,month} ...]]
                              [--cache {none,day,week,month}]
 
@@ -63,18 +64,17 @@ optional arguments:
                         `30m`, `1h`, `1d`).
   --strategy-list STRATEGY_LIST [STRATEGY_LIST ...]
                         Provide a space-separated list of strategies to
-                        backtest. Please note that timeframe needs to be
+                        backtest. Please note that timeframe needs to be set
-                        set either in config or via command line. When using
+                        either in config or via command line. When using this
-                        this together with `--export trades`, the strategy-
+                        together with `--export trades`, the strategy-name is
-                        name is injected into the filename (so `backtest-
+                        injected into the filename (so `backtest-data.json`
-                        data.json` becomes `backtest-data-SampleStrategy.json`
+                        becomes `backtest-data-SampleStrategy.json`
-  --export {none,trades}
+  --export {none,trades,signals}
                         Export backtest results (default: trades).
-  --export-filename PATH
+  --export-filename PATH, --backtest-filename PATH
-                        Save backtest results to the file with this filename.
+                        Use this filename for backtest results.Requires
-                        Requires `--export` to be set as well. Example:
+                        `--export` to be set as well. Example: `--export-filen
-                        `--export-filename=user_data/backtest_results/backtest
+                        ame=user_data/backtest_results/backtest_today.json`
-                        _today.json`
   --breakdown {day,week,month} [{day,week,month} ...]
                         Show backtesting breakdown per [day, week, month].
   --cache {none,day,week,month}

docs/configuration.md

@@ -11,7 +11,7 @@ Per default, the bot loads the configuration from the `config.json` file, locate
 
 You can specify a different configuration file used by the bot with the `-c/--config` command-line option.
 
-If you used the [Quick start](installation.md/#quick-start) method for installing 
+If you used the [Quick start](installation.md/#quick-start) method for installing
 the bot, the installation script should have already created the default configuration file (`config.json`) for you.
 
 If the default configuration file is not created we recommend to use `freqtrade new-config --config config.json` to generate a basic configuration file.
@@ -64,7 +64,7 @@ This is similar to using multiple `--config` parameters, but simpler in usage as
         "config-private.json"
     ]
     ```
-    
+
     ``` bash
     freqtrade trade --config user_data/config.json <...>
     ```
@@ -100,7 +100,7 @@ This is similar to using multiple `--config` parameters, but simpler in usage as
         "stake_amount": "unlimited",
     }
     ```
-    
+
     Resulting combined configuration:
 
     ``` json title="Result"

docs/data-analysis.md

@@ -122,5 +122,6 @@ Best avoid relative paths, since this starts at the storage location of the jupy
 
 * [Strategy debugging](strategy_analysis_example.md) - also available as Jupyter notebook (`user_data/notebooks/strategy_analysis_example.ipynb`)
 * [Plotting](plotting.md)
+* [Tag Analysis](advanced-backtesting.md)
 
 Feel free to submit an issue or Pull Request enhancing this document if you would like to share ideas on how to best analyze the data.

docs/strategy_analysis_example.md

@@ -93,7 +93,7 @@ from freqtrade.data.btanalysis import load_backtest_data, load_backtest_stats
 
 # if backtest_dir points to a directory, it'll automatically load the last backtest file.
 backtest_dir = config["user_data_dir"] / "backtest_results"
-# backtest_dir can also point to a specific file 
+# backtest_dir can also point to a specific file
 # backtest_dir = config["user_data_dir"] / "backtest_results/backtest-result-2020-07-01_20-04-22.json"
 ```
 

freqtrade/constants.py

@@ -14,7 +14,7 @@ PROCESS_THROTTLE_SECS = 5  # sec
 HYPEROPT_EPOCH = 100  # epochs
 RETRY_TIMEOUT = 30  # sec
 TIMEOUT_UNITS = ['minutes', 'seconds']
-EXPORT_OPTIONS = ['none', 'trades']
+EXPORT_OPTIONS = ['none', 'trades', 'signals']
 DEFAULT_DB_PROD_URL = 'sqlite:///tradesv3.sqlite'
 DEFAULT_DB_DRYRUN_URL = 'sqlite:///tradesv3.dryrun.sqlite'
 UNLIMITED_STAKE_AMOUNT = 'unlimited'

freqtrade/misc.py

@@ -12,6 +12,7 @@ from typing import Any, Iterator, List, Union
 from typing.io import IO
 from urllib.parse import urlparse
 
+import joblib
 import rapidjson
 
 from freqtrade.constants import DECIMAL_PER_COIN_FALLBACK, DECIMALS_PER_COIN
@@ -86,6 +87,21 @@ def file_dump_json(filename: Path, data: Any, is_zip: bool = False, log: bool =
     logger.debug(f'done json to "{filename}"')
 
 
+def file_dump_joblib(filename: Path, data: Any, log: bool = True) -> None:
+    """
+    Dump object data into a file
+    :param filename: file to create
+    :param data: Object data to save
+    :return:
+    """
+
+    if log:
+        logger.info(f'dumping joblib to "{filename}"')
+    with open(filename, 'wb') as fp:
+        joblib.dump(data, fp)
+    logger.debug(f'done joblib dump to "{filename}"')
+
+
 def json_load(datafile: IO) -> Any:
     """
     load data with rapidjson

freqtrade/optimize/backtesting.py

@@ -19,13 +19,15 @@ from freqtrade.data import history
 from freqtrade.data.btanalysis import find_existing_backtest_stats, trade_list_to_dataframe
 from freqtrade.data.converter import trim_dataframe, trim_dataframes
 from freqtrade.data.dataprovider import DataProvider
-from freqtrade.enums import BacktestState, CandleType, ExitCheckTuple, ExitType, TradingMode
+from freqtrade.enums import (BacktestState, CandleType, ExitCheckTuple, ExitType, RunMode,
+                             TradingMode)
 from freqtrade.exceptions import DependencyException, OperationalException
 from freqtrade.exchange import timeframe_to_minutes, timeframe_to_seconds
 from freqtrade.misc import get_strategy_run_id
 from freqtrade.mixins import LoggingMixin
 from freqtrade.optimize.bt_progress import BTProgress
 from freqtrade.optimize.optimize_reports import (generate_backtest_stats, show_backtest_results,
+                                                 store_backtest_signal_candles,
                                                  store_backtest_stats)
 from freqtrade.persistence import LocalTrade, Order, PairLocks, Trade
 from freqtrade.plugins.pairlistmanager import PairListManager
@@ -73,6 +75,8 @@ class Backtesting:
         self.run_ids: Dict[str, str] = {}
         self.strategylist: List[IStrategy] = []
         self.all_results: Dict[str, Dict] = {}
+        self.processed_dfs: Dict[str, Dict] = {}
+
         self._exchange_name = self.config['exchange']['name']
         self.exchange = ExchangeResolver.load_exchange(self._exchange_name, self.config)
         self.dataprovider = DataProvider(self.config, self.exchange)
@@ -1070,8 +1074,31 @@ class Backtesting:
         })
         self.all_results[self.strategy.get_strategy_name()] = results
 
+        if (self.config.get('export', 'none') == 'signals' and
+                self.dataprovider.runmode == RunMode.BACKTEST):
+            self._generate_trade_signal_candles(preprocessed_tmp, results)
+
         return min_date, max_date
 
+    def _generate_trade_signal_candles(self, preprocessed_df, bt_results):
+        signal_candles_only = {}
+        for pair in preprocessed_df.keys():
+            signal_candles_only_df = DataFrame()
+
+            pairdf = preprocessed_df[pair]
+            resdf = bt_results['results']
+            pairresults = resdf.loc[(resdf["pair"] == pair)]
+
+            if pairdf.shape[0] > 0:
+                for t, v in pairresults.open_date.items():
+                    allinds = pairdf.loc[(pairdf['date'] < v)]
+                    signal_inds = allinds.iloc[[-1]]
+                    signal_candles_only_df = signal_candles_only_df.append(signal_inds)
+
+                signal_candles_only[pair] = signal_candles_only_df
+
+        self.processed_dfs[self.strategy.get_strategy_name()] = signal_candles_only
+
     def _get_min_cached_backtest_date(self):
         min_backtest_date = None
         backtest_cache_age = self.config.get('backtest_cache', constants.BACKTEST_CACHE_DEFAULT)
@@ -1130,9 +1157,13 @@ class Backtesting:
             else:
                 self.results = results
 
-            if self.config.get('export', 'none') == 'trades':
+            if self.config.get('export', 'none') in ('trades', 'signals'):
                 store_backtest_stats(self.config['exportfilename'], self.results)
 
+            if (self.config.get('export', 'none') == 'signals' and
+                    self.dataprovider.runmode == RunMode.BACKTEST):
+                store_backtest_signal_candles(self.config['exportfilename'], self.processed_dfs)
+
         # Results may be mixed up now. Sort them so they follow --strategy-list order.
         if 'strategy_list' in self.config and len(self.results) > 0:
             self.results['strategy_comparison'] = sorted(

freqtrade/optimize/optimize_reports.py

@@ -11,8 +11,8 @@ from tabulate import tabulate
 from freqtrade.constants import DATETIME_PRINT_FORMAT, LAST_BT_RESULT_FN, UNLIMITED_STAKE_AMOUNT
 from freqtrade.data.btanalysis import (calculate_csum, calculate_market_change,
                                        calculate_max_drawdown)
-from freqtrade.misc import (decimals_per_coin, file_dump_json, get_backtest_metadata_filename,
+from freqtrade.misc import (decimals_per_coin, file_dump_joblib, file_dump_json,
-                            round_coin_value)
+                            get_backtest_metadata_filename, round_coin_value)
 
 
 logger = logging.getLogger(__name__)
@@ -45,6 +45,29 @@ def store_backtest_stats(recordfilename: Path, stats: Dict[str, DataFrame]) -> N
     file_dump_json(latest_filename, {'latest_backtest': str(filename.name)})
 
 
+def store_backtest_signal_candles(recordfilename: Path, candles: Dict[str, Dict]) -> Path:
+    """
+    Stores backtest trade signal candles
+    :param recordfilename: Path object, which can either be a filename or a directory.
+        Filenames will be appended with a timestamp right before the suffix
+        while for directories, <directory>/backtest-result-<datetime>_signals.pkl will be used
+        as filename
+    :param stats: Dict containing the backtesting signal candles
+    """
+    if recordfilename.is_dir():
+        filename = (recordfilename /
+                    f'backtest-result-{datetime.now().strftime("%Y-%m-%d_%H-%M-%S")}_signals.pkl')
+    else:
+        filename = Path.joinpath(
+            recordfilename.parent,
+            f'{recordfilename.stem}-{datetime.now().strftime("%Y-%m-%d_%H-%M-%S")}_signals.pkl'
+        )
+
+    file_dump_joblib(filename, candles)
+
+    return filename
+
+
 def _get_line_floatfmt(stake_currency: str) -> List[str]:
     """
     Generate floatformat (goes in line with _generate_result_line())

mkdocs.yml

@@ -29,6 +29,7 @@ nav:
     - Data Analysis:
         - Jupyter Notebooks: data-analysis.md
         - Strategy analysis: strategy_analysis_example.md
+        - Backtest analysis: advanced-backtesting.md
     - Advanced Topics:
         - Advanced Post-installation Tasks: advanced-setup.md
         - Edge Positioning: edge.md

tests/optimize/test_backtesting.py

@@ -384,14 +384,16 @@ def test_backtesting_start(default_conf, mocker, testdatadir, caplog) -> None:
     mocker.patch('freqtrade.optimize.backtesting.generate_backtest_stats')
     mocker.patch('freqtrade.optimize.backtesting.show_backtest_results')
     sbs = mocker.patch('freqtrade.optimize.backtesting.store_backtest_stats')
+    sbc = mocker.patch('freqtrade.optimize.backtesting.store_backtest_signal_candles')
     mocker.patch('freqtrade.plugins.pairlistmanager.PairListManager.whitelist',
                  PropertyMock(return_value=['UNITTEST/BTC']))
 
     default_conf['timeframe'] = '1m'
     default_conf['datadir'] = testdatadir
-    default_conf['export'] = 'trades'
+    default_conf['export'] = 'signals'
     default_conf['exportfilename'] = 'export.txt'
     default_conf['timerange'] = '-1510694220'
+    default_conf['runmode'] = RunMode.BACKTEST
 
     backtesting = Backtesting(default_conf)
     backtesting._set_strategy(backtesting.strategylist[0])
@@ -407,6 +409,7 @@ def test_backtesting_start(default_conf, mocker, testdatadir, caplog) -> None:
     assert backtesting.strategy.dp._pairlists is not None
     assert backtesting.strategy.bot_loop_start.call_count == 1
     assert sbs.call_count == 1
+    assert sbc.call_count == 1
 
 
 def test_backtesting_start_no_data(default_conf, mocker, caplog, testdatadir) -> None:

tests/optimize/test_optimize_reports.py

@@ -2,6 +2,7 @@ import re
 from datetime import timedelta
 from pathlib import Path
 
+import joblib
 import pandas as pd
 import pytest
 from arrow import Arrow
@@ -19,6 +20,7 @@ from freqtrade.optimize.optimize_reports import (_get_resample_from_period, gene
                                                  generate_periodic_breakdown_stats,
                                                  generate_strategy_comparison,
                                                  generate_trading_stats, show_sorted_pairlist,
+                                                 store_backtest_signal_candles,
                                                  store_backtest_stats, text_table_bt_results,
                                                  text_table_exit_reason, text_table_strategy)
 from freqtrade.resolvers.strategy_resolver import StrategyResolver
@@ -201,6 +203,62 @@ def test_store_backtest_stats(testdatadir, mocker):
     assert str(dump_mock.call_args_list[0][0][0]).startswith(str(testdatadir / 'testresult'))
 
 
+def test_store_backtest_candles(testdatadir, mocker):
+
+    dump_mock = mocker.patch('freqtrade.optimize.optimize_reports.file_dump_joblib')
+
+    candle_dict = {'DefStrat': {'UNITTEST/BTC': pd.DataFrame()}}
+
+    # mock directory exporting
+    store_backtest_signal_candles(testdatadir, candle_dict)
+
+    assert dump_mock.call_count == 1
+    assert isinstance(dump_mock.call_args_list[0][0][0], Path)
+    assert str(dump_mock.call_args_list[0][0][0]).endswith(str('_signals.pkl'))
+
+    dump_mock.reset_mock()
+    # mock file exporting
+    filename = Path(testdatadir / 'testresult')
+    store_backtest_signal_candles(filename, candle_dict)
+    assert dump_mock.call_count == 1
+    assert isinstance(dump_mock.call_args_list[0][0][0], Path)
+    # result will be testdatadir / testresult-<timestamp>_signals.pkl
+    assert str(dump_mock.call_args_list[0][0][0]).endswith(str('_signals.pkl'))
+    dump_mock.reset_mock()
+
+
+def test_write_read_backtest_candles(tmpdir):
+
+    candle_dict = {'DefStrat': {'UNITTEST/BTC': pd.DataFrame()}}
+
+    # test directory exporting
+    stored_file = store_backtest_signal_candles(Path(tmpdir), candle_dict)
+    scp = open(stored_file, "rb")
+    pickled_signal_candles = joblib.load(scp)
+    scp.close()
+
+    assert pickled_signal_candles.keys() == candle_dict.keys()
+    assert pickled_signal_candles['DefStrat'].keys() == pickled_signal_candles['DefStrat'].keys()
+    assert pickled_signal_candles['DefStrat']['UNITTEST/BTC'] \
+        .equals(pickled_signal_candles['DefStrat']['UNITTEST/BTC'])
+
+    _clean_test_file(stored_file)
+
+    # test file exporting
+    filename = Path(tmpdir / 'testresult')
+    stored_file = store_backtest_signal_candles(filename, candle_dict)
+    scp = open(stored_file, "rb")
+    pickled_signal_candles = joblib.load(scp)
+    scp.close()
+
+    assert pickled_signal_candles.keys() == candle_dict.keys()
+    assert pickled_signal_candles['DefStrat'].keys() == pickled_signal_candles['DefStrat'].keys()
+    assert pickled_signal_candles['DefStrat']['UNITTEST/BTC'] \
+        .equals(pickled_signal_candles['DefStrat']['UNITTEST/BTC'])
+
+    _clean_test_file(stored_file)
+
+
 def test_generate_pair_metrics():
 
     results = pd.DataFrame(