Merge pull request #5413 from freqtrade/feat/backtest_detail

Add backtest-detail to allow intra-candle backtests

docs/backtesting.md

@@ -18,6 +18,7 @@ usage: freqtrade backtesting [-h] [-v] [--logfile FILE] [-V] [-c PATH]
                              [-p PAIRS [PAIRS ...]] [--eps] [--dmmp]
                              [--enable-protections]
                              [--dry-run-wallet DRY_RUN_WALLET]
+                             [--timeframe-detail TIMEFRAME_DETAIL]
                              [--strategy-list STRATEGY_LIST [STRATEGY_LIST ...]]
                              [--export {none,trades}] [--export-filename PATH]
 
@@ -55,6 +56,9 @@ optional arguments:
   --dry-run-wallet DRY_RUN_WALLET, --starting-balance DRY_RUN_WALLET
                         Starting balance, used for backtesting / hyperopt and
                         dry-runs.
+  --timeframe-detail TIMEFRAME_DETAIL
+                        Specify detail timeframe for backtesting (`1m`, `5m`,
+                        `30m`, `1h`, `1d`).
   --strategy-list STRATEGY_LIST [STRATEGY_LIST ...]
                         Provide a space-separated list of strategies to
                         backtest. Please note that ticker-interval needs to be
@@ -425,7 +429,12 @@ It contains some useful key metrics about performance of your strategy on backte
 - `Drawdown Start` / `Drawdown End`: Start and end datetime for this largest drawdown (can also be visualized via the `plot-dataframe` sub-command).
 - `Market change`: Change of the market during the backtest period. Calculated as average of all pairs changes from the first to the last candle using the "close" column.
 
-### Assumptions made by backtesting
+### Further backtest-result analysis
+
+To further analyze your backtest results, you can [export the trades](#exporting-trades-to-file).
+You can then load the trades to perform further analysis as shown in our [data analysis](data-analysis.md#backtesting) backtesting section.
+
+## Assumptions made by backtesting
 
 Since backtesting lacks some detailed information about what happens within a candle, it needs to take a few assumptions:
 
@@ -456,10 +465,30 @@ Also, keep in mind that past results don't guarantee future success.
 
 In addition to the above assumptions, strategy authors should carefully read the [Common Mistakes](strategy-customization.md#common-mistakes-when-developing-strategies) section, to avoid using data in backtesting which is not available in real market conditions.
 
-### Further backtest-result analysis
+### Improved backtest accuracy
 
-To further analyze your backtest results, you can [export the trades](#exporting-trades-to-file).
+One big limitation of backtesting is it's inability to know how prices moved intra-candle (was high before close, or viceversa?).
-You can then load the trades to perform further analysis as shown in our [data analysis](data-analysis.md#backtesting) backtesting section.
+So assuming you run backtesting with a 1h timeframe, there will be 4 prices for that candle (Open, High, Low, Close).
+
+While backtesting does take some assumptions (read above) about this - this can never be perfect, and will always be biased in one way or the other.
+To mitigate this, freqtrade can use a lower (faster) timeframe to simulate intra-candle movements.
+
+To utilize this, you can append `--timeframe-detail 5m` to your regular backtesting command.
+
+``` bash
+freqtrade backtesting --strategy AwesomeStrategy --timeframe 1h --timeframe-detail 5m
+```
+
+This will load 1h data as well as 5m data for the timeframe. The strategy will be analyzed with the 1h timeframe - and for every "open trade candle" (candles where a trade is open) the 5m data will be used to simulate intra-candle movements.
+All callback functions (`custom_sell()`, `custom_stoploss()`, ... ) will be running for each 5m candle once the trade is opened (so 12 times in the above example of 1h timeframe, and 5m detailed timeframe).
+
+`--timeframe-detail` must be smaller than the original timeframe, otherwise backtesting will fail to start.
+
+Obviously this will require more memory (5m data is bigger than 1h data), and will also impact runtime (depending on the amount of trades and trade durations).
+Also, data must be available / downloaded already.
+
+!!! Tip
+    You can use this function as the last part of strategy development, to ensure your strategy is not exploiting one of the [backtesting assumptions](#assumptions-made-by-backtesting). Strategies that perform similarly well with this mode have a good chance to perform well in dry/live modes too (although only forward-testing (dry-mode) can really confirm a strategy).
 
 ## Backtesting multiple strategies
 

freqtrade/commands/arguments.py

@@ -22,7 +22,7 @@ ARGS_COMMON_OPTIMIZE = ["timeframe", "timerange", "dataformat_ohlcv",
                         "max_open_trades", "stake_amount", "fee", "pairs"]
 
 ARGS_BACKTEST = ARGS_COMMON_OPTIMIZE + ["position_stacking", "use_max_market_positions",
-                                        "enable_protections", "dry_run_wallet",
+                                        "enable_protections", "dry_run_wallet", "timeframe_detail",
                                         "strategy_list", "export", "exportfilename"]
 
 ARGS_HYPEROPT = ARGS_COMMON_OPTIMIZE + ["hyperopt", "hyperopt_path",

freqtrade/commands/cli_options.py

@@ -135,6 +135,10 @@ AVAILABLE_CLI_OPTIONS = {
         help='Override the value of the `stake_amount` configuration setting.',
     ),
     # Backtesting
+    "timeframe_detail": Arg(
+        '--timeframe-detail',
+        help='Specify detail timeframe for backtesting (`1m`, `5m`, `30m`, `1h`, `1d`).',
+    ),
     "position_stacking": Arg(
         '--eps', '--enable-position-stacking',
         help='Allow buying the same pair multiple times (position stacking).',

freqtrade/configuration/configuration.py

@@ -242,6 +242,9 @@ class Configuration:
             except ValueError:
                 pass
 
+        self._args_to_config(config, argname='timeframe_detail',
+                             logstring='Parameter --timeframe-detail detected, '
+                             'using {} for intra-candle backtesting ...')
         self._args_to_config(config, argname='stake_amount',
                              logstring='Parameter --stake-amount detected, '
                              'overriding stake_amount to: {} ...')

freqtrade/optimize/backtesting.py

@@ -86,6 +86,17 @@ class Backtesting:
                                        "configuration or as cli argument `--timeframe 5m`")
         self.timeframe = str(self.config.get('timeframe'))
         self.timeframe_min = timeframe_to_minutes(self.timeframe)
+        # Load detail timeframe if specified
+        self.timeframe_detail = str(self.config.get('timeframe_detail', ''))
+        if self.timeframe_detail:
+            self.timeframe_detail_min = timeframe_to_minutes(self.timeframe_detail)
+            if self.timeframe_min <= self.timeframe_detail_min:
+                raise OperationalException(
+                    "Detail timeframe must be smaller than strategy timeframe.")
+
+        else:
+            self.timeframe_detail_min = 0
+        self.detail_data: Dict[str, DataFrame] = {}
 
         self.pairlists = PairListManager(self.exchange, self.config)
         if 'VolumePairList' in self.pairlists.name_list:
@@ -188,6 +199,23 @@ class Backtesting:
         self.progress.set_new_value(1)
         return data, self.timerange
 
+    def load_bt_data_detail(self) -> None:
+        """
+        Loads backtest detail data (smaller timeframe) if necessary.
+        """
+        if self.timeframe_detail:
+            self.detail_data = history.load_data(
+                datadir=self.config['datadir'],
+                pairs=self.pairlists.whitelist,
+                timeframe=self.timeframe_detail,
+                timerange=self.timerange,
+                startup_candles=0,
+                fail_without_data=True,
+                data_format=self.config.get('dataformat_ohlcv', 'json'),
+            )
+        else:
+            self.detail_data = {}
+
     def prepare_backtest(self, enable_protections):
         """
         Backtesting setup method - called once for every call to "backtest()".
@@ -318,7 +346,8 @@ class Backtesting:
         else:
             return sell_row[OPEN_IDX]
 
-    def _get_sell_trade_entry(self, trade: LocalTrade, sell_row: Tuple) -> Optional[LocalTrade]:
+    def _get_sell_trade_entry_for_candle(self, trade: LocalTrade,
+                                         sell_row: Tuple) -> Optional[LocalTrade]:
         sell_candle_time = sell_row[DATE_IDX].to_pydatetime()
         sell = self.strategy.should_sell(trade, sell_row[OPEN_IDX],  # type: ignore
                                          sell_candle_time, sell_row[BUY_IDX],
@@ -346,6 +375,32 @@ class Backtesting:
 
         return None
 
+    def _get_sell_trade_entry(self, trade: LocalTrade, sell_row: Tuple) -> Optional[LocalTrade]:
+        if self.timeframe_detail and trade.pair in self.detail_data:
+            sell_candle_time = sell_row[DATE_IDX].to_pydatetime()
+            sell_candle_end = sell_candle_time + timedelta(minutes=self.timeframe_min)
+
+            detail_data = self.detail_data[trade.pair]
+            detail_data = detail_data.loc[
+                (detail_data['date'] >= sell_candle_time) &
+                (detail_data['date'] < sell_candle_end)
+             ]
+            if len(detail_data) == 0:
+                # Fall back to "regular" data if no detail data was found for this candle
+                return self._get_sell_trade_entry_for_candle(trade, sell_row)
+            detail_data['buy'] = sell_row[BUY_IDX]
+            detail_data['sell'] = sell_row[SELL_IDX]
+            headers = ['date', 'buy', 'open', 'close', 'sell', 'low', 'high']
+            for det_row in detail_data[headers].values.tolist():
+                res = self._get_sell_trade_entry_for_candle(trade, det_row)
+                if res:
+                    return res
+
+            return None
+
+        else:
+            return self._get_sell_trade_entry_for_candle(trade, sell_row)
+
     def _enter_trade(self, pair: str, row: List) -> Optional[LocalTrade]:
         try:
             stake_amount = self.wallets.get_trade_stake_amount(pair, None)
@@ -592,6 +647,7 @@ class Backtesting:
         data: Dict[str, Any] = {}
 
         data, timerange = self.load_bt_data()
+        self.load_bt_data_detail()
         logger.info("Dataload complete. Calculating indicators")
 
         for strat in self.strategylist:

freqtrade/optimize/optimize_reports.py

@@ -368,6 +368,7 @@ def generate_strategy_stats(btdata: Dict[str, DataFrame],
         'max_open_trades_setting': (config['max_open_trades']
                                     if config['max_open_trades'] != float('inf') else -1),
         'timeframe': config['timeframe'],
+        'timeframe_detail': config.get('timeframe_detail', ''),
         'timerange': config.get('timerange', ''),
         'enable_protections': config.get('enable_protections', False),
         'strategy_name': strategy,

freqtrade/rpc/api_server/api_backtest.py

@@ -46,11 +46,14 @@ async def api_start_backtest(bt_settings: BacktestRequest, background_tasks: Bac
             if (
                 not ApiServer._bt
                 or lastconfig.get('timeframe') != strat.timeframe
+                or lastconfig.get('timeframe_detail') != btconfig.get('timeframe_detail')
                 or lastconfig.get('dry_run_wallet') != btconfig.get('dry_run_wallet', 0)
                 or lastconfig.get('timerange') != btconfig['timerange']
             ):
                 from freqtrade.optimize.backtesting import Backtesting
                 ApiServer._bt = Backtesting(btconfig)
+                if ApiServer._bt.timeframe_detail:
+                    ApiServer._bt.load_bt_data_detail()
 
             # Only reload data if timeframe changed.
             if (

freqtrade/rpc/api_server/api_schemas.py

@@ -324,6 +324,7 @@ class PairHistory(BaseModel):
 class BacktestRequest(BaseModel):
     strategy: str
     timeframe: Optional[str]
+    timeframe_detail: Optional[str]
     timerange: Optional[str]
     max_open_trades: Optional[int]
     stake_amount: Optional[Union[float, str]]

tests/optimize/test_backtesting.py

@@ -1,7 +1,7 @@
 # pragma pylint: disable=missing-docstring, W0212, line-too-long, C0103, unused-argument
 
 import random
-from datetime import timedelta
+from datetime import datetime, timedelta, timezone
 from pathlib import Path
 from unittest.mock import MagicMock, PropertyMock
 
@@ -441,6 +441,15 @@ def test_backtesting_no_pair_left(default_conf, mocker, caplog, testdatadir) ->
     with pytest.raises(OperationalException, match='VolumePairList not allowed for backtesting.'):
         Backtesting(default_conf)
 
+    default_conf.update({
+        'pairlists': [{"method": "StaticPairList"}],
+        'timeframe_detail': '1d',
+    })
+
+    with pytest.raises(OperationalException,
+                       match='Detail timeframe must be smaller than strategy timeframe.'):
+        Backtesting(default_conf)
+
 
 def test_backtesting_pairlist_list(default_conf, mocker, caplog, testdatadir, tickers) -> None:
     mocker.patch('freqtrade.exchange.Exchange.exchange_has', MagicMock(return_value=True))
@@ -491,7 +500,7 @@ def test_backtest__enter_trade(default_conf, fee, mocker) -> None:
     pair = 'UNITTEST/BTC'
     row = [
         pd.Timestamp(year=2020, month=1, day=1, hour=5, minute=0),
-        1,  # Sell
+        1,  # Buy
         0.001,  # Open
         0.0011,  # Close
         0,  # Sell
@@ -539,6 +548,88 @@ def test_backtest__enter_trade(default_conf, fee, mocker) -> None:
     backtesting.cleanup()
 
 
+def test_backtest__get_sell_trade_entry(default_conf, fee, mocker) -> None:
+    default_conf['use_sell_signal'] = False
+    mocker.patch('freqtrade.exchange.Exchange.get_fee', fee)
+    mocker.patch("freqtrade.exchange.Exchange.get_min_pair_stake_amount", return_value=0.00001)
+    patch_exchange(mocker)
+    default_conf['timeframe_detail'] = '1m'
+    default_conf['max_open_trades'] = 2
+    backtesting = Backtesting(default_conf)
+    backtesting._set_strategy(backtesting.strategylist[0])
+    pair = 'UNITTEST/BTC'
+    row = [
+        pd.Timestamp(year=2020, month=1, day=1, hour=4, minute=55, tzinfo=timezone.utc),
+        1,  # Buy
+        200,  # Open
+        201,  # Close
+        0,  # Sell
+        195,  # Low
+        201.5,  # High
+        '',  # Buy Signal Name
+    ]
+
+    trade = backtesting._enter_trade(pair, row=row)
+    assert isinstance(trade, LocalTrade)
+
+    row_sell = [
+        pd.Timestamp(year=2020, month=1, day=1, hour=5, minute=0, tzinfo=timezone.utc),
+        0,  # Buy
+        200,  # Open
+        201,  # Close
+        0,  # Sell
+        195,  # Low
+        210.5,  # High
+        '',  # Buy Signal Name
+    ]
+    row_detail = pd.DataFrame(
+        [
+            [
+                pd.Timestamp(year=2020, month=1, day=1, hour=5, minute=0, tzinfo=timezone.utc),
+                1, 200, 199, 0, 197, 200.1, '',
+            ], [
+                pd.Timestamp(year=2020, month=1, day=1, hour=5, minute=1, tzinfo=timezone.utc),
+                0, 199, 199.5, 0, 199, 199.7, '',
+            ], [
+                pd.Timestamp(year=2020, month=1, day=1, hour=5, minute=2, tzinfo=timezone.utc),
+                0, 199.5, 200.5, 0, 199, 200.8, '',
+            ], [
+                pd.Timestamp(year=2020, month=1, day=1, hour=5, minute=3, tzinfo=timezone.utc),
+                0, 200.5, 210.5, 0, 193, 210.5, '',  # ROI sell (?)
+            ], [
+                pd.Timestamp(year=2020, month=1, day=1, hour=5, minute=4, tzinfo=timezone.utc),
+                0, 200, 199, 0, 193, 200.1, '',
+            ],
+        ], columns=["date", "buy", "open", "close", "sell", "low", "high", "buy_tag"]
+    )
+
+    # No data available.
+    res = backtesting._get_sell_trade_entry(trade, row_sell)
+    assert res is not None
+    assert res.sell_reason == SellType.ROI.value
+    assert res.close_date_utc == datetime(2020, 1, 1, 5, 0, tzinfo=timezone.utc)
+
+    # Enter new trade
+    trade = backtesting._enter_trade(pair, row=row)
+    assert isinstance(trade, LocalTrade)
+    # Assign empty ... no result.
+    backtesting.detail_data[pair] = pd.DataFrame(
+        [], columns=["date", "buy", "open", "close", "sell", "low", "high", "buy_tag"])
+
+    res = backtesting._get_sell_trade_entry(trade, row)
+    assert res is None
+
+    # Assign backtest-detail data
+    backtesting.detail_data[pair] = row_detail
+
+    res = backtesting._get_sell_trade_entry(trade, row_sell)
+    assert res is not None
+    assert res.sell_reason == SellType.ROI.value
+    # Sell at minute 3 (not available above!)
+    assert res.close_date_utc == datetime(2020, 1, 1, 5, 3, tzinfo=timezone.utc)
+    assert round(res.close_rate, 3) == round(209.0225, 3)
+
+
 def test_backtest_one(default_conf, fee, mocker, testdatadir) -> None:
     default_conf['use_sell_signal'] = False
     mocker.patch('freqtrade.exchange.Exchange.get_fee', fee)
@@ -1042,3 +1133,102 @@ def test_backtest_start_multi_strat_nomock(default_conf, mocker, caplog, testdat
     assert 'LEFT OPEN TRADES REPORT' in captured.out
     assert '2017-11-14 21:17:00 -> 2017-11-14 22:58:00 | Max open trades : 1' in captured.out
     assert 'STRATEGY SUMMARY' in captured.out
+
+
+@pytest.mark.filterwarnings("ignore:deprecated")
+def test_backtest_start_multi_strat_nomock_detail(default_conf, mocker,
+                                                  caplog, testdatadir, capsys):
+    # Tests detail-data loading
+    default_conf.update({
+        "use_sell_signal": True,
+        "sell_profit_only": False,
+        "sell_profit_offset": 0.0,
+        "ignore_roi_if_buy_signal": False,
+    })
+    patch_exchange(mocker)
+    result1 = pd.DataFrame({'pair': ['XRP/BTC', 'LTC/BTC'],
+                            'profit_ratio': [0.0, 0.0],
+                            'profit_abs': [0.0, 0.0],
+                            'open_date': pd.to_datetime(['2018-01-29 18:40:00',
+                                                         '2018-01-30 03:30:00', ], utc=True
+                                                        ),
+                            'close_date': pd.to_datetime(['2018-01-29 20:45:00',
+                                                          '2018-01-30 05:35:00', ], utc=True),
+                            'trade_duration': [235, 40],
+                            'is_open': [False, False],
+                            'stake_amount': [0.01, 0.01],
+                            'open_rate': [0.104445, 0.10302485],
+                            'close_rate': [0.104969, 0.103541],
+                            'sell_reason': [SellType.ROI, SellType.ROI]
+                            })
+    result2 = pd.DataFrame({'pair': ['XRP/BTC', 'LTC/BTC', 'ETH/BTC'],
+                            'profit_ratio': [0.03, 0.01, 0.1],
+                            'profit_abs': [0.01, 0.02, 0.2],
+                            'open_date': pd.to_datetime(['2018-01-29 18:40:00',
+                                                         '2018-01-30 03:30:00',
+                                                         '2018-01-30 05:30:00'], utc=True
+                                                        ),
+                            'close_date': pd.to_datetime(['2018-01-29 20:45:00',
+                                                          '2018-01-30 05:35:00',
+                                                          '2018-01-30 08:30:00'], utc=True),
+                            'trade_duration': [47, 40, 20],
+                            'is_open': [False, False, False],
+                            'stake_amount': [0.01, 0.01, 0.01],
+                            'open_rate': [0.104445, 0.10302485, 0.122541],
+                            'close_rate': [0.104969, 0.103541, 0.123541],
+                            'sell_reason': [SellType.ROI, SellType.ROI, SellType.STOP_LOSS]
+                            })
+    backtestmock = MagicMock(side_effect=[
+        {
+            'results': result1,
+            'config': default_conf,
+            'locks': [],
+            'rejected_signals': 20,
+            'final_balance': 1000,
+        },
+        {
+            'results': result2,
+            'config': default_conf,
+            'locks': [],
+            'rejected_signals': 20,
+            'final_balance': 1000,
+        }
+    ])
+    mocker.patch('freqtrade.plugins.pairlistmanager.PairListManager.whitelist',
+                 PropertyMock(return_value=['XRP/ETH']))
+    mocker.patch('freqtrade.optimize.backtesting.Backtesting.backtest', backtestmock)
+
+    patched_configuration_load_config_file(mocker, default_conf)
+
+    args = [
+        'backtesting',
+        '--config', 'config.json',
+        '--datadir', str(testdatadir),
+        '--strategy-path', str(Path(__file__).parents[1] / 'strategy/strats'),
+        '--timeframe', '5m',
+        '--timeframe-detail', '1m',
+        '--strategy-list',
+        'StrategyTestV2'
+    ]
+    args = get_args(args)
+    start_backtesting(args)
+
+    # check the logs, that will contain the backtest result
+    exists = [
+        'Parameter -i/--timeframe detected ... Using timeframe: 5m ...',
+        'Parameter --timeframe-detail detected, using 1m for intra-candle backtesting ...',
+        f'Using data directory: {testdatadir} ...',
+        'Loading data from 2019-10-11 00:00:00 '
+        'up to 2019-10-13 11:10:00 (2 days).',
+        'Backtesting with data from 2019-10-11 01:40:00 '
+        'up to 2019-10-13 11:10:00 (2 days).',
+        'Running backtesting for Strategy StrategyTestV2',
+    ]
+
+    for line in exists:
+        assert log_has(line, caplog)
+
+    captured = capsys.readouterr()
+    assert 'BACKTESTING REPORT' in captured.out
+    assert 'SELL REASON STATS' in captured.out
+    assert 'LEFT OPEN TRADES REPORT' in captured.out