adelorenzo/stable
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Merge pull request #5413 from freqtrade/feat/backtest_detail

Browse Source 
Add backtest-detail to allow intra-candle backtests
This commit is contained in:
Matthias 2021-09-01 19:48:12 +02:00 committed by GitHub
commit f59ba92920
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
9 changed files with 295 additions and 8 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

37
docs/backtesting.md
View File

@ -18,6 +18,7 @@ usage: freqtrade backtesting [-h] [-v] [--logfile FILE] [-V] [-c PATH]
[-p PAIRS [PAIRS ...]] [--eps] [--dmmp]
[--enable-protections]
[--dry-run-wallet DRY_RUN_WALLET]
[--timeframe-detail TIMEFRAME_DETAIL]
[--strategy-list STRATEGY_LIST [STRATEGY_LIST ...]]
[--export {none,trades}] [--export-filename PATH]
@ -55,6 +56,9 @@ optional arguments:
--dry-run-wallet DRY_RUN_WALLET, --starting-balance DRY_RUN_WALLET
Starting balance, used for backtesting / hyperopt and
dry-runs.
--timeframe-detail TIMEFRAME_DETAIL
Specify detail timeframe for backtesting (`1m`, `5m`,
`30m`, `1h`, `1d`).
--strategy-list STRATEGY_LIST [STRATEGY_LIST ...]
Provide a space-separated list of strategies to
backtest. Please note that ticker-interval needs to be
@ -425,7 +429,12 @@ It contains some useful key metrics about performance of your strategy on backte
- `Drawdown Start` / `Drawdown End`: Start and end datetime for this largest drawdown (can also be visualized via the `plot-dataframe` sub-command).
- `Market change`: Change of the market during the backtest period. Calculated as average of all pairs changes from the first to the last candle using the "close" column.
### Assumptions made by backtesting
### Further backtest-result analysis
To further analyze your backtest results, you can [export the trades](#exporting-trades-to-file).
You can then load the trades to perform further analysis as shown in our [data analysis](data-analysis.md#backtesting) backtesting section.
## Assumptions made by backtesting
Since backtesting lacks some detailed information about what happens within a candle, it needs to take a few assumptions:
@ -456,10 +465,30 @@ Also, keep in mind that past results don't guarantee future success.
In addition to the above assumptions, strategy authors should carefully read the [Common Mistakes](strategy-customization.md#common-mistakes-when-developing-strategies) section, to avoid using data in backtesting which is not available in real market conditions.
### Further backtest-result analysis
### Improved backtest accuracy
To further analyze your backtest results, you can [export the trades](#exporting-trades-to-file).
You can then load the trades to perform further analysis as shown in our [data analysis](data-analysis.md#backtesting) backtesting section.
One big limitation of backtesting is it's inability to know how prices moved intra-candle (was high before close, or viceversa?).
So assuming you run backtesting with a 1h timeframe, there will be 4 prices for that candle (Open, High, Low, Close).
While backtesting does take some assumptions (read above) about this - this can never be perfect, and will always be biased in one way or the other.
To mitigate this, freqtrade can use a lower (faster) timeframe to simulate intra-candle movements.
To utilize this, you can append `--timeframe-detail 5m` to your regular backtesting command.
``` bash
freqtrade backtesting --strategy AwesomeStrategy --timeframe 1h --timeframe-detail 5m
```
This will load 1h data as well as 5m data for the timeframe. The strategy will be analyzed with the 1h timeframe - and for every "open trade candle" (candles where a trade is open) the 5m data will be used to simulate intra-candle movements.
All callback functions (`custom_sell()`, `custom_stoploss()`, ... ) will be running for each 5m candle once the trade is opened (so 12 times in the above example of 1h timeframe, and 5m detailed timeframe).
`--timeframe-detail` must be smaller than the original timeframe, otherwise backtesting will fail to start.
Obviously this will require more memory (5m data is bigger than 1h data), and will also impact runtime (depending on the amount of trades and trade durations).
Also, data must be available / downloaded already.
!!! Tip
You can use this function as the last part of strategy development, to ensure your strategy is not exploiting one of the [backtesting assumptions](#assumptions-made-by-backtesting). Strategies that perform similarly well with this mode have a good chance to perform well in dry/live modes too (although only forward-testing (dry-mode) can really confirm a strategy).
## Backtesting multiple strategies

2
freqtrade/commands/arguments.py
View File

@ -22,7 +22,7 @@ ARGS_COMMON_OPTIMIZE = ["timeframe", "timerange", "dataformat_ohlcv",
"max_open_trades", "stake_amount", "fee", "pairs"]
ARGS_BACKTEST = ARGS_COMMON_OPTIMIZE + ["position_stacking", "use_max_market_positions",
"enable_protections", "dry_run_wallet",
"enable_protections", "dry_run_wallet", "timeframe_detail",
"strategy_list", "export", "exportfilename"]
ARGS_HYPEROPT = ARGS_COMMON_OPTIMIZE + ["hyperopt", "hyperopt_path",

4
freqtrade/commands/cli_options.py
View File

@ -135,6 +135,10 @@ AVAILABLE_CLI_OPTIONS = {
help='Override the value of the `stake_amount` configuration setting.',
),
# Backtesting
"timeframe_detail": Arg(
'--timeframe-detail',
help='Specify detail timeframe for backtesting (`1m`, `5m`, `30m`, `1h`, `1d`).',
),
"position_stacking": Arg(
'--eps', '--enable-position-stacking',
help='Allow buying the same pair multiple times (position stacking).',

3
freqtrade/configuration/configuration.py
View File

@ -242,6 +242,9 @@ class Configuration:
except ValueError:
pass
self._args_to_config(config, argname='timeframe_detail',
logstring='Parameter --timeframe-detail detected, '
'using {} for intra-candle backtesting ...')
self._args_to_config(config, argname='stake_amount',
logstring='Parameter --stake-amount detected, '
'overriding stake_amount to: {} ...')

58
freqtrade/optimize/backtesting.py
View File

@ -86,6 +86,17 @@ class Backtesting:
"configuration or as cli argument `--timeframe 5m`")
self.timeframe = str(self.config.get('timeframe'))
self.timeframe_min = timeframe_to_minutes(self.timeframe)
# Load detail timeframe if specified
self.timeframe_detail = str(self.config.get('timeframe_detail', ''))
if self.timeframe_detail:
self.timeframe_detail_min = timeframe_to_minutes(self.timeframe_detail)
if self.timeframe_min <= self.timeframe_detail_min:
raise OperationalException(
"Detail timeframe must be smaller than strategy timeframe.")
else:
self.timeframe_detail_min = 0
self.detail_data: Dict[str, DataFrame] = {}
self.pairlists = PairListManager(self.exchange, self.config)
if 'VolumePairList' in self.pairlists.name_list:
@ -188,6 +199,23 @@ class Backtesting:
self.progress.set_new_value(1)
return data, self.timerange
def load_bt_data_detail(self) -> None:
"""
Loads backtest detail data (smaller timeframe) if necessary.
"""
if self.timeframe_detail:
self.detail_data = history.load_data(
datadir=self.config['datadir'],
pairs=self.pairlists.whitelist,
timeframe=self.timeframe_detail,
timerange=self.timerange,
startup_candles=0,
fail_without_data=True,
data_format=self.config.get('dataformat_ohlcv', 'json'),
)
else:
self.detail_data = {}
def prepare_backtest(self, enable_protections):
"""
Backtesting setup method - called once for every call to "backtest()".
@ -318,7 +346,8 @@ class Backtesting:
else:
return sell_row[OPEN_IDX]
def _get_sell_trade_entry(self, trade: LocalTrade, sell_row: Tuple) -> Optional[LocalTrade]:
def _get_sell_trade_entry_for_candle(self, trade: LocalTrade,
sell_row: Tuple) -> Optional[LocalTrade]:
sell_candle_time = sell_row[DATE_IDX].to_pydatetime()
sell = self.strategy.should_sell(trade, sell_row[OPEN_IDX], # type: ignore
sell_candle_time, sell_row[BUY_IDX],
@ -346,6 +375,32 @@ class Backtesting:
return None
def _get_sell_trade_entry(self, trade: LocalTrade, sell_row: Tuple) -> Optional[LocalTrade]:
if self.timeframe_detail and trade.pair in self.detail_data:
sell_candle_time = sell_row[DATE_IDX].to_pydatetime()
sell_candle_end = sell_candle_time + timedelta(minutes=self.timeframe_min)
detail_data = self.detail_data[trade.pair]
detail_data = detail_data.loc[
(detail_data['date'] >= sell_candle_time) &
(detail_data['date'] < sell_candle_end)
]
if len(detail_data) == 0:
# Fall back to "regular" data if no detail data was found for this candle
return self._get_sell_trade_entry_for_candle(trade, sell_row)
detail_data['buy'] = sell_row[BUY_IDX]
detail_data['sell'] = sell_row[SELL_IDX]
headers = ['date', 'buy', 'open', 'close', 'sell', 'low', 'high']
for det_row in detail_data[headers].values.tolist():
res = self._get_sell_trade_entry_for_candle(trade, det_row)
if res:
return res
return None
else:
return self._get_sell_trade_entry_for_candle(trade, sell_row)
def _enter_trade(self, pair: str, row: List) -> Optional[LocalTrade]:
try:
stake_amount = self.wallets.get_trade_stake_amount(pair, None)
@ -592,6 +647,7 @@ class Backtesting:
data: Dict[str, Any] = {}
data, timerange = self.load_bt_data()
self.load_bt_data_detail()
logger.info("Dataload complete. Calculating indicators")
for strat in self.strategylist:

1
freqtrade/optimize/optimize_reports.py
View File

@ -368,6 +368,7 @@ def generate_strategy_stats(btdata: Dict[str, DataFrame],
'max_open_trades_setting': (config['max_open_trades']
if config['max_open_trades'] != float('inf') else -1),
'timeframe': config['timeframe'],
'timeframe_detail': config.get('timeframe_detail', ''),
'timerange': config.get('timerange', ''),
'enable_protections': config.get('enable_protections', False),
'strategy_name': strategy,

3
freqtrade/rpc/api_server/api_backtest.py
View File

@ -46,11 +46,14 @@ async def api_start_backtest(bt_settings: BacktestRequest, background_tasks: Bac
if (
not ApiServer._bt
or lastconfig.get('timeframe') != strat.timeframe
or lastconfig.get('timeframe_detail') != btconfig.get('timeframe_detail')
or lastconfig.get('dry_run_wallet') != btconfig.get('dry_run_wallet', 0)
or lastconfig.get('timerange') != btconfig['timerange']
):
from freqtrade.optimize.backtesting import Backtesting
ApiServer._bt = Backtesting(btconfig)
if ApiServer._bt.timeframe_detail:
ApiServer._bt.load_bt_data_detail()
# Only reload data if timeframe changed.
if (

1
freqtrade/rpc/api_server/api_schemas.py
View File

@ -324,6 +324,7 @@ class PairHistory(BaseModel):
class BacktestRequest(BaseModel):
strategy: str
timeframe: Optional[str]
timeframe_detail: Optional[str]
timerange: Optional[str]
max_open_trades: Optional[int]
stake_amount: Optional[Union[float, str]]

194
tests/optimize/test_backtesting.py
View File

@ -1,7 +1,7 @@
# pragma pylint: disable=missing-docstring, W0212, line-too-long, C0103, unused-argument
import random
from datetime import timedelta
from datetime import datetime, timedelta, timezone
from pathlib import Path
from unittest.mock import MagicMock, PropertyMock
@ -441,6 +441,15 @@ def test_backtesting_no_pair_left(default_conf, mocker, caplog, testdatadir) ->
with pytest.raises(OperationalException, match='VolumePairList not allowed for backtesting.'):
Backtesting(default_conf)
default_conf.update({
'pairlists': [{"method": "StaticPairList"}],
'timeframe_detail': '1d',
})
with pytest.raises(OperationalException,
match='Detail timeframe must be smaller than strategy timeframe.'):
Backtesting(default_conf)
def test_backtesting_pairlist_list(default_conf, mocker, caplog, testdatadir, tickers) -> None:
mocker.patch('freqtrade.exchange.Exchange.exchange_has', MagicMock(return_value=True))
@ -491,7 +500,7 @@ def test_backtest__enter_trade(default_conf, fee, mocker) -> None:
pair = 'UNITTEST/BTC'
row = [
pd.Timestamp(year=2020, month=1, day=1, hour=5, minute=0),
1, # Sell
1, # Buy
0.001, # Open
0.0011, # Close
0, # Sell
@ -539,6 +548,88 @@ def test_backtest__enter_trade(default_conf, fee, mocker) -> None:
backtesting.cleanup()
def test_backtest__get_sell_trade_entry(default_conf, fee, mocker) -> None:
default_conf['use_sell_signal'] = False
mocker.patch('freqtrade.exchange.Exchange.get_fee', fee)
mocker.patch("freqtrade.exchange.Exchange.get_min_pair_stake_amount", return_value=0.00001)
patch_exchange(mocker)
default_conf['timeframe_detail'] = '1m'
default_conf['max_open_trades'] = 2
backtesting = Backtesting(default_conf)
backtesting._set_strategy(backtesting.strategylist[0])
pair = 'UNITTEST/BTC'
row = [
pd.Timestamp(year=2020, month=1, day=1, hour=4, minute=55, tzinfo=timezone.utc),
1, # Buy
200, # Open
201, # Close
0, # Sell
195, # Low
201.5, # High
'', # Buy Signal Name
]
trade = backtesting._enter_trade(pair, row=row)
assert isinstance(trade, LocalTrade)
row_sell = [
pd.Timestamp(year=2020, month=1, day=1, hour=5, minute=0, tzinfo=timezone.utc),
0, # Buy
200, # Open
201, # Close
0, # Sell
195, # Low
210.5, # High
'', # Buy Signal Name
]
row_detail = pd.DataFrame(
[
[
pd.Timestamp(year=2020, month=1, day=1, hour=5, minute=0, tzinfo=timezone.utc),
1, 200, 199, 0, 197, 200.1, '',
], [
pd.Timestamp(year=2020, month=1, day=1, hour=5, minute=1, tzinfo=timezone.utc),
0, 199, 199.5, 0, 199, 199.7, '',
], [
pd.Timestamp(year=2020, month=1, day=1, hour=5, minute=2, tzinfo=timezone.utc),
0, 199.5, 200.5, 0, 199, 200.8, '',
], [
pd.Timestamp(year=2020, month=1, day=1, hour=5, minute=3, tzinfo=timezone.utc),
0, 200.5, 210.5, 0, 193, 210.5, '', # ROI sell (?)
], [
pd.Timestamp(year=2020, month=1, day=1, hour=5, minute=4, tzinfo=timezone.utc),
0, 200, 199, 0, 193, 200.1, '',
],
], columns=["date", "buy", "open", "close", "sell", "low", "high", "buy_tag"]
)
# No data available.
res = backtesting._get_sell_trade_entry(trade, row_sell)
assert res is not None
assert res.sell_reason == SellType.ROI.value
assert res.close_date_utc == datetime(2020, 1, 1, 5, 0, tzinfo=timezone.utc)
# Enter new trade
trade = backtesting._enter_trade(pair, row=row)
assert isinstance(trade, LocalTrade)
# Assign empty ... no result.
backtesting.detail_data[pair] = pd.DataFrame(
[], columns=["date", "buy", "open", "close", "sell", "low", "high", "buy_tag"])
res = backtesting._get_sell_trade_entry(trade, row)
assert res is None
# Assign backtest-detail data
backtesting.detail_data[pair] = row_detail
res = backtesting._get_sell_trade_entry(trade, row_sell)
assert res is not None
assert res.sell_reason == SellType.ROI.value
# Sell at minute 3 (not available above!)
assert res.close_date_utc == datetime(2020, 1, 1, 5, 3, tzinfo=timezone.utc)
assert round(res.close_rate, 3) == round(209.0225, 3)
def test_backtest_one(default_conf, fee, mocker, testdatadir) -> None:
default_conf['use_sell_signal'] = False
mocker.patch('freqtrade.exchange.Exchange.get_fee', fee)
@ -1042,3 +1133,102 @@ def test_backtest_start_multi_strat_nomock(default_conf, mocker, caplog, testdat
assert 'LEFT OPEN TRADES REPORT' in captured.out
assert '2017-11-14 21:17:00 -> 2017-11-14 22:58:00 | Max open trades : 1' in captured.out
assert 'STRATEGY SUMMARY' in captured.out
@pytest.mark.filterwarnings("ignore:deprecated")
def test_backtest_start_multi_strat_nomock_detail(default_conf, mocker,
caplog, testdatadir, capsys):
# Tests detail-data loading
default_conf.update({
"use_sell_signal": True,
"sell_profit_only": False,
"sell_profit_offset": 0.0,
"ignore_roi_if_buy_signal": False,
})
patch_exchange(mocker)
result1 = pd.DataFrame({'pair': ['XRP/BTC', 'LTC/BTC'],
'profit_ratio': [0.0, 0.0],
'profit_abs': [0.0, 0.0],
'open_date': pd.to_datetime(['2018-01-29 18:40:00',
'2018-01-30 03:30:00', ], utc=True
),
'close_date': pd.to_datetime(['2018-01-29 20:45:00',
'2018-01-30 05:35:00', ], utc=True),
'trade_duration': [235, 40],
'is_open': [False, False],
'stake_amount': [0.01, 0.01],
'open_rate': [0.104445, 0.10302485],
'close_rate': [0.104969, 0.103541],
'sell_reason': [SellType.ROI, SellType.ROI]
})
result2 = pd.DataFrame({'pair': ['XRP/BTC', 'LTC/BTC', 'ETH/BTC'],
'profit_ratio': [0.03, 0.01, 0.1],
'profit_abs': [0.01, 0.02, 0.2],
'open_date': pd.to_datetime(['2018-01-29 18:40:00',
'2018-01-30 03:30:00',
'2018-01-30 05:30:00'], utc=True
),
'close_date': pd.to_datetime(['2018-01-29 20:45:00',
'2018-01-30 05:35:00',
'2018-01-30 08:30:00'], utc=True),
'trade_duration': [47, 40, 20],
'is_open': [False, False, False],
'stake_amount': [0.01, 0.01, 0.01],
'open_rate': [0.104445, 0.10302485, 0.122541],
'close_rate': [0.104969, 0.103541, 0.123541],
'sell_reason': [SellType.ROI, SellType.ROI, SellType.STOP_LOSS]
})
backtestmock = MagicMock(side_effect=[
{
'results': result1,
'config': default_conf,
'locks': [],
'rejected_signals': 20,
'final_balance': 1000,
},
{
'results': result2,
'config': default_conf,
'locks': [],
'rejected_signals': 20,
'final_balance': 1000,
}
])
mocker.patch('freqtrade.plugins.pairlistmanager.PairListManager.whitelist',
PropertyMock(return_value=['XRP/ETH']))
mocker.patch('freqtrade.optimize.backtesting.Backtesting.backtest', backtestmock)
patched_configuration_load_config_file(mocker, default_conf)
args = [
'backtesting',
'--config', 'config.json',
'--datadir', str(testdatadir),
'--strategy-path', str(Path(__file__).parents[1] / 'strategy/strats'),
'--timeframe', '5m',
'--timeframe-detail', '1m',
'--strategy-list',
'StrategyTestV2'
]
args = get_args(args)
start_backtesting(args)
# check the logs, that will contain the backtest result
exists = [
'Parameter -i/--timeframe detected ... Using timeframe: 5m ...',
'Parameter --timeframe-detail detected, using 1m for intra-candle backtesting ...',
f'Using data directory: {testdatadir} ...',
'Loading data from 2019-10-11 00:00:00 '
'up to 2019-10-13 11:10:00 (2 days).',
'Backtesting with data from 2019-10-11 01:40:00 '
'up to 2019-10-13 11:10:00 (2 days).',
'Running backtesting for Strategy StrategyTestV2',
]
for line in exists:
assert log_has(line, caplog)
captured = capsys.readouterr()
assert 'BACKTESTING REPORT' in captured.out
assert 'SELL REASON STATS' in captured.out
assert 'LEFT OPEN TRADES REPORT' in captured.out