From b1a3e213ae9eab1479ba852589593591f425a485 Mon Sep 17 00:00:00 2001 From: Matthias Date: Sat, 21 Sep 2019 10:09:14 +0200 Subject: [PATCH 1/6] Improve backtesting docs --- docs/backtesting.md | 117 +++++++++++++++++------------------------- docs/data-download.md | 39 ++++++++++++++ docs/hyperopt.md | 3 ++ mkdocs.yml | 1 + 4 files changed, 89 insertions(+), 71 deletions(-) create mode 100644 docs/data-download.md diff --git a/docs/backtesting.md b/docs/backtesting.md index 45d5f486f..b3f22c664 100644 --- a/docs/backtesting.md +++ b/docs/backtesting.md @@ -1,41 +1,9 @@ # Backtesting -This page explains how to validate your strategy performance by using -Backtesting. +This page explains how to validate your strategy performance by using Backtesting. -## Getting data for backtesting and hyperopt - -To download data (candles / OHLCV) needed for backtesting and hyperoptimization use the `freqtrade download-data` command. - -If no additional parameter is specified, freqtrade will download data for `"1m"` and `"5m"` timeframes. -Exchange and pairs will come from `config.json` (if specified using `-c/--config`). Otherwise `--exchange` becomes mandatory. - -Alternatively, a `pairs.json` file can be used. - -If you are using Binance for example: - -- create a directory `user_data/data/binance` and copy `pairs.json` in that directory. -- update the `pairs.json` to contain the currency pairs you are interested in. - -```bash -mkdir -p user_data/data/binance -cp freqtrade/tests/testdata/pairs.json user_data/data/binance -``` - -Then run: - -```bash -freqtrade download-data --exchange binance -``` - -This will download ticker data for all the currency pairs you defined in `pairs.json`. - -- To use a different directory than the exchange specific default, use `--datadir user_data/data/some_directory`. -- To change the exchange used to download the tickers, please use a different configuration file (you'll probably need to adjust ratelimits etc.) -- To use `pairs.json` from some other directory, use `--pairs-file some_other_dir/pairs.json`. -- To download ticker data for only 10 days, use `--days 10` (defaults to 30 days). -- Use `--timeframes` to specify which tickers to download. Default is `--timeframes 1m 5m` which will download 1-minute and 5-minute tickers. -- To use exchange, timeframe and list of pairs as defined in your configuration file, use the `-c/--config` option. With this, the script uses the whitelist defined in the config as the list of currency pairs to download data for and does not require the pairs.json file. You can combine `-c/--config` with most other options. +Backtesting requires historic data to be available. +To learn how to get data for the pairs and exchange you're interrested in, head over to the [Data downloading](data-download.md) section of the documentation. ## Test your strategy with Backtesting @@ -46,14 +14,13 @@ real data. This is what we call Backtesting will use the crypto-currencies (pairs) from your config file and load ticker data from `user_data/data/` by default. If no data is available for the exchange / pair / ticker interval combination, backtesting will -ask you to download them first using `freqtrade download-data`. -For details on downloading, please refer to the [relevant section](#Getting-data-for-backtesting-and-hyperopt) in the documentation. +ask you to download them first using `freqtrade download-data`. +For details on downloading, please refer to the [Downloading data](data-download.md) section in the documentation. -The result of backtesting will confirm you if your bot has better odds of making a profit than a loss. - -The backtesting is very easy with freqtrade. +The result of backtesting will confirm if your bot has better odds of making a profit than a loss. ### Run a backtesting against the currencies listed in your config file + #### With 5 min tickers (Per default) ```bash @@ -107,31 +74,33 @@ freqtrade backtesting --export trades --export-filename=backtest_samplestrategy. #### Running backtest with smaller testset -Use the `--timerange` argument to change how much of the testset -you want to use. The last N ticks/timeframes will be used. +Use the `--timerange` argument to change how much of the testset you want to use. Example: ```bash -freqtrade backtesting --timerange=-200 +freqtrade backtesting --timerange=20190501- ``` #### Advanced use of timerange -Doing `--timerange=-200` will get the last 200 timeframes -from your inputdata. You can also specify specific dates, -or a range span indexed by start and stop. +Doing `--timerange=20190101-` will all available data starting with January 1st, 2019 from your inputdata. +You can also specify specific dates, or a range span indexed by start and stop. The full timerange specification: -- Use last 123 tickframes of data: `--timerange=-123` -- Use first 123 tickframes of data: `--timerange=123-` -- Use tickframes from line 123 through 456: `--timerange=123-456` + - Use tickframes till 2018/01/31: `--timerange=-20180131` - Use tickframes since 2018/01/31: `--timerange=20180131-` - Use tickframes since 2018/01/31 till 2018/03/01 : `--timerange=20180131-20180301` - Use tickframes between POSIX timestamps 1527595200 1527618600: `--timerange=1527595200-1527618600` +- Use last 123 tickframes of data: `--timerange=-123` +- Use first 123 tickframes of data: `--timerange=123-` +- Use tickframes from line 123 through 456: `--timerange=123-456` + +!!! warning + Be carefull when using non-date functions - these do not allow you to specify precise dates, so if you updated the test-data it will probably use a different dataset. ## Understand the backtesting result @@ -177,11 +146,12 @@ A backtesting result will look like that: | TOTAL | 2 | 0.78 | 1.57 | 0.00007855 | 0.78 | 4:00:00 | 2 | 0 | ``` -The 1st table will contain all trades the bot made. +The 1st table contains all trades the bot made, including "left open trades". -The 2nd table will contain a recap of sell reasons. +The 2nd table contains a recap of sell reasons. -The 3rd table will contain all trades the bot had to `forcesell` at the end of the backtest period to present a full picture. +The 3rd table contains all trades the bot had to `forcesell` at the end of the backtest period to present a full picture. +This is necessary to simulate realistic behaviour, since the backtest period has to end at some point, while realistically, you could leave the bot running forever. These trades are also included in the first table, but are extracted separately for clarity. The last line will give you the overall performance of your strategy, @@ -191,22 +161,16 @@ here: | TOTAL | 429 | 0.36 | 152.41 | 0.00762792 | 76.20 | 4:12:00 | 186 | 243 | ``` -We understand the bot has made `429` trades for an average duration of -`4:12:00`, with a performance of `76.20%` (profit), that means it has +The bot has made `429` trades for an average duration of `4:12:00`, with a performance of `76.20%` (profit), that means it has earned a total of `0.00762792 BTC` starting with a capital of 0.01 BTC. -The column `avg profit %` shows the average profit for all trades made while the column `cum profit %` sums all the profits/losses. -The column `tot profit %` shows instead the total profit % in relation to allocated capital -(`max_open_trades * stake_amount`). In the above results we have `max_open_trades=2 stake_amount=0.005` in config -so `(76.20/100) * (0.005 * 2) =~ 0.00762792 BTC`. +The column `avg profit %` shows the average profit for all trades made while the column `cum profit %` sums all the profits/losses. +The column `tot profit %` shows instead the total profit % in relation to allocated capital (`max_open_trades * stake_amount`). +In the above results we have `max_open_trades=2 stake_amount=0.005` in config so `(76.20/100) * (0.005 * 2) =~ 0.00762792 BTC`. -As you will see your strategy performance will be influenced by your buy -strategy, your sell strategy, and also by the `minimal_roi` and -`stop_loss` you have set. +Your strategy performance is influenced by your buy strategy, your sell strategy, and also by the `minimal_roi` and `stop_loss` you have set. -As for an example if your minimal_roi is only `"0": 0.01`. You cannot -expect the bot to make more profit than 1% (because it will sell every -time a trade will reach 1%). +For example, if your minimal_roi is only `"0": 0.01`. You cannot expect the bot to make more profit than 1% (because it will sell every time a trade will reach 1%). ```json "minimal_roi": { @@ -215,22 +179,33 @@ time a trade will reach 1%). ``` On the other hand, if you set a too high `minimal_roi` like `"0": 0.55` -(55%), there is a lot of chance that the bot will never reach this -profit. Hence, keep in mind that your performance is a mix of your -strategies, your configuration, and the crypto-currency you have set up. +(55%), there is almost no chance that the bot will never reach this profit. +Hence, keep in mind that your performance is a mix of the different elements of the strategy, your configuration, and the crypto-currency pairs you have set up. + +### Assumptions made by backtesting + +Since backtesting lacks some detailed information about what happens within a candle, it needs to take a few assumptions: + +- Buys happen at open-price +- Low happens before high for stoploss, protecting capital first. +- ROI sells are compared to high - but the ROI value is used (e.g. ROI = 2%, high=5% - so the sell will be at 2%) +- Stoploss sells happen exactly at stoploss price, even if low was lower +- Trailing stoploss + - High happens first - adjusting stoploss + - Low uses the adjusted stoploss (so sells with large high-low difference are backtested correctly) +- Sell-reason does not explain if a trade was positive or negative, just what triggered the sell (this can look odd if negative ROI values are used) ### Further backtest-result analysis To further analyze your backtest results, you can [export the trades](#exporting-trades-to-file). You can then load the trades to perform further analysis as shown in our [data analysis](data-analysis.md#backtesting) backtesting section. - ## Backtesting multiple strategies -To backtest multiple strategies, a list of Strategies can be provided. +To compare multiple strategies, a list of Strategies can be provided to backtesting. This is limited to 1 ticker-interval per run, however, data is only loaded once from disk so if you have multiple -strategies you'd like to compare, this should give a nice runtime boost. +strategies you'd like to compare, this will give a nice runtime boost. All listed Strategies need to be in the same directory. @@ -240,7 +215,7 @@ freqtrade backtesting --timerange 20180401-20180410 --ticker-interval 5m --strat This will save the results to `user_data/backtest_results/backtest-result-.json`, injecting the strategy-name into the target filename. There will be an additional table comparing win/losses of the different strategies (identical to the "Total" row in the first table). -Detailed output for all strategies one after the other will be available, so make sure to scroll up. +Detailed output for all strategies one after the other will be available, so make sure to scroll up to see the details per strategy. ``` =========================================================== Strategy Summary =========================================================== diff --git a/docs/data-download.md b/docs/data-download.md new file mode 100644 index 000000000..f6b1e9904 --- /dev/null +++ b/docs/data-download.md @@ -0,0 +1,39 @@ +# Data download + +## Getting data for backtesting and hyperopt + +To download data (candles / OHLCV) needed for backtesting and hyperoptimization use the `freqtrade download-data` command. + +If no additional parameter is specified, freqtrade will download data for `"1m"` and `"5m"` timeframes. +Exchange and pairs will come from `config.json` (if specified using `-c/--config`). Otherwise `--exchange` becomes mandatory. + +Alternatively, a `pairs.json` file can be used. + +If you are using Binance for example: + +- create a directory `user_data/data/binance` and copy `pairs.json` in that directory. +- update the `pairs.json` to contain the currency pairs you are interested in. + +```bash +mkdir -p user_data/data/binance +cp freqtrade/tests/testdata/pairs.json user_data/data/binance +``` + +Then run: + +```bash +freqtrade download-data --exchange binance +``` + +This will download ticker data for all the currency pairs you defined in `pairs.json`. + +- To use a different directory than the exchange specific default, use `--datadir user_data/data/some_directory`. +- To change the exchange used to download the tickers, please use a different configuration file (you'll probably need to adjust ratelimits etc.) +- To use `pairs.json` from some other directory, use `--pairs-file some_other_dir/pairs.json`. +- To download ticker data for only 10 days, use `--days 10` (defaults to 30 days). +- Use `--timeframes` to specify which tickers to download. Default is `--timeframes 1m 5m` which will download 1-minute and 5-minute tickers. +- To use exchange, timeframe and list of pairs as defined in your configuration file, use the `-c/--config` option. With this, the script uses the whitelist defined in the config as the list of currency pairs to download data for and does not require the pairs.json file. You can combine `-c/--config` with most other options. + +## Next step + +Great, you now have backtest data downloaded, so you can now start [backtesting](backtesting.md) your strategy. diff --git a/docs/hyperopt.md b/docs/hyperopt.md index becf572ee..00aec458c 100644 --- a/docs/hyperopt.md +++ b/docs/hyperopt.md @@ -6,6 +6,9 @@ algorithms included in the `scikit-optimize` package to accomplish this. The search will burn all your CPU cores, make your laptop sound like a fighter jet and still take a long time. +Hyperopt requires historic data to be available, just as backtesting does. +To learn how to get data for the pairs and exchange you're interrested in, head over to the [Data downloading](data-download.md) section of the documentation. + !!! Bug Hyperopt will crash when used with only 1 CPU Core as found out in [Issue #1133](https://github.com/freqtrade/freqtrade/issues/1133) diff --git a/mkdocs.yml b/mkdocs.yml index 869c6565c..069f060db 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -11,6 +11,7 @@ nav: - Telegram: telegram-usage.md - Web Hook: webhook-config.md - REST API: rest-api.md + - Data Download: data-download.md - Backtesting: backtesting.md - Hyperopt: hyperopt.md - Edge positioning: edge.md From cc9fc413188105f1af6e87df38406bf031481c8a Mon Sep 17 00:00:00 2001 From: Matthias Date: Tue, 24 Sep 2019 06:56:19 +0200 Subject: [PATCH 2/6] Rename section to data-downloading, implement some feedback --- docs/backtesting.md | 12 ++++++------ docs/hyperopt.md | 2 +- mkdocs.yml | 2 +- 3 files changed, 8 insertions(+), 8 deletions(-) diff --git a/docs/backtesting.md b/docs/backtesting.md index b3f22c664..9ddc6bbb3 100644 --- a/docs/backtesting.md +++ b/docs/backtesting.md @@ -3,7 +3,7 @@ This page explains how to validate your strategy performance by using Backtesting. Backtesting requires historic data to be available. -To learn how to get data for the pairs and exchange you're interrested in, head over to the [Data downloading](data-download.md) section of the documentation. +To learn how to get data for the pairs and exchange you're interrested in, head over to the [Data Downloading](data-download.md) section of the documentation. ## Test your strategy with Backtesting @@ -164,13 +164,13 @@ here: The bot has made `429` trades for an average duration of `4:12:00`, with a performance of `76.20%` (profit), that means it has earned a total of `0.00762792 BTC` starting with a capital of 0.01 BTC. -The column `avg profit %` shows the average profit for all trades made while the column `cum profit %` sums all the profits/losses. +The column `avg profit %` shows the average profit for all trades made while the column `cum profit %` sums up all the profits/losses. The column `tot profit %` shows instead the total profit % in relation to allocated capital (`max_open_trades * stake_amount`). -In the above results we have `max_open_trades=2 stake_amount=0.005` in config so `(76.20/100) * (0.005 * 2) =~ 0.00762792 BTC`. +In the above results we have `max_open_trades=2` and `stake_amount=0.005` in config so `tot_profit %` will be `(76.20/100) * (0.005 * 2) =~ 0.00762792 BTC`. Your strategy performance is influenced by your buy strategy, your sell strategy, and also by the `minimal_roi` and `stop_loss` you have set. -For example, if your minimal_roi is only `"0": 0.01`. You cannot expect the bot to make more profit than 1% (because it will sell every time a trade will reach 1%). +For example, if your `minimal_roi` is only `"0": 0.01` you cannot expect the bot to make more profit than 1% (because it will sell every time a trade reaches 1%). ```json "minimal_roi": { @@ -179,8 +179,8 @@ For example, if your minimal_roi is only `"0": 0.01`. You cannot expect the bot ``` On the other hand, if you set a too high `minimal_roi` like `"0": 0.55` -(55%), there is almost no chance that the bot will never reach this profit. -Hence, keep in mind that your performance is a mix of the different elements of the strategy, your configuration, and the crypto-currency pairs you have set up. +(55%), there is almost no chance that the bot will ever reach this profit. +Hence, keep in mind that your performance is an integral mix of all different elements of the strategy, your configuration, and the crypto-currency pairs you have set up. ### Assumptions made by backtesting diff --git a/docs/hyperopt.md b/docs/hyperopt.md index 00aec458c..9542d5725 100644 --- a/docs/hyperopt.md +++ b/docs/hyperopt.md @@ -7,7 +7,7 @@ search will burn all your CPU cores, make your laptop sound like a fighter jet and still take a long time. Hyperopt requires historic data to be available, just as backtesting does. -To learn how to get data for the pairs and exchange you're interrested in, head over to the [Data downloading](data-download.md) section of the documentation. +To learn how to get data for the pairs and exchange you're interrested in, head over to the [Data Downloading](data-download.md) section of the documentation. !!! Bug Hyperopt will crash when used with only 1 CPU Core as found out in [Issue #1133](https://github.com/freqtrade/freqtrade/issues/1133) diff --git a/mkdocs.yml b/mkdocs.yml index 069f060db..7aedb4bba 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -11,7 +11,7 @@ nav: - Telegram: telegram-usage.md - Web Hook: webhook-config.md - REST API: rest-api.md - - Data Download: data-download.md + - Data Downloading: data-download.md - Backtesting: backtesting.md - Hyperopt: hyperopt.md - Edge positioning: edge.md From 6aa1ec2a4ce054f174c668f00b624257913be1ac Mon Sep 17 00:00:00 2001 From: Matthias Date: Tue, 24 Sep 2019 07:05:30 +0200 Subject: [PATCH 3/6] Some small restructuring --- docs/backtesting.md | 8 ++++---- docs/data-download.md | 17 ++++++++++++----- 2 files changed, 16 insertions(+), 9 deletions(-) diff --git a/docs/backtesting.md b/docs/backtesting.md index 9ddc6bbb3..24df545fd 100644 --- a/docs/backtesting.md +++ b/docs/backtesting.md @@ -3,7 +3,7 @@ This page explains how to validate your strategy performance by using Backtesting. Backtesting requires historic data to be available. -To learn how to get data for the pairs and exchange you're interrested in, head over to the [Data Downloading](data-download.md) section of the documentation. +To learn how to get data for the pairs and exchange you're interested in, head over to the [Data Downloading](data-download.md) section of the documentation. ## Test your strategy with Backtesting @@ -15,7 +15,7 @@ Backtesting will use the crypto-currencies (pairs) from your config file and load ticker data from `user_data/data/` by default. If no data is available for the exchange / pair / ticker interval combination, backtesting will ask you to download them first using `freqtrade download-data`. -For details on downloading, please refer to the [Downloading data](data-download.md) section in the documentation. +For details on downloading, please refer to the [Data Downloading](data-download.md) section in the documentation. The result of backtesting will confirm if your bot has better odds of making a profit than a loss. @@ -84,8 +84,8 @@ freqtrade backtesting --timerange=20190501- #### Advanced use of timerange -Doing `--timerange=20190101-` will all available data starting with January 1st, 2019 from your inputdata. -You can also specify specific dates, or a range span indexed by start and stop. +Using `--timerange=20190101-` will use all available data starting with January 1st, 2019 from your inputdata. +You can also specify particular dates or a range span indexed by start and stop. The full timerange specification: diff --git a/docs/data-download.md b/docs/data-download.md index f6b1e9904..b576e5c01 100644 --- a/docs/data-download.md +++ b/docs/data-download.md @@ -4,21 +4,26 @@ To download data (candles / OHLCV) needed for backtesting and hyperoptimization use the `freqtrade download-data` command. -If no additional parameter is specified, freqtrade will download data for `"1m"` and `"5m"` timeframes. -Exchange and pairs will come from `config.json` (if specified using `-c/--config`). Otherwise `--exchange` becomes mandatory. +If no additional parameter is specified, freqtrade will download data for `"1m"` and `"5m"` timeframes for the last 30 days. +Exchange and pairs will come from `config.json` (if specified using `-c/--config`). +Otherwise `--exchange` becomes mandatory. -Alternatively, a `pairs.json` file can be used. +### Pairs file + +In alternative to the whitelist from `config.json`, a `pairs.json` file can be used. If you are using Binance for example: -- create a directory `user_data/data/binance` and copy `pairs.json` in that directory. -- update the `pairs.json` to contain the currency pairs you are interested in. +- create a directory `user_data/data/binance` and copy or create the `pairs.json` file in that directory. +- update the `pairs.json` file to contain the currency pairs you are interested in. ```bash mkdir -p user_data/data/binance cp freqtrade/tests/testdata/pairs.json user_data/data/binance ``` +### start download + Then run: ```bash @@ -27,6 +32,8 @@ freqtrade download-data --exchange binance This will download ticker data for all the currency pairs you defined in `pairs.json`. +### Other Notes + - To use a different directory than the exchange specific default, use `--datadir user_data/data/some_directory`. - To change the exchange used to download the tickers, please use a different configuration file (you'll probably need to adjust ratelimits etc.) - To use `pairs.json` from some other directory, use `--pairs-file some_other_dir/pairs.json`. From 93b262165161089583d2ab02a5bb37bab464d995 Mon Sep 17 00:00:00 2001 From: Matthias Date: Tue, 24 Sep 2019 07:07:06 +0200 Subject: [PATCH 4/6] Add format description for pairs.json file --- docs/data-download.md | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/docs/data-download.md b/docs/data-download.md index b576e5c01..17565bb98 100644 --- a/docs/data-download.md +++ b/docs/data-download.md @@ -22,6 +22,18 @@ mkdir -p user_data/data/binance cp freqtrade/tests/testdata/pairs.json user_data/data/binance ``` +The format of the `pairs.json` file is a simple json list. +Mixing different stake-currencies is allowed for this file, since it's only used for downloading. + +``` json +[ + "ETH/BTC", + "ETH/USDT", + "BTC/USDT", + "XRP/ETH" +] +``` + ### start download Then run: From cc91ccad3ee39ec0a501878132219492df332a25 Mon Sep 17 00:00:00 2001 From: Matthias Date: Wed, 25 Sep 2019 06:26:28 +0200 Subject: [PATCH 5/6] Improve documentation wording --- docs/backtesting.md | 2 +- docs/data-download.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/backtesting.md b/docs/backtesting.md index 24df545fd..041a47ba2 100644 --- a/docs/backtesting.md +++ b/docs/backtesting.md @@ -84,7 +84,7 @@ freqtrade backtesting --timerange=20190501- #### Advanced use of timerange -Using `--timerange=20190101-` will use all available data starting with January 1st, 2019 from your inputdata. +For Example, running backtesting with the `--timerange=20190101-` option will use all available data starting with January 1st, 2019 from your inputdata. You can also specify particular dates or a range span indexed by start and stop. The full timerange specification: diff --git a/docs/data-download.md b/docs/data-download.md index 17565bb98..c370b3919 100644 --- a/docs/data-download.md +++ b/docs/data-download.md @@ -1,4 +1,4 @@ -# Data download +# Data Downloading ## Getting data for backtesting and hyperopt From 60e3e626e4744172469a576dceec497086f7820a Mon Sep 17 00:00:00 2001 From: Matthias Date: Thu, 26 Sep 2019 11:00:26 +0200 Subject: [PATCH 6/6] Improve timerange section of the docs --- docs/backtesting.md | 9 +++------ 1 file changed, 3 insertions(+), 6 deletions(-) diff --git a/docs/backtesting.md b/docs/backtesting.md index 041a47ba2..75aba6c73 100644 --- a/docs/backtesting.md +++ b/docs/backtesting.md @@ -72,24 +72,21 @@ The exported trades can be used for [further analysis](#further-backtest-result- freqtrade backtesting --export trades --export-filename=backtest_samplestrategy.json ``` -#### Running backtest with smaller testset +#### Running backtest with smaller testset by using timerange Use the `--timerange` argument to change how much of the testset you want to use. -Example: + +For example, running backtesting with the `--timerange=20190501-` option will use all available data starting with May 1st, 2019 from your inputdata. ```bash freqtrade backtesting --timerange=20190501- ``` -#### Advanced use of timerange - -For Example, running backtesting with the `--timerange=20190101-` option will use all available data starting with January 1st, 2019 from your inputdata. You can also specify particular dates or a range span indexed by start and stop. The full timerange specification: - - Use tickframes till 2018/01/31: `--timerange=-20180131` - Use tickframes since 2018/01/31: `--timerange=20180131-` - Use tickframes since 2018/01/31 till 2018/03/01 : `--timerange=20180131-20180301`