adelorenzo/stable
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Merge pull request #2146 from freqtrade/download_module

Browse Source 
Download module
This commit is contained in:
Matthias 2019-08-20 06:59:30 +02:00 committed by GitHub
commit af51ff4162
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
16 changed files with 416 additions and 219 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

67
docs/backtesting.md
View File

@ -3,9 +3,43 @@
This page explains how to validate your strategy performance by using
Backtesting.
## Getting data for backtesting and hyperopt
To download backtesting data (candles / OHLCV) and hyperoptimization using the `freqtrade download-data` command.
If no additional parameter is specified, freqtrade will download data for `"1m"` and `"5m"` timeframes.
Exchange and pairs will come from `config.json` (if specified using `-c/--config`). Otherwise `--exchange` becomes mandatory.
Alternatively, a `pairs.json` file can be used.
If you are using Binance for example:
- create a directory `user_data/data/binance` and copy `pairs.json` in that directory.
- update the `pairs.json` to contain the currency pairs you are interested in.
```bash
mkdir -p user_data/data/binance
cp freqtrade/tests/testdata/pairs.json user_data/data/binance
```
Then run:
```bash
freqtrade download-data --exchange binance
```
This will download ticker data for all the currency pairs you defined in `pairs.json`.
- To use a different directory than the exchange specific default, use `--datadir user_data/data/some_directory`.
- To change the exchange used to download the tickers, please use a different configuration file (you'll probably need to adjust ratelimits etc.)
- To use `pairs.json` from some other directory, use `--pairs-file some_other_dir/pairs.json`.
- To download ticker data for only 10 days, use `--days 10` (defaults to 30 days).
- Use `--timeframes` to specify which tickers to download. Default is `--timeframes 1m 5m` which will download 1-minute and 5-minute tickers.
- To use exchange, timeframe and list of pairs as defined in your configuration file, use the `-c/--config` option. With this, the script uses the whitelist defined in the config as the list of currency pairs to download data for and does not require the pairs.json file. You can combine `-c/--config` with most other options.
## Test your strategy with Backtesting
Now you have good Buy and Sell strategies, you want to test it against
Now you have good Buy and Sell strategies and some historic data, you want to test it against
real data. This is what we call
[backtesting](https://en.wikipedia.org/wiki/Backtesting).
@ -109,37 +143,6 @@ The full timerange specification:
- Use tickframes between POSIX timestamps 1527595200 1527618600:
`--timerange=1527595200-1527618600`
#### Downloading new set of ticker data
To download new set of backtesting ticker data, you can use a download script.
If you are using Binance for example:
- create a directory `user_data/data/binance` and copy `pairs.json` in that directory.
- update the `pairs.json` to contain the currency pairs you are interested in.
```bash
mkdir -p user_data/data/binance
cp freqtrade/tests/testdata/pairs.json user_data/data/binance
```
Then run:
```bash
python scripts/download_backtest_data.py --exchange binance
```
This will download ticker data for all the currency pairs you defined in `pairs.json`.
- To use a different directory than the exchange specific default, use `--datadir user_data/data/some_directory`.
- To change the exchange used to download the tickers, use `--exchange`. Default is `bittrex`.
- To use `pairs.json` from some other directory, use `--pairs-file some_other_dir/pairs.json`.
- To download ticker data for only 10 days, use `--days 10`.
- Use `--timeframes` to specify which tickers to download. Default is `--timeframes 1m 5m` which will download 1-minute and 5-minute tickers.
- To use exchange, timeframe and list of pairs as defined in your configuration file, use the `-c/--config` option. With this, the script uses the whitelist defined in the config as the list of currency pairs to download data for and does not require the pairs.json file. You can combine `-c/--config` with other options.
For help about backtesting usage, please refer to [Backtesting commands](#backtesting-commands).
## Understand the backtesting result
The most important in the backtesting is to understand the result.

16
docs/bot-usage.md
View File

@ -184,19 +184,11 @@ optional arguments:
result.json)
```
### How to use **--refresh-pairs-cached** parameter?
### Getting historic data for backtesting
The first time your run Backtesting, it will take the pairs you have
set in your config file and download data from the Exchange.
If for any reason you want to update your data set, you use
`--refresh-pairs-cached` to force Backtesting to update the data it has.
!!! Note
Use it only if you want to update your data set. You will not be able to come back to the previous version.
To test your strategy with latest data, we recommend continuing using
the parameter `-l` or `--live`.
The first time your run Backtesting, you will need to download some historic data first.
This can be accomplished by using `freqtrade download-data`.
Check the corresponding [help page section](backtesting.md#Getting-data-for-backtesting-and-hyperopt) for more details
## Hyperopt commands

19
freqtrade/configuration/arguments.py
View File

@ -30,7 +30,7 @@ ARGS_EDGE = ARGS_COMMON_OPTIMIZE + ["stoploss_range"]
ARGS_LIST_EXCHANGES = ["print_one_column"]
ARGS_DOWNLOADER = ARGS_COMMON + ["pairs", "pairs_file", "days", "exchange", "timeframes", "erase"]
ARGS_DOWNLOAD_DATA = ["pairs", "pairs_file", "days", "exchange", "timeframes", "erase"]
ARGS_PLOT_DATAFRAME = (ARGS_COMMON + ARGS_STRATEGY +
["pairs", "indicators1", "indicators2", "plot_limit", "db_url",
@ -40,6 +40,8 @@ ARGS_PLOT_DATAFRAME = (ARGS_COMMON + ARGS_STRATEGY +
ARGS_PLOT_PROFIT = (ARGS_COMMON + ARGS_STRATEGY +
["pairs", "timerange", "export", "exportfilename", "db_url", "trade_source"])
NO_CONF_REQURIED = ["start_download_data"]
class Arguments(object):
"""
@ -75,7 +77,10 @@ class Arguments(object):
# Workaround issue in argparse with action='append' and default value
# (see https://bugs.python.org/issue16399)
if not self._no_default_config and parsed_arg.config is None:
# Allow no-config for certain commands (like downloading / plotting)
if (not self._no_default_config and parsed_arg.config is None
and not (hasattr(parsed_arg, 'func')
and parsed_arg.func.__name__ in NO_CONF_REQURIED)):
parsed_arg.config = [constants.DEFAULT_CONFIG]
return parsed_arg
@ -93,7 +98,7 @@ class Arguments(object):
:return: None
"""
from freqtrade.optimize import start_backtesting, start_hyperopt, start_edge
from freqtrade.utils import start_list_exchanges
from freqtrade.utils import start_download_data, start_list_exchanges
subparsers = self.parser.add_subparsers(dest='subparser')
@ -119,3 +124,11 @@ class Arguments(object):
)
list_exchanges_cmd.set_defaults(func=start_list_exchanges)
self._build_args(optionlist=ARGS_LIST_EXCHANGES, parser=list_exchanges_cmd)
# Add download-data subcommand
download_data_cmd = subparsers.add_parser(
'download-data',
help='Download backtesting data.'
)
download_data_cmd.set_defaults(func=start_download_data)
self._build_args(optionlist=ARGS_DOWNLOAD_DATA, parser=download_data_cmd)

6
freqtrade/configuration/cli_options.py
View File

@ -254,7 +254,8 @@ AVAILABLE_CLI_OPTIONS = {
# Script options
"pairs": Arg(
'-p', '--pairs',
help='Show profits for only these pairs. Pairs are comma-separated.',
help='Show profits for only these pairs. Pairs are space-separated.',
nargs='+',
),
# Download data
"pairs_file": Arg(
@ -276,9 +277,10 @@ AVAILABLE_CLI_OPTIONS = {
"timeframes": Arg(
'-t', '--timeframes',
help=f'Specify which tickers to download. Space-separated list. '
f'Default: `{constants.DEFAULT_DOWNLOAD_TICKER_INTERVALS}`.',
f'Default: `1m 5m`.',
choices=['1m', '3m', '5m', '15m', '30m', '1h', '2h', '4h',
'6h', '8h', '12h', '1d', '3d', '1w'],
default=['1m', '5m'],
nargs='+',
),
"erase": Arg(

64
freqtrade/configuration/configuration.py
View File

@ -4,16 +4,17 @@ This module contains the configuration class
import logging
import warnings
from argparse import Namespace
from pathlib import Path
from typing import Any, Callable, Dict, List, Optional
from freqtrade import constants
from freqtrade import constants, OperationalException
from freqtrade.configuration.check_exchange import check_exchange
from freqtrade.configuration.create_datadir import create_datadir
from freqtrade.configuration.config_validation import (validate_config_schema,
validate_config_consistency)
from freqtrade.configuration.load_config import load_config_file
from freqtrade.loggers import setup_logging
from freqtrade.misc import deep_merge_dicts
from freqtrade.misc import deep_merge_dicts, json_load
from freqtrade.state import RunMode
logger = logging.getLogger(__name__)
@ -53,6 +54,9 @@ class Configuration(object):
# Keep this method as staticmethod, so it can be used from interactive environments
config: Dict[str, Any] = {}
if not files:
return constants.MINIMAL_CONFIG.copy()
# We expect here a list of config filenames
for path in files:
logger.info(f'Using config: {path} ...')
@ -86,6 +90,11 @@ class Configuration(object):
self._process_runmode(config)
# Check if the exchange set by the user is supported
check_exchange(config, config.get('experimental', {}).get('block_bad_exchanges', True))
self._resolve_pairs_list(config)
validate_config_consistency(config)
return config
@ -148,9 +157,6 @@ class Configuration(object):
if 'sd_notify' in self.args and self.args.sd_notify:
config['internals'].update({'sd_notify': True})
# Check if the exchange set by the user is supported
check_exchange(config, config.get('experimental', {}).get('block_bad_exchanges', True))
def _process_datadir_options(self, config: Dict[str, Any]) -> None:
"""
Extract information for sys.argv and load datadir configuration:
@ -277,6 +283,19 @@ class Configuration(object):
self._args_to_config(config, argname='trade_source',
logstring='Using trades from: {}')
self._args_to_config(config, argname='erase',
logstring='Erase detected. Deleting existing data.')
self._args_to_config(config, argname='timeframes',
logstring='timeframes --timeframes: {}')
self._args_to_config(config, argname='days',
logstring='Detected --days: {}')
if "exchange" in self.args and self.args.exchange:
config['exchange']['name'] = self.args.exchange
logger.info(f"Using exchange {config['exchange']['name']}")
def _process_runmode(self, config: Dict[str, Any]) -> None:
if not self.runmode:
@ -307,3 +326,38 @@ class Configuration(object):
logger.info(logstring.format(config[argname]))
if deprecated_msg:
warnings.warn(f"DEPRECATED: {deprecated_msg}", DeprecationWarning)
def _resolve_pairs_list(self, config: Dict[str, Any]) -> None:
"""
Helper for download script.
Takes first found:
* -p (pairs argument)
* --pairs-file
* whitelist from config
"""
if "pairs" in config:
return
if "pairs_file" in self.args and self.args.pairs_file:
pairs_file = Path(self.args.pairs_file)
logger.info(f'Reading pairs file "{pairs_file}".')
# Download pairs from the pairs file if no config is specified
# or if pairs file is specified explicitely
if not pairs_file.exists():
raise OperationalException(f'No pairs file found with path "{pairs_file}".')
config['pairs'] = json_load(pairs_file)
config['pairs'].sort()
return
if "config" in self.args and self.args.config:
logger.info("Using pairlist from configuration.")
config['pairs'] = config.get('exchange', {}).get('pair_whitelist')
else:
# Fall back to /dl_path/pairs.json
pairs_file = Path(config['datadir']) / "pairs.json"
if pairs_file.exists():
config['pairs'] = json_load(pairs_file)
config['pairs'].sort()

15
freqtrade/constants.py
View File

@ -22,7 +22,6 @@ ORDERTYPE_POSSIBILITIES = ['limit', 'market']
ORDERTIF_POSSIBILITIES = ['gtc', 'fok', 'ioc']
AVAILABLE_PAIRLISTS = ['StaticPairList', 'VolumePairList']
DRY_RUN_WALLET = 999.9
DEFAULT_DOWNLOAD_TICKER_INTERVALS = '1m 5m'
TICKER_INTERVALS = [
'1m', '3m', '5m', '15m', '30m',
@ -38,6 +37,20 @@ SUPPORTED_FIAT = [
"BTC", "XBT", "ETH", "XRP", "LTC", "BCH", "USDT"
]
MINIMAL_CONFIG = {
'stake_currency': '',
'dry_run': True,
'exchange': {
'name': '',
'key': '',
'secret': '',
'pair_whitelist': [],
'ccxt_async_config': {
'enableRateLimit': True,
}
}
}
# Required json-schema for user specified config
CONF_SCHEMA = {
'type': 'object',

2
freqtrade/data/history.py
View File

@ -129,7 +129,7 @@ def load_pair_history(pair: str,
else:
logger.warning(
f'No history data for pair: "{pair}", interval: {ticker_interval}. '
'Use --refresh-pairs-cached option or download_backtest_data.py '
'Use --refresh-pairs-cached option or `freqtrade download-data` '
'script to download the data'
)
return None

2
freqtrade/plot/plotting.py
View File

@ -37,7 +37,7 @@ def init_plotscript(config):
strategy = StrategyResolver(config).strategy
if "pairs" in config:
pairs = config["pairs"].split(',')
pairs = config["pairs"]
else:
pairs = config["exchange"]["pair_whitelist"]

4
freqtrade/tests/data/test_history.py
View File

@ -74,7 +74,7 @@ def test_load_data_7min_ticker(mocker, caplog, default_conf) -> None:
assert ld is None
assert log_has(
'No history data for pair: "UNITTEST/BTC", interval: 7m. '
'Use --refresh-pairs-cached option or download_backtest_data.py '
'Use --refresh-pairs-cached option or `freqtrade download-data` '
'script to download the data', caplog
)
@ -109,7 +109,7 @@ def test_load_data_with_new_pair_1min(ticker_history_list, mocker, caplog, defau
assert os.path.isfile(file) is False
assert log_has(
'No history data for pair: "MEME/BTC", interval: 1m. '
'Use --refresh-pairs-cached option or download_backtest_data.py '
'Use --refresh-pairs-cached option or `freqtrade download-data` '
'script to download the data', caplog
)

20
freqtrade/tests/test_arguments.py
View File

@ -4,7 +4,7 @@ import argparse
import pytest
from freqtrade.configuration import Arguments
from freqtrade.configuration.arguments import ARGS_DOWNLOADER, ARGS_PLOT_DATAFRAME
from freqtrade.configuration.arguments import ARGS_PLOT_DATAFRAME
from freqtrade.configuration.cli_options import check_int_positive
@ -50,10 +50,10 @@ def test_parse_args_verbose() -> None:
def test_common_scripts_options() -> None:
arguments = Arguments(['-p', 'ETH/BTC'], '')
arguments._build_args(ARGS_DOWNLOADER)
args = arguments._parse_args()
assert args.pairs == 'ETH/BTC'
args = Arguments(['download-data', '-p', 'ETH/BTC', 'XRP/BTC'], '').get_parsed_arg()
assert args.pairs == ['ETH/BTC', 'XRP/BTC']
assert hasattr(args, "func")
def test_parse_args_version() -> None:
@ -135,14 +135,14 @@ def test_parse_args_hyperopt_custom() -> None:
def test_download_data_options() -> None:
args = [
'--pairs-file', 'file_with_pairs',
'--datadir', 'datadir/directory',
'download-data',
'--pairs-file', 'file_with_pairs',
'--days', '30',
'--exchange', 'binance'
]
arguments = Arguments(args, '')
arguments._build_args(ARGS_DOWNLOADER)
args = arguments._parse_args()
args = Arguments(args, '').get_parsed_arg()
assert args.pairs_file == 'file_with_pairs'
assert args.datadir == 'datadir/directory'
assert args.days == 30
@ -162,7 +162,7 @@ def test_plot_dataframe_options() -> None:
assert pargs.indicators1 == "sma10,sma100"
assert pargs.indicators2 == "macd,fastd,fastk"
assert pargs.plot_limit == 30
assert pargs.pairs == "UNITTEST/BTC"
assert pargs.pairs == ["UNITTEST/BTC"]
def test_check_int_positive() -> None:

106
freqtrade/tests/test_configuration.py
View File

@ -716,3 +716,109 @@ def test_load_config_default_subkeys(all_conf, keys) -> None:
validate_config_schema(all_conf)
assert subkey in all_conf[key]
assert all_conf[key][subkey] == keys[2]
def test_pairlist_resolving():
arglist = [
'download-data',
'--pairs', 'ETH/BTC', 'XRP/BTC',
'--exchange', 'binance'
]
args = Arguments(arglist, '').get_parsed_arg()
configuration = Configuration(args)
config = configuration.get_config()
assert config['pairs'] == ['ETH/BTC', 'XRP/BTC']
assert config['exchange']['name'] == 'binance'
def test_pairlist_resolving_with_config(mocker, default_conf):
patched_configuration_load_config_file(mocker, default_conf)
arglist = [
'--config', 'config.json',
'download-data',
]
args = Arguments(arglist, '').get_parsed_arg()
configuration = Configuration(args)
config = configuration.get_config()
assert config['pairs'] == default_conf['exchange']['pair_whitelist']
assert config['exchange']['name'] == default_conf['exchange']['name']
# Override pairs
arglist = [
'--config', 'config.json',
'download-data',
'--pairs', 'ETH/BTC', 'XRP/BTC',
]
args = Arguments(arglist, '').get_parsed_arg()
configuration = Configuration(args)
config = configuration.get_config()
assert config['pairs'] == ['ETH/BTC', 'XRP/BTC']
assert config['exchange']['name'] == default_conf['exchange']['name']
def test_pairlist_resolving_with_config_pl(mocker, default_conf):
patched_configuration_load_config_file(mocker, default_conf)
load_mock = mocker.patch("freqtrade.configuration.configuration.json_load",
MagicMock(return_value=['XRP/BTC', 'ETH/BTC']))
mocker.patch.object(Path, "exists", MagicMock(return_value=True))
arglist = [
'--config', 'config.json',
'download-data',
'--pairs-file', 'pairs.json',
]
args = Arguments(arglist, '').get_parsed_arg()
configuration = Configuration(args)
config = configuration.get_config()
assert load_mock.call_count == 1
assert config['pairs'] == ['ETH/BTC', 'XRP/BTC']
assert config['exchange']['name'] == default_conf['exchange']['name']
def test_pairlist_resolving_with_config_pl_not_exists(mocker, default_conf):
patched_configuration_load_config_file(mocker, default_conf)
mocker.patch("freqtrade.configuration.configuration.json_load",
MagicMock(return_value=['XRP/BTC', 'ETH/BTC']))
mocker.patch.object(Path, "exists", MagicMock(return_value=False))
arglist = [
'--config', 'config.json',
'download-data',
'--pairs-file', 'pairs.json',
]
args = Arguments(arglist, '').get_parsed_arg()
with pytest.raises(OperationalException, match=r"No pairs file found with path.*"):
configuration = Configuration(args)
configuration.get_config()
def test_pairlist_resolving_fallback(mocker):
mocker.patch.object(Path, "exists", MagicMock(return_value=True))
mocker.patch("freqtrade.configuration.configuration.json_load",
MagicMock(return_value=['XRP/BTC', 'ETH/BTC']))
arglist = [
'download-data',
'--exchange', 'binance'
]
args = Arguments(arglist, '').get_parsed_arg()
configuration = Configuration(args)
config = configuration.get_config()
assert config['pairs'] == ['ETH/BTC', 'XRP/BTC']
assert config['exchange']['name'] == 'binance'

4
freqtrade/tests/test_main.py
View File

@ -1,7 +1,7 @@
# pragma pylint: disable=missing-docstring
from copy import deepcopy
from unittest.mock import MagicMock
from unittest.mock import MagicMock, PropertyMock
import pytest
@ -21,6 +21,7 @@ def test_parse_args_backtesting(mocker) -> None:
further argument parsing is done in test_arguments.py
"""
backtesting_mock = mocker.patch('freqtrade.optimize.start_backtesting', MagicMock())
backtesting_mock.__name__ = PropertyMock("start_backtesting")
# it's sys.exit(0) at the end of backtesting
with pytest.raises(SystemExit):
main(['backtesting'])
@ -36,6 +37,7 @@ def test_parse_args_backtesting(mocker) -> None:
def test_main_start_hyperopt(mocker) -> None:
hyperopt_mock = mocker.patch('freqtrade.optimize.start_hyperopt', MagicMock())
hyperopt_mock.__name__ = PropertyMock("start_hyperopt")
# it's sys.exit(0) at the end of hyperopt
with pytest.raises(SystemExit):
main(['hyperopt'])

2
freqtrade/tests/test_plotting.py
View File

@ -50,7 +50,7 @@ def test_init_plotscript(default_conf, mocker):
assert "pairs" in ret
assert "strategy" in ret
default_conf['pairs'] = "POWR/BTC,XLM/BTC"
default_conf['pairs'] = ["POWR/BTC", "XLM/BTC"]
ret = init_plotscript(default_conf)
assert "tickers" in ret
assert "POWR/BTC" in ret["tickers"]

97
freqtrade/tests/test_utils.py
View File

@ -1,8 +1,13 @@
from freqtrade.utils import setup_utils_configuration, start_list_exchanges
from freqtrade.tests.conftest import get_args
from freqtrade.state import RunMode
import re
from pathlib import Path
from unittest.mock import MagicMock, PropertyMock
import pytest
from freqtrade.state import RunMode
from freqtrade.tests.conftest import get_args, log_has, patch_exchange
from freqtrade.utils import (setup_utils_configuration, start_download_data,
start_list_exchanges)
def test_setup_utils_configuration():
@ -40,3 +45,87 @@ def test_list_exchanges(capsys):
assert not re.match(r"Exchanges supported by ccxt and available.*", captured.out)
assert re.search(r"^binance$", captured.out, re.MULTILINE)
assert re.search(r"^bittrex$", captured.out, re.MULTILINE)
def test_download_data(mocker, markets, caplog):
dl_mock = mocker.patch('freqtrade.utils.download_pair_history', MagicMock())
patch_exchange(mocker)
mocker.patch(
'freqtrade.exchange.Exchange.markets', PropertyMock(return_value=markets)
)
mocker.patch.object(Path, "exists", MagicMock(return_value=True))
mocker.patch.object(Path, "unlink", MagicMock())
args = [
"download-data",
"--exchange", "binance",
"--pairs", "ETH/BTC", "XRP/BTC",
"--erase",
]
start_download_data(get_args(args))
assert dl_mock.call_count == 4
assert dl_mock.call_args[1]['timerange'].starttype is None
assert dl_mock.call_args[1]['timerange'].stoptype is None
assert log_has("Deleting existing data for pair ETH/BTC, interval 1m.", caplog)
assert log_has("Downloading pair ETH/BTC, interval 1m.", caplog)
def test_download_data_days(mocker, markets, caplog):
dl_mock = mocker.patch('freqtrade.utils.download_pair_history', MagicMock())
patch_exchange(mocker)
mocker.patch(
'freqtrade.exchange.Exchange.markets', PropertyMock(return_value=markets)
)
mocker.patch.object(Path, "exists", MagicMock(return_value=True))
mocker.patch.object(Path, "unlink", MagicMock())
args = [
"download-data",
"--exchange", "binance",
"--pairs", "ETH/BTC", "XRP/BTC",
"--days", "20",
]
start_download_data(get_args(args))
assert dl_mock.call_count == 4
assert dl_mock.call_args[1]['timerange'].starttype == 'date'
assert log_has("Downloading pair ETH/BTC, interval 1m.", caplog)
def test_download_data_no_markets(mocker, caplog):
dl_mock = mocker.patch('freqtrade.utils.download_pair_history', MagicMock())
patch_exchange(mocker)
mocker.patch(
'freqtrade.exchange.Exchange.markets', PropertyMock(return_value={})
)
args = [
"download-data",
"--exchange", "binance",
"--pairs", "ETH/BTC", "XRP/BTC",
]
start_download_data(get_args(args))
assert dl_mock.call_count == 0
assert log_has("Skipping pair ETH/BTC...", caplog)
assert log_has("Pairs [ETH/BTC,XRP/BTC] not available on exchange binance.", caplog)
def test_download_data_keyboardInterrupt(mocker, caplog, markets):
dl_mock = mocker.patch('freqtrade.utils.download_pair_history',
MagicMock(side_effect=KeyboardInterrupt))
patch_exchange(mocker)
mocker.patch(
'freqtrade.exchange.Exchange.markets', PropertyMock(return_value=markets)
)
args = [
"download-data",
"--exchange", "binance",
"--pairs", "ETH/BTC", "XRP/BTC",
]
with pytest.raises(SystemExit):
start_download_data(get_args(args))
assert dl_mock.call_count == 1

66
freqtrade/utils.py
View File

@ -1,11 +1,16 @@
import logging
import sys
from argparse import Namespace
from pathlib import Path
from typing import Any, Dict
from freqtrade.configuration import Configuration
from freqtrade.exchange import available_exchanges
from freqtrade.state import RunMode
import arrow
from freqtrade.configuration import Configuration, TimeRange
from freqtrade.data.history import download_pair_history
from freqtrade.exchange import available_exchanges
from freqtrade.resolvers import ExchangeResolver
from freqtrade.state import RunMode
logger = logging.getLogger(__name__)
@ -17,7 +22,7 @@ def setup_utils_configuration(args: Namespace, method: RunMode) -> Dict[str, Any
:return: Configuration
"""
configuration = Configuration(args, method)
config = configuration.load_config()
config = configuration.get_config()
config['exchange']['dry_run'] = True
# Ensure we do not use Exchange credentials
@ -39,3 +44,56 @@ def start_list_exchanges(args: Namespace) -> None:
else:
print(f"Exchanges supported by ccxt and available for Freqtrade: "
f"{', '.join(available_exchanges())}")
def start_download_data(args: Namespace) -> None:
"""
Download data (former download_backtest_data.py script)
"""
config = setup_utils_configuration(args, RunMode.OTHER)
timerange = TimeRange()
if 'days' in config:
time_since = arrow.utcnow().shift(days=-config['days']).strftime("%Y%m%d")
timerange = TimeRange.parse_timerange(f'{time_since}-')
dl_path = Path(config['datadir'])
logger.info(f'About to download pairs: {config["pairs"]}, '
f'intervals: {config["timeframes"]} to {dl_path}')
pairs_not_available = []
try:
# Init exchange
exchange = ExchangeResolver(config['exchange']['name'], config).exchange
for pair in config["pairs"]:
if pair not in exchange.markets:
pairs_not_available.append(pair)
logger.info(f"Skipping pair {pair}...")
continue
for ticker_interval in config["timeframes"]:
pair_print = pair.replace('/', '_')
filename = f'{pair_print}-{ticker_interval}.json'
dl_file = dl_path.joinpath(filename)
if config.get("erase") and dl_file.exists():
logger.info(
f'Deleting existing data for pair {pair}, interval {ticker_interval}.')
dl_file.unlink()
logger.info(f'Downloading pair {pair}, interval {ticker_interval}.')
download_pair_history(datadir=dl_path, exchange=exchange,
pair=pair, ticker_interval=str(ticker_interval),
timerange=timerange)
except KeyboardInterrupt:
sys.exit("SIGINT received, aborting ...")
finally:
if pairs_not_available:
logger.info(
f"Pairs [{','.join(pairs_not_available)}] not available "
f"on exchange {config['exchange']['name']}.")
# configuration.resolve_pairs_list()
print(config)

145
scripts/download_backtest_data.py
View File

@ -1,144 +1,9 @@
#!/usr/bin/env python3
"""
This script generates json files with pairs history data
"""
import arrow
import json
import sys
from pathlib import Path
from typing import Any, Dict, List
from freqtrade.configuration import Arguments, TimeRange
from freqtrade.configuration import Configuration
from freqtrade.configuration.arguments import ARGS_DOWNLOADER
from freqtrade.configuration.check_exchange import check_exchange
from freqtrade.configuration.load_config import load_config_file
from freqtrade.data.history import download_pair_history
from freqtrade.exchange import Exchange
from freqtrade.misc import deep_merge_dicts
import logging
print("This script has been integrated into freqtrade "
"and it's functionality is available by calling `freqtrade download-data`.")
print("Please check the documentation on https://www.freqtrade.io/en/latest/backtesting/ "
"for details.")
logger = logging.getLogger('download_backtest_data')
DEFAULT_DL_PATH = 'user_data/data'
# Do not read the default config if config is not specified
# in the command line options explicitely
arguments = Arguments(sys.argv[1:], 'Download backtest data',
no_default_config=True)
arguments._build_args(optionlist=ARGS_DOWNLOADER)
args = arguments._parse_args()
# Use bittrex as default exchange
exchange_name = args.exchange or 'bittrex'
pairs: List = []
configuration = Configuration(args)
config: Dict[str, Any] = {}
if args.config:
# Now expecting a list of config filenames here, not a string
for path in args.config:
logger.info(f"Using config: {path}...")
# Merge config options, overwriting old values
config = deep_merge_dicts(load_config_file(path), config)
config['stake_currency'] = ''
# Ensure we do not use Exchange credentials
config['exchange']['dry_run'] = True
config['exchange']['key'] = ''
config['exchange']['secret'] = ''
pairs = config['exchange']['pair_whitelist']
if config.get('ticker_interval'):
timeframes = args.timeframes or [config.get('ticker_interval')]
else:
timeframes = args.timeframes or ['1m', '5m']
else:
config = {
'stake_currency': '',
'dry_run': True,
'exchange': {
'name': exchange_name,
'key': '',
'secret': '',
'pair_whitelist': [],
'ccxt_async_config': {
'enableRateLimit': True,
'rateLimit': 200
}
}
}
timeframes = args.timeframes or ['1m', '5m']
configuration._process_logging_options(config)
if args.config and args.exchange:
logger.warning("The --exchange option is ignored, "
"using exchange settings from the configuration file.")
# Check if the exchange set by the user is supported
check_exchange(config)
configuration._process_datadir_options(config)
dl_path = Path(config['datadir'])
pairs_file = Path(args.pairs_file) if args.pairs_file else dl_path.joinpath('pairs.json')
if not pairs or args.pairs_file:
logger.info(f'Reading pairs file "{pairs_file}".')
# Download pairs from the pairs file if no config is specified
# or if pairs file is specified explicitely
if not pairs_file.exists():
sys.exit(f'No pairs file found with path "{pairs_file}".')
with pairs_file.open() as file:
pairs = list(set(json.load(file)))
pairs.sort()
timerange = TimeRange()
if args.days:
time_since = arrow.utcnow().shift(days=-args.days).strftime("%Y%m%d")
timerange = TimeRange.parse_timerange(f'{time_since}-')
logger.info(f'About to download pairs: {pairs}, intervals: {timeframes} to {dl_path}')
pairs_not_available = []
try:
# Init exchange
exchange = Exchange(config)
for pair in pairs:
if pair not in exchange._api.markets:
pairs_not_available.append(pair)
logger.info(f"Skipping pair {pair}...")
continue
for ticker_interval in timeframes:
pair_print = pair.replace('/', '_')
filename = f'{pair_print}-{ticker_interval}.json'
dl_file = dl_path.joinpath(filename)
if args.erase and dl_file.exists():
logger.info(
f'Deleting existing data for pair {pair}, interval {ticker_interval}.')
dl_file.unlink()
logger.info(f'Downloading pair {pair}, interval {ticker_interval}.')
download_pair_history(datadir=dl_path, exchange=exchange,
pair=pair, ticker_interval=str(ticker_interval),
timerange=timerange)
except KeyboardInterrupt:
sys.exit("SIGINT received, aborting ...")
finally:
if pairs_not_available:
logger.info(
f"Pairs [{','.join(pairs_not_available)}] not available "
f"on exchange {config['exchange']['name']}.")
sys.exit(1)