Document expansion of --pairs, add download-inactive
This commit is contained in:
Matthias
parent
9bb2dd1851
commit
d4d57f0002
@ -22,6 +22,7 @@ usage: freqtrade download-data [-h] [-v] [--logfile FILE] [-V] [-c PATH]
|
||||
[-d PATH] [--userdir PATH]
|
||||
[-p PAIRS [PAIRS ...]] [--pairs-file FILE]
|
||||
[--days INT] [--new-pairs-days INT]
|
||||
[--include-inactive-pairs]
|
||||
[--timerange TIMERANGE] [--dl-trades]
|
||||
[--exchange EXCHANGE]
|
||||
[-t {1m,3m,5m,15m,30m,1h,2h,4h,6h,8h,12h,1d,3d,1w,2w,1M,1y} [{1m,3m,5m,15m,30m,1h,2h,4h,6h,8h,12h,1d,3d,1w,2w,1M,1y} ...]]
|
||||
@ -38,6 +39,8 @@ optional arguments:
|
||||
--days INT Download data for given number of days.
|
||||
--new-pairs-days INT Download data of new pairs for given number of days.
|
||||
Default: `None`.
|
||||
--include-inactive-pairs
|
||||
Also download data from inactive pairs.
|
||||
--timerange TIMERANGE
|
||||
Specify what timerange of data to use.
|
||||
--dl-trades Download trades instead of OHLCV data. The bot will
|
||||
@ -52,10 +55,10 @@ optional arguments:
|
||||
exchange/pairs/timeframes.
|
||||
--data-format-ohlcv {json,jsongz,hdf5}
|
||||
Storage format for downloaded candle (OHLCV) data.
|
||||
(default: `None`).
|
||||
(default: `json`).
|
||||
--data-format-trades {json,jsongz,hdf5}
|
||||
Storage format for downloaded trades data. (default:
|
||||
`None`).
|
||||
`jsongz`).
|
||||
|
||||
Common arguments:
|
||||
-v, --verbose Verbose mode (-vv for more, -vvv to get all messages).
|
||||
@ -80,6 +83,82 @@ Common arguments:
|
||||
|
||||
For that reason, `download-data` does not care about the "startup-period" defined in a strategy. It's up to the user to download additional days if the backtest should start at a specific point in time (while respecting startup period).
|
||||
|
||||
### Pairs file
|
||||
|
||||
In alternative to the whitelist from `config.json`, a `pairs.json` file can be used.
|
||||
If you are using Binance for example:
|
||||
|
||||
- create a directory `user_data/data/binance` and copy or create the `pairs.json` file in that directory.
|
||||
- update the `pairs.json` file to contain the currency pairs you are interested in.
|
||||
|
||||
```bash
|
||||
mkdir -p user_data/data/binance
|
||||
touch user_data/data/binance/pairs.json
|
||||
```
|
||||
|
||||
The format of the `pairs.json` file is a simple json list.
|
||||
Mixing different stake-currencies is allowed for this file, since it's only used for downloading.
|
||||
|
||||
``` json
|
||||
[
|
||||
"ETH/BTC",
|
||||
"ETH/USDT",
|
||||
"BTC/USDT",
|
||||
"XRP/ETH"
|
||||
]
|
||||
```
|
||||
|
||||
!!! Tip "Downloading all data for one quote currency"
|
||||
Often, you'll want to download data for all pairs of a specific quote-currency. In such cases, you can use the following shorthand:
|
||||
`freqtrade download-data --exchange binance --pairs .*/USDT <...>`. The provided "pairs" string will be expanded to contain all active pairs on the exchange.
|
||||
To also download data for inactive (delisted) pairs, add `--include-inactive-pairs` to the command.
|
||||
|
||||
??? Note "Permission denied errors"
|
||||
If your configuration directory `user_data` was made by docker, you may get the following error:
|
||||
|
||||
```
|
||||
cp: cannot create regular file 'user_data/data/binance/pairs.json': Permission denied
|
||||
```
|
||||
|
||||
You can fix the permissions of your user-data directory as follows:
|
||||
|
||||
```
|
||||
sudo chown -R $UID:$GID user_data
|
||||
```
|
||||
|
||||
### Start download
|
||||
|
||||
Then run:
|
||||
|
||||
```bash
|
||||
freqtrade download-data --exchange binance
|
||||
```
|
||||
|
||||
This will download historical candle (OHLCV) data for all the currency pairs you defined in `pairs.json`.
|
||||
|
||||
Alternatively, specify the pairs directly
|
||||
|
||||
```bash
|
||||
freqtrade download-data --exchange binance --pairs ETH/USDT XRP/USDT BTC/USDT
|
||||
```
|
||||
|
||||
or as regex (to download all active USDT pairs)
|
||||
|
||||
```bash
|
||||
freqtrade download-data --exchange binance --pairs .*/USDT
|
||||
```
|
||||
|
||||
### Other Notes
|
||||
|
||||
- To use a different directory than the exchange specific default, use `--datadir user_data/data/some_directory`.
|
||||
- To change the exchange used to download the historical data from, please use a different configuration file (you'll probably need to adjust rate limits etc.)
|
||||
- To use `pairs.json` from some other directory, use `--pairs-file some_other_dir/pairs.json`.
|
||||
- To download historical candle (OHLCV) data for only 10 days, use `--days 10` (defaults to 30 days).
|
||||
- To download historical candle (OHLCV) data from a fixed starting point, use `--timerange 20200101-` - which will download all data from January 1st, 2020. Eventually set end dates are ignored.
|
||||
- Use `--timeframes` to specify what timeframe download the historical candle (OHLCV) data for. Default is `--timeframes 1m 5m` which will download 1-minute and 5-minute data.
|
||||
- To use exchange, timeframe and list of pairs as defined in your configuration file, use the `-c/--config` option. With this, the script uses the whitelist defined in the config as the list of currency pairs to download data for and does not require the pairs.json file. You can combine `-c/--config` with most other options.
|
||||
|
||||
|
||||
### Data format
|
||||
|
||||
Freqtrade currently supports 3 data-formats for both OHLCV and trades data:
|
||||
@ -312,64 +391,6 @@ ETH/BTC 5m, 15m, 30m, 1h, 2h, 4h, 6h, 12h, 1d
|
||||
ETH/USDT 5m, 15m, 30m, 1h, 2h, 4h
|
||||
```
|
||||
|
||||
### Pairs file
|
||||
|
||||
In alternative to the whitelist from `config.json`, a `pairs.json` file can be used.
|
||||
|
||||
If you are using Binance for example:
|
||||
|
||||
- create a directory `user_data/data/binance` and copy or create the `pairs.json` file in that directory.
|
||||
- update the `pairs.json` file to contain the currency pairs you are interested in.
|
||||
|
||||
```bash
|
||||
mkdir -p user_data/data/binance
|
||||
cp tests/testdata/pairs.json user_data/data/binance
|
||||
```
|
||||
|
||||
If your configuration directory `user_data` was made by docker, you may get the following error:
|
||||
|
||||
```
|
||||
cp: cannot create regular file 'user_data/data/binance/pairs.json': Permission denied
|
||||
```
|
||||
|
||||
You can fix the permissions of your user-data directory as follows:
|
||||
|
||||
```
|
||||
sudo chown -R $UID:$GID user_data
|
||||
```
|
||||
|
||||
The format of the `pairs.json` file is a simple json list.
|
||||
Mixing different stake-currencies is allowed for this file, since it's only used for downloading.
|
||||
|
||||
``` json
|
||||
[
|
||||
"ETH/BTC",
|
||||
"ETH/USDT",
|
||||
"BTC/USDT",
|
||||
"XRP/ETH"
|
||||
]
|
||||
```
|
||||
|
||||
### Start download
|
||||
|
||||
Then run:
|
||||
|
||||
```bash
|
||||
freqtrade download-data --exchange binance
|
||||
```
|
||||
|
||||
This will download historical candle (OHLCV) data for all the currency pairs you defined in `pairs.json`.
|
||||
|
||||
### Other Notes
|
||||
|
||||
- To use a different directory than the exchange specific default, use `--datadir user_data/data/some_directory`.
|
||||
- To change the exchange used to download the historical data from, please use a different configuration file (you'll probably need to adjust rate limits etc.)
|
||||
- To use `pairs.json` from some other directory, use `--pairs-file some_other_dir/pairs.json`.
|
||||
- To download historical candle (OHLCV) data for only 10 days, use `--days 10` (defaults to 30 days).
|
||||
- To download historical candle (OHLCV) data from a fixed starting point, use `--timerange 20200101-` - which will download all data from January 1st, 2020. Eventually set end dates are ignored.
|
||||
- Use `--timeframes` to specify what timeframe download the historical candle (OHLCV) data for. Default is `--timeframes 1m 5m` which will download 1-minute and 5-minute data.
|
||||
- To use exchange, timeframe and list of pairs as defined in your configuration file, use the `-c/--config` option. With this, the script uses the whitelist defined in the config as the list of currency pairs to download data for and does not require the pairs.json file. You can combine `-c/--config` with most other options.
|
||||
|
||||
### Trades (tick) data
|
||||
|
||||
By default, `download-data` sub-command downloads Candles (OHLCV) data. Some exchanges also provide historic trade-data via their API.
|
||||
|
||||
@ -63,9 +63,9 @@ ARGS_CONVERT_TRADES = ["pairs", "timeframes", "exchange", "dataformat_ohlcv", "d
|
||||
|
||||
ARGS_LIST_DATA = ["exchange", "dataformat_ohlcv", "pairs"]
|
||||
|
||||
ARGS_DOWNLOAD_DATA = ["pairs", "pairs_file", "days", "new_pairs_days", "timerange",
|
||||
"download_trades", "exchange", "timeframes", "erase", "dataformat_ohlcv",
|
||||
"dataformat_trades"]
|
||||
ARGS_DOWNLOAD_DATA = ["pairs", "pairs_file", "days", "new_pairs_days", "include_inactive",
|
||||
"timerange", "download_trades", "exchange", "timeframes",
|
||||
"erase", "dataformat_ohlcv", "dataformat_trades"]
|
||||
|
||||
ARGS_PLOT_DATAFRAME = ["pairs", "indicators1", "indicators2", "plot_limit",
|
||||
"db_url", "trade_source", "export", "exportfilename",
|
||||
|
||||
@ -355,6 +355,11 @@ AVAILABLE_CLI_OPTIONS = {
|
||||
type=check_int_positive,
|
||||
metavar='INT',
|
||||
),
|
||||
"include_inactive": Arg(
|
||||
'--include-inactive-pairs',
|
||||
help='Also download data from inactive pairs.',
|
||||
action='store_true',
|
||||
),
|
||||
"new_pairs_days": Arg(
|
||||
'--new-pairs-days',
|
||||
help='Download data of new pairs for given number of days. Default: `%(default)s`.',
|
||||
|
||||
@ -11,6 +11,7 @@ from freqtrade.data.history import (convert_trades_to_ohlcv, refresh_backtest_oh
|
||||
from freqtrade.enums import RunMode
|
||||
from freqtrade.exceptions import OperationalException
|
||||
from freqtrade.exchange import timeframe_to_minutes
|
||||
from freqtrade.exchange.exchange import market_is_active
|
||||
from freqtrade.plugins.pairlist.pairlist_helpers import expand_pairlist
|
||||
from freqtrade.resolvers import ExchangeResolver
|
||||
|
||||
@ -47,11 +48,13 @@ def start_download_data(args: Dict[str, Any]) -> None:
|
||||
|
||||
# Init exchange
|
||||
exchange = ExchangeResolver.load_exchange(config['exchange']['name'], config, validate=False)
|
||||
markets = [p for p, m in exchange.markets.items() if market_is_active(m)
|
||||
or config.get('include_inactive')]
|
||||
expanded_pairs = expand_pairlist(config['pairs'], markets)
|
||||
|
||||
# Manual validations of relevant settings
|
||||
if not config['exchange'].get('skip_pair_validation', False):
|
||||
exchange.validate_pairs(config['pairs'])
|
||||
expanded_pairs = expand_pairlist(config['pairs'], list(exchange.markets))
|
||||
|
||||
exchange.validate_pairs(expanded_pairs)
|
||||
logger.info(f"About to download pairs: {expanded_pairs}, "
|
||||
f"intervals: {config['timeframes']} to {config['datadir']}")
|
||||
|
||||
|
||||
@ -407,6 +407,9 @@ class Configuration:
|
||||
self._args_to_config(config, argname='days',
|
||||
logstring='Detected --days: {}')
|
||||
|
||||
self._args_to_config(config, argname='include_inactive',
|
||||
logstring='Detected --include-inactive-pairs: {}')
|
||||
|
||||
self._args_to_config(config, argname='download_trades',
|
||||
logstring='Detected --dl-trades: {}')
|
||||
|
||||
|
||||
@ -754,6 +754,46 @@ def test_download_data_no_pairs(mocker, caplog):
|
||||
start_download_data(pargs)
|
||||
|
||||
|
||||
def test_download_data_all_pairs(mocker, markets):
|
||||
|
||||
mocker.patch.object(Path, "exists", MagicMock(return_value=False))
|
||||
|
||||
dl_mock = mocker.patch('freqtrade.commands.data_commands.refresh_backtest_ohlcv_data',
|
||||
MagicMock(return_value=["ETH/BTC", "XRP/BTC"]))
|
||||
patch_exchange(mocker)
|
||||
mocker.patch(
|
||||
'freqtrade.exchange.Exchange.markets', PropertyMock(return_value=markets)
|
||||
)
|
||||
args = [
|
||||
"download-data",
|
||||
"--exchange",
|
||||
"binance",
|
||||
"--pairs",
|
||||
".*/USDT"
|
||||
]
|
||||
pargs = get_args(args)
|
||||
pargs['config'] = None
|
||||
start_download_data(pargs)
|
||||
expected = set(['ETH/USDT', 'XRP/USDT', 'NEO/USDT', 'TKN/USDT'])
|
||||
assert set(dl_mock.call_args_list[0][1]['pairs']) == expected
|
||||
assert dl_mock.call_count == 1
|
||||
|
||||
dl_mock.reset_mock()
|
||||
args = [
|
||||
"download-data",
|
||||
"--exchange",
|
||||
"binance",
|
||||
"--pairs",
|
||||
".*/USDT",
|
||||
"--include-inactive-pairs",
|
||||
]
|
||||
pargs = get_args(args)
|
||||
pargs['config'] = None
|
||||
start_download_data(pargs)
|
||||
expected = set(['ETH/USDT', 'LTC/USDT', 'XRP/USDT', 'NEO/USDT', 'TKN/USDT'])
|
||||
assert set(dl_mock.call_args_list[0][1]['pairs']) == expected
|
||||
|
||||
|
||||
def test_download_data_trades(mocker, caplog):
|
||||
dl_mock = mocker.patch('freqtrade.commands.data_commands.refresh_backtest_trades_data',
|
||||
MagicMock(return_value=[]))
|
||||
|
||||
Loading…
Reference in New Issue
Block a user