diff --git a/docs/backtesting.md b/docs/backtesting.md index 8f8310afe..474498371 100644 --- a/docs/backtesting.md +++ b/docs/backtesting.md @@ -201,6 +201,8 @@ Since backtesting lacks some detailed information about what happens within a ca Taking these assumptions, backtesting tries to mirror real trading as closely as possible. However, backtesting will **never** replace running a strategy in dry-run mode. Also, keep in mind that past results don't guarantee future success. +In addition to the above assumptions, strategy authors should carefully read the [Common Mistakes](strategy-customization.md#common-mistakes-when-developing-strategies) section, to avoid using data in backtesting which is not available in real market conditions. + ### Further backtest-result analysis To further analyze your backtest results, you can [export the trades](#exporting-trades-to-file). diff --git a/docs/strategy-customization.md b/docs/strategy-customization.md index 177192c45..e401d3ab8 100644 --- a/docs/strategy-customization.md +++ b/docs/strategy-customization.md @@ -60,8 +60,7 @@ file as reference.** !!! Warning Using future data Since backtesting passes the full time interval to the `populate_*()` methods, the strategy author needs to take care to avoid having the strategy utilize data from the future. - Samples for usage of future data are `dataframe.shift(-1)`, `dataframe.resample("1h")` (this uses the left border of the interval, so moves data from an hour to the start of the hour). - They all use data which is not available during regular operations, so these strategies will perform well during backtesting, but will fail / perform badly in dry-runs. + Some common patterns for this are listed in the [Common Mistakes](#common-mistakes-when-developing-strategies) section of this document. ### Customize Indicators @@ -399,10 +398,10 @@ def populate_buy_trend(self, dataframe: DataFrame, metadata: dict) -> DataFrame: Printing more than a few rows is also possible (simply use `print(dataframe)` instead of `print(dataframe.tail())`), however not recommended, as that will be very verbose (~500 lines per pair every 5 seconds). -### Where is the default strategy? +### Where can i find a strategy template? -The default buy strategy is located in the file -[freqtrade/default_strategy.py](https://github.com/freqtrade/freqtrade/blob/develop/freqtrade/strategy/default_strategy.py). +The strategy template is located in the file +[user_data/strategies/sample_strategy.py](https://github.com/freqtrade/freqtrade/blob/develop/user_data/strategies/sample_strategy.py). ### Specify custom strategy location @@ -412,6 +411,18 @@ If you want to use a strategy from a different directory you can pass `--strateg freqtrade trade --strategy AwesomeStrategy --strategy-path /some/directory ``` +### Common mistakes when developing strategies + +Backtesting analyzes the whole time-range at once for performance reasons. Because of this, strategy authors need to make sure that strategies do not look-ahead into the future. +This is a common pain-point, which can cause huge differences between backtesting and dry/live run methods, since they all use data which is not available during dry/live runs, so these strategies will perform well during backtesting, but will fail / perform badly in real conditions. + +The following lists some common patterns which should be avoided to prevent frustration: + +- don't use `shift(-1)`. This uses data from the future, which is not available. +- don't use `.iloc[-1]` or any other absolute position in the dataframe, this will be different between dry-run and backtesting. +- don't use `dataframe['volume'].mean()`. This uses the full DataFrame for backtesting, including data from the future. Use `dataframe['volume'].rolling().mean()` instead +- don't use `.resample('1h')`. This uses the left border of the interval, so moves data from an hour to the start of the hour. Use `.resample('1h', label='right')` instead. + ### Further strategy ideas To get additional Ideas for strategies, head over to our [strategy repository](https://github.com/freqtrade/freqtrade-strategies). Feel free to use them as they are - but results will depend on the current market situation, pairs used etc. - therefore please backtest the strategy for your exchange/desired pairs first, evaluate carefully, use at your own risk. diff --git a/docs/utils.md b/docs/utils.md index 93162aca2..9f5792660 100644 --- a/docs/utils.md +++ b/docs/utils.md @@ -54,3 +54,73 @@ Timeframes available for the exchange `binance`: 1m, 3m, 5m, 15m, 30m, 1h, 2h, 4 ``` $ for i in `freqtrade list-exchanges -1`; do freqtrade list-timeframes --exchange $i; done ``` + +## List pairs/list markets + +The `list-pairs` and `list-markets` subcommands allow to see the pairs/markets available on exchange. + +Pairs are markets with the '/' character between the base currency part and the quote currency part in the market symbol. +For example, in the 'ETH/BTC' pair 'ETH' is the base currency, while 'BTC' is the quote currency. + +For pairs traded by Freqtrade the pair quote currency is defined by the value of the `stake_currency` configuration setting. + +You can print info about any pair/market with these subcommands - and you can filter output by quote-currency using `--quote BTC`, or by base-currency using `--base ETH` options correspondingly. + +These subcommands have same usage and same set of available options: + +``` +usage: freqtrade list-markets [-h] [--exchange EXCHANGE] [--print-list] + [--print-json] [-1] [--print-csv] + [--base BASE_CURRENCY [BASE_CURRENCY ...]] + [--quote QUOTE_CURRENCY [QUOTE_CURRENCY ...]] + [-a] + +usage: freqtrade list-pairs [-h] [--exchange EXCHANGE] [--print-list] + [--print-json] [-1] [--print-csv] + [--base BASE_CURRENCY [BASE_CURRENCY ...]] + [--quote QUOTE_CURRENCY [QUOTE_CURRENCY ...]] [-a] + +optional arguments: + -h, --help show this help message and exit + --exchange EXCHANGE Exchange name (default: `bittrex`). Only valid if no + config is provided. + --print-list Print list of pairs or market symbols. By default data + is printed in the tabular format. + --print-json Print list of pairs or market symbols in JSON format. + -1, --one-column Print output in one column. + --print-csv Print exchange pair or market data in the csv format. + --base BASE_CURRENCY [BASE_CURRENCY ...] + Specify base currency(-ies). Space-separated list. + --quote QUOTE_CURRENCY [QUOTE_CURRENCY ...] + Specify quote currency(-ies). Space-separated list. + -a, --all Print all pairs or market symbols. By default only + active ones are shown. +``` + +By default, only active pairs/markets are shown. Active pairs/markets are those that can currently be traded +on the exchange. The see the list of all pairs/markets (not only the active ones), use the `-a`/`-all` option. + +Pairs/markets are sorted by its symbol string in the printed output. + +### Examples + +* Print the list of active pairs with quote currency USD on exchange, specified in the default +configuration file (i.e. pairs on the "Bittrex" exchange) in JSON format: + +``` +$ freqtrade list-pairs --quote USD --print-json +``` + +* Print the list of all pairs on the exchange, specified in the `config_binance.json` configuration file +(i.e. on the "Binance" exchange) with base currencies BTC or ETH and quote currencies USDT or USD, as the +human-readable list with summary: + +``` +$ freqtrade -c config_binance.json list-pairs --all --base BTC ETH --quote USDT USD --print-list +``` + +* Print all markets on exchange "Kraken", in the tabular format: + +``` +$ freqtrade list-markets --exchange kraken --all +``` diff --git a/freqtrade/configuration/arguments.py b/freqtrade/configuration/arguments.py index bc58fc8e8..e62921af8 100644 --- a/freqtrade/configuration/arguments.py +++ b/freqtrade/configuration/arguments.py @@ -2,6 +2,7 @@ This module contains the argument manager class """ import argparse +from functools import partial from pathlib import Path from typing import Any, Dict, List, Optional @@ -33,6 +34,9 @@ ARGS_LIST_EXCHANGES = ["print_one_column", "list_exchanges_all"] ARGS_LIST_TIMEFRAMES = ["exchange", "print_one_column"] +ARGS_LIST_PAIRS = ["exchange", "print_list", "list_pairs_print_json", "print_one_column", + "print_csv", "base_currencies", "quote_currencies", "list_pairs_all"] + ARGS_CREATE_USERDIR = ["user_data_dir"] ARGS_DOWNLOAD_DATA = ["pairs", "pairs_file", "days", "download_trades", "exchange", @@ -45,7 +49,8 @@ ARGS_PLOT_DATAFRAME = ["pairs", "indicators1", "indicators2", "plot_limit", ARGS_PLOT_PROFIT = ["pairs", "timerange", "export", "exportfilename", "db_url", "trade_source", "ticker_interval"] -NO_CONF_REQURIED = ["download-data", "list-timeframes", "plot-dataframe", "plot-profit"] +NO_CONF_REQURIED = ["download-data", "list-timeframes", "list-markets", "list-pairs", + "plot-dataframe", "plot-profit"] NO_CONF_ALLOWED = ["create-userdir", "list-exchanges"] @@ -111,7 +116,8 @@ class Arguments: from freqtrade.optimize import start_backtesting, start_hyperopt, start_edge from freqtrade.utils import (start_create_userdir, start_download_data, - start_list_exchanges, start_list_timeframes, start_trading) + start_list_exchanges, start_list_markets, + start_list_timeframes, start_trading) from freqtrade.plot.plot_utils import start_plot_dataframe, start_plot_profit subparsers = self.parser.add_subparsers(dest='command', @@ -170,6 +176,22 @@ class Arguments: list_timeframes_cmd.set_defaults(func=start_list_timeframes) self._build_args(optionlist=ARGS_LIST_TIMEFRAMES, parser=list_timeframes_cmd) + # Add list-markets subcommand + list_markets_cmd = subparsers.add_parser( + 'list-markets', + help='Print markets on exchange.' + ) + list_markets_cmd.set_defaults(func=partial(start_list_markets, pairs_only=False)) + self._build_args(optionlist=ARGS_LIST_PAIRS, parser=list_markets_cmd) + + # Add list-pairs subcommand + list_pairs_cmd = subparsers.add_parser( + 'list-pairs', + help='Print pairs on exchange.' + ) + list_pairs_cmd.set_defaults(func=partial(start_list_markets, pairs_only=True)) + self._build_args(optionlist=ARGS_LIST_PAIRS, parser=list_pairs_cmd) + # Add download-data subcommand download_data_cmd = subparsers.add_parser( 'download-data', diff --git a/freqtrade/configuration/cli_options.py b/freqtrade/configuration/cli_options.py index ac72cff0b..ff2178108 100644 --- a/freqtrade/configuration/cli_options.py +++ b/freqtrade/configuration/cli_options.py @@ -255,6 +255,42 @@ AVAILABLE_CLI_OPTIONS = { help='Print all exchanges known to the ccxt library.', action='store_true', ), + # List pairs / markets + "list_pairs_all": Arg( + '-a', '--all', + help='Print all pairs or market symbols. By default only active ' + 'ones are shown.', + action='store_true', + ), + "print_list": Arg( + '--print-list', + help='Print list of pairs or market symbols. By default data is ' + 'printed in the tabular format.', + action='store_true', + ), + "list_pairs_print_json": Arg( + '--print-json', + help='Print list of pairs or market symbols in JSON format.', + action='store_true', + default=False, + ), + "print_csv": Arg( + '--print-csv', + help='Print exchange pair or market data in the csv format.', + action='store_true', + ), + "quote_currencies": Arg( + '--quote', + help='Specify quote currency(-ies). Space-separated list.', + nargs='+', + metavar='QUOTE_CURRENCY', + ), + "base_currencies": Arg( + '--base', + help='Specify base currency(-ies). Space-separated list.', + nargs='+', + metavar='BASE_CURRENCY', + ), # Script options "pairs": Arg( '-p', '--pairs', diff --git a/freqtrade/exchange/__init__.py b/freqtrade/exchange/__init__.py index 1b3e94951..0948692f1 100644 --- a/freqtrade/exchange/__init__.py +++ b/freqtrade/exchange/__init__.py @@ -10,5 +10,7 @@ from freqtrade.exchange.exchange import (timeframe_to_seconds, # noqa: F401 timeframe_to_msecs, timeframe_to_next_date, timeframe_to_prev_date) +from freqtrade.exchange.exchange import (market_is_active, # noqa: F401 + symbol_is_pair) from freqtrade.exchange.kraken import Kraken # noqa: F401 from freqtrade.exchange.binance import Binance # noqa: F401 diff --git a/freqtrade/exchange/exchange.py b/freqtrade/exchange/exchange.py index d53c580cb..71f0737ef 100644 --- a/freqtrade/exchange/exchange.py +++ b/freqtrade/exchange/exchange.py @@ -22,6 +22,7 @@ from freqtrade import (DependencyException, InvalidOrderException, from freqtrade.data.converter import parse_ticker_dataframe from freqtrade.misc import deep_merge_dicts + logger = logging.getLogger(__name__) @@ -165,7 +166,7 @@ class Exchange: } _ft_has: Dict = {} - def __init__(self, config: dict) -> None: + def __init__(self, config: dict, validate: bool = True) -> None: """ Initializes this module with the given config, it does basic validation whether the specified exchange and pairs are valid. @@ -216,19 +217,21 @@ class Exchange: logger.info('Using Exchange "%s"', self.name) - # Check if timeframe is available - self.validate_timeframes(config.get('ticker_interval')) + if validate: + # Check if timeframe is available + self.validate_timeframes(config.get('ticker_interval')) + + # Initial markets load + self._load_markets() + + # Check if all pairs are available + self.validate_pairs(config['exchange']['pair_whitelist']) + self.validate_ordertypes(config.get('order_types', {})) + self.validate_order_time_in_force(config.get('order_time_in_force', {})) # Converts the interval provided in minutes in config to seconds self.markets_refresh_interval: int = exchange_config.get( "markets_refresh_interval", 60) * 60 - # Initial markets load - self._load_markets() - - # Check if all pairs are available - self.validate_pairs(config['exchange']['pair_whitelist']) - self.validate_ordertypes(config.get('order_types', {})) - self.validate_order_time_in_force(config.get('order_time_in_force', {})) def __del__(self): """ @@ -293,6 +296,28 @@ class Exchange: self._load_markets() return self._api.markets + def get_markets(self, base_currencies: List[str] = None, quote_currencies: List[str] = None, + pairs_only: bool = False, active_only: bool = False) -> Dict: + """ + Return exchange ccxt markets, filtered out by base currency and quote currency + if this was requested in parameters. + + TODO: consider moving it to the Dataprovider + """ + markets = self.markets + if not markets: + raise OperationalException("Markets were not loaded.") + + if base_currencies: + markets = {k: v for k, v in markets.items() if v['base'] in base_currencies} + if quote_currencies: + markets = {k: v for k, v in markets.items() if v['quote'] in quote_currencies} + if pairs_only: + markets = {k: v for k, v in markets.items() if symbol_is_pair(v['symbol'])} + if active_only: + markets = {k: v for k, v in markets.items() if market_is_active(v)} + return markets + def klines(self, pair_interval: Tuple[str, str], copy=True) -> DataFrame: if pair_interval in self._klines: return self._klines[pair_interval].copy() if copy else self._klines[pair_interval] @@ -1074,3 +1099,27 @@ def timeframe_to_next_date(timeframe: str, date: datetime = None) -> datetime: new_timestamp = ccxt.Exchange.round_timeframe(timeframe, date.timestamp() * 1000, ROUND_UP) // 1000 return datetime.fromtimestamp(new_timestamp, tz=timezone.utc) + + +def symbol_is_pair(market_symbol: str, base_currency: str = None, quote_currency: str = None): + """ + Check if the market symbol is a pair, i.e. that its symbol consists of the base currency and the + quote currency separated by '/' character. If base_currency and/or quote_currency is passed, + it also checks that the symbol contains appropriate base and/or quote currency part before + and after the separating character correspondingly. + """ + symbol_parts = market_symbol.split('/') + return (len(symbol_parts) == 2 and + (symbol_parts[0] == base_currency if base_currency else len(symbol_parts[0]) > 0) and + (symbol_parts[1] == quote_currency if quote_currency else len(symbol_parts[1]) > 0)) + + +def market_is_active(market): + """ + Return True if the market is active. + """ + # "It's active, if the active flag isn't explicitly set to false. If it's missing or + # true then it's true. If it's undefined, then it's most likely true, but not 100% )" + # See https://github.com/ccxt/ccxt/issues/4874, + # https://github.com/ccxt/ccxt/issues/4075#issuecomment-434760520 + return market.get('active', True) is not False diff --git a/freqtrade/misc.py b/freqtrade/misc.py index b8e89b040..7682b5285 100644 --- a/freqtrade/misc.py +++ b/freqtrade/misc.py @@ -123,3 +123,7 @@ def round_dict(d, n): Rounds float values in the dict to n digits after the decimal point. """ return {k: (round(v, n) if isinstance(v, float) else v) for k, v in d.items()} + + +def plural(num, singular: str, plural: str = None) -> str: + return singular if (num == 1 or num == -1) else plural or singular + 's' diff --git a/freqtrade/pairlist/IPairList.py b/freqtrade/pairlist/IPairList.py index a112c63b4..5afb0c4c2 100644 --- a/freqtrade/pairlist/IPairList.py +++ b/freqtrade/pairlist/IPairList.py @@ -8,6 +8,9 @@ import logging from abc import ABC, abstractmethod from typing import List +from freqtrade.exchange import market_is_active + + logger = logging.getLogger(__name__) @@ -77,7 +80,7 @@ class IPairList(ABC): continue # Check if market is active market = markets[pair] - if not market['active']: + if not market_is_active(market): logger.info(f"Ignoring {pair} from whitelist. Market is not active.") continue sanitized_whitelist.add(pair) diff --git a/freqtrade/resolvers/exchange_resolver.py b/freqtrade/resolvers/exchange_resolver.py index 8ae40d971..60f37b1c9 100644 --- a/freqtrade/resolvers/exchange_resolver.py +++ b/freqtrade/resolvers/exchange_resolver.py @@ -17,7 +17,7 @@ class ExchangeResolver(IResolver): __slots__ = ['exchange'] - def __init__(self, exchange_name: str, config: dict) -> None: + def __init__(self, exchange_name: str, config: dict, validate: bool = True) -> None: """ Load the custom class from config parameter :param config: configuration dictionary @@ -26,12 +26,13 @@ class ExchangeResolver(IResolver): exchange_name = MAP_EXCHANGE_CHILDCLASS.get(exchange_name, exchange_name) exchange_name = exchange_name.title() try: - self.exchange = self._load_exchange(exchange_name, kwargs={'config': config}) + self.exchange = self._load_exchange(exchange_name, kwargs={'config': config, + 'validate': validate}) except ImportError: logger.info( f"No {exchange_name} specific subclass found. Using the generic class instead.") if not hasattr(self, "exchange"): - self.exchange = Exchange(config) + self.exchange = Exchange(config, validate=validate) def _load_exchange( self, exchange_name: str, kwargs: dict) -> Exchange: @@ -45,7 +46,7 @@ class ExchangeResolver(IResolver): try: ex_class = getattr(exchanges, exchange_name) - exchange = ex_class(kwargs['config']) + exchange = ex_class(**kwargs) if exchange: logger.info(f"Using resolved exchange '{exchange_name}'...") return exchange diff --git a/freqtrade/rpc/api_server.py b/freqtrade/rpc/api_server.py index 711202b27..67bbfdc78 100644 --- a/freqtrade/rpc/api_server.py +++ b/freqtrade/rpc/api_server.py @@ -2,7 +2,7 @@ import logging import threading from datetime import date, datetime from ipaddress import IPv4Address -from typing import Dict +from typing import Dict, Callable, Any from arrow import Arrow from flask import Flask, jsonify, request @@ -34,41 +34,45 @@ class ArrowJSONEncoder(JSONEncoder): return JSONEncoder.default(self, obj) +# Type should really be Callable[[ApiServer, Any], Any], but that will create a circular dependency +def require_login(func: Callable[[Any, Any], Any]): + + def func_wrapper(obj, *args, **kwargs): + + auth = request.authorization + if auth and obj.check_auth(auth.username, auth.password): + return func(obj, *args, **kwargs) + else: + return jsonify({"error": "Unauthorized"}), 401 + + return func_wrapper + + +# Type should really be Callable[[ApiServer], Any], but that will create a circular dependency +def rpc_catch_errors(func: Callable[[Any], Any]): + + def func_wrapper(obj, *args, **kwargs): + + try: + return func(obj, *args, **kwargs) + except RPCException as e: + logger.exception("API Error calling %s: %s", func.__name__, e) + return obj.rest_error(f"Error querying {func.__name__}: {e}") + + return func_wrapper + + class ApiServer(RPC): """ This class runs api server and provides rpc.rpc functionality to it - This class starts a none blocking thread the api server runs within + This class starts a non-blocking thread the api server runs within """ - def rpc_catch_errors(func): - - def func_wrapper(self, *args, **kwargs): - - try: - return func(self, *args, **kwargs) - except RPCException as e: - logger.exception("API Error calling %s: %s", func.__name__, e) - return self.rest_error(f"Error querying {func.__name__}: {e}") - - return func_wrapper - def check_auth(self, username, password): return (username == self._config['api_server'].get('username') and password == self._config['api_server'].get('password')) - def require_login(func): - - def func_wrapper(self, *args, **kwargs): - - auth = request.authorization - if auth and self.check_auth(auth.username, auth.password): - return func(self, *args, **kwargs) - else: - return jsonify({"error": "Unauthorized"}), 401 - - return func_wrapper - def __init__(self, freqtrade) -> None: """ Init the api server, and init the super class RPC diff --git a/freqtrade/utils.py b/freqtrade/utils.py index aff917e4a..506ed0a31 100644 --- a/freqtrade/utils.py +++ b/freqtrade/utils.py @@ -1,9 +1,13 @@ import logging import sys +from collections import OrderedDict from pathlib import Path from typing import Any, Dict, List import arrow +import csv +import rapidjson +from tabulate import tabulate from freqtrade import OperationalException from freqtrade.configuration import Configuration, TimeRange @@ -11,7 +15,9 @@ from freqtrade.configuration.directory_operations import create_userdata_dir from freqtrade.data.history import (convert_trades_to_ohlcv, refresh_backtest_ohlcv_data, refresh_backtest_trades_data) -from freqtrade.exchange import available_exchanges, ccxt_exchanges +from freqtrade.exchange import (available_exchanges, ccxt_exchanges, market_is_active, + symbol_is_pair) +from freqtrade.misc import plural from freqtrade.resolvers import ExchangeResolver from freqtrade.state import RunMode @@ -133,10 +139,94 @@ def start_list_timeframes(args: Dict[str, Any]) -> None: config['ticker_interval'] = None # Init exchange - exchange = ExchangeResolver(config['exchange']['name'], config).exchange + exchange = ExchangeResolver(config['exchange']['name'], config, validate=False).exchange if args['print_one_column']: print('\n'.join(exchange.timeframes)) else: print(f"Timeframes available for the exchange `{config['exchange']['name']}`: " - f"{', '.join(exchange.timeframes)}") + f"{', '.join(exchange.timeframes)}.") + + +def start_list_markets(args: Dict[str, Any], pairs_only: bool = False) -> None: + """ + Print pairs/markets on the exchange + :param args: Cli args from Arguments() + :param pairs_only: if True print only pairs, otherwise print all instruments (markets) + :return: None + """ + config = setup_utils_configuration(args, RunMode.OTHER) + + # Init exchange + exchange = ExchangeResolver(config['exchange']['name'], config, validate=False).exchange + + # By default only active pairs/markets are to be shown + active_only = not args.get('list_pairs_all', False) + + base_currencies = args.get('base_currencies', []) + quote_currencies = args.get('quote_currencies', []) + + try: + pairs = exchange.get_markets(base_currencies=base_currencies, + quote_currencies=quote_currencies, + pairs_only=pairs_only, + active_only=active_only) + # Sort the pairs/markets by symbol + pairs = OrderedDict(sorted(pairs.items())) + except Exception as e: + raise OperationalException(f"Cannot get markets. Reason: {e}") from e + + else: + summary_str = ((f"Exchange {exchange.name} has {len(pairs)} ") + + ("active " if active_only else "") + + (plural(len(pairs), "pair" if pairs_only else "market")) + + (f" with {', '.join(base_currencies)} as base " + f"{plural(len(base_currencies), 'currency', 'currencies')}" + if base_currencies else "") + + (" and" if base_currencies and quote_currencies else "") + + (f" with {', '.join(quote_currencies)} as quote " + f"{plural(len(quote_currencies), 'currency', 'currencies')}" + if quote_currencies else "")) + + headers = ["Id", "Symbol", "Base", "Quote", "Active", + *(['Is pair'] if not pairs_only else [])] + + tabular_data = [] + for _, v in pairs.items(): + tabular_data.append({'Id': v['id'], 'Symbol': v['symbol'], + 'Base': v['base'], 'Quote': v['quote'], + 'Active': market_is_active(v), + **({'Is pair': symbol_is_pair(v['symbol'])} + if not pairs_only else {})}) + + if (args.get('print_one_column', False) or + args.get('list_pairs_print_json', False) or + args.get('print_csv', False)): + # Print summary string in the log in case of machine-readable + # regular formats. + logger.info(f"{summary_str}.") + else: + # Print empty string separating leading logs and output in case of + # human-readable formats. + print() + + if len(pairs): + if args.get('print_list', False): + # print data as a list, with human-readable summary + print(f"{summary_str}: {', '.join(pairs.keys())}.") + elif args.get('print_one_column', False): + print('\n'.join(pairs.keys())) + elif args.get('list_pairs_print_json', False): + print(rapidjson.dumps(list(pairs.keys()), default=str)) + elif args.get('print_csv', False): + writer = csv.DictWriter(sys.stdout, fieldnames=headers) + writer.writeheader() + writer.writerows(tabular_data) + else: + # print data as a table, with the human-readable summary + print(f"{summary_str}:") + print(tabulate(tabular_data, headers='keys', tablefmt='pipe')) + elif not (args.get('print_one_column', False) or + args.get('list_pairs_print_json', False) or + args.get('print_csv', False)): + print(f"{summary_str}.") diff --git a/requirements-common.txt b/requirements-common.txt index c2c5317dc..1e42d8a04 100644 --- a/requirements-common.txt +++ b/requirements-common.txt @@ -1,8 +1,8 @@ # requirements without requirements installable via conda # mainly used for Raspberry pi installs -ccxt==1.18.1260 +ccxt==1.18.1306 SQLAlchemy==1.3.10 -python-telegram-bot==12.1.1 +python-telegram-bot==12.2.0 arrow==0.15.2 cachetools==3.1.1 requests==2.22.0 diff --git a/requirements-dev.txt b/requirements-dev.txt index 6cf378c03..f5cde59e8 100644 --- a/requirements-dev.txt +++ b/requirements-dev.txt @@ -6,8 +6,8 @@ coveralls==1.8.2 flake8==3.7.8 flake8-type-annotations==0.1.0 -flake8-tidy-imports==2.0.0 -mypy==0.730 +flake8-tidy-imports==3.0.0 +mypy==0.740 pytest==5.2.1 pytest-asyncio==0.10.0 pytest-cov==2.8.1 diff --git a/requirements-plot.txt b/requirements-plot.txt index 1f1df4ecc..235c71896 100644 --- a/requirements-plot.txt +++ b/requirements-plot.txt @@ -1,5 +1,5 @@ # Include all requirements to run the bot. -r requirements.txt -plotly==4.1.1 +plotly==4.2.1 diff --git a/requirements.txt b/requirements.txt index 2767180ac..8d9b4953f 100644 --- a/requirements.txt +++ b/requirements.txt @@ -1,5 +1,5 @@ # Load common requirements -r requirements-common.txt -numpy==1.17.2 -pandas==0.25.1 +numpy==1.17.3 +pandas==0.25.2 diff --git a/tests/conftest.py b/tests/conftest.py index 1a3c80f6c..4946c4d42 100644 --- a/tests/conftest.py +++ b/tests/conftest.py @@ -319,7 +319,8 @@ def markets(): 'symbol': 'TKN/BTC', 'base': 'TKN', 'quote': 'BTC', - 'active': True, + # According to ccxt, markets without active item set are also active + # 'active': True, 'precision': { 'price': 8, 'amount': 8, @@ -510,6 +511,50 @@ def markets(): } }, 'info': {}, + }, + 'LTC/USD': { + 'id': 'USD-LTC', + 'symbol': 'LTC/USD', + 'base': 'LTC', + 'quote': 'USD', + 'active': True, + 'precision': { + 'amount': 8, + 'price': 8 + }, + 'limits': { + 'amount': { + 'min': 0.06646786, + 'max': None + }, + 'price': { + 'min': 1e-08, + 'max': None + } + }, + 'info': {}, + }, + 'XLTCUSDT': { + 'id': 'xLTCUSDT', + 'symbol': 'XLTCUSDT', + 'base': 'LTC', + 'quote': 'USDT', + 'active': True, + 'precision': { + 'amount': 8, + 'price': 8 + }, + 'limits': { + 'amount': { + 'min': 0.06646786, + 'max': None + }, + 'price': { + 'min': 1e-08, + 'max': None + } + }, + 'info': {}, } } diff --git a/tests/exchange/test_exchange.py b/tests/exchange/test_exchange.py index 14d886085..1e0a5fdc3 100644 --- a/tests/exchange/test_exchange.py +++ b/tests/exchange/test_exchange.py @@ -18,7 +18,9 @@ from freqtrade.exchange.exchange import (API_RETRY_COUNT, timeframe_to_minutes, timeframe_to_msecs, timeframe_to_next_date, timeframe_to_prev_date, - timeframe_to_seconds) + timeframe_to_seconds, + symbol_is_pair, + market_is_active) from freqtrade.resolvers.exchange_resolver import ExchangeResolver from tests.conftest import get_patched_exchange, log_has, log_has_re @@ -929,17 +931,17 @@ def test_get_balances_prod(default_conf, mocker, exchange_name): def test_get_tickers(default_conf, mocker, exchange_name): api_mock = MagicMock() tick = {'ETH/BTC': { - 'symbol': 'ETH/BTC', - 'bid': 0.5, - 'ask': 1, - 'last': 42, - }, 'BCH/BTC': { - 'symbol': 'BCH/BTC', - 'bid': 0.6, - 'ask': 0.5, - 'last': 41, - } - } + 'symbol': 'ETH/BTC', + 'bid': 0.5, + 'ask': 1, + 'last': 42, + }, 'BCH/BTC': { + 'symbol': 'BCH/BTC', + 'bid': 0.6, + 'ask': 0.5, + 'last': 41, + } + } api_mock.fetch_tickers = MagicMock(return_value=tick) exchange = get_patched_exchange(mocker, default_conf, api_mock, id=exchange_name) # retrieve original ticker @@ -1693,6 +1695,74 @@ def test_get_valid_pair_combination(default_conf, mocker, markets): ex.get_valid_pair_combination("NOPAIR", "ETH") +@pytest.mark.parametrize( + "base_currencies, quote_currencies, pairs_only, active_only, expected_keys", [ + # Testing markets (in conftest.py): + # 'BLK/BTC': 'active': True + # 'BTT/BTC': 'active': True + # 'ETH/BTC': 'active': True + # 'ETH/USDT': 'active': True + # 'LTC/BTC': 'active': False + # 'LTC/USD': 'active': True + # 'LTC/USDT': 'active': True + # 'NEO/BTC': 'active': False + # 'TKN/BTC': 'active' not set + # 'XLTCUSDT': 'active': True, not a pair + # 'XRP/BTC': 'active': False + # all markets + ([], [], False, False, + ['BLK/BTC', 'BTT/BTC', 'ETH/BTC', 'ETH/USDT', 'LTC/BTC', 'LTC/USD', + 'LTC/USDT', 'NEO/BTC', 'TKN/BTC', 'XLTCUSDT', 'XRP/BTC']), + # active markets + ([], [], False, True, + ['BLK/BTC', 'BTT/BTC', 'ETH/BTC', 'ETH/USDT', 'LTC/USD', 'LTC/USDT', + 'TKN/BTC', 'XLTCUSDT']), + # all pairs + ([], [], True, False, + ['BLK/BTC', 'BTT/BTC', 'ETH/BTC', 'ETH/USDT', 'LTC/BTC', 'LTC/USD', + 'LTC/USDT', 'NEO/BTC', 'TKN/BTC', 'XRP/BTC']), + # active pairs + ([], [], True, True, + ['BLK/BTC', 'BTT/BTC', 'ETH/BTC', 'ETH/USDT', 'LTC/USD', 'LTC/USDT', 'TKN/BTC']), + # all markets, base=ETH, LTC + (['ETH', 'LTC'], [], False, False, + ['ETH/BTC', 'ETH/USDT', 'LTC/BTC', 'LTC/USD', 'LTC/USDT', 'XLTCUSDT']), + # all markets, base=LTC + (['LTC'], [], False, False, + ['LTC/BTC', 'LTC/USD', 'LTC/USDT', 'XLTCUSDT']), + # all markets, quote=USDT + ([], ['USDT'], False, False, + ['ETH/USDT', 'LTC/USDT', 'XLTCUSDT']), + # all markets, quote=USDT, USD + ([], ['USDT', 'USD'], False, False, + ['ETH/USDT', 'LTC/USD', 'LTC/USDT', 'XLTCUSDT']), + # all markets, base=LTC, quote=USDT + (['LTC'], ['USDT'], False, False, + ['LTC/USDT', 'XLTCUSDT']), + # all pairs, base=LTC, quote=USDT + (['LTC'], ['USDT'], True, False, + ['LTC/USDT']), + # all markets, base=LTC, quote=USDT, NONEXISTENT + (['LTC'], ['USDT', 'NONEXISTENT'], False, False, + ['LTC/USDT', 'XLTCUSDT']), + # all markets, base=LTC, quote=NONEXISTENT + (['LTC'], ['NONEXISTENT'], False, False, + []), + ]) +def test_get_markets(default_conf, mocker, markets, + base_currencies, quote_currencies, pairs_only, active_only, + expected_keys): + mocker.patch.multiple('freqtrade.exchange.Exchange', + _init_ccxt=MagicMock(return_value=MagicMock()), + _load_async_markets=MagicMock(), + validate_pairs=MagicMock(), + validate_timeframes=MagicMock(), + markets=PropertyMock(return_value=markets)) + ex = Exchange(default_conf) + pairs = ex.get_markets(base_currencies, quote_currencies, pairs_only, active_only) + assert sorted(pairs.keys()) == sorted(expected_keys) + + def test_timeframe_to_minutes(): assert timeframe_to_minutes("5m") == 5 assert timeframe_to_minutes("10m") == 10 @@ -1762,3 +1832,33 @@ def test_timeframe_to_next_date(): date = datetime.now(tz=timezone.utc) assert timeframe_to_next_date("5m") > date + + +@pytest.mark.parametrize("market_symbol,base_currency,quote_currency,expected_result", [ + ("BTC/USDT", None, None, True), + ("USDT/BTC", None, None, True), + ("BTCUSDT", None, None, False), + ("BTC/USDT", None, "USDT", True), + ("USDT/BTC", None, "USDT", False), + ("BTCUSDT", None, "USDT", False), + ("BTC/USDT", "BTC", None, True), + ("USDT/BTC", "BTC", None, False), + ("BTCUSDT", "BTC", None, False), + ("BTC/USDT", "BTC", "USDT", True), + ("BTC/USDT", "USDT", "BTC", False), + ("BTC/USDT", "BTC", "USD", False), + ("BTCUSDT", "BTC", "USDT", False), + ("BTC/", None, None, False), + ("/USDT", None, None, False), +]) +def test_symbol_is_pair(market_symbol, base_currency, quote_currency, expected_result) -> None: + assert symbol_is_pair(market_symbol, base_currency, quote_currency) == expected_result + + +@pytest.mark.parametrize("market,expected_result", [ + ({'symbol': 'ETH/BTC', 'active': True}, True), + ({'symbol': 'ETH/BTC', 'active': False}, False), + ({'symbol': 'ETH/BTC', }, True), +]) +def test_market_is_active(market, expected_result) -> None: + assert market_is_active(market) == expected_result diff --git a/tests/test_misc.py b/tests/test_misc.py index 320ed208c..23231e2f0 100644 --- a/tests/test_misc.py +++ b/tests/test_misc.py @@ -7,7 +7,7 @@ from unittest.mock import MagicMock from freqtrade.data.converter import parse_ticker_dataframe from freqtrade.data.history import pair_data_filename from freqtrade.misc import (datesarray_to_datetimearray, file_dump_json, - file_load_json, format_ms_time, shorten_date) + file_load_json, format_ms_time, plural, shorten_date) def test_shorten_date() -> None: @@ -69,3 +69,35 @@ def test_format_ms_time() -> None: # Date 2017-12-13 08:02:01 date_in_epoch_ms = 1513152121000 assert format_ms_time(date_in_epoch_ms) == res.astimezone(None).strftime('%Y-%m-%dT%H:%M:%S') + + +def test_plural() -> None: + assert plural(0, "page") == "pages" + assert plural(0.0, "page") == "pages" + assert plural(1, "page") == "page" + assert plural(1.0, "page") == "page" + assert plural(2, "page") == "pages" + assert plural(2.0, "page") == "pages" + assert plural(-1, "page") == "page" + assert plural(-1.0, "page") == "page" + assert plural(-2, "page") == "pages" + assert plural(-2.0, "page") == "pages" + assert plural(0.5, "page") == "pages" + assert plural(1.5, "page") == "pages" + assert plural(-0.5, "page") == "pages" + assert plural(-1.5, "page") == "pages" + + assert plural(0, "ox", "oxen") == "oxen" + assert plural(0.0, "ox", "oxen") == "oxen" + assert plural(1, "ox", "oxen") == "ox" + assert plural(1.0, "ox", "oxen") == "ox" + assert plural(2, "ox", "oxen") == "oxen" + assert plural(2.0, "ox", "oxen") == "oxen" + assert plural(-1, "ox", "oxen") == "ox" + assert plural(-1.0, "ox", "oxen") == "ox" + assert plural(-2, "ox", "oxen") == "oxen" + assert plural(-2.0, "ox", "oxen") == "oxen" + assert plural(0.5, "ox", "oxen") == "oxen" + assert plural(1.5, "ox", "oxen") == "oxen" + assert plural(-0.5, "ox", "oxen") == "oxen" + assert plural(-1.5, "ox", "oxen") == "oxen" diff --git a/tests/test_utils.py b/tests/test_utils.py index 4e76bb6ca..c598cfd76 100644 --- a/tests/test_utils.py +++ b/tests/test_utils.py @@ -8,7 +8,7 @@ from freqtrade import OperationalException from freqtrade.state import RunMode from freqtrade.utils import (setup_utils_configuration, start_create_userdir, start_download_data, start_list_exchanges, - start_list_timeframes) + start_list_markets, start_list_timeframes) from tests.conftest import get_args, log_has, patch_exchange @@ -164,6 +164,247 @@ def test_list_timeframes(mocker, capsys): assert re.search(r"^1d$", captured.out, re.MULTILINE) +def test_list_markets(mocker, markets, capsys): + + api_mock = MagicMock() + api_mock.markets = markets + patch_exchange(mocker, api_mock=api_mock) + + # Test with no --config + args = [ + "list-markets", + ] + pargs = get_args(args) + pargs['config'] = None + with pytest.raises(OperationalException, + match=r"This command requires a configured exchange.*"): + start_list_markets(pargs, False) + + # Test with --config config.json.example + args = [ + '--config', 'config.json.example', + "list-markets", + "--print-list", + ] + start_list_markets(get_args(args), False) + captured = capsys.readouterr() + assert ("Exchange Bittrex has 8 active markets: " + "BLK/BTC, BTT/BTC, ETH/BTC, ETH/USDT, LTC/USD, LTC/USDT, TKN/BTC, XLTCUSDT.\n" + in captured.out) + + patch_exchange(mocker, api_mock=api_mock, id="binance") + # Test with --exchange + args = [ + "list-markets", + "--exchange", "binance" + ] + pargs = get_args(args) + pargs['config'] = None + start_list_markets(pargs, False) + captured = capsys.readouterr() + assert re.match("\nExchange Binance has 8 active markets:\n", + captured.out) + + patch_exchange(mocker, api_mock=api_mock, id="bittrex") + # Test with --all: all markets + args = [ + '--config', 'config.json.example', + "list-markets", "--all", + "--print-list", + ] + start_list_markets(get_args(args), False) + captured = capsys.readouterr() + assert ("Exchange Bittrex has 11 markets: " + "BLK/BTC, BTT/BTC, ETH/BTC, ETH/USDT, LTC/BTC, LTC/USD, LTC/USDT, NEO/BTC, " + "TKN/BTC, XLTCUSDT, XRP/BTC.\n" + in captured.out) + + # Test list-pairs subcommand: active pairs + args = [ + '--config', 'config.json.example', + "list-pairs", + "--print-list", + ] + start_list_markets(get_args(args), True) + captured = capsys.readouterr() + assert ("Exchange Bittrex has 7 active pairs: " + "BLK/BTC, BTT/BTC, ETH/BTC, ETH/USDT, LTC/USD, LTC/USDT, TKN/BTC.\n" + in captured.out) + + # Test list-pairs subcommand with --all: all pairs + args = [ + '--config', 'config.json.example', + "list-pairs", "--all", + "--print-list", + ] + start_list_markets(get_args(args), True) + captured = capsys.readouterr() + assert ("Exchange Bittrex has 10 pairs: " + "BLK/BTC, BTT/BTC, ETH/BTC, ETH/USDT, LTC/BTC, LTC/USD, LTC/USDT, NEO/BTC, " + "TKN/BTC, XRP/BTC.\n" + in captured.out) + + # active markets, base=ETH, LTC + args = [ + '--config', 'config.json.example', + "list-markets", + "--base", "ETH", "LTC", + "--print-list", + ] + start_list_markets(get_args(args), False) + captured = capsys.readouterr() + assert ("Exchange Bittrex has 5 active markets with ETH, LTC as base currencies: " + "ETH/BTC, ETH/USDT, LTC/USD, LTC/USDT, XLTCUSDT.\n" + in captured.out) + + # active markets, base=LTC + args = [ + '--config', 'config.json.example', + "list-markets", + "--base", "LTC", + "--print-list", + ] + start_list_markets(get_args(args), False) + captured = capsys.readouterr() + assert ("Exchange Bittrex has 3 active markets with LTC as base currency: " + "LTC/USD, LTC/USDT, XLTCUSDT.\n" + in captured.out) + + # active markets, quote=USDT, USD + args = [ + '--config', 'config.json.example', + "list-markets", + "--quote", "USDT", "USD", + "--print-list", + ] + start_list_markets(get_args(args), False) + captured = capsys.readouterr() + assert ("Exchange Bittrex has 4 active markets with USDT, USD as quote currencies: " + "ETH/USDT, LTC/USD, LTC/USDT, XLTCUSDT.\n" + in captured.out) + + # active markets, quote=USDT + args = [ + '--config', 'config.json.example', + "list-markets", + "--quote", "USDT", + "--print-list", + ] + start_list_markets(get_args(args), False) + captured = capsys.readouterr() + assert ("Exchange Bittrex has 3 active markets with USDT as quote currency: " + "ETH/USDT, LTC/USDT, XLTCUSDT.\n" + in captured.out) + + # active markets, base=LTC, quote=USDT + args = [ + '--config', 'config.json.example', + "list-markets", + "--base", "LTC", "--quote", "USDT", + "--print-list", + ] + start_list_markets(get_args(args), False) + captured = capsys.readouterr() + assert ("Exchange Bittrex has 2 active markets with LTC as base currency and " + "with USDT as quote currency: LTC/USDT, XLTCUSDT.\n" + in captured.out) + + # active pairs, base=LTC, quote=USDT + args = [ + '--config', 'config.json.example', + "list-pairs", + "--base", "LTC", "--quote", "USDT", + "--print-list", + ] + start_list_markets(get_args(args), True) + captured = capsys.readouterr() + assert ("Exchange Bittrex has 1 active pair with LTC as base currency and " + "with USDT as quote currency: LTC/USDT.\n" + in captured.out) + + # active markets, base=LTC, quote=USDT, NONEXISTENT + args = [ + '--config', 'config.json.example', + "list-markets", + "--base", "LTC", "--quote", "USDT", "NONEXISTENT", + "--print-list", + ] + start_list_markets(get_args(args), False) + captured = capsys.readouterr() + assert ("Exchange Bittrex has 2 active markets with LTC as base currency and " + "with USDT, NONEXISTENT as quote currencies: LTC/USDT, XLTCUSDT.\n" + in captured.out) + + # active markets, base=LTC, quote=NONEXISTENT + args = [ + '--config', 'config.json.example', + "list-markets", + "--base", "LTC", "--quote", "NONEXISTENT", + "--print-list", + ] + start_list_markets(get_args(args), False) + captured = capsys.readouterr() + assert ("Exchange Bittrex has 0 active markets with LTC as base currency and " + "with NONEXISTENT as quote currency.\n" + in captured.out) + + # Test tabular output + args = [ + '--config', 'config.json.example', + "list-markets", + ] + start_list_markets(get_args(args), False) + captured = capsys.readouterr() + assert ("Exchange Bittrex has 8 active markets:\n" + in captured.out) + + # Test tabular output, no markets found + args = [ + '--config', 'config.json.example', + "list-markets", + "--base", "LTC", "--quote", "NONEXISTENT", + ] + start_list_markets(get_args(args), False) + captured = capsys.readouterr() + assert ("Exchange Bittrex has 0 active markets with LTC as base currency and " + "with NONEXISTENT as quote currency.\n" + in captured.out) + + # Test --print-json + args = [ + '--config', 'config.json.example', + "list-markets", + "--print-json" + ] + start_list_markets(get_args(args), False) + captured = capsys.readouterr() + assert ('["BLK/BTC","BTT/BTC","ETH/BTC","ETH/USDT","LTC/USD","LTC/USDT","TKN/BTC","XLTCUSDT"]' + in captured.out) + + # Test --print-csv + args = [ + '--config', 'config.json.example', + "list-markets", + "--print-csv" + ] + start_list_markets(get_args(args), False) + captured = capsys.readouterr() + assert ("Id,Symbol,Base,Quote,Active,Is pair" in captured.out) + assert ("blkbtc,BLK/BTC,BLK,BTC,True,True" in captured.out) + assert ("BTTBTC,BTT/BTC,BTT,BTC,True,True" in captured.out) + + # Test --one-column + args = [ + '--config', 'config.json.example', + "list-markets", + "--one-column" + ] + start_list_markets(get_args(args), False) + captured = capsys.readouterr() + assert re.search(r"^BLK/BTC$", captured.out, re.MULTILINE) + assert re.search(r"^BTT/BTC$", captured.out, re.MULTILINE) + + def test_create_datadir_failed(caplog): args = [