diff --git a/README.md b/README.md index 571709e3b..0d0724d3a 100644 --- a/README.md +++ b/README.md @@ -34,13 +34,15 @@ hesitate to read the source code and understand the mechanism of this bot. - [x] **Dry-run**: Run the bot without playing money. - [x] **Backtesting**: Run a simulation of your buy/sell strategy. - [x] **Strategy Optimization by machine learning**: Use machine learning to optimize your buy/sell strategy parameters with real exchange data. -- [x] **Whitelist crypto-currencies**: Select which crypto-currency you want to trade. +- [x] **Edge position sizing** Calculate your win rate, risk reward ratio, the best stoploss and adjust your position size before taking a position for each specific market. [Learn more](https://github.com/freqtrade/freqtrade/blob/develop/docs/edge.md) +- [x] **Whitelist crypto-currencies**: Select which crypto-currency you want to trade or use dynamic whitelists. - [x] **Blacklist crypto-currencies**: Select which crypto-currency you want to avoid. - [x] **Manageable via Telegram**: Manage the bot with Telegram - [x] **Display profit/loss in fiat**: Display your profit/loss in 33 fiat. - [x] **Daily summary of profit/loss**: Provide a daily summary of your profit/loss. - [x] **Performance status report**: Provide a performance status of your current trades. + ## Table of Contents - [Quick start](#quick-start) @@ -51,6 +53,7 @@ hesitate to read the source code and understand the mechanism of this bot. - [Backtesting](https://github.com/freqtrade/freqtrade/blob/develop/docs/backtesting.md) - [Hyperopt](https://github.com/freqtrade/freqtrade/blob/develop/docs/hyperopt.md) - [Sandbox Testing](https://github.com/freqtrade/freqtrade/blob/develop/docs/sandbox-testing.md) + - [Edge](https://github.com/freqtrade/freqtrade/blob/develop/docs/edge.md) - [Basic Usage](#basic-usage) - [Bot commands](#bot-commands) - [Telegram RPC commands](#telegram-rpc-commands) @@ -62,7 +65,8 @@ hesitate to read the source code and understand the mechanism of this bot. - [Requirements](#requirements) - [Min hardware required](#min-hardware-required) - [Software requirements](#software-requirements) -- [Wanna help?] +- [Wanna help?](https://github.com/freqtrade/freqtrade/blob/develop/CONTRIBUTING.md) + - [Dev - getting started](https://github.com/freqtrade/freqtrade/blob/develop/docs/developer.md) (WIP) ## Quick start diff --git a/docs/bot-usage.md b/docs/bot-usage.md index 114e7613e..5451c8459 100644 --- a/docs/bot-usage.md +++ b/docs/bot-usage.md @@ -240,6 +240,8 @@ optional arguments: --stoplosses=-0.01,-0.1,-0.001 ``` +To understand edge and how to read the results, please read the [edge documentation](edge.md). + ## A parameter missing in the configuration? All parameters for `main.py`, `backtesting`, `hyperopt` are referenced diff --git a/docs/configuration.md b/docs/configuration.md index 521f330ec..6f54cfa8d 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -39,7 +39,7 @@ The table below will list all configuration parameters. | `ask_strategy.use_order_book` | false | No | Allows selling of open traded pair using the rates in Order Book Asks. | `ask_strategy.order_book_min` | 0 | No | Bot will scan from the top min to max Order Book Asks searching for a profitable rate. | `ask_strategy.order_book_max` | 0 | No | Bot will scan from the top min to max Order Book Asks searching for a profitable rate. -| `order_types` | None | No | Configure order-types depending on the action (`"buy"`, `"sell"`, `"stoploss"`, `"stoploss_on_exchange"`). +| `order_types` | None | No | Configure order-types depending on the action (`"buy"`, `"sell"`, `"stoploss"`, `"stoploss_on_exchange"`). [More information below](#understand-order_types). | `order_time_in_force` | None | No | Configure time in force for buy and sell orders. [More information below](#understand-order_time_in_force). | `exchange.name` | bittrex | Yes | Name of the exchange class to use. [List below](#user-content-what-values-for-exchangename). | `exchange.key` | key | No | API key to use for the exchange. Only required when you are in production mode. diff --git a/docs/index.md b/docs/index.md index 9eb0d445c..8fa24e996 100644 --- a/docs/index.md +++ b/docs/index.md @@ -26,7 +26,7 @@ Pull-request. Do not hesitate to reach us on - [Change your strategy](https://github.com/freqtrade/freqtrade/blob/develop/docs/bot-optimization.md#change-your-strategy) - [Add more Indicator](https://github.com/freqtrade/freqtrade/blob/develop/docs/bot-optimization.md#add-more-indicator) - [Test your strategy with Backtesting](https://github.com/freqtrade/freqtrade/blob/develop/docs/backtesting.md) - - [Edge positioning](https://github.com/freqtrade/freqtrade/blob/money_mgt/docs/edge.md) + - [Edge positioning](https://github.com/freqtrade/freqtrade/blob/develop/docs/edge.md) - [Find optimal parameters with Hyperopt](https://github.com/freqtrade/freqtrade/blob/develop/docs/hyperopt.md) - [Control the bot with telegram](https://github.com/freqtrade/freqtrade/blob/develop/docs/telegram-usage.md) - [Receive notifications via webhook](https://github.com/freqtrade/freqtrade/blob/develop/docs/webhook-config.md) diff --git a/freqtrade/edge/__init__.py b/freqtrade/edge/__init__.py index 49acbd3e7..9bd43e37f 100644 --- a/freqtrade/edge/__init__.py +++ b/freqtrade/edge/__init__.py @@ -144,20 +144,6 @@ class Edge(): self._cached_pairs = self._process_expectancy(trades_df) self._last_updated = arrow.utcnow().timestamp - # Not a nice hack but probably simplest solution: - # When backtest load data it loads the delta between disk and exchange - # The problem is that exchange consider that recent. - # it is but it is incomplete (c.f. _async_get_candle_history) - # So it causes get_signal to exit cause incomplete ticker_hist - # A patch to that would be update _pairs_last_refresh_time of exchange - # so it will download again all pairs - # Another solution is to add new data to klines instead of reassigning it: - # self.klines[pair].update(data) instead of self.klines[pair] = data in exchange package. - # But that means indexing timestamp and having a verification so that - # there is no empty range between two timestaps (recently added and last - # one) - self.exchange._pairs_last_refresh_time = {} - return True def stake_amount(self, pair: str, free_capital: float, diff --git a/freqtrade/exchange/__init__.py b/freqtrade/exchange/__init__.py index e9d819beb..b59e8407a 100644 --- a/freqtrade/exchange/__init__.py +++ b/freqtrade/exchange/__init__.py @@ -7,12 +7,14 @@ from typing import List, Dict, Tuple, Any, Optional from datetime import datetime from math import floor, ceil +import arrow import asyncio import ccxt import ccxt.async_support as ccxt_async -import arrow +from pandas import DataFrame from freqtrade import constants, OperationalException, DependencyException, TemporaryError +from freqtrade.exchange.exchange_helpers import parse_ticker_dataframe logger = logging.getLogger(__name__) @@ -64,14 +66,8 @@ def retrier(f): class Exchange(object): - # Current selected exchange - _api: ccxt.Exchange = None - _api_async: ccxt_async.Exchange = None _conf: Dict = {} - # Holds all open sell orders for dry_run - _dry_run_open_orders: Dict[str, Any] = {} - def __init__(self, config: dict) -> None: """ Initializes this module with the given config, @@ -87,15 +83,19 @@ class Exchange(object): self._pairs_last_refresh_time: Dict[str, int] = {} # Holds candles - self.klines: Dict[str, Any] = {} + self._klines: Dict[str, DataFrame] = {} + + # Holds all open sell orders for dry_run + self._dry_run_open_orders: Dict[str, Any] = {} if config['dry_run']: logger.info('Instance is running with dry_run enabled') exchange_config = config['exchange'] - self._api = self._init_ccxt(exchange_config, ccxt_kwargs=exchange_config.get('ccxt_config')) - self._api_async = self._init_ccxt(exchange_config, ccxt_async, - ccxt_kwargs=exchange_config.get('ccxt_async_config')) + self._api: ccxt.Exchange = self._init_ccxt( + exchange_config, ccxt_kwargs=exchange_config.get('ccxt_config')) + self._api_async: ccxt_async.Exchange = self._init_ccxt( + exchange_config, ccxt_async, ccxt_kwargs=exchange_config.get('ccxt_async_config')) logger.info('Using Exchange "%s"', self.name) @@ -129,12 +129,12 @@ class Exchange(object): raise OperationalException(f'Exchange {name} is not supported') ex_config = { - 'apiKey': exchange_config.get('key'), - 'secret': exchange_config.get('secret'), - 'password': exchange_config.get('password'), - 'uid': exchange_config.get('uid', ''), - 'enableRateLimit': exchange_config.get('ccxt_rate_limit', True) - } + 'apiKey': exchange_config.get('key'), + 'secret': exchange_config.get('secret'), + 'password': exchange_config.get('password'), + 'uid': exchange_config.get('uid', ''), + 'enableRateLimit': exchange_config.get('ccxt_rate_limit', True) + } if ccxt_kwargs: logger.info('Applying additional ccxt config: %s', ccxt_kwargs) ex_config.update(ccxt_kwargs) @@ -158,6 +158,12 @@ class Exchange(object): """exchange ccxt id""" return self._api.id + def klines(self, pair: str) -> DataFrame: + if pair in self._klines: + return self._klines[pair].copy() + else: + return None + def set_sandbox(self, api, exchange_config: dict, name: str): if exchange_config.get('sandbox'): if api.urls.get('test'): @@ -513,9 +519,9 @@ class Exchange(object): # Combine tickers data: List = [] - for tick in tickers: - if tick[0] == pair: - data.extend(tick[1]) + for p, ticker in tickers: + if p == pair: + data.extend(ticker) # Sort data again after extending the result - above calls return in "async order" order data = sorted(data, key=lambda x: x[0]) logger.info("downloaded %s with length %s.", pair, len(data)) @@ -523,7 +529,7 @@ class Exchange(object): def refresh_tickers(self, pair_list: List[str], ticker_interval: str) -> None: """ - Refresh tickers asyncronously and return the result. + Refresh tickers asyncronously and set `_klines` of this object with the result """ logger.debug("Refreshing klines for %d pairs", len(pair_list)) asyncio.get_event_loop().run_until_complete( @@ -532,9 +538,27 @@ class Exchange(object): async def async_get_candles_history(self, pairs: List[str], tick_interval: str) -> List[Tuple[str, List]]: """Download ohlcv history for pair-list asyncronously """ - input_coroutines = [self._async_get_candle_history( - symbol, tick_interval) for symbol in pairs] + # Calculating ticker interval in second + interval_in_sec = constants.TICKER_INTERVAL_MINUTES[tick_interval] * 60 + input_coroutines = [] + + # Gather corotines to run + for pair in pairs: + if not (self._pairs_last_refresh_time.get(pair, 0) + interval_in_sec >= + arrow.utcnow().timestamp and pair in self._klines): + input_coroutines.append(self._async_get_candle_history(pair, tick_interval)) + else: + logger.debug("Using cached klines data for %s ...", pair) + tickers = await asyncio.gather(*input_coroutines, return_exceptions=True) + + # handle caching + for pair, ticks in tickers: + # keeping last candle time as last refreshed time of the pair + if ticks: + self._pairs_last_refresh_time[pair] = ticks[-1][0] // 1000 + # keeping parsed dataframe in cache + self._klines[pair] = parse_ticker_dataframe(ticks) return tickers @retrier_async @@ -544,20 +568,8 @@ class Exchange(object): # fetch ohlcv asynchronously logger.debug("fetching %s since %s ...", pair, since_ms) - # Calculating ticker interval in second - interval_in_sec = constants.TICKER_INTERVAL_MINUTES[tick_interval] * 60 - - # If (last update time) + (interval in second) is greater or equal than now - # that means we don't have to hit the API as there is no new candle - # so we fetch it from local cache - if (not since_ms and - self._pairs_last_refresh_time.get(pair, 0) + interval_in_sec >= - arrow.utcnow().timestamp): - data = self.klines[pair] - logger.debug("Using cached klines data for %s ...", pair) - else: - data = await self._api_async.fetch_ohlcv(pair, timeframe=tick_interval, - since=since_ms) + data = await self._api_async.fetch_ohlcv(pair, timeframe=tick_interval, + since=since_ms) # Because some exchange sort Tickers ASC and other DESC. # Ex: Bittrex returns a list of tickers ASC (oldest first, newest last) @@ -566,13 +578,6 @@ class Exchange(object): if data and data[0][0] > data[-1][0]: data = sorted(data, key=lambda x: x[0]) - # keeping last candle time as last refreshed time of the pair - if data: - self._pairs_last_refresh_time[pair] = data[-1][0] // 1000 - - # keeping candles in cache - self.klines[pair] = data - logger.debug("done fetching %s ...", pair) return pair, data diff --git a/freqtrade/exchange/exchange_helpers.py b/freqtrade/exchange/exchange_helpers.py index 84e68d4bb..729a4a987 100644 --- a/freqtrade/exchange/exchange_helpers.py +++ b/freqtrade/exchange/exchange_helpers.py @@ -14,6 +14,7 @@ def parse_ticker_dataframe(ticker: list) -> DataFrame: :param ticker: ticker list, as returned by exchange.async_get_candle_history :return: DataFrame """ + logger.debug("Parsing tickerlist to dataframe") cols = ['date', 'open', 'high', 'low', 'close', 'volume'] frame = DataFrame(ticker, columns=cols) diff --git a/freqtrade/freqtradebot.py b/freqtrade/freqtradebot.py index 8e0440dc3..9e84e74ec 100644 --- a/freqtrade/freqtradebot.py +++ b/freqtrade/freqtradebot.py @@ -317,7 +317,7 @@ class FreqtradeBot(object): # running get_signal on historical data fetched for _pair in whitelist: - (buy, sell) = self.strategy.get_signal(_pair, interval, self.exchange.klines.get(_pair)) + (buy, sell) = self.strategy.get_signal(_pair, interval, self.exchange.klines(_pair)) if buy and not sell: stake_amount = self._get_trade_stake_amount(_pair) if not stake_amount: @@ -578,9 +578,8 @@ class FreqtradeBot(object): (buy, sell) = (False, False) experimental = self.config.get('experimental', {}) if experimental.get('use_sell_signal') or experimental.get('ignore_roi_if_buy_signal'): - ticker = self.exchange.klines.get(trade.pair) (buy, sell) = self.strategy.get_signal(trade.pair, self.strategy.ticker_interval, - ticker) + self.exchange.klines(trade.pair)) config_ask_strategy = self.config.get('ask_strategy', {}) if config_ask_strategy.get('use_order_book', False): diff --git a/freqtrade/optimize/backtesting.py b/freqtrade/optimize/backtesting.py index f950ddb3c..61a2c3e9d 100644 --- a/freqtrade/optimize/backtesting.py +++ b/freqtrade/optimize/backtesting.py @@ -362,7 +362,7 @@ class Backtesting(object): if self.config.get('live'): logger.info('Downloading data for all pairs in whitelist ...') self.exchange.refresh_tickers(pairs, self.ticker_interval) - data = self.exchange.klines + data = self.exchange._klines else: logger.info('Using local backtesting data (using whitelist in given config) ...') diff --git a/freqtrade/optimize/edge_cli.py b/freqtrade/optimize/edge_cli.py index a2189f6c1..a98f0c877 100644 --- a/freqtrade/optimize/edge_cli.py +++ b/freqtrade/optimize/edge_cli.py @@ -1,7 +1,7 @@ # pragma pylint: disable=missing-docstring, W0212, too-many-arguments """ -This module contains the backtesting logic +This module contains the edge backtesting interface """ import logging from argparse import Namespace @@ -19,11 +19,11 @@ logger = logging.getLogger(__name__) class EdgeCli(object): """ - Backtesting class, this class contains all the logic to run a backtest + EdgeCli class, this class contains all the logic to run edge backtesting - To run a backtest: - backtesting = Backtesting(config) - backtesting.start() + To run a edge backtest: + edge = EdgeCli(config) + edge.start() """ def __init__(self, config: Dict[str, Any]) -> None: @@ -77,7 +77,7 @@ class EdgeCli(object): def setup_configuration(args: Namespace) -> Dict[str, Any]: """ - Prepare the configuration for the backtesting + Prepare the configuration for edge backtesting :param args: Cli args from Arguments() :return: Configuration """ diff --git a/freqtrade/fiat_convert.py b/freqtrade/rpc/fiat_convert.py similarity index 100% rename from freqtrade/fiat_convert.py rename to freqtrade/rpc/fiat_convert.py diff --git a/freqtrade/rpc/rpc.py b/freqtrade/rpc/rpc.py index 6ca32e4be..e83d9d41b 100644 --- a/freqtrade/rpc/rpc.py +++ b/freqtrade/rpc/rpc.py @@ -14,9 +14,9 @@ from numpy import mean, nan_to_num, NAN from pandas import DataFrame from freqtrade import TemporaryError, DependencyException -from freqtrade.fiat_convert import CryptoToFiatConverter from freqtrade.misc import shorten_date from freqtrade.persistence import Trade +from freqtrade.rpc.fiat_convert import CryptoToFiatConverter from freqtrade.state import State from freqtrade.strategy.interface import SellType diff --git a/freqtrade/rpc/telegram.py b/freqtrade/rpc/telegram.py index 56c58f2cc..be2498d78 100644 --- a/freqtrade/rpc/telegram.py +++ b/freqtrade/rpc/telegram.py @@ -12,8 +12,8 @@ from telegram.error import NetworkError, TelegramError from telegram.ext import CommandHandler, Updater from freqtrade.__init__ import __version__ -from freqtrade.fiat_convert import CryptoToFiatConverter from freqtrade.rpc import RPC, RPCException, RPCMessageType +from freqtrade.rpc.fiat_convert import CryptoToFiatConverter logger = logging.getLogger(__name__) diff --git a/freqtrade/strategy/interface.py b/freqtrade/strategy/interface.py index 9c9f08399..785e3b9c3 100644 --- a/freqtrade/strategy/interface.py +++ b/freqtrade/strategy/interface.py @@ -6,7 +6,7 @@ import logging from abc import ABC, abstractmethod from datetime import datetime from enum import Enum -from typing import Dict, List, NamedTuple, Optional, Tuple +from typing import Dict, List, NamedTuple, Tuple import warnings import arrow @@ -128,19 +128,17 @@ class IStrategy(ABC): """ return self.__class__.__name__ - def analyze_ticker(self, ticker_history: List[Dict], metadata: dict) -> DataFrame: + def analyze_ticker(self, dataframe: DataFrame, metadata: dict) -> DataFrame: """ Parses the given ticker history and returns a populated DataFrame add several TA indicators and buy signal to it :return DataFrame with ticker data and indicator data """ - dataframe = parse_ticker_dataframe(ticker_history) - pair = str(metadata.get('pair')) # Test if seen this pair and last candle before. - # always run if process_only_new_candles is set to true + # always run if process_only_new_candles is set to false if (not self.process_only_new_candles or self._last_candle_seen_per_pair.get(pair, None) != dataframe.iloc[-1]['date']): # Defs that only make change on new candle data. @@ -161,19 +159,20 @@ class IStrategy(ABC): return dataframe def get_signal(self, pair: str, interval: str, - ticker_hist: Optional[List[Dict]]) -> Tuple[bool, bool]: + dataframe: DataFrame) -> Tuple[bool, bool]: """ Calculates current signal based several technical analysis indicators :param pair: pair in format ANT/BTC :param interval: Interval to use (in min) + :param dataframe: Dataframe to analyze :return: (Buy, Sell) A bool-tuple indicating buy/sell signal """ - if not ticker_hist: + if not isinstance(dataframe, DataFrame) or dataframe.empty: logger.warning('Empty ticker history for pair %s', pair) return False, False try: - dataframe = self.analyze_ticker(ticker_hist, {'pair': pair}) + dataframe = self.analyze_ticker(dataframe, {'pair': pair}) except ValueError as error: logger.warning( 'Unable to analyze ticker for pair %s: %s', diff --git a/freqtrade/tests/conftest.py b/freqtrade/tests/conftest.py index df1a1cdc4..59c8835b5 100644 --- a/freqtrade/tests/conftest.py +++ b/freqtrade/tests/conftest.py @@ -82,7 +82,6 @@ def get_patched_freqtradebot(mocker, config) -> FreqtradeBot: :param config: Config to pass to the bot :return: None """ - # mocker.patch('freqtrade.fiat_convert.Market', {'price_usd': 12345.0}) patch_coinmarketcap(mocker, {'price_usd': 12345.0}) mocker.patch('freqtrade.freqtradebot.RPCManager', MagicMock()) mocker.patch('freqtrade.freqtradebot.persistence.init', MagicMock()) @@ -107,7 +106,7 @@ def patch_coinmarketcap(mocker, value: Optional[Dict[str, float]] = None) -> Non 'website_slug': 'ethereum'} ]}) mocker.patch.multiple( - 'freqtrade.fiat_convert.Market', + 'freqtrade.rpc.fiat_convert.Market', ticker=tickermock, listings=listmock, @@ -482,7 +481,7 @@ def order_book_l2(): @pytest.fixture -def ticker_history(): +def ticker_history_list(): return [ [ 1511686200000, # unix timestamp ms @@ -511,6 +510,11 @@ def ticker_history(): ] +@pytest.fixture +def ticker_history(ticker_history_list): + return parse_ticker_dataframe(ticker_history_list) + + @pytest.fixture def tickers(): return MagicMock(return_value={ diff --git a/freqtrade/tests/exchange/test_exchange.py b/freqtrade/tests/exchange/test_exchange.py index 02314b32c..647440223 100644 --- a/freqtrade/tests/exchange/test_exchange.py +++ b/freqtrade/tests/exchange/test_exchange.py @@ -9,6 +9,7 @@ from unittest.mock import Mock, MagicMock, PropertyMock import arrow import ccxt import pytest +from pandas import DataFrame from freqtrade import DependencyException, OperationalException, TemporaryError from freqtrade.exchange import API_RETRY_COUNT, Exchange @@ -780,12 +781,20 @@ def test_get_history(default_conf, mocker, caplog): def test_refresh_tickers(mocker, default_conf, caplog) -> None: tick = [ [ - 1511686200000, # unix timestamp ms + (arrow.utcnow().timestamp - 1) * 1000, # unix timestamp ms 1, # open 2, # high 3, # low 4, # close 5, # volume (in quote currency) + ], + [ + arrow.utcnow().timestamp * 1000, # unix timestamp ms + 3, # open + 1, # high + 4, # low + 6, # close + 5, # volume (in quote currency) ] ] @@ -795,13 +804,21 @@ def test_refresh_tickers(mocker, default_conf, caplog) -> None: pairs = ['IOTA/ETH', 'XRP/ETH'] # empty dicts - assert not exchange.klines + assert not exchange._klines exchange.refresh_tickers(['IOTA/ETH', 'XRP/ETH'], '5m') assert log_has(f'Refreshing klines for {len(pairs)} pairs', caplog.record_tuples) - assert exchange.klines + assert exchange._klines + assert exchange._api_async.fetch_ohlcv.call_count == 2 for pair in pairs: - assert exchange.klines[pair] + assert isinstance(exchange.klines(pair), DataFrame) + assert len(exchange.klines(pair)) > 0 + + # test caching + exchange.refresh_tickers(['IOTA/ETH', 'XRP/ETH'], '5m') + + assert exchange._api_async.fetch_ohlcv.call_count == 2 + assert log_has(f"Using cached klines data for {pairs[0]} ...", caplog.record_tuples) @pytest.mark.asyncio @@ -831,10 +848,6 @@ async def test__async_get_candle_history(default_conf, mocker, caplog): assert res[1] == tick assert exchange._api_async.fetch_ohlcv.call_count == 1 assert not log_has(f"Using cached klines data for {pair} ...", caplog.record_tuples) - # test caching - res = await exchange._async_get_candle_history(pair, "5m") - assert exchange._api_async.fetch_ohlcv.call_count == 1 - assert log_has(f"Using cached klines data for {pair} ...", caplog.record_tuples) # exchange = Exchange(default_conf) await async_ccxt_exception(mocker, default_conf, MagicMock(), diff --git a/freqtrade/tests/exchange/test_exchange_helpers.py b/freqtrade/tests/exchange/test_exchange_helpers.py index 82525e805..57a24c69c 100644 --- a/freqtrade/tests/exchange/test_exchange_helpers.py +++ b/freqtrade/tests/exchange/test_exchange_helpers.py @@ -1,6 +1,8 @@ # pragma pylint: disable=missing-docstring, C0103 +import logging from freqtrade.exchange.exchange_helpers import parse_ticker_dataframe +from freqtrade.tests.conftest import log_has def test_dataframe_correct_length(result): @@ -13,9 +15,11 @@ def test_dataframe_correct_columns(result): ['date', 'open', 'high', 'low', 'close', 'volume'] -def test_parse_ticker_dataframe(ticker_history): +def test_parse_ticker_dataframe(ticker_history, caplog): columns = ['date', 'open', 'high', 'low', 'close', 'volume'] + caplog.set_level(logging.DEBUG) # Test file with BV data dataframe = parse_ticker_dataframe(ticker_history) assert dataframe.columns.tolist() == columns + assert log_has('Parsing tickerlist to dataframe', caplog.record_tuples) diff --git a/freqtrade/tests/optimize/test_backtesting.py b/freqtrade/tests/optimize/test_backtesting.py index e832e3a9b..4f80d618f 100644 --- a/freqtrade/tests/optimize/test_backtesting.py +++ b/freqtrade/tests/optimize/test_backtesting.py @@ -840,7 +840,7 @@ def test_backtest_start_live(default_conf, mocker, caplog): 'Using stake_currency: BTC ...', 'Using stake_amount: 0.001 ...', 'Downloading data for all pairs in whitelist ...', - 'Measuring data from 2017-11-14T19:31:00+00:00 up to 2017-11-14T22:58:00+00:00 (0 days)..', + 'Measuring data from 2017-11-14T19:31:00+00:00 up to 2017-11-14T22:57:00+00:00 (0 days)..', 'Parameter --enable-position-stacking detected ...' ] @@ -899,7 +899,7 @@ def test_backtest_start_multi_strat(default_conf, mocker, caplog): 'Using stake_currency: BTC ...', 'Using stake_amount: 0.001 ...', 'Downloading data for all pairs in whitelist ...', - 'Measuring data from 2017-11-14T19:31:00+00:00 up to 2017-11-14T22:58:00+00:00 (0 days)..', + 'Measuring data from 2017-11-14T19:31:00+00:00 up to 2017-11-14T22:57:00+00:00 (0 days)..', 'Parameter --enable-position-stacking detected ...', 'Running backtesting for Strategy DefaultStrategy', 'Running backtesting for Strategy TestStrategy', diff --git a/freqtrade/tests/optimize/test_optimize.py b/freqtrade/tests/optimize/test_optimize.py index d73f31ad5..dde76332b 100644 --- a/freqtrade/tests/optimize/test_optimize.py +++ b/freqtrade/tests/optimize/test_optimize.py @@ -85,11 +85,11 @@ def test_load_data_1min_ticker(ticker_history, mocker, caplog) -> None: _clean_test_file(file) -def test_load_data_with_new_pair_1min(ticker_history, mocker, caplog, default_conf) -> None: +def test_load_data_with_new_pair_1min(ticker_history_list, mocker, caplog, default_conf) -> None: """ Test load_data() with 1 min ticker """ - mocker.patch('freqtrade.exchange.Exchange.get_history', return_value=ticker_history) + mocker.patch('freqtrade.exchange.Exchange.get_history', return_value=ticker_history_list) exchange = get_patched_exchange(mocker, default_conf) file = os.path.join(os.path.dirname(__file__), '..', 'testdata', 'MEME_BTC-1m.json') @@ -119,8 +119,8 @@ def test_testdata_path() -> None: assert os.path.join('freqtrade', 'tests', 'testdata') in make_testdata_path(None) -def test_download_pairs(ticker_history, mocker, default_conf) -> None: - mocker.patch('freqtrade.exchange.Exchange.get_history', return_value=ticker_history) +def test_download_pairs(ticker_history_list, mocker, default_conf) -> None: + mocker.patch('freqtrade.exchange.Exchange.get_history', return_value=ticker_history_list) exchange = get_patched_exchange(mocker, default_conf) file1_1 = os.path.join(os.path.dirname(__file__), '..', 'testdata', 'MEME_BTC-1m.json') file1_5 = os.path.join(os.path.dirname(__file__), '..', 'testdata', 'MEME_BTC-5m.json') @@ -280,16 +280,18 @@ def test_download_pairs_exception(ticker_history, mocker, caplog, default_conf) assert log_has('Failed to download the pair: "MEME/BTC", Interval: 1m', caplog.record_tuples) -def test_download_backtesting_testdata(ticker_history, mocker, default_conf) -> None: - mocker.patch('freqtrade.exchange.Exchange.get_history', return_value=ticker_history) +def test_download_backtesting_testdata(ticker_history_list, mocker, default_conf) -> None: + mocker.patch('freqtrade.exchange.Exchange.get_history', return_value=ticker_history_list) exchange = get_patched_exchange(mocker, default_conf) - + # Tst that pairs-cached is not touched. + assert not exchange._pairs_last_refresh_time # Download a 1 min ticker file file1 = os.path.join(os.path.dirname(__file__), '..', 'testdata', 'XEL_BTC-1m.json') _backup_file(file1) download_backtesting_testdata(None, exchange, pair="XEL/BTC", tick_interval='1m') assert os.path.isfile(file1) is True _clean_test_file(file1) + assert not exchange._pairs_last_refresh_time # Download a 5 min ticker file file2 = os.path.join(os.path.dirname(__file__), '..', 'testdata', 'STORJ_BTC-5m.json') @@ -298,6 +300,7 @@ def test_download_backtesting_testdata(ticker_history, mocker, default_conf) -> download_backtesting_testdata(None, exchange, pair="STORJ/BTC", tick_interval='5m') assert os.path.isfile(file2) is True _clean_test_file(file2) + assert not exchange._pairs_last_refresh_time def test_download_backtesting_testdata2(mocker, default_conf) -> None: diff --git a/freqtrade/tests/test_fiat_convert.py b/freqtrade/tests/rpc/test_fiat_convert.py similarity index 91% rename from freqtrade/tests/test_fiat_convert.py rename to freqtrade/tests/rpc/test_fiat_convert.py index 8fd3b66b4..7d857d2f1 100644 --- a/freqtrade/tests/test_fiat_convert.py +++ b/freqtrade/tests/rpc/test_fiat_convert.py @@ -7,7 +7,7 @@ from unittest.mock import MagicMock import pytest from requests.exceptions import RequestException -from freqtrade.fiat_convert import CryptoFiat, CryptoToFiatConverter +from freqtrade.rpc.fiat_convert import CryptoFiat, CryptoToFiatConverter from freqtrade.tests.conftest import log_has, patch_coinmarketcap @@ -81,16 +81,18 @@ def test_fiat_convert_find_price(mocker): assert fiat_convert.get_price(crypto_symbol='XRP', fiat_symbol='USD') == 0.0 - mocker.patch('freqtrade.fiat_convert.CryptoToFiatConverter._find_price', return_value=12345.0) + mocker.patch('freqtrade.rpc.fiat_convert.CryptoToFiatConverter._find_price', + return_value=12345.0) assert fiat_convert.get_price(crypto_symbol='BTC', fiat_symbol='USD') == 12345.0 assert fiat_convert.get_price(crypto_symbol='btc', fiat_symbol='usd') == 12345.0 - mocker.patch('freqtrade.fiat_convert.CryptoToFiatConverter._find_price', return_value=13000.2) + mocker.patch('freqtrade.rpc.fiat_convert.CryptoToFiatConverter._find_price', + return_value=13000.2) assert fiat_convert.get_price(crypto_symbol='BTC', fiat_symbol='EUR') == 13000.2 def test_fiat_convert_unsupported_crypto(mocker, caplog): - mocker.patch('freqtrade.fiat_convert.CryptoToFiatConverter._cryptomap', return_value=[]) + mocker.patch('freqtrade.rpc.fiat_convert.CryptoToFiatConverter._cryptomap', return_value=[]) patch_coinmarketcap(mocker) fiat_convert = CryptoToFiatConverter() assert fiat_convert._find_price(crypto_symbol='CRYPTO_123', fiat_symbol='EUR') == 0.0 @@ -100,7 +102,8 @@ def test_fiat_convert_unsupported_crypto(mocker, caplog): def test_fiat_convert_get_price(mocker): patch_coinmarketcap(mocker) - mocker.patch('freqtrade.fiat_convert.CryptoToFiatConverter._find_price', return_value=28000.0) + mocker.patch('freqtrade.rpc.fiat_convert.CryptoToFiatConverter._find_price', + return_value=28000.0) fiat_convert = CryptoToFiatConverter() @@ -157,7 +160,7 @@ def test_fiat_init_network_exception(mocker): # Because CryptoToFiatConverter is a Singleton we reset the listings listmock = MagicMock(side_effect=RequestException) mocker.patch.multiple( - 'freqtrade.fiat_convert.Market', + 'freqtrade.rpc.fiat_convert.Market', listings=listmock, ) # with pytest.raises(RequestEsxception): @@ -187,7 +190,7 @@ def test_fiat_invalid_response(mocker, caplog): # Because CryptoToFiatConverter is a Singleton we reset the listings listmock = MagicMock(return_value="{'novalidjson':DEADBEEFf}") mocker.patch.multiple( - 'freqtrade.fiat_convert.Market', + 'freqtrade.rpc.fiat_convert.Market', listings=listmock, ) # with pytest.raises(RequestEsxception): @@ -203,7 +206,7 @@ def test_fiat_invalid_response(mocker, caplog): def test_convert_amount(mocker): patch_coinmarketcap(mocker) - mocker.patch('freqtrade.fiat_convert.CryptoToFiatConverter.get_price', return_value=12345.0) + mocker.patch('freqtrade.rpc.fiat_convert.CryptoToFiatConverter.get_price', return_value=12345.0) fiat_convert = CryptoToFiatConverter() result = fiat_convert.convert_amount( diff --git a/freqtrade/tests/rpc/test_rpc.py b/freqtrade/tests/rpc/test_rpc.py index 2b271af31..bb685cad5 100644 --- a/freqtrade/tests/rpc/test_rpc.py +++ b/freqtrade/tests/rpc/test_rpc.py @@ -8,10 +8,10 @@ import pytest from numpy import isnan from freqtrade import TemporaryError, DependencyException -from freqtrade.fiat_convert import CryptoToFiatConverter from freqtrade.freqtradebot import FreqtradeBot from freqtrade.persistence import Trade from freqtrade.rpc import RPC, RPCException +from freqtrade.rpc.fiat_convert import CryptoToFiatConverter from freqtrade.state import State from freqtrade.tests.test_freqtradebot import patch_get_signal from freqtrade.tests.conftest import patch_coinmarketcap, patch_exchange @@ -171,7 +171,7 @@ def test_rpc_daily_profit(default_conf, update, ticker, fee, def test_rpc_trade_statistics(default_conf, ticker, ticker_sell_up, fee, limit_buy_order, limit_sell_order, markets, mocker) -> None: mocker.patch.multiple( - 'freqtrade.fiat_convert.Market', + 'freqtrade.rpc.fiat_convert.Market', ticker=MagicMock(return_value={'price_usd': 15000.0}), ) patch_coinmarketcap(mocker) @@ -260,10 +260,11 @@ def test_rpc_trade_statistics_closed(mocker, default_conf, ticker, fee, markets, ticker_sell_up, limit_buy_order, limit_sell_order): patch_exchange(mocker) mocker.patch.multiple( - 'freqtrade.fiat_convert.Market', + 'freqtrade.rpc.fiat_convert.Market', ticker=MagicMock(return_value={'price_usd': 15000.0}), ) - mocker.patch('freqtrade.fiat_convert.CryptoToFiatConverter._find_price', return_value=15000.0) + mocker.patch('freqtrade.rpc.fiat_convert.CryptoToFiatConverter._find_price', + return_value=15000.0) mocker.patch('freqtrade.rpc.telegram.Telegram', MagicMock()) mocker.patch.multiple( 'freqtrade.exchange.Exchange', @@ -328,7 +329,7 @@ def test_rpc_balance_handle(default_conf, mocker): # ETH will be skipped due to mocked Error below mocker.patch.multiple( - 'freqtrade.fiat_convert.Market', + 'freqtrade.rpc.fiat_convert.Market', ticker=MagicMock(return_value={'price_usd': 15000.0}), ) patch_coinmarketcap(mocker) diff --git a/freqtrade/tests/rpc/test_rpc_telegram.py b/freqtrade/tests/rpc/test_rpc_telegram.py index cd4445a1e..686a92469 100644 --- a/freqtrade/tests/rpc/test_rpc_telegram.py +++ b/freqtrade/tests/rpc/test_rpc_telegram.py @@ -737,7 +737,8 @@ def test_forcesell_handle(default_conf, update, ticker, fee, def test_forcesell_down_handle(default_conf, update, ticker, fee, ticker_sell_down, markets, mocker) -> None: patch_coinmarketcap(mocker, value={'price_usd': 15000.0}) - mocker.patch('freqtrade.fiat_convert.CryptoToFiatConverter._find_price', return_value=15000.0) + mocker.patch('freqtrade.rpc.fiat_convert.CryptoToFiatConverter._find_price', + return_value=15000.0) rpc_mock = mocker.patch('freqtrade.rpc.telegram.Telegram.send_msg', MagicMock()) mocker.patch('freqtrade.rpc.telegram.Telegram._init', MagicMock()) mocker.patch.multiple( @@ -791,7 +792,8 @@ def test_forcesell_down_handle(default_conf, update, ticker, fee, def test_forcesell_all_handle(default_conf, update, ticker, fee, markets, mocker) -> None: patch_coinmarketcap(mocker, value={'price_usd': 15000.0}) patch_exchange(mocker) - mocker.patch('freqtrade.fiat_convert.CryptoToFiatConverter._find_price', return_value=15000.0) + mocker.patch('freqtrade.rpc.fiat_convert.CryptoToFiatConverter._find_price', + return_value=15000.0) rpc_mock = mocker.patch('freqtrade.rpc.telegram.Telegram.send_msg', MagicMock()) mocker.patch('freqtrade.rpc.telegram.Telegram._init', MagicMock()) mocker.patch('freqtrade.exchange.Exchange.get_pair_detail_url', MagicMock()) @@ -836,7 +838,8 @@ def test_forcesell_all_handle(default_conf, update, ticker, fee, markets, mocker def test_forcesell_handle_invalid(default_conf, update, mocker) -> None: patch_coinmarketcap(mocker, value={'price_usd': 15000.0}) - mocker.patch('freqtrade.fiat_convert.CryptoToFiatConverter._find_price', return_value=15000.0) + mocker.patch('freqtrade.rpc.fiat_convert.CryptoToFiatConverter._find_price', + return_value=15000.0) msg_mock = MagicMock() mocker.patch.multiple( 'freqtrade.rpc.telegram.Telegram', diff --git a/freqtrade/tests/strategy/test_interface.py b/freqtrade/tests/strategy/test_interface.py index 79c485590..d42296462 100644 --- a/freqtrade/tests/strategy/test_interface.py +++ b/freqtrade/tests/strategy/test_interface.py @@ -16,62 +16,69 @@ from freqtrade.strategy.default_strategy import DefaultStrategy _STRATEGY = DefaultStrategy(config={}) -def test_returns_latest_buy_signal(mocker, default_conf): +def test_returns_latest_buy_signal(mocker, default_conf, ticker_history): mocker.patch.object( _STRATEGY, 'analyze_ticker', return_value=DataFrame([{'buy': 1, 'sell': 0, 'date': arrow.utcnow()}]) ) - assert _STRATEGY.get_signal('ETH/BTC', '5m', MagicMock()) == (True, False) + assert _STRATEGY.get_signal('ETH/BTC', '5m', ticker_history) == (True, False) mocker.patch.object( _STRATEGY, 'analyze_ticker', return_value=DataFrame([{'buy': 0, 'sell': 1, 'date': arrow.utcnow()}]) ) - assert _STRATEGY.get_signal('ETH/BTC', '5m', MagicMock()) == (False, True) + assert _STRATEGY.get_signal('ETH/BTC', '5m', ticker_history) == (False, True) -def test_returns_latest_sell_signal(mocker, default_conf): +def test_returns_latest_sell_signal(mocker, default_conf, ticker_history): mocker.patch.object( _STRATEGY, 'analyze_ticker', return_value=DataFrame([{'sell': 1, 'buy': 0, 'date': arrow.utcnow()}]) ) - assert _STRATEGY.get_signal('ETH/BTC', '5m', MagicMock()) == (False, True) + assert _STRATEGY.get_signal('ETH/BTC', '5m', ticker_history) == (False, True) mocker.patch.object( _STRATEGY, 'analyze_ticker', return_value=DataFrame([{'sell': 0, 'buy': 1, 'date': arrow.utcnow()}]) ) - assert _STRATEGY.get_signal('ETH/BTC', '5m', MagicMock()) == (True, False) + assert _STRATEGY.get_signal('ETH/BTC', '5m', ticker_history) == (True, False) def test_get_signal_empty(default_conf, mocker, caplog): assert (False, False) == _STRATEGY.get_signal('foo', default_conf['ticker_interval'], - None) + DataFrame()) assert log_has('Empty ticker history for pair foo', caplog.record_tuples) + caplog.clear() + + assert (False, False) == _STRATEGY.get_signal('bar', default_conf['ticker_interval'], + []) + assert log_has('Empty ticker history for pair bar', caplog.record_tuples) -def test_get_signal_exception_valueerror(default_conf, mocker, caplog): +def test_get_signal_exception_valueerror(default_conf, mocker, caplog, ticker_history): caplog.set_level(logging.INFO) mocker.patch.object( _STRATEGY, 'analyze_ticker', side_effect=ValueError('xyz') ) - assert (False, False) == _STRATEGY.get_signal('foo', default_conf['ticker_interval'], 1) + assert (False, False) == _STRATEGY.get_signal('foo', default_conf['ticker_interval'], + ticker_history) assert log_has('Unable to analyze ticker for pair foo: xyz', caplog.record_tuples) -def test_get_signal_empty_dataframe(default_conf, mocker, caplog): +def test_get_signal_empty_dataframe(default_conf, mocker, caplog, ticker_history): caplog.set_level(logging.INFO) mocker.patch.object( _STRATEGY, 'analyze_ticker', return_value=DataFrame([]) ) - assert (False, False) == _STRATEGY.get_signal('xyz', default_conf['ticker_interval'], 1) + assert (False, False) == _STRATEGY.get_signal('xyz', default_conf['ticker_interval'], + ticker_history) assert log_has('Empty dataframe for pair xyz', caplog.record_tuples) -def test_get_signal_old_dataframe(default_conf, mocker, caplog): +def test_get_signal_old_dataframe(default_conf, mocker, caplog, ticker_history): caplog.set_level(logging.INFO) # default_conf defines a 5m interval. we check interval * 2 + 5m # this is necessary as the last candle is removed (partial candles) by default @@ -81,7 +88,8 @@ def test_get_signal_old_dataframe(default_conf, mocker, caplog): _STRATEGY, 'analyze_ticker', return_value=DataFrame(ticks) ) - assert (False, False) == _STRATEGY.get_signal('xyz', default_conf['ticker_interval'], 1) + assert (False, False) == _STRATEGY.get_signal('xyz', default_conf['ticker_interval'], + ticker_history) assert log_has( 'Outdated history for pair xyz. Last tick is 16 minutes old', caplog.record_tuples diff --git a/freqtrade/tests/test_misc.py b/freqtrade/tests/test_misc.py index 26e0c5ee6..e405457a1 100644 --- a/freqtrade/tests/test_misc.py +++ b/freqtrade/tests/test_misc.py @@ -16,8 +16,8 @@ def test_shorten_date() -> None: assert shorten_date(str_data) == str_shorten_data -def test_datesarray_to_datetimearray(ticker_history): - dataframes = parse_ticker_dataframe(ticker_history) +def test_datesarray_to_datetimearray(ticker_history_list): + dataframes = parse_ticker_dataframe(ticker_history_list) dates = datesarray_to_datetimearray(dataframes['date']) assert isinstance(dates[0], datetime.datetime) diff --git a/requirements.txt b/requirements.txt index 72c416b60..6ad6cd2d4 100644 --- a/requirements.txt +++ b/requirements.txt @@ -1,9 +1,9 @@ -ccxt==1.18.13 -SQLAlchemy==1.2.14 +ccxt==1.18.37 +SQLAlchemy==1.2.15 python-telegram-bot==11.1.0 arrow==0.12.1 cachetools==3.0.0 -requests==2.20.1 +requests==2.21.0 urllib3==1.24.1 wrapt==1.10.11 pandas==0.23.4 @@ -13,7 +13,7 @@ scipy==1.1.0 jsonschema==2.6.0 numpy==1.15.4 TA-Lib==0.4.17 -pytest==4.0.1 +pytest==4.0.2 pytest-mock==1.10.0 pytest-asyncio==0.9.0 pytest-cov==2.6.0 diff --git a/scripts/plot_dataframe.py b/scripts/plot_dataframe.py index 8fd3a43bd..5a8a25309 100755 --- a/scripts/plot_dataframe.py +++ b/scripts/plot_dataframe.py @@ -139,7 +139,7 @@ def plot_analyzed_dataframe(args: Namespace) -> None: if args.live: logger.info('Downloading pair.') exchange.refresh_tickers([pair], tick_interval) - tickers[pair] = exchange.klines[pair] + tickers[pair] = exchange.klines(pair) else: tickers = optimize.load_data( datadir=_CONF.get("datadir"),