From f0456bb802257af9ddbabe088142b566908be00e Mon Sep 17 00:00:00 2001 From: Gerald Lonlas Date: Tue, 5 Jun 2018 00:22:44 -0700 Subject: [PATCH 01/64] Update the README structure --- README.md | 169 ++++++++++++++++++++----------------------- docs/installation.md | 23 ++++++ 2 files changed, 102 insertions(+), 90 deletions(-) diff --git a/README.md b/README.md index 24e01531c..94ebb7bf8 100644 --- a/README.md +++ b/README.md @@ -22,33 +22,10 @@ expect. We strongly recommend you to have coding and Python knowledge. Do not hesitate to read the source code and understand the mechanism of this bot. -## Table of Contents -- [Features](#features) -- [Quick start](#quick-start) -- [Documentations](https://github.com/freqtrade/freqtrade/blob/develop/docs/index.md) - - [Installation](https://github.com/freqtrade/freqtrade/blob/develop/docs/installation.md) - - [Configuration](https://github.com/freqtrade/freqtrade/blob/develop/docs/configuration.md) - - [Strategy Optimization](https://github.com/freqtrade/freqtrade/blob/develop/docs/bot-optimization.md) - - [Backtesting](https://github.com/freqtrade/freqtrade/blob/develop/docs/backtesting.md) - - [Hyperopt](https://github.com/freqtrade/freqtrade/blob/develop/docs/hyperopt.md) -- [Support](#support) - - [Help](#help--slack) - - [Bugs](#bugs--issues) - - [Feature Requests](#feature-requests) - - [Pull Requests](#pull-requests) -- [Basic Usage](#basic-usage) - - [Bot commands](#bot-commands) - - [Telegram RPC commands](#telegram-rpc-commands) -- [Requirements](#requirements) - - [Min hardware required](#min-hardware-required) - - [Software requirements](#software-requirements) - -## Branches -The project is currently setup in two main branches: -- `develop` - This branch has often new features, but might also cause -breaking changes. -- `master` - This branch contains the latest stable release. The bot -'should' be stable on this branch, and is generally well tested. +## Exchange marketplaces supported +- [X] [Bittrex](https://bittrex.com/) +- [X] [Binance](https://www.binance.com/) +- [ ] [113 others to tests](https://github.com/ccxt/ccxt/). _(We cannot guarantee they will work)_ ## Features - [x] **Based on Python 3.6+**: For botting on any operating system - @@ -65,74 +42,50 @@ strategy parameters with real exchange data. - [x] **Daily summary of profit/loss**: Provide a daily summary of your profit/loss. - [x] **Performance status report**: Provide a performance status of your current trades. -### Exchange marketplaces supported -- [X] [Bittrex](https://bittrex.com/) -- [X] [Binance](https://www.binance.com/) -- [ ] [113 others to tests](https://github.com/ccxt/ccxt/). _(We cannot guarantee they will work)_ +## Table of Contents +- [Quick start](#quick-start) +- [Documentations](https://github.com/freqtrade/freqtrade/blob/develop/docs/index.md) + - [Installation](https://github.com/freqtrade/freqtrade/blob/develop/docs/installation.md) + - [Configuration](https://github.com/freqtrade/freqtrade/blob/develop/docs/configuration.md) + - [Strategy Optimization](https://github.com/freqtrade/freqtrade/blob/develop/docs/bot-optimization.md) + - [Backtesting](https://github.com/freqtrade/freqtrade/blob/develop/docs/backtesting.md) + - [Hyperopt](https://github.com/freqtrade/freqtrade/blob/develop/docs/hyperopt.md) +- [Basic Usage](#basic-usage) + - [Bot commands](#bot-commands) + - [Telegram RPC commands](#telegram-rpc-commands) +- [Support](#support) + - [Help](#help--slack) + - [Bugs](#bugs--issues) + - [Feature Requests](#feature-requests) + - [Pull Requests](#pull-requests) +- [Requirements](#requirements) + - [Min hardware required](#min-hardware-required) + - [Software requirements](#software-requirements) ## Quick start -This quick start section is a very short explanation on how to test the -bot in dry-run. We invite you to read the -[bot documentation](https://github.com/freqtrade/freqtrade/blob/develop/docs/index.md) -to ensure you understand how the bot is working. - -### Easy installation -The script below will install all dependencies and help you to configure the bot. -```bash -./setup.sh --install -``` - -### Manual installation -The following steps are made for Linux/MacOS environment - -**1. Clone the repo** +Freqtrade provides a Linux/macOS script to install all dependencies and help you to configure the bot. ```bash git clone git@github.com:freqtrade/freqtrade.git git checkout develop cd freqtrade +./setup.sh --install ``` -**2. Create the config file** -Switch `"dry_run": true,` -```bash -cp config.json.example config.json -vi config.json -``` -**3. Build your docker image and run it** -```bash -docker build -t freqtrade . -docker run --rm -v /etc/localtime:/etc/localtime:ro -v `pwd`/config.json:/freqtrade/config.json -it freqtrade -``` +_Windows installation is explained in [Installation doc](https://github.com/freqtrade/freqtrade/blob/develop/docs/installation.md)_ -### Help / Slack -For any questions not covered by the documentation or for further -information about the bot, we encourage you to join our slack channel. -- [Click here to join Slack channel](https://join.slack.com/t/highfrequencybot/shared_invite/enQtMjQ5NTM0OTYzMzY3LWMxYzE3M2MxNDdjMGM3ZTYwNzFjMGIwZGRjNTc3ZGU3MGE3NzdmZGMwNmU3NDM5ZTNmM2Y3NjRiNzk4NmM4OGE). +## Documentation +We invite you to read the bot documentation to ensure you understand how the bot is working. +- [Index](https://github.com/freqtrade/freqtrade/blob/develop/docs/index.md) +- [Installation](https://github.com/freqtrade/freqtrade/blob/develop/docs/installation.md) +- [Configuration](https://github.com/freqtrade/freqtrade/blob/develop/docs/configuration.md) +- [Bot usage](https://github.com/freqtrade/freqtrade/blob/develop/docs/bot-usage.md) + - [How to run the bot](https://github.com/freqtrade/freqtrade/blob/develop/docs/bot-usage.md#bot-commands) + - [How to use Backtesting](https://github.com/freqtrade/freqtrade/blob/develop/docs/bot-usage.md#backtesting-commands) + - [How to use Hyperopt](https://github.com/freqtrade/freqtrade/blob/develop/docs/bot-usage.md#hyperopt-commands) +- [Strategy Optimization](https://github.com/freqtrade/freqtrade/blob/develop/docs/bot-optimization.md) +- [Backtesting](https://github.com/freqtrade/freqtrade/blob/develop/docs/backtesting.md) +- [Hyperopt](https://github.com/freqtrade/freqtrade/blob/develop/docs/hyperopt.md) -### [Bugs / Issues](https://github.com/freqtrade/freqtrade/issues?q=is%3Aissue) -If you discover a bug in the bot, please -[search our issue tracker](https://github.com/freqtrade/freqtrade/issues?q=is%3Aissue) -first. If it hasn't been reported, please -[create a new issue](https://github.com/freqtrade/freqtrade/issues/new) and -ensure you follow the template guide so that our team can assist you as -quickly as possible. - -### [Feature Requests](https://github.com/freqtrade/freqtrade/labels/enhancement) -Have you a great idea to improve the bot you want to share? Please, -first search if this feature was not [already discussed](https://github.com/freqtrade/freqtrade/labels/enhancement). -If it hasn't been requested, please -[create a new request](https://github.com/freqtrade/freqtrade/issues/new) -and ensure you follow the template guide so that it does not get lost -in the bug reports. - -### [Pull Requests](https://github.com/freqtrade/freqtrade/pulls) -Feel like our bot is missing a feature? We welcome your pull requests! -Please read our -[Contributing document](https://github.com/freqtrade/freqtrade/blob/develop/CONTRIBUTING.md) -to understand the requirements before sending your pull-requests. - -**Important:** Always create your PR against the `develop` branch, not -`master`. ## Basic Usage @@ -170,11 +123,7 @@ optional arguments: "tradesv3.dry_run.sqlite" instead of memory DB. Work only if dry_run is enabled. ``` -More details on: -- [How to run the bot](https://github.com/freqtrade/freqtrade/blob/develop/docs/bot-usage.md#bot-commands) -- [How to use Backtesting](https://github.com/freqtrade/freqtrade/blob/develop/docs/bot-usage.md#backtesting-commands) -- [How to use Hyperopt](https://github.com/freqtrade/freqtrade/blob/develop/docs/bot-usage.md#hyperopt-commands) - + ### Telegram RPC commands Telegram is not mandatory. However, this is a great way to control your bot. More details on our @@ -193,6 +142,46 @@ bot. More details on our - `/help`: Show help message - `/version`: Show version + +## Development branches +The project is currently setup in two main branches: +- `develop` - This branch has often new features, but might also cause +breaking changes. +- `master` - This branch contains the latest stable release. The bot +'should' be stable on this branch, and is generally well tested. + + +## Support +### Help / Slack +For any questions not covered by the documentation or for further +information about the bot, we encourage you to join our slack channel. +- [Click here to join Slack channel](https://join.slack.com/t/highfrequencybot/shared_invite/enQtMjQ5NTM0OTYzMzY3LWMxYzE3M2MxNDdjMGM3ZTYwNzFjMGIwZGRjNTc3ZGU3MGE3NzdmZGMwNmU3NDM5ZTNmM2Y3NjRiNzk4NmM4OGE). + +### [Bugs / Issues](https://github.com/freqtrade/freqtrade/issues?q=is%3Aissue) +If you discover a bug in the bot, please +[search our issue tracker](https://github.com/freqtrade/freqtrade/issues?q=is%3Aissue) +first. If it hasn't been reported, please +[create a new issue](https://github.com/freqtrade/freqtrade/issues/new) and +ensure you follow the template guide so that our team can assist you as +quickly as possible. + +### [Feature Requests](https://github.com/freqtrade/freqtrade/labels/enhancement) +Have you a great idea to improve the bot you want to share? Please, +first search if this feature was not [already discussed](https://github.com/freqtrade/freqtrade/labels/enhancement). +If it hasn't been requested, please +[create a new request](https://github.com/freqtrade/freqtrade/issues/new) +and ensure you follow the template guide so that it does not get lost +in the bug reports. + +### [Pull Requests](https://github.com/freqtrade/freqtrade/pulls) +Feel like our bot is missing a feature? We welcome your pull requests! +Please read our +[Contributing document](https://github.com/freqtrade/freqtrade/blob/develop/CONTRIBUTING.md) +to understand the requirements before sending your pull-requests. + +**Important:** Always create your PR against the `develop` branch, not +`master`. + ## Requirements ### Min hardware required diff --git a/docs/installation.md b/docs/installation.md index 9818529f6..9fec6d25e 100644 --- a/docs/installation.md +++ b/docs/installation.md @@ -8,6 +8,7 @@ To understand how to set up the bot please read the [Bot Configuration](https:// * [Table of Contents](#table-of-contents) * [Easy Installation - Linux Script](#easy-installation---linux-script) +* [Manual installation](#manual-installation) * [Automatic Installation - Docker](#automatic-installation---docker) * [Custom Linux MacOS Installation](#custom-installation) - [Requirements](#requirements) @@ -55,6 +56,28 @@ Reset parameter will hard reset your branch (only if you are on `master` or `dev Config parameter is a `config.json` configurator. This script will ask you questions to setup your bot and create your `config.json`. + +## Manual installation - Linux/MacOS +The following steps are made for Linux/MacOS environment + +**1. Clone the repo** +```bash +git clone git@github.com:freqtrade/freqtrade.git +git checkout develop +cd freqtrade +``` +**2. Create the config file** +Switch `"dry_run": true,` +```bash +cp config.json.example config.json +vi config.json +``` +**3. Build your docker image and run it** +```bash +docker build -t freqtrade . +docker run --rm -v /etc/localtime:/etc/localtime:ro -v `pwd`/config.json:/freqtrade/config.json -it freqtrade +``` + ------ ## Automatic Installation - Docker From fa00157d12f7faad9cd778d4d825019aa88e5ba4 Mon Sep 17 00:00:00 2001 From: xmatthias Date: Sat, 16 Jun 2018 13:42:25 +0200 Subject: [PATCH 02/64] Fix fiat_convert missing mockups --- freqtrade/tests/test_fiat_convert.py | 26 +++++++++++++------------- 1 file changed, 13 insertions(+), 13 deletions(-) diff --git a/freqtrade/tests/test_fiat_convert.py b/freqtrade/tests/test_fiat_convert.py index 24f0f776b..2fb9219ca 100644 --- a/freqtrade/tests/test_fiat_convert.py +++ b/freqtrade/tests/test_fiat_convert.py @@ -40,7 +40,8 @@ def test_pair_convertion_object(): assert pair_convertion.price == 30000.123 -def test_fiat_convert_is_supported(): +def test_fiat_convert_is_supported(mocker): + patch_coinmarketcap(mocker) fiat_convert = CryptoToFiatConverter() assert fiat_convert._is_supported_fiat(fiat='USD') is True assert fiat_convert._is_supported_fiat(fiat='usd') is True @@ -48,7 +49,9 @@ def test_fiat_convert_is_supported(): assert fiat_convert._is_supported_fiat(fiat='ABC') is False -def test_fiat_convert_add_pair(): +def test_fiat_convert_add_pair(mocker): + patch_coinmarketcap(mocker) + fiat_convert = CryptoToFiatConverter() pair_len = len(fiat_convert._pairs) @@ -70,11 +73,8 @@ def test_fiat_convert_add_pair(): def test_fiat_convert_find_price(mocker): - api_mock = MagicMock(return_value={ - 'price_usd': 12345.0, - 'price_eur': 13000.2 - }) - mocker.patch('freqtrade.fiat_convert.Market.ticker', api_mock) + patch_coinmarketcap(mocker) + fiat_convert = CryptoToFiatConverter() with pytest.raises(ValueError, match=r'The fiat ABC is not supported.'): @@ -92,17 +92,15 @@ def test_fiat_convert_find_price(mocker): def test_fiat_convert_unsupported_crypto(mocker, caplog): mocker.patch('freqtrade.fiat_convert.CryptoToFiatConverter._cryptomap', return_value=[]) + patch_coinmarketcap(mocker) fiat_convert = CryptoToFiatConverter() assert fiat_convert._find_price(crypto_symbol='CRYPTO_123', fiat_symbol='EUR') == 0.0 assert log_has('unsupported crypto-symbol CRYPTO_123 - returning 0.0', caplog.record_tuples) def test_fiat_convert_get_price(mocker): - api_mock = MagicMock(return_value={ - 'price_usd': 28000.0, - 'price_eur': 15000.0 - }) - mocker.patch('freqtrade.fiat_convert.Market.ticker', api_mock) + patch_coinmarketcap(mocker) + mocker.patch('freqtrade.fiat_convert.CryptoToFiatConverter._find_price', return_value=28000.0) fiat_convert = CryptoToFiatConverter() @@ -172,8 +170,9 @@ def test_fiat_init_network_exception(mocker): assert length_cryptomap == 0 -def test_fiat_convert_without_network(): +def test_fiat_convert_without_network(mocker): # Because CryptoToFiatConverter is a Singleton we reset the value of _coinmarketcap + patch_coinmarketcap(mocker) fiat_convert = CryptoToFiatConverter() @@ -186,6 +185,7 @@ def test_fiat_convert_without_network(): def test_convert_amount(mocker): + patch_coinmarketcap(mocker) mocker.patch('freqtrade.fiat_convert.CryptoToFiatConverter.get_price', return_value=12345.0) fiat_convert = CryptoToFiatConverter() From 7564f7e526cfa8a1f8bf04155d26e3e9e9eadade Mon Sep 17 00:00:00 2001 From: xmatthias Date: Sat, 16 Jun 2018 13:49:03 +0200 Subject: [PATCH 03/64] fix hyperopt test when no config.json exists --- freqtrade/tests/optimize/test_hyperopt.py | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/freqtrade/tests/optimize/test_hyperopt.py b/freqtrade/tests/optimize/test_hyperopt.py index f8dce5fd6..9b66a29e2 100644 --- a/freqtrade/tests/optimize/test_hyperopt.py +++ b/freqtrade/tests/optimize/test_hyperopt.py @@ -61,6 +61,10 @@ def test_start(mocker, default_conf, caplog) -> None: Test start() function """ start_mock = MagicMock() + mocker.patch( + 'freqtrade.configuration.Configuration._load_config_file', + lambda *args, **kwargs: default_conf + ) mocker.patch('freqtrade.optimize.hyperopt.Hyperopt.start', start_mock) mocker.patch('freqtrade.freqtradebot.exchange.validate_pairs', MagicMock()) From 972736f0abd34945b4505a351aece6e4436c82a2 Mon Sep 17 00:00:00 2001 From: xmatthias Date: Sat, 16 Jun 2018 20:55:35 +0200 Subject: [PATCH 04/64] increase test-coverate for configureation --- freqtrade/tests/test_configuration.py | 38 +++++++++++++++++++++++++++ 1 file changed, 38 insertions(+) diff --git a/freqtrade/tests/test_configuration.py b/freqtrade/tests/test_configuration.py index 212df2e96..019c0c09d 100644 --- a/freqtrade/tests/test_configuration.py +++ b/freqtrade/tests/test_configuration.py @@ -13,6 +13,7 @@ from jsonschema import ValidationError from freqtrade.arguments import Arguments from freqtrade.configuration import Configuration +from freqtrade.constants import DEFAULT_DB_PROD_URL, DEFAULT_DB_DRYRUN_URL from freqtrade.tests.conftest import log_has from freqtrade import OperationalException @@ -140,6 +141,43 @@ def test_load_config_with_params(default_conf, mocker) -> None: assert validated_conf.get('strategy_path') == '/some/path' assert validated_conf.get('db_url') == 'sqlite:///someurl' + conf = default_conf.copy() + conf["dry_run"] = False + del conf["db_url"] + mocker.patch('freqtrade.configuration.open', mocker.mock_open( + read_data=json.dumps(conf) + )) + + arglist = [ + '--dynamic-whitelist', '10', + '--strategy', 'TestStrategy', + '--strategy-path', '/some/path' + ] + args = Arguments(arglist, '').get_parsed_arg() + + configuration = Configuration(args) + validated_conf = configuration.load_config() + assert validated_conf.get('db_url') == DEFAULT_DB_PROD_URL + + # Test dry=run with ProdURL + conf = default_conf.copy() + conf["dry_run"] = True + conf["db_url"] = DEFAULT_DB_PROD_URL + mocker.patch('freqtrade.configuration.open', mocker.mock_open( + read_data=json.dumps(conf) + )) + + arglist = [ + '--dynamic-whitelist', '10', + '--strategy', 'TestStrategy', + '--strategy-path', '/some/path' + ] + args = Arguments(arglist, '').get_parsed_arg() + + configuration = Configuration(args) + validated_conf = configuration.load_config() + assert validated_conf.get('db_url') == DEFAULT_DB_DRYRUN_URL + def test_load_custom_strategy(default_conf, mocker) -> None: """ From 7cfd99d17f574de139ccd11e5cdd3b2a5b4ca124 Mon Sep 17 00:00:00 2001 From: xmatthias Date: Sat, 16 Jun 2018 21:00:45 +0200 Subject: [PATCH 05/64] exclude __main__.py from coveralls - if __name__ == '__main__' is close to untestable - and should do nothing other than calling another function. --- .coveragerc | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/.coveragerc b/.coveragerc index 95eea4f8f..80ed2af23 100644 --- a/.coveragerc +++ b/.coveragerc @@ -2,4 +2,5 @@ omit = scripts/* freqtrade/tests/* - freqtrade/vendor/* \ No newline at end of file + freqtrade/vendor/* + freatrade/__main__.py From 90a7fb603d50e52aced0cfaf1ed6221c469a8e10 Mon Sep 17 00:00:00 2001 From: xmatthias Date: Sat, 16 Jun 2018 21:28:41 +0200 Subject: [PATCH 06/64] fix typo in coverage-omit --- .coveragerc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.coveragerc b/.coveragerc index 80ed2af23..4bd5b63fa 100644 --- a/.coveragerc +++ b/.coveragerc @@ -3,4 +3,4 @@ omit = scripts/* freqtrade/tests/* freqtrade/vendor/* - freatrade/__main__.py + freqtrade/__main__.py From ad0549414b7434fbd31841dbffe36dbd34dce1ac Mon Sep 17 00:00:00 2001 From: xmatthias Date: Sun, 17 Jun 2018 11:34:12 +0200 Subject: [PATCH 07/64] Revert "also unit tests now need config.json" This reverts commit 7e2e7946c5b12abffd222420d74156208ef41e59. --- .travis.yml | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/.travis.yml b/.travis.yml index 3f041f5dd..88121945f 100644 --- a/.travis.yml +++ b/.travis.yml @@ -16,15 +16,16 @@ install: - pip install --upgrade flake8 coveralls pytest-random-order mypy - pip install -r requirements.txt - pip install -e . -- cp config.json.example config.json jobs: include: - script: - pytest --cov=freqtrade --cov-config=.coveragerc freqtrade/tests/ - coveralls - script: + - cp config.json.example config.json - python freqtrade/main.py --datadir freqtrade/tests/testdata backtesting - script: + - cp config.json.example config.json - python freqtrade/main.py --datadir freqtrade/tests/testdata hyperopt -e 5 - script: flake8 freqtrade - script: mypy freqtrade From fef267a0dcf8f9256961866a48dba763f1652699 Mon Sep 17 00:00:00 2001 From: pyup-bot Date: Sun, 17 Jun 2018 14:23:05 +0200 Subject: [PATCH 08/64] Update ccxt from 1.14.201 to 1.14.202 --- requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/requirements.txt b/requirements.txt index 7e01b10aa..9d572304d 100644 --- a/requirements.txt +++ b/requirements.txt @@ -1,4 +1,4 @@ -ccxt==1.14.201 +ccxt==1.14.202 SQLAlchemy==1.2.8 python-telegram-bot==10.1.0 arrow==0.12.1 From 21edcbdc2747e8ea7b31a8a61eec3420835725ec Mon Sep 17 00:00:00 2001 From: xmatthias Date: Sun, 17 Jun 2018 12:41:33 +0200 Subject: [PATCH 09/64] Refactor exchange to class --- freqtrade/analyze.py | 6 +- freqtrade/exchange/__init__.py | 658 +++++++++++++++--------------- freqtrade/freqtradebot.py | 49 +-- freqtrade/optimize/__init__.py | 6 +- freqtrade/optimize/backtesting.py | 8 +- 5 files changed, 356 insertions(+), 371 deletions(-) diff --git a/freqtrade/analyze.py b/freqtrade/analyze.py index f18ae291c..aa64f04e5 100644 --- a/freqtrade/analyze.py +++ b/freqtrade/analyze.py @@ -10,7 +10,7 @@ import arrow from pandas import DataFrame, to_datetime from freqtrade import constants -from freqtrade.exchange import get_ticker_history +from freqtrade.exchange import Exchange from freqtrade.persistence import Trade from freqtrade.strategy.resolver import StrategyResolver, IStrategy @@ -110,14 +110,14 @@ class Analyze(object): dataframe = self.populate_sell_trend(dataframe) return dataframe - def get_signal(self, pair: str, interval: str) -> Tuple[bool, bool]: + def get_signal(self, exchange: Exchange, pair: str, interval: str) -> Tuple[bool, bool]: """ Calculates current signal based several technical analysis indicators :param pair: pair in format ANT/BTC :param interval: Interval to use (in min) :return: (Buy, Sell) A bool-tuple indicating buy/sell signal """ - ticker_hist = get_ticker_history(pair, interval) + ticker_hist = exchange.get_ticker_history(pair, interval) if not ticker_hist: logger.warning('Empty ticker history for pair %s', pair) return False, False diff --git a/freqtrade/exchange/__init__.py b/freqtrade/exchange/__init__.py index 54d564f04..5ac5f8a0b 100644 --- a/freqtrade/exchange/__init__.py +++ b/freqtrade/exchange/__init__.py @@ -12,16 +12,8 @@ from freqtrade import constants, OperationalException, DependencyException, Temp logger = logging.getLogger(__name__) -# Current selected exchange -_API: ccxt.Exchange = None - -_CONF: Dict = {} API_RETRY_COUNT = 4 -_CACHED_TICKER: Dict[str, Any] = {} - -# Holds all open sell orders for dry_run -_DRY_RUN_OPEN_ORDERS: Dict[str, Any] = {} # Urls to exchange markets, insert quote and base with .format() _EXCHANGE_URLS = { @@ -74,364 +66,354 @@ def init_ccxt(exchange_config: dict) -> ccxt.Exchange: return api -def init(config: dict) -> None: - """ - Initializes this module with the given config, - it does basic validation whether the specified - exchange and pairs are valid. - :param config: config to use - :return: None - """ - global _CONF, _API +class Exchange(object): - _CONF.update(config) + # Current selected exchange + _API: ccxt.Exchange = None + _CONF: Dict = {} + _CACHED_TICKER: Dict[str, Any] = {} - if config['dry_run']: - logger.info('Instance is running with dry_run enabled') + # Holds all open sell orders for dry_run + _DRY_RUN_OPEN_ORDERS: Dict[str, Any] = {} - exchange_config = config['exchange'] - _API = init_ccxt(exchange_config) + def __init__(self, config: dict) -> None: + """ + Initializes this module with the given config, + it does basic validation whether the specified + exchange and pairs are valid. + :param config: config to use + :return: None + """ + self._API - logger.info('Using Exchange "%s"', get_name()) + self._CONF.update(config) - # Check if all pairs are available - validate_pairs(config['exchange']['pair_whitelist']) + if config['dry_run']: + logger.info('Instance is running with dry_run enabled') + exchange_config = config['exchange'] + self._API = init_ccxt(exchange_config) -def validate_pairs(pairs: List[str]) -> None: - """ - Checks if all given pairs are tradable on the current exchange. - Raises OperationalException if one pair is not available. - :param pairs: list of pairs - :return: None - """ + logger.info('Using Exchange "%s"', self.get_name()) - try: - markets = _API.load_markets() - except ccxt.BaseError as e: - logger.warning('Unable to validate pairs (assuming they are correct). Reason: %s', e) - return + # Check if all pairs are available + self.validate_pairs(config['exchange']['pair_whitelist']) - stake_cur = _CONF['stake_currency'] - for pair in pairs: - # Note: ccxt has BaseCurrency/QuoteCurrency format for pairs - # TODO: add a support for having coins in BTC/USDT format - if not pair.endswith(stake_cur): - raise OperationalException( - f'Pair {pair} not compatible with stake_currency: {stake_cur}') - if pair not in markets: - raise OperationalException( - f'Pair {pair} is not available at {get_name()}') + def get_name(self) -> str: + return self._API.name + def get_id(self) -> str: + return self._API.id -def exchange_has(endpoint: str) -> bool: - """ - Checks if exchange implements a specific API endpoint. - Wrapper around ccxt 'has' attribute - :param endpoint: Name of endpoint (e.g. 'fetchOHLCV', 'fetchTickers') - :return: bool - """ - return endpoint in _API.has and _API.has[endpoint] + def validate_pairs(self, pairs: List[str]) -> None: + """ + Checks if all given pairs are tradable on the current exchange. + Raises OperationalException if one pair is not available. + :param pairs: list of pairs + :return: None + """ - -def buy(pair: str, rate: float, amount: float) -> Dict: - if _CONF['dry_run']: - global _DRY_RUN_OPEN_ORDERS - order_id = f'dry_run_buy_{randint(0, 10**6)}' - _DRY_RUN_OPEN_ORDERS[order_id] = { - 'pair': pair, - 'price': rate, - 'amount': amount, - 'type': 'limit', - 'side': 'buy', - 'remaining': 0.0, - 'datetime': arrow.utcnow().isoformat(), - 'status': 'closed', - 'fee': None - } - return {'id': order_id} - - try: - return _API.create_limit_buy_order(pair, amount, rate) - except ccxt.InsufficientFunds as e: - raise DependencyException( - f'Insufficient funds to create limit buy order on market {pair}.' - f'Tried to buy amount {amount} at rate {rate} (total {rate*amount}).' - f'Message: {e}') - except ccxt.InvalidOrder as e: - raise DependencyException( - f'Could not create limit buy order on market {pair}.' - f'Tried to buy amount {amount} at rate {rate} (total {rate*amount}).' - f'Message: {e}') - except (ccxt.NetworkError, ccxt.ExchangeError) as e: - raise TemporaryError( - f'Could not place buy order due to {e.__class__.__name__}. Message: {e}') - except ccxt.BaseError as e: - raise OperationalException(e) - - -def sell(pair: str, rate: float, amount: float) -> Dict: - if _CONF['dry_run']: - global _DRY_RUN_OPEN_ORDERS - order_id = f'dry_run_sell_{randint(0, 10**6)}' - _DRY_RUN_OPEN_ORDERS[order_id] = { - 'pair': pair, - 'price': rate, - 'amount': amount, - 'type': 'limit', - 'side': 'sell', - 'remaining': 0.0, - 'datetime': arrow.utcnow().isoformat(), - 'status': 'closed' - } - return {'id': order_id} - - try: - return _API.create_limit_sell_order(pair, amount, rate) - except ccxt.InsufficientFunds as e: - raise DependencyException( - f'Insufficient funds to create limit sell order on market {pair}.' - f'Tried to sell amount {amount} at rate {rate} (total {rate*amount}).' - f'Message: {e}') - except ccxt.InvalidOrder as e: - raise DependencyException( - f'Could not create limit sell order on market {pair}.' - f'Tried to sell amount {amount} at rate {rate} (total {rate*amount}).' - f'Message: {e}') - except (ccxt.NetworkError, ccxt.ExchangeError) as e: - raise TemporaryError( - f'Could not place sell order due to {e.__class__.__name__}. Message: {e}') - except ccxt.BaseError as e: - raise OperationalException(e) - - -@retrier -def get_balance(currency: str) -> float: - if _CONF['dry_run']: - return 999.9 - - # ccxt exception is already handled by get_balances - balances = get_balances() - balance = balances.get(currency) - if balance is None: - raise TemporaryError( - f'Could not get {currency} balance due to malformed exchange response: {balances}') - return balance['free'] - - -@retrier -def get_balances() -> dict: - if _CONF['dry_run']: - return {} - - try: - balances = _API.fetch_balance() - # Remove additional info from ccxt results - balances.pop("info", None) - balances.pop("free", None) - balances.pop("total", None) - balances.pop("used", None) - - return balances - except (ccxt.NetworkError, ccxt.ExchangeError) as e: - raise TemporaryError( - f'Could not get balance due to {e.__class__.__name__}. Message: {e}') - except ccxt.BaseError as e: - raise OperationalException(e) - - -@retrier -def get_tickers() -> Dict: - try: - return _API.fetch_tickers() - except ccxt.NotSupported as e: - raise OperationalException( - f'Exchange {_API.name} does not support fetching tickers in batch.' - f'Message: {e}') - except (ccxt.NetworkError, ccxt.ExchangeError) as e: - raise TemporaryError( - f'Could not load tickers due to {e.__class__.__name__}. Message: {e}') - except ccxt.BaseError as e: - raise OperationalException(e) - - -@retrier -def get_ticker(pair: str, refresh: Optional[bool] = True) -> dict: - global _CACHED_TICKER - if refresh or pair not in _CACHED_TICKER.keys(): try: - data = _API.fetch_ticker(pair) + markets = self._API.load_markets() + except ccxt.BaseError as e: + logger.warning('Unable to validate pairs (assuming they are correct). Reason: %s', e) + return + + stake_cur = self._CONF['stake_currency'] + for pair in pairs: + # Note: ccxt has BaseCurrency/QuoteCurrency format for pairs + # TODO: add a support for having coins in BTC/USDT format + if not pair.endswith(stake_cur): + raise OperationalException( + f'Pair {pair} not compatible with stake_currency: {stake_cur}') + if pair not in markets: + raise OperationalException( + f'Pair {pair} is not available at {self.get_name()}') + + def exchange_has(self, endpoint: str) -> bool: + """ + Checks if exchange implements a specific API endpoint. + Wrapper around ccxt 'has' attribute + :param endpoint: Name of endpoint (e.g. 'fetchOHLCV', 'fetchTickers') + :return: bool + """ + return endpoint in self._API.has and self._API.has[endpoint] + + def buy(self, pair: str, rate: float, amount: float) -> Dict: + if self._CONF['dry_run']: + order_id = f'dry_run_buy_{randint(0, 10**6)}' + self._DRY_RUN_OPEN_ORDERS[order_id] = { + 'pair': pair, + 'price': rate, + 'amount': amount, + 'type': 'limit', + 'side': 'buy', + 'remaining': 0.0, + 'datetime': arrow.utcnow().isoformat(), + 'status': 'closed', + 'fee': None + } + return {'id': order_id} + + try: + return self._API.create_limit_buy_order(pair, amount, rate) + except ccxt.InsufficientFunds as e: + raise DependencyException( + f'Insufficient funds to create limit buy order on market {pair}.' + f'Tried to buy amount {amount} at rate {rate} (total {rate*amount}).' + f'Message: {e}') + except ccxt.InvalidOrder as e: + raise DependencyException( + f'Could not create limit buy order on market {pair}.' + f'Tried to buy amount {amount} at rate {rate} (total {rate*amount}).' + f'Message: {e}') + except (ccxt.NetworkError, ccxt.ExchangeError) as e: + raise TemporaryError( + f'Could not place buy order due to {e.__class__.__name__}. Message: {e}') + except ccxt.BaseError as e: + raise OperationalException(e) + + def sell(self, pair: str, rate: float, amount: float) -> Dict: + if self._CONF['dry_run']: + order_id = f'dry_run_sell_{randint(0, 10**6)}' + self._DRY_RUN_OPEN_ORDERS[order_id] = { + 'pair': pair, + 'price': rate, + 'amount': amount, + 'type': 'limit', + 'side': 'sell', + 'remaining': 0.0, + 'datetime': arrow.utcnow().isoformat(), + 'status': 'closed' + } + return {'id': order_id} + + try: + return self._API.create_limit_sell_order(pair, amount, rate) + except ccxt.InsufficientFunds as e: + raise DependencyException( + f'Insufficient funds to create limit sell order on market {pair}.' + f'Tried to sell amount {amount} at rate {rate} (total {rate*amount}).' + f'Message: {e}') + except ccxt.InvalidOrder as e: + raise DependencyException( + f'Could not create limit sell order on market {pair}.' + f'Tried to sell amount {amount} at rate {rate} (total {rate*amount}).' + f'Message: {e}') + except (ccxt.NetworkError, ccxt.ExchangeError) as e: + raise TemporaryError( + f'Could not place sell order due to {e.__class__.__name__}. Message: {e}') + except ccxt.BaseError as e: + raise OperationalException(e) + + @retrier + def get_balance(self, currency: str) -> float: + if self._CONF['dry_run']: + return 999.9 + + # ccxt exception is already handled by get_balances + balances = self.get_balances() + balance = balances.get(currency) + if balance is None: + raise TemporaryError( + f'Could not get {currency} balance due to malformed exchange response: {balances}') + return balance['free'] + + @retrier + def get_balances(self) -> dict: + if self._CONF['dry_run']: + return {} + + try: + balances = self._API.fetch_balance() + # Remove additional info from ccxt results + balances.pop("info", None) + balances.pop("free", None) + balances.pop("total", None) + balances.pop("used", None) + + return balances + except (ccxt.NetworkError, ccxt.ExchangeError) as e: + raise TemporaryError( + f'Could not get balance due to {e.__class__.__name__}. Message: {e}') + except ccxt.BaseError as e: + raise OperationalException(e) + + @retrier + def get_tickers(self) -> Dict: + try: + return self._API.fetch_tickers() + except ccxt.NotSupported as e: + raise OperationalException( + f'Exchange {self._API.name} does not support fetching tickers in batch.' + f'Message: {e}') + except (ccxt.NetworkError, ccxt.ExchangeError) as e: + raise TemporaryError( + f'Could not load tickers due to {e.__class__.__name__}. Message: {e}') + except ccxt.BaseError as e: + raise OperationalException(e) + + @retrier + def get_ticker(self, pair: str, refresh: Optional[bool] = True) -> dict: + if refresh or pair not in self._CACHED_TICKER.keys(): try: - _CACHED_TICKER[pair] = { - 'bid': float(data['bid']), - 'ask': float(data['ask']), - } - except KeyError: - logger.debug("Could not cache ticker data for %s", pair) + data = self._API.fetch_ticker(pair) + try: + self._CACHED_TICKER[pair] = { + 'bid': float(data['bid']), + 'ask': float(data['ask']), + } + except KeyError: + logger.debug("Could not cache ticker data for %s", pair) + return data + except (ccxt.NetworkError, ccxt.ExchangeError) as e: + raise TemporaryError( + f'Could not load ticker history due to {e.__class__.__name__}. Message: {e}') + except ccxt.BaseError as e: + raise OperationalException(e) + else: + logger.info("returning cached ticker-data for %s", pair) + return self._CACHED_TICKER[pair] + + @retrier + def get_ticker_history(self, pair: str, tick_interval: str, + since_ms: Optional[int] = None) -> List[Dict]: + try: + # last item should be in the time interval [now - tick_interval, now] + till_time_ms = arrow.utcnow().shift( + minutes=-constants.TICKER_INTERVAL_MINUTES[tick_interval] + ).timestamp * 1000 + # it looks as if some exchanges return cached data + # and they update it one in several minute, so 10 mins interval + # is necessary to skeep downloading of an empty array when all + # chached data was already downloaded + till_time_ms = min(till_time_ms, arrow.utcnow().shift(minutes=-10).timestamp * 1000) + + data: List[Dict[Any, Any]] = [] + while not since_ms or since_ms < till_time_ms: + data_part = self._API.fetch_ohlcv(pair, timeframe=tick_interval, since=since_ms) + + # Because some exchange sort Tickers ASC and other DESC. + # Ex: Bittrex returns a list of tickers ASC (oldest first, newest last) + # when GDAX returns a list of tickers DESC (newest first, oldest last) + data_part = sorted(data_part, key=lambda x: x[0]) + + if not data_part: + break + + logger.debug('Downloaded data for %s time range [%s, %s]', + pair, + arrow.get(data_part[0][0] / 1000).format(), + arrow.get(data_part[-1][0] / 1000).format()) + + data.extend(data_part) + since_ms = data[-1][0] + 1 + return data + except ccxt.NotSupported as e: + raise OperationalException( + f'Exchange {self._API.name} does not support fetching historical candlestick data.' + f'Message: {e}') except (ccxt.NetworkError, ccxt.ExchangeError) as e: raise TemporaryError( f'Could not load ticker history due to {e.__class__.__name__}. Message: {e}') + except ccxt.BaseError as e: + raise OperationalException(f'Could not fetch ticker data. Msg: {e}') + + @retrier + def cancel_order(self, order_id: str, pair: str) -> None: + if self._CONF['dry_run']: + return + + try: + return self._API.cancel_order(order_id, pair) + except ccxt.InvalidOrder as e: + raise DependencyException( + f'Could not cancel order. Message: {e}') + except (ccxt.NetworkError, ccxt.ExchangeError) as e: + raise TemporaryError( + f'Could not cancel order due to {e.__class__.__name__}. Message: {e}') except ccxt.BaseError as e: raise OperationalException(e) - else: - logger.info("returning cached ticker-data for %s", pair) - return _CACHED_TICKER[pair] + @retrier + def get_order(self, order_id: str, pair: str) -> Dict: + if self._CONF['dry_run']: + order = self._DRY_RUN_OPEN_ORDERS[order_id] + order.update({ + 'id': order_id + }) + return order + try: + return self._API.fetch_order(order_id, pair) + except ccxt.InvalidOrder as e: + raise DependencyException( + f'Could not get order. Message: {e}') + except (ccxt.NetworkError, ccxt.ExchangeError) as e: + raise TemporaryError( + f'Could not get order due to {e.__class__.__name__}. Message: {e}') + except ccxt.BaseError as e: + raise OperationalException(e) -@retrier -def get_ticker_history(pair: str, tick_interval: str, since_ms: Optional[int] = None) -> List[Dict]: - try: - # last item should be in the time interval [now - tick_interval, now] - till_time_ms = arrow.utcnow().shift( - minutes=-constants.TICKER_INTERVAL_MINUTES[tick_interval] - ).timestamp * 1000 - # it looks as if some exchanges return cached data - # and they update it one in several minute, so 10 mins interval - # is necessary to skeep downloading of an empty array when all - # chached data was already downloaded - till_time_ms = min(till_time_ms, arrow.utcnow().shift(minutes=-10).timestamp * 1000) + @retrier + def get_trades_for_order(self, order_id: str, pair: str, since: datetime) -> List: + if self._CONF['dry_run']: + return [] + if not self.exchange_has('fetchMyTrades'): + return [] + try: + my_trades = self._API.fetch_my_trades(pair, since.timestamp()) + matched_trades = [trade for trade in my_trades if trade['order'] == order_id] - data: List[Dict[Any, Any]] = [] - while not since_ms or since_ms < till_time_ms: - data_part = _API.fetch_ohlcv(pair, timeframe=tick_interval, since=since_ms) + return matched_trades - # Because some exchange sort Tickers ASC and other DESC. - # Ex: Bittrex returns a list of tickers ASC (oldest first, newest last) - # when GDAX returns a list of tickers DESC (newest first, oldest last) - data_part = sorted(data_part, key=lambda x: x[0]) + except ccxt.NetworkError as e: + raise TemporaryError( + f'Could not get trades due to networking error. Message: {e}') + except ccxt.BaseError as e: + raise OperationalException(e) - if not data_part: - break + def get_pair_detail_url(self, pair: str) -> str: + try: + url_base = self._API.urls.get('www') + base, quote = pair.split('/') - logger.debug('Downloaded data for %s time range [%s, %s]', - pair, - arrow.get(data_part[0][0] / 1000).format(), - arrow.get(data_part[-1][0] / 1000).format()) + return url_base + _EXCHANGE_URLS[self._API.id].format(base=base, quote=quote) + except KeyError: + logger.warning('Could not get exchange url for %s', self.get_name()) + return "" - data.extend(data_part) - since_ms = data[-1][0] + 1 + @retrier + def get_markets(self) -> List[dict]: + try: + return self._API.fetch_markets() + except (ccxt.NetworkError, ccxt.ExchangeError) as e: + raise TemporaryError( + f'Could not load markets due to {e.__class__.__name__}. Message: {e}') + except ccxt.BaseError as e: + raise OperationalException(e) - return data - except ccxt.NotSupported as e: - raise OperationalException( - f'Exchange {_API.name} does not support fetching historical candlestick data.' - f'Message: {e}') - except (ccxt.NetworkError, ccxt.ExchangeError) as e: - raise TemporaryError( - f'Could not load ticker history due to {e.__class__.__name__}. Message: {e}') - except ccxt.BaseError as e: - raise OperationalException(f'Could not fetch ticker data. Msg: {e}') + @retrier + def get_fee(self, symbol='ETH/BTC', type='', side='', amount=1, + price=1, taker_or_maker='maker') -> float: + try: + # validate that markets are loaded before trying to get fee + if self._API.markets is None or len(self._API.markets) == 0: + self._API.load_markets() + return self._API.calculate_fee(symbol=symbol, type=type, side=side, amount=amount, + price=price, takerOrMaker=taker_or_maker)['rate'] + except (ccxt.NetworkError, ccxt.ExchangeError) as e: + raise TemporaryError( + f'Could not get fee info due to {e.__class__.__name__}. Message: {e}') + except ccxt.BaseError as e: + raise OperationalException(e) -@retrier -def cancel_order(order_id: str, pair: str) -> None: - if _CONF['dry_run']: - return - - try: - return _API.cancel_order(order_id, pair) - except ccxt.InvalidOrder as e: - raise DependencyException( - f'Could not cancel order. Message: {e}') - except (ccxt.NetworkError, ccxt.ExchangeError) as e: - raise TemporaryError( - f'Could not cancel order due to {e.__class__.__name__}. Message: {e}') - except ccxt.BaseError as e: - raise OperationalException(e) - - -@retrier -def get_order(order_id: str, pair: str) -> Dict: - if _CONF['dry_run']: - order = _DRY_RUN_OPEN_ORDERS[order_id] - order.update({ - 'id': order_id - }) - return order - try: - return _API.fetch_order(order_id, pair) - except ccxt.InvalidOrder as e: - raise DependencyException( - f'Could not get order. Message: {e}') - except (ccxt.NetworkError, ccxt.ExchangeError) as e: - raise TemporaryError( - f'Could not get order due to {e.__class__.__name__}. Message: {e}') - except ccxt.BaseError as e: - raise OperationalException(e) - - -@retrier -def get_trades_for_order(order_id: str, pair: str, since: datetime) -> List: - if _CONF['dry_run']: - return [] - if not exchange_has('fetchMyTrades'): - return [] - try: - my_trades = _API.fetch_my_trades(pair, since.timestamp()) - matched_trades = [trade for trade in my_trades if trade['order'] == order_id] - - return matched_trades - - except ccxt.NetworkError as e: - raise TemporaryError( - f'Could not get trades due to networking error. Message: {e}') - except ccxt.BaseError as e: - raise OperationalException(e) - - -def get_pair_detail_url(pair: str) -> str: - try: - url_base = _API.urls.get('www') - base, quote = pair.split('/') - - return url_base + _EXCHANGE_URLS[_API.id].format(base=base, quote=quote) - except KeyError: - logger.warning('Could not get exchange url for %s', get_name()) - return "" - - -@retrier -def get_markets() -> List[dict]: - try: - return _API.fetch_markets() - except (ccxt.NetworkError, ccxt.ExchangeError) as e: - raise TemporaryError( - f'Could not load markets due to {e.__class__.__name__}. Message: {e}') - except ccxt.BaseError as e: - raise OperationalException(e) - - -def get_name() -> str: - return _API.name - - -def get_id() -> str: - return _API.id - - -@retrier -def get_fee(symbol='ETH/BTC', type='', side='', amount=1, - price=1, taker_or_maker='maker') -> float: - try: + def get_amount_lots(self, pair: str, amount: float) -> float: + """ + get buyable amount rounding, .. + """ # validate that markets are loaded before trying to get fee - if _API.markets is None or len(_API.markets) == 0: - _API.load_markets() - - return _API.calculate_fee(symbol=symbol, type=type, side=side, amount=amount, - price=price, takerOrMaker=taker_or_maker)['rate'] - except (ccxt.NetworkError, ccxt.ExchangeError) as e: - raise TemporaryError( - f'Could not get fee info due to {e.__class__.__name__}. Message: {e}') - except ccxt.BaseError as e: - raise OperationalException(e) - - -def get_amount_lots(pair: str, amount: float) -> float: - """ - get buyable amount rounding, .. - """ - # validate that markets are loaded before trying to get fee - if not _API.markets: - _API.load_markets() - return _API.amount_to_lots(pair, amount) + if not self._API.markets: + self._API.load_markets() + return self._API.amount_to_lots(pair, amount) diff --git a/freqtrade/freqtradebot.py b/freqtrade/freqtradebot.py index 157de862f..5f69b8115 100644 --- a/freqtrade/freqtradebot.py +++ b/freqtrade/freqtradebot.py @@ -14,11 +14,11 @@ import requests from cachetools import TTLCache, cached from freqtrade import ( - DependencyException, OperationalException, TemporaryError, - exchange, persistence, __version__, + DependencyException, OperationalException, TemporaryError, persistence, __version__, ) from freqtrade import constants from freqtrade.analyze import Analyze +from freqtrade.exchange import Exchange from freqtrade.fiat_convert import CryptoToFiatConverter from freqtrade.persistence import Trade from freqtrade.rpc.rpc_manager import RPCManager @@ -66,7 +66,7 @@ class FreqtradeBot(object): # Initialize all modules persistence.init(self.config) - exchange.init(self.config) + self.exchange = Exchange(self.config) # Set initial application state initial_state = self.config.get('initial_state') @@ -186,13 +186,13 @@ class FreqtradeBot(object): :return: List of pairs """ - if not exchange.exchange_has('fetchTickers'): + if not self.exchange.exchange_has('fetchTickers'): raise OperationalException( 'Exchange does not support dynamic whitelist.' 'Please edit your config and restart the bot' ) - tickers = exchange.get_tickers() + tickers = self.exchange.get_tickers() # check length so that we make sure that '/' is actually in the string tickers = [v for k, v in tickers.items() if len(k.split('/')) == 2 and k.split('/')[1] == base_currency] @@ -210,7 +210,7 @@ class FreqtradeBot(object): black_listed """ sanitized_whitelist = whitelist - markets = exchange.get_markets() + markets = self.exchange.get_markets() markets = [m for m in markets if m['quote'] == self.config['stake_currency']] known_pairs = set() @@ -255,7 +255,7 @@ class FreqtradeBot(object): interval = self.analyze.get_ticker_interval() stake_currency = self.config['stake_currency'] fiat_currency = self.config['fiat_display_currency'] - exc_name = exchange.get_name() + exc_name = self.exchange.get_name() logger.info( 'Checking buy signals to create a new trade with stake_amount: %f ...', @@ -263,7 +263,7 @@ class FreqtradeBot(object): ) whitelist = copy.deepcopy(self.config['exchange']['pair_whitelist']) # Check if stake_amount is fulfilled - if exchange.get_balance(stake_currency) < stake_amount: + if self.exchange.get_balance(stake_currency) < stake_amount: raise DependencyException( f'stake amount is not fulfilled (currency={stake_currency})') @@ -278,19 +278,19 @@ class FreqtradeBot(object): # Pick pair based on buy signals for _pair in whitelist: - (buy, sell) = self.analyze.get_signal(_pair, interval) + (buy, sell) = self.analyze.get_signal(self.exchange, _pair, interval) if buy and not sell: pair = _pair break else: return False pair_s = pair.replace('_', '/') - pair_url = exchange.get_pair_detail_url(pair) + pair_url = self.exchange.get_pair_detail_url(pair) # Calculate amount - buy_limit = self.get_target_bid(exchange.get_ticker(pair)) + buy_limit = self.get_target_bid(self.exchange.get_ticker(pair)) amount = stake_amount / buy_limit - order_id = exchange.buy(pair, buy_limit, amount)['id'] + order_id = self.exchange.buy(pair, buy_limit, amount)['id'] stake_amount_fiat = self.fiat_converter.convert_amount( stake_amount, @@ -305,7 +305,7 @@ with limit `{buy_limit:.8f} ({stake_amount:.6f} \ {stake_currency}, {stake_amount_fiat:.3f} {fiat_currency})`""" ) # Fee is applied twice because we make a LIMIT_BUY and LIMIT_SELL - fee = exchange.get_fee(symbol=pair, taker_or_maker='maker') + fee = self.exchange.get_fee(symbol=pair, taker_or_maker='maker') trade = Trade( pair=pair, stake_amount=stake_amount, @@ -315,7 +315,7 @@ with limit `{buy_limit:.8f} ({stake_amount:.6f} \ open_rate=buy_limit, open_rate_requested=buy_limit, open_date=datetime.utcnow(), - exchange=exchange.get_id(), + exchange=self.exchange.get_id(), open_order_id=order_id ) Trade.session.add(trade) @@ -348,7 +348,7 @@ with limit `{buy_limit:.8f} ({stake_amount:.6f} \ if trade.open_order_id: # Update trade with order values logger.info('Found open order for %s', trade) - order = exchange.get_order(trade.open_order_id, trade.pair) + order = self.exchange.get_order(trade.open_order_id, trade.pair) # Try update amount (binance-fix) try: new_amount = self.get_real_amount(trade, order) @@ -372,7 +372,7 @@ with limit `{buy_limit:.8f} ({stake_amount:.6f} \ def get_real_amount(self, trade: Trade, order: Dict) -> float: """ Get real amount for the trade - Necessary for exchanges which charge fees in base currency (e.g. binance) + Necessary for self.exchanges which charge fees in base currency (e.g. binance) """ order_amount = order['amount'] # Only run for closed orders @@ -388,7 +388,8 @@ with limit `{buy_limit:.8f} ({stake_amount:.6f} \ return new_amount # Fallback to Trades - trades = exchange.get_trades_for_order(trade.open_order_id, trade.pair, trade.open_date) + trades = self.exchange.get_trades_for_order(trade.open_order_id, trade.pair, + trade.open_date) if len(trades) == 0: logger.info("Applying fee on amount for %s failed: myTrade-Dict empty found", trade) @@ -420,7 +421,7 @@ with limit `{buy_limit:.8f} ({stake_amount:.6f} \ raise ValueError(f'attempt to handle closed trade: {trade}') logger.debug('Handling %s ...', trade) - current_rate = exchange.get_ticker(trade.pair)['bid'] + current_rate = self.exchange.get_ticker(trade.pair)['bid'] (buy, sell) = (False, False) @@ -449,7 +450,7 @@ with limit `{buy_limit:.8f} ({stake_amount:.6f} \ # updated via /forcesell in a different thread. if not trade.open_order_id: continue - order = exchange.get_order(trade.open_order_id, trade.pair) + order = self.exchange.get_order(trade.open_order_id, trade.pair) except requests.exceptions.RequestException: logger.info( 'Cannot query order for %s due to %s', @@ -475,7 +476,7 @@ with limit `{buy_limit:.8f} ({stake_amount:.6f} \ :return: True if order was fully cancelled """ pair_s = trade.pair.replace('_', '/') - exchange.cancel_order(trade.open_order_id, trade.pair) + self.exchange.cancel_order(trade.open_order_id, trade.pair) if order['remaining'] == order['amount']: # if trade is not partially completed, just delete the trade Trade.session.delete(trade) @@ -502,7 +503,7 @@ with limit `{buy_limit:.8f} ({stake_amount:.6f} \ pair_s = trade.pair.replace('_', '/') if order['remaining'] == order['amount']: # if trade is not partially completed, just cancel the trade - exchange.cancel_order(trade.open_order_id, trade.pair) + self.exchange.cancel_order(trade.open_order_id, trade.pair) trade.close_rate = None trade.close_profit = None trade.close_date = None @@ -525,15 +526,15 @@ with limit `{buy_limit:.8f} ({stake_amount:.6f} \ exc = trade.exchange pair = trade.pair # Execute sell and update trade record - order_id = exchange.sell(str(trade.pair), limit, trade.amount)['id'] + order_id = self.exchange.sell(str(trade.pair), limit, trade.amount)['id'] trade.open_order_id = order_id trade.close_rate_requested = limit fmt_exp_profit = round(trade.calc_profit_percent(rate=limit) * 100, 2) profit_trade = trade.calc_profit(rate=limit) - current_rate = exchange.get_ticker(trade.pair)['bid'] + current_rate = self.exchange.get_ticker(trade.pair)['bid'] profit = trade.calc_profit_percent(limit) - pair_url = exchange.get_pair_detail_url(trade.pair) + pair_url = self.exchange.get_pair_detail_url(trade.pair) gain = "profit" if fmt_exp_profit > 0 else "loss" message = f"*{exc}:* Selling\n" \ diff --git a/freqtrade/optimize/__init__.py b/freqtrade/optimize/__init__.py index 867e8c7dc..3e2fb4554 100644 --- a/freqtrade/optimize/__init__.py +++ b/freqtrade/optimize/__init__.py @@ -8,7 +8,7 @@ from typing import Optional, List, Dict, Tuple, Any import arrow from freqtrade import misc, constants -from freqtrade.exchange import get_ticker_history +from freqtrade.exchange import Exchange from freqtrade.arguments import TimeRange logger = logging.getLogger(__name__) @@ -183,6 +183,7 @@ def load_cached_data_for_updating(filename: str, def download_backtesting_testdata(datadir: str, + exchange: Exchange, pair: str, tick_interval: str = '5m', timerange: Optional[TimeRange] = None) -> None: @@ -216,7 +217,8 @@ def download_backtesting_testdata(datadir: str, logger.debug("Current Start: %s", misc.format_ms_time(data[1][0]) if data else 'None') logger.debug("Current End: %s", misc.format_ms_time(data[-1][0]) if data else 'None') - new_data = get_ticker_history(pair=pair, tick_interval=tick_interval, since_ms=since_ms) + new_data = exchange.get_ticker_history(pair=pair, tick_interval=tick_interval, + since_ms=since_ms) data.extend(new_data) logger.debug("New Start: %s", misc.format_ms_time(data[0][0])) diff --git a/freqtrade/optimize/backtesting.py b/freqtrade/optimize/backtesting.py index 23bd7741a..58b552237 100644 --- a/freqtrade/optimize/backtesting.py +++ b/freqtrade/optimize/backtesting.py @@ -14,7 +14,7 @@ from pandas import DataFrame from tabulate import tabulate import freqtrade.optimize as optimize -from freqtrade import exchange +from freqtrade.exchange import Exchange from freqtrade.analyze import Analyze from freqtrade.arguments import Arguments from freqtrade.configuration import Configuration @@ -61,7 +61,7 @@ class Backtesting(object): self.config['exchange']['password'] = '' self.config['exchange']['uid'] = '' self.config['dry_run'] = True - exchange.init(self.config) + self.exchange = Exchange(self.config) @staticmethod def get_timeframe(data: Dict[str, DataFrame]) -> Tuple[arrow.Arrow, arrow.Arrow]: @@ -130,7 +130,7 @@ class Backtesting(object): stake_amount = args['stake_amount'] max_open_trades = args.get('max_open_trades', 0) - fee = exchange.get_fee() + fee = self.exchange.get_fee() trade = Trade( open_rate=buy_row.close, open_date=buy_row.date, @@ -256,7 +256,7 @@ class Backtesting(object): if self.config.get('live'): logger.info('Downloading data for all pairs in whitelist ...') for pair in pairs: - data[pair] = exchange.get_ticker_history(pair, self.ticker_interval) + data[pair] = self.exchange.get_ticker_history(pair, self.ticker_interval) else: logger.info('Using local backtesting data (using whitelist in given config) ...') From dea26fadfe5499d5fd66391476864adc6bf71050 Mon Sep 17 00:00:00 2001 From: xmatthias Date: Sun, 17 Jun 2018 13:09:23 +0200 Subject: [PATCH 10/64] move init_ccxt to class --- freqtrade/exchange/__init__.py | 53 +++++++++++++++++----------------- 1 file changed, 26 insertions(+), 27 deletions(-) diff --git a/freqtrade/exchange/__init__.py b/freqtrade/exchange/__init__.py index 5ac5f8a0b..0d303fa4a 100644 --- a/freqtrade/exchange/__init__.py +++ b/freqtrade/exchange/__init__.py @@ -40,32 +40,6 @@ def retrier(f): return wrapper -def init_ccxt(exchange_config: dict) -> ccxt.Exchange: - """ - Initialize ccxt with given config and return valid - ccxt instance. - :param config: config to use - :return: ccxt - """ - # Find matching class for the given exchange name - name = exchange_config['name'] - - if name not in ccxt.exchanges: - raise OperationalException(f'Exchange {name} is not supported') - try: - api = getattr(ccxt, name.lower())({ - 'apiKey': exchange_config.get('key'), - 'secret': exchange_config.get('secret'), - 'password': exchange_config.get('password'), - 'uid': exchange_config.get('uid', ''), - 'enableRateLimit': True, - }) - except (KeyError, AttributeError): - raise OperationalException(f'Exchange {name} is not supported') - - return api - - class Exchange(object): # Current selected exchange @@ -92,13 +66,38 @@ class Exchange(object): logger.info('Instance is running with dry_run enabled') exchange_config = config['exchange'] - self._API = init_ccxt(exchange_config) + self._API = self._init_ccxt(exchange_config) logger.info('Using Exchange "%s"', self.get_name()) # Check if all pairs are available self.validate_pairs(config['exchange']['pair_whitelist']) + def _init_ccxt(self, exchange_config: dict) -> ccxt.Exchange: + """ + Initialize ccxt with given config and return valid + ccxt instance. + :param config: config to use + :return: ccxt + """ + # Find matching class for the given exchange name + name = exchange_config['name'] + + if name not in ccxt.exchanges: + raise OperationalException(f'Exchange {name} is not supported') + try: + api = getattr(ccxt, name.lower())({ + 'apiKey': exchange_config.get('key'), + 'secret': exchange_config.get('secret'), + 'password': exchange_config.get('password'), + 'uid': exchange_config.get('uid', ''), + 'enableRateLimit': True, + }) + except (KeyError, AttributeError): + raise OperationalException(f'Exchange {name} is not supported') + + return api + def get_name(self) -> str: return self._API.name From a159db68637bec69e4438c272d76e4ce1fa36eb2 Mon Sep 17 00:00:00 2001 From: xmatthias Date: Sun, 17 Jun 2018 13:32:05 +0200 Subject: [PATCH 11/64] get_exchange --- freqtrade/tests/conftest.py | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/freqtrade/tests/conftest.py b/freqtrade/tests/conftest.py index 1311687b7..ed900e6bb 100644 --- a/freqtrade/tests/conftest.py +++ b/freqtrade/tests/conftest.py @@ -13,6 +13,7 @@ from telegram import Chat, Message, Update from freqtrade.analyze import Analyze from freqtrade import constants +from freqtrade.exchange import Exchange from freqtrade.freqtradebot import FreqtradeBot logging.getLogger('').setLevel(logging.INFO) @@ -26,6 +27,17 @@ def log_has(line, logs): False) +def get_patched_exchange(mocker, config, api_mock=None) -> Exchange: + + mocker.patch('freqtrade.exchange.Exchange.validate_pairs', MagicMock()) + if api_mock: + mocker.patch('freqtrade.exchange.Exchange._init_ccxt', MagicMock(return_value=api_mock)) + else: + mocker.patch('freqtrade.exchange.Exchange._init_ccxt', MagicMock()) + exchange = Exchange(config) + return exchange + + # Functions for recurrent object patching def get_patched_freqtradebot(mocker, config) -> FreqtradeBot: """ From 67d345bc0859fac1184fae0552b9e967dd279ca6 Mon Sep 17 00:00:00 2001 From: xmatthias Date: Sun, 17 Jun 2018 19:54:51 +0200 Subject: [PATCH 12/64] fix tests for objectify exchange --- freqtrade/tests/test_acl_pair.py | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/freqtrade/tests/test_acl_pair.py b/freqtrade/tests/test_acl_pair.py index 07375260e..608e6a4a1 100644 --- a/freqtrade/tests/test_acl_pair.py +++ b/freqtrade/tests/test_acl_pair.py @@ -32,7 +32,7 @@ def test_refresh_market_pair_not_in_whitelist(mocker, markets): freqtradebot = tt.get_patched_freqtradebot(mocker, conf) - mocker.patch('freqtrade.freqtradebot.exchange.get_markets', markets) + mocker.patch('freqtrade.exchange.Exchange.get_markets', markets) refreshedwhitelist = freqtradebot._refresh_whitelist( conf['exchange']['pair_whitelist'] + ['XXX/BTC'] ) @@ -46,7 +46,7 @@ def test_refresh_whitelist(mocker, markets): conf = whitelist_conf() freqtradebot = tt.get_patched_freqtradebot(mocker, conf) - mocker.patch('freqtrade.freqtradebot.exchange.get_markets', markets) + mocker.patch('freqtrade.exchange.Exchange.get_markets', markets) refreshedwhitelist = freqtradebot._refresh_whitelist(conf['exchange']['pair_whitelist']) # List ordered by BaseVolume @@ -59,7 +59,7 @@ def test_refresh_whitelist_dynamic(mocker, markets, tickers): conf = whitelist_conf() freqtradebot = tt.get_patched_freqtradebot(mocker, conf) mocker.patch.multiple( - 'freqtrade.freqtradebot.exchange', + 'freqtrade.exchange.Exchange', get_markets=markets, get_tickers=tickers, exchange_has=MagicMock(return_value=True) @@ -78,7 +78,7 @@ def test_refresh_whitelist_dynamic(mocker, markets, tickers): def test_refresh_whitelist_dynamic_empty(mocker, markets_empty): conf = whitelist_conf() freqtradebot = tt.get_patched_freqtradebot(mocker, conf) - mocker.patch('freqtrade.freqtradebot.exchange.get_markets', markets_empty) + mocker.patch('freqtrade.exchange.Exchange.get_markets', markets_empty) # argument: use the whitelist dynamically by exchange-volume whitelist = [] From 68f6423d39519cf1f30dcb9baab49e82efba4f80 Mon Sep 17 00:00:00 2001 From: xmatthias Date: Sun, 17 Jun 2018 20:13:39 +0200 Subject: [PATCH 13/64] fix most tests --- freqtrade/tests/exchange/test_exchange.py | 283 ++++++++++------------ 1 file changed, 130 insertions(+), 153 deletions(-) diff --git a/freqtrade/tests/exchange/test_exchange.py b/freqtrade/tests/exchange/test_exchange.py index 97a723929..2479d26b0 100644 --- a/freqtrade/tests/exchange/test_exchange.py +++ b/freqtrade/tests/exchange/test_exchange.py @@ -8,28 +8,14 @@ from unittest.mock import MagicMock, PropertyMock import ccxt import pytest -import freqtrade.exchange as exchange from freqtrade import OperationalException, DependencyException, TemporaryError -from freqtrade.exchange import (init, validate_pairs, buy, sell, get_balance, get_balances, - get_ticker, get_ticker_history, cancel_order, get_name, get_fee, - get_id, get_pair_detail_url, get_amount_lots) -from freqtrade.tests.conftest import log_has - -API_INIT = False - - -def maybe_init_api(conf, mocker, force=False): - global API_INIT - if force or not API_INIT: - mocker.patch('freqtrade.exchange.validate_pairs', - side_effect=lambda s: True) - init(config=conf) - API_INIT = True +from freqtrade.exchange import Exchange, API_RETRY_COUNT +from freqtrade.tests.conftest import log_has, get_patched_exchange def test_init(default_conf, mocker, caplog): caplog.set_level(logging.INFO) - maybe_init_api(default_conf, mocker, True) + get_patched_exchange(mocker, default_conf) assert log_has('Instance is running with dry_run enabled', caplog.record_tuples) @@ -39,7 +25,7 @@ def test_init_exception(default_conf): with pytest.raises( OperationalException, match='Exchange {} is not supported'.format(default_conf['exchange']['name'])): - init(config=default_conf) + Exchange(default_conf) def test_validate_pairs(default_conf, mocker): @@ -50,18 +36,17 @@ def test_validate_pairs(default_conf, mocker): id_mock = PropertyMock(return_value='test_exchange') type(api_mock).id = id_mock - mocker.patch('freqtrade.exchange._API', api_mock) - mocker.patch.dict('freqtrade.exchange._CONF', default_conf) - validate_pairs(default_conf['exchange']['pair_whitelist']) + mocker.patch('freqtrade.exchange.Exchange._init_ccxt', MagicMock(return_value=api_mock)) + Exchange(default_conf) def test_validate_pairs_not_available(default_conf, mocker): api_mock = MagicMock() api_mock.load_markets = MagicMock(return_value={}) - mocker.patch('freqtrade.exchange._API', api_mock) - mocker.patch.dict('freqtrade.exchange._CONF', default_conf) + mocker.patch('freqtrade.exchange.Exchange._init_ccxt', MagicMock(return_value=api_mock)) + with pytest.raises(OperationalException, match=r'not available'): - validate_pairs(default_conf['exchange']['pair_whitelist']) + Exchange(default_conf) def test_validate_pairs_not_compatible(default_conf, mocker): @@ -71,25 +56,24 @@ def test_validate_pairs_not_compatible(default_conf, mocker): }) conf = deepcopy(default_conf) conf['stake_currency'] = 'ETH' - mocker.patch('freqtrade.exchange._API', api_mock) - mocker.patch.dict('freqtrade.exchange._CONF', conf) + mocker.patch('freqtrade.exchange.Exchange._init_ccxt', MagicMock(return_value=api_mock)) + with pytest.raises(OperationalException, match=r'not compatible'): - validate_pairs(conf['exchange']['pair_whitelist']) + Exchange(conf) def test_validate_pairs_exception(default_conf, mocker, caplog): caplog.set_level(logging.INFO) api_mock = MagicMock() api_mock.name = 'Binance' - mocker.patch('freqtrade.exchange._API', api_mock) - mocker.patch.dict('freqtrade.exchange._CONF', default_conf) - + mocker.patch('freqtrade.exchange.Exchange._init_ccxt', MagicMock()) + exchange = Exchange(default_conf) api_mock.load_markets = MagicMock(return_value={}) with pytest.raises(OperationalException, match=r'Pair ETH/BTC is not available at Binance'): - validate_pairs(default_conf['exchange']['pair_whitelist']) + exchange.validate_pairs(default_conf['exchange']['pair_whitelist']) api_mock.load_markets = MagicMock(side_effect=ccxt.BaseError()) - validate_pairs(default_conf['exchange']['pair_whitelist']) + exchange.validate_pairs(default_conf['exchange']['pair_whitelist']) assert log_has('Unable to validate pairs (assuming they are correct). Reason: ', caplog.record_tuples) @@ -100,21 +84,20 @@ def test_validate_pairs_stake_exception(default_conf, mocker, caplog): conf['stake_currency'] = 'ETH' api_mock = MagicMock() api_mock.name = 'binance' - mocker.patch('freqtrade.exchange._API', api_mock) - mocker.patch.dict('freqtrade.exchange._CONF', conf) + exchange = get_patched_exchange(mocker, default_conf, api_mock) with pytest.raises( OperationalException, match=r'Pair ETH/BTC not compatible with stake_currency: ETH' ): - validate_pairs(default_conf['exchange']['pair_whitelist']) + exchange.validate_pairs(default_conf['exchange']['pair_whitelist']) def test_buy_dry_run(default_conf, mocker): default_conf['dry_run'] = True - mocker.patch.dict('freqtrade.exchange._CONF', default_conf) + exchange = get_patched_exchange(mocker, default_conf) - order = buy(pair='ETH/BTC', rate=200, amount=1) + order = exchange.buy(pair='ETH/BTC', rate=200, amount=1) assert 'id' in order assert 'dry_run_buy_' in order['id'] @@ -128,12 +111,10 @@ def test_buy_prod(default_conf, mocker): 'foo': 'bar' } }) - mocker.patch('freqtrade.exchange._API', api_mock) - default_conf['dry_run'] = False - mocker.patch.dict('freqtrade.exchange._CONF', default_conf) + exchange = get_patched_exchange(mocker, default_conf, api_mock) - order = buy(pair='ETH/BTC', rate=200, amount=1) + order = exchange.buy(pair='ETH/BTC', rate=200, amount=1) assert 'id' in order assert 'info' in order assert order['id'] == order_id @@ -141,30 +122,30 @@ def test_buy_prod(default_conf, mocker): # test exception handling with pytest.raises(DependencyException): api_mock.create_limit_buy_order = MagicMock(side_effect=ccxt.InsufficientFunds) - mocker.patch('freqtrade.exchange._API', api_mock) - buy(pair='ETH/BTC', rate=200, amount=1) + exchange = get_patched_exchange(mocker, default_conf, api_mock) + exchange.buy(pair='ETH/BTC', rate=200, amount=1) with pytest.raises(DependencyException): api_mock.create_limit_buy_order = MagicMock(side_effect=ccxt.InvalidOrder) - mocker.patch('freqtrade.exchange._API', api_mock) - buy(pair='ETH/BTC', rate=200, amount=1) + exchange = get_patched_exchange(mocker, default_conf, api_mock) + exchange.buy(pair='ETH/BTC', rate=200, amount=1) with pytest.raises(TemporaryError): api_mock.create_limit_buy_order = MagicMock(side_effect=ccxt.NetworkError) - mocker.patch('freqtrade.exchange._API', api_mock) - buy(pair='ETH/BTC', rate=200, amount=1) + exchange = get_patched_exchange(mocker, default_conf, api_mock) + exchange.buy(pair='ETH/BTC', rate=200, amount=1) with pytest.raises(OperationalException): api_mock.create_limit_buy_order = MagicMock(side_effect=ccxt.BaseError) - mocker.patch('freqtrade.exchange._API', api_mock) - buy(pair='ETH/BTC', rate=200, amount=1) + exchange = get_patched_exchange(mocker, default_conf, api_mock) + exchange.buy(pair='ETH/BTC', rate=200, amount=1) def test_sell_dry_run(default_conf, mocker): default_conf['dry_run'] = True - mocker.patch.dict('freqtrade.exchange._CONF', default_conf) + exchange = get_patched_exchange(mocker, default_conf) - order = sell(pair='ETH/BTC', rate=200, amount=1) + order = exchange.sell(pair='ETH/BTC', rate=200, amount=1) assert 'id' in order assert 'dry_run_sell_' in order['id'] @@ -178,12 +159,11 @@ def test_sell_prod(default_conf, mocker): 'foo': 'bar' } }) - mocker.patch('freqtrade.exchange._API', api_mock) - default_conf['dry_run'] = False - mocker.patch.dict('freqtrade.exchange._CONF', default_conf) - order = sell(pair='ETH/BTC', rate=200, amount=1) + exchange = get_patched_exchange(mocker, default_conf, api_mock) + + order = exchange.sell(pair='ETH/BTC', rate=200, amount=1) assert 'id' in order assert 'info' in order assert order['id'] == order_id @@ -191,53 +171,52 @@ def test_sell_prod(default_conf, mocker): # test exception handling with pytest.raises(DependencyException): api_mock.create_limit_sell_order = MagicMock(side_effect=ccxt.InsufficientFunds) - mocker.patch('freqtrade.exchange._API', api_mock) - sell(pair='ETH/BTC', rate=200, amount=1) + exchange = get_patched_exchange(mocker, default_conf, api_mock) + exchange.sell(pair='ETH/BTC', rate=200, amount=1) with pytest.raises(DependencyException): api_mock.create_limit_sell_order = MagicMock(side_effect=ccxt.InvalidOrder) - mocker.patch('freqtrade.exchange._API', api_mock) - sell(pair='ETH/BTC', rate=200, amount=1) + exchange = get_patched_exchange(mocker, default_conf, api_mock) + exchange.sell(pair='ETH/BTC', rate=200, amount=1) with pytest.raises(TemporaryError): api_mock.create_limit_sell_order = MagicMock(side_effect=ccxt.NetworkError) - mocker.patch('freqtrade.exchange._API', api_mock) - sell(pair='ETH/BTC', rate=200, amount=1) + exchange = get_patched_exchange(mocker, default_conf, api_mock) + exchange.sell(pair='ETH/BTC', rate=200, amount=1) with pytest.raises(OperationalException): api_mock.create_limit_sell_order = MagicMock(side_effect=ccxt.BaseError) - mocker.patch('freqtrade.exchange._API', api_mock) - sell(pair='ETH/BTC', rate=200, amount=1) + exchange = get_patched_exchange(mocker, default_conf, api_mock) + exchange.sell(pair='ETH/BTC', rate=200, amount=1) def test_get_balance_dry_run(default_conf, mocker): default_conf['dry_run'] = True - mocker.patch.dict('freqtrade.exchange._CONF', default_conf) - assert get_balance(currency='BTC') == 999.9 + exchange = get_patched_exchange(mocker, default_conf) + assert exchange.get_balance(currency='BTC') == 999.9 def test_get_balance_prod(default_conf, mocker): api_mock = MagicMock() api_mock.fetch_balance = MagicMock(return_value={'BTC': {'free': 123.4}}) - mocker.patch('freqtrade.exchange._API', api_mock) - default_conf['dry_run'] = False - mocker.patch.dict('freqtrade.exchange._CONF', default_conf) - assert get_balance(currency='BTC') == 123.4 + exchange = get_patched_exchange(mocker, default_conf, api_mock) + + assert exchange.get_balance(currency='BTC') == 123.4 with pytest.raises(OperationalException): api_mock.fetch_balance = MagicMock(side_effect=ccxt.BaseError) - mocker.patch('freqtrade.exchange._API', api_mock) - get_balance(currency='BTC') + exchange = get_patched_exchange(mocker, default_conf, api_mock) + + exchange.get_balance(currency='BTC') def test_get_balances_dry_run(default_conf, mocker): default_conf['dry_run'] = True - mocker.patch.dict('freqtrade.exchange._CONF', default_conf) - - assert get_balances() == {} + exchange = get_patched_exchange(mocker, default_conf) + assert exchange.get_balances() == {} def test_get_balances_prod(default_conf, mocker): @@ -253,33 +232,29 @@ def test_get_balances_prod(default_conf, mocker): '2ST': balance_item, '3ST': balance_item }) - mocker.patch('freqtrade.exchange._API', api_mock) - default_conf['dry_run'] = False - mocker.patch.dict('freqtrade.exchange._CONF', default_conf) - - assert len(get_balances()) == 3 - assert get_balances()['1ST']['free'] == 10.0 - assert get_balances()['1ST']['total'] == 10.0 - assert get_balances()['1ST']['used'] == 0.0 + exchange = get_patched_exchange(mocker, default_conf, api_mock) + assert len(exchange.get_balances()) == 3 + assert exchange.get_balances()['1ST']['free'] == 10.0 + assert exchange.get_balances()['1ST']['total'] == 10.0 + assert exchange.get_balances()['1ST']['used'] == 0.0 with pytest.raises(TemporaryError): api_mock.fetch_balance = MagicMock(side_effect=ccxt.NetworkError) - mocker.patch('freqtrade.exchange._API', api_mock) - get_balances() - assert api_mock.fetch_balance.call_count == exchange.API_RETRY_COUNT + 1 + exchange = get_patched_exchange(mocker, default_conf, api_mock) + exchange.get_balances() + assert api_mock.fetch_balance.call_count == API_RETRY_COUNT + 1 with pytest.raises(OperationalException): api_mock.fetch_balance = MagicMock(side_effect=ccxt.BaseError) - mocker.patch('freqtrade.exchange._API', api_mock) - get_balances() + exchange = get_patched_exchange(mocker, default_conf, api_mock) + exchange.get_balances() assert api_mock.fetch_balance.call_count == 1 # This test is somewhat redundant with # test_exchange_bittrex.py::test_exchange_bittrex_get_ticker def test_get_ticker(default_conf, mocker): - maybe_init_api(default_conf, mocker) api_mock = MagicMock() tick = { 'symbol': 'ETH/BTC', @@ -288,10 +263,9 @@ def test_get_ticker(default_conf, mocker): 'last': 0.0001, } api_mock.fetch_ticker = MagicMock(return_value=tick) - mocker.patch('freqtrade.exchange._API', api_mock) - + exchange = get_patched_exchange(mocker, default_conf, api_mock) # retrieve original ticker - ticker = get_ticker(pair='ETH/BTC') + ticker = exchange.get_ticker(pair='ETH/BTC') assert ticker['bid'] == 0.00001098 assert ticker['ask'] == 0.00001099 @@ -304,11 +278,11 @@ def test_get_ticker(default_conf, mocker): 'last': 42, } api_mock.fetch_ticker = MagicMock(return_value=tick) - mocker.patch('freqtrade.exchange._API', api_mock) + exchange = get_patched_exchange(mocker, default_conf, api_mock) # if not caching the result we should get the same ticker # if not fetching a new result we should get the cached ticker - ticker = get_ticker(pair='ETH/BTC') + ticker = exchange.get_ticker(pair='ETH/BTC') assert api_mock.fetch_ticker.call_count == 1 assert ticker['bid'] == 0.5 @@ -320,22 +294,22 @@ def test_get_ticker(default_conf, mocker): # Test caching api_mock.fetch_ticker = MagicMock() - get_ticker(pair='ETH/BTC', refresh=False) + exchange.get_ticker(pair='ETH/BTC', refresh=False) assert api_mock.fetch_ticker.call_count == 0 with pytest.raises(TemporaryError): # test retrier api_mock.fetch_ticker = MagicMock(side_effect=ccxt.NetworkError) - mocker.patch('freqtrade.exchange._API', api_mock) - get_ticker(pair='ETH/BTC', refresh=True) + exchange = get_patched_exchange(mocker, default_conf, api_mock) + exchange.get_ticker(pair='ETH/BTC', refresh=True) with pytest.raises(OperationalException): api_mock.fetch_ticker = MagicMock(side_effect=ccxt.BaseError) - mocker.patch('freqtrade.exchange._API', api_mock) - get_ticker(pair='ETH/BTC', refresh=True) + exchange = get_patched_exchange(mocker, default_conf, api_mock) + exchange.get_ticker(pair='ETH/BTC', refresh=True) api_mock.fetch_ticker = MagicMock(return_value={}) - mocker.patch('freqtrade.exchange._API', api_mock) - get_ticker(pair='ETH/BTC', refresh=True) + exchange = get_patched_exchange(mocker, default_conf, api_mock) + exchange.get_ticker(pair='ETH/BTC', refresh=True) def make_fetch_ohlcv_mock(data): @@ -361,10 +335,10 @@ def test_get_ticker_history(default_conf, mocker): ] type(api_mock).has = PropertyMock(return_value={'fetchOHLCV': True}) api_mock.fetch_ohlcv = MagicMock(side_effect=make_fetch_ohlcv_mock(tick)) - mocker.patch('freqtrade.exchange._API', api_mock) + exchange = get_patched_exchange(mocker, default_conf, api_mock) # retrieve original ticker - ticks = get_ticker_history('ETH/BTC', default_conf['ticker_interval']) + ticks = exchange.get_ticker_history('ETH/BTC', default_conf['ticker_interval']) assert ticks[0][0] == 1511686200000 assert ticks[0][1] == 1 assert ticks[0][2] == 2 @@ -384,9 +358,9 @@ def test_get_ticker_history(default_conf, mocker): ] ] api_mock.fetch_ohlcv = MagicMock(side_effect=make_fetch_ohlcv_mock(new_tick)) - mocker.patch('freqtrade.exchange._API', api_mock) + exchange = get_patched_exchange(mocker, default_conf, api_mock) - ticks = get_ticker_history('ETH/BTC', default_conf['ticker_interval']) + ticks = exchange.get_ticker_history('ETH/BTC', default_conf['ticker_interval']) assert ticks[0][0] == 1511686210000 assert ticks[0][1] == 6 assert ticks[0][2] == 7 @@ -396,15 +370,15 @@ def test_get_ticker_history(default_conf, mocker): with pytest.raises(TemporaryError): # test retrier api_mock.fetch_ohlcv = MagicMock(side_effect=ccxt.NetworkError) - mocker.patch('freqtrade.exchange._API', api_mock) + exchange = get_patched_exchange(mocker, default_conf, api_mock) # new symbol to get around cache - get_ticker_history('ABCD/BTC', default_conf['ticker_interval']) + exchange.get_ticker_history('ABCD/BTC', default_conf['ticker_interval']) with pytest.raises(OperationalException): api_mock.fetch_ohlcv = MagicMock(side_effect=ccxt.BaseError) - mocker.patch('freqtrade.exchange._API', api_mock) + exchange = get_patched_exchange(mocker, default_conf, api_mock) # new symbol to get around cache - get_ticker_history('EFGH/BTC', default_conf['ticker_interval']) + exchange.get_ticker_history('EFGH/BTC', default_conf['ticker_interval']) def test_get_ticker_history_sort(default_conf, mocker): @@ -426,10 +400,11 @@ def test_get_ticker_history_sort(default_conf, mocker): ] type(api_mock).has = PropertyMock(return_value={'fetchOHLCV': True}) api_mock.fetch_ohlcv = MagicMock(side_effect=make_fetch_ohlcv_mock(tick)) - mocker.patch('freqtrade.exchange._API', api_mock) + + exchange = get_patched_exchange(mocker, default_conf, api_mock) # Test the ticker history sort - ticks = get_ticker_history('ETH/BTC', default_conf['ticker_interval']) + ticks = exchange.get_ticker_history('ETH/BTC', default_conf['ticker_interval']) assert ticks[0][0] == 1527830400000 assert ticks[0][1] == 0.07649 assert ticks[0][2] == 0.07651 @@ -460,10 +435,9 @@ def test_get_ticker_history_sort(default_conf, mocker): ] type(api_mock).has = PropertyMock(return_value={'fetchOHLCV': True}) api_mock.fetch_ohlcv = MagicMock(side_effect=make_fetch_ohlcv_mock(tick)) - mocker.patch('freqtrade.exchange._API', api_mock) - + exchange = get_patched_exchange(mocker, default_conf, api_mock) # Test the ticker history sort - ticks = get_ticker_history('ETH/BTC', default_conf['ticker_interval']) + ticks = exchange.get_ticker_history('ETH/BTC', default_conf['ticker_interval']) assert ticks[0][0] == 1527827700000 assert ticks[0][1] == 0.07659999 assert ticks[0][2] == 0.0766 @@ -481,114 +455,115 @@ def test_get_ticker_history_sort(default_conf, mocker): def test_cancel_order_dry_run(default_conf, mocker): default_conf['dry_run'] = True - mocker.patch.dict('freqtrade.exchange._CONF', default_conf) - - assert cancel_order(order_id='123', pair='TKN/BTC') is None + exchange = get_patched_exchange(mocker, default_conf) + assert exchange.cancel_order(order_id='123', pair='TKN/BTC') is None # Ensure that if not dry_run, we should call API def test_cancel_order(default_conf, mocker): default_conf['dry_run'] = False - mocker.patch.dict('freqtrade.exchange._CONF', default_conf) + # mocker.patch.dict('freqtrade.exchange.._CONF', default_conf) api_mock = MagicMock() api_mock.cancel_order = MagicMock(return_value=123) - mocker.patch('freqtrade.exchange._API', api_mock) - assert cancel_order(order_id='_', pair='TKN/BTC') == 123 + exchange = get_patched_exchange(mocker, default_conf, api_mock) + assert exchange.cancel_order(order_id='_', pair='TKN/BTC') == 123 with pytest.raises(TemporaryError): api_mock.cancel_order = MagicMock(side_effect=ccxt.NetworkError) - mocker.patch('freqtrade.exchange._API', api_mock) - cancel_order(order_id='_', pair='TKN/BTC') - assert api_mock.cancel_order.call_count == exchange.API_RETRY_COUNT + 1 + + exchange = get_patched_exchange(mocker, default_conf, api_mock) + exchange.cancel_order(order_id='_', pair='TKN/BTC') + assert api_mock.cancel_order.call_count == API_RETRY_COUNT + 1 with pytest.raises(DependencyException): api_mock.cancel_order = MagicMock(side_effect=ccxt.InvalidOrder) - mocker.patch('freqtrade.exchange._API', api_mock) - cancel_order(order_id='_', pair='TKN/BTC') - assert api_mock.cancel_order.call_count == exchange.API_RETRY_COUNT + 1 + exchange = get_patched_exchange(mocker, default_conf, api_mock) + exchange.cancel_order(order_id='_', pair='TKN/BTC') + assert api_mock.cancel_order.call_count == API_RETRY_COUNT + 1 with pytest.raises(OperationalException): api_mock.cancel_order = MagicMock(side_effect=ccxt.BaseError) - mocker.patch('freqtrade.exchange._API', api_mock) - cancel_order(order_id='_', pair='TKN/BTC') + exchange = get_patched_exchange(mocker, default_conf, api_mock) + exchange.cancel_order(order_id='_', pair='TKN/BTC') assert api_mock.cancel_order.call_count == 1 def test_get_order(default_conf, mocker): default_conf['dry_run'] = True - mocker.patch.dict('freqtrade.exchange._CONF', default_conf) + mocker.patch.dict('freqtrade.exchange.Exchange._CONF', default_conf) order = MagicMock() order.myid = 123 + exchange = Exchange(default_conf) exchange._DRY_RUN_OPEN_ORDERS['X'] = order print(exchange.get_order('X', 'TKN/BTC')) assert exchange.get_order('X', 'TKN/BTC').myid == 123 default_conf['dry_run'] = False - mocker.patch.dict('freqtrade.exchange._CONF', default_conf) + mocker.patch.dict('freqtrade.exchange.Exchange._CONF', default_conf) api_mock = MagicMock() api_mock.fetch_order = MagicMock(return_value=456) - mocker.patch('freqtrade.exchange._API', api_mock) + exchange = get_patched_exchange(mocker, default_conf, api_mock) assert exchange.get_order('X', 'TKN/BTC') == 456 with pytest.raises(TemporaryError): api_mock.fetch_order = MagicMock(side_effect=ccxt.NetworkError) - mocker.patch('freqtrade.exchange._API', api_mock) + exchange = get_patched_exchange(mocker, default_conf, api_mock) exchange.get_order(order_id='_', pair='TKN/BTC') - assert api_mock.fetch_order.call_count == exchange.API_RETRY_COUNT + 1 + assert api_mock.fetch_order.call_count == API_RETRY_COUNT + 1 with pytest.raises(DependencyException): api_mock.fetch_order = MagicMock(side_effect=ccxt.InvalidOrder) - mocker.patch('freqtrade.exchange._API', api_mock) + exchange = get_patched_exchange(mocker, default_conf, api_mock) exchange.get_order(order_id='_', pair='TKN/BTC') - assert api_mock.fetch_order.call_count == exchange.API_RETRY_COUNT + 1 + assert api_mock.fetch_order.call_count == API_RETRY_COUNT + 1 with pytest.raises(OperationalException): api_mock.fetch_order = MagicMock(side_effect=ccxt.BaseError) - mocker.patch('freqtrade.exchange._API', api_mock) + exchange = get_patched_exchange(mocker, default_conf, api_mock) exchange.get_order(order_id='_', pair='TKN/BTC') assert api_mock.fetch_order.call_count == 1 def test_get_name(default_conf, mocker): - mocker.patch('freqtrade.exchange.validate_pairs', + mocker.patch('freqtrade.exchange.Exchange.validate_pairs', side_effect=lambda s: True) default_conf['exchange']['name'] = 'binance' - init(default_conf) + exchange = Exchange(default_conf) - assert get_name() == 'Binance' + assert exchange.get_name() == 'Binance' def test_get_id(default_conf, mocker): - mocker.patch('freqtrade.exchange.validate_pairs', + mocker.patch('freqtrade.exchange.Exchange.validate_pairs', side_effect=lambda s: True) default_conf['exchange']['name'] = 'binance' - init(default_conf) - - assert get_id() == 'binance' + exchange = Exchange(default_conf) + assert exchange.get_id() == 'binance' def test_get_pair_detail_url(default_conf, mocker): - mocker.patch('freqtrade.exchange.validate_pairs', + mocker.patch('freqtrade.exchange.Exchange.validate_pairs', side_effect=lambda s: True) default_conf['exchange']['name'] = 'binance' - init(default_conf) + # exchange = get_patched_exchange(mocker, default_conf, api_mock) + exchange = Exchange(default_conf) - url = get_pair_detail_url('TKN/ETH') + url = exchange.get_pair_detail_url('TKN/ETH') assert 'TKN' in url assert 'ETH' in url - url = get_pair_detail_url('LOOONG/BTC') + url = exchange.get_pair_detail_url('LOOONG/BTC') assert 'LOOONG' in url assert 'BTC' in url default_conf['exchange']['name'] = 'bittrex' - init(default_conf) + exchange = Exchange(default_conf) - url = get_pair_detail_url('TKN/ETH') + url = exchange.get_pair_detail_url('TKN/ETH') assert 'TKN' in url assert 'ETH' in url - url = get_pair_detail_url('LOOONG/BTC') + url = exchange.get_pair_detail_url('LOOONG/BTC') assert 'LOOONG' in url assert 'BTC' in url @@ -601,12 +576,14 @@ def test_get_fee(default_conf, mocker): 'rate': 0.025, 'cost': 0.05 }) - mocker.patch('freqtrade.exchange._API', api_mock) - assert get_fee() == 0.025 + exchange = get_patched_exchange(mocker, default_conf, api_mock) + + assert exchange.get_fee() == 0.025 def test_get_amount_lots(default_conf, mocker): api_mock = MagicMock() api_mock.amount_to_lots = MagicMock(return_value=1.0) - mocker.patch('freqtrade.exchange._API', api_mock) - assert get_amount_lots('LTC/BTC', 1.54) == 1 + exchange = get_patched_exchange(mocker, default_conf, api_mock) + + assert exchange.get_amount_lots('LTC/BTC', 1.54) == 1 From 495f15f13c8ccf96e0409c1b49e5bdf659240526 Mon Sep 17 00:00:00 2001 From: xmatthias Date: Sun, 17 Jun 2018 20:35:52 +0200 Subject: [PATCH 14/64] fix exchange tests --- freqtrade/tests/exchange/test_exchange.py | 19 +++++++++++-------- 1 file changed, 11 insertions(+), 8 deletions(-) diff --git a/freqtrade/tests/exchange/test_exchange.py b/freqtrade/tests/exchange/test_exchange.py index 2479d26b0..9c23b36af 100644 --- a/freqtrade/tests/exchange/test_exchange.py +++ b/freqtrade/tests/exchange/test_exchange.py @@ -65,15 +65,18 @@ def test_validate_pairs_not_compatible(default_conf, mocker): def test_validate_pairs_exception(default_conf, mocker, caplog): caplog.set_level(logging.INFO) api_mock = MagicMock() - api_mock.name = 'Binance' - mocker.patch('freqtrade.exchange.Exchange._init_ccxt', MagicMock()) - exchange = Exchange(default_conf) + mocker.patch('freqtrade.exchange.Exchange.get_name', MagicMock(return_value='Binance')) + api_mock.load_markets = MagicMock(return_value={}) + mocker.patch('freqtrade.exchange.Exchange._init_ccxt', api_mock) + with pytest.raises(OperationalException, match=r'Pair ETH/BTC is not available at Binance'): - exchange.validate_pairs(default_conf['exchange']['pair_whitelist']) + Exchange(default_conf) api_mock.load_markets = MagicMock(side_effect=ccxt.BaseError()) - exchange.validate_pairs(default_conf['exchange']['pair_whitelist']) + + mocker.patch('freqtrade.exchange.Exchange._init_ccxt', MagicMock(return_value=api_mock)) + Exchange(default_conf) assert log_has('Unable to validate pairs (assuming they are correct). Reason: ', caplog.record_tuples) @@ -83,14 +86,14 @@ def test_validate_pairs_stake_exception(default_conf, mocker, caplog): conf = deepcopy(default_conf) conf['stake_currency'] = 'ETH' api_mock = MagicMock() - api_mock.name = 'binance' - exchange = get_patched_exchange(mocker, default_conf, api_mock) + api_mock.name = MagicMock(return_value='binance') + mocker.patch('freqtrade.exchange.Exchange._init_ccxt', api_mock) with pytest.raises( OperationalException, match=r'Pair ETH/BTC not compatible with stake_currency: ETH' ): - exchange.validate_pairs(default_conf['exchange']['pair_whitelist']) + Exchange(conf) def test_buy_dry_run(default_conf, mocker): From e8ab76f55bd771109f6418b75c1238406c216782 Mon Sep 17 00:00:00 2001 From: xmatthias Date: Sun, 17 Jun 2018 20:37:10 +0200 Subject: [PATCH 15/64] fix small in tests --- freqtrade/tests/exchange/test_exchange.py | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/freqtrade/tests/exchange/test_exchange.py b/freqtrade/tests/exchange/test_exchange.py index 9c23b36af..c66116ae0 100644 --- a/freqtrade/tests/exchange/test_exchange.py +++ b/freqtrade/tests/exchange/test_exchange.py @@ -493,16 +493,14 @@ def test_cancel_order(default_conf, mocker): def test_get_order(default_conf, mocker): default_conf['dry_run'] = True - mocker.patch.dict('freqtrade.exchange.Exchange._CONF', default_conf) order = MagicMock() order.myid = 123 - exchange = Exchange(default_conf) + exchange = get_patched_exchange(mocker, default_conf) exchange._DRY_RUN_OPEN_ORDERS['X'] = order print(exchange.get_order('X', 'TKN/BTC')) assert exchange.get_order('X', 'TKN/BTC').myid == 123 default_conf['dry_run'] = False - mocker.patch.dict('freqtrade.exchange.Exchange._CONF', default_conf) api_mock = MagicMock() api_mock.fetch_order = MagicMock(return_value=456) exchange = get_patched_exchange(mocker, default_conf, api_mock) From 082b6077e9390c8af983738d45749d7af992e7eb Mon Sep 17 00:00:00 2001 From: xmatthias Date: Sun, 17 Jun 2018 20:48:12 +0200 Subject: [PATCH 16/64] Fix tests analyze --- freqtrade/tests/test_analyze.py | 48 ++++++++++++++++++--------------- 1 file changed, 27 insertions(+), 21 deletions(-) diff --git a/freqtrade/tests/test_analyze.py b/freqtrade/tests/test_analyze.py index e8d0816aa..89dac21c6 100644 --- a/freqtrade/tests/test_analyze.py +++ b/freqtrade/tests/test_analyze.py @@ -14,7 +14,7 @@ from pandas import DataFrame from freqtrade.analyze import Analyze, SignalType from freqtrade.optimize.__init__ import load_tickerdata_file from freqtrade.arguments import TimeRange -from freqtrade.tests.conftest import log_has +from freqtrade.tests.conftest import log_has, get_patched_exchange # Avoid to reinit the same object again and again _ANALYZE = Analyze({'strategy': 'DefaultStrategy'}) @@ -66,16 +66,16 @@ def test_populates_sell_trend(result): assert 'sell' in dataframe.columns -def test_returns_latest_buy_signal(mocker): - mocker.patch('freqtrade.analyze.get_ticker_history', return_value=MagicMock()) - +def test_returns_latest_buy_signal(mocker, default_conf): + mocker.patch('freqtrade.exchange.Exchange.get_ticker_history', return_value=MagicMock()) + exchange = get_patched_exchange(mocker, default_conf) mocker.patch.multiple( 'freqtrade.analyze.Analyze', analyze_ticker=MagicMock( return_value=DataFrame([{'buy': 1, 'sell': 0, 'date': arrow.utcnow()}]) ) ) - assert _ANALYZE.get_signal('ETH/BTC', '5m') == (True, False) + assert _ANALYZE.get_signal(exchange, 'ETH/BTC', '5m') == (True, False) mocker.patch.multiple( 'freqtrade.analyze.Analyze', @@ -83,11 +83,12 @@ def test_returns_latest_buy_signal(mocker): return_value=DataFrame([{'buy': 0, 'sell': 1, 'date': arrow.utcnow()}]) ) ) - assert _ANALYZE.get_signal('ETH/BTC', '5m') == (False, True) + assert _ANALYZE.get_signal(exchange, 'ETH/BTC', '5m') == (False, True) -def test_returns_latest_sell_signal(mocker): - mocker.patch('freqtrade.analyze.get_ticker_history', return_value=MagicMock()) +def test_returns_latest_sell_signal(mocker, default_conf): + mocker.patch('freqtrade.exchange.Exchange.get_ticker_history', return_value=MagicMock()) + exchange = get_patched_exchange(mocker, default_conf) mocker.patch.multiple( 'freqtrade.analyze.Analyze', analyze_ticker=MagicMock( @@ -95,7 +96,7 @@ def test_returns_latest_sell_signal(mocker): ) ) - assert _ANALYZE.get_signal('ETH/BTC', '5m') == (False, True) + assert _ANALYZE.get_signal(exchange, 'ETH/BTC', '5m') == (False, True) mocker.patch.multiple( 'freqtrade.analyze.Analyze', @@ -103,45 +104,49 @@ def test_returns_latest_sell_signal(mocker): return_value=DataFrame([{'sell': 0, 'buy': 1, 'date': arrow.utcnow()}]) ) ) - assert _ANALYZE.get_signal('ETH/BTC', '5m') == (True, False) + assert _ANALYZE.get_signal(exchange, 'ETH/BTC', '5m') == (True, False) def test_get_signal_empty(default_conf, mocker, caplog): caplog.set_level(logging.INFO) - mocker.patch('freqtrade.analyze.get_ticker_history', return_value=None) - assert (False, False) == _ANALYZE.get_signal('foo', default_conf['ticker_interval']) + mocker.patch('freqtrade.exchange.Exchange.get_ticker_history', return_value=None) + exchange = get_patched_exchange(mocker, default_conf) + assert (False, False) == _ANALYZE.get_signal(exchange, 'foo', default_conf['ticker_interval']) assert log_has('Empty ticker history for pair foo', caplog.record_tuples) def test_get_signal_exception_valueerror(default_conf, mocker, caplog): caplog.set_level(logging.INFO) - mocker.patch('freqtrade.analyze.get_ticker_history', return_value=1) + mocker.patch('freqtrade.exchange.Exchange.get_ticker_history', return_value=1) + exchange = get_patched_exchange(mocker, default_conf) mocker.patch.multiple( 'freqtrade.analyze.Analyze', analyze_ticker=MagicMock( side_effect=ValueError('xyz') ) ) - assert (False, False) == _ANALYZE.get_signal('foo', default_conf['ticker_interval']) + assert (False, False) == _ANALYZE.get_signal(exchange, 'foo', default_conf['ticker_interval']) assert log_has('Unable to analyze ticker for pair foo: xyz', caplog.record_tuples) def test_get_signal_empty_dataframe(default_conf, mocker, caplog): caplog.set_level(logging.INFO) - mocker.patch('freqtrade.analyze.get_ticker_history', return_value=1) + mocker.patch('freqtrade.exchange.Exchange.get_ticker_history', return_value=1) + exchange = get_patched_exchange(mocker, default_conf) mocker.patch.multiple( 'freqtrade.analyze.Analyze', analyze_ticker=MagicMock( return_value=DataFrame([]) ) ) - assert (False, False) == _ANALYZE.get_signal('xyz', default_conf['ticker_interval']) + assert (False, False) == _ANALYZE.get_signal(exchange, 'xyz', default_conf['ticker_interval']) assert log_has('Empty dataframe for pair xyz', caplog.record_tuples) def test_get_signal_old_dataframe(default_conf, mocker, caplog): caplog.set_level(logging.INFO) - mocker.patch('freqtrade.analyze.get_ticker_history', return_value=1) + mocker.patch('freqtrade.exchange.Exchange.get_ticker_history', return_value=1) + exchange = get_patched_exchange(mocker, default_conf) # FIX: The get_signal function has hardcoded 10, which we must inturn hardcode oldtime = arrow.utcnow() - datetime.timedelta(minutes=11) ticks = DataFrame([{'buy': 1, 'date': oldtime}]) @@ -151,15 +156,16 @@ def test_get_signal_old_dataframe(default_conf, mocker, caplog): return_value=DataFrame(ticks) ) ) - assert (False, False) == _ANALYZE.get_signal('xyz', default_conf['ticker_interval']) + assert (False, False) == _ANALYZE.get_signal(exchange, 'xyz', default_conf['ticker_interval']) assert log_has( 'Outdated history for pair xyz. Last tick is 11 minutes old', caplog.record_tuples ) -def test_get_signal_handles_exceptions(mocker): - mocker.patch('freqtrade.analyze.get_ticker_history', return_value=MagicMock()) +def test_get_signal_handles_exceptions(mocker, default_conf): + mocker.patch('freqtrade.exchange.Exchange.get_ticker_history', return_value=MagicMock()) + exchange = get_patched_exchange(mocker, default_conf) mocker.patch.multiple( 'freqtrade.analyze.Analyze', analyze_ticker=MagicMock( @@ -167,7 +173,7 @@ def test_get_signal_handles_exceptions(mocker): ) ) - assert _ANALYZE.get_signal('ETH/BTC', '5m') == (False, False) + assert _ANALYZE.get_signal(exchange, 'ETH/BTC', '5m') == (False, False) def test_parse_ticker_dataframe(ticker_history): From 75d02df60dc6fbb347d30eaee63ec82427d399ec Mon Sep 17 00:00:00 2001 From: xmatthias Date: Sun, 17 Jun 2018 21:10:42 +0200 Subject: [PATCH 17/64] add exchange to call get_singal --- freqtrade/freqtradebot.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/freqtrade/freqtradebot.py b/freqtrade/freqtradebot.py index 5f69b8115..e708ba57f 100644 --- a/freqtrade/freqtradebot.py +++ b/freqtrade/freqtradebot.py @@ -426,7 +426,7 @@ with limit `{buy_limit:.8f} ({stake_amount:.6f} \ (buy, sell) = (False, False) if self.config.get('experimental', {}).get('use_sell_signal'): - (buy, sell) = self.analyze.get_signal(trade.pair, self.analyze.get_ticker_interval()) + (buy, sell) = self.analyze.get_signal(self.exchange, trade.pair, self.analyze.get_ticker_interval()) if self.analyze.should_sell(trade, current_rate, datetime.utcnow(), buy, sell): self.execute_sell(trade, current_rate) From 975b42caa37e320c6b602d7ee4964d8b56ec5dc8 Mon Sep 17 00:00:00 2001 From: xmatthias Date: Sun, 17 Jun 2018 21:11:10 +0200 Subject: [PATCH 18/64] fix tests for exchange objectify --- freqtrade/tests/test_freqtradebot.py | 133 ++++++++++++++------------- 1 file changed, 69 insertions(+), 64 deletions(-) diff --git a/freqtrade/tests/test_freqtradebot.py b/freqtrade/tests/test_freqtradebot.py index 1d272428e..797456328 100644 --- a/freqtrade/tests/test_freqtradebot.py +++ b/freqtrade/tests/test_freqtradebot.py @@ -32,7 +32,8 @@ def get_patched_freqtradebot(mocker, config) -> FreqtradeBot: mocker.patch('freqtrade.freqtradebot.Analyze', MagicMock()) mocker.patch('freqtrade.freqtradebot.RPCManager', MagicMock()) mocker.patch('freqtrade.freqtradebot.persistence.init', MagicMock()) - mocker.patch('freqtrade.freqtradebot.exchange.init', MagicMock()) + mocker.patch('freqtrade.exchange.Exchange._init_ccxt', MagicMock()) + mocker.patch('freqtrade.exchange.Exchange.validate_pairs', MagicMock()) patch_coinmarketcap(mocker) return FreqtradeBot(config) @@ -47,7 +48,7 @@ def patch_get_signal(mocker, value=(True, False)) -> None: """ mocker.patch( 'freqtrade.freqtradebot.Analyze.get_signal', - side_effect=lambda s, t: value + side_effect=lambda e, s, t: value ) @@ -187,9 +188,9 @@ def test_gen_pair_whitelist(mocker, default_conf, tickers) -> None: Test _gen_pair_whitelist() method """ freqtrade = get_patched_freqtradebot(mocker, default_conf) - mocker.patch('freqtrade.freqtradebot.exchange.get_tickers', tickers) - mocker.patch('freqtrade.freqtradebot.exchange.exchange_has', MagicMock(return_value=True)) - mocker.patch('freqtrade.exchange.validate_pairs', MagicMock(return_value=True)) + mocker.patch('freqtrade.exchange.Exchange.get_tickers', tickers) + mocker.patch('freqtrade.exchange.Exchange.exchange_has', MagicMock(return_value=True)) + # mocker.patch('freqtrade.exchange.validate_pairs', MagicMock(return_value=True)) # Test to retrieved BTC sorted on quoteVolume (default) whitelist = freqtrade._gen_pair_whitelist(base_currency='BTC') @@ -224,7 +225,7 @@ def test_create_trade(default_conf, ticker, limit_buy_order, fee, mocker) -> Non patch_RPCManager(mocker) patch_coinmarketcap(mocker) mocker.patch.multiple( - 'freqtrade.freqtradebot.exchange', + 'freqtrade.exchange.Exchange', validate_pairs=MagicMock(), get_ticker=ticker, buy=MagicMock(return_value={'id': limit_buy_order['id']}), @@ -261,7 +262,7 @@ def test_create_trade_minimal_amount(default_conf, ticker, limit_buy_order, fee, patch_coinmarketcap(mocker) buy_mock = MagicMock(return_value={'id': limit_buy_order['id']}) mocker.patch.multiple( - 'freqtrade.freqtradebot.exchange', + 'freqtrade.exchange.Exchange', validate_pairs=MagicMock(), get_ticker=ticker, buy=buy_mock, @@ -285,7 +286,7 @@ def test_create_trade_no_stake_amount(default_conf, ticker, limit_buy_order, fee patch_RPCManager(mocker) patch_coinmarketcap(mocker) mocker.patch.multiple( - 'freqtrade.freqtradebot.exchange', + 'freqtrade.exchange.Exchange', validate_pairs=MagicMock(), get_ticker=ticker, buy=MagicMock(return_value={'id': limit_buy_order['id']}), @@ -306,7 +307,7 @@ def test_create_trade_no_pairs(default_conf, ticker, limit_buy_order, fee, mocke patch_RPCManager(mocker) patch_coinmarketcap(mocker) mocker.patch.multiple( - 'freqtrade.freqtradebot.exchange', + 'freqtrade.exchange.Exchange', validate_pairs=MagicMock(), get_ticker=ticker, buy=MagicMock(return_value={'id': limit_buy_order['id']}), @@ -333,7 +334,7 @@ def test_create_trade_no_pairs_after_blacklist(default_conf, ticker, patch_RPCManager(mocker) patch_coinmarketcap(mocker) mocker.patch.multiple( - 'freqtrade.freqtradebot.exchange', + 'freqtrade.exchange.Exchange', validate_pairs=MagicMock(), get_ticker=ticker, buy=MagicMock(return_value={'id': limit_buy_order['id']}), @@ -362,7 +363,7 @@ def test_create_trade_no_signal(default_conf, fee, mocker) -> None: patch_RPCManager(mocker) patch_coinmarketcap(mocker) mocker.patch.multiple( - 'freqtrade.freqtradebot.exchange', + 'freqtrade.exchange.Exchange', validate_pairs=MagicMock(), get_ticker_history=MagicMock(return_value=20), get_balance=MagicMock(return_value=20), @@ -387,7 +388,7 @@ def test_process_trade_creation(default_conf, ticker, limit_buy_order, patch_RPCManager(mocker) patch_coinmarketcap(mocker, value={'price_usd': 12345.0}) mocker.patch.multiple( - 'freqtrade.freqtradebot.exchange', + 'freqtrade.exchange.Exchange', validate_pairs=MagicMock(), get_ticker=ticker, get_markets=markets, @@ -428,7 +429,7 @@ def test_process_exchange_failures(default_conf, ticker, markets, mocker) -> Non patch_RPCManager(mocker) patch_coinmarketcap(mocker, value={'price_usd': 12345.0}) mocker.patch.multiple( - 'freqtrade.freqtradebot.exchange', + 'freqtrade.exchange.Exchange', validate_pairs=MagicMock(), get_ticker=ticker, get_markets=markets, @@ -450,7 +451,7 @@ def test_process_operational_exception(default_conf, ticker, markets, mocker) -> msg_mock = patch_RPCManager(mocker) patch_coinmarketcap(mocker, value={'price_usd': 12345.0}) mocker.patch.multiple( - 'freqtrade.freqtradebot.exchange', + 'freqtrade.exchange.Exchange', validate_pairs=MagicMock(), get_ticker=ticker, get_markets=markets, @@ -474,7 +475,7 @@ def test_process_trade_handling( patch_RPCManager(mocker) patch_coinmarketcap(mocker, value={'price_usd': 12345.0}) mocker.patch.multiple( - 'freqtrade.freqtradebot.exchange', + 'freqtrade.exchange.Exchange', validate_pairs=MagicMock(), get_ticker=ticker, get_markets=markets, @@ -495,29 +496,32 @@ def test_process_trade_handling( assert result is False -def test_balance_fully_ask_side(mocker) -> None: +def test_balance_fully_ask_side(mocker, default_conf) -> None: """ Test get_target_bid() method """ - freqtrade = get_patched_freqtradebot(mocker, {'bid_strategy': {'ask_last_balance': 0.0}}) + default_conf['bid_strategy']['ask_last_balance'] = 0.0 + freqtrade = get_patched_freqtradebot(mocker, default_conf) assert freqtrade.get_target_bid({'ask': 20, 'last': 10}) == 20 -def test_balance_fully_last_side(mocker) -> None: +def test_balance_fully_last_side(mocker, default_conf) -> None: """ Test get_target_bid() method """ - freqtrade = get_patched_freqtradebot(mocker, {'bid_strategy': {'ask_last_balance': 1.0}}) + default_conf['bid_strategy']['ask_last_balance'] = 1.0 + freqtrade = get_patched_freqtradebot(mocker, default_conf) assert freqtrade.get_target_bid({'ask': 20, 'last': 10}) == 10 -def test_balance_bigger_last_ask(mocker) -> None: +def test_balance_bigger_last_ask(mocker, default_conf) -> None: """ Test get_target_bid() method """ - freqtrade = get_patched_freqtradebot(mocker, {'bid_strategy': {'ask_last_balance': 1.0}}) + default_conf['bid_strategy']['ask_last_balance'] = 1.0 + freqtrade = get_patched_freqtradebot(mocker, default_conf) assert freqtrade.get_target_bid({'ask': 5, 'last': 10}) == 5 @@ -556,8 +560,8 @@ def test_process_maybe_execute_sell(mocker, default_conf, limit_buy_order, caplo freqtrade = get_patched_freqtradebot(mocker, default_conf) mocker.patch('freqtrade.freqtradebot.FreqtradeBot.handle_trade', MagicMock(return_value=True)) - mocker.patch('freqtrade.freqtradebot.exchange.get_order', return_value=limit_buy_order) - mocker.patch('freqtrade.freqtradebot.exchange.get_trades_for_order', return_value=[]) + mocker.patch('freqtrade.exchange.Exchange.get_order', return_value=limit_buy_order) + mocker.patch('freqtrade.exchange.Exchange.get_trades_for_order', return_value=[]) mocker.patch('freqtrade.freqtradebot.FreqtradeBot.get_real_amount', return_value=limit_buy_order['amount']) @@ -590,7 +594,7 @@ def test_process_maybe_execute_sell_exception(mocker, default_conf, Test the exceptions in process_maybe_execute_sell() """ freqtrade = get_patched_freqtradebot(mocker, default_conf) - mocker.patch('freqtrade.freqtradebot.exchange.get_order', return_value=limit_buy_order) + mocker.patch('freqtrade.exchange.Exchange.get_order', return_value=limit_buy_order) trade = MagicMock() trade.open_order_id = '123' @@ -620,7 +624,7 @@ def test_handle_trade(default_conf, limit_buy_order, limit_sell_order, fee, mock patch_get_signal(mocker) patch_RPCManager(mocker) mocker.patch.multiple( - 'freqtrade.freqtradebot.exchange', + 'freqtrade.exchange.Exchange', validate_pairs=MagicMock(), get_ticker=MagicMock(return_value={ 'bid': 0.00001172, @@ -669,7 +673,7 @@ def test_handle_overlpapping_signals(default_conf, ticker, limit_buy_order, fee, patch_coinmarketcap(mocker) mocker.patch('freqtrade.freqtradebot.Analyze.min_roi_reached', return_value=False) mocker.patch.multiple( - 'freqtrade.freqtradebot.exchange', + 'freqtrade.exchange.Exchange', validate_pairs=MagicMock(), get_ticker=ticker, buy=MagicMock(return_value={'id': limit_buy_order['id']}), @@ -727,7 +731,7 @@ def test_handle_trade_roi(default_conf, ticker, limit_buy_order, fee, mocker, ca patch_RPCManager(mocker) patch_coinmarketcap(mocker) mocker.patch.multiple( - 'freqtrade.freqtradebot.exchange', + 'freqtrade.exchange.Exchange', validate_pairs=MagicMock(), get_ticker=ticker, buy=MagicMock(return_value={'id': limit_buy_order['id']}), @@ -764,7 +768,7 @@ def test_handle_trade_experimental( patch_RPCManager(mocker) patch_coinmarketcap(mocker) mocker.patch.multiple( - 'freqtrade.freqtradebot.exchange', + 'freqtrade.exchange.Exchange', validate_pairs=MagicMock(), get_ticker=ticker, buy=MagicMock(return_value={'id': limit_buy_order['id']}), @@ -794,7 +798,7 @@ def test_close_trade(default_conf, ticker, limit_buy_order, limit_sell_order, fe patch_RPCManager(mocker) patch_coinmarketcap(mocker) mocker.patch.multiple( - 'freqtrade.freqtradebot.exchange', + 'freqtrade.exchange.Exchange', validate_pairs=MagicMock(), get_ticker=ticker, buy=MagicMock(return_value={'id': limit_buy_order['id']}), @@ -824,7 +828,7 @@ def test_check_handle_timedout_buy(default_conf, ticker, limit_buy_order_old, fe cancel_order_mock = MagicMock() patch_coinmarketcap(mocker) mocker.patch.multiple( - 'freqtrade.freqtradebot.exchange', + 'freqtrade.exchange.Exchange', validate_pairs=MagicMock(), get_ticker=ticker, get_order=MagicMock(return_value=limit_buy_order_old), @@ -865,7 +869,7 @@ def test_check_handle_timedout_sell(default_conf, ticker, limit_sell_order_old, patch_coinmarketcap(mocker) cancel_order_mock = MagicMock() mocker.patch.multiple( - 'freqtrade.freqtradebot.exchange', + 'freqtrade.exchange.Exchange', validate_pairs=MagicMock(), get_ticker=ticker, get_order=MagicMock(return_value=limit_sell_order_old), @@ -905,7 +909,7 @@ def test_check_handle_timedout_partial(default_conf, ticker, limit_buy_order_old patch_coinmarketcap(mocker) cancel_order_mock = MagicMock() mocker.patch.multiple( - 'freqtrade.freqtradebot.exchange', + 'freqtrade.exchange.Exchange', validate_pairs=MagicMock(), get_ticker=ticker, get_order=MagicMock(return_value=limit_buy_order_old_partial), @@ -953,7 +957,7 @@ def test_check_handle_timedout_exception(default_conf, ticker, mocker, caplog) - handle_timedout_limit_sell=MagicMock(), ) mocker.patch.multiple( - 'freqtrade.freqtradebot.exchange', + 'freqtrade.exchange.Exchange', validate_pairs=MagicMock(), get_ticker=ticker, get_order=MagicMock(side_effect=requests.exceptions.RequestException('Oh snap')), @@ -993,7 +997,7 @@ def test_handle_timedout_limit_buy(mocker, default_conf) -> None: patch_coinmarketcap(mocker) cancel_order_mock = MagicMock() mocker.patch.multiple( - 'freqtrade.freqtradebot.exchange', + 'freqtrade.exchange.Exchange', validate_pairs=MagicMock(), cancel_order=cancel_order_mock ) @@ -1019,7 +1023,7 @@ def test_handle_timedout_limit_sell(mocker, default_conf) -> None: cancel_order_mock = MagicMock() patch_coinmarketcap(mocker) mocker.patch.multiple( - 'freqtrade.freqtradebot.exchange', + 'freqtrade.exchange.Exchange', validate_pairs=MagicMock(), cancel_order=cancel_order_mock ) @@ -1045,7 +1049,7 @@ def test_execute_sell_up(default_conf, ticker, fee, ticker_sell_up, mocker) -> N rpc_mock = patch_RPCManager(mocker) patch_coinmarketcap(mocker) mocker.patch.multiple( - 'freqtrade.freqtradebot.exchange', + 'freqtrade.exchange.Exchange', validate_pairs=MagicMock(), get_ticker=ticker, get_fee=fee @@ -1061,7 +1065,7 @@ def test_execute_sell_up(default_conf, ticker, fee, ticker_sell_up, mocker) -> N # Increase the price and sell it mocker.patch.multiple( - 'freqtrade.freqtradebot.exchange', + 'freqtrade.exchange.Exchange', validate_pairs=MagicMock(), get_ticker=ticker_sell_up ) @@ -1087,7 +1091,7 @@ def test_execute_sell_down(default_conf, ticker, fee, ticker_sell_down, mocker) patch_coinmarketcap(mocker) mocker.patch('freqtrade.fiat_convert.CryptoToFiatConverter._find_price', return_value=15000.0) mocker.patch.multiple( - 'freqtrade.freqtradebot.exchange', + 'freqtrade.exchange.Exchange', validate_pairs=MagicMock(), get_ticker=ticker, get_fee=fee @@ -1102,7 +1106,7 @@ def test_execute_sell_down(default_conf, ticker, fee, ticker_sell_down, mocker) # Decrease the price and sell it mocker.patch.multiple( - 'freqtrade.freqtradebot.exchange', + 'freqtrade.exchange.Exchange', validate_pairs=MagicMock(), get_ticker=ticker_sell_down ) @@ -1127,7 +1131,7 @@ def test_execute_sell_without_conf_sell_up(default_conf, ticker, fee, rpc_mock = patch_RPCManager(mocker) patch_coinmarketcap(mocker, value={'price_usd': 12345.0}) mocker.patch.multiple( - 'freqtrade.freqtradebot.exchange', + 'freqtrade.exchange.Exchange', validate_pairs=MagicMock(), get_ticker=ticker, get_fee=fee @@ -1142,7 +1146,7 @@ def test_execute_sell_without_conf_sell_up(default_conf, ticker, fee, # Increase the price and sell it mocker.patch.multiple( - 'freqtrade.freqtradebot.exchange', + 'freqtrade.exchange.Exchange', validate_pairs=MagicMock(), get_ticker=ticker_sell_up ) @@ -1168,7 +1172,7 @@ def test_execute_sell_without_conf_sell_down(default_conf, ticker, fee, rpc_mock = patch_RPCManager(mocker) patch_coinmarketcap(mocker, value={'price_usd': 12345.0}) mocker.patch.multiple( - 'freqtrade.freqtradebot.exchange', + 'freqtrade.exchange.Exchange', validate_pairs=MagicMock(), get_ticker=ticker, get_fee=fee @@ -1183,7 +1187,7 @@ def test_execute_sell_without_conf_sell_down(default_conf, ticker, fee, # Decrease the price and sell it mocker.patch.multiple( - 'freqtrade.freqtradebot.exchange', + 'freqtrade.exchange.Exchange', validate_pairs=MagicMock(), get_ticker=ticker_sell_down ) @@ -1207,7 +1211,7 @@ def test_sell_profit_only_enable_profit(default_conf, limit_buy_order, fee, mock patch_coinmarketcap(mocker) mocker.patch('freqtrade.freqtradebot.Analyze.min_roi_reached', return_value=False) mocker.patch.multiple( - 'freqtrade.freqtradebot.exchange', + 'freqtrade.exchange.Exchange', validate_pairs=MagicMock(), get_ticker=MagicMock(return_value={ 'bid': 0.00002172, @@ -1240,7 +1244,7 @@ def test_sell_profit_only_disable_profit(default_conf, limit_buy_order, fee, moc patch_coinmarketcap(mocker) mocker.patch('freqtrade.freqtradebot.Analyze.min_roi_reached', return_value=False) mocker.patch.multiple( - 'freqtrade.freqtradebot.exchange', + 'freqtrade.exchange.Exchange', validate_pairs=MagicMock(), get_ticker=MagicMock(return_value={ 'bid': 0.00002172, @@ -1273,7 +1277,7 @@ def test_sell_profit_only_enable_loss(default_conf, limit_buy_order, fee, mocker patch_coinmarketcap(mocker) mocker.patch('freqtrade.freqtradebot.Analyze.min_roi_reached', return_value=False) mocker.patch.multiple( - 'freqtrade.freqtradebot.exchange', + 'freqtrade.exchange.Exchange', validate_pairs=MagicMock(), get_ticker=MagicMock(return_value={ 'bid': 0.00000172, @@ -1306,7 +1310,7 @@ def test_sell_profit_only_disable_loss(default_conf, limit_buy_order, fee, mocke patch_coinmarketcap(mocker) mocker.patch('freqtrade.freqtradebot.Analyze.min_roi_reached', return_value=False) mocker.patch.multiple( - 'freqtrade.freqtradebot.exchange', + 'freqtrade.exchange.Exchange', validate_pairs=MagicMock(), get_ticker=MagicMock(return_value={ 'bid': 0.00000172, @@ -1337,12 +1341,12 @@ def test_get_real_amount_quote(default_conf, trades_for_order, buy_order_fee, ca Test get_real_amount - fee in quote currency """ - mocker.patch('freqtrade.exchange.get_trades_for_order', return_value=trades_for_order) + mocker.patch('freqtrade.exchange.Exchange.get_trades_for_order', return_value=trades_for_order) patch_get_signal(mocker) patch_RPCManager(mocker) patch_coinmarketcap(mocker) - mocker.patch('freqtrade.exchange.validate_pairs', MagicMock(return_value=True)) + mocker.patch('freqtrade.exchange.Exchange.validate_pairs', MagicMock(return_value=True)) amount = sum(x['amount'] for x in trades_for_order) trade = Trade( pair='LTC/ETH', @@ -1364,12 +1368,12 @@ def test_get_real_amount_no_trade(default_conf, buy_order_fee, caplog, mocker): Test get_real_amount - fee in quote currency """ - mocker.patch('freqtrade.exchange.get_trades_for_order', return_value=[]) + mocker.patch('freqtrade.exchange.Exchange.get_trades_for_order', return_value=[]) patch_get_signal(mocker) patch_RPCManager(mocker) patch_coinmarketcap(mocker) - mocker.patch('freqtrade.exchange.validate_pairs', MagicMock(return_value=True)) + mocker.patch('freqtrade.exchange.Exchange.validate_pairs', MagicMock(return_value=True)) amount = buy_order_fee['amount'] trade = Trade( pair='LTC/ETH', @@ -1395,8 +1399,8 @@ def test_get_real_amount_stake(default_conf, trades_for_order, buy_order_fee, mo patch_get_signal(mocker) patch_RPCManager(mocker) patch_coinmarketcap(mocker) - mocker.patch('freqtrade.exchange.validate_pairs', MagicMock(return_value=True)) - mocker.patch('freqtrade.exchange.get_trades_for_order', return_value=trades_for_order) + mocker.patch('freqtrade.exchange.Exchange.validate_pairs', MagicMock(return_value=True)) + mocker.patch('freqtrade.exchange.Exchange.get_trades_for_order', return_value=trades_for_order) amount = sum(x['amount'] for x in trades_for_order) trade = Trade( pair='LTC/ETH', @@ -1421,8 +1425,8 @@ def test_get_real_amount_BNB(default_conf, trades_for_order, buy_order_fee, mock patch_get_signal(mocker) patch_RPCManager(mocker) patch_coinmarketcap(mocker) - mocker.patch('freqtrade.exchange.validate_pairs', MagicMock(return_value=True)) - mocker.patch('freqtrade.exchange.get_trades_for_order', return_value=trades_for_order) + mocker.patch('freqtrade.exchange.Exchange.validate_pairs', MagicMock(return_value=True)) + mocker.patch('freqtrade.exchange.Exchange.get_trades_for_order', return_value=trades_for_order) amount = sum(x['amount'] for x in trades_for_order) trade = Trade( pair='LTC/ETH', @@ -1444,8 +1448,8 @@ def test_get_real_amount_multi(default_conf, trades_for_order2, buy_order_fee, c patch_get_signal(mocker) patch_RPCManager(mocker) patch_coinmarketcap(mocker) - mocker.patch('freqtrade.exchange.validate_pairs', MagicMock(return_value=True)) - mocker.patch('freqtrade.exchange.get_trades_for_order', return_value=trades_for_order2) + mocker.patch('freqtrade.exchange.Exchange.validate_pairs', MagicMock(return_value=True)) + mocker.patch('freqtrade.exchange.Exchange.get_trades_for_order', return_value=trades_for_order2) amount = float(sum(x['amount'] for x in trades_for_order2)) trade = Trade( pair='LTC/ETH', @@ -1472,8 +1476,9 @@ def test_get_real_amount_fromorder(default_conf, trades_for_order, buy_order_fee patch_get_signal(mocker) patch_RPCManager(mocker) patch_coinmarketcap(mocker) - mocker.patch('freqtrade.exchange.validate_pairs', MagicMock(return_value=True)) - mocker.patch('freqtrade.exchange.get_trades_for_order', return_value=[trades_for_order]) + mocker.patch('freqtrade.exchange.Exchange.validate_pairs', MagicMock(return_value=True)) + mocker.patch('freqtrade.exchange.Exchange.get_trades_for_order', + return_value=[trades_for_order]) amount = float(sum(x['amount'] for x in trades_for_order)) trade = Trade( pair='LTC/ETH', @@ -1500,8 +1505,8 @@ def test_get_real_amount_invalid_order(default_conf, trades_for_order, buy_order patch_get_signal(mocker) patch_RPCManager(mocker) patch_coinmarketcap(mocker) - mocker.patch('freqtrade.exchange.validate_pairs', MagicMock(return_value=True)) - mocker.patch('freqtrade.exchange.get_trades_for_order', return_value=[]) + mocker.patch('freqtrade.exchange.Exchange.validate_pairs', MagicMock(return_value=True)) + mocker.patch('freqtrade.exchange.Exchange.get_trades_for_order', return_value=[]) amount = float(sum(x['amount'] for x in trades_for_order)) trade = Trade( pair='LTC/ETH', @@ -1525,8 +1530,8 @@ def test_get_real_amount_invalid(default_conf, trades_for_order, buy_order_fee, patch_get_signal(mocker) patch_RPCManager(mocker) patch_coinmarketcap(mocker) - mocker.patch('freqtrade.exchange.validate_pairs', MagicMock(return_value=True)) - mocker.patch('freqtrade.exchange.get_trades_for_order', return_value=trades_for_order) + mocker.patch('freqtrade.exchange.Exchange.validate_pairs', MagicMock(return_value=True)) + mocker.patch('freqtrade.exchange.Exchange.get_trades_for_order', return_value=trades_for_order) amount = sum(x['amount'] for x in trades_for_order) trade = Trade( pair='LTC/ETH', @@ -1547,7 +1552,7 @@ def test_get_real_amount_open_trade(default_conf, mocker): patch_get_signal(mocker) patch_RPCManager(mocker) patch_coinmarketcap(mocker) - mocker.patch('freqtrade.exchange.validate_pairs', MagicMock(return_value=True)) + mocker.patch('freqtrade.exchange.Exchange.validate_pairs', MagicMock(return_value=True)) amount = 12345 trade = Trade( pair='LTC/ETH', From 63b568989aeb7abb8fdbc5201586113f8134fa87 Mon Sep 17 00:00:00 2001 From: xmatthias Date: Sun, 17 Jun 2018 21:24:36 +0200 Subject: [PATCH 19/64] Fix rpc for exchange objectify --- freqtrade/rpc/rpc.py | 24 ++++++++++++------------ 1 file changed, 12 insertions(+), 12 deletions(-) diff --git a/freqtrade/rpc/rpc.py b/freqtrade/rpc/rpc.py index 34802f920..d58c265a7 100644 --- a/freqtrade/rpc/rpc.py +++ b/freqtrade/rpc/rpc.py @@ -12,7 +12,7 @@ import sqlalchemy as sql from numpy import mean, nan_to_num from pandas import DataFrame -from freqtrade import exchange +from freqtrade.exchange import Exchange from freqtrade.misc import shorten_date from freqtrade.persistence import Trade from freqtrade.state import State @@ -71,9 +71,9 @@ class RPC(object): for trade in trades: order = None if trade.open_order_id: - order = exchange.get_order(trade.open_order_id, trade.pair) + order = self._freqtrade.exchange.get_order(trade.open_order_id, trade.pair) # calculate profit and send message to user - current_rate = exchange.get_ticker(trade.pair, False)['bid'] + current_rate = self._freqtrade.exchange.get_ticker(trade.pair, False)['bid'] current_profit = trade.calc_profit_percent(current_rate) fmt_close_profit = '{:.2f}%'.format( round(trade.close_profit * 100, 2) @@ -91,7 +91,7 @@ class RPC(object): .format( trade_id=trade.id, pair=trade.pair, - market_url=exchange.get_pair_detail_url(trade.pair), + market_url=self._freqtrade.exchange.get_pair_detail_url(trade.pair), date=arrow.get(trade.open_date).humanize(), open_rate=trade.open_rate, close_rate=trade.close_rate, @@ -116,7 +116,7 @@ class RPC(object): trades_list = [] for trade in trades: # calculate profit and send message to user - current_rate = exchange.get_ticker(trade.pair, False)['bid'] + current_rate = self._freqtrade.exchange.get_ticker(trade.pair, False)['bid'] trades_list.append([ trade.id, trade.pair, @@ -201,7 +201,7 @@ class RPC(object): profit_closed_percent.append(profit_percent) else: # Get current rate - current_rate = exchange.get_ticker(trade.pair, False)['bid'] + current_rate = self._freqtrade.exchange.get_ticker(trade.pair, False)['bid'] profit_percent = trade.calc_profit_percent(rate=current_rate) profit_all_coin.append( @@ -258,7 +258,7 @@ class RPC(object): """ Returns current account balance per crypto """ output = [] total = 0.0 - for coin, balance in exchange.get_balances().items(): + for coin, balance in self._freqtrade.exchange.get_balances().items(): if not balance['total']: continue @@ -266,9 +266,9 @@ class RPC(object): rate = 1.0 else: if coin == 'USDT': - rate = 1.0 / exchange.get_ticker('BTC/USDT', False)['bid'] + rate = 1.0 / self._freqtrade.exchange.get_ticker('BTC/USDT', False)['bid'] else: - rate = exchange.get_ticker(coin + '/BTC', False)['bid'] + rate = self._freqtrade.exchange.get_ticker(coin + '/BTC', False)['bid'] est_btc: float = rate * balance['total'] total = total + est_btc output.append( @@ -318,13 +318,13 @@ class RPC(object): def _exec_forcesell(trade: Trade) -> None: # Check if there is there is an open order if trade.open_order_id: - order = exchange.get_order(trade.open_order_id, trade.pair) + order = self._freqtrade.exchange.get_order(trade.open_order_id, trade.pair) # Cancel open LIMIT_BUY orders and close trade if order and order['status'] == 'open' \ and order['type'] == 'limit' \ and order['side'] == 'buy': - exchange.cancel_order(trade.open_order_id, trade.pair) + self._freqtrade.exchange.cancel_order(trade.open_order_id, trade.pair) trade.close(order.get('price') or trade.open_rate) # Do the best effort, if we don't know 'filled' amount, don't try selling if order['filled'] is None: @@ -338,7 +338,7 @@ class RPC(object): return # Get current rate and execute sell - current_rate = exchange.get_ticker(trade.pair, False)['bid'] + current_rate = self._freqtrade.exchange.get_ticker(trade.pair, False)['bid'] self._freqtrade.execute_sell(trade, current_rate) # ---- EOF def _exec_forcesell ---- From 64e09f74a1d271ba93274ecf1edb048073764b17 Mon Sep 17 00:00:00 2001 From: xmatthias Date: Sun, 17 Jun 2018 21:24:51 +0200 Subject: [PATCH 20/64] fix rpc tests --- freqtrade/tests/rpc/test_rpc.py | 34 ++++++++++++++++----------------- 1 file changed, 17 insertions(+), 17 deletions(-) diff --git a/freqtrade/tests/rpc/test_rpc.py b/freqtrade/tests/rpc/test_rpc.py index b49b7fdcb..cc3a78a0e 100644 --- a/freqtrade/tests/rpc/test_rpc.py +++ b/freqtrade/tests/rpc/test_rpc.py @@ -33,7 +33,7 @@ def test_rpc_trade_status(default_conf, ticker, fee, mocker) -> None: patch_coinmarketcap(mocker) mocker.patch('freqtrade.rpc.telegram.Telegram', MagicMock()) mocker.patch.multiple( - 'freqtrade.freqtradebot.exchange', + 'freqtrade.exchange.Exchange', validate_pairs=MagicMock(), get_ticker=ticker, get_fee=fee @@ -79,7 +79,7 @@ def test_rpc_status_table(default_conf, ticker, fee, mocker) -> None: patch_coinmarketcap(mocker) mocker.patch('freqtrade.rpc.telegram.Telegram', MagicMock()) mocker.patch.multiple( - 'freqtrade.freqtradebot.exchange', + 'freqtrade.exchange.Exchange', validate_pairs=MagicMock(), get_ticker=ticker, get_fee=fee @@ -112,7 +112,7 @@ def test_rpc_daily_profit(default_conf, update, ticker, fee, patch_coinmarketcap(mocker, value={'price_usd': 15000.0}) mocker.patch('freqtrade.rpc.telegram.Telegram', MagicMock()) mocker.patch.multiple( - 'freqtrade.freqtradebot.exchange', + 'freqtrade.exchange.Exchange', validate_pairs=MagicMock(), get_ticker=ticker, get_fee=fee @@ -167,7 +167,7 @@ def test_rpc_trade_statistics(default_conf, ticker, ticker_sell_up, fee, mocker.patch('freqtrade.fiat_convert.CryptoToFiatConverter._find_price', return_value=15000.0) mocker.patch('freqtrade.rpc.telegram.Telegram', MagicMock()) mocker.patch.multiple( - 'freqtrade.freqtradebot.exchange', + 'freqtrade.exchange.Exchange', validate_pairs=MagicMock(), get_ticker=ticker, get_fee=fee @@ -190,7 +190,7 @@ def test_rpc_trade_statistics(default_conf, ticker, ticker_sell_up, fee, # Update the ticker with a market going up mocker.patch.multiple( - 'freqtrade.freqtradebot.exchange', + 'freqtrade.exchange.Exchange', validate_pairs=MagicMock(), get_ticker=ticker_sell_up ) @@ -205,7 +205,7 @@ def test_rpc_trade_statistics(default_conf, ticker, ticker_sell_up, fee, # Update the ticker with a market going up mocker.patch.multiple( - 'freqtrade.freqtradebot.exchange', + 'freqtrade.exchange.Exchange', validate_pairs=MagicMock(), get_ticker=ticker_sell_up ) @@ -243,7 +243,7 @@ def test_rpc_trade_statistics_closed(mocker, default_conf, ticker, fee, mocker.patch('freqtrade.fiat_convert.CryptoToFiatConverter._find_price', return_value=15000.0) mocker.patch('freqtrade.rpc.telegram.Telegram', MagicMock()) mocker.patch.multiple( - 'freqtrade.freqtradebot.exchange', + 'freqtrade.exchange.Exchange', validate_pairs=MagicMock(), get_ticker=ticker, get_fee=fee @@ -262,7 +262,7 @@ def test_rpc_trade_statistics_closed(mocker, default_conf, ticker, fee, trade.update(limit_buy_order) # Update the ticker with a market going up mocker.patch.multiple( - 'freqtrade.freqtradebot.exchange', + 'freqtrade.exchange.Exchange', validate_pairs=MagicMock(), get_ticker=ticker_sell_up, get_fee=fee @@ -314,7 +314,7 @@ def test_rpc_balance_handle(default_conf, mocker): mocker.patch('freqtrade.fiat_convert.CryptoToFiatConverter._find_price', return_value=15000.0) mocker.patch('freqtrade.rpc.telegram.Telegram', MagicMock()) mocker.patch.multiple( - 'freqtrade.freqtradebot.exchange', + 'freqtrade.exchange.Exchange', validate_pairs=MagicMock(), get_balances=MagicMock(return_value=mock_balance) ) @@ -342,7 +342,7 @@ def test_rpc_start(mocker, default_conf) -> None: patch_coinmarketcap(mocker) mocker.patch('freqtrade.rpc.telegram.Telegram', MagicMock()) mocker.patch.multiple( - 'freqtrade.freqtradebot.exchange', + 'freqtrade.exchange.Exchange', validate_pairs=MagicMock(), get_ticker=MagicMock() ) @@ -368,7 +368,7 @@ def test_rpc_stop(mocker, default_conf) -> None: patch_coinmarketcap(mocker) mocker.patch('freqtrade.rpc.telegram.Telegram', MagicMock()) mocker.patch.multiple( - 'freqtrade.freqtradebot.exchange', + 'freqtrade.exchange.Exchange', validate_pairs=MagicMock(), get_ticker=MagicMock() ) @@ -396,7 +396,7 @@ def test_rpc_forcesell(default_conf, ticker, fee, mocker) -> None: cancel_order_mock = MagicMock() mocker.patch.multiple( - 'freqtrade.freqtradebot.exchange', + 'freqtrade.exchange.Exchange', validate_pairs=MagicMock(), get_ticker=ticker, cancel_order=cancel_order_mock, @@ -441,7 +441,7 @@ def test_rpc_forcesell(default_conf, ticker, fee, mocker) -> None: trade = Trade.query.filter(Trade.id == '1').first() filled_amount = trade.amount / 2 mocker.patch( - 'freqtrade.freqtradebot.exchange.get_order', + 'freqtrade.exchange.Exchange.get_order', return_value={ 'status': 'open', 'type': 'limit', @@ -460,7 +460,7 @@ def test_rpc_forcesell(default_conf, ticker, fee, mocker) -> None: amount = trade.amount # make an limit-buy open trade, if there is no 'filled', don't sell it mocker.patch( - 'freqtrade.freqtradebot.exchange.get_order', + 'freqtrade.exchange.Exchange.get_order', return_value={ 'status': 'open', 'type': 'limit', @@ -476,7 +476,7 @@ def test_rpc_forcesell(default_conf, ticker, fee, mocker) -> None: freqtradebot.create_trade() # make an limit-sell open trade mocker.patch( - 'freqtrade.freqtradebot.exchange.get_order', + 'freqtrade.exchange.Exchange.get_order', return_value={ 'status': 'open', 'type': 'limit', @@ -497,7 +497,7 @@ def test_performance_handle(default_conf, ticker, limit_buy_order, fee, patch_coinmarketcap(mocker) mocker.patch('freqtrade.rpc.telegram.Telegram', MagicMock()) mocker.patch.multiple( - 'freqtrade.freqtradebot.exchange', + 'freqtrade.exchange.Exchange', validate_pairs=MagicMock(), get_balances=MagicMock(return_value=ticker), get_ticker=ticker, @@ -535,7 +535,7 @@ def test_rpc_count(mocker, default_conf, ticker, fee) -> None: patch_coinmarketcap(mocker) mocker.patch('freqtrade.rpc.telegram.Telegram', MagicMock()) mocker.patch.multiple( - 'freqtrade.freqtradebot.exchange', + 'freqtrade.exchange.Exchange', validate_pairs=MagicMock(), get_balances=MagicMock(return_value=ticker), get_ticker=ticker, From c83e8b7cb585bc7fafc97407b4c1ebc56f264073 Mon Sep 17 00:00:00 2001 From: xmatthias Date: Sun, 17 Jun 2018 21:46:56 +0200 Subject: [PATCH 21/64] fix rpc_test --- freqtrade/tests/conftest.py | 9 ++- freqtrade/tests/rpc/test_rpc_telegram.py | 93 ++++++++++-------------- 2 files changed, 43 insertions(+), 59 deletions(-) diff --git a/freqtrade/tests/conftest.py b/freqtrade/tests/conftest.py index ed900e6bb..ce22cd193 100644 --- a/freqtrade/tests/conftest.py +++ b/freqtrade/tests/conftest.py @@ -27,13 +27,16 @@ def log_has(line, logs): False) -def get_patched_exchange(mocker, config, api_mock=None) -> Exchange: - +def patch_exchange(mocker, api_mock=None) -> None: mocker.patch('freqtrade.exchange.Exchange.validate_pairs', MagicMock()) if api_mock: mocker.patch('freqtrade.exchange.Exchange._init_ccxt', MagicMock(return_value=api_mock)) else: mocker.patch('freqtrade.exchange.Exchange._init_ccxt', MagicMock()) + + +def get_patched_exchange(mocker, config, api_mock=None) -> Exchange: + patch_exchange(mocker, api_mock) exchange = Exchange(config) return exchange @@ -51,7 +54,7 @@ def get_patched_freqtradebot(mocker, config) -> FreqtradeBot: mocker.patch('freqtrade.freqtradebot.Analyze', MagicMock()) mocker.patch('freqtrade.freqtradebot.RPCManager', MagicMock()) mocker.patch('freqtrade.freqtradebot.persistence.init', MagicMock()) - mocker.patch('freqtrade.freqtradebot.exchange.init', MagicMock()) + patch_exchange(mocker, None) mocker.patch('freqtrade.freqtradebot.RPCManager._init', MagicMock()) mocker.patch('freqtrade.freqtradebot.RPCManager.send_msg', MagicMock()) mocker.patch('freqtrade.freqtradebot.Analyze.get_signal', MagicMock()) diff --git a/freqtrade/tests/rpc/test_rpc_telegram.py b/freqtrade/tests/rpc/test_rpc_telegram.py index f022c09e4..1b07a2143 100644 --- a/freqtrade/tests/rpc/test_rpc_telegram.py +++ b/freqtrade/tests/rpc/test_rpc_telegram.py @@ -20,7 +20,7 @@ from freqtrade.persistence import Trade from freqtrade.rpc.telegram import Telegram from freqtrade.rpc.telegram import authorized_only from freqtrade.state import State -from freqtrade.tests.conftest import get_patched_freqtradebot, log_has +from freqtrade.tests.conftest import get_patched_freqtradebot,patch_exchange, log_has from freqtrade.tests.test_freqtradebot import patch_get_signal, patch_coinmarketcap @@ -100,7 +100,7 @@ def test_authorized_only(default_conf, mocker, caplog) -> None: """ patch_get_signal(mocker, (True, False)) patch_coinmarketcap(mocker) - mocker.patch('freqtrade.freqtradebot.exchange.init', MagicMock()) + patch_exchange(mocker, None) chat = Chat(0, 0) update = Update(randint(1, 100)) @@ -131,8 +131,7 @@ def test_authorized_only_unauthorized(default_conf, mocker, caplog) -> None: """ patch_get_signal(mocker, (True, False)) patch_coinmarketcap(mocker) - mocker.patch('freqtrade.freqtradebot.exchange.init', MagicMock()) - + patch_exchange(mocker, None) chat = Chat(0xdeadbeef, 0) update = Update(randint(1, 100)) update.message = Message(randint(1, 100), 0, datetime.utcnow(), chat) @@ -162,7 +161,7 @@ def test_authorized_only_exception(default_conf, mocker, caplog) -> None: """ patch_get_signal(mocker, (True, False)) patch_coinmarketcap(mocker) - mocker.patch('freqtrade.freqtradebot.exchange.init', MagicMock()) + patch_exchange(mocker) update = Update(randint(1, 100)) update.message = Message(randint(1, 100), 0, datetime.utcnow(), Chat(0, 0)) @@ -198,7 +197,7 @@ def test_status(default_conf, update, mocker, fee, ticker) -> None: patch_get_signal(mocker, (True, False)) patch_coinmarketcap(mocker) mocker.patch.multiple( - 'freqtrade.freqtradebot.exchange', + 'freqtrade.exchange.Exchange', validate_pairs=MagicMock(), get_ticker=ticker, get_pair_detail_url=MagicMock(), @@ -238,7 +237,7 @@ def test_status_handle(default_conf, update, ticker, fee, mocker) -> None: patch_get_signal(mocker, (True, False)) patch_coinmarketcap(mocker) mocker.patch.multiple( - 'freqtrade.freqtradebot.exchange', + 'freqtrade.exchange.Exchange', validate_pairs=MagicMock(), get_ticker=ticker, get_fee=fee, @@ -284,7 +283,7 @@ def test_status_table_handle(default_conf, update, ticker, fee, mocker) -> None: patch_get_signal(mocker, (True, False)) patch_coinmarketcap(mocker) mocker.patch.multiple( - 'freqtrade.freqtradebot.exchange', + 'freqtrade.exchange.Exchange', validate_pairs=MagicMock(), get_ticker=ticker, buy=MagicMock(return_value={'id': 'mocked_order_id'}), @@ -341,7 +340,7 @@ def test_daily_handle(default_conf, update, ticker, limit_buy_order, fee, return_value=15000.0 ) mocker.patch.multiple( - 'freqtrade.freqtradebot.exchange', + 'freqtrade.exchange.Exchange', validate_pairs=MagicMock(), get_ticker=ticker, get_fee=fee @@ -410,7 +409,7 @@ def test_daily_wrong_input(default_conf, update, ticker, mocker) -> None: patch_get_signal(mocker, (True, False)) patch_coinmarketcap(mocker, value={'price_usd': 15000.0}) mocker.patch.multiple( - 'freqtrade.freqtradebot.exchange', + 'freqtrade.exchange.Exchange', validate_pairs=MagicMock(), get_ticker=ticker ) @@ -450,7 +449,7 @@ def test_profit_handle(default_conf, update, ticker, ticker_sell_up, fee, patch_coinmarketcap(mocker, value={'price_usd': 15000.0}) mocker.patch('freqtrade.fiat_convert.CryptoToFiatConverter._find_price', return_value=15000.0) mocker.patch.multiple( - 'freqtrade.freqtradebot.exchange', + 'freqtrade.exchange.Exchange', validate_pairs=MagicMock(), get_ticker=ticker, get_fee=fee @@ -484,7 +483,7 @@ def test_profit_handle(default_conf, update, ticker, ticker_sell_up, fee, msg_mock.reset_mock() # Update the ticker with a market going up - mocker.patch('freqtrade.freqtradebot.exchange.get_ticker', ticker_sell_up) + mocker.patch('freqtrade.exchange.Exchange.get_ticker', ticker_sell_up) trade.update(limit_sell_order) trade.close_date = datetime.utcnow() @@ -549,9 +548,8 @@ def test_telegram_balance_handle(default_conf, update, mocker) -> None: patch_get_signal(mocker, (True, False)) patch_coinmarketcap(mocker, value={'price_usd': 15000.0}) - mocker.patch('freqtrade.freqtradebot.exchange.init', MagicMock()) - mocker.patch('freqtrade.freqtradebot.exchange.get_balances', return_value=mock_balance) - mocker.patch('freqtrade.freqtradebot.exchange.get_ticker', side_effect=mock_ticker) + mocker.patch('freqtrade.exchange.Exchange.get_balances', return_value=mock_balance) + mocker.patch('freqtrade.exchange.Exchange.get_ticker', side_effect=mock_ticker) msg_mock = MagicMock() mocker.patch.multiple( @@ -560,7 +558,7 @@ def test_telegram_balance_handle(default_conf, update, mocker) -> None: _send_msg=msg_mock ) - freqtradebot = FreqtradeBot(default_conf) + freqtradebot = get_patched_freqtradebot(mocker, default_conf) telegram = Telegram(freqtradebot) telegram._balance(bot=MagicMock(), update=update) @@ -579,9 +577,7 @@ def test_zero_balance_handle(default_conf, update, mocker) -> None: Test _balance() method when the Exchange platform returns nothing """ patch_get_signal(mocker, (True, False)) - patch_coinmarketcap(mocker, value={'price_usd': 15000.0}) - mocker.patch('freqtrade.freqtradebot.exchange.init', MagicMock()) - mocker.patch('freqtrade.freqtradebot.exchange.get_balances', return_value={}) + mocker.patch('freqtrade.exchange.Exchange.get_balances', return_value={}) msg_mock = MagicMock() mocker.patch.multiple( @@ -590,7 +586,7 @@ def test_zero_balance_handle(default_conf, update, mocker) -> None: _send_msg=msg_mock ) - freqtradebot = FreqtradeBot(default_conf) + freqtradebot = get_patched_freqtradebot(mocker, default_conf) telegram = Telegram(freqtradebot) telegram._balance(bot=MagicMock(), update=update) @@ -603,17 +599,14 @@ def test_start_handle(default_conf, update, mocker) -> None: """ Test _start() method """ - patch_coinmarketcap(mocker) - mocker.patch('freqtrade.freqtradebot.exchange.init', MagicMock()) msg_mock = MagicMock() mocker.patch.multiple( 'freqtrade.rpc.telegram.Telegram', _init=MagicMock(), _send_msg=msg_mock ) - mocker.patch('freqtrade.freqtradebot.RPCManager', MagicMock()) - freqtradebot = FreqtradeBot(default_conf) + freqtradebot = get_patched_freqtradebot(mocker, default_conf) telegram = Telegram(freqtradebot) freqtradebot.state = State.STOPPED @@ -627,17 +620,14 @@ def test_start_handle_already_running(default_conf, update, mocker) -> None: """ Test _start() method """ - patch_coinmarketcap(mocker) - mocker.patch('freqtrade.freqtradebot.exchange.init', MagicMock()) msg_mock = MagicMock() mocker.patch.multiple( 'freqtrade.rpc.telegram.Telegram', _init=MagicMock(), _send_msg=msg_mock ) - mocker.patch('freqtrade.freqtradebot.RPCManager', MagicMock()) - freqtradebot = FreqtradeBot(default_conf) + freqtradebot = get_patched_freqtradebot(mocker, default_conf) telegram = Telegram(freqtradebot) freqtradebot.state = State.RUNNING @@ -653,16 +643,14 @@ def test_stop_handle(default_conf, update, mocker) -> None: Test _stop() method """ patch_coinmarketcap(mocker) - mocker.patch('freqtrade.freqtradebot.exchange.init', MagicMock()) msg_mock = MagicMock() mocker.patch.multiple( 'freqtrade.rpc.telegram.Telegram', _init=MagicMock(), _send_msg=msg_mock ) - mocker.patch('freqtrade.freqtradebot.RPCManager', MagicMock()) - freqtradebot = FreqtradeBot(default_conf) + freqtradebot = get_patched_freqtradebot(mocker, default_conf) telegram = Telegram(freqtradebot) freqtradebot.state = State.RUNNING @@ -678,16 +666,14 @@ def test_stop_handle_already_stopped(default_conf, update, mocker) -> None: Test _stop() method """ patch_coinmarketcap(mocker) - mocker.patch('freqtrade.freqtradebot.exchange.init', MagicMock()) msg_mock = MagicMock() mocker.patch.multiple( 'freqtrade.rpc.telegram.Telegram', _init=MagicMock(), _send_msg=msg_mock ) - mocker.patch('freqtrade.freqtradebot.RPCManager', MagicMock()) - freqtradebot = FreqtradeBot(default_conf) + freqtradebot = get_patched_freqtradebot(mocker, default_conf) telegram = Telegram(freqtradebot) freqtradebot.state = State.STOPPED @@ -701,16 +687,14 @@ def test_stop_handle_already_stopped(default_conf, update, mocker) -> None: def test_reload_conf_handle(default_conf, update, mocker) -> None: """ Test _reload_conf() method """ patch_coinmarketcap(mocker) - mocker.patch('freqtrade.freqtradebot.exchange.init', MagicMock()) msg_mock = MagicMock() mocker.patch.multiple( 'freqtrade.rpc.telegram.Telegram', _init=MagicMock(), _send_msg=msg_mock ) - mocker.patch('freqtrade.freqtradebot.RPCManager', MagicMock()) - freqtradebot = FreqtradeBot(default_conf) + freqtradebot = get_patched_freqtradebot(mocker, default_conf) telegram = Telegram(freqtradebot) freqtradebot.state = State.RUNNING @@ -731,7 +715,7 @@ def test_forcesell_handle(default_conf, update, ticker, fee, ticker_sell_up, moc rpc_mock = mocker.patch('freqtrade.rpc.telegram.Telegram.send_msg', MagicMock()) mocker.patch('freqtrade.rpc.telegram.Telegram._init', MagicMock()) mocker.patch.multiple( - 'freqtrade.freqtradebot.exchange', + 'freqtrade.exchange.Exchange', validate_pairs=MagicMock(), get_ticker=ticker, get_fee=fee @@ -747,7 +731,7 @@ def test_forcesell_handle(default_conf, update, ticker, fee, ticker_sell_up, moc assert trade # Increase the price and sell it - mocker.patch('freqtrade.freqtradebot.exchange.get_ticker', ticker_sell_up) + mocker.patch('freqtrade.exchange.Exchange.get_ticker', ticker_sell_up) update.message.text = '/forcesell 1' telegram._forcesell(bot=MagicMock(), update=update) @@ -771,7 +755,7 @@ def test_forcesell_down_handle(default_conf, update, ticker, fee, ticker_sell_do rpc_mock = mocker.patch('freqtrade.rpc.telegram.Telegram.send_msg', MagicMock()) mocker.patch('freqtrade.rpc.telegram.Telegram._init', MagicMock()) mocker.patch.multiple( - 'freqtrade.freqtradebot.exchange', + 'freqtrade.exchange.Exchange', validate_pairs=MagicMock(), get_ticker=ticker, get_fee=fee @@ -785,7 +769,7 @@ def test_forcesell_down_handle(default_conf, update, ticker, fee, ticker_sell_do # Decrease the price and sell it mocker.patch.multiple( - 'freqtrade.freqtradebot.exchange', + 'freqtrade.exchange.Exchange', validate_pairs=MagicMock(), get_ticker=ticker_sell_down ) @@ -814,9 +798,9 @@ def test_forcesell_all_handle(default_conf, update, ticker, fee, mocker) -> None mocker.patch('freqtrade.fiat_convert.CryptoToFiatConverter._find_price', return_value=15000.0) rpc_mock = mocker.patch('freqtrade.rpc.telegram.Telegram.send_msg', MagicMock()) mocker.patch('freqtrade.rpc.telegram.Telegram._init', MagicMock()) - mocker.patch('freqtrade.exchange.get_pair_detail_url', MagicMock()) + mocker.patch('freqtrade.exchange.Exchange.get_pair_detail_url', MagicMock()) mocker.patch.multiple( - 'freqtrade.freqtradebot.exchange', + 'freqtrade.exchange.Exchange', validate_pairs=MagicMock(), get_ticker=ticker, get_fee=fee @@ -853,7 +837,7 @@ def test_forcesell_handle_invalid(default_conf, update, mocker) -> None: _init=MagicMock(), _send_msg=msg_mock ) - mocker.patch('freqtrade.freqtradebot.exchange.validate_pairs', MagicMock()) + mocker.patch('freqtrade.exchange.Exchange.validate_pairs', MagicMock()) freqtradebot = FreqtradeBot(default_conf) telegram = Telegram(freqtradebot) @@ -896,7 +880,7 @@ def test_performance_handle(default_conf, update, ticker, fee, _send_msg=msg_mock ) mocker.patch.multiple( - 'freqtrade.freqtradebot.exchange', + 'freqtrade.exchange.Exchange', validate_pairs=MagicMock(), get_ticker=ticker, get_fee=fee @@ -936,7 +920,7 @@ def test_performance_handle_invalid(default_conf, update, mocker) -> None: _init=MagicMock(), _send_msg=msg_mock ) - mocker.patch('freqtrade.freqtradebot.exchange.validate_pairs', MagicMock()) + mocker.patch('freqtrade.exchange.Exchange.validate_pairs', MagicMock()) freqtradebot = FreqtradeBot(default_conf) telegram = Telegram(freqtradebot) @@ -960,12 +944,12 @@ def test_count_handle(default_conf, update, ticker, fee, mocker) -> None: _send_msg=msg_mock ) mocker.patch.multiple( - 'freqtrade.freqtradebot.exchange', + 'freqtrade.exchange.Exchange', validate_pairs=MagicMock(), get_ticker=ticker, buy=MagicMock(return_value={'id': 'mocked_order_id'}) ) - mocker.patch('freqtrade.optimize.backtesting.exchange.get_fee', fee) + mocker.patch('freqtrade.exchange.Exchange.get_fee', fee) freqtradebot = FreqtradeBot(default_conf) telegram = Telegram(freqtradebot) @@ -995,14 +979,14 @@ def test_help_handle(default_conf, update, mocker) -> None: Test _help() method """ patch_coinmarketcap(mocker) - mocker.patch('freqtrade.freqtradebot.exchange.init', MagicMock()) msg_mock = MagicMock() mocker.patch.multiple( 'freqtrade.rpc.telegram.Telegram', _init=MagicMock(), _send_msg=msg_mock ) - freqtradebot = FreqtradeBot(default_conf) + freqtradebot = get_patched_freqtradebot(mocker, default_conf) + telegram = Telegram(freqtradebot) telegram._help(bot=MagicMock(), update=update) @@ -1015,14 +999,13 @@ def test_version_handle(default_conf, update, mocker) -> None: Test _version() method """ patch_coinmarketcap(mocker) - mocker.patch('freqtrade.freqtradebot.exchange.init', MagicMock()) msg_mock = MagicMock() mocker.patch.multiple( 'freqtrade.rpc.telegram.Telegram', _init=MagicMock(), _send_msg=msg_mock ) - freqtradebot = FreqtradeBot(default_conf) + freqtradebot = get_patched_freqtradebot(mocker, default_conf) telegram = Telegram(freqtradebot) telegram._version(bot=MagicMock(), update=update) @@ -1035,11 +1018,10 @@ def test_send_msg(default_conf, mocker) -> None: Test send_msg() method """ patch_coinmarketcap(mocker) - mocker.patch('freqtrade.freqtradebot.exchange.init', MagicMock()) mocker.patch('freqtrade.rpc.telegram.Telegram._init', MagicMock()) conf = deepcopy(default_conf) bot = MagicMock() - freqtradebot = FreqtradeBot(conf) + freqtradebot = get_patched_freqtradebot(mocker, conf) telegram = Telegram(freqtradebot) telegram._config['telegram']['enabled'] = True @@ -1052,12 +1034,11 @@ def test_send_msg_network_error(default_conf, mocker, caplog) -> None: Test send_msg() method """ patch_coinmarketcap(mocker) - mocker.patch('freqtrade.freqtradebot.exchange.init', MagicMock()) mocker.patch('freqtrade.rpc.telegram.Telegram._init', MagicMock()) conf = deepcopy(default_conf) bot = MagicMock() bot.send_message = MagicMock(side_effect=NetworkError('Oh snap')) - freqtradebot = FreqtradeBot(conf) + freqtradebot = get_patched_freqtradebot(mocker, conf) telegram = Telegram(freqtradebot) telegram._config['telegram']['enabled'] = True From 251f7db3ca1ec7ef123a36eefd070bc8aeead60c Mon Sep 17 00:00:00 2001 From: xmatthias Date: Sun, 17 Jun 2018 22:11:32 +0200 Subject: [PATCH 22/64] require exchange object to delete pairs --- freqtrade/optimize/__init__.py | 11 ++++++++--- freqtrade/optimize/backtesting.py | 1 + 2 files changed, 9 insertions(+), 3 deletions(-) diff --git a/freqtrade/optimize/__init__.py b/freqtrade/optimize/__init__.py index 3e2fb4554..1e76808e7 100644 --- a/freqtrade/optimize/__init__.py +++ b/freqtrade/optimize/__init__.py @@ -7,7 +7,7 @@ import os from typing import Optional, List, Dict, Tuple, Any import arrow -from freqtrade import misc, constants +from freqtrade import misc, constants, OperationalException from freqtrade.exchange import Exchange from freqtrade.arguments import TimeRange @@ -83,6 +83,7 @@ def load_data(datadir: str, ticker_interval: str, pairs: List[str], refresh_pairs: Optional[bool] = False, + exchange: Optional[Exchange] = None, timerange: TimeRange = TimeRange(None, None, 0, 0)) -> Dict[str, List]: """ Loads ticker history data for the given parameters @@ -93,7 +94,10 @@ def load_data(datadir: str, # If the user force the refresh of pairs if refresh_pairs: logger.info('Download data for all pairs and store them in %s', datadir) - download_pairs(datadir, pairs, ticker_interval, timerange=timerange) + if not exchange: + raise OperationalException("Exchange needs to be initialized when " + "calling load_data with refresh_pairs=True") + download_pairs(datadir, exchange, pairs, ticker_interval, timerange=timerange) for pair in pairs: pairdata = load_tickerdata_file(datadir, pair, ticker_interval, timerange=timerange) @@ -119,13 +123,14 @@ def make_testdata_path(datadir: str) -> str: ) -def download_pairs(datadir, pairs: List[str], +def download_pairs(datadir, exchange: Exchange, pairs: List[str], ticker_interval: str, timerange: TimeRange = TimeRange(None, None, 0, 0)) -> bool: """For each pairs passed in parameters, download the ticker intervals""" for pair in pairs: try: download_backtesting_testdata(datadir, + exchange=exchange, pair=pair, tick_interval=ticker_interval, timerange=timerange) diff --git a/freqtrade/optimize/backtesting.py b/freqtrade/optimize/backtesting.py index 58b552237..9a19d1412 100644 --- a/freqtrade/optimize/backtesting.py +++ b/freqtrade/optimize/backtesting.py @@ -267,6 +267,7 @@ class Backtesting(object): pairs=pairs, ticker_interval=self.ticker_interval, refresh_pairs=self.config.get('refresh_pairs', False), + exchange=self.exchange, timerange=timerange ) From 52d36c33cf6e905753ef411741727d0f03a10691 Mon Sep 17 00:00:00 2001 From: xmatthias Date: Sun, 17 Jun 2018 22:12:22 +0200 Subject: [PATCH 23/64] fix optimie test --- freqtrade/tests/optimize/test_optimize.py | 60 +++++++++++++---------- 1 file changed, 33 insertions(+), 27 deletions(-) diff --git a/freqtrade/tests/optimize/test_optimize.py b/freqtrade/tests/optimize/test_optimize.py index bac8a6b36..624f8ed77 100644 --- a/freqtrade/tests/optimize/test_optimize.py +++ b/freqtrade/tests/optimize/test_optimize.py @@ -12,7 +12,7 @@ from freqtrade.optimize.__init__ import make_testdata_path, download_pairs, \ download_backtesting_testdata, load_tickerdata_file, trim_tickerlist, \ load_cached_data_for_updating from freqtrade.arguments import TimeRange -from freqtrade.tests.conftest import log_has +from freqtrade.tests.conftest import log_has, get_patched_exchange # Change this if modifying UNITTEST/BTC testdatafile _BTC_UNITTEST_LENGTH = 13681 @@ -49,12 +49,11 @@ def _clean_test_file(file: str) -> None: os.rename(file_swp, file) -def test_load_data_30min_ticker(ticker_history, mocker, caplog) -> None: +def test_load_data_30min_ticker(ticker_history, mocker, caplog, default_conf) -> None: """ Test load_data() with 30 min ticker """ - mocker.patch('freqtrade.optimize.get_ticker_history', return_value=ticker_history) - + mocker.patch('freqtrade.exchange.Exchange.get_ticker_history', return_value=ticker_history) file = os.path.join(os.path.dirname(__file__), '..', 'testdata', 'UNITTEST_BTC-30m.json') _backup_file(file, copy_file=True) optimize.load_data(None, pairs=['UNITTEST/BTC'], ticker_interval='30m') @@ -63,11 +62,11 @@ def test_load_data_30min_ticker(ticker_history, mocker, caplog) -> None: _clean_test_file(file) -def test_load_data_5min_ticker(ticker_history, mocker, caplog) -> None: +def test_load_data_5min_ticker(ticker_history, mocker, caplog, default_conf) -> None: """ Test load_data() with 5 min ticker """ - mocker.patch('freqtrade.optimize.get_ticker_history', return_value=ticker_history) + mocker.patch('freqtrade.exchange.Exchange.get_ticker_history', return_value=ticker_history) file = os.path.join(os.path.dirname(__file__), '..', 'testdata', 'UNITTEST_BTC-5m.json') _backup_file(file, copy_file=True) @@ -81,7 +80,7 @@ def test_load_data_1min_ticker(ticker_history, mocker, caplog) -> None: """ Test load_data() with 1 min ticker """ - mocker.patch('freqtrade.optimize.get_ticker_history', return_value=ticker_history) + mocker.patch('freqtrade.exchange.Exchange.get_ticker_history', return_value=ticker_history) file = os.path.join(os.path.dirname(__file__), '..', 'testdata', 'UNITTEST_BTC-1m.json') _backup_file(file, copy_file=True) @@ -91,12 +90,12 @@ def test_load_data_1min_ticker(ticker_history, mocker, caplog) -> None: _clean_test_file(file) -def test_load_data_with_new_pair_1min(ticker_history, mocker, caplog) -> None: +def test_load_data_with_new_pair_1min(ticker_history, mocker, caplog, default_conf) -> None: """ Test load_data() with 1 min ticker """ - mocker.patch('freqtrade.optimize.get_ticker_history', return_value=ticker_history) - + mocker.patch('freqtrade.exchange.Exchange.get_ticker_history', return_value=ticker_history) + exchange = get_patched_exchange(mocker, default_conf) file = os.path.join(os.path.dirname(__file__), '..', 'testdata', 'MEME_BTC-1m.json') _backup_file(file) @@ -114,6 +113,7 @@ def test_load_data_with_new_pair_1min(ticker_history, mocker, caplog) -> None: optimize.load_data(None, ticker_interval='1m', refresh_pairs=True, + exchange=exchange, pairs=['MEME/BTC']) assert os.path.isfile(file) is True assert log_has('Download the pair: "MEME/BTC", Interval: 1m', caplog.record_tuples) @@ -124,9 +124,9 @@ def test_testdata_path() -> None: assert os.path.join('freqtrade', 'tests', 'testdata') in make_testdata_path(None) -def test_download_pairs(ticker_history, mocker) -> None: - mocker.patch('freqtrade.optimize.__init__.get_ticker_history', return_value=ticker_history) - +def test_download_pairs(ticker_history, mocker, default_conf) -> None: + mocker.patch('freqtrade.exchange.Exchange.get_ticker_history', return_value=ticker_history) + exchange = get_patched_exchange(mocker, default_conf) file1_1 = os.path.join(os.path.dirname(__file__), '..', 'testdata', 'MEME_BTC-1m.json') file1_5 = os.path.join(os.path.dirname(__file__), '..', 'testdata', 'MEME_BTC-5m.json') file2_1 = os.path.join(os.path.dirname(__file__), '..', 'testdata', 'CFI_BTC-1m.json') @@ -140,7 +140,8 @@ def test_download_pairs(ticker_history, mocker) -> None: assert os.path.isfile(file1_1) is False assert os.path.isfile(file2_1) is False - assert download_pairs(None, pairs=['MEME/BTC', 'CFI/BTC'], ticker_interval='1m') is True + assert download_pairs(None, exchange, + pairs=['MEME/BTC', 'CFI/BTC'], ticker_interval='1m') is True assert os.path.isfile(file1_1) is True assert os.path.isfile(file2_1) is True @@ -152,7 +153,8 @@ def test_download_pairs(ticker_history, mocker) -> None: assert os.path.isfile(file1_5) is False assert os.path.isfile(file2_5) is False - assert download_pairs(None, pairs=['MEME/BTC', 'CFI/BTC'], ticker_interval='5m') is True + assert download_pairs(None, exchange, + pairs=['MEME/BTC', 'CFI/BTC'], ticker_interval='5m') is True assert os.path.isfile(file1_5) is True assert os.path.isfile(file2_5) is True @@ -265,30 +267,32 @@ def test_load_cached_data_for_updating(mocker) -> None: assert start_ts is None -def test_download_pairs_exception(ticker_history, mocker, caplog) -> None: - mocker.patch('freqtrade.optimize.__init__.get_ticker_history', return_value=ticker_history) +def test_download_pairs_exception(ticker_history, mocker, caplog, default_conf) -> None: + mocker.patch('freqtrade.exchange.Exchange.get_ticker_history', return_value=ticker_history) mocker.patch('freqtrade.optimize.__init__.download_backtesting_testdata', side_effect=BaseException('File Error')) + exchange = get_patched_exchange(mocker, default_conf) file1_1 = os.path.join(os.path.dirname(__file__), '..', 'testdata', 'MEME_BTC-1m.json') file1_5 = os.path.join(os.path.dirname(__file__), '..', 'testdata', 'MEME_BTC-5m.json') _backup_file(file1_1) _backup_file(file1_5) - download_pairs(None, pairs=['MEME/BTC'], ticker_interval='1m') + download_pairs(None, exchange, pairs=['MEME/BTC'], ticker_interval='1m') # clean files freshly downloaded _clean_test_file(file1_1) _clean_test_file(file1_5) assert log_has('Failed to download the pair: "MEME/BTC", Interval: 1m', caplog.record_tuples) -def test_download_backtesting_testdata(ticker_history, mocker) -> None: - mocker.patch('freqtrade.optimize.__init__.get_ticker_history', return_value=ticker_history) +def test_download_backtesting_testdata(ticker_history, mocker, default_conf) -> None: + mocker.patch('freqtrade.exchange.Exchange.get_ticker_history', return_value=ticker_history) + exchange = get_patched_exchange(mocker, default_conf) # Download a 1 min ticker file file1 = os.path.join(os.path.dirname(__file__), '..', 'testdata', 'XEL_BTC-1m.json') _backup_file(file1) - download_backtesting_testdata(None, pair="XEL/BTC", tick_interval='1m') + download_backtesting_testdata(None, exchange, pair="XEL/BTC", tick_interval='1m') assert os.path.isfile(file1) is True _clean_test_file(file1) @@ -296,21 +300,21 @@ def test_download_backtesting_testdata(ticker_history, mocker) -> None: file2 = os.path.join(os.path.dirname(__file__), '..', 'testdata', 'STORJ_BTC-5m.json') _backup_file(file2) - download_backtesting_testdata(None, pair="STORJ/BTC", tick_interval='5m') + download_backtesting_testdata(None, exchange, pair="STORJ/BTC", tick_interval='5m') assert os.path.isfile(file2) is True _clean_test_file(file2) -def test_download_backtesting_testdata2(mocker) -> None: +def test_download_backtesting_testdata2(mocker, default_conf) -> None: tick = [ [1509836520000, 0.00162008, 0.00162008, 0.00162008, 0.00162008, 108.14853839], [1509836580000, 0.00161, 0.00161, 0.00161, 0.00161, 82.390199] ] json_dump_mock = mocker.patch('freqtrade.misc.file_dump_json', return_value=None) - mocker.patch('freqtrade.optimize.__init__.get_ticker_history', return_value=tick) - - download_backtesting_testdata(None, pair="UNITTEST/BTC", tick_interval='1m') - download_backtesting_testdata(None, pair="UNITTEST/BTC", tick_interval='3m') + mocker.patch('freqtrade.exchange.Exchange.get_ticker_history', return_value=tick) + exchange = get_patched_exchange(mocker, default_conf) + download_backtesting_testdata(None, exchange, pair="UNITTEST/BTC", tick_interval='1m') + download_backtesting_testdata(None, exchange, pair="UNITTEST/BTC", tick_interval='3m') assert json_dump_mock.call_count == 2 @@ -326,8 +330,10 @@ def test_load_tickerdata_file() -> None: def test_init(default_conf, mocker) -> None: + exchange = get_patched_exchange(mocker, default_conf) assert {} == optimize.load_data( '', + exchange=exchange, pairs=[], refresh_pairs=True, ticker_interval=default_conf['ticker_interval'] From ace519847536f53980be2eba89ed1bcc944031a6 Mon Sep 17 00:00:00 2001 From: xmatthias Date: Sun, 17 Jun 2018 22:29:57 +0200 Subject: [PATCH 24/64] fix optimize tests --- freqtrade/tests/optimize/test_backtesting.py | 58 ++++++++++---------- freqtrade/tests/optimize/test_hyperopt.py | 15 +++-- 2 files changed, 35 insertions(+), 38 deletions(-) diff --git a/freqtrade/tests/optimize/test_backtesting.py b/freqtrade/tests/optimize/test_backtesting.py index 31080c503..58fa3a3f3 100644 --- a/freqtrade/tests/optimize/test_backtesting.py +++ b/freqtrade/tests/optimize/test_backtesting.py @@ -83,7 +83,7 @@ def load_data_test(what): def simple_backtest(config, contour, num_results, mocker) -> None: - mocker.patch('freqtrade.exchange.validate_pairs', MagicMock(return_value=True)) + mocker.patch('freqtrade.exchange.Exchange.validate_pairs', MagicMock(return_value=True)) backtesting = Backtesting(config) data = load_data_test(contour) @@ -101,7 +101,8 @@ def simple_backtest(config, contour, num_results, mocker) -> None: assert len(results) == num_results -def mocked_load_data(datadir, pairs=[], ticker_interval='0m', refresh_pairs=False, timerange=None): +def mocked_load_data(datadir, pairs=[], ticker_interval='0m', refresh_pairs=False, + timerange=None, exchange=None): tickerdata = optimize.load_tickerdata_file(datadir, 'UNITTEST/BTC', '1m', timerange=timerange) pairdata = {'UNITTEST/BTC': tickerdata} return pairdata @@ -118,7 +119,7 @@ def _load_pair_as_ticks(pair, tickfreq): def _make_backtest_conf(mocker, conf=None, pair='UNITTEST/BTC', record=None): data = optimize.load_data(None, ticker_interval='8m', pairs=[pair]) data = trim_dictlist(data, -201) - mocker.patch('freqtrade.exchange.validate_pairs', MagicMock(return_value=True)) + mocker.patch('freqtrade.exchange.Exchange.validate_pairs', MagicMock(return_value=True)) backtesting = Backtesting(conf) return { 'stake_amount': conf['stake_amount'], @@ -272,8 +273,8 @@ def test_start(mocker, fee, default_conf, caplog) -> None: Test start() function """ start_mock = MagicMock() - mocker.patch('freqtrade.exchange.get_fee', fee) - mocker.patch('freqtrade.exchange.validate_pairs', MagicMock(return_value=True)) + mocker.patch('freqtrade.exchange.Exchange.get_fee', fee) + mocker.patch('freqtrade.exchange.Exchange.validate_pairs', MagicMock(return_value=True)) mocker.patch('freqtrade.optimize.backtesting.Backtesting.start', start_mock) mocker.patch('freqtrade.configuration.open', mocker.mock_open( read_data=json.dumps(default_conf) @@ -296,7 +297,7 @@ def test_backtesting_init(mocker, default_conf) -> None: """ Test Backtesting._init() method """ - mocker.patch('freqtrade.exchange.validate_pairs', MagicMock(return_value=True)) + mocker.patch('freqtrade.exchange.Exchange.validate_pairs', MagicMock(return_value=True)) backtesting = Backtesting(default_conf) assert backtesting.config == default_conf assert isinstance(backtesting.analyze, Analyze) @@ -310,7 +311,7 @@ def test_tickerdata_to_dataframe(default_conf, mocker) -> None: """ Test Backtesting.tickerdata_to_dataframe() method """ - mocker.patch('freqtrade.exchange.validate_pairs', MagicMock(return_value=True)) + mocker.patch('freqtrade.exchange.Exchange.validate_pairs', MagicMock(return_value=True)) timerange = TimeRange(None, 'line', 0, -100) tick = optimize.load_tickerdata_file(None, 'UNITTEST/BTC', '1m', timerange=timerange) tickerlist = {'UNITTEST/BTC': tick} @@ -329,7 +330,7 @@ def test_get_timeframe(default_conf, mocker) -> None: """ Test Backtesting.get_timeframe() method """ - mocker.patch('freqtrade.exchange.validate_pairs', MagicMock(return_value=True)) + mocker.patch('freqtrade.exchange.Exchange.validate_pairs', MagicMock(return_value=True)) backtesting = Backtesting(default_conf) data = backtesting.tickerdata_to_dataframe( @@ -348,7 +349,7 @@ def test_generate_text_table(default_conf, mocker): """ Test Backtesting.generate_text_table() method """ - mocker.patch('freqtrade.exchange.validate_pairs', MagicMock(return_value=True)) + mocker.patch('freqtrade.exchange.Exchange.validate_pairs', MagicMock(return_value=True)) backtesting = Backtesting(default_conf) results = pd.DataFrame( @@ -385,8 +386,8 @@ def test_backtesting_start(default_conf, mocker, caplog) -> None: mocker.patch('freqtrade.freqtradebot.Analyze', MagicMock()) mocker.patch('freqtrade.optimize.load_data', mocked_load_data) - mocker.patch('freqtrade.exchange.get_ticker_history') - mocker.patch('freqtrade.exchange.validate_pairs', MagicMock(return_value=True)) + mocker.patch('freqtrade.exchange.Exchange.get_ticker_history') + mocker.patch('freqtrade.exchange.Exchange.validate_pairs', MagicMock(return_value=True)) mocker.patch.multiple( 'freqtrade.optimize.backtesting.Backtesting', backtest=MagicMock(), @@ -426,8 +427,8 @@ def test_backtesting_start_no_data(default_conf, mocker, caplog) -> None: mocker.patch('freqtrade.freqtradebot.Analyze', MagicMock()) mocker.patch('freqtrade.optimize.load_data', MagicMock(return_value={})) - mocker.patch('freqtrade.exchange.get_ticker_history') - mocker.patch('freqtrade.exchange.validate_pairs', MagicMock(return_value=True)) + mocker.patch('freqtrade.exchange.Exchange.get_ticker_history') + mocker.patch('freqtrade.exchange.Exchange.validate_pairs', MagicMock(return_value=True)) mocker.patch.multiple( 'freqtrade.optimize.backtesting.Backtesting', backtest=MagicMock(), @@ -454,8 +455,8 @@ def test_backtest(default_conf, fee, mocker) -> None: """ Test Backtesting.backtest() method """ - mocker.patch('freqtrade.exchange.get_fee', fee) - mocker.patch('freqtrade.exchange.validate_pairs', MagicMock(return_value=True)) + mocker.patch('freqtrade.exchange.Exchange.get_fee', fee) + mocker.patch('freqtrade.exchange.Exchange.validate_pairs', MagicMock(return_value=True)) backtesting = Backtesting(default_conf) data = optimize.load_data(None, ticker_interval='5m', pairs=['UNITTEST/BTC']) @@ -476,8 +477,8 @@ def test_backtest_1min_ticker_interval(default_conf, fee, mocker) -> None: """ Test Backtesting.backtest() method with 1 min ticker """ - mocker.patch('freqtrade.exchange.get_fee', fee) - mocker.patch('freqtrade.exchange.validate_pairs', MagicMock(return_value=True)) + mocker.patch('freqtrade.exchange.Exchange.get_fee', fee) + mocker.patch('freqtrade.exchange.Exchange.validate_pairs', MagicMock(return_value=True)) backtesting = Backtesting(default_conf) # Run a backtesting for an exiting 5min ticker_interval @@ -499,7 +500,7 @@ def test_processed(default_conf, mocker) -> None: """ Test Backtesting.backtest() method with offline data """ - mocker.patch('freqtrade.exchange.validate_pairs', MagicMock(return_value=True)) + mocker.patch('freqtrade.exchange.Exchange.validate_pairs', MagicMock(return_value=True)) backtesting = Backtesting(default_conf) dict_of_tickerrows = load_data_test('raise') @@ -513,7 +514,7 @@ def test_processed(default_conf, mocker) -> None: def test_backtest_pricecontours(default_conf, fee, mocker) -> None: - mocker.patch('freqtrade.optimize.backtesting.exchange.get_fee', fee) + mocker.patch('freqtrade.exchange.Exchange.get_fee', fee) tests = [['raise', 18], ['lower', 0], ['sine', 16]] for [contour, numres] in tests: simple_backtest(default_conf, contour, numres, mocker) @@ -521,8 +522,8 @@ def test_backtest_pricecontours(default_conf, fee, mocker) -> None: # Test backtest using offline data (testdata directory) def test_backtest_ticks(default_conf, fee, mocker): - mocker.patch('freqtrade.exchange.get_fee', fee) - mocker.patch('freqtrade.exchange.validate_pairs', MagicMock(return_value=True)) + mocker.patch('freqtrade.exchange.Exchange.get_fee', fee) + mocker.patch('freqtrade.exchange.Exchange.validate_pairs', MagicMock(return_value=True)) ticks = [1, 5] fun = Backtesting(default_conf).populate_buy_trend for _ in ticks: @@ -541,7 +542,6 @@ def test_backtest_clash_buy_sell(mocker, default_conf): sell_value = 1 return _trend(dataframe, buy_value, sell_value) - mocker.patch('freqtrade.exchange.validate_pairs', MagicMock(return_value=True)) backtest_conf = _make_backtest_conf(mocker, conf=default_conf) backtesting = Backtesting(default_conf) backtesting.populate_buy_trend = fun # Override @@ -557,7 +557,6 @@ def test_backtest_only_sell(mocker, default_conf): sell_value = 1 return _trend(dataframe, buy_value, sell_value) - mocker.patch('freqtrade.exchange.validate_pairs', MagicMock(return_value=True)) backtest_conf = _make_backtest_conf(mocker, conf=default_conf) backtesting = Backtesting(default_conf) backtesting.populate_buy_trend = fun # Override @@ -567,8 +566,7 @@ def test_backtest_only_sell(mocker, default_conf): def test_backtest_alternate_buy_sell(default_conf, fee, mocker): - mocker.patch('freqtrade.optimize.backtesting.exchange.get_fee', fee) - mocker.patch('freqtrade.exchange.validate_pairs', MagicMock(return_value=True)) + mocker.patch('freqtrade.exchange.Exchange.get_fee', fee) backtest_conf = _make_backtest_conf(mocker, conf=default_conf, pair='UNITTEST/BTC') backtesting = Backtesting(default_conf) backtesting.populate_buy_trend = _trend_alternate # Override @@ -583,8 +581,8 @@ def test_backtest_alternate_buy_sell(default_conf, fee, mocker): def test_backtest_record(default_conf, fee, mocker): names = [] records = [] - mocker.patch('freqtrade.exchange.validate_pairs', MagicMock(return_value=True)) - mocker.patch('freqtrade.optimize.backtesting.exchange.get_fee', fee) + mocker.patch('freqtrade.exchange.Exchange.validate_pairs', MagicMock(return_value=True)) + mocker.patch('freqtrade.exchange.Exchange.get_fee', fee) mocker.patch( 'freqtrade.optimize.backtesting.file_dump_json', new=lambda n, r: (names.append(n), records.append(r)) @@ -632,9 +630,9 @@ def test_backtest_record(default_conf, fee, mocker): def test_backtest_start_live(default_conf, mocker, caplog): conf = deepcopy(default_conf) conf['exchange']['pair_whitelist'] = ['UNITTEST/BTC'] - mocker.patch('freqtrade.exchange.get_ticker_history', - new=lambda n, i: _load_pair_as_ticks(n, i)) - mocker.patch('freqtrade.exchange.validate_pairs', MagicMock()) + mocker.patch('freqtrade.exchange.Exchange.get_ticker_history', + new=lambda s, n, i: _load_pair_as_ticks(n, i)) + mocker.patch('freqtrade.exchange.Exchange.validate_pairs', MagicMock()) mocker.patch('freqtrade.optimize.backtesting.Backtesting.backtest', MagicMock()) mocker.patch('freqtrade.optimize.backtesting.Backtesting._generate_text_table', MagicMock()) mocker.patch('freqtrade.configuration.open', mocker.mock_open( diff --git a/freqtrade/tests/optimize/test_hyperopt.py b/freqtrade/tests/optimize/test_hyperopt.py index 9b66a29e2..616d56c8f 100644 --- a/freqtrade/tests/optimize/test_hyperopt.py +++ b/freqtrade/tests/optimize/test_hyperopt.py @@ -22,8 +22,7 @@ _HYPEROPT = None def init_hyperopt(default_conf, mocker): global _HYPEROPT_INITIALIZED, _HYPEROPT if not _HYPEROPT_INITIALIZED: - mocker.patch('freqtrade.exchange.validate_pairs', MagicMock(return_value=True)) - mocker.patch('freqtrade.exchange.validate_pairs', MagicMock()) + mocker.patch('freqtrade.exchange.Exchange.validate_pairs', MagicMock(return_value=True)) _HYPEROPT = Hyperopt(default_conf) _HYPEROPT_INITIALIZED = True @@ -66,7 +65,7 @@ def test_start(mocker, default_conf, caplog) -> None: lambda *args, **kwargs: default_conf ) mocker.patch('freqtrade.optimize.hyperopt.Hyperopt.start', start_mock) - mocker.patch('freqtrade.freqtradebot.exchange.validate_pairs', MagicMock()) + mocker.patch('freqtrade.exchange.Exchange.validate_pairs', MagicMock()) args = [ '--config', 'config.json', @@ -182,7 +181,7 @@ def test_fmin_best_results(mocker, init_hyperopt, default_conf, caplog) -> None: mocker.patch('freqtrade.optimize.hyperopt.load_data', MagicMock()) mocker.patch('freqtrade.optimize.hyperopt.fmin', return_value=fmin_result) - mocker.patch('freqtrade.freqtradebot.exchange.validate_pairs', MagicMock()) + mocker.patch('freqtrade.exchange.Exchange.validate_pairs', MagicMock()) StrategyResolver({'strategy': 'DefaultStrategy'}) hyperopt = Hyperopt(conf) @@ -226,7 +225,7 @@ def test_fmin_throw_value_error(mocker, init_hyperopt, default_conf, caplog) -> conf.update({'epochs': 1}) conf.update({'timerange': None}) conf.update({'spaces': 'all'}) - mocker.patch('freqtrade.freqtradebot.exchange.validate_pairs', MagicMock()) + mocker.patch('freqtrade.exchange.Exchange.validate_pairs', MagicMock()) StrategyResolver({'strategy': 'DefaultStrategy'}) hyperopt = Hyperopt(conf) @@ -267,7 +266,7 @@ def test_resuming_previous_hyperopt_results_succeeds(mocker, init_hyperopt, defa mocker.patch('freqtrade.optimize.hyperopt.sorted', return_value=trials.results) mocker.patch('freqtrade.optimize.hyperopt.load_data', MagicMock()) mocker.patch('freqtrade.optimize.hyperopt.fmin', return_value={}) - mocker.patch('freqtrade.exchange.validate_pairs', MagicMock()) + mocker.patch('freqtrade.exchange.Exchange.validate_pairs', MagicMock()) StrategyResolver({'strategy': 'DefaultStrategy'}) hyperopt = Hyperopt(conf) @@ -338,7 +337,7 @@ def test_start_calls_fmin(mocker, init_hyperopt, default_conf) -> None: trials = create_trials(mocker) mocker.patch('freqtrade.optimize.hyperopt.sorted', return_value=trials.results) mocker.patch('freqtrade.optimize.hyperopt.load_data', MagicMock()) - mocker.patch('freqtrade.exchange.validate_pairs', MagicMock()) + mocker.patch('freqtrade.exchange.Exchange.validate_pairs', MagicMock()) mock_fmin = mocker.patch('freqtrade.optimize.hyperopt.fmin', return_value={}) conf = deepcopy(default_conf) @@ -503,7 +502,7 @@ def test_generate_optimizer(mocker, init_hyperopt, default_conf) -> None: 'freqtrade.optimize.hyperopt.Hyperopt.backtest', MagicMock(return_value=backtest_result) ) - mocker.patch('freqtrade.exchange.validate_pairs', MagicMock()) + mocker.patch('freqtrade.exchange.Exchange.validate_pairs', MagicMock()) optimizer_param = { 'adx': {'enabled': False}, From e194af8d254d599f662e069220cbecb092c1a3fc Mon Sep 17 00:00:00 2001 From: xmatthias Date: Sun, 17 Jun 2018 22:32:56 +0200 Subject: [PATCH 25/64] Streamline validate_pair patching --- freqtrade/tests/optimize/test_backtesting.py | 32 ++++++++++---------- freqtrade/tests/optimize/test_hyperopt.py | 18 ++++++----- 2 files changed, 26 insertions(+), 24 deletions(-) diff --git a/freqtrade/tests/optimize/test_backtesting.py b/freqtrade/tests/optimize/test_backtesting.py index 58fa3a3f3..1702818b1 100644 --- a/freqtrade/tests/optimize/test_backtesting.py +++ b/freqtrade/tests/optimize/test_backtesting.py @@ -15,7 +15,7 @@ from freqtrade import optimize from freqtrade.analyze import Analyze from freqtrade.arguments import Arguments, TimeRange from freqtrade.optimize.backtesting import Backtesting, start, setup_configuration -from freqtrade.tests.conftest import log_has +from freqtrade.tests.conftest import log_has, patch_exchange def get_args(args) -> List[str]: @@ -83,7 +83,7 @@ def load_data_test(what): def simple_backtest(config, contour, num_results, mocker) -> None: - mocker.patch('freqtrade.exchange.Exchange.validate_pairs', MagicMock(return_value=True)) + patch_exchange(mocker) backtesting = Backtesting(config) data = load_data_test(contour) @@ -119,7 +119,7 @@ def _load_pair_as_ticks(pair, tickfreq): def _make_backtest_conf(mocker, conf=None, pair='UNITTEST/BTC', record=None): data = optimize.load_data(None, ticker_interval='8m', pairs=[pair]) data = trim_dictlist(data, -201) - mocker.patch('freqtrade.exchange.Exchange.validate_pairs', MagicMock(return_value=True)) + patch_exchange(mocker) backtesting = Backtesting(conf) return { 'stake_amount': conf['stake_amount'], @@ -274,7 +274,7 @@ def test_start(mocker, fee, default_conf, caplog) -> None: """ start_mock = MagicMock() mocker.patch('freqtrade.exchange.Exchange.get_fee', fee) - mocker.patch('freqtrade.exchange.Exchange.validate_pairs', MagicMock(return_value=True)) + patch_exchange(mocker) mocker.patch('freqtrade.optimize.backtesting.Backtesting.start', start_mock) mocker.patch('freqtrade.configuration.open', mocker.mock_open( read_data=json.dumps(default_conf) @@ -297,7 +297,7 @@ def test_backtesting_init(mocker, default_conf) -> None: """ Test Backtesting._init() method """ - mocker.patch('freqtrade.exchange.Exchange.validate_pairs', MagicMock(return_value=True)) + patch_exchange(mocker) backtesting = Backtesting(default_conf) assert backtesting.config == default_conf assert isinstance(backtesting.analyze, Analyze) @@ -311,7 +311,7 @@ def test_tickerdata_to_dataframe(default_conf, mocker) -> None: """ Test Backtesting.tickerdata_to_dataframe() method """ - mocker.patch('freqtrade.exchange.Exchange.validate_pairs', MagicMock(return_value=True)) + patch_exchange(mocker) timerange = TimeRange(None, 'line', 0, -100) tick = optimize.load_tickerdata_file(None, 'UNITTEST/BTC', '1m', timerange=timerange) tickerlist = {'UNITTEST/BTC': tick} @@ -330,7 +330,7 @@ def test_get_timeframe(default_conf, mocker) -> None: """ Test Backtesting.get_timeframe() method """ - mocker.patch('freqtrade.exchange.Exchange.validate_pairs', MagicMock(return_value=True)) + patch_exchange(mocker) backtesting = Backtesting(default_conf) data = backtesting.tickerdata_to_dataframe( @@ -349,7 +349,7 @@ def test_generate_text_table(default_conf, mocker): """ Test Backtesting.generate_text_table() method """ - mocker.patch('freqtrade.exchange.Exchange.validate_pairs', MagicMock(return_value=True)) + patch_exchange(mocker) backtesting = Backtesting(default_conf) results = pd.DataFrame( @@ -387,7 +387,7 @@ def test_backtesting_start(default_conf, mocker, caplog) -> None: mocker.patch('freqtrade.freqtradebot.Analyze', MagicMock()) mocker.patch('freqtrade.optimize.load_data', mocked_load_data) mocker.patch('freqtrade.exchange.Exchange.get_ticker_history') - mocker.patch('freqtrade.exchange.Exchange.validate_pairs', MagicMock(return_value=True)) + patch_exchange(mocker) mocker.patch.multiple( 'freqtrade.optimize.backtesting.Backtesting', backtest=MagicMock(), @@ -428,7 +428,7 @@ def test_backtesting_start_no_data(default_conf, mocker, caplog) -> None: mocker.patch('freqtrade.freqtradebot.Analyze', MagicMock()) mocker.patch('freqtrade.optimize.load_data', MagicMock(return_value={})) mocker.patch('freqtrade.exchange.Exchange.get_ticker_history') - mocker.patch('freqtrade.exchange.Exchange.validate_pairs', MagicMock(return_value=True)) + patch_exchange(mocker) mocker.patch.multiple( 'freqtrade.optimize.backtesting.Backtesting', backtest=MagicMock(), @@ -456,7 +456,7 @@ def test_backtest(default_conf, fee, mocker) -> None: Test Backtesting.backtest() method """ mocker.patch('freqtrade.exchange.Exchange.get_fee', fee) - mocker.patch('freqtrade.exchange.Exchange.validate_pairs', MagicMock(return_value=True)) + patch_exchange(mocker) backtesting = Backtesting(default_conf) data = optimize.load_data(None, ticker_interval='5m', pairs=['UNITTEST/BTC']) @@ -478,7 +478,7 @@ def test_backtest_1min_ticker_interval(default_conf, fee, mocker) -> None: Test Backtesting.backtest() method with 1 min ticker """ mocker.patch('freqtrade.exchange.Exchange.get_fee', fee) - mocker.patch('freqtrade.exchange.Exchange.validate_pairs', MagicMock(return_value=True)) + patch_exchange(mocker) backtesting = Backtesting(default_conf) # Run a backtesting for an exiting 5min ticker_interval @@ -500,7 +500,7 @@ def test_processed(default_conf, mocker) -> None: """ Test Backtesting.backtest() method with offline data """ - mocker.patch('freqtrade.exchange.Exchange.validate_pairs', MagicMock(return_value=True)) + patch_exchange(mocker) backtesting = Backtesting(default_conf) dict_of_tickerrows = load_data_test('raise') @@ -523,7 +523,7 @@ def test_backtest_pricecontours(default_conf, fee, mocker) -> None: # Test backtest using offline data (testdata directory) def test_backtest_ticks(default_conf, fee, mocker): mocker.patch('freqtrade.exchange.Exchange.get_fee', fee) - mocker.patch('freqtrade.exchange.Exchange.validate_pairs', MagicMock(return_value=True)) + patch_exchange(mocker) ticks = [1, 5] fun = Backtesting(default_conf).populate_buy_trend for _ in ticks: @@ -581,7 +581,7 @@ def test_backtest_alternate_buy_sell(default_conf, fee, mocker): def test_backtest_record(default_conf, fee, mocker): names = [] records = [] - mocker.patch('freqtrade.exchange.Exchange.validate_pairs', MagicMock(return_value=True)) + patch_exchange(mocker) mocker.patch('freqtrade.exchange.Exchange.get_fee', fee) mocker.patch( 'freqtrade.optimize.backtesting.file_dump_json', @@ -632,7 +632,7 @@ def test_backtest_start_live(default_conf, mocker, caplog): conf['exchange']['pair_whitelist'] = ['UNITTEST/BTC'] mocker.patch('freqtrade.exchange.Exchange.get_ticker_history', new=lambda s, n, i: _load_pair_as_ticks(n, i)) - mocker.patch('freqtrade.exchange.Exchange.validate_pairs', MagicMock()) + patch_exchange(mocker) mocker.patch('freqtrade.optimize.backtesting.Backtesting.backtest', MagicMock()) mocker.patch('freqtrade.optimize.backtesting.Backtesting._generate_text_table', MagicMock()) mocker.patch('freqtrade.configuration.open', mocker.mock_open( diff --git a/freqtrade/tests/optimize/test_hyperopt.py b/freqtrade/tests/optimize/test_hyperopt.py index 616d56c8f..7de6bcb7c 100644 --- a/freqtrade/tests/optimize/test_hyperopt.py +++ b/freqtrade/tests/optimize/test_hyperopt.py @@ -10,7 +10,7 @@ import pytest from freqtrade.optimize.__init__ import load_tickerdata_file from freqtrade.optimize.hyperopt import Hyperopt, start from freqtrade.strategy.resolver import StrategyResolver -from freqtrade.tests.conftest import log_has +from freqtrade.tests.conftest import log_has, patch_exchange from freqtrade.tests.optimize.test_backtesting import get_args # Avoid to reinit the same object again and again @@ -22,7 +22,7 @@ _HYPEROPT = None def init_hyperopt(default_conf, mocker): global _HYPEROPT_INITIALIZED, _HYPEROPT if not _HYPEROPT_INITIALIZED: - mocker.patch('freqtrade.exchange.Exchange.validate_pairs', MagicMock(return_value=True)) + patch_exchange(mocker) _HYPEROPT = Hyperopt(default_conf) _HYPEROPT_INITIALIZED = True @@ -65,7 +65,8 @@ def test_start(mocker, default_conf, caplog) -> None: lambda *args, **kwargs: default_conf ) mocker.patch('freqtrade.optimize.hyperopt.Hyperopt.start', start_mock) - mocker.patch('freqtrade.exchange.Exchange.validate_pairs', MagicMock()) + patch_exchange(mocker) + args = [ '--config', 'config.json', @@ -181,7 +182,8 @@ def test_fmin_best_results(mocker, init_hyperopt, default_conf, caplog) -> None: mocker.patch('freqtrade.optimize.hyperopt.load_data', MagicMock()) mocker.patch('freqtrade.optimize.hyperopt.fmin', return_value=fmin_result) - mocker.patch('freqtrade.exchange.Exchange.validate_pairs', MagicMock()) + patch_exchange(mocker) + StrategyResolver({'strategy': 'DefaultStrategy'}) hyperopt = Hyperopt(conf) @@ -225,7 +227,7 @@ def test_fmin_throw_value_error(mocker, init_hyperopt, default_conf, caplog) -> conf.update({'epochs': 1}) conf.update({'timerange': None}) conf.update({'spaces': 'all'}) - mocker.patch('freqtrade.exchange.Exchange.validate_pairs', MagicMock()) + patch_exchange(mocker) StrategyResolver({'strategy': 'DefaultStrategy'}) hyperopt = Hyperopt(conf) @@ -266,7 +268,7 @@ def test_resuming_previous_hyperopt_results_succeeds(mocker, init_hyperopt, defa mocker.patch('freqtrade.optimize.hyperopt.sorted', return_value=trials.results) mocker.patch('freqtrade.optimize.hyperopt.load_data', MagicMock()) mocker.patch('freqtrade.optimize.hyperopt.fmin', return_value={}) - mocker.patch('freqtrade.exchange.Exchange.validate_pairs', MagicMock()) + patch_exchange(mocker) StrategyResolver({'strategy': 'DefaultStrategy'}) hyperopt = Hyperopt(conf) @@ -337,7 +339,7 @@ def test_start_calls_fmin(mocker, init_hyperopt, default_conf) -> None: trials = create_trials(mocker) mocker.patch('freqtrade.optimize.hyperopt.sorted', return_value=trials.results) mocker.patch('freqtrade.optimize.hyperopt.load_data', MagicMock()) - mocker.patch('freqtrade.exchange.Exchange.validate_pairs', MagicMock()) + patch_exchange(mocker) mock_fmin = mocker.patch('freqtrade.optimize.hyperopt.fmin', return_value={}) conf = deepcopy(default_conf) @@ -502,7 +504,7 @@ def test_generate_optimizer(mocker, init_hyperopt, default_conf) -> None: 'freqtrade.optimize.hyperopt.Hyperopt.backtest', MagicMock(return_value=backtest_result) ) - mocker.patch('freqtrade.exchange.Exchange.validate_pairs', MagicMock()) + patch_exchange(mocker) optimizer_param = { 'adx': {'enabled': False}, From 6e6ec969ebe72f60929846d4e2f25e0ea5d08b56 Mon Sep 17 00:00:00 2001 From: xmatthias Date: Sun, 17 Jun 2018 22:38:31 +0200 Subject: [PATCH 26/64] cleanup mockings --- freqtrade/tests/test_freqtradebot.py | 5 ++--- freqtrade/tests/test_main.py | 7 ++++++- 2 files changed, 8 insertions(+), 4 deletions(-) diff --git a/freqtrade/tests/test_freqtradebot.py b/freqtrade/tests/test_freqtradebot.py index 797456328..9eae2fdf5 100644 --- a/freqtrade/tests/test_freqtradebot.py +++ b/freqtrade/tests/test_freqtradebot.py @@ -18,7 +18,7 @@ from freqtrade import DependencyException, OperationalException, TemporaryError from freqtrade.freqtradebot import FreqtradeBot from freqtrade.persistence import Trade from freqtrade.state import State -from freqtrade.tests.conftest import log_has, patch_coinmarketcap +from freqtrade.tests.conftest import log_has, patch_coinmarketcap, patch_exchange # Functions for recurrent object patching @@ -32,8 +32,7 @@ def get_patched_freqtradebot(mocker, config) -> FreqtradeBot: mocker.patch('freqtrade.freqtradebot.Analyze', MagicMock()) mocker.patch('freqtrade.freqtradebot.RPCManager', MagicMock()) mocker.patch('freqtrade.freqtradebot.persistence.init', MagicMock()) - mocker.patch('freqtrade.exchange.Exchange._init_ccxt', MagicMock()) - mocker.patch('freqtrade.exchange.Exchange.validate_pairs', MagicMock()) + patch_exchange(mocker) patch_coinmarketcap(mocker) return FreqtradeBot(config) diff --git a/freqtrade/tests/test_main.py b/freqtrade/tests/test_main.py index 9640a7350..414dcdbe9 100644 --- a/freqtrade/tests/test_main.py +++ b/freqtrade/tests/test_main.py @@ -13,7 +13,7 @@ from freqtrade.arguments import Arguments from freqtrade.freqtradebot import FreqtradeBot from freqtrade.main import main, set_loggers, reconfigure from freqtrade.state import State -from freqtrade.tests.conftest import log_has +from freqtrade.tests.conftest import log_has, patch_exchange def test_parse_args_backtesting(mocker) -> None: @@ -70,6 +70,7 @@ def test_main_fatal_exception(mocker, default_conf, caplog) -> None: Test main() function In this test we are skipping the while True loop by throwing an exception. """ + patch_exchange(mocker) mocker.patch.multiple( 'freqtrade.freqtradebot.FreqtradeBot', _init_modules=MagicMock(), @@ -97,6 +98,7 @@ def test_main_keyboard_interrupt(mocker, default_conf, caplog) -> None: Test main() function In this test we are skipping the while True loop by throwing an exception. """ + patch_exchange(mocker) mocker.patch.multiple( 'freqtrade.freqtradebot.FreqtradeBot', _init_modules=MagicMock(), @@ -124,6 +126,7 @@ def test_main_operational_exception(mocker, default_conf, caplog) -> None: Test main() function In this test we are skipping the while True loop by throwing an exception. """ + patch_exchange(mocker) mocker.patch.multiple( 'freqtrade.freqtradebot.FreqtradeBot', _init_modules=MagicMock(), @@ -151,6 +154,7 @@ def test_main_reload_conf(mocker, default_conf, caplog) -> None: Test main() function In this test we are skipping the while True loop by throwing an exception. """ + patch_exchange(mocker) mocker.patch.multiple( 'freqtrade.freqtradebot.FreqtradeBot', _init_modules=MagicMock(), @@ -178,6 +182,7 @@ def test_main_reload_conf(mocker, default_conf, caplog) -> None: def test_reconfigure(mocker, default_conf) -> None: """ Test recreate() function """ + patch_exchange(mocker) mocker.patch.multiple( 'freqtrade.freqtradebot.FreqtradeBot', _init_modules=MagicMock(), From 2b099a89e430f3e78ad5b7fa6f7cc0f2bbf9d40c Mon Sep 17 00:00:00 2001 From: xmatthias Date: Sun, 17 Jun 2018 22:42:28 +0200 Subject: [PATCH 27/64] fix styling issues --- freqtrade/freqtradebot.py | 6 +++--- freqtrade/rpc/rpc.py | 1 - freqtrade/tests/optimize/test_hyperopt.py | 2 -- freqtrade/tests/rpc/test_rpc_telegram.py | 2 +- 4 files changed, 4 insertions(+), 7 deletions(-) diff --git a/freqtrade/freqtradebot.py b/freqtrade/freqtradebot.py index e708ba57f..dd35700a4 100644 --- a/freqtrade/freqtradebot.py +++ b/freqtrade/freqtradebot.py @@ -54,7 +54,7 @@ class FreqtradeBot(object): self.fiat_converter = CryptoToFiatConverter() self.rpc: RPCManager = RPCManager(self) self.persistence = None - self.exchange = None + self.exchange = Exchange(self.config) self._init_modules() @@ -66,7 +66,6 @@ class FreqtradeBot(object): # Initialize all modules persistence.init(self.config) - self.exchange = Exchange(self.config) # Set initial application state initial_state = self.config.get('initial_state') @@ -426,7 +425,8 @@ with limit `{buy_limit:.8f} ({stake_amount:.6f} \ (buy, sell) = (False, False) if self.config.get('experimental', {}).get('use_sell_signal'): - (buy, sell) = self.analyze.get_signal(self.exchange, trade.pair, self.analyze.get_ticker_interval()) + (buy, sell) = self.analyze.get_signal(self.exchange, + trade.pair, self.analyze.get_ticker_interval()) if self.analyze.should_sell(trade, current_rate, datetime.utcnow(), buy, sell): self.execute_sell(trade, current_rate) diff --git a/freqtrade/rpc/rpc.py b/freqtrade/rpc/rpc.py index d58c265a7..ee6ecb770 100644 --- a/freqtrade/rpc/rpc.py +++ b/freqtrade/rpc/rpc.py @@ -12,7 +12,6 @@ import sqlalchemy as sql from numpy import mean, nan_to_num from pandas import DataFrame -from freqtrade.exchange import Exchange from freqtrade.misc import shorten_date from freqtrade.persistence import Trade from freqtrade.state import State diff --git a/freqtrade/tests/optimize/test_hyperopt.py b/freqtrade/tests/optimize/test_hyperopt.py index 7de6bcb7c..8ad1932af 100644 --- a/freqtrade/tests/optimize/test_hyperopt.py +++ b/freqtrade/tests/optimize/test_hyperopt.py @@ -67,7 +67,6 @@ def test_start(mocker, default_conf, caplog) -> None: mocker.patch('freqtrade.optimize.hyperopt.Hyperopt.start', start_mock) patch_exchange(mocker) - args = [ '--config', 'config.json', '--strategy', 'DefaultStrategy', @@ -184,7 +183,6 @@ def test_fmin_best_results(mocker, init_hyperopt, default_conf, caplog) -> None: mocker.patch('freqtrade.optimize.hyperopt.fmin', return_value=fmin_result) patch_exchange(mocker) - StrategyResolver({'strategy': 'DefaultStrategy'}) hyperopt = Hyperopt(conf) hyperopt.trials = create_trials(mocker) diff --git a/freqtrade/tests/rpc/test_rpc_telegram.py b/freqtrade/tests/rpc/test_rpc_telegram.py index 1b07a2143..0a7d9690f 100644 --- a/freqtrade/tests/rpc/test_rpc_telegram.py +++ b/freqtrade/tests/rpc/test_rpc_telegram.py @@ -20,7 +20,7 @@ from freqtrade.persistence import Trade from freqtrade.rpc.telegram import Telegram from freqtrade.rpc.telegram import authorized_only from freqtrade.state import State -from freqtrade.tests.conftest import get_patched_freqtradebot,patch_exchange, log_has +from freqtrade.tests.conftest import get_patched_freqtradebot, patch_exchange, log_has from freqtrade.tests.test_freqtradebot import patch_get_signal, patch_coinmarketcap From d156de39f1ee7fb2416508328e90cc60a9479fbe Mon Sep 17 00:00:00 2001 From: xmatthias Date: Sun, 17 Jun 2018 22:57:49 +0200 Subject: [PATCH 28/64] Increase test-coverage --- freqtrade/tests/exchange/test_exchange.py | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/freqtrade/tests/exchange/test_exchange.py b/freqtrade/tests/exchange/test_exchange.py index c66116ae0..bb2cdc6e0 100644 --- a/freqtrade/tests/exchange/test_exchange.py +++ b/freqtrade/tests/exchange/test_exchange.py @@ -585,6 +585,10 @@ def test_get_fee(default_conf, mocker): def test_get_amount_lots(default_conf, mocker): api_mock = MagicMock() api_mock.amount_to_lots = MagicMock(return_value=1.0) + api_mock.markets = None + marketmock = MagicMock() + api_mock.load_markets = marketmock exchange = get_patched_exchange(mocker, default_conf, api_mock) assert exchange.get_amount_lots('LTC/BTC', 1.54) == 1 + assert marketmock.call_count == 1 From c9f8dfc6c55198d93a165721bdf1601ebae6e1b3 Mon Sep 17 00:00:00 2001 From: xmatthias Date: Sun, 17 Jun 2018 23:05:44 +0200 Subject: [PATCH 29/64] increase get_fee coverage --- freqtrade/tests/exchange/test_exchange.py | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/freqtrade/tests/exchange/test_exchange.py b/freqtrade/tests/exchange/test_exchange.py index bb2cdc6e0..5541c63c6 100644 --- a/freqtrade/tests/exchange/test_exchange.py +++ b/freqtrade/tests/exchange/test_exchange.py @@ -581,6 +581,20 @@ def test_get_fee(default_conf, mocker): assert exchange.get_fee() == 0.025 + # test Exceptions + api_mock = MagicMock() + api_mock.calculate_fee = MagicMock(side_effect=ccxt.BaseError) + exchange = get_patched_exchange(mocker, default_conf, api_mock) + with pytest.raises(OperationalException): + exchange.get_fee() + + api_mock = MagicMock() + api_mock.calculate_fee = MagicMock(side_effect=ccxt.NetworkError) + exchange = get_patched_exchange(mocker, default_conf, api_mock) + with pytest.raises(TemporaryError): + exchange.get_fee() + assert api_mock.calculate_fee.call_count == API_RETRY_COUNT + 1 + def test_get_amount_lots(default_conf, mocker): api_mock = MagicMock() From 1e3d722bc2998da849f20a2fe84c9e9fe26c8371 Mon Sep 17 00:00:00 2001 From: xmatthias Date: Sun, 17 Jun 2018 23:22:28 +0200 Subject: [PATCH 30/64] add test for get_trades --- freqtrade/tests/exchange/test_exchange.py | 49 +++++++++++++++++++++++ 1 file changed, 49 insertions(+) diff --git a/freqtrade/tests/exchange/test_exchange.py b/freqtrade/tests/exchange/test_exchange.py index 5541c63c6..4d442d534 100644 --- a/freqtrade/tests/exchange/test_exchange.py +++ b/freqtrade/tests/exchange/test_exchange.py @@ -3,6 +3,7 @@ import logging from copy import deepcopy from random import randint +from datetime import datetime from unittest.mock import MagicMock, PropertyMock import ccxt @@ -569,6 +570,54 @@ def test_get_pair_detail_url(default_conf, mocker): assert 'BTC' in url +def test_get_trades_for_order(default_conf, mocker): + order_id = 'ABCD-ABCD' + since = datetime(2018, 5, 5) + default_conf["dry_run"] = False + mocker.patch('freqtrade.exchange.Exchange.exchange_has', return_value=True) + api_mock = MagicMock() + + api_mock.fetch_my_trades = MagicMock(return_value=[{'id': 'TTR67E-3PFBD-76IISV', + 'order': 'ABCD-ABCD', + 'info': {'pair': 'XLTCZBTC', + 'time': 1519860024.4388, + 'type': 'buy', + 'ordertype': 'limit', + 'price': '20.00000', + 'cost': '38.62000', + 'fee': '0.06179', + 'vol': '5', + 'id': 'ABCD-ABCD'}, + 'timestamp': 1519860024438, + 'datetime': '2018-02-28T23:20:24.438Z', + 'symbol': 'LTC/BTC', + 'type': 'limit', + 'side': 'buy', + 'price': 165.0, + 'amount': 0.2340606, + 'fee': {'cost': 0.06179, 'currency': 'BTC'} + }]) + exchange = get_patched_exchange(mocker, default_conf, api_mock) + + orders = exchange.get_trades_for_order(order_id, 'LTC/BTC', since) + assert len(orders) == 1 + assert orders[0]['price'] == 165 + + # test Exceptions + api_mock = MagicMock() + api_mock.fetch_my_trades = MagicMock(side_effect=ccxt.BaseError) + exchange = get_patched_exchange(mocker, default_conf, api_mock) + with pytest.raises(OperationalException): + exchange.get_trades_for_order(order_id, 'LTC/BTC', since) + + api_mock = MagicMock() + api_mock.fetch_my_trades = MagicMock(side_effect=ccxt.NetworkError) + exchange = get_patched_exchange(mocker, default_conf, api_mock) + with pytest.raises(TemporaryError): + exchange.get_trades_for_order(order_id, 'LTC/BTC', since) + assert api_mock.fetch_my_trades.call_count == API_RETRY_COUNT + 1 + + def test_get_fee(default_conf, mocker): api_mock = MagicMock() api_mock.calculate_fee = MagicMock(return_value={ From 520c7feeab12df5a14df3e5862b6f827d86e7c38 Mon Sep 17 00:00:00 2001 From: xmatthias Date: Sun, 17 Jun 2018 23:30:48 +0200 Subject: [PATCH 31/64] Add test for fetch_tickers --- freqtrade/tests/exchange/test_exchange.py | 43 +++++++++++++++++++++-- 1 file changed, 41 insertions(+), 2 deletions(-) diff --git a/freqtrade/tests/exchange/test_exchange.py b/freqtrade/tests/exchange/test_exchange.py index 4d442d534..0e7457532 100644 --- a/freqtrade/tests/exchange/test_exchange.py +++ b/freqtrade/tests/exchange/test_exchange.py @@ -256,8 +256,47 @@ def test_get_balances_prod(default_conf, mocker): assert api_mock.fetch_balance.call_count == 1 -# This test is somewhat redundant with -# test_exchange_bittrex.py::test_exchange_bittrex_get_ticker +def test_get_tickers(default_conf, mocker): + api_mock = MagicMock() + tick = {'ETH/BTC': { + 'symbol': 'ETH/BTC', + 'bid': 0.5, + 'ask': 1, + 'last': 42, + }, 'BCH/BTC': { + 'symbol': 'BCH/BTC', + 'bid': 0.6, + 'ask': 0.5, + 'last': 41, + } + } + api_mock.fetch_tickers = MagicMock(return_value=tick) + exchange = get_patched_exchange(mocker, default_conf, api_mock) + # retrieve original ticker + tickers = exchange.get_tickers() + + assert 'ETH/BTC' in tickers + assert 'BCH/BTC' in tickers + assert tickers['ETH/BTC']['bid'] == 0.5 + assert tickers['ETH/BTC']['ask'] == 1 + assert tickers['BCH/BTC']['bid'] == 0.6 + assert tickers['BCH/BTC']['ask'] == 0.5 + + with pytest.raises(TemporaryError): # test retrier + api_mock.fetch_tickers = MagicMock(side_effect=ccxt.NetworkError) + exchange = get_patched_exchange(mocker, default_conf, api_mock) + exchange.get_tickers() + + with pytest.raises(OperationalException): + api_mock.fetch_tickers = MagicMock(side_effect=ccxt.BaseError) + exchange = get_patched_exchange(mocker, default_conf, api_mock) + exchange.get_tickers() + + api_mock.fetch_tickers = MagicMock(return_value={}) + exchange = get_patched_exchange(mocker, default_conf, api_mock) + exchange.get_tickers() + + def test_get_ticker(default_conf, mocker): api_mock = MagicMock() tick = { From 9bc833166781a44b98818234a29f905d8395deb5 Mon Sep 17 00:00:00 2001 From: pyup-bot Date: Mon, 18 Jun 2018 14:23:05 +0200 Subject: [PATCH 32/64] Update ccxt from 1.14.202 to 1.14.211 --- requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/requirements.txt b/requirements.txt index 9d572304d..96f3169c8 100644 --- a/requirements.txt +++ b/requirements.txt @@ -1,4 +1,4 @@ -ccxt==1.14.202 +ccxt==1.14.211 SQLAlchemy==1.2.8 python-telegram-bot==10.1.0 arrow==0.12.1 From 695beecf1429665ceef354c264746e7b041e3b70 Mon Sep 17 00:00:00 2001 From: xmatthias Date: Mon, 18 Jun 2018 19:36:36 +0200 Subject: [PATCH 33/64] add test for get_markets --- freqtrade/tests/exchange/test_exchange.py | 26 +++++++++++++++++++++++ 1 file changed, 26 insertions(+) diff --git a/freqtrade/tests/exchange/test_exchange.py b/freqtrade/tests/exchange/test_exchange.py index 0e7457532..16992b53f 100644 --- a/freqtrade/tests/exchange/test_exchange.py +++ b/freqtrade/tests/exchange/test_exchange.py @@ -657,6 +657,32 @@ def test_get_trades_for_order(default_conf, mocker): assert api_mock.fetch_my_trades.call_count == API_RETRY_COUNT + 1 +def test_get_markets(default_conf, mocker, markets): + api_mock = MagicMock() + api_mock.fetch_markets = markets + exchange = get_patched_exchange(mocker, default_conf, api_mock) + ret = exchange.get_markets() + assert isinstance(ret, list) + assert len(ret) == 3 + + assert ret[0]["id"] == "ethbtc" + assert ret[0]["symbol"] == "ETH/BTC" + + # test Exceptions + api_mock = MagicMock() + api_mock.fetch_markets = MagicMock(side_effect=ccxt.BaseError) + exchange = get_patched_exchange(mocker, default_conf, api_mock) + with pytest.raises(OperationalException): + exchange.get_markets() + + api_mock = MagicMock() + api_mock.fetch_markets = MagicMock(side_effect=ccxt.NetworkError) + exchange = get_patched_exchange(mocker, default_conf, api_mock) + with pytest.raises(TemporaryError): + exchange.get_markets() + assert api_mock.fetch_markets.call_count == API_RETRY_COUNT + 1 + + def test_get_fee(default_conf, mocker): api_mock = MagicMock() api_mock.calculate_fee = MagicMock(return_value={ From ae4c4e77bffbdce1f3612ee84c27ff7ef875115d Mon Sep 17 00:00:00 2001 From: xmatthias Date: Mon, 18 Jun 2018 19:46:42 +0200 Subject: [PATCH 34/64] standardize exception tests - add one more --- freqtrade/tests/exchange/test_exchange.py | 41 +++++++++++++---------- 1 file changed, 23 insertions(+), 18 deletions(-) diff --git a/freqtrade/tests/exchange/test_exchange.py b/freqtrade/tests/exchange/test_exchange.py index 16992b53f..d748bdad2 100644 --- a/freqtrade/tests/exchange/test_exchange.py +++ b/freqtrade/tests/exchange/test_exchange.py @@ -292,6 +292,11 @@ def test_get_tickers(default_conf, mocker): exchange = get_patched_exchange(mocker, default_conf, api_mock) exchange.get_tickers() + with pytest.raises(OperationalException): + api_mock.fetch_tickers = MagicMock(side_effect=ccxt.NotSupported) + exchange = get_patched_exchange(mocker, default_conf, api_mock) + exchange.get_tickers() + api_mock.fetch_tickers = MagicMock(return_value={}) exchange = get_patched_exchange(mocker, default_conf, api_mock) exchange.get_tickers() @@ -643,16 +648,16 @@ def test_get_trades_for_order(default_conf, mocker): assert orders[0]['price'] == 165 # test Exceptions - api_mock = MagicMock() - api_mock.fetch_my_trades = MagicMock(side_effect=ccxt.BaseError) - exchange = get_patched_exchange(mocker, default_conf, api_mock) with pytest.raises(OperationalException): + api_mock = MagicMock() + api_mock.fetch_my_trades = MagicMock(side_effect=ccxt.BaseError) + exchange = get_patched_exchange(mocker, default_conf, api_mock) exchange.get_trades_for_order(order_id, 'LTC/BTC', since) - api_mock = MagicMock() - api_mock.fetch_my_trades = MagicMock(side_effect=ccxt.NetworkError) - exchange = get_patched_exchange(mocker, default_conf, api_mock) with pytest.raises(TemporaryError): + api_mock = MagicMock() + api_mock.fetch_my_trades = MagicMock(side_effect=ccxt.NetworkError) + exchange = get_patched_exchange(mocker, default_conf, api_mock) exchange.get_trades_for_order(order_id, 'LTC/BTC', since) assert api_mock.fetch_my_trades.call_count == API_RETRY_COUNT + 1 @@ -669,16 +674,16 @@ def test_get_markets(default_conf, mocker, markets): assert ret[0]["symbol"] == "ETH/BTC" # test Exceptions - api_mock = MagicMock() - api_mock.fetch_markets = MagicMock(side_effect=ccxt.BaseError) - exchange = get_patched_exchange(mocker, default_conf, api_mock) with pytest.raises(OperationalException): + api_mock = MagicMock() + api_mock.fetch_markets = MagicMock(side_effect=ccxt.BaseError) + exchange = get_patched_exchange(mocker, default_conf, api_mock) exchange.get_markets() - api_mock = MagicMock() - api_mock.fetch_markets = MagicMock(side_effect=ccxt.NetworkError) - exchange = get_patched_exchange(mocker, default_conf, api_mock) with pytest.raises(TemporaryError): + api_mock = MagicMock() + api_mock.fetch_markets = MagicMock(side_effect=ccxt.NetworkError) + exchange = get_patched_exchange(mocker, default_conf, api_mock) exchange.get_markets() assert api_mock.fetch_markets.call_count == API_RETRY_COUNT + 1 @@ -696,16 +701,16 @@ def test_get_fee(default_conf, mocker): assert exchange.get_fee() == 0.025 # test Exceptions - api_mock = MagicMock() - api_mock.calculate_fee = MagicMock(side_effect=ccxt.BaseError) - exchange = get_patched_exchange(mocker, default_conf, api_mock) with pytest.raises(OperationalException): + api_mock = MagicMock() + api_mock.calculate_fee = MagicMock(side_effect=ccxt.BaseError) + exchange = get_patched_exchange(mocker, default_conf, api_mock) exchange.get_fee() - api_mock = MagicMock() - api_mock.calculate_fee = MagicMock(side_effect=ccxt.NetworkError) - exchange = get_patched_exchange(mocker, default_conf, api_mock) with pytest.raises(TemporaryError): + api_mock = MagicMock() + api_mock.calculate_fee = MagicMock(side_effect=ccxt.NetworkError) + exchange = get_patched_exchange(mocker, default_conf, api_mock) exchange.get_fee() assert api_mock.calculate_fee.call_count == API_RETRY_COUNT + 1 From 162f94872990e56f3f5caa8355d36fa1e40e95d9 Mon Sep 17 00:00:00 2001 From: xmatthias Date: Mon, 18 Jun 2018 19:56:23 +0200 Subject: [PATCH 35/64] add test for non-configured exchange --- freqtrade/tests/exchange/test_exchange.py | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/freqtrade/tests/exchange/test_exchange.py b/freqtrade/tests/exchange/test_exchange.py index d748bdad2..44bf539d0 100644 --- a/freqtrade/tests/exchange/test_exchange.py +++ b/freqtrade/tests/exchange/test_exchange.py @@ -587,11 +587,10 @@ def test_get_id(default_conf, mocker): assert exchange.get_id() == 'binance' -def test_get_pair_detail_url(default_conf, mocker): +def test_get_pair_detail_url(default_conf, mocker, caplog): mocker.patch('freqtrade.exchange.Exchange.validate_pairs', side_effect=lambda s: True) default_conf['exchange']['name'] = 'binance' - # exchange = get_patched_exchange(mocker, default_conf, api_mock) exchange = Exchange(default_conf) url = exchange.get_pair_detail_url('TKN/ETH') @@ -613,6 +612,12 @@ def test_get_pair_detail_url(default_conf, mocker): assert 'LOOONG' in url assert 'BTC' in url + default_conf['exchange']['name'] = 'poloniex' + exchange = Exchange(default_conf) + url = exchange.get_pair_detail_url('LOOONG/BTC') + assert '' == url + assert log_has('Could not get exchange url for Poloniex', caplog.record_tuples) + def test_get_trades_for_order(default_conf, mocker): order_id = 'ABCD-ABCD' From c31519fdb29ae21133343786d7f4769c67b2a454 Mon Sep 17 00:00:00 2001 From: xmatthias Date: Mon, 18 Jun 2018 22:07:15 +0200 Subject: [PATCH 36/64] lowercase _api object --- freqtrade/exchange/__init__.py | 55 ++++++++++++++++------------------ 1 file changed, 26 insertions(+), 29 deletions(-) diff --git a/freqtrade/exchange/__init__.py b/freqtrade/exchange/__init__.py index 0d303fa4a..9f07d66d2 100644 --- a/freqtrade/exchange/__init__.py +++ b/freqtrade/exchange/__init__.py @@ -43,7 +43,7 @@ def retrier(f): class Exchange(object): # Current selected exchange - _API: ccxt.Exchange = None + _api: ccxt.Exchange = None _CONF: Dict = {} _CACHED_TICKER: Dict[str, Any] = {} @@ -55,18 +55,15 @@ class Exchange(object): Initializes this module with the given config, it does basic validation whether the specified exchange and pairs are valid. - :param config: config to use :return: None """ - self._API - self._CONF.update(config) if config['dry_run']: logger.info('Instance is running with dry_run enabled') exchange_config = config['exchange'] - self._API = self._init_ccxt(exchange_config) + self._api = self._init_ccxt(exchange_config) logger.info('Using Exchange "%s"', self.get_name()) @@ -99,10 +96,10 @@ class Exchange(object): return api def get_name(self) -> str: - return self._API.name + return self._api.name def get_id(self) -> str: - return self._API.id + return self._api.id def validate_pairs(self, pairs: List[str]) -> None: """ @@ -113,7 +110,7 @@ class Exchange(object): """ try: - markets = self._API.load_markets() + markets = self._api.load_markets() except ccxt.BaseError as e: logger.warning('Unable to validate pairs (assuming they are correct). Reason: %s', e) return @@ -136,7 +133,7 @@ class Exchange(object): :param endpoint: Name of endpoint (e.g. 'fetchOHLCV', 'fetchTickers') :return: bool """ - return endpoint in self._API.has and self._API.has[endpoint] + return endpoint in self._api.has and self._api.has[endpoint] def buy(self, pair: str, rate: float, amount: float) -> Dict: if self._CONF['dry_run']: @@ -155,7 +152,7 @@ class Exchange(object): return {'id': order_id} try: - return self._API.create_limit_buy_order(pair, amount, rate) + return self._api.create_limit_buy_order(pair, amount, rate) except ccxt.InsufficientFunds as e: raise DependencyException( f'Insufficient funds to create limit buy order on market {pair}.' @@ -188,7 +185,7 @@ class Exchange(object): return {'id': order_id} try: - return self._API.create_limit_sell_order(pair, amount, rate) + return self._api.create_limit_sell_order(pair, amount, rate) except ccxt.InsufficientFunds as e: raise DependencyException( f'Insufficient funds to create limit sell order on market {pair}.' @@ -224,7 +221,7 @@ class Exchange(object): return {} try: - balances = self._API.fetch_balance() + balances = self._api.fetch_balance() # Remove additional info from ccxt results balances.pop("info", None) balances.pop("free", None) @@ -241,10 +238,10 @@ class Exchange(object): @retrier def get_tickers(self) -> Dict: try: - return self._API.fetch_tickers() + return self._api.fetch_tickers() except ccxt.NotSupported as e: raise OperationalException( - f'Exchange {self._API.name} does not support fetching tickers in batch.' + f'Exchange {self._api.name} does not support fetching tickers in batch.' f'Message: {e}') except (ccxt.NetworkError, ccxt.ExchangeError) as e: raise TemporaryError( @@ -256,7 +253,7 @@ class Exchange(object): def get_ticker(self, pair: str, refresh: Optional[bool] = True) -> dict: if refresh or pair not in self._CACHED_TICKER.keys(): try: - data = self._API.fetch_ticker(pair) + data = self._api.fetch_ticker(pair) try: self._CACHED_TICKER[pair] = { 'bid': float(data['bid']), @@ -290,7 +287,7 @@ class Exchange(object): data: List[Dict[Any, Any]] = [] while not since_ms or since_ms < till_time_ms: - data_part = self._API.fetch_ohlcv(pair, timeframe=tick_interval, since=since_ms) + data_part = self._api.fetch_ohlcv(pair, timeframe=tick_interval, since=since_ms) # Because some exchange sort Tickers ASC and other DESC. # Ex: Bittrex returns a list of tickers ASC (oldest first, newest last) @@ -311,7 +308,7 @@ class Exchange(object): return data except ccxt.NotSupported as e: raise OperationalException( - f'Exchange {self._API.name} does not support fetching historical candlestick data.' + f'Exchange {self._api.name} does not support fetching historical candlestick data.' f'Message: {e}') except (ccxt.NetworkError, ccxt.ExchangeError) as e: raise TemporaryError( @@ -325,7 +322,7 @@ class Exchange(object): return try: - return self._API.cancel_order(order_id, pair) + return self._api.cancel_order(order_id, pair) except ccxt.InvalidOrder as e: raise DependencyException( f'Could not cancel order. Message: {e}') @@ -344,7 +341,7 @@ class Exchange(object): }) return order try: - return self._API.fetch_order(order_id, pair) + return self._api.fetch_order(order_id, pair) except ccxt.InvalidOrder as e: raise DependencyException( f'Could not get order. Message: {e}') @@ -361,7 +358,7 @@ class Exchange(object): if not self.exchange_has('fetchMyTrades'): return [] try: - my_trades = self._API.fetch_my_trades(pair, since.timestamp()) + my_trades = self._api.fetch_my_trades(pair, since.timestamp()) matched_trades = [trade for trade in my_trades if trade['order'] == order_id] return matched_trades @@ -374,10 +371,10 @@ class Exchange(object): def get_pair_detail_url(self, pair: str) -> str: try: - url_base = self._API.urls.get('www') + url_base = self._api.urls.get('www') base, quote = pair.split('/') - return url_base + _EXCHANGE_URLS[self._API.id].format(base=base, quote=quote) + return url_base + _EXCHANGE_URLS[self._api.id].format(base=base, quote=quote) except KeyError: logger.warning('Could not get exchange url for %s', self.get_name()) return "" @@ -385,7 +382,7 @@ class Exchange(object): @retrier def get_markets(self) -> List[dict]: try: - return self._API.fetch_markets() + return self._api.fetch_markets() except (ccxt.NetworkError, ccxt.ExchangeError) as e: raise TemporaryError( f'Could not load markets due to {e.__class__.__name__}. Message: {e}') @@ -397,10 +394,10 @@ class Exchange(object): price=1, taker_or_maker='maker') -> float: try: # validate that markets are loaded before trying to get fee - if self._API.markets is None or len(self._API.markets) == 0: - self._API.load_markets() + if self._api.markets is None or len(self._api.markets) == 0: + self._api.load_markets() - return self._API.calculate_fee(symbol=symbol, type=type, side=side, amount=amount, + return self._api.calculate_fee(symbol=symbol, type=type, side=side, amount=amount, price=price, takerOrMaker=taker_or_maker)['rate'] except (ccxt.NetworkError, ccxt.ExchangeError) as e: raise TemporaryError( @@ -413,6 +410,6 @@ class Exchange(object): get buyable amount rounding, .. """ # validate that markets are loaded before trying to get fee - if not self._API.markets: - self._API.load_markets() - return self._API.amount_to_lots(pair, amount) + if not self._api.markets: + self._api.load_markets() + return self._api.amount_to_lots(pair, amount) From ef5313449908bcdd2f44d33eee49763b93b228ea Mon Sep 17 00:00:00 2001 From: xmatthias Date: Mon, 18 Jun 2018 22:09:46 +0200 Subject: [PATCH 37/64] lowercase variables --- freqtrade/exchange/__init__.py | 36 +++++++++++------------ freqtrade/tests/exchange/test_exchange.py | 8 ++--- 2 files changed, 22 insertions(+), 22 deletions(-) diff --git a/freqtrade/exchange/__init__.py b/freqtrade/exchange/__init__.py index 9f07d66d2..971a09bd8 100644 --- a/freqtrade/exchange/__init__.py +++ b/freqtrade/exchange/__init__.py @@ -44,11 +44,11 @@ class Exchange(object): # Current selected exchange _api: ccxt.Exchange = None - _CONF: Dict = {} - _CACHED_TICKER: Dict[str, Any] = {} + _conf: Dict = {} + _cached_ticker: Dict[str, Any] = {} # Holds all open sell orders for dry_run - _DRY_RUN_OPEN_ORDERS: Dict[str, Any] = {} + _dry_run_open_orders: Dict[str, Any] = {} def __init__(self, config: dict) -> None: """ @@ -57,7 +57,7 @@ class Exchange(object): exchange and pairs are valid. :return: None """ - self._CONF.update(config) + self._conf.update(config) if config['dry_run']: logger.info('Instance is running with dry_run enabled') @@ -115,7 +115,7 @@ class Exchange(object): logger.warning('Unable to validate pairs (assuming they are correct). Reason: %s', e) return - stake_cur = self._CONF['stake_currency'] + stake_cur = self._conf['stake_currency'] for pair in pairs: # Note: ccxt has BaseCurrency/QuoteCurrency format for pairs # TODO: add a support for having coins in BTC/USDT format @@ -136,9 +136,9 @@ class Exchange(object): return endpoint in self._api.has and self._api.has[endpoint] def buy(self, pair: str, rate: float, amount: float) -> Dict: - if self._CONF['dry_run']: + if self._conf['dry_run']: order_id = f'dry_run_buy_{randint(0, 10**6)}' - self._DRY_RUN_OPEN_ORDERS[order_id] = { + self._dry_run_open_orders[order_id] = { 'pair': pair, 'price': rate, 'amount': amount, @@ -170,9 +170,9 @@ class Exchange(object): raise OperationalException(e) def sell(self, pair: str, rate: float, amount: float) -> Dict: - if self._CONF['dry_run']: + if self._conf['dry_run']: order_id = f'dry_run_sell_{randint(0, 10**6)}' - self._DRY_RUN_OPEN_ORDERS[order_id] = { + self._dry_run_open_orders[order_id] = { 'pair': pair, 'price': rate, 'amount': amount, @@ -204,7 +204,7 @@ class Exchange(object): @retrier def get_balance(self, currency: str) -> float: - if self._CONF['dry_run']: + if self._conf['dry_run']: return 999.9 # ccxt exception is already handled by get_balances @@ -217,7 +217,7 @@ class Exchange(object): @retrier def get_balances(self) -> dict: - if self._CONF['dry_run']: + if self._conf['dry_run']: return {} try: @@ -251,11 +251,11 @@ class Exchange(object): @retrier def get_ticker(self, pair: str, refresh: Optional[bool] = True) -> dict: - if refresh or pair not in self._CACHED_TICKER.keys(): + if refresh or pair not in self._cached_ticker.keys(): try: data = self._api.fetch_ticker(pair) try: - self._CACHED_TICKER[pair] = { + self._cached_ticker[pair] = { 'bid': float(data['bid']), 'ask': float(data['ask']), } @@ -269,7 +269,7 @@ class Exchange(object): raise OperationalException(e) else: logger.info("returning cached ticker-data for %s", pair) - return self._CACHED_TICKER[pair] + return self._cached_ticker[pair] @retrier def get_ticker_history(self, pair: str, tick_interval: str, @@ -318,7 +318,7 @@ class Exchange(object): @retrier def cancel_order(self, order_id: str, pair: str) -> None: - if self._CONF['dry_run']: + if self._conf['dry_run']: return try: @@ -334,8 +334,8 @@ class Exchange(object): @retrier def get_order(self, order_id: str, pair: str) -> Dict: - if self._CONF['dry_run']: - order = self._DRY_RUN_OPEN_ORDERS[order_id] + if self._conf['dry_run']: + order = self._dry_run_open_orders[order_id] order.update({ 'id': order_id }) @@ -353,7 +353,7 @@ class Exchange(object): @retrier def get_trades_for_order(self, order_id: str, pair: str, since: datetime) -> List: - if self._CONF['dry_run']: + if self._conf['dry_run']: return [] if not self.exchange_has('fetchMyTrades'): return [] diff --git a/freqtrade/tests/exchange/test_exchange.py b/freqtrade/tests/exchange/test_exchange.py index 44bf539d0..43cd0076c 100644 --- a/freqtrade/tests/exchange/test_exchange.py +++ b/freqtrade/tests/exchange/test_exchange.py @@ -336,9 +336,9 @@ def test_get_ticker(default_conf, mocker): assert ticker['bid'] == 0.5 assert ticker['ask'] == 1 - assert 'ETH/BTC' in exchange._CACHED_TICKER - assert exchange._CACHED_TICKER['ETH/BTC']['bid'] == 0.5 - assert exchange._CACHED_TICKER['ETH/BTC']['ask'] == 1 + assert 'ETH/BTC' in exchange._cached_ticker + assert exchange._cached_ticker['ETH/BTC']['bid'] == 0.5 + assert exchange._cached_ticker['ETH/BTC']['ask'] == 1 # Test caching api_mock.fetch_ticker = MagicMock() @@ -541,7 +541,7 @@ def test_get_order(default_conf, mocker): order = MagicMock() order.myid = 123 exchange = get_patched_exchange(mocker, default_conf) - exchange._DRY_RUN_OPEN_ORDERS['X'] = order + exchange._dry_run_open_orders['X'] = order print(exchange.get_order('X', 'TKN/BTC')) assert exchange.get_order('X', 'TKN/BTC').myid == 123 From 896afe711836bfb0285add761d16f20e5aadc511 Mon Sep 17 00:00:00 2001 From: xmatthias Date: Mon, 18 Jun 2018 22:20:50 +0200 Subject: [PATCH 38/64] convert get_name and get_id to properties --- freqtrade/exchange/__init__.py | 14 +++++++++----- freqtrade/freqtradebot.py | 4 ++-- freqtrade/tests/exchange/test_exchange.py | 10 +++++----- 3 files changed, 16 insertions(+), 12 deletions(-) diff --git a/freqtrade/exchange/__init__.py b/freqtrade/exchange/__init__.py index 971a09bd8..481469701 100644 --- a/freqtrade/exchange/__init__.py +++ b/freqtrade/exchange/__init__.py @@ -65,7 +65,7 @@ class Exchange(object): exchange_config = config['exchange'] self._api = self._init_ccxt(exchange_config) - logger.info('Using Exchange "%s"', self.get_name()) + logger.info('Using Exchange "%s"', self.name) # Check if all pairs are available self.validate_pairs(config['exchange']['pair_whitelist']) @@ -95,10 +95,14 @@ class Exchange(object): return api - def get_name(self) -> str: + @property + def name(self) -> str: + """exchange Name (from ccxt)""" return self._api.name - def get_id(self) -> str: + @property + def id(self) -> str: + """exchange ccxt id""" return self._api.id def validate_pairs(self, pairs: List[str]) -> None: @@ -124,7 +128,7 @@ class Exchange(object): f'Pair {pair} not compatible with stake_currency: {stake_cur}') if pair not in markets: raise OperationalException( - f'Pair {pair} is not available at {self.get_name()}') + f'Pair {pair} is not available at {self.name}') def exchange_has(self, endpoint: str) -> bool: """ @@ -376,7 +380,7 @@ class Exchange(object): return url_base + _EXCHANGE_URLS[self._api.id].format(base=base, quote=quote) except KeyError: - logger.warning('Could not get exchange url for %s', self.get_name()) + logger.warning('Could not get exchange url for %s', self.name) return "" @retrier diff --git a/freqtrade/freqtradebot.py b/freqtrade/freqtradebot.py index dd35700a4..02375ca80 100644 --- a/freqtrade/freqtradebot.py +++ b/freqtrade/freqtradebot.py @@ -254,7 +254,7 @@ class FreqtradeBot(object): interval = self.analyze.get_ticker_interval() stake_currency = self.config['stake_currency'] fiat_currency = self.config['fiat_display_currency'] - exc_name = self.exchange.get_name() + exc_name = self.exchange.name logger.info( 'Checking buy signals to create a new trade with stake_amount: %f ...', @@ -314,7 +314,7 @@ with limit `{buy_limit:.8f} ({stake_amount:.6f} \ open_rate=buy_limit, open_rate_requested=buy_limit, open_date=datetime.utcnow(), - exchange=self.exchange.get_id(), + exchange=self.exchange.id, open_order_id=order_id ) Trade.session.add(trade) diff --git a/freqtrade/tests/exchange/test_exchange.py b/freqtrade/tests/exchange/test_exchange.py index 43cd0076c..3c68e8493 100644 --- a/freqtrade/tests/exchange/test_exchange.py +++ b/freqtrade/tests/exchange/test_exchange.py @@ -66,7 +66,7 @@ def test_validate_pairs_not_compatible(default_conf, mocker): def test_validate_pairs_exception(default_conf, mocker, caplog): caplog.set_level(logging.INFO) api_mock = MagicMock() - mocker.patch('freqtrade.exchange.Exchange.get_name', MagicMock(return_value='Binance')) + mocker.patch('freqtrade.exchange.Exchange.name', PropertyMock(return_value='Binance')) api_mock.load_markets = MagicMock(return_value={}) mocker.patch('freqtrade.exchange.Exchange._init_ccxt', api_mock) @@ -570,21 +570,21 @@ def test_get_order(default_conf, mocker): assert api_mock.fetch_order.call_count == 1 -def test_get_name(default_conf, mocker): +def test_name(default_conf, mocker): mocker.patch('freqtrade.exchange.Exchange.validate_pairs', side_effect=lambda s: True) default_conf['exchange']['name'] = 'binance' exchange = Exchange(default_conf) - assert exchange.get_name() == 'Binance' + assert exchange.name == 'Binance' -def test_get_id(default_conf, mocker): +def test_id(default_conf, mocker): mocker.patch('freqtrade.exchange.Exchange.validate_pairs', side_effect=lambda s: True) default_conf['exchange']['name'] = 'binance' exchange = Exchange(default_conf) - assert exchange.get_id() == 'binance' + assert exchange.id == 'binance' def test_get_pair_detail_url(default_conf, mocker, caplog): From 2b0ef5459546a99a7e753cf55427490493fc8c99 Mon Sep 17 00:00:00 2001 From: xmatthias Date: Mon, 18 Jun 2018 22:28:51 +0200 Subject: [PATCH 39/64] update download_script for exchange objectify --- scripts/download_backtest_data.py | 20 +++++++++++++------- 1 file changed, 13 insertions(+), 7 deletions(-) diff --git a/scripts/download_backtest_data.py b/scripts/download_backtest_data.py index 9aedbecb9..2f76c1232 100755 --- a/scripts/download_backtest_data.py +++ b/scripts/download_backtest_data.py @@ -6,7 +6,8 @@ import sys import os import arrow -from freqtrade import (exchange, arguments, misc) +from freqtrade import (arguments, misc) +from freqtrade.exchange import Exchange DEFAULT_DL_PATH = 'user_data/data' @@ -39,16 +40,21 @@ if args.days: print(f'About to download pairs: {PAIRS} to {dl_path}') + # Init exchange -exchange._API = exchange.init_ccxt({'key': '', - 'secret': '', - 'name': args.exchange}) +exchange = Exchange({'key': '', + 'secret': '', + 'stake_currency': '', + 'dry_run': True, + 'exchange': { + 'name': args.exchange, + 'pair_whitelist': [] + } + }) pairs_not_available = [] -# Make sure API markets is initialized -exchange._API.load_markets() for pair in PAIRS: - if pair not in exchange._API.markets: + if pair not in exchange._api.markets: pairs_not_available.append(pair) print(f"skipping pair {pair}") continue From 488f1717a19649e06c298d5dae5ca8d16ca740c0 Mon Sep 17 00:00:00 2001 From: xmatthias Date: Mon, 18 Jun 2018 22:32:29 +0200 Subject: [PATCH 40/64] update plot_dataframe script to objectify exchange --- scripts/plot_dataframe.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/scripts/plot_dataframe.py b/scripts/plot_dataframe.py index ce1a4b819..7f9641222 100755 --- a/scripts/plot_dataframe.py +++ b/scripts/plot_dataframe.py @@ -35,10 +35,10 @@ from plotly import tools from plotly.offline import plot import freqtrade.optimize as optimize -from freqtrade import exchange from freqtrade import persistence from freqtrade.analyze import Analyze from freqtrade.arguments import Arguments +from freqtrade.exchange import Exchange from freqtrade.optimize.backtesting import setup_configuration from freqtrade.persistence import Trade @@ -73,7 +73,7 @@ def plot_analyzed_dataframe(args: Namespace) -> None: # Load the strategy try: analyze = Analyze(_CONF) - exchange.init(_CONF) + exchange = Exchange(_CONF) except AttributeError: logger.critical( 'Impossible to load the strategy. Please check the file "user_data/strategies/%s.py"', From f7b46d5404ece4e430377a005cf71930b3d47848 Mon Sep 17 00:00:00 2001 From: xmatthias Date: Mon, 18 Jun 2018 22:34:28 +0200 Subject: [PATCH 41/64] update docstring --- freqtrade/exchange/__init__.py | 2 -- 1 file changed, 2 deletions(-) diff --git a/freqtrade/exchange/__init__.py b/freqtrade/exchange/__init__.py index 481469701..acfefdad4 100644 --- a/freqtrade/exchange/__init__.py +++ b/freqtrade/exchange/__init__.py @@ -74,8 +74,6 @@ class Exchange(object): """ Initialize ccxt with given config and return valid ccxt instance. - :param config: config to use - :return: ccxt """ # Find matching class for the given exchange name name = exchange_config['name'] From a7be15d72f9a3d534042040fdf19e1a3f4974c57 Mon Sep 17 00:00:00 2001 From: xmatthias Date: Mon, 18 Jun 2018 22:42:14 +0200 Subject: [PATCH 42/64] Update Documentation to include backtesting with docker --- docs/installation.md | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) diff --git a/docs/installation.md b/docs/installation.md index b6688885b..87d3dc5cf 100644 --- a/docs/installation.md +++ b/docs/installation.md @@ -184,6 +184,26 @@ docker start freqtrade You do not need to rebuild the image for configuration changes, it will suffice to edit `config.json` and restart the container. +### 7. Backtest with docker + +The following assumes that the above steps (1-4) have been completed successfully. +Also, backtest-data should be available at `~/.freqtrade/user_data/`. + + +``` bash +docker run -d \ + --name freqtrade \ + -v /etc/localtime:/etc/localtime:ro \ + -v ~/.freqtrade/config.json:/freqtrade/config.json \ + -v ~/.freqtrade/tradesv3.sqlite:/freqtrade/tradesv3.sqlite \ + -v ~/.freqtrade/user_data/:/freqtrade/user_data/ \ + freqtrade --strategy AwsomelyProfitableStrategy backtesting +``` + +Head over to the [Backtesting Documentation](https://github.com/freqtrade/freqtrade/blob/develop/docs/backtesting.md) for more details. + +*Note*: Additional parameters can be appended after the image name (`freqtrade` in the above example). + ------ ## Custom Installation From e66b861c9e93cec7079dbca6ab09ee7d1febeedb Mon Sep 17 00:00:00 2001 From: pyup-bot Date: Tue, 19 Jun 2018 14:23:05 +0200 Subject: [PATCH 43/64] Update ccxt from 1.14.211 to 1.14.224 --- requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/requirements.txt b/requirements.txt index 96f3169c8..104634890 100644 --- a/requirements.txt +++ b/requirements.txt @@ -1,4 +1,4 @@ -ccxt==1.14.211 +ccxt==1.14.224 SQLAlchemy==1.2.8 python-telegram-bot==10.1.0 arrow==0.12.1 From a493a2ceef49a5420e6c883a50f6326f45f2c70d Mon Sep 17 00:00:00 2001 From: pyup-bot Date: Wed, 20 Jun 2018 14:23:06 +0200 Subject: [PATCH 44/64] Update ccxt from 1.14.224 to 1.14.230 --- requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/requirements.txt b/requirements.txt index 104634890..ed01a35d4 100644 --- a/requirements.txt +++ b/requirements.txt @@ -1,4 +1,4 @@ -ccxt==1.14.224 +ccxt==1.14.230 SQLAlchemy==1.2.8 python-telegram-bot==10.1.0 arrow==0.12.1 From 36cfea3d0f413ab51d9042bc461c44617fdc93d2 Mon Sep 17 00:00:00 2001 From: pyup-bot Date: Wed, 20 Jun 2018 14:23:08 +0200 Subject: [PATCH 45/64] Update pytest from 3.6.1 to 3.6.2 --- requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/requirements.txt b/requirements.txt index ed01a35d4..a86cdc020 100644 --- a/requirements.txt +++ b/requirements.txt @@ -12,7 +12,7 @@ scipy==1.1.0 jsonschema==2.6.0 numpy==1.14.5 TA-Lib==0.4.17 -pytest==3.6.1 +pytest==3.6.2 pytest-mock==1.10.0 pytest-cov==2.5.1 hyperopt==0.1 From c7976f51e20ff2bf2e24cb26e455d843eaa44c16 Mon Sep 17 00:00:00 2001 From: pyup-bot Date: Thu, 21 Jun 2018 14:24:06 +0200 Subject: [PATCH 46/64] Update ccxt from 1.14.230 to 1.14.242 --- requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/requirements.txt b/requirements.txt index a86cdc020..41e246d50 100644 --- a/requirements.txt +++ b/requirements.txt @@ -1,4 +1,4 @@ -ccxt==1.14.230 +ccxt==1.14.242 SQLAlchemy==1.2.8 python-telegram-bot==10.1.0 arrow==0.12.1 From 7f927b4d7a636feb36a7e28a348b36a5d1ee6e32 Mon Sep 17 00:00:00 2001 From: xmatthias Date: Thu, 21 Jun 2018 20:47:53 +0200 Subject: [PATCH 47/64] Squashed commit of the following: commit 435f299bcf52c7fe0be1b44c5ea7a9983f31b811 Author: Gert Wohlgemuth Date: Wed Jun 20 01:57:28 2018 -0700 improve readability of outdated history code --- freqtrade/analyze.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/freqtrade/analyze.py b/freqtrade/analyze.py index f18ae291c..17a8f27b9 100644 --- a/freqtrade/analyze.py +++ b/freqtrade/analyze.py @@ -148,7 +148,7 @@ class Analyze(object): # Check if dataframe is out of date signal_date = arrow.get(latest['date']) interval_minutes = constants.TICKER_INTERVAL_MINUTES[interval] - if signal_date < arrow.utcnow() - timedelta(minutes=(interval_minutes + 5)): + if signal_date < (arrow.utcnow() - timedelta(minutes=(interval_minutes + 5))): logger.warning( 'Outdated history for pair %s. Last tick is %s minutes old', pair, From 98cd8970f996ca64f96ca32193fe746ec1bddd40 Mon Sep 17 00:00:00 2001 From: pyup-bot Date: Fri, 22 Jun 2018 14:24:06 +0200 Subject: [PATCH 48/64] Update ccxt from 1.14.242 to 1.14.253 --- requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/requirements.txt b/requirements.txt index 41e246d50..a31976c8c 100644 --- a/requirements.txt +++ b/requirements.txt @@ -1,4 +1,4 @@ -ccxt==1.14.242 +ccxt==1.14.253 SQLAlchemy==1.2.8 python-telegram-bot==10.1.0 arrow==0.12.1 From c73b9f5c77ab9f55b9d51326f9c372dc02af02b1 Mon Sep 17 00:00:00 2001 From: Janne Sinivirta Date: Fri, 22 Jun 2018 21:04:07 +0300 Subject: [PATCH 49/64] avoid calling exchange.get_fee inside loop --- freqtrade/optimize/backtesting.py | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/freqtrade/optimize/backtesting.py b/freqtrade/optimize/backtesting.py index 9a19d1412..ffb808a24 100644 --- a/freqtrade/optimize/backtesting.py +++ b/freqtrade/optimize/backtesting.py @@ -62,6 +62,7 @@ class Backtesting(object): self.config['exchange']['uid'] = '' self.config['dry_run'] = True self.exchange = Exchange(self.config) + self.fee = self.exchange.get_fee() @staticmethod def get_timeframe(data: Dict[str, DataFrame]) -> Tuple[arrow.Arrow, arrow.Arrow]: @@ -130,14 +131,13 @@ class Backtesting(object): stake_amount = args['stake_amount'] max_open_trades = args.get('max_open_trades', 0) - fee = self.exchange.get_fee() trade = Trade( open_rate=buy_row.close, open_date=buy_row.date, stake_amount=stake_amount, amount=stake_amount / buy_row.open, - fee_open=fee, - fee_close=fee + fee_open=self.fee, + fee_close=self.fee ) # calculate win/lose forwards from buy point From 8a44dff595da271b35e8fbc4696f15bd8271b80e Mon Sep 17 00:00:00 2001 From: xmatthias Date: Fri, 22 Jun 2018 20:10:05 +0200 Subject: [PATCH 50/64] don't sell if buy is still active --- config.json.example | 3 +- config_full.json.example | 3 +- docs/configuration.md | 1 + freqtrade/analyze.py | 4 ++ freqtrade/constants.py | 3 +- freqtrade/freqtradebot.py | 3 +- freqtrade/tests/test_freqtradebot.py | 77 ++++++++++++++++++++++++++++ 7 files changed, 90 insertions(+), 4 deletions(-) diff --git a/config.json.example b/config.json.example index d3dbeb52e..e5bdc95b1 100644 --- a/config.json.example +++ b/config.json.example @@ -31,7 +31,8 @@ }, "experimental": { "use_sell_signal": false, - "sell_profit_only": false + "sell_profit_only": false, + "ignore_roi_if_buy_signal": false }, "telegram": { "enabled": true, diff --git a/config_full.json.example b/config_full.json.example index c17d22a15..231384acc 100644 --- a/config_full.json.example +++ b/config_full.json.example @@ -38,7 +38,8 @@ }, "experimental": { "use_sell_signal": false, - "sell_profit_only": false + "sell_profit_only": false, + "ignore_roi_if_buy_signal": false }, "telegram": { "enabled": true, diff --git a/docs/configuration.md b/docs/configuration.md index d5d53860b..c7ba9febe 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -31,6 +31,7 @@ The table below will list all configuration parameters. | `exchange.pair_blacklist` | [] | No | List of currency the bot must avoid. Useful when using `--dynamic-whitelist` param. | `experimental.use_sell_signal` | false | No | Use your sell strategy in addition of the `minimal_roi`. | `experimental.sell_profit_only` | false | No | waits until you have made a positive profit before taking a sell decision. +| `experimental.ignore_roi_if_buy_signal` | false | No | Does not sell if the buy-signal is still active. Takes preference over `minimal_roi` and `use_sell_signal` | `telegram.enabled` | true | Yes | Enable or not the usage of Telegram. | `telegram.token` | token | No | Your Telegram bot token. Only required if `telegram.enabled` is `true`. | `telegram.chat_id` | chat_id | No | Your personal Telegram account id. Only required if `telegram.enabled` is `true`. diff --git a/freqtrade/analyze.py b/freqtrade/analyze.py index cf2d5519b..b986b9611 100644 --- a/freqtrade/analyze.py +++ b/freqtrade/analyze.py @@ -172,6 +172,10 @@ class Analyze(object): if the threshold is reached and updates the trade record. :return: True if trade should be sold, False otherwise """ + if buy and self.config.get('experimental', {}).get('ignore_roi_if_buy_signal', False): + logger.debug('Buy signal still active - not selling.') + return False + # Check if minimal roi has been reached and no longer in buy conditions (avoiding a fee) if self.min_roi_reached(trade=trade, current_rate=rate, current_time=date): logger.debug('Required profit reached. Selling..') diff --git a/freqtrade/constants.py b/freqtrade/constants.py index 5be01f977..bf661aecc 100644 --- a/freqtrade/constants.py +++ b/freqtrade/constants.py @@ -73,7 +73,8 @@ CONF_SCHEMA = { 'type': 'object', 'properties': { 'use_sell_signal': {'type': 'boolean'}, - 'sell_profit_only': {'type': 'boolean'} + 'sell_profit_only': {'type': 'boolean'}, + "ignore_roi_if_buy_signal_true": {'type': 'boolean'} } }, 'telegram': { diff --git a/freqtrade/freqtradebot.py b/freqtrade/freqtradebot.py index 02375ca80..f06460706 100644 --- a/freqtrade/freqtradebot.py +++ b/freqtrade/freqtradebot.py @@ -424,7 +424,8 @@ with limit `{buy_limit:.8f} ({stake_amount:.6f} \ (buy, sell) = (False, False) - if self.config.get('experimental', {}).get('use_sell_signal'): + if (self.config.get('experimental', {}).get('use_sell_signal') + or self.config.get('experimental', {}).get('ignore_roi_if_buy_signal')): (buy, sell) = self.analyze.get_signal(self.exchange, trade.pair, self.analyze.get_ticker_interval()) diff --git a/freqtrade/tests/test_freqtradebot.py b/freqtrade/tests/test_freqtradebot.py index 9eae2fdf5..c18950177 100644 --- a/freqtrade/tests/test_freqtradebot.py +++ b/freqtrade/tests/test_freqtradebot.py @@ -1335,6 +1335,83 @@ def test_sell_profit_only_disable_loss(default_conf, limit_buy_order, fee, mocke assert freqtrade.handle_trade(trade) is True +def ignore_roi_if_buy_signal(default_conf, limit_buy_order, fee, mocker) -> None: + """ + Test sell_profit_only feature when enabled and we have a loss + """ + patch_get_signal(mocker) + patch_RPCManager(mocker) + patch_coinmarketcap(mocker) + mocker.patch('freqtrade.freqtradebot.Analyze.min_roi_reached', return_value=True) + mocker.patch.multiple( + 'freqtrade.exchange.Exchange', + validate_pairs=MagicMock(), + get_ticker=MagicMock(return_value={ + 'bid': 0.00000172, + 'ask': 0.00000173, + 'last': 0.00000172 + }), + buy=MagicMock(return_value={'id': limit_buy_order['id']}), + get_fee=fee, + ) + + conf = deepcopy(default_conf) + conf['experimental'] = { + 'ignore_roi_if_buy_signal': True + } + + freqtrade = FreqtradeBot(conf) + freqtrade.create_trade() + + trade = Trade.query.first() + trade.update(limit_buy_order) + patch_get_signal(mocker, value=(True, True)) + assert freqtrade.handle_trade(trade) is False + + # Test if buy-signal is absent (should sell due to roi = true) + patch_get_signal(mocker, value=(False, True)) + assert freqtrade.handle_trade(trade) is True + + +def test_disable_ignore_roi_if_buy_signal(default_conf, limit_buy_order, fee, mocker) -> None: + """ + Test sell_profit_only feature when enabled and we have a loss + """ + patch_get_signal(mocker) + patch_RPCManager(mocker) + patch_coinmarketcap(mocker) + mocker.patch('freqtrade.freqtradebot.Analyze.min_roi_reached', return_value=True) + mocker.patch.multiple( + 'freqtrade.exchange.Exchange', + validate_pairs=MagicMock(), + get_ticker=MagicMock(return_value={ + 'bid': 0.00000172, + 'ask': 0.00000173, + 'last': 0.00000172 + }), + buy=MagicMock(return_value={'id': limit_buy_order['id']}), + get_fee=fee, + ) + + conf = deepcopy(default_conf) + conf['experimental'] = { + 'ignore_roi_if_buy_signal': False + } + + freqtrade = FreqtradeBot(conf) + freqtrade.create_trade() + + trade = Trade.query.first() + trade.update(limit_buy_order) + # Sell due to min_roi_reached + patch_get_signal(mocker, value=(True, True)) + assert freqtrade.handle_trade(trade) is True + + # Test if buy-signal is absent + patch_get_signal(mocker, value=(False, True)) + assert freqtrade.handle_trade(trade) is True + + def test_get_real_amount_quote(default_conf, trades_for_order, buy_order_fee, caplog, mocker): """ Test get_real_amount - fee in quote currency From cbfee51f321992ba17501605514e87cf8a6361f3 Mon Sep 17 00:00:00 2001 From: xmatthias Date: Fri, 22 Jun 2018 20:51:21 +0200 Subject: [PATCH 51/64] introduce experimental variable and fix test naming --- freqtrade/analyze.py | 9 ++++----- freqtrade/freqtradebot.py | 5 ++--- freqtrade/tests/test_freqtradebot.py | 2 +- 3 files changed, 7 insertions(+), 9 deletions(-) diff --git a/freqtrade/analyze.py b/freqtrade/analyze.py index b986b9611..fc81f3fb9 100644 --- a/freqtrade/analyze.py +++ b/freqtrade/analyze.py @@ -172,7 +172,8 @@ class Analyze(object): if the threshold is reached and updates the trade record. :return: True if trade should be sold, False otherwise """ - if buy and self.config.get('experimental', {}).get('ignore_roi_if_buy_signal', False): + experimental = self.config.get('experimental', {}) + if buy and experimental.get('ignore_roi_if_buy_signal', False): logger.debug('Buy signal still active - not selling.') return False @@ -181,13 +182,11 @@ class Analyze(object): logger.debug('Required profit reached. Selling..') return True - # Experimental: Check if the trade is profitable before selling it (avoid selling at loss) - if self.config.get('experimental', {}).get('sell_profit_only', False): + if experimental.get('sell_profit_only', False): logger.debug('Checking if trade is profitable..') if trade.calc_profit(rate=rate) <= 0: return False - - if sell and not buy and self.config.get('experimental', {}).get('use_sell_signal', False): + if sell and not buy and experimental.get('use_sell_signal', False): logger.debug('Sell signal received. Selling..') return True diff --git a/freqtrade/freqtradebot.py b/freqtrade/freqtradebot.py index f06460706..221d32e9e 100644 --- a/freqtrade/freqtradebot.py +++ b/freqtrade/freqtradebot.py @@ -423,9 +423,8 @@ with limit `{buy_limit:.8f} ({stake_amount:.6f} \ current_rate = self.exchange.get_ticker(trade.pair)['bid'] (buy, sell) = (False, False) - - if (self.config.get('experimental', {}).get('use_sell_signal') - or self.config.get('experimental', {}).get('ignore_roi_if_buy_signal')): + experimental = self.config.get('experimental', {}) + if experimental.get('use_sell_signal') or experimental.get('ignore_roi_if_buy_signal'): (buy, sell) = self.analyze.get_signal(self.exchange, trade.pair, self.analyze.get_ticker_interval()) diff --git a/freqtrade/tests/test_freqtradebot.py b/freqtrade/tests/test_freqtradebot.py index c18950177..bad6d9502 100644 --- a/freqtrade/tests/test_freqtradebot.py +++ b/freqtrade/tests/test_freqtradebot.py @@ -1335,7 +1335,7 @@ def test_sell_profit_only_disable_loss(default_conf, limit_buy_order, fee, mocke assert freqtrade.handle_trade(trade) is True -def ignore_roi_if_buy_signal(default_conf, limit_buy_order, fee, mocker) -> None: +def test_ignore_roi_if_buy_signal(default_conf, limit_buy_order, fee, mocker) -> None: """ Test sell_profit_only feature when enabled and we have a loss """ From f7e5d2c3a572d7f1af3f28c164341ba942d480d9 Mon Sep 17 00:00:00 2001 From: Janne Sinivirta Date: Fri, 22 Jun 2018 21:55:09 +0300 Subject: [PATCH 52/64] check that we set fee on backtesting init --- freqtrade/tests/optimize/test_backtesting.py | 3 +++ 1 file changed, 3 insertions(+) diff --git a/freqtrade/tests/optimize/test_backtesting.py b/freqtrade/tests/optimize/test_backtesting.py index 1702818b1..22536e42f 100644 --- a/freqtrade/tests/optimize/test_backtesting.py +++ b/freqtrade/tests/optimize/test_backtesting.py @@ -298,6 +298,7 @@ def test_backtesting_init(mocker, default_conf) -> None: Test Backtesting._init() method """ patch_exchange(mocker) + get_fee = mocker.patch('freqtrade.exchange.Exchange.get_fee', MagicMock(return_value=0.5)) backtesting = Backtesting(default_conf) assert backtesting.config == default_conf assert isinstance(backtesting.analyze, Analyze) @@ -305,6 +306,8 @@ def test_backtesting_init(mocker, default_conf) -> None: assert callable(backtesting.tickerdata_to_dataframe) assert callable(backtesting.populate_buy_trend) assert callable(backtesting.populate_sell_trend) + get_fee.assert_called() + assert backtesting.fee == 0.5 def test_tickerdata_to_dataframe(default_conf, mocker) -> None: From e2a2a0be9b7bed68504d7adac5048c4c41d5980a Mon Sep 17 00:00:00 2001 From: xmatthias Date: Fri, 22 Jun 2018 21:21:34 +0200 Subject: [PATCH 53/64] extract stop_loss_reached to allow check before ignore_roi_if_buy_signal --- freqtrade/analyze.py | 20 ++++++++++++++------ freqtrade/tests/test_freqtradebot.py | 2 +- 2 files changed, 15 insertions(+), 7 deletions(-) diff --git a/freqtrade/analyze.py b/freqtrade/analyze.py index fc81f3fb9..4f4b4f391 100644 --- a/freqtrade/analyze.py +++ b/freqtrade/analyze.py @@ -172,13 +172,17 @@ class Analyze(object): if the threshold is reached and updates the trade record. :return: True if trade should be sold, False otherwise """ + current_profit = trade.calc_profit_percent(rate) experimental = self.config.get('experimental', {}) + if self.stop_loss_reached(current_profit=current_profit): + return True + if buy and experimental.get('ignore_roi_if_buy_signal', False): logger.debug('Buy signal still active - not selling.') return False # Check if minimal roi has been reached and no longer in buy conditions (avoiding a fee) - if self.min_roi_reached(trade=trade, current_rate=rate, current_time=date): + if self.min_roi_reached(trade=trade, current_profit=current_profit, current_time=date): logger.debug('Required profit reached. Selling..') return True @@ -192,16 +196,20 @@ class Analyze(object): return False - def min_roi_reached(self, trade: Trade, current_rate: float, current_time: datetime) -> bool: + def stop_loss_reached(self, current_profit: float) -> bool: + """Based on current profit of the trade and configured stoploss, decides to sell or not""" + + if self.strategy.stoploss is not None and current_profit < self.strategy.stoploss: + logger.debug('Stop loss hit.') + return True + return False + + def min_roi_reached(self, trade: Trade, current_profit: float, current_time: datetime) -> bool: """ Based an earlier trade and current price and ROI configuration, decides whether bot should sell :return True if bot should sell at current rate """ - current_profit = trade.calc_profit_percent(current_rate) - if self.strategy.stoploss is not None and current_profit < self.strategy.stoploss: - logger.debug('Stop loss hit.') - return True # Check if time matches and current rate is above threshold time_diff = (current_time.timestamp() - trade.open_date.timestamp()) / 60 diff --git a/freqtrade/tests/test_freqtradebot.py b/freqtrade/tests/test_freqtradebot.py index bad6d9502..33d9ae2d9 100644 --- a/freqtrade/tests/test_freqtradebot.py +++ b/freqtrade/tests/test_freqtradebot.py @@ -1274,7 +1274,7 @@ def test_sell_profit_only_enable_loss(default_conf, limit_buy_order, fee, mocker patch_get_signal(mocker) patch_RPCManager(mocker) patch_coinmarketcap(mocker) - mocker.patch('freqtrade.freqtradebot.Analyze.min_roi_reached', return_value=False) + mocker.patch('freqtrade.freqtradebot.Analyze.stop_loss_reached', return_value=False) mocker.patch.multiple( 'freqtrade.exchange.Exchange', validate_pairs=MagicMock(), From 2be7b3d9eb71b08543b5757947ded059e9eb4e4c Mon Sep 17 00:00:00 2001 From: xmatthias Date: Fri, 22 Jun 2018 21:24:21 +0200 Subject: [PATCH 54/64] fix mocked bid-value to match limt_buy_order config --- freqtrade/tests/test_freqtradebot.py | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/freqtrade/tests/test_freqtradebot.py b/freqtrade/tests/test_freqtradebot.py index 33d9ae2d9..0d4256c42 100644 --- a/freqtrade/tests/test_freqtradebot.py +++ b/freqtrade/tests/test_freqtradebot.py @@ -1312,9 +1312,9 @@ def test_sell_profit_only_disable_loss(default_conf, limit_buy_order, fee, mocke 'freqtrade.exchange.Exchange', validate_pairs=MagicMock(), get_ticker=MagicMock(return_value={ - 'bid': 0.00000172, - 'ask': 0.00000173, - 'last': 0.00000172 + 'bid': 0.0000172, + 'ask': 0.0000173, + 'last': 0.0000172 }), buy=MagicMock(return_value={'id': limit_buy_order['id']}), get_fee=fee, @@ -1347,9 +1347,9 @@ def test_ignore_roi_if_buy_signal(default_conf, limit_buy_order, fee, mocker) -> 'freqtrade.exchange.Exchange', validate_pairs=MagicMock(), get_ticker=MagicMock(return_value={ - 'bid': 0.00000172, - 'ask': 0.00000173, - 'last': 0.00000172 + 'bid': 0.0000172, + 'ask': 0.0000173, + 'last': 0.0000172 }), buy=MagicMock(return_value={'id': limit_buy_order['id']}), get_fee=fee, From 9a07d57ed7b8f45352cf78ae7f6d45cb01acfa54 Mon Sep 17 00:00:00 2001 From: Janne Sinivirta Date: Sat, 23 Jun 2018 07:58:25 +0300 Subject: [PATCH 55/64] fix flake8 --- freqtrade/tests/optimize/test_backtesting.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/freqtrade/tests/optimize/test_backtesting.py b/freqtrade/tests/optimize/test_backtesting.py index 22536e42f..65aa00a70 100644 --- a/freqtrade/tests/optimize/test_backtesting.py +++ b/freqtrade/tests/optimize/test_backtesting.py @@ -307,7 +307,7 @@ def test_backtesting_init(mocker, default_conf) -> None: assert callable(backtesting.populate_buy_trend) assert callable(backtesting.populate_sell_trend) get_fee.assert_called() - assert backtesting.fee == 0.5 + assert backtesting.fee == 0.5 def test_tickerdata_to_dataframe(default_conf, mocker) -> None: From 3360bf400125cf4d058534e9edfacd8ac3e54c1e Mon Sep 17 00:00:00 2001 From: gcarq Date: Sat, 23 Jun 2018 10:25:03 +0200 Subject: [PATCH 56/64] wrap strategies with HyperoptStrategy for module lookups with pickle --- freqtrade/optimize/hyperopt.py | 20 +++++++++++++++++++- 1 file changed, 19 insertions(+), 1 deletion(-) diff --git a/freqtrade/optimize/hyperopt.py b/freqtrade/optimize/hyperopt.py index 19e732d7b..e52581491 100644 --- a/freqtrade/optimize/hyperopt.py +++ b/freqtrade/optimize/hyperopt.py @@ -11,6 +11,7 @@ import pickle import signal import sys from argparse import Namespace +from copy import deepcopy from functools import reduce from math import exp from operator import itemgetter @@ -26,10 +27,25 @@ from freqtrade.arguments import Arguments from freqtrade.configuration import Configuration from freqtrade.optimize import load_data from freqtrade.optimize.backtesting import Backtesting +from freqtrade.strategy.interface import IStrategy logger = logging.getLogger(__name__) +HyperoptStrategy = None + + +def wrap_strategy(strategy: IStrategy) -> Optional[HyperoptStrategy]: + """Wraps a given Strategy instance to HyperoptStrategy""" + global HyperoptStrategy + + attr = deepcopy(dict(strategy.__class__.__dict__)) + # Patch module name to make it compatible with pickle + attr['__module__'] = 'freqtrade.optimize.hyperopt' + HyperoptStrategy = type('HyperoptStrategy', (IStrategy,), attr) + return HyperoptStrategy() + + class Hyperopt(Backtesting): """ Hyperopt class, this class contains all the logic to run a hyperopt simulation @@ -39,7 +55,6 @@ class Hyperopt(Backtesting): hyperopt.start() """ def __init__(self, config: Dict[str, Any]) -> None: - super().__init__(config) # set TARGET_TRADES to suit your number concurrent trades so its realistic # to the number of days @@ -57,6 +72,9 @@ class Hyperopt(Backtesting): # check that the reported Σ% values do not exceed this! self.expected_max_profit = 3.0 + # Wrap strategy to make it compatible with pickle + self.analyze.strategy = wrap_strategy(self.analyze.strategy) + # Configuration and data used by hyperopt self.processed: Optional[Dict[str, Any]] = None From c40e6a12d1ca4f302f876c10a3a107dcb5aae1c9 Mon Sep 17 00:00:00 2001 From: gcarq Date: Sat, 23 Jun 2018 11:13:49 +0200 Subject: [PATCH 57/64] move logic from hyperopt to freqtrade.strategy --- freqtrade/optimize/hyperopt.py | 19 ------------------- freqtrade/strategy/__init__.py | 32 ++++++++++++++++++++++++++++++++ freqtrade/strategy/resolver.py | 5 +++-- 3 files changed, 35 insertions(+), 21 deletions(-) diff --git a/freqtrade/optimize/hyperopt.py b/freqtrade/optimize/hyperopt.py index e52581491..7a313a3ac 100644 --- a/freqtrade/optimize/hyperopt.py +++ b/freqtrade/optimize/hyperopt.py @@ -11,7 +11,6 @@ import pickle import signal import sys from argparse import Namespace -from copy import deepcopy from functools import reduce from math import exp from operator import itemgetter @@ -27,25 +26,10 @@ from freqtrade.arguments import Arguments from freqtrade.configuration import Configuration from freqtrade.optimize import load_data from freqtrade.optimize.backtesting import Backtesting -from freqtrade.strategy.interface import IStrategy logger = logging.getLogger(__name__) -HyperoptStrategy = None - - -def wrap_strategy(strategy: IStrategy) -> Optional[HyperoptStrategy]: - """Wraps a given Strategy instance to HyperoptStrategy""" - global HyperoptStrategy - - attr = deepcopy(dict(strategy.__class__.__dict__)) - # Patch module name to make it compatible with pickle - attr['__module__'] = 'freqtrade.optimize.hyperopt' - HyperoptStrategy = type('HyperoptStrategy', (IStrategy,), attr) - return HyperoptStrategy() - - class Hyperopt(Backtesting): """ Hyperopt class, this class contains all the logic to run a hyperopt simulation @@ -72,9 +56,6 @@ class Hyperopt(Backtesting): # check that the reported Σ% values do not exceed this! self.expected_max_profit = 3.0 - # Wrap strategy to make it compatible with pickle - self.analyze.strategy = wrap_strategy(self.analyze.strategy) - # Configuration and data used by hyperopt self.processed: Optional[Dict[str, Any]] = None diff --git a/freqtrade/strategy/__init__.py b/freqtrade/strategy/__init__.py index e69de29bb..e1dc7bb3f 100644 --- a/freqtrade/strategy/__init__.py +++ b/freqtrade/strategy/__init__.py @@ -0,0 +1,32 @@ +import logging +from copy import deepcopy + +from freqtrade.strategy.interface import IStrategy + + +logger = logging.getLogger(__name__) + + +def import_strategy(strategy: IStrategy) -> IStrategy: + """ + Imports given Strategy instance to global scope + of freqtrade.strategy and returns an instance of it + """ + # Copy all attributes from base class and class + attr = deepcopy({**strategy.__class__.__dict__, **strategy.__dict__}) + # Adjust module name + attr['__module__'] = 'freqtrade.strategy' + + name = strategy.__class__.__name__ + clazz = type(name, (IStrategy,), attr) + + logger.debug( + 'Imported strategy %s.%s as %s.%s', + strategy.__module__, strategy.__class__.__name__, + clazz.__module__, strategy.__class__.__name__, + ) + + # Modify global scope to declare class + globals()[name] = clazz + + return clazz() diff --git a/freqtrade/strategy/resolver.py b/freqtrade/strategy/resolver.py index 3fd39bca3..3c7836291 100644 --- a/freqtrade/strategy/resolver.py +++ b/freqtrade/strategy/resolver.py @@ -11,6 +11,7 @@ from collections import OrderedDict from typing import Optional, Dict, Type from freqtrade import constants +from freqtrade.strategy import import_strategy from freqtrade.strategy.interface import IStrategy @@ -83,7 +84,7 @@ class StrategyResolver(object): strategy = self._search_strategy(path, strategy_name) if strategy: logger.info('Using resolved strategy %s from \'%s\'', strategy_name, path) - return strategy + return import_strategy(strategy) raise ImportError( "Impossible to load Strategy '{}'. This class does not exist" @@ -100,7 +101,7 @@ class StrategyResolver(object): """ # Generate spec based on absolute path - spec = importlib.util.spec_from_file_location('user_data.strategies', module_path) + spec = importlib.util.spec_from_file_location('unknown', module_path) module = importlib.util.module_from_spec(spec) spec.loader.exec_module(module) # type: ignore # importlib does not use typehints From 4bd61df3a7e7405c821e38f5bce4f06ae2fe4917 Mon Sep 17 00:00:00 2001 From: gcarq Date: Sat, 23 Jun 2018 11:14:31 +0200 Subject: [PATCH 58/64] implement test for import_strategy --- freqtrade/tests/strategy/test_strategy.py | 27 +++++++++++++++++++++-- 1 file changed, 25 insertions(+), 2 deletions(-) diff --git a/freqtrade/tests/strategy/test_strategy.py b/freqtrade/tests/strategy/test_strategy.py index 244910790..26cd798f4 100644 --- a/freqtrade/tests/strategy/test_strategy.py +++ b/freqtrade/tests/strategy/test_strategy.py @@ -5,10 +5,34 @@ import os import pytest +from freqtrade.strategy import import_strategy +from freqtrade.strategy.default_strategy import DefaultStrategy from freqtrade.strategy.interface import IStrategy from freqtrade.strategy.resolver import StrategyResolver +def test_import_strategy(caplog): + caplog.set_level(logging.DEBUG) + + strategy = DefaultStrategy() + strategy.some_method = lambda *args, **kwargs: 42 + + assert strategy.__module__ == 'freqtrade.strategy.default_strategy' + assert strategy.some_method() == 42 + + imported_strategy = import_strategy(strategy) + + assert imported_strategy.__module__ == 'freqtrade.strategy' + assert imported_strategy.some_method() == 42 + + assert ( + 'freqtrade.strategy', + logging.DEBUG, + 'Imported strategy freqtrade.strategy.default_strategy.DefaultStrategy ' + 'as freqtrade.strategy.DefaultStrategy', + ) in caplog.record_tuples + + def test_search_strategy(): default_location = os.path.join(os.path.dirname( os.path.realpath(__file__)), '..', '..', 'strategy' @@ -20,8 +44,7 @@ def test_search_strategy(): def test_load_strategy(result): - resolver = StrategyResolver() - resolver._load_strategy('TestStrategy') + resolver = StrategyResolver({'strategy': 'TestStrategy'}) assert hasattr(resolver.strategy, 'populate_indicators') assert 'adx' in resolver.strategy.populate_indicators(result) From 818a6b12ed6cad47a4b7e758534b7c5721365e06 Mon Sep 17 00:00:00 2001 From: gcarq Date: Sat, 23 Jun 2018 11:57:26 +0200 Subject: [PATCH 59/64] tests: add dir() assertion --- freqtrade/tests/strategy/test_strategy.py | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/freqtrade/tests/strategy/test_strategy.py b/freqtrade/tests/strategy/test_strategy.py index 26cd798f4..13eac3261 100644 --- a/freqtrade/tests/strategy/test_strategy.py +++ b/freqtrade/tests/strategy/test_strategy.py @@ -1,5 +1,4 @@ # pragma pylint: disable=missing-docstring, protected-access, C0103 - import logging import os @@ -22,6 +21,8 @@ def test_import_strategy(caplog): imported_strategy = import_strategy(strategy) + assert dir(strategy) == dir(imported_strategy) + assert imported_strategy.__module__ == 'freqtrade.strategy' assert imported_strategy.some_method() == 42 From fc219b4e940d84e1d0e45efb5b48bc1a2631858f Mon Sep 17 00:00:00 2001 From: xmatthias Date: Sat, 23 Jun 2018 13:10:08 +0200 Subject: [PATCH 60/64] move experimental eval below stop_loss_reached to improve performance --- freqtrade/analyze.py | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/freqtrade/analyze.py b/freqtrade/analyze.py index 4f4b4f391..a0f133b22 100644 --- a/freqtrade/analyze.py +++ b/freqtrade/analyze.py @@ -173,10 +173,11 @@ class Analyze(object): :return: True if trade should be sold, False otherwise """ current_profit = trade.calc_profit_percent(rate) - experimental = self.config.get('experimental', {}) if self.stop_loss_reached(current_profit=current_profit): return True + experimental = self.config.get('experimental', {}) + if buy and experimental.get('ignore_roi_if_buy_signal', False): logger.debug('Buy signal still active - not selling.') return False From d23cd73ba82abb0d670ccdbccfb4dcd821a557c9 Mon Sep 17 00:00:00 2001 From: xmatthias Date: Sat, 23 Jun 2018 13:12:36 +0200 Subject: [PATCH 61/64] update plotly --- requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/requirements.txt b/requirements.txt index a31976c8c..e26c9c8f6 100644 --- a/requirements.txt +++ b/requirements.txt @@ -22,4 +22,4 @@ tabulate==0.8.2 coinmarketcap==5.0.3 # Required for plotting data -#plotly==2.3.0 +#plotly==2.7.0 From 295dfe26523ba0cd959bb953e95c620650d31a63 Mon Sep 17 00:00:00 2001 From: gcarq Date: Sat, 23 Jun 2018 13:50:22 +0200 Subject: [PATCH 62/64] persistence: remove obsolete global _CONF variable --- freqtrade/persistence.py | 7 ++----- freqtrade/tests/test_persistence.py | 11 ++--------- 2 files changed, 4 insertions(+), 14 deletions(-) diff --git a/freqtrade/persistence.py b/freqtrade/persistence.py index 7fd8fdeb9..20a7db4bb 100644 --- a/freqtrade/persistence.py +++ b/freqtrade/persistence.py @@ -21,7 +21,6 @@ from freqtrade import OperationalException logger = logging.getLogger(__name__) -_CONF = {} _DECL_BASE: Any = declarative_base() @@ -33,9 +32,7 @@ def init(config: Dict) -> None: :param config: config to use :return: None """ - _CONF.update(config) - - db_url = _CONF.get('db_url', None) + db_url = config.get('db_url', None) kwargs = {} # Take care of thread ownership if in-memory db @@ -61,7 +58,7 @@ def init(config: Dict) -> None: check_migrate(engine) # Clean dry_run DB if the db is not in-memory - if _CONF.get('dry_run', False) and db_url != 'sqlite://': + if config.get('dry_run', False) and db_url != 'sqlite://': clean_dry_run_db() diff --git a/freqtrade/tests/test_persistence.py b/freqtrade/tests/test_persistence.py index c50ad7d2c..30ad239a1 100644 --- a/freqtrade/tests/test_persistence.py +++ b/freqtrade/tests/test_persistence.py @@ -14,9 +14,7 @@ def init_persistence(default_conf): init(default_conf) -def test_init_create_session(default_conf, mocker): - mocker.patch.dict('freqtrade.persistence._CONF', default_conf) - +def test_init_create_session(default_conf): # Check if init create a session init(default_conf) assert hasattr(Trade, 'session') @@ -29,20 +27,17 @@ def test_init_custom_db_url(default_conf, mocker): # Update path to a value other than default, but still in-memory conf.update({'db_url': 'sqlite:///tmp/freqtrade2_test.sqlite'}) create_engine_mock = mocker.patch('freqtrade.persistence.create_engine', MagicMock()) - mocker.patch.dict('freqtrade.persistence._CONF', conf) init(conf) assert create_engine_mock.call_count == 1 assert create_engine_mock.mock_calls[0][1][0] == 'sqlite:///tmp/freqtrade2_test.sqlite' -def test_init_invalid_db_url(default_conf, mocker): +def test_init_invalid_db_url(default_conf): conf = deepcopy(default_conf) # Update path to a value other than default, but still in-memory conf.update({'db_url': 'unknown:///some.url'}) - mocker.patch.dict('freqtrade.persistence._CONF', conf) - with pytest.raises(OperationalException, match=r'.*no valid database URL*'): init(conf) @@ -53,7 +48,6 @@ def test_init_prod_db(default_conf, mocker): conf.update({'db_url': constants.DEFAULT_DB_PROD_URL}) create_engine_mock = mocker.patch('freqtrade.persistence.create_engine', MagicMock()) - mocker.patch.dict('freqtrade.persistence._CONF', conf) init(conf) assert create_engine_mock.call_count == 1 @@ -66,7 +60,6 @@ def test_init_dryrun_db(default_conf, mocker): conf.update({'db_url': constants.DEFAULT_DB_DRYRUN_URL}) create_engine_mock = mocker.patch('freqtrade.persistence.create_engine', MagicMock()) - mocker.patch.dict('freqtrade.persistence._CONF', conf) init(conf) assert create_engine_mock.call_count == 1 From 0b3e4f6bcd07d416f5ef3508ebfc76fd4cdfb2c5 Mon Sep 17 00:00:00 2001 From: gcarq Date: Sat, 23 Jun 2018 13:50:49 +0200 Subject: [PATCH 63/64] remove dead code --- freqtrade/tests/exchange/test_exchange.py | 1 - 1 file changed, 1 deletion(-) diff --git a/freqtrade/tests/exchange/test_exchange.py b/freqtrade/tests/exchange/test_exchange.py index 3c68e8493..620113be1 100644 --- a/freqtrade/tests/exchange/test_exchange.py +++ b/freqtrade/tests/exchange/test_exchange.py @@ -510,7 +510,6 @@ def test_cancel_order_dry_run(default_conf, mocker): # Ensure that if not dry_run, we should call API def test_cancel_order(default_conf, mocker): default_conf['dry_run'] = False - # mocker.patch.dict('freqtrade.exchange.._CONF', default_conf) api_mock = MagicMock() api_mock.cancel_order = MagicMock(return_value=123) exchange = get_patched_exchange(mocker, default_conf, api_mock) From 925b9b0c1920c86c1e0e5df0abcb93602673b824 Mon Sep 17 00:00:00 2001 From: pyup-bot Date: Sat, 23 Jun 2018 14:23:07 +0200 Subject: [PATCH 64/64] Update ccxt from 1.14.253 to 1.14.256 --- requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/requirements.txt b/requirements.txt index e26c9c8f6..51312791e 100644 --- a/requirements.txt +++ b/requirements.txt @@ -1,4 +1,4 @@ -ccxt==1.14.253 +ccxt==1.14.256 SQLAlchemy==1.2.8 python-telegram-bot==10.1.0 arrow==0.12.1