From 1f30c3d7f1314d387da8e546987158ba8ed7e132 Mon Sep 17 00:00:00 2001 From: Matthias Date: Wed, 24 Feb 2021 06:46:07 +0100 Subject: [PATCH 001/187] Refresh slack link --- CONTRIBUTING.md | 2 +- README.md | 4 ++-- docs/developer.md | 2 +- docs/faq.md | 2 +- docs/index.md | 2 +- 5 files changed, 6 insertions(+), 6 deletions(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index afa41ed33..c29d6e632 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -12,7 +12,7 @@ Few pointers for contributions: - New features need to contain unit tests, must conform to PEP8 (max-line-length = 100) and should be documented with the introduction PR. - PR's can be declared as `[WIP]` - which signify Work in Progress Pull Requests (which are not finished). -If you are unsure, discuss the feature on our [discord server](https://discord.gg/MA9v74M), on [Slack](https://join.slack.com/t/highfrequencybot/shared_invite/zt-l9d9iqgl-9cVBIeBkCBa8j6upSmd_NA) or in a [issue](https://github.com/freqtrade/freqtrade/issues) before a PR. +If you are unsure, discuss the feature on our [discord server](https://discord.gg/MA9v74M), on [Slack](https://join.slack.com/t/highfrequencybot/shared_invite/zt-mm786y93-Fxo37glxMY9g8OQC5AoOIw) or in a [issue](https://github.com/freqtrade/freqtrade/issues) before a PR. ## Getting started diff --git a/README.md b/README.md index 7ef0d4ce7..c3a665c47 100644 --- a/README.md +++ b/README.md @@ -147,7 +147,7 @@ For any questions not covered by the documentation or for further information ab Please check out our [discord server](https://discord.gg/MA9v74M). -You can also join our [Slack channel](https://join.slack.com/t/highfrequencybot/shared_invite/zt-l9d9iqgl-9cVBIeBkCBa8j6upSmd_NA). +You can also join our [Slack channel](https://join.slack.com/t/highfrequencybot/shared_invite/zt-mm786y93-Fxo37glxMY9g8OQC5AoOIw). ### [Bugs / Issues](https://github.com/freqtrade/freqtrade/issues?q=is%3Aissue) @@ -178,7 +178,7 @@ to understand the requirements before sending your pull-requests. Coding is not a necessity to contribute - maybe start with improving our documentation? Issues labeled [good first issue](https://github.com/freqtrade/freqtrade/labels/good%20first%20issue) can be good first contributions, and will help get you familiar with the codebase. -**Note** before starting any major new feature work, *please open an issue describing what you are planning to do* or talk to us on [discord](https://discord.gg/MA9v74M) or [Slack](https://join.slack.com/t/highfrequencybot/shared_invite/zt-k9o2v5ut-jX8Mc4CwNM8CDc2Dyg96YA). This will ensure that interested parties can give valuable feedback on the feature, and let others know that you are working on it. +**Note** before starting any major new feature work, *please open an issue describing what you are planning to do* or talk to us on [discord](https://discord.gg/MA9v74M) or [Slack](https://join.slack.com/t/highfrequencybot/shared_invite/zt-mm786y93-Fxo37glxMY9g8OQC5AoOIw). This will ensure that interested parties can give valuable feedback on the feature, and let others know that you are working on it. **Important:** Always create your PR against the `develop` branch, not `stable`. diff --git a/docs/developer.md b/docs/developer.md index c09e528bf..4b8c64530 100644 --- a/docs/developer.md +++ b/docs/developer.md @@ -2,7 +2,7 @@ This page is intended for developers of Freqtrade, people who want to contribute to the Freqtrade codebase or documentation, or people who want to understand the source code of the application they're running. -All contributions, bug reports, bug fixes, documentation improvements, enhancements and ideas are welcome. We [track issues](https://github.com/freqtrade/freqtrade/issues) on [GitHub](https://github.com) and also have a dev channel on [discord](https://discord.gg/MA9v74M) or [slack](https://join.slack.com/t/highfrequencybot/shared_invite/zt-l9d9iqgl-9cVBIeBkCBa8j6upSmd_NA) where you can ask questions. +All contributions, bug reports, bug fixes, documentation improvements, enhancements and ideas are welcome. We [track issues](https://github.com/freqtrade/freqtrade/issues) on [GitHub](https://github.com) and also have a dev channel on [discord](https://discord.gg/MA9v74M) or [slack](https://join.slack.com/t/highfrequencybot/shared_invite/zt-mm786y93-Fxo37glxMY9g8OQC5AoOIw) where you can ask questions. ## Documentation diff --git a/docs/faq.md b/docs/faq.md index 87b0893bd..93b806dca 100644 --- a/docs/faq.md +++ b/docs/faq.md @@ -142,7 +142,7 @@ freqtrade hyperopt --hyperopt SampleHyperopt --hyperopt-loss SharpeHyperOptLossD ### Why does it take a long time to run hyperopt? -* Discovering a great strategy with Hyperopt takes time. Study www.freqtrade.io, the Freqtrade Documentation page, join the Freqtrade [Slack community](https://join.slack.com/t/highfrequencybot/shared_invite/zt-l9d9iqgl-9cVBIeBkCBa8j6upSmd_NA) - or the Freqtrade [discord community](https://discord.gg/X89cVG). While you patiently wait for the most advanced, free crypto bot in the world, to hand you a possible golden strategy specially designed just for you. +* Discovering a great strategy with Hyperopt takes time. Study www.freqtrade.io, the Freqtrade Documentation page, join the Freqtrade [Slack community](https://join.slack.com/t/highfrequencybot/shared_invite/zt-mm786y93-Fxo37glxMY9g8OQC5AoOIw) - or the Freqtrade [discord community](https://discord.gg/X89cVG). While you patiently wait for the most advanced, free crypto bot in the world, to hand you a possible golden strategy specially designed just for you. * If you wonder why it can take from 20 minutes to days to do 1000 epochs here are some answers: diff --git a/docs/index.md b/docs/index.md index db5088707..9d1a1532e 100644 --- a/docs/index.md +++ b/docs/index.md @@ -81,7 +81,7 @@ For any questions not covered by the documentation or for further information ab Please check out our [discord server](https://discord.gg/MA9v74M). -You can also join our [Slack channel](https://join.slack.com/t/highfrequencybot/shared_invite/zt-l9d9iqgl-9cVBIeBkCBa8j6upSmd_NA). +You can also join our [Slack channel](https://join.slack.com/t/highfrequencybot/shared_invite/zt-mm786y93-Fxo37glxMY9g8OQC5AoOIw). ## Ready to try? From 117f0064ed12945a415d53059d52ae9006ec29fc Mon Sep 17 00:00:00 2001 From: Th0masL Date: Thu, 25 Feb 2021 05:02:08 +0200 Subject: [PATCH 002/187] Allow changing the order_type for forcesell --- docs/configuration.md | 6 ++++-- docs/stoploss.md | 4 ++++ freqtrade/freqtradebot.py | 3 +++ 3 files changed, 11 insertions(+), 2 deletions(-) diff --git a/docs/configuration.md b/docs/configuration.md index 00d2830e4..2b13bdd3d 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -275,7 +275,7 @@ For example, if your strategy is using a 1h timeframe, and you only want to buy ### Understand order_types -The `order_types` configuration parameter maps actions (`buy`, `sell`, `stoploss`, `emergencysell`) to order-types (`market`, `limit`, ...) as well as configures stoploss to be on the exchange and defines stoploss on exchange update interval in seconds. +The `order_types` configuration parameter maps actions (`buy`, `sell`, `stoploss`, `emergencysell`, `forcesell`) to order-types (`market`, `limit`, ...) as well as configures stoploss to be on the exchange and defines stoploss on exchange update interval in seconds. This allows to buy using limit orders, sell using limit-orders, and create stoplosses using market orders. It also allows to set the @@ -287,7 +287,7 @@ the buy order is fulfilled. If this is configured, the following 4 values (`buy`, `sell`, `stoploss` and `stoploss_on_exchange`) need to be present, otherwise the bot will fail to start. -For information on (`emergencysell`,`stoploss_on_exchange`,`stoploss_on_exchange_interval`,`stoploss_on_exchange_limit_ratio`) please see stop loss documentation [stop loss on exchange](stoploss.md) +For information on (`emergencysell`,`forcesell`, `stoploss_on_exchange`,`stoploss_on_exchange_interval`,`stoploss_on_exchange_limit_ratio`) please see stop loss documentation [stop loss on exchange](stoploss.md) Syntax for Strategy: @@ -296,6 +296,7 @@ order_types = { "buy": "limit", "sell": "limit", "emergencysell": "market", + "forcesell": "market", "stoploss": "market", "stoploss_on_exchange": False, "stoploss_on_exchange_interval": 60, @@ -310,6 +311,7 @@ Configuration: "buy": "limit", "sell": "limit", "emergencysell": "market", + "forcesell": "market", "stoploss": "market", "stoploss_on_exchange": false, "stoploss_on_exchange_interval": 60 diff --git a/docs/stoploss.md b/docs/stoploss.md index 671e643b0..4a4391655 100644 --- a/docs/stoploss.md +++ b/docs/stoploss.md @@ -51,6 +51,10 @@ The bot cannot do these every 5 seconds (at each iteration), otherwise it would So this parameter will tell the bot how often it should update the stoploss order. The default value is 60 (1 minute). This same logic will reapply a stoploss order on the exchange should you cancel it accidentally. +### forcesell + +`forcesell` is an optional value, which defaults to the same value as `sell` and is used when sending a `/forcesell` command from Telegram or from the Rest API. + ### emergencysell `emergencysell` is an optional value, which defaults to `market` and is used when creating stop loss on exchange orders fails. diff --git a/freqtrade/freqtradebot.py b/freqtrade/freqtradebot.py index d546dd6d2..be35380bb 100644 --- a/freqtrade/freqtradebot.py +++ b/freqtrade/freqtradebot.py @@ -1156,6 +1156,9 @@ class FreqtradeBot(LoggingMixin): if sell_reason == SellType.EMERGENCY_SELL: # Emergency sells (default to market!) order_type = self.strategy.order_types.get("emergencysell", "market") + if sell_reason == SellType.FORCE_SELL: + # Force sells (default to the sell_type defined in the strategy, but we allow this value to be changed) + order_type = self.strategy.order_types.get("forcesell", order_type) amount = self._safe_sell_amount(trade.pair, trade.amount) time_in_force = self.strategy.order_time_in_force['sell'] From 006f31129e73fbe2ad994ef7be797395c640ae6a Mon Sep 17 00:00:00 2001 From: Th0masL Date: Thu, 25 Feb 2021 05:23:24 +0200 Subject: [PATCH 003/187] Reduced length of the line --- freqtrade/freqtradebot.py | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/freqtrade/freqtradebot.py b/freqtrade/freqtradebot.py index be35380bb..2f64f3dac 100644 --- a/freqtrade/freqtradebot.py +++ b/freqtrade/freqtradebot.py @@ -1157,7 +1157,8 @@ class FreqtradeBot(LoggingMixin): # Emergency sells (default to market!) order_type = self.strategy.order_types.get("emergencysell", "market") if sell_reason == SellType.FORCE_SELL: - # Force sells (default to the sell_type defined in the strategy, but we allow this value to be changed) + # Force sells (default to the sell_type defined in the strategy, + # but we allow this value to be changed) order_type = self.strategy.order_types.get("forcesell", order_type) amount = self._safe_sell_amount(trade.pair, trade.amount) From 262394e112aa66f17797089a5a8bcb51e0eb4bdd Mon Sep 17 00:00:00 2001 From: Matthias Date: Thu, 25 Feb 2021 19:24:30 +0100 Subject: [PATCH 004/187] Add psutils to support OOM Gracefull shutdown closes #4436, #4439 #3990 --- requirements-hyperopt.txt | 1 + setup.py | 1 + 2 files changed, 2 insertions(+) diff --git a/requirements-hyperopt.txt b/requirements-hyperopt.txt index 8e87a434c..8cdb6fd28 100644 --- a/requirements-hyperopt.txt +++ b/requirements-hyperopt.txt @@ -7,4 +7,5 @@ scikit-learn==0.24.1 scikit-optimize==0.8.1 filelock==3.0.12 joblib==1.0.1 +psutil==5.8.0 progressbar2==3.53.1 diff --git a/setup.py b/setup.py index 148803cd6..118bc8485 100644 --- a/setup.py +++ b/setup.py @@ -28,6 +28,7 @@ hyperopt = [ 'filelock', 'joblib', 'progressbar2', + 'psutil', ] develop = [ From 6d38a2e6598299949eeefc83919a0252abbae39f Mon Sep 17 00:00:00 2001 From: Matthias Date: Thu, 25 Feb 2021 19:54:57 +0100 Subject: [PATCH 005/187] Small enhancements to docs --- docs/bot-basics.md | 16 ++++++++-------- docs/configuration.md | 4 ++-- mkdocs.yml | 2 +- 3 files changed, 11 insertions(+), 11 deletions(-) diff --git a/docs/bot-basics.md b/docs/bot-basics.md index 30a25d4fc..13694c316 100644 --- a/docs/bot-basics.md +++ b/docs/bot-basics.md @@ -4,14 +4,14 @@ This page provides you some basic concepts on how Freqtrade works and operates. ## Freqtrade terminology -* Strategy: Your trading strategy, telling the bot what to do. -* Trade: Open position. -* Open Order: Order which is currently placed on the exchange, and is not yet complete. -* Pair: Tradable pair, usually in the format of Quote/Base (e.g. XRP/USDT). -* Timeframe: Candle length to use (e.g. `"5m"`, `"1h"`, ...). -* Indicators: Technical indicators (SMA, EMA, RSI, ...). -* Limit order: Limit orders which execute at the defined limit price or better. -* Market order: Guaranteed to fill, may move price depending on the order size. +* **Strategy**: Your trading strategy, telling the bot what to do. +* **Trade**: Open position. +* **Open Order**: Order which is currently placed on the exchange, and is not yet complete. +* **Pair**: Tradable pair, usually in the format of Quote/Base (e.g. XRP/USDT). +* **Timeframe**: Candle length to use (e.g. `"5m"`, `"1h"`, ...). +* **Indicators**: Technical indicators (SMA, EMA, RSI, ...). +* **Limit order**: Limit orders which execute at the defined limit price or better. +* **Market order**: Guaranteed to fill, may move price depending on the order size. ## Fee handling diff --git a/docs/configuration.md b/docs/configuration.md index 00d2830e4..e84455c22 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -224,6 +224,8 @@ To allow the bot to trade all the available `stake_currency` in your account (mi !!! Note "When using Dry-Run Mode" When using `"stake_amount" : "unlimited",` in combination with Dry-Run, the balance will be simulated starting with a stake of `dry_run_wallet` which will evolve over time. It is therefore important to set `dry_run_wallet` to a sensible value (like 0.05 or 0.01 for BTC and 1000 or 100 for USDT, for example), otherwise it may simulate trades with 100 BTC (or more) or 0.05 USDT (or less) at once - which may not correspond to your real available balance or is less than the exchange minimal limit for the order amount for the stake currency. +--8<-- "includes/pricing.md" + ### Understand minimal_roi The `minimal_roi` configuration parameter is a JSON object where the key is a duration @@ -449,8 +451,6 @@ The valid values are: "BTC", "ETH", "XRP", "LTC", "BCH", "USDT" ``` ---8<-- "includes/pricing.md" - ## Using Dry-run mode We recommend starting the bot in the Dry-run mode to see how your bot will diff --git a/mkdocs.yml b/mkdocs.yml index 18fccc333..ca52627cb 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -19,10 +19,10 @@ nav: - Backtesting: backtesting.md - Hyperopt: hyperopt.md - Utility Sub-commands: utils.md + - Plotting: plotting.md - Data Analysis: - Jupyter Notebooks: data-analysis.md - Strategy analysis: strategy_analysis_example.md - - Plotting: plotting.md - Exchange-specific Notes: exchanges.md - Advanced Topics: - Advanced Post-installation Tasks: advanced-setup.md From d877e3c1df72261f3b6c81ffca0fe9cab9148837 Mon Sep 17 00:00:00 2001 From: Matthias Date: Fri, 26 Feb 2021 06:51:32 +0100 Subject: [PATCH 006/187] Fix failing CI due to unavailable pairs --- config_bittrex.json.example | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/config_bittrex.json.example b/config_bittrex.json.example index 0f0bbec4b..172cfcfc3 100644 --- a/config_bittrex.json.example +++ b/config_bittrex.json.example @@ -41,13 +41,13 @@ "ETH/BTC", "LTC/BTC", "ETC/BTC", - "DASH/BTC", - "ZEC/BTC", + "RVN/BTC", + "CRO/BTC", "XLM/BTC", "XRP/BTC", "TRX/BTC", "ADA/BTC", - "XMR/BTC" + "DOT/BTC" ], "pair_blacklist": [ "DOGE/BTC" From c4979fd87fc121eabd897ec4e831fbc1489f2752 Mon Sep 17 00:00:00 2001 From: Matthias Date: Fri, 26 Feb 2021 06:57:51 +0100 Subject: [PATCH 007/187] Add note to check configuration settings to docker quickstart part of #4441 --- docs/docker_quickstart.md | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/docs/docker_quickstart.md b/docs/docker_quickstart.md index 9cccfa93d..5c7be3dde 100644 --- a/docs/docker_quickstart.md +++ b/docs/docker_quickstart.md @@ -75,7 +75,7 @@ The last 2 steps in the snippet create the directory with `user_data`, as well a 1. The configuration is now available as `user_data/config.json` 2. Copy a custom strategy to the directory `user_data/strategies/` -3. add the Strategy' class name to the `docker-compose.yml` file +3. Add the Strategy' class name to the `docker-compose.yml` file The `SampleStrategy` is run by default. @@ -90,6 +90,9 @@ Once this is done, you're ready to launch the bot in trading mode (Dry-run or Li docker-compose up -d ``` +!!! Warning "Default configuration" + While the configuration generated will be mostly functional, you will still need to verify that all options correspond to what you want (like Pricing, pairlist, ...) before starting the bot. + #### Monitoring the bot You can check for running instances with `docker-compose ps`. From 1b3b3891090483d40e78973a0147450388d0cbc6 Mon Sep 17 00:00:00 2001 From: Matthias Date: Fri, 26 Feb 2021 07:58:15 +0100 Subject: [PATCH 008/187] Remove binanceje, add ftx to config selector closes #4441 --- docs/exchanges.md | 3 --- freqtrade/commands/build_config_commands.py | 5 ++++- freqtrade/templates/base_config.json.j2 | 3 ++- 3 files changed, 6 insertions(+), 5 deletions(-) diff --git a/docs/exchanges.md b/docs/exchanges.md index d877e6da2..4b3833e72 100644 --- a/docs/exchanges.md +++ b/docs/exchanges.md @@ -92,9 +92,6 @@ To use subaccounts with FTX, you need to edit the configuration and add the foll } ``` -!!! Note - Older versions of freqtrade may require this key to be added to `"ccxt_async_config"` as well. - ## All exchanges Should you experience constant errors with Nonce (like `InvalidNonce`), it is best to regenerate the API keys. Resetting Nonce is difficult and it's usually easier to regenerate the API keys. diff --git a/freqtrade/commands/build_config_commands.py b/freqtrade/commands/build_config_commands.py index 7bdbcc057..3c34ff162 100644 --- a/freqtrade/commands/build_config_commands.py +++ b/freqtrade/commands/build_config_commands.py @@ -93,10 +93,10 @@ def ask_user_config() -> Dict[str, Any]: "message": "Select exchange", "choices": [ "binance", - "binanceje", "binanceus", "bittrex", "kraken", + "ftx", Separator(), "other", ], @@ -173,6 +173,9 @@ def deploy_new_config(config_path: Path, selections: Dict[str, Any]) -> None: arguments=selections) logger.info(f"Writing config to `{config_path}`.") + logger.info( + "Please make sure to check the configuration contents and adjust settings to your needs.") + config_path.write_text(config_text) diff --git a/freqtrade/templates/base_config.json.j2 b/freqtrade/templates/base_config.json.j2 index f920843b2..226bf1a81 100644 --- a/freqtrade/templates/base_config.json.j2 +++ b/freqtrade/templates/base_config.json.j2 @@ -57,7 +57,8 @@ "enabled": false, "listen_ip_address": "127.0.0.1", "listen_port": 8080, - "verbosity": "info", + "verbosity": "error", + "enable_openapi": false, "jwt_secret_key": "somethingrandom", "CORS_origins": [], "username": "", From 622ff771ec09ba7d8a7325c13a89922fc4cbd44d Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Fri, 26 Feb 2021 07:21:04 +0000 Subject: [PATCH 009/187] Bump aiohttp from 3.7.3 to 3.7.4 Bumps [aiohttp](https://github.com/aio-libs/aiohttp) from 3.7.3 to 3.7.4. - [Release notes](https://github.com/aio-libs/aiohttp/releases) - [Changelog](https://github.com/aio-libs/aiohttp/blob/master/CHANGES.rst) - [Commits](https://github.com/aio-libs/aiohttp/compare/v3.7.3...v3.7.4) Signed-off-by: dependabot[bot] --- requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/requirements.txt b/requirements.txt index 51b1ed3d1..d17070e34 100644 --- a/requirements.txt +++ b/requirements.txt @@ -4,7 +4,7 @@ pandas==1.2.2 ccxt==1.42.19 # Pin cryptography for now due to rust build errors with piwheels cryptography==3.4.6 -aiohttp==3.7.3 +aiohttp==3.7.4 SQLAlchemy==1.3.23 python-telegram-bot==13.3 arrow==0.17.0 From 51d73a58892252c8860e1584eb982ba5b82e0efe Mon Sep 17 00:00:00 2001 From: Marco Seguri Date: Fri, 26 Feb 2021 11:11:27 +0100 Subject: [PATCH 010/187] Fix #4441 --- docs/docker_quickstart.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/docker_quickstart.md b/docs/docker_quickstart.md index 5c7be3dde..017264569 100644 --- a/docs/docker_quickstart.md +++ b/docs/docker_quickstart.md @@ -1,5 +1,7 @@ # Using Freqtrade with Docker +This page explains how to run the bot with Docker. It is not meant to work out of the box. You'll still need to read through the documentation and understand how to properly configure it. + ## Install Docker Start by downloading and installing Docker CE for your platform: From fc69240e6dee03d482c85fe84b9d4cbe6b6b6fca Mon Sep 17 00:00:00 2001 From: Xanders Date: Fri, 26 Feb 2021 17:46:23 +0300 Subject: [PATCH 011/187] Add JSON-encoded webhooks --- freqtrade/rpc/webhook.py | 14 +++++++++++++- 1 file changed, 13 insertions(+), 1 deletion(-) diff --git a/freqtrade/rpc/webhook.py b/freqtrade/rpc/webhook.py index 5796201b5..6d9b718ff 100644 --- a/freqtrade/rpc/webhook.py +++ b/freqtrade/rpc/webhook.py @@ -28,6 +28,11 @@ class Webhook(RPCHandler): self._url = self._config['webhook']['url'] + self._format = self._config['webhook'].get('format', 'form') + + if self._format != 'form' and self._format != 'json': + raise NotImplementedError('Unknown webhook format `{}`, possible values are `form` (default) and `json`'.format(self._format)) + def cleanup(self) -> None: """ Cleanup pending module resources. @@ -66,7 +71,14 @@ class Webhook(RPCHandler): def _send_msg(self, payload: dict) -> None: """do the actual call to the webhook""" + if self._format == 'form': + kwargs = {'data': payload} + elif self._format == 'json': + kwargs = {'json': payload} + else: + raise NotImplementedError('Unknown format: {}'.format(self._format)) + try: - post(self._url, data=payload) + post(self._url, **kwargs) except RequestException as exc: logger.warning("Could not call webhook url. Exception: %s", exc) From a2cd3ed5ba0c17125787965cb5dd9e9528cce19f Mon Sep 17 00:00:00 2001 From: Xanders Date: Fri, 26 Feb 2021 17:59:38 +0300 Subject: [PATCH 012/187] Add documentation for JSON webhook format --- docs/webhook-config.md | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/docs/webhook-config.md b/docs/webhook-config.md index db6d4d1ef..14ac7e7ee 100644 --- a/docs/webhook-config.md +++ b/docs/webhook-config.md @@ -40,6 +40,19 @@ Sample configuration (tested using IFTTT). The url in `webhook.url` should point to the correct url for your webhook. If you're using [IFTTT](https://ifttt.com) (as shown in the sample above) please insert our event and key to the url. +You can set the POST body format to Form-Encoded (default) or JSON-Encoded. Use `"format": "form"` or `"format": "json"` respectively. Example configuration for Mattermost Cloud integration: + +```json + "webhook": { + "enabled": true, + "url": "https://.cloud.mattermost.com/hooks/", + "format": "json", + "webhookstatus": { + "text": "Status: {status}" + } + }, +``` + Different payloads can be configured for different events. Not all fields are necessary, but you should configure at least one of the dicts, otherwise the webhook will never be called. ### Webhookbuy From 52641aaa315ce6903cade30c9db904816f2cfabd Mon Sep 17 00:00:00 2001 From: Xanders Date: Fri, 26 Feb 2021 18:12:10 +0300 Subject: [PATCH 013/187] Add test for webhook JSON format --- tests/rpc/test_rpc_webhook.py | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/tests/rpc/test_rpc_webhook.py b/tests/rpc/test_rpc_webhook.py index 4ca547390..025b7326f 100644 --- a/tests/rpc/test_rpc_webhook.py +++ b/tests/rpc/test_rpc_webhook.py @@ -225,3 +225,14 @@ def test__send_msg(default_conf, mocker, caplog): mocker.patch("freqtrade.rpc.webhook.post", post) webhook._send_msg(msg) assert log_has('Could not call webhook url. Exception: ', caplog) + +def test__send_msg_with_json_format(default_conf, mocker, caplog): + default_conf["webhook"] = get_webhook_dict() + default_conf["webhook"]["format"] = "json" + webhook = Webhook(RPC(get_patched_freqtradebot(mocker, default_conf)), default_conf) + msg = {'text': 'Hello'} + post = MagicMock() + mocker.patch("freqtrade.rpc.webhook.post", post) + webhook._send_msg(msg) + + assert post.call_args[1] == {'json': msg} From 984e70d4e8dd8dc8e2303ac8997143b382c7f9ed Mon Sep 17 00:00:00 2001 From: Xanders Date: Fri, 26 Feb 2021 21:15:40 +0300 Subject: [PATCH 014/187] Add webhook result example to documentation --- docs/webhook-config.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/webhook-config.md b/docs/webhook-config.md index 14ac7e7ee..4d2b31ec9 100644 --- a/docs/webhook-config.md +++ b/docs/webhook-config.md @@ -53,6 +53,8 @@ You can set the POST body format to Form-Encoded (default) or JSON-Encoded. Use }, ``` +The result would be e.g. `Status: running` message in the Mattermost channel. + Different payloads can be configured for different events. Not all fields are necessary, but you should configure at least one of the dicts, otherwise the webhook will never be called. ### Webhookbuy From 9a926c155df00784bb1345736b6519a2b16776bb Mon Sep 17 00:00:00 2001 From: Matthias Date: Fri, 26 Feb 2021 19:30:42 +0100 Subject: [PATCH 015/187] Add forcesell entry to full config --- config_full.json.example | 1 + 1 file changed, 1 insertion(+) diff --git a/config_full.json.example b/config_full.json.example index 6593750b4..9a613c0a1 100644 --- a/config_full.json.example +++ b/config_full.json.example @@ -49,6 +49,7 @@ "buy": "limit", "sell": "limit", "emergencysell": "market", + "forcesell": "market", "stoploss": "market", "stoploss_on_exchange": false, "stoploss_on_exchange_interval": 60 From 7281e794b4fe547b147774c72cf4ce2b82dee99c Mon Sep 17 00:00:00 2001 From: Xanders Date: Fri, 26 Feb 2021 21:31:33 +0300 Subject: [PATCH 016/187] Fix too long line at webhook.py --- freqtrade/rpc/webhook.py | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/freqtrade/rpc/webhook.py b/freqtrade/rpc/webhook.py index 6d9b718ff..5a30a9be8 100644 --- a/freqtrade/rpc/webhook.py +++ b/freqtrade/rpc/webhook.py @@ -31,7 +31,8 @@ class Webhook(RPCHandler): self._format = self._config['webhook'].get('format', 'form') if self._format != 'form' and self._format != 'json': - raise NotImplementedError('Unknown webhook format `{}`, possible values are `form` (default) and `json`'.format(self._format)) + raise NotImplementedError('Unknown webhook format `{}`, possible values are ' + '`form` (default) and `json`'.format(self._format)) def cleanup(self) -> None: """ From efa50be145eb08e4496d9c0114ffc14d5b2abcf2 Mon Sep 17 00:00:00 2001 From: Xanders Date: Fri, 26 Feb 2021 21:32:41 +0300 Subject: [PATCH 017/187] Fix blank lines rule at test_rpc_webhook.py --- tests/rpc/test_rpc_webhook.py | 1 + 1 file changed, 1 insertion(+) diff --git a/tests/rpc/test_rpc_webhook.py b/tests/rpc/test_rpc_webhook.py index 025b7326f..5361cd947 100644 --- a/tests/rpc/test_rpc_webhook.py +++ b/tests/rpc/test_rpc_webhook.py @@ -226,6 +226,7 @@ def test__send_msg(default_conf, mocker, caplog): webhook._send_msg(msg) assert log_has('Could not call webhook url. Exception: ', caplog) + def test__send_msg_with_json_format(default_conf, mocker, caplog): default_conf["webhook"] = get_webhook_dict() default_conf["webhook"]["format"] = "json" From f0391d3761a1aec4f1ad2e03236d8552a4f135d7 Mon Sep 17 00:00:00 2001 From: Xanders Date: Fri, 26 Feb 2021 21:40:45 +0300 Subject: [PATCH 018/187] Better JSON webhook result description --- docs/webhook-config.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/webhook-config.md b/docs/webhook-config.md index 4d2b31ec9..2e41ad2cc 100644 --- a/docs/webhook-config.md +++ b/docs/webhook-config.md @@ -53,7 +53,7 @@ You can set the POST body format to Form-Encoded (default) or JSON-Encoded. Use }, ``` -The result would be e.g. `Status: running` message in the Mattermost channel. +The result would be POST request with e.g. `{"text":"Status: running"}` body and `Content-Type: application/json` header which results `Status: running` message in the Mattermost channel. Different payloads can be configured for different events. Not all fields are necessary, but you should configure at least one of the dicts, otherwise the webhook will never be called. From 642e3be7c549669e6068d694b875dfbe40661ee8 Mon Sep 17 00:00:00 2001 From: JoeSchr Date: Fri, 26 Feb 2021 23:17:59 +0100 Subject: [PATCH 019/187] Fix(strategy/interface.py): comment typo `advice_buy` -> `advise_buy` --- freqtrade/strategy/interface.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/freqtrade/strategy/interface.py b/freqtrade/strategy/interface.py index da4ce6c50..8a0b27e96 100644 --- a/freqtrade/strategy/interface.py +++ b/freqtrade/strategy/interface.py @@ -659,7 +659,7 @@ class IStrategy(ABC): def ohlcvdata_to_dataframe(self, data: Dict[str, DataFrame]) -> Dict[str, DataFrame]: """ Populates indicators for given candle (OHLCV) data (for multiple pairs) - Does not run advice_buy or advise_sell! + Does not run advise_buy or advise_sell! Used by optimize operations only, not during dry / live runs. Using .copy() to get a fresh copy of the dataframe for every strategy run. Has positive effects on memory usage for whatever reason - also when From 9968e4e49c58334c3541d2e5c153fc9eb266f4d6 Mon Sep 17 00:00:00 2001 From: Matthias Date: Sat, 27 Feb 2021 08:26:05 +0100 Subject: [PATCH 020/187] Add warning about downloading data from kraken closes #4449 --- docs/exchanges.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/docs/exchanges.md b/docs/exchanges.md index 4b3833e72..2e5bdfadd 100644 --- a/docs/exchanges.md +++ b/docs/exchanges.md @@ -40,6 +40,10 @@ Due to the heavy rate-limiting applied by Kraken, the following configuration se }, ``` +!!! Warning "Downloading data from kraken" + Downloading kraken data will require significantly more memory (RAM) than any other exchange, as the trades-data needs to be converted into candles on your machine. + It will also take a long time, as freqtrade will need to download every single trade that happened on the exchange for the pair / timerange combination, therefore please be patient. + ## Bittrex ### Order types From f0a154692de57912c1d5d9596133c17808f152f5 Mon Sep 17 00:00:00 2001 From: Matthias Date: Thu, 28 Jan 2021 06:37:42 +0100 Subject: [PATCH 021/187] Wallets should use trade_proxy --- freqtrade/wallets.py | 21 ++++++++++++--------- 1 file changed, 12 insertions(+), 9 deletions(-) diff --git a/freqtrade/wallets.py b/freqtrade/wallets.py index d7dcfd487..078bcd07e 100644 --- a/freqtrade/wallets.py +++ b/freqtrade/wallets.py @@ -11,6 +11,7 @@ from freqtrade.constants import UNLIMITED_STAKE_AMOUNT from freqtrade.exceptions import DependencyException from freqtrade.exchange import Exchange from freqtrade.persistence import Trade +from freqtrade.state import RunMode logger = logging.getLogger(__name__) @@ -26,13 +27,14 @@ class Wallet(NamedTuple): class Wallets: - def __init__(self, config: dict, exchange: Exchange) -> None: + def __init__(self, config: dict, exchange: Exchange, skip_update: bool = False) -> None: self._config = config self._exchange = exchange self._wallets: Dict[str, Wallet] = {} self.start_cap = config['dry_run_wallet'] self._last_wallet_refresh = 0 - self.update() + if not skip_update: + self.update() def get_free(self, currency: str) -> float: balance = self._wallets.get(currency) @@ -64,8 +66,8 @@ class Wallets: """ # Recreate _wallets to reset closed trade balances _wallets = {} - closed_trades = Trade.get_trades(Trade.is_open.is_(False)).all() - open_trades = Trade.get_trades(Trade.is_open.is_(True)).all() + closed_trades = Trade.get_trades_proxy(is_open=False) + open_trades = Trade.get_trades_proxy(is_open=True) tot_profit = sum([trade.calc_profit() for trade in closed_trades]) tot_in_trades = sum([trade.stake_amount for trade in open_trades]) @@ -102,7 +104,7 @@ class Wallets: if currency not in balances: del self._wallets[currency] - def update(self, require_update: bool = True) -> None: + def update(self, require_update: bool = True, log: bool = True) -> None: """ Updates wallets from the configured version. By default, updates from the exchange. @@ -111,11 +113,12 @@ class Wallets: :param require_update: Allow skipping an update if balances were recently refreshed """ if (require_update or (self._last_wallet_refresh + 3600 < arrow.utcnow().int_timestamp)): - if self._config['dry_run']: - self._update_dry() - else: + if (not self._config['dry_run'] or self._config.get('runmode') == RunMode.LIVE): self._update_live() - logger.info('Wallets synced.') + else: + self._update_dry() + if log: + logger.info('Wallets synced.') self._last_wallet_refresh = arrow.utcnow().int_timestamp def get_all_balances(self) -> Dict[str, Any]: From 9361aa1c95d5a408a4015e4ae5b04d29fd129b58 Mon Sep 17 00:00:00 2001 From: Matthias Date: Thu, 28 Jan 2021 07:06:58 +0100 Subject: [PATCH 022/187] Add wallets to backtesting --- freqtrade/optimize/backtesting.py | 17 +++++++++++++++-- 1 file changed, 15 insertions(+), 2 deletions(-) diff --git a/freqtrade/optimize/backtesting.py b/freqtrade/optimize/backtesting.py index 3186313e1..b68732d5c 100644 --- a/freqtrade/optimize/backtesting.py +++ b/freqtrade/optimize/backtesting.py @@ -28,6 +28,7 @@ from freqtrade.plugins.protectionmanager import ProtectionManager from freqtrade.resolvers import ExchangeResolver, StrategyResolver from freqtrade.strategy.interface import IStrategy, SellCheckTuple, SellType from freqtrade.strategy.strategy_wrapper import strategy_safe_wrapper +from freqtrade.wallets import Wallets logger = logging.getLogger(__name__) @@ -114,6 +115,8 @@ class Backtesting: if self.config.get('enable_protections', False): self.protections = ProtectionManager(self.config) + self.wallets = Wallets(self.config, self.exchange) + # Get maximum required startup period self.required_startup = max([strat.startup_candle_count for strat in self.strategylist]) # Load one (first) strategy @@ -176,6 +179,10 @@ class Backtesting: PairLocks.reset_locks() Trade.reset_trades() + def update_wallets(self): + if self.wallets: + self.wallets.update(log=False) + def _get_ohlcv_as_lists(self, processed: Dict[str, DataFrame]) -> Dict[str, Tuple]: """ Helper function to convert a processed dataframes into lists for performance reasons. @@ -276,8 +283,10 @@ class Backtesting: trade.close_date = sell_row[DATE_IDX] trade.sell_reason = SellType.FORCE_SELL trade.close(sell_row[OPEN_IDX], show_msg=False) - trade.is_open = True - trades.append(trade) + # Deepcopy object to have wallets update correctly + trade1 = deepcopy(trade) + trade1.is_open = True + trades.append(trade1) return trades def backtest(self, processed: Dict, stake_amount: float, @@ -346,6 +355,7 @@ class Backtesting: and tmp != end_date and row[BUY_IDX] == 1 and row[SELL_IDX] != 1 and not PairLocks.is_pair_locked(pair, row[DATE_IDX])): + self.update_wallets() # Enter trade trade = Trade( pair=pair, @@ -372,6 +382,7 @@ class Backtesting: trade_entry = self._get_sell_trade_entry(trade, row) # Sell occured if trade_entry: + self.update_wallets() # logger.debug(f"{pair} - Backtesting sell {trade}") open_trade_count -= 1 open_trades[pair].remove(trade) @@ -384,6 +395,7 @@ class Backtesting: tmp += timedelta(minutes=self.timeframe_min) trades += self.handle_left_open(open_trades, data=data) + self.update_wallets() return trade_list_to_dataframe(trades) @@ -425,6 +437,7 @@ class Backtesting: enable_protections=self.config.get('enable_protections', False), ) backtest_end_time = datetime.now(timezone.utc) + print(self.wallets.get_all_balances()) self.all_results[self.strategy.get_strategy_name()] = { 'results': results, 'config': self.strategy.config, From 4ce4eadc2366f6f58ab0811bc2fc01e2018627e9 Mon Sep 17 00:00:00 2001 From: Matthias Date: Sat, 30 Jan 2021 07:15:04 +0100 Subject: [PATCH 023/187] remove only ccxt objects when hyperopting --- freqtrade/optimize/hyperopt.py | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/freqtrade/optimize/hyperopt.py b/freqtrade/optimize/hyperopt.py index eee0f13b3..9cc5f2059 100644 --- a/freqtrade/optimize/hyperopt.py +++ b/freqtrade/optimize/hyperopt.py @@ -661,7 +661,9 @@ class Hyperopt: dump(preprocessed, self.data_pickle_file) # We don't need exchange instance anymore while running hyperopt - self.backtesting.exchange = None # type: ignore + self.backtesting.exchange._api = None # type: ignore + self.backtesting.exchange._api_async = None # type: ignore + # self.backtesting.exchange = None # type: ignore self.backtesting.pairlists = None # type: ignore self.backtesting.strategy.dp = None # type: ignore IStrategy.dp = None # type: ignore From b5177eadabe2a349382d7ee537aaae423942435f Mon Sep 17 00:00:00 2001 From: Matthias Date: Sat, 6 Feb 2021 10:22:59 +0100 Subject: [PATCH 024/187] Extract close method for exchange --- freqtrade/exchange/exchange.py | 3 +++ freqtrade/optimize/hyperopt.py | 1 + 2 files changed, 4 insertions(+) diff --git a/freqtrade/exchange/exchange.py b/freqtrade/exchange/exchange.py index 617cd6c26..0e9a90548 100644 --- a/freqtrade/exchange/exchange.py +++ b/freqtrade/exchange/exchange.py @@ -147,6 +147,9 @@ class Exchange: """ Destructor - clean up async stuff """ + self.close() + + def close(self): logger.debug("Exchange object destroyed, closing async loop") if self._api_async and inspect.iscoroutinefunction(self._api_async.close): asyncio.get_event_loop().run_until_complete(self._api_async.close()) diff --git a/freqtrade/optimize/hyperopt.py b/freqtrade/optimize/hyperopt.py index 9cc5f2059..155f1e69b 100644 --- a/freqtrade/optimize/hyperopt.py +++ b/freqtrade/optimize/hyperopt.py @@ -661,6 +661,7 @@ class Hyperopt: dump(preprocessed, self.data_pickle_file) # We don't need exchange instance anymore while running hyperopt + self.backtesting.exchange.close() self.backtesting.exchange._api = None # type: ignore self.backtesting.exchange._api_async = None # type: ignore # self.backtesting.exchange = None # type: ignore From 712d503e6ca51acde5a676f833f073018d465cab Mon Sep 17 00:00:00 2001 From: Matthias Date: Sat, 6 Feb 2021 10:30:50 +0100 Subject: [PATCH 025/187] Use sell-reason value in backtesting, not the enum object --- freqtrade/optimize/backtesting.py | 9 +++++---- freqtrade/optimize/optimize_reports.py | 2 +- freqtrade/persistence/models.py | 12 ++++++++++-- tests/optimize/test_backtest_detail.py | 2 +- tests/optimize/test_backtesting.py | 2 +- tests/optimize/test_optimize_reports.py | 2 +- 6 files changed, 19 insertions(+), 10 deletions(-) diff --git a/freqtrade/optimize/backtesting.py b/freqtrade/optimize/backtesting.py index b68732d5c..718fd2c42 100644 --- a/freqtrade/optimize/backtesting.py +++ b/freqtrade/optimize/backtesting.py @@ -259,11 +259,11 @@ class Backtesting: sell_row[BUY_IDX], sell_row[SELL_IDX], low=sell_row[LOW_IDX], high=sell_row[HIGH_IDX]) if sell.sell_flag: - trade_dur = int((sell_row[DATE_IDX] - trade.open_date).total_seconds() // 60) - closerate = self._get_close_rate(sell_row, trade, sell, trade_dur) trade.close_date = sell_row[DATE_IDX] - trade.sell_reason = sell.sell_type + trade.sell_reason = sell.sell_type.value + trade_dur = int((trade.close_date_utc - trade.open_date_utc).total_seconds() // 60) + closerate = self._get_close_rate(sell_row, trade, sell, trade_dur) trade.close(closerate, show_msg=False) return trade @@ -281,7 +281,7 @@ class Backtesting: sell_row = data[pair][-1] trade.close_date = sell_row[DATE_IDX] - trade.sell_reason = SellType.FORCE_SELL + trade.sell_reason = SellType.FORCE_SELL.value trade.close(sell_row[OPEN_IDX], show_msg=False) # Deepcopy object to have wallets update correctly trade1 = deepcopy(trade) @@ -366,6 +366,7 @@ class Backtesting: fee_open=self.fee, fee_close=self.fee, is_open=True, + exchange='backtesting', ) # TODO: hacky workaround to avoid opening > max_open_trades # This emulates previous behaviour - not sure if this is correct diff --git a/freqtrade/optimize/optimize_reports.py b/freqtrade/optimize/optimize_reports.py index 88b2028ba..6338b1d71 100644 --- a/freqtrade/optimize/optimize_reports.py +++ b/freqtrade/optimize/optimize_reports.py @@ -132,7 +132,7 @@ def generate_sell_reason_stats(max_open_trades: int, results: DataFrame) -> List tabular_data.append( { - 'sell_reason': reason.value, + 'sell_reason': reason, 'trades': count, 'wins': len(result[result['profit_abs'] > 0]), 'draws': len(result[result['profit_abs'] == 0]), diff --git a/freqtrade/persistence/models.py b/freqtrade/persistence/models.py index dff59819c..a05aa2c96 100644 --- a/freqtrade/persistence/models.py +++ b/freqtrade/persistence/models.py @@ -268,6 +268,14 @@ class Trade(_DECL_BASE): return (f'Trade(id={self.id}, pair={self.pair}, amount={self.amount:.8f}, ' f'open_rate={self.open_rate:.8f}, open_since={open_since})') + @property + def open_date_utc(self): + return self.open_date.replace(tzinfo=timezone.utc) + + @property + def close_date_utc(self): + return self.close_date.replace(tzinfo=timezone.utc) + def to_json(self) -> Dict[str, Any]: return { 'trade_id': self.id, @@ -306,9 +314,9 @@ class Trade(_DECL_BASE): 'close_profit_pct': round(self.close_profit * 100, 2) if self.close_profit else None, 'close_profit_abs': self.close_profit_abs, # Deprecated - 'trade_duration_s': (int((self.close_date - self.open_date).total_seconds()) + 'trade_duration_s': (int((self.close_date_utc - self.open_date_utc).total_seconds()) if self.close_date else None), - 'trade_duration': (int((self.close_date - self.open_date).total_seconds() // 60) + 'trade_duration': (int((self.close_date_utc - self.open_date_utc).total_seconds() // 60) if self.close_date else None), 'profit_ratio': self.close_profit, diff --git a/tests/optimize/test_backtest_detail.py b/tests/optimize/test_backtest_detail.py index daf7c2053..c9499cc42 100644 --- a/tests/optimize/test_backtest_detail.py +++ b/tests/optimize/test_backtest_detail.py @@ -514,6 +514,6 @@ def test_backtest_results(default_conf, fee, mocker, caplog, data) -> None: for c, trade in enumerate(data.trades): res = results.iloc[c] - assert res.sell_reason == trade.sell_reason + assert res.sell_reason == trade.sell_reason.value assert res.open_date == _get_frame_time_from_offset(trade.open_tick) assert res.close_date == _get_frame_time_from_offset(trade.close_tick) diff --git a/tests/optimize/test_backtesting.py b/tests/optimize/test_backtesting.py index c8d4338af..db14749c3 100644 --- a/tests/optimize/test_backtesting.py +++ b/tests/optimize/test_backtesting.py @@ -486,7 +486,7 @@ def test_backtest_one(default_conf, fee, mocker, testdatadir) -> None: 'trade_duration': [235, 40], 'profit_ratio': [0.0, 0.0], 'profit_abs': [0.0, 0.0], - 'sell_reason': [SellType.ROI, SellType.ROI], + 'sell_reason': [SellType.ROI.value, SellType.ROI.value], 'initial_stop_loss_abs': [0.0940005, 0.09272236], 'initial_stop_loss_ratio': [-0.1, -0.1], 'stop_loss_abs': [0.0940005, 0.09272236], diff --git a/tests/optimize/test_optimize_reports.py b/tests/optimize/test_optimize_reports.py index 51a78c7cc..8b64c2764 100644 --- a/tests/optimize/test_optimize_reports.py +++ b/tests/optimize/test_optimize_reports.py @@ -265,7 +265,7 @@ def test_generate_sell_reason_stats(): 'wins': [2, 0, 0], 'draws': [0, 0, 0], 'losses': [0, 0, 1], - 'sell_reason': [SellType.ROI, SellType.ROI, SellType.STOP_LOSS] + 'sell_reason': [SellType.ROI.value, SellType.ROI.value, SellType.STOP_LOSS.value] } ) From e32b2097f0010127f0bbb095a3968ccc5c71f6c9 Mon Sep 17 00:00:00 2001 From: Matthias Date: Sun, 7 Feb 2021 10:20:43 +0100 Subject: [PATCH 026/187] Use timestamp in UTC timezone for ROI comparisons --- freqtrade/freqtradebot.py | 2 +- freqtrade/strategy/interface.py | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/freqtrade/freqtradebot.py b/freqtrade/freqtradebot.py index 2f64f3dac..fd2f8bdd0 100644 --- a/freqtrade/freqtradebot.py +++ b/freqtrade/freqtradebot.py @@ -932,7 +932,7 @@ class FreqtradeBot(LoggingMixin): Check and execute sell """ should_sell = self.strategy.should_sell( - trade, sell_rate, datetime.utcnow(), buy, sell, + trade, sell_rate, datetime.now(timezone.utc), buy, sell, force_stoploss=self.edge.stoploss(trade.pair) if self.edge else 0 ) diff --git a/freqtrade/strategy/interface.py b/freqtrade/strategy/interface.py index 8a0b27e96..6d40e56cc 100644 --- a/freqtrade/strategy/interface.py +++ b/freqtrade/strategy/interface.py @@ -649,7 +649,7 @@ class IStrategy(ABC): :return: True if bot should sell at current rate """ # Check if time matches and current rate is above threshold - trade_dur = int((current_time.timestamp() - trade.open_date.timestamp()) // 60) + trade_dur = int((current_time.timestamp() - trade.open_date_utc.timestamp()) // 60) _, roi = self.min_roi_reached_entry(trade_dur) if roi is None: return False From 081b9be45c072d4d39f5672122d3c5dbbbf5aa07 Mon Sep 17 00:00:00 2001 From: Matthias Date: Sun, 7 Feb 2021 10:49:47 +0100 Subject: [PATCH 027/187] use get_all_locks to get locks for backtest result --- freqtrade/optimize/backtesting.py | 2 +- freqtrade/persistence/pairlock_middleware.py | 8 ++++++++ 2 files changed, 9 insertions(+), 1 deletion(-) diff --git a/freqtrade/optimize/backtesting.py b/freqtrade/optimize/backtesting.py index 718fd2c42..f37107767 100644 --- a/freqtrade/optimize/backtesting.py +++ b/freqtrade/optimize/backtesting.py @@ -442,7 +442,7 @@ class Backtesting: self.all_results[self.strategy.get_strategy_name()] = { 'results': results, 'config': self.strategy.config, - 'locks': PairLocks.locks, + 'locks': PairLocks.get_all_locks(), 'backtest_start_time': int(backtest_start_time.timestamp()), 'backtest_end_time': int(backtest_end_time.timestamp()), } diff --git a/freqtrade/persistence/pairlock_middleware.py b/freqtrade/persistence/pairlock_middleware.py index 8644146d8..f0048bb52 100644 --- a/freqtrade/persistence/pairlock_middleware.py +++ b/freqtrade/persistence/pairlock_middleware.py @@ -123,3 +123,11 @@ class PairLocks(): now = datetime.now(timezone.utc) return len(PairLocks.get_pair_locks(pair, now)) > 0 or PairLocks.is_global_lock(now) + + @staticmethod + def get_all_locks() -> List[PairLock]: + + if PairLocks.use_db: + return PairLock.query.all() + else: + return PairLocks.locks From 20455de2a9533ebbf3990b9efe241a1bec1543e3 Mon Sep 17 00:00:00 2001 From: Matthias Date: Tue, 9 Feb 2021 20:22:33 +0100 Subject: [PATCH 028/187] Small enhancements to docs --- docs/strategy-customization.md | 2 +- freqtrade/persistence/migrations.py | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/strategy-customization.md b/docs/strategy-customization.md index fdc95a3c1..fd733c88e 100644 --- a/docs/strategy-customization.md +++ b/docs/strategy-customization.md @@ -709,7 +709,7 @@ To verify if a pair is currently locked, use `self.is_pair_locked(pair)`. Locked pairs will always be rounded up to the next candle. So assuming a `5m` timeframe, a lock with `until` set to 10:18 will lock the pair until the candle from 10:15-10:20 will be finished. !!! Warning - Locking pairs is not available during backtesting. + Manually locking pairs is not available during backtesting, only locks via Protections are allowed. #### Pair locking example diff --git a/freqtrade/persistence/migrations.py b/freqtrade/persistence/migrations.py index ed976c2a9..961363b0e 100644 --- a/freqtrade/persistence/migrations.py +++ b/freqtrade/persistence/migrations.py @@ -141,7 +141,7 @@ def check_migrate(engine, decl_base, previous_tables) -> None: inspector = inspect(engine) cols = inspector.get_columns('trades') - if 'orders' not in previous_tables: + if 'orders' not in previous_tables and 'trades' in previous_tables: logger.info('Moving open orders to Orders table.') migrate_open_orders_to_trades(engine) else: From 0faa6f84dcd6ee8d5990fd8f2bbe0c7fed80dac9 Mon Sep 17 00:00:00 2001 From: Matthias Date: Wed, 10 Feb 2021 19:23:11 +0100 Subject: [PATCH 029/187] Improve Wallet logging disabling for backtesting --- freqtrade/optimize/backtesting.py | 3 ++- freqtrade/wallets.py | 5 +++-- 2 files changed, 5 insertions(+), 3 deletions(-) diff --git a/freqtrade/optimize/backtesting.py b/freqtrade/optimize/backtesting.py index f37107767..7f2ba60f2 100644 --- a/freqtrade/optimize/backtesting.py +++ b/freqtrade/optimize/backtesting.py @@ -116,6 +116,7 @@ class Backtesting: self.protections = ProtectionManager(self.config) self.wallets = Wallets(self.config, self.exchange) + self.wallets._log = False # Get maximum required startup period self.required_startup = max([strat.startup_candle_count for strat in self.strategylist]) @@ -181,7 +182,7 @@ class Backtesting: def update_wallets(self): if self.wallets: - self.wallets.update(log=False) + self.wallets.update() def _get_ohlcv_as_lists(self, processed: Dict[str, DataFrame]) -> Dict[str, Tuple]: """ diff --git a/freqtrade/wallets.py b/freqtrade/wallets.py index 078bcd07e..9562f34e6 100644 --- a/freqtrade/wallets.py +++ b/freqtrade/wallets.py @@ -29,6 +29,7 @@ class Wallets: def __init__(self, config: dict, exchange: Exchange, skip_update: bool = False) -> None: self._config = config + self._log = True self._exchange = exchange self._wallets: Dict[str, Wallet] = {} self.start_cap = config['dry_run_wallet'] @@ -104,7 +105,7 @@ class Wallets: if currency not in balances: del self._wallets[currency] - def update(self, require_update: bool = True, log: bool = True) -> None: + def update(self, require_update: bool = True) -> None: """ Updates wallets from the configured version. By default, updates from the exchange. @@ -117,7 +118,7 @@ class Wallets: self._update_live() else: self._update_dry() - if log: + if self._log: logger.info('Wallets synced.') self._last_wallet_refresh = arrow.utcnow().int_timestamp From 0754a7a78f36d379d9332de0bfef3124780debe0 Mon Sep 17 00:00:00 2001 From: Matthias Date: Wed, 10 Feb 2021 19:33:39 +0100 Subject: [PATCH 030/187] total_open_trades_stake should support no-db mode --- freqtrade/persistence/models.py | 10 +++++++--- tests/conftest.py | 20 +++++++++++++------- tests/conftest_trades.py | 4 ++++ tests/test_persistence.py | 8 ++++++-- 4 files changed, 30 insertions(+), 12 deletions(-) diff --git a/freqtrade/persistence/models.py b/freqtrade/persistence/models.py index a05aa2c96..f72705c34 100644 --- a/freqtrade/persistence/models.py +++ b/freqtrade/persistence/models.py @@ -671,9 +671,13 @@ class Trade(_DECL_BASE): Calculates total invested amount in open trades in stake currency """ - total_open_stake_amount = Trade.session.query(func.sum(Trade.stake_amount))\ - .filter(Trade.is_open.is_(True))\ - .scalar() + if Trade.use_db: + total_open_stake_amount = Trade.session.query(func.sum(Trade.stake_amount))\ + .filter(Trade.is_open.is_(True))\ + .scalar() + else: + total_open_stake_amount = sum( + t.stake_amount for t in Trade.get_trades_proxy(is_open=True)) return total_open_stake_amount or 0 @staticmethod diff --git a/tests/conftest.py b/tests/conftest.py index 61899dd53..946ae1fb5 100644 --- a/tests/conftest.py +++ b/tests/conftest.py @@ -183,28 +183,34 @@ def patch_get_signal(freqtrade: FreqtradeBot, value=(True, False)) -> None: freqtrade.exchange.refresh_latest_ohlcv = lambda p: None -def create_mock_trades(fee): +def create_mock_trades(fee, use_db: bool = True): """ Create some fake trades ... """ + def add_trade(trade): + if use_db: + Trade.session.add(trade) + else: + Trade.trades.append(trade) + # Simulate dry_run entries trade = mock_trade_1(fee) - Trade.session.add(trade) + add_trade(trade) trade = mock_trade_2(fee) - Trade.session.add(trade) + add_trade(trade) trade = mock_trade_3(fee) - Trade.session.add(trade) + add_trade(trade) trade = mock_trade_4(fee) - Trade.session.add(trade) + add_trade(trade) trade = mock_trade_5(fee) - Trade.session.add(trade) + add_trade(trade) trade = mock_trade_6(fee) - Trade.session.add(trade) + add_trade(trade) @pytest.fixture(autouse=True) diff --git a/tests/conftest_trades.py b/tests/conftest_trades.py index fa9910b8d..6a42d04e3 100644 --- a/tests/conftest_trades.py +++ b/tests/conftest_trades.py @@ -28,6 +28,7 @@ def mock_trade_1(fee): amount_requested=123.0, fee_open=fee.return_value, fee_close=fee.return_value, + is_open=True, open_rate=0.123, exchange='bittrex', open_order_id='dry_run_buy_12345', @@ -180,6 +181,7 @@ def mock_trade_4(fee): amount_requested=124.0, fee_open=fee.return_value, fee_close=fee.return_value, + is_open=True, open_rate=0.123, exchange='bittrex', open_order_id='prod_buy_12345', @@ -230,6 +232,7 @@ def mock_trade_5(fee): amount_requested=124.0, fee_open=fee.return_value, fee_close=fee.return_value, + is_open=True, open_rate=0.123, exchange='bittrex', strategy='SampleStrategy', @@ -281,6 +284,7 @@ def mock_trade_6(fee): amount_requested=2.0, fee_open=fee.return_value, fee_close=fee.return_value, + is_open=True, open_rate=0.15, exchange='bittrex', strategy='SampleStrategy', diff --git a/tests/test_persistence.py b/tests/test_persistence.py index d0d29f142..1fced3e16 100644 --- a/tests/test_persistence.py +++ b/tests/test_persistence.py @@ -1039,14 +1039,18 @@ def test_fee_updated(fee): @pytest.mark.usefixtures("init_persistence") -def test_total_open_trades_stakes(fee): +@pytest.mark.parametrize('use_db', [True, False]) +def test_total_open_trades_stakes(fee, use_db): + Trade.use_db = use_db res = Trade.total_open_trades_stakes() assert res == 0 - create_mock_trades(fee) + create_mock_trades(fee, use_db) res = Trade.total_open_trades_stakes() assert res == 0.004 + Trade.use_db = True + @pytest.mark.usefixtures("init_persistence") def test_get_overall_performance(fee): From 959ff990460acd0e137c0c2aaccbb6cfc1efd932 Mon Sep 17 00:00:00 2001 From: Matthias Date: Wed, 10 Feb 2021 19:45:59 +0100 Subject: [PATCH 031/187] Add Dry-run wallet CLI option --- docs/backtesting.md | 4 ++++ docs/bot-usage.md | 4 ++++ docs/configuration.md | 2 +- docs/hyperopt.md | 6 +++++- freqtrade/commands/arguments.py | 6 +++--- freqtrade/commands/cli_options.py | 5 +++++ freqtrade/configuration/configuration.py | 4 +++- freqtrade/wallets.py | 1 + 8 files changed, 26 insertions(+), 6 deletions(-) diff --git a/docs/backtesting.md b/docs/backtesting.md index a14c8f2e4..38d1af45a 100644 --- a/docs/backtesting.md +++ b/docs/backtesting.md @@ -16,6 +16,7 @@ usage: freqtrade backtesting [-h] [-v] [--logfile FILE] [-V] [-c PATH] [--max-open-trades INT] [--stake-amount STAKE_AMOUNT] [--fee FLOAT] [--eps] [--dmmp] [--enable-protections] + [--dry-run-wallet DRY_RUN_WALLET] [--strategy-list STRATEGY_LIST [STRATEGY_LIST ...]] [--export EXPORT] [--export-filename PATH] @@ -48,6 +49,9 @@ optional arguments: Enable protections for backtesting.Will slow backtesting down by a considerable amount, but will include configured protections + --dry-run-wallet DRY_RUN_WALLET + Starting balance, used for backtesting / hyperopt and + dry-runs. --strategy-list STRATEGY_LIST [STRATEGY_LIST ...] Provide a space-separated list of strategies to backtest. Please note that ticker-interval needs to be diff --git a/docs/bot-usage.md b/docs/bot-usage.md index c7fe8634d..4ff6168a0 100644 --- a/docs/bot-usage.md +++ b/docs/bot-usage.md @@ -56,6 +56,7 @@ optional arguments: usage: freqtrade trade [-h] [-v] [--logfile FILE] [-V] [-c PATH] [-d PATH] [--userdir PATH] [-s NAME] [--strategy-path PATH] [--db-url PATH] [--sd-notify] [--dry-run] + [--dry-run-wallet DRY_RUN_WALLET] optional arguments: -h, --help show this help message and exit @@ -66,6 +67,9 @@ optional arguments: --sd-notify Notify systemd service manager. --dry-run Enforce dry-run for trading (removes Exchange secrets and simulates trades). + --dry-run-wallet DRY_RUN_WALLET + Starting balance, used for backtesting / hyperopt and + dry-runs. Common arguments: -v, --verbose Verbose mode (-vv for more, -vvv to get all messages). diff --git a/docs/configuration.md b/docs/configuration.md index 0163e1671..663d9c5b2 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -49,7 +49,7 @@ Mandatory parameters are marked as **Required**, which means that they are requi | `timeframe` | The timeframe (former ticker interval) to use (e.g `1m`, `5m`, `15m`, `30m`, `1h` ...). [Strategy Override](#parameters-in-the-strategy).
**Datatype:** String | `fiat_display_currency` | Fiat currency used to show your profits. [More information below](#what-values-can-be-used-for-fiat_display_currency).
**Datatype:** String | `dry_run` | **Required.** Define if the bot must be in Dry Run or production mode.
*Defaults to `true`.*
**Datatype:** Boolean -| `dry_run_wallet` | Define the starting amount in stake currency for the simulated wallet used by the bot running in the Dry Run mode.
*Defaults to `1000`.*
**Datatype:** Float +| `dry_run_wallet` | Define the starting amount in stake currency for the simulated wallet used by the bot running in Dry Run mode.
*Defaults to `1000`.*
**Datatype:** Float | `cancel_open_orders_on_exit` | Cancel open orders when the `/stop` RPC command is issued, `Ctrl+C` is pressed or the bot dies unexpectedly. When set to `true`, this allows you to use `/stop` to cancel unfilled and partially filled orders in the event of a market crash. It does not impact open positions.
*Defaults to `false`.*
**Datatype:** Boolean | `process_only_new_candles` | Enable processing of indicators only when new candles arrive. If false each loop populates the indicators, this will mean the same candle is processed many times creating system load but can be useful of your strategy depends on tick data not only candle. [Strategy Override](#parameters-in-the-strategy).
*Defaults to `false`.*
**Datatype:** Boolean | `minimal_roi` | **Required.** Set the threshold as ratio the bot will use to sell a trade. [More information below](#understand-minimal_roi). [Strategy Override](#parameters-in-the-strategy).
**Datatype:** Dict diff --git a/docs/hyperopt.md b/docs/hyperopt.md index ec155062f..ee3d75d0b 100644 --- a/docs/hyperopt.md +++ b/docs/hyperopt.md @@ -43,7 +43,8 @@ usage: freqtrade hyperopt [-h] [-v] [--logfile FILE] [-V] [-c PATH] [-d PATH] [--max-open-trades INT] [--stake-amount STAKE_AMOUNT] [--fee FLOAT] [--hyperopt NAME] [--hyperopt-path PATH] [--eps] - [--dmmp] [--enable-protections] [-e INT] + [--dmmp] [--enable-protections] + [--dry-run-wallet DRY_RUN_WALLET] [-e INT] [--spaces {all,buy,sell,roi,stoploss,trailing,default} [{all,buy,sell,roi,stoploss,trailing,default} ...]] [--print-all] [--no-color] [--print-json] [-j JOBS] [--random-state INT] [--min-trades INT] @@ -82,6 +83,9 @@ optional arguments: Enable protections for backtesting.Will slow backtesting down by a considerable amount, but will include configured protections + --dry-run-wallet DRY_RUN_WALLET + Starting balance, used for backtesting / hyperopt and + dry-runs. -e INT, --epochs INT Specify number of epochs (default: 100). --spaces {all,buy,sell,roi,stoploss,trailing,default} [{all,buy,sell,roi,stoploss,trailing,default} ...] Specify which parameters to hyperopt. Space-separated diff --git a/freqtrade/commands/arguments.py b/freqtrade/commands/arguments.py index c64c11a18..88cec7b3e 100644 --- a/freqtrade/commands/arguments.py +++ b/freqtrade/commands/arguments.py @@ -14,18 +14,18 @@ ARGS_COMMON = ["verbosity", "logfile", "version", "config", "datadir", "user_dat ARGS_STRATEGY = ["strategy", "strategy_path"] -ARGS_TRADE = ["db_url", "sd_notify", "dry_run"] +ARGS_TRADE = ["db_url", "sd_notify", "dry_run", "dry_run_wallet", ] ARGS_COMMON_OPTIMIZE = ["timeframe", "timerange", "dataformat_ohlcv", "max_open_trades", "stake_amount", "fee"] ARGS_BACKTEST = ARGS_COMMON_OPTIMIZE + ["position_stacking", "use_max_market_positions", - "enable_protections", + "enable_protections", "dry_run_wallet", "strategy_list", "export", "exportfilename"] ARGS_HYPEROPT = ARGS_COMMON_OPTIMIZE + ["hyperopt", "hyperopt_path", "position_stacking", "use_max_market_positions", - "enable_protections", + "enable_protections", "dry_run_wallet", "epochs", "spaces", "print_all", "print_colorized", "print_json", "hyperopt_jobs", "hyperopt_random_state", "hyperopt_min_trades", diff --git a/freqtrade/commands/cli_options.py b/freqtrade/commands/cli_options.py index 7dc85377d..90ebb5e6a 100644 --- a/freqtrade/commands/cli_options.py +++ b/freqtrade/commands/cli_options.py @@ -110,6 +110,11 @@ AVAILABLE_CLI_OPTIONS = { help='Enforce dry-run for trading (removes Exchange secrets and simulates trades).', action='store_true', ), + "dry_run_wallet": Arg( + '--dry-run-wallet', + help='Starting balance, used for backtesting / hyperopt and dry-runs.', + type=float, + ), # Optimize common "timeframe": Arg( '-i', '--timeframe', '--ticker-interval', diff --git a/freqtrade/configuration/configuration.py b/freqtrade/configuration/configuration.py index 7bf3e6bf2..6295d01d4 100644 --- a/freqtrade/configuration/configuration.py +++ b/freqtrade/configuration/configuration.py @@ -232,7 +232,9 @@ class Configuration: self._args_to_config(config, argname='stake_amount', logstring='Parameter --stake-amount detected, ' 'overriding stake_amount to: {} ...') - + self._args_to_config(config, argname='dry_run_wallet', + logstring='Parameter --dry-run-wallet detected, ' + 'overriding dry_run_wallet to: {} ...') self._args_to_config(config, argname='fee', logstring='Parameter --fee detected, ' 'setting fee to: {} ...') diff --git a/freqtrade/wallets.py b/freqtrade/wallets.py index 9562f34e6..f5ce4c102 100644 --- a/freqtrade/wallets.py +++ b/freqtrade/wallets.py @@ -158,6 +158,7 @@ class Wallets: Check if stake amount can be fulfilled with the available balance for the stake currency :return: float: Stake amount + :raise: DependencyException if balance is lower than stake-amount """ available_amount = self._get_available_stake_amount() From e4abe902fc924b30c41ddc67edb744eb2095951e Mon Sep 17 00:00:00 2001 From: Matthias Date: Wed, 10 Feb 2021 20:37:55 +0100 Subject: [PATCH 032/187] Enable compounding for backtesting --- freqtrade/optimize/backtesting.py | 66 ++++++++++++++------------ freqtrade/optimize/hyperopt.py | 1 - tests/optimize/test_backtest_detail.py | 1 - tests/optimize/test_backtesting.py | 6 --- 4 files changed, 36 insertions(+), 38 deletions(-) diff --git a/freqtrade/optimize/backtesting.py b/freqtrade/optimize/backtesting.py index 7f2ba60f2..7ed5064e7 100644 --- a/freqtrade/optimize/backtesting.py +++ b/freqtrade/optimize/backtesting.py @@ -17,7 +17,7 @@ from freqtrade.data import history from freqtrade.data.btanalysis import trade_list_to_dataframe from freqtrade.data.converter import trim_dataframe from freqtrade.data.dataprovider import DataProvider -from freqtrade.exceptions import OperationalException +from freqtrade.exceptions import DependencyException, OperationalException from freqtrade.exchange import timeframe_to_minutes, timeframe_to_seconds from freqtrade.mixins import LoggingMixin from freqtrade.optimize.optimize_reports import (generate_backtest_stats, show_backtest_results, @@ -270,6 +270,30 @@ class Backtesting: return None + def _enter_trade(self, pair: str, row, max_open_trades: int, + open_trade_count: int) -> Optional[Trade]: + self.update_wallets() + try: + stake_amount = self.wallets.get_trade_stake_amount( + pair, max_open_trades - open_trade_count, None) + except DependencyException: + stake_amount = 0 + if stake_amount: + # Enter trade + trade = Trade( + pair=pair, + open_rate=row[OPEN_IDX], + open_date=row[DATE_IDX], + stake_amount=stake_amount, + amount=round(stake_amount / row[OPEN_IDX], 8), + fee_open=self.fee, + fee_close=self.fee, + is_open=True, + exchange='backtesting', + ) + return trade + return None + def handle_left_open(self, open_trades: Dict[str, List[Trade]], data: Dict[str, List[Tuple]]) -> List[Trade]: """ @@ -290,7 +314,7 @@ class Backtesting: trades.append(trade1) return trades - def backtest(self, processed: Dict, stake_amount: float, + def backtest(self, processed: Dict, start_date: datetime, end_date: datetime, max_open_trades: int = 0, position_stacking: bool = False, enable_protections: bool = False) -> DataFrame: @@ -302,7 +326,6 @@ class Backtesting: Avoid extensive logging in this method and functions it calls. :param processed: a processed dictionary with format {pair, data} - :param stake_amount: amount to use for each trade :param start_date: backtesting timerange start datetime :param end_date: backtesting timerange end datetime :param max_open_trades: maximum number of concurrent trades, <= 0 means unlimited @@ -310,10 +333,6 @@ class Backtesting: :param enable_protections: Should protections be enabled? :return: DataFrame with trades (results of backtesting) """ - logger.debug(f"Run backtest, stake_amount: {stake_amount}, " - f"start_date: {start_date}, end_date: {end_date}, " - f"max_open_trades: {max_open_trades}, position_stacking: {position_stacking}" - ) trades: List[Trade] = [] self.prepare_backtest(enable_protections) @@ -356,30 +375,18 @@ class Backtesting: and tmp != end_date and row[BUY_IDX] == 1 and row[SELL_IDX] != 1 and not PairLocks.is_pair_locked(pair, row[DATE_IDX])): - self.update_wallets() - # Enter trade - trade = Trade( - pair=pair, - open_rate=row[OPEN_IDX], - open_date=row[DATE_IDX], - stake_amount=stake_amount, - amount=round(stake_amount / row[OPEN_IDX], 8), - fee_open=self.fee, - fee_close=self.fee, - is_open=True, - exchange='backtesting', - ) - # TODO: hacky workaround to avoid opening > max_open_trades - # This emulates previous behaviour - not sure if this is correct - # Prevents buying if the trade-slot was freed in this candle - open_trade_count_start += 1 - open_trade_count += 1 - # logger.debug(f"{pair} - Backtesting emulates creation of new trade: {trade}.") - open_trades[pair].append(trade) - Trade.trades.append(trade) + trade = self._enter_trade(pair, row, max_open_trades, open_trade_count_start) + if trade: + # TODO: hacky workaround to avoid opening > max_open_trades + # This emulates previous behaviour - not sure if this is correct + # Prevents buying if the trade-slot was freed in this candle + open_trade_count_start += 1 + open_trade_count += 1 + # logger.debug(f"{pair} - Backtesting emulates creation of new trade: {trade}.") + open_trades[pair].append(trade) + Trade.trades.append(trade) for trade in open_trades[pair]: - # since indexes has been incremented before, we need to go one step back to # also check the buying candle for sell conditions. trade_entry = self._get_sell_trade_entry(trade, row) # Sell occured @@ -431,7 +438,6 @@ class Backtesting: # Execute backtest and store results results = self.backtest( processed=preprocessed, - stake_amount=self.config['stake_amount'], start_date=min_date.datetime, end_date=max_date.datetime, max_open_trades=max_open_trades, diff --git a/freqtrade/optimize/hyperopt.py b/freqtrade/optimize/hyperopt.py index 155f1e69b..79ecb6052 100644 --- a/freqtrade/optimize/hyperopt.py +++ b/freqtrade/optimize/hyperopt.py @@ -537,7 +537,6 @@ class Hyperopt: backtesting_results = self.backtesting.backtest( processed=processed, - stake_amount=self.config['stake_amount'], start_date=min_date.datetime, end_date=max_date.datetime, max_open_trades=self.max_open_trades, diff --git a/tests/optimize/test_backtest_detail.py b/tests/optimize/test_backtest_detail.py index c9499cc42..4d6605b9f 100644 --- a/tests/optimize/test_backtest_detail.py +++ b/tests/optimize/test_backtest_detail.py @@ -503,7 +503,6 @@ def test_backtest_results(default_conf, fee, mocker, caplog, data) -> None: min_date, max_date = get_timerange({pair: frame}) results = backtesting.backtest( processed=data_processed, - stake_amount=default_conf['stake_amount'], start_date=min_date, end_date=max_date, max_open_trades=10, diff --git a/tests/optimize/test_backtesting.py b/tests/optimize/test_backtesting.py index db14749c3..620bd1df5 100644 --- a/tests/optimize/test_backtesting.py +++ b/tests/optimize/test_backtesting.py @@ -90,7 +90,6 @@ def simple_backtest(config, contour, mocker, testdatadir) -> None: assert isinstance(processed, dict) results = backtesting.backtest( processed=processed, - stake_amount=config['stake_amount'], start_date=min_date, end_date=max_date, max_open_trades=1, @@ -111,7 +110,6 @@ def _make_backtest_conf(mocker, datadir, conf=None, pair='UNITTEST/BTC'): min_date, max_date = get_timerange(processed) return { 'processed': processed, - 'stake_amount': conf['stake_amount'], 'start_date': min_date, 'end_date': max_date, 'max_open_trades': 10, @@ -461,7 +459,6 @@ def test_backtest_one(default_conf, fee, mocker, testdatadir) -> None: min_date, max_date = get_timerange(processed) results = backtesting.backtest( processed=processed, - stake_amount=default_conf['stake_amount'], start_date=min_date, end_date=max_date, max_open_trades=10, @@ -523,7 +520,6 @@ def test_backtest_1min_timeframe(default_conf, fee, mocker, testdatadir) -> None min_date, max_date = get_timerange(processed) results = backtesting.backtest( processed=processed, - stake_amount=default_conf['stake_amount'], start_date=min_date, end_date=max_date, max_open_trades=1, @@ -678,7 +674,6 @@ def test_backtest_multi_pair(default_conf, fee, mocker, tres, pair, testdatadir) min_date, max_date = get_timerange(processed) backtest_conf = { 'processed': processed, - 'stake_amount': default_conf['stake_amount'], 'start_date': min_date, 'end_date': max_date, 'max_open_trades': 3, @@ -694,7 +689,6 @@ def test_backtest_multi_pair(default_conf, fee, mocker, tres, pair, testdatadir) backtest_conf = { 'processed': processed, - 'stake_amount': default_conf['stake_amount'], 'start_date': min_date, 'end_date': max_date, 'max_open_trades': 1, From 8d61a263823943cdbdb911d2c40bc283ba415903 Mon Sep 17 00:00:00 2001 From: Matthias Date: Fri, 12 Feb 2021 20:20:32 +0100 Subject: [PATCH 033/187] Allow dynamic stake for backtesting and hyperopt --- freqtrade/commands/optimize_commands.py | 14 +++++++++----- freqtrade/optimize/backtesting.py | 2 +- tests/optimize/test_backtesting.py | 7 ++++--- tests/optimize/test_hyperopt.py | 8 ++++---- 4 files changed, 18 insertions(+), 13 deletions(-) diff --git a/freqtrade/commands/optimize_commands.py b/freqtrade/commands/optimize_commands.py index 7411ca9c6..bf36972c4 100644 --- a/freqtrade/commands/optimize_commands.py +++ b/freqtrade/commands/optimize_commands.py @@ -3,7 +3,7 @@ from typing import Any, Dict from freqtrade import constants from freqtrade.configuration import setup_utils_configuration -from freqtrade.exceptions import DependencyException, OperationalException +from freqtrade.exceptions import OperationalException from freqtrade.state import RunMode @@ -23,10 +23,14 @@ def setup_optimize_configuration(args: Dict[str, Any], method: RunMode) -> Dict[ RunMode.HYPEROPT: 'hyperoptimization', } if (method in no_unlimited_runmodes.keys() and - config['stake_amount'] == constants.UNLIMITED_STAKE_AMOUNT): - raise DependencyException( - f'The value of `stake_amount` cannot be set as "{constants.UNLIMITED_STAKE_AMOUNT}" ' - f'for {no_unlimited_runmodes[method]}') + config['stake_amount'] != constants.UNLIMITED_STAKE_AMOUNT and + config['max_open_trades'] != float('inf')): + pass + # config['dry_run_wallet'] = config['stake_amount'] * \ + # config['max_open_trades'] * (2 - config['tradable_balance_ratio']) + + # logger.warning(f"Changing dry-run-wallet to {config['dry_run_wallet']} " + # "(max_open_trades * stake_amount).") return config diff --git a/freqtrade/optimize/backtesting.py b/freqtrade/optimize/backtesting.py index 7ed5064e7..29559126b 100644 --- a/freqtrade/optimize/backtesting.py +++ b/freqtrade/optimize/backtesting.py @@ -445,11 +445,11 @@ class Backtesting: enable_protections=self.config.get('enable_protections', False), ) backtest_end_time = datetime.now(timezone.utc) - print(self.wallets.get_all_balances()) self.all_results[self.strategy.get_strategy_name()] = { 'results': results, 'config': self.strategy.config, 'locks': PairLocks.get_all_locks(), + 'final_balance': self.wallets.get_total(self.strategy.config['stake_currency']), 'backtest_start_time': int(backtest_start_time.timestamp()), 'backtest_end_time': int(backtest_end_time.timestamp()), } diff --git a/tests/optimize/test_backtesting.py b/tests/optimize/test_backtesting.py index 620bd1df5..061bcbaa0 100644 --- a/tests/optimize/test_backtesting.py +++ b/tests/optimize/test_backtesting.py @@ -17,7 +17,7 @@ from freqtrade.data.btanalysis import BT_DATA_COLUMNS, evaluate_result_multi from freqtrade.data.converter import clean_ohlcv_dataframe from freqtrade.data.dataprovider import DataProvider from freqtrade.data.history import get_timerange -from freqtrade.exceptions import DependencyException, OperationalException +from freqtrade.exceptions import OperationalException from freqtrade.optimize.backtesting import Backtesting from freqtrade.resolvers import StrategyResolver from freqtrade.state import RunMode @@ -242,8 +242,9 @@ def test_setup_optimize_configuration_unlimited_stake_amount(mocker, default_con '--strategy', 'DefaultStrategy', ] - with pytest.raises(DependencyException, match=r'.`stake_amount`.*'): - setup_optimize_configuration(get_args(args), RunMode.BACKTEST) + # TODO: does this test still make sense? + conf = setup_optimize_configuration(get_args(args), RunMode.BACKTEST) + assert isinstance(conf, dict) def test_start(mocker, fee, default_conf, caplog) -> None: diff --git a/tests/optimize/test_hyperopt.py b/tests/optimize/test_hyperopt.py index 68eb3d6f7..88a4cea2d 100644 --- a/tests/optimize/test_hyperopt.py +++ b/tests/optimize/test_hyperopt.py @@ -15,7 +15,7 @@ from filelock import Timeout from freqtrade import constants from freqtrade.commands.optimize_commands import setup_optimize_configuration, start_hyperopt from freqtrade.data.history import load_data -from freqtrade.exceptions import DependencyException, OperationalException +from freqtrade.exceptions import OperationalException from freqtrade.optimize.hyperopt import Hyperopt from freqtrade.resolvers.hyperopt_resolver import HyperOptResolver from freqtrade.state import RunMode @@ -140,9 +140,9 @@ def test_setup_hyperopt_configuration_unlimited_stake_amount(mocker, default_con '--config', 'config.json', '--hyperopt', 'DefaultHyperOpt', ] - - with pytest.raises(DependencyException, match=r'.`stake_amount`.*'): - setup_optimize_configuration(get_args(args), RunMode.HYPEROPT) + # TODO: does this test still make sense? + conf = setup_optimize_configuration(get_args(args), RunMode.HYPEROPT) + assert isinstance(conf, dict) def test_hyperoptresolver(mocker, default_conf, caplog) -> None: From 35e6a9ab3aa70cad59d8cc75b50bedecdce56e0a Mon Sep 17 00:00:00 2001 From: Matthias Date: Sat, 13 Feb 2021 09:01:05 +0100 Subject: [PATCH 034/187] Backtest-reports should calculate total gains based on starting capital --- docs/backtesting.md | 18 +++++++++++--- freqtrade/optimize/optimize_reports.py | 32 +++++++++++++++++-------- tests/conftest.py | 1 + tests/optimize/test_optimize_reports.py | 10 ++++---- 4 files changed, 44 insertions(+), 17 deletions(-) diff --git a/docs/backtesting.md b/docs/backtesting.md index 38d1af45a..eab64a7a9 100644 --- a/docs/backtesting.md +++ b/docs/backtesting.md @@ -252,7 +252,10 @@ A backtesting result will look like that: | Max open trades | 3 | | | | | Total trades | 429 | -| Total Profit % | 152.41% | +| Starting capital | 0.01000000 BTC | +| End capital | 0.01762792 BTC | +| Absolute profit | 0.00762792 BTC | +| Total Profit % | 76.2% | | Trades per day | 3.575 | | | | | Best Pair | LSK/BTC 26.26% | @@ -261,6 +264,7 @@ A backtesting result will look like that: | Worst Trade | ZEC/BTC -10.25% | | Best day | 25.27% | | Worst day | -30.67% | +| Days win/draw/lose | 12 / 82 / 25 | | Avg. Duration Winners | 4:23:00 | | Avg. Duration Loser | 6:55:00 | | | | @@ -328,7 +332,10 @@ It contains some useful key metrics about performance of your strategy on backte | Max open trades | 3 | | | | | Total trades | 429 | -| Total Profit % | 152.41% | +| Starting capital | 0.01000000 BTC | +| End capital | 0.01762792 BTC | +| Absolute profit | 0.00762792 BTC | +| Total Profit % | 76.2% | | Trades per day | 3.575 | | | | | Best Pair | LSK/BTC 26.26% | @@ -337,6 +344,7 @@ It contains some useful key metrics about performance of your strategy on backte | Worst Trade | ZEC/BTC -10.25% | | Best day | 25.27% | | Worst day | -30.67% | +| Days win/draw/lose | 12 / 82 / 25 | | Avg. Duration Winners | 4:23:00 | | Avg. Duration Loser | 6:55:00 | | | | @@ -351,11 +359,15 @@ It contains some useful key metrics about performance of your strategy on backte - `Backtesting from` / `Backtesting to`: Backtesting range (usually defined with the `--timerange` option). - `Max open trades`: Setting of `max_open_trades` (or `--max-open-trades`) - or number of pairs in the pairlist (whatever is lower). - `Total trades`: Identical to the total trades of the backtest output table. -- `Total Profit %`: Total profit. Aligned to the `TOTAL` row's `Tot Profit %` from the first table. +- `Starting capital`: Start capital - as given by dry-run-wallet (config or command line). +- `End capital`: Final capital - starting capital + absolute profit. +- `Absolute profit`: Profit made in stake currency. +- `Total Profit %`: Total profit. Aligned to the `TOTAL` row's `Tot Profit %` from the first table. Calculated as `(End capital − Starting capital) / Starting capital`. - `Trades per day`: Total trades divided by the backtesting duration in days (this will give you information about how many trades to expect from the strategy). - `Best Pair` / `Worst Pair`: Best and worst performing pair, and it's corresponding `Cum Profit %`. - `Best Trade` / `Worst Trade`: Biggest winning trade and biggest losing trade - `Best day` / `Worst day`: Best and worst day based on daily profit. +- `Days win/draw/lose`: Winning / Losing days (draws are usually days without closed trade). - `Avg. Duration Winners` / `Avg. Duration Loser`: Average durations for winning and losing trades. - `Max Drawdown`: Maximum drawdown experienced. For example, the value of 50% means that from highest to subsequent lowest point, a 50% drop was experienced). - `Drawdown Start` / `Drawdown End`: Start and end datetime for this largest drawdown (can also be visualized via the `plot-dataframe` sub-command). diff --git a/freqtrade/optimize/optimize_reports.py b/freqtrade/optimize/optimize_reports.py index 6338b1d71..d6adfdf50 100644 --- a/freqtrade/optimize/optimize_reports.py +++ b/freqtrade/optimize/optimize_reports.py @@ -56,12 +56,13 @@ def _get_line_header(first_column: str, stake_currency: str) -> List[str]: 'Wins', 'Draws', 'Losses'] -def _generate_result_line(result: DataFrame, max_open_trades: int, first_column: str) -> Dict: +def _generate_result_line(result: DataFrame, starting_balance: int, first_column: str) -> Dict: """ Generate one result dict, with "first_column" as key. """ profit_sum = result['profit_ratio'].sum() - profit_total = profit_sum / max_open_trades + # (end-capital - starting capital) / starting capital + profit_total = result['profit_abs'].sum() / starting_balance return { 'key': first_column, @@ -88,13 +89,13 @@ def _generate_result_line(result: DataFrame, max_open_trades: int, first_column: } -def generate_pair_metrics(data: Dict[str, Dict], stake_currency: str, max_open_trades: int, +def generate_pair_metrics(data: Dict[str, Dict], stake_currency: str, starting_balance: int, results: DataFrame, skip_nan: bool = False) -> List[Dict]: """ Generates and returns a list for the given backtest data and the results dataframe :param data: Dict of containing data that was used during backtesting. :param stake_currency: stake-currency - used to correctly name headers - :param max_open_trades: Maximum allowed open trades + :param starting_balance: Starting balance :param results: Dataframe containing the backtest results :param skip_nan: Print "left open" open trades :return: List of Dicts containing the metrics per pair @@ -107,10 +108,10 @@ def generate_pair_metrics(data: Dict[str, Dict], stake_currency: str, max_open_t if skip_nan and result['profit_abs'].isnull().all(): continue - tabular_data.append(_generate_result_line(result, max_open_trades, pair)) + tabular_data.append(_generate_result_line(result, starting_balance, pair)) # Append Total - tabular_data.append(_generate_result_line(results, max_open_trades, 'TOTAL')) + tabular_data.append(_generate_result_line(results, starting_balance, 'TOTAL')) return tabular_data @@ -159,7 +160,7 @@ def generate_strategy_metrics(all_results: Dict) -> List[Dict]: tabular_data = [] for strategy, results in all_results.items(): tabular_data.append(_generate_result_line( - results['results'], results['config']['max_open_trades'], strategy) + results['results'], results['config']['dry_run_wallet'], strategy) ) return tabular_data @@ -246,15 +247,16 @@ def generate_backtest_stats(btdata: Dict[str, DataFrame], continue config = content['config'] max_open_trades = min(config['max_open_trades'], len(btdata.keys())) + starting_balance = config['dry_run_wallet'] stake_currency = config['stake_currency'] pair_results = generate_pair_metrics(btdata, stake_currency=stake_currency, - max_open_trades=max_open_trades, + starting_balance=starting_balance, results=results, skip_nan=False) sell_reason_stats = generate_sell_reason_stats(max_open_trades=max_open_trades, results=results) left_open_results = generate_pair_metrics(btdata, stake_currency=stake_currency, - max_open_trades=max_open_trades, + starting_balance=starting_balance, results=results.loc[results['is_open']], skip_nan=True) daily_stats = generate_daily_stats(results) @@ -276,7 +278,7 @@ def generate_backtest_stats(btdata: Dict[str, DataFrame], 'left_open_trades': left_open_results, 'total_trades': len(results), 'profit_mean': results['profit_ratio'].mean() if len(results) > 0 else 0, - 'profit_total': results['profit_ratio'].sum() / max_open_trades, + 'profit_total': results['profit_abs'].sum() / starting_balance, 'profit_total_abs': results['profit_abs'].sum(), 'backtest_start': min_date.datetime, 'backtest_start_ts': min_date.int_timestamp * 1000, @@ -292,6 +294,9 @@ def generate_backtest_stats(btdata: Dict[str, DataFrame], 'pairlist': list(btdata.keys()), 'stake_amount': config['stake_amount'], 'stake_currency': config['stake_currency'], + 'starting_balance': starting_balance, + 'dry_run_wallet': starting_balance, + 'final_balance': content['final_balance'], 'max_open_trades': max_open_trades, 'max_open_trades_setting': (config['max_open_trades'] if config['max_open_trades'] != float('inf') else -1), @@ -431,6 +436,13 @@ def text_table_add_metrics(strat_results: Dict) -> str: ('Max open trades', strat_results['max_open_trades']), ('', ''), # Empty line to improve readability ('Total trades', strat_results['total_trades']), + ('Starting capital', round_coin_value(strat_results['starting_balance'], + strat_results['stake_currency'])), + ('End capital', round_coin_value(strat_results['final_balance'], + strat_results['stake_currency'])), + ('Absolute profit ', round_coin_value(strat_results['profit_total_abs'], + strat_results['stake_currency'])), + ('Total Profit %', f"{round(strat_results['profit_total'] * 100, 2)}%"), ('Trades per day', strat_results['trades_per_day']), ('', ''), # Empty line to improve readability diff --git a/tests/conftest.py b/tests/conftest.py index 946ae1fb5..6e70603b1 100644 --- a/tests/conftest.py +++ b/tests/conftest.py @@ -261,6 +261,7 @@ def get_default_conf(testdatadir): "20": 0.02, "0": 0.04 }, + "dry_run_wallet": 1000, "stoploss": -0.10, "unfilledtimeout": { "buy": 10, diff --git a/tests/optimize/test_optimize_reports.py b/tests/optimize/test_optimize_reports.py index 8b64c2764..405cc599b 100644 --- a/tests/optimize/test_optimize_reports.py +++ b/tests/optimize/test_optimize_reports.py @@ -48,7 +48,7 @@ def test_text_table_bt_results(): ) pair_results = generate_pair_metrics(data={'ETH/BTC': {}}, stake_currency='BTC', - max_open_trades=2, results=results) + starting_balance=4, results=results) assert text_table_bt_results(pair_results, stake_currency='BTC') == result_str @@ -78,6 +78,7 @@ def test_generate_backtest_stats(default_conf, testdatadir): }), 'config': default_conf, 'locks': [], + 'final_balance': 1000.02, 'backtest_start_time': Arrow.utcnow().int_timestamp, 'backtest_end_time': Arrow.utcnow().int_timestamp, } @@ -189,7 +190,7 @@ def test_generate_pair_metrics(): ) pair_results = generate_pair_metrics(data={'ETH/BTC': {}}, stake_currency='BTC', - max_open_trades=2, results=results) + starting_balance=2, results=results) assert isinstance(pair_results, list) assert len(pair_results) == 2 assert pair_results[-1]['key'] == 'TOTAL' @@ -291,6 +292,7 @@ def test_generate_sell_reason_stats(): def test_text_table_strategy(default_conf): default_conf['max_open_trades'] = 2 + default_conf['dry_run_wallet'] = 3 results = {} results['TestStrategy1'] = {'results': pd.DataFrame( { @@ -323,9 +325,9 @@ def test_text_table_strategy(default_conf): '|---------------+--------+----------------+----------------+------------------+' '----------------+----------------+--------+---------+----------|\n' '| TestStrategy1 | 3 | 20.00 | 60.00 | 1.10000000 |' - ' 30.00 | 0:17:00 | 3 | 0 | 0 |\n' + ' 36.67 | 0:17:00 | 3 | 0 | 0 |\n' '| TestStrategy2 | 3 | 30.00 | 90.00 | 1.30000000 |' - ' 45.00 | 0:20:00 | 3 | 0 | 0 |' + ' 43.33 | 0:20:00 | 3 | 0 | 0 |' ) strategy_results = generate_strategy_metrics(all_results=results) From 72f21fc5ec90ae15f622c362d2a24bd31f068613 Mon Sep 17 00:00:00 2001 From: Matthias Date: Sun, 14 Feb 2021 13:08:49 +0100 Subject: [PATCH 035/187] Add trade-volume metric --- docs/backtesting.md | 3 +++ freqtrade/optimize/optimize_reports.py | 5 ++++- tests/optimize/test_backtesting.py | 2 ++ tests/optimize/test_optimize_reports.py | 1 + 4 files changed, 10 insertions(+), 1 deletion(-) diff --git a/docs/backtesting.md b/docs/backtesting.md index eab64a7a9..ada788da9 100644 --- a/docs/backtesting.md +++ b/docs/backtesting.md @@ -257,6 +257,7 @@ A backtesting result will look like that: | Absolute profit | 0.00762792 BTC | | Total Profit % | 76.2% | | Trades per day | 3.575 | +| Total trade volume | 0.429 BTC | | | | | Best Pair | LSK/BTC 26.26% | | Worst Pair | ZEC/BTC -10.18% | @@ -337,6 +338,7 @@ It contains some useful key metrics about performance of your strategy on backte | Absolute profit | 0.00762792 BTC | | Total Profit % | 76.2% | | Trades per day | 3.575 | +| Total trade volume | 0.429 BTC | | | | | Best Pair | LSK/BTC 26.26% | | Worst Pair | ZEC/BTC -10.18% | @@ -364,6 +366,7 @@ It contains some useful key metrics about performance of your strategy on backte - `Absolute profit`: Profit made in stake currency. - `Total Profit %`: Total profit. Aligned to the `TOTAL` row's `Tot Profit %` from the first table. Calculated as `(End capital − Starting capital) / Starting capital`. - `Trades per day`: Total trades divided by the backtesting duration in days (this will give you information about how many trades to expect from the strategy). +- `Total trade volume`: Volume generated on the exchange to reach the above profit. - `Best Pair` / `Worst Pair`: Best and worst performing pair, and it's corresponding `Cum Profit %`. - `Best Trade` / `Worst Trade`: Biggest winning trade and biggest losing trade - `Best day` / `Worst day`: Best and worst day based on daily profit. diff --git a/freqtrade/optimize/optimize_reports.py b/freqtrade/optimize/optimize_reports.py index d6adfdf50..dde0f8dd2 100644 --- a/freqtrade/optimize/optimize_reports.py +++ b/freqtrade/optimize/optimize_reports.py @@ -277,6 +277,7 @@ def generate_backtest_stats(btdata: Dict[str, DataFrame], 'sell_reason_summary': sell_reason_stats, 'left_open_trades': left_open_results, 'total_trades': len(results), + 'total_volume': results['stake_amount'].sum(), 'profit_mean': results['profit_ratio'].mean() if len(results) > 0 else 0, 'profit_total': results['profit_abs'].sum() / starting_balance, 'profit_total_abs': results['profit_abs'].sum(), @@ -442,9 +443,11 @@ def text_table_add_metrics(strat_results: Dict) -> str: strat_results['stake_currency'])), ('Absolute profit ', round_coin_value(strat_results['profit_total_abs'], strat_results['stake_currency'])), - ('Total Profit %', f"{round(strat_results['profit_total'] * 100, 2)}%"), ('Trades per day', strat_results['trades_per_day']), + ('Total trade volume', round_coin_value(strat_results['total_volume'], + strat_results['stake_currency'])), + ('', ''), # Empty line to improve readability ('Best Pair', f"{strat_results['best_pair']['key']} " f"{round(strat_results['best_pair']['profit_sum_pct'], 2)}%"), diff --git a/tests/optimize/test_backtesting.py b/tests/optimize/test_backtesting.py index 061bcbaa0..8fba8724b 100644 --- a/tests/optimize/test_backtesting.py +++ b/tests/optimize/test_backtesting.py @@ -817,6 +817,7 @@ def test_backtest_start_multi_strat_nomock(default_conf, mocker, caplog, testdat '2018-01-30 05:35:00', ], utc=True), 'trade_duration': [235, 40], 'is_open': [False, False], + 'stake_amount': [0.01, 0.01], 'open_rate': [0.104445, 0.10302485], 'close_rate': [0.104969, 0.103541], 'sell_reason': [SellType.ROI, SellType.ROI] @@ -833,6 +834,7 @@ def test_backtest_start_multi_strat_nomock(default_conf, mocker, caplog, testdat '2018-01-30 08:30:00'], utc=True), 'trade_duration': [47, 40, 20], 'is_open': [False, False, False], + 'stake_amount': [0.01, 0.01, 0.01], 'open_rate': [0.104445, 0.10302485, 0.122541], 'close_rate': [0.104969, 0.103541, 0.123541], 'sell_reason': [SellType.ROI, SellType.ROI, SellType.STOP_LOSS] diff --git a/tests/optimize/test_optimize_reports.py b/tests/optimize/test_optimize_reports.py index 405cc599b..ca6a4ab01 100644 --- a/tests/optimize/test_optimize_reports.py +++ b/tests/optimize/test_optimize_reports.py @@ -73,6 +73,7 @@ def test_generate_backtest_stats(default_conf, testdatadir): "close_rate": [0.002546, 0.003014, 0.003103, 0.003217], "trade_duration": [123, 34, 31, 14], "is_open": [False, False, False, True], + "stake_amount": [0.01, 0.01, 0.01, 0.01], "sell_reason": [SellType.ROI, SellType.STOP_LOSS, SellType.ROI, SellType.FORCE_SELL] }), From 74fc4bdab5e108c72015f80112df8ff7aa12bdcd Mon Sep 17 00:00:00 2001 From: Matthias Date: Tue, 16 Feb 2021 07:56:35 +0100 Subject: [PATCH 036/187] Shorten debug log --- freqtrade/optimize/backtesting.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/freqtrade/optimize/backtesting.py b/freqtrade/optimize/backtesting.py index 29559126b..c60cfa9b7 100644 --- a/freqtrade/optimize/backtesting.py +++ b/freqtrade/optimize/backtesting.py @@ -382,7 +382,7 @@ class Backtesting: # Prevents buying if the trade-slot was freed in this candle open_trade_count_start += 1 open_trade_count += 1 - # logger.debug(f"{pair} - Backtesting emulates creation of new trade: {trade}.") + # logger.debug(f"{pair} - Emulate creation of new trade: {trade}.") open_trades[pair].append(trade) Trade.trades.append(trade) From 0d2f877e77b17dc87c4efede8097dc90fd6202a1 Mon Sep 17 00:00:00 2001 From: Matthias Date: Sun, 14 Feb 2021 19:30:17 +0100 Subject: [PATCH 037/187] Use absolute drawdown calc --- freqtrade/data/btanalysis.py | 10 ++++++--- freqtrade/optimize/optimize_reports.py | 17 ++++++++++++++- freqtrade/plot/plotting.py | 2 +- .../protections/max_drawdown_protection.py | 2 +- tests/data/test_btanalysis.py | 21 ++++++++++++------- 5 files changed, 38 insertions(+), 14 deletions(-) diff --git a/freqtrade/data/btanalysis.py b/freqtrade/data/btanalysis.py index 8e851a8e8..117278585 100644 --- a/freqtrade/data/btanalysis.py +++ b/freqtrade/data/btanalysis.py @@ -360,13 +360,14 @@ def create_cum_profit(df: pd.DataFrame, trades: pd.DataFrame, col_name: str, def calculate_max_drawdown(trades: pd.DataFrame, *, date_col: str = 'close_date', value_col: str = 'profit_ratio' - ) -> Tuple[float, pd.Timestamp, pd.Timestamp]: + ) -> Tuple[float, pd.Timestamp, pd.Timestamp, float, float]: """ Calculate max drawdown and the corresponding close dates :param trades: DataFrame containing trades (requires columns close_date and profit_ratio) :param date_col: Column in DataFrame to use for dates (defaults to 'close_date') :param value_col: Column in DataFrame to use for values (defaults to 'profit_ratio') - :return: Tuple (float, highdate, lowdate) with absolute max drawdown, high and low time + :return: Tuple (float, highdate, lowdate, highvalue, lowvalue) with absolute max drawdown, + high and low time and high and low value. :raise: ValueError if trade-dataframe was found empty. """ if len(trades) == 0: @@ -382,7 +383,10 @@ def calculate_max_drawdown(trades: pd.DataFrame, *, date_col: str = 'close_date' raise ValueError("No losing trade, therefore no drawdown.") high_date = profit_results.loc[max_drawdown_df.iloc[:idxmin]['high_value'].idxmax(), date_col] low_date = profit_results.loc[idxmin, date_col] - return abs(min(max_drawdown_df['drawdown'])), high_date, low_date + high_val = max_drawdown_df.loc[max_drawdown_df.iloc[:idxmin] + ['high_value'].idxmax(), 'cumulative'] + low_val = max_drawdown_df.loc[idxmin, 'cumulative'] + return abs(min(max_drawdown_df['drawdown'])), high_date, low_date, high_val, low_val def calculate_csum(trades: pd.DataFrame) -> Tuple[float, float]: diff --git a/freqtrade/optimize/optimize_reports.py b/freqtrade/optimize/optimize_reports.py index dde0f8dd2..5b3f813f2 100644 --- a/freqtrade/optimize/optimize_reports.py +++ b/freqtrade/optimize/optimize_reports.py @@ -322,14 +322,20 @@ def generate_backtest_stats(btdata: Dict[str, DataFrame], result['strategy'][strategy] = strat_stats try: - max_drawdown, drawdown_start, drawdown_end = calculate_max_drawdown( + max_drawdown, _, _, _, _ = calculate_max_drawdown( results, value_col='profit_ratio') + drawdown_abs, drawdown_start, drawdown_end, high_val, low_val = calculate_max_drawdown( + results, value_col='profit_abs') strat_stats.update({ 'max_drawdown': max_drawdown, + 'max_drawdown_abs': drawdown_abs, 'drawdown_start': drawdown_start, 'drawdown_start_ts': drawdown_start.timestamp() * 1000, 'drawdown_end': drawdown_end, 'drawdown_end_ts': drawdown_end.timestamp() * 1000, + + 'max_drawdown_low': low_val, + 'max_drawdown_high': high_val, }) csum_min, csum_max = calculate_csum(results) @@ -341,6 +347,9 @@ def generate_backtest_stats(btdata: Dict[str, DataFrame], except ValueError: strat_stats.update({ 'max_drawdown': 0.0, + 'max_drawdown_abs': 0.0, + 'max_drawdown_low': 0.0, + 'max_drawdown_high': 0.0, 'drawdown_start': datetime(1970, 1, 1, tzinfo=timezone.utc), 'drawdown_start_ts': 0, 'drawdown_end': datetime(1970, 1, 1, tzinfo=timezone.utc), @@ -471,6 +480,12 @@ def text_table_add_metrics(strat_results: Dict) -> str: strat_results['stake_currency'])), ('Max Drawdown', f"{round(strat_results['max_drawdown'] * 100, 2)}%"), + ('Max Drawdown', round_coin_value(strat_results['max_drawdown_abs'], + strat_results['stake_currency'])), + ('Max Drawdown high', round_coin_value(strat_results['max_drawdown_high'], + strat_results['stake_currency'])), + ('Max Drawdown low', round_coin_value(strat_results['max_drawdown_low'], + strat_results['stake_currency'])), ('Drawdown Start', strat_results['drawdown_start'].strftime(DATETIME_PRINT_FORMAT)), ('Drawdown End', strat_results['drawdown_end'].strftime(DATETIME_PRINT_FORMAT)), ('Market change', f"{round(strat_results['market_change'] * 100, 2)}%"), diff --git a/freqtrade/plot/plotting.py b/freqtrade/plot/plotting.py index 4325e537e..682c2b018 100644 --- a/freqtrade/plot/plotting.py +++ b/freqtrade/plot/plotting.py @@ -145,7 +145,7 @@ def add_max_drawdown(fig, row, trades: pd.DataFrame, df_comb: pd.DataFrame, Add scatter points indicating max drawdown """ try: - max_drawdown, highdate, lowdate = calculate_max_drawdown(trades) + max_drawdown, highdate, lowdate, _, _ = calculate_max_drawdown(trades) drawdown = go.Scatter( x=[highdate, lowdate], diff --git a/freqtrade/plugins/protections/max_drawdown_protection.py b/freqtrade/plugins/protections/max_drawdown_protection.py index d54e6699b..d1c6b192d 100644 --- a/freqtrade/plugins/protections/max_drawdown_protection.py +++ b/freqtrade/plugins/protections/max_drawdown_protection.py @@ -55,7 +55,7 @@ class MaxDrawdown(IProtection): # Drawdown is always positive try: - drawdown, _, _ = calculate_max_drawdown(trades_df, value_col='close_profit') + drawdown, _, _, _, _ = calculate_max_drawdown(trades_df, value_col='close_profit') except ValueError: return False, None, None diff --git a/tests/data/test_btanalysis.py b/tests/data/test_btanalysis.py index 3c4687745..555808679 100644 --- a/tests/data/test_btanalysis.py +++ b/tests/data/test_btanalysis.py @@ -274,15 +274,17 @@ def test_create_cum_profit1(testdatadir): def test_calculate_max_drawdown(testdatadir): filename = testdatadir / "backtest-result_test.json" bt_data = load_backtest_data(filename) - drawdown, h, low = calculate_max_drawdown(bt_data) + drawdown, hdate, lowdate, hval, lval = calculate_max_drawdown(bt_data) assert isinstance(drawdown, float) assert pytest.approx(drawdown) == 0.21142322 - assert isinstance(h, Timestamp) - assert isinstance(low, Timestamp) - assert h == Timestamp('2018-01-24 14:25:00', tz='UTC') - assert low == Timestamp('2018-01-30 04:45:00', tz='UTC') + assert isinstance(hdate, Timestamp) + assert isinstance(lowdate, Timestamp) + assert isinstance(hval, float) + assert isinstance(lval, float) + assert hdate == Timestamp('2018-01-24 14:25:00', tz='UTC') + assert lowdate == Timestamp('2018-01-30 04:45:00', tz='UTC') with pytest.raises(ValueError, match='Trade dataframe empty.'): - drawdown, h, low = calculate_max_drawdown(DataFrame()) + drawdown, hdate, lowdate, hval, lval = calculate_max_drawdown(DataFrame()) def test_calculate_csum(testdatadir): @@ -310,13 +312,16 @@ def test_calculate_max_drawdown2(): # sort by profit and reset index df = df.sort_values('profit').reset_index(drop=True) df1 = df.copy() - drawdown, h, low = calculate_max_drawdown(df, date_col='open_date', value_col='profit') + drawdown, hdate, ldate, hval, lval = calculate_max_drawdown( + df, date_col='open_date', value_col='profit') # Ensure df has not been altered. assert df.equals(df1) assert isinstance(drawdown, float) # High must be before low - assert h < low + assert hdate < ldate + # High value must be higher than low value + assert hval > lval assert drawdown == 0.091755 df = DataFrame(zip(values[:5], dates[:5]), columns=['profit', 'open_date']) From aed23d55c280806af19fe9c7927fe1088ff1e0b7 Mon Sep 17 00:00:00 2001 From: Matthias Date: Tue, 16 Feb 2021 20:12:59 +0100 Subject: [PATCH 038/187] Add starting balance to profit cumsum calculation --- freqtrade/data/btanalysis.py | 7 ++++--- tests/data/test_btanalysis.py | 5 +++++ 2 files changed, 9 insertions(+), 3 deletions(-) diff --git a/freqtrade/data/btanalysis.py b/freqtrade/data/btanalysis.py index 117278585..3adee8775 100644 --- a/freqtrade/data/btanalysis.py +++ b/freqtrade/data/btanalysis.py @@ -389,10 +389,11 @@ def calculate_max_drawdown(trades: pd.DataFrame, *, date_col: str = 'close_date' return abs(min(max_drawdown_df['drawdown'])), high_date, low_date, high_val, low_val -def calculate_csum(trades: pd.DataFrame) -> Tuple[float, float]: +def calculate_csum(trades: pd.DataFrame, starting_balance: float = 0) -> Tuple[float, float]: """ Calculate min/max cumsum of trades, to show if the wallet/stake amount ratio is sane :param trades: DataFrame containing trades (requires columns close_date and profit_percent) + :param starting_balance: Add starting balance to results, to show the wallets high / low points :return: Tuple (float, float) with cumsum of profit_abs :raise: ValueError if trade-dataframe was found empty. """ @@ -401,7 +402,7 @@ def calculate_csum(trades: pd.DataFrame) -> Tuple[float, float]: csum_df = pd.DataFrame() csum_df['sum'] = trades['profit_abs'].cumsum() - csum_min = csum_df['sum'].min() - csum_max = csum_df['sum'].max() + csum_min = csum_df['sum'].min() + starting_balance + csum_max = csum_df['sum'].max() + starting_balance return csum_min, csum_max diff --git a/tests/data/test_btanalysis.py b/tests/data/test_btanalysis.py index 555808679..538c89a90 100644 --- a/tests/data/test_btanalysis.py +++ b/tests/data/test_btanalysis.py @@ -296,6 +296,11 @@ def test_calculate_csum(testdatadir): assert isinstance(csum_max, float) assert csum_min < 0.01 assert csum_max > 0.02 + csum_min1, csum_max1 = calculate_csum(bt_data, 5) + + assert csum_min1 == csum_min + 5 + assert csum_max1 == csum_max + 5 + with pytest.raises(ValueError, match='Trade dataframe empty.'): csum_min, csum_max = calculate_csum(DataFrame()) From f367375e5b6240625ddd0ba51120ad48faa41409 Mon Sep 17 00:00:00 2001 From: Matthias Date: Tue, 16 Feb 2021 20:39:50 +0100 Subject: [PATCH 039/187] ABS drawdown should show wallet high and low values --- freqtrade/optimize/optimize_reports.py | 16 ++++++++++++---- 1 file changed, 12 insertions(+), 4 deletions(-) diff --git a/freqtrade/optimize/optimize_reports.py b/freqtrade/optimize/optimize_reports.py index 5b3f813f2..1ac0ae1d6 100644 --- a/freqtrade/optimize/optimize_reports.py +++ b/freqtrade/optimize/optimize_reports.py @@ -334,11 +334,11 @@ def generate_backtest_stats(btdata: Dict[str, DataFrame], 'drawdown_end': drawdown_end, 'drawdown_end_ts': drawdown_end.timestamp() * 1000, - 'max_drawdown_low': low_val, - 'max_drawdown_high': high_val, + 'max_drawdown_low': low_val + starting_balance, + 'max_drawdown_high': high_val + starting_balance, }) - csum_min, csum_max = calculate_csum(results) + csum_min, csum_max = calculate_csum(results, starting_balance) strat_stats.update({ 'csum_min': csum_min, 'csum_max': csum_max @@ -493,7 +493,15 @@ def text_table_add_metrics(strat_results: Dict) -> str: return tabulate(metrics, headers=["Metric", "Value"], tablefmt="orgtbl") else: - return '' + start_balance = round_coin_value(strat_results['starting_balance'], + strat_results['stake_currency']) + stake_amount = round_coin_value(strat_results['stake_amount'], + strat_results['stake_currency']) + message = ("No trades made. " + f"Your starting balance was {start_balance}, " + f"and your stake was {stake_amount}." + ) + return message def show_backtest_results(config: Dict, backtest_stats: Dict): From 37d7d2afd5c953c413024964d0078172ba1e3e1e Mon Sep 17 00:00:00 2001 From: Matthias Date: Wed, 17 Feb 2021 19:50:10 +0100 Subject: [PATCH 040/187] Wallets should not recalculate close_profit for closed trades --- freqtrade/wallets.py | 2 +- tests/conftest_trades.py | 2 ++ tests/test_freqtradebot.py | 2 ++ 3 files changed, 5 insertions(+), 1 deletion(-) diff --git a/freqtrade/wallets.py b/freqtrade/wallets.py index f5ce4c102..c2085641e 100644 --- a/freqtrade/wallets.py +++ b/freqtrade/wallets.py @@ -69,7 +69,7 @@ class Wallets: _wallets = {} closed_trades = Trade.get_trades_proxy(is_open=False) open_trades = Trade.get_trades_proxy(is_open=True) - tot_profit = sum([trade.calc_profit() for trade in closed_trades]) + tot_profit = sum([trade.close_profit_abs for trade in closed_trades]) tot_in_trades = sum([trade.stake_amount for trade in open_trades]) current_stake = self.start_cap + tot_profit - tot_in_trades diff --git a/tests/conftest_trades.py b/tests/conftest_trades.py index 6a42d04e3..025aac1b6 100644 --- a/tests/conftest_trades.py +++ b/tests/conftest_trades.py @@ -82,6 +82,7 @@ def mock_trade_2(fee): open_rate=0.123, close_rate=0.128, close_profit=0.005, + close_profit_abs=0.000584127, exchange='bittrex', is_open=False, open_order_id='dry_run_sell_12345', @@ -141,6 +142,7 @@ def mock_trade_3(fee): open_rate=0.05, close_rate=0.06, close_profit=0.01, + close_profit_abs=0.000155, exchange='bittrex', is_open=False, strategy='DefaultStrategy', diff --git a/tests/test_freqtradebot.py b/tests/test_freqtradebot.py index 3bd2f5607..d7d2e19f6 100644 --- a/tests/test_freqtradebot.py +++ b/tests/test_freqtradebot.py @@ -2243,6 +2243,7 @@ def test_check_handle_timedout_sell_usercustom(default_conf, ticker, limit_sell_ open_trade.open_date = arrow.utcnow().shift(hours=-5).datetime open_trade.close_date = arrow.utcnow().shift(minutes=-601).datetime + open_trade.close_profit_abs = 0.001 open_trade.is_open = False Trade.session.add(open_trade) @@ -2290,6 +2291,7 @@ def test_check_handle_timedout_sell(default_conf, ticker, limit_sell_order_old, open_trade.open_date = arrow.utcnow().shift(hours=-5).datetime open_trade.close_date = arrow.utcnow().shift(minutes=-601).datetime + open_trade.close_profit_abs = 0.001 open_trade.is_open = False Trade.session.add(open_trade) From 7913166453518733fcd793879d0150d1339796d9 Mon Sep 17 00:00:00 2001 From: Matthias Date: Wed, 17 Feb 2021 20:07:27 +0100 Subject: [PATCH 041/187] Improve performance by updating wallets only when necessary --- freqtrade/optimize/backtesting.py | 8 +------- 1 file changed, 1 insertion(+), 7 deletions(-) diff --git a/freqtrade/optimize/backtesting.py b/freqtrade/optimize/backtesting.py index c60cfa9b7..f921f64c3 100644 --- a/freqtrade/optimize/backtesting.py +++ b/freqtrade/optimize/backtesting.py @@ -180,10 +180,6 @@ class Backtesting: PairLocks.reset_locks() Trade.reset_trades() - def update_wallets(self): - if self.wallets: - self.wallets.update() - def _get_ohlcv_as_lists(self, processed: Dict[str, DataFrame]) -> Dict[str, Tuple]: """ Helper function to convert a processed dataframes into lists for performance reasons. @@ -272,7 +268,6 @@ class Backtesting: def _enter_trade(self, pair: str, row, max_open_trades: int, open_trade_count: int) -> Optional[Trade]: - self.update_wallets() try: stake_amount = self.wallets.get_trade_stake_amount( pair, max_open_trades - open_trade_count, None) @@ -391,7 +386,6 @@ class Backtesting: trade_entry = self._get_sell_trade_entry(trade, row) # Sell occured if trade_entry: - self.update_wallets() # logger.debug(f"{pair} - Backtesting sell {trade}") open_trade_count -= 1 open_trades[pair].remove(trade) @@ -404,7 +398,7 @@ class Backtesting: tmp += timedelta(minutes=self.timeframe_min) trades += self.handle_left_open(open_trades, data=data) - self.update_wallets() + self.wallets.update() return trade_list_to_dataframe(trades) From f04f07299c7689841eaf7eab15c574c09c5774b2 Mon Sep 17 00:00:00 2001 From: Matthias Date: Wed, 17 Feb 2021 20:19:03 +0100 Subject: [PATCH 042/187] Improve backtesting metrics --- docs/backtesting.md | 39 ++++++++++++++++++-------- freqtrade/optimize/optimize_reports.py | 36 +++++++++++++----------- 2 files changed, 47 insertions(+), 28 deletions(-) diff --git a/docs/backtesting.md b/docs/backtesting.md index ada788da9..bac12dae0 100644 --- a/docs/backtesting.md +++ b/docs/backtesting.md @@ -252,11 +252,12 @@ A backtesting result will look like that: | Max open trades | 3 | | | | | Total trades | 429 | -| Starting capital | 0.01000000 BTC | -| End capital | 0.01762792 BTC | +| Starting balance | 0.01000000 BTC | +| Final balance | 0.01762792 BTC | | Absolute profit | 0.00762792 BTC | -| Total Profit % | 76.2% | +| Total profit % | 76.2% | | Trades per day | 3.575 | +| Avg. stake amount | 0.001 | | Total trade volume | 0.429 BTC | | | | | Best Pair | LSK/BTC 26.26% | @@ -269,7 +270,12 @@ A backtesting result will look like that: | Avg. Duration Winners | 4:23:00 | | Avg. Duration Loser | 6:55:00 | | | | -| Max Drawdown | 50.63% | +| Min balance | 0.00945123 BTC | +| Max balance | 0.01846651 BTC | +| Drawdown | 50.63% | +| Drawdown | 0.0015 BTC | +| Drawdown high | 0.0013 BTC | +| Drawdown low | -0.0002 BTC | | Drawdown Start | 2019-02-15 14:10:00 | | Drawdown End | 2019-04-11 18:15:00 | | Market change | -5.88% | @@ -333,11 +339,12 @@ It contains some useful key metrics about performance of your strategy on backte | Max open trades | 3 | | | | | Total trades | 429 | -| Starting capital | 0.01000000 BTC | -| End capital | 0.01762792 BTC | +| Starting balance | 0.01000000 BTC | +| Final balance | 0.01762792 BTC | | Absolute profit | 0.00762792 BTC | -| Total Profit % | 76.2% | +| Total profit % | 76.2% | | Trades per day | 3.575 | +| Avg. stake amount | 0.001 | | Total trade volume | 0.429 BTC | | | | | Best Pair | LSK/BTC 26.26% | @@ -350,7 +357,12 @@ It contains some useful key metrics about performance of your strategy on backte | Avg. Duration Winners | 4:23:00 | | Avg. Duration Loser | 6:55:00 | | | | -| Max Drawdown | 50.63% | +| Min balance | 0.00945123 BTC | +| Max balance | 0.01846651 BTC | +| Drawdown | 50.63% | +| Drawdown | 0.0015 BTC | +| Drawdown high | 0.0013 BTC | +| Drawdown low | -0.0002 BTC | | Drawdown Start | 2019-02-15 14:10:00 | | Drawdown End | 2019-04-11 18:15:00 | | Market change | -5.88% | @@ -361,18 +373,21 @@ It contains some useful key metrics about performance of your strategy on backte - `Backtesting from` / `Backtesting to`: Backtesting range (usually defined with the `--timerange` option). - `Max open trades`: Setting of `max_open_trades` (or `--max-open-trades`) - or number of pairs in the pairlist (whatever is lower). - `Total trades`: Identical to the total trades of the backtest output table. -- `Starting capital`: Start capital - as given by dry-run-wallet (config or command line). -- `End capital`: Final capital - starting capital + absolute profit. +- `Starting balance`: Start balance - as given by dry-run-wallet (config or command line). +- `End balance`: Final balance - starting balance + absolute profit. - `Absolute profit`: Profit made in stake currency. -- `Total Profit %`: Total profit. Aligned to the `TOTAL` row's `Tot Profit %` from the first table. Calculated as `(End capital − Starting capital) / Starting capital`. +- `Total profit %`: Total profit. Aligned to the `TOTAL` row's `Tot Profit %` from the first table. Calculated as `(End capital − Starting capital) / Starting capital`. - `Trades per day`: Total trades divided by the backtesting duration in days (this will give you information about how many trades to expect from the strategy). +- `Avg. stake amount`: Average stake amount, either `stake_amount` or the average when using dynamic stake amount. - `Total trade volume`: Volume generated on the exchange to reach the above profit. - `Best Pair` / `Worst Pair`: Best and worst performing pair, and it's corresponding `Cum Profit %`. - `Best Trade` / `Worst Trade`: Biggest winning trade and biggest losing trade - `Best day` / `Worst day`: Best and worst day based on daily profit. - `Days win/draw/lose`: Winning / Losing days (draws are usually days without closed trade). - `Avg. Duration Winners` / `Avg. Duration Loser`: Average durations for winning and losing trades. -- `Max Drawdown`: Maximum drawdown experienced. For example, the value of 50% means that from highest to subsequent lowest point, a 50% drop was experienced). +- `Min balance` / `Max balance`: Lowest and Highest Wallet balance during the backtest period. +- `Drawdown`: Maximum drawdown experienced. For example, the value of 50% means that from highest to subsequent lowest point, a 50% drop was experienced). +- `Drawdown high` / `Drawdown low`: Profit at the beginning and end of the largest drawdown period. A negative low value means initial capital lost. - `Drawdown Start` / `Drawdown End`: Start and end datetime for this largest drawdown (can also be visualized via the `plot-dataframe` sub-command). - `Market change`: Change of the market during the backtest period. Calculated as average of all pairs changes from the first to the last candle using the "close" column. diff --git a/freqtrade/optimize/optimize_reports.py b/freqtrade/optimize/optimize_reports.py index 1ac0ae1d6..cee0bb1ce 100644 --- a/freqtrade/optimize/optimize_reports.py +++ b/freqtrade/optimize/optimize_reports.py @@ -278,6 +278,7 @@ def generate_backtest_stats(btdata: Dict[str, DataFrame], 'left_open_trades': left_open_results, 'total_trades': len(results), 'total_volume': results['stake_amount'].sum(), + 'avg_stake_amount': results['stake_amount'].mean(), 'profit_mean': results['profit_ratio'].mean() if len(results) > 0 else 0, 'profit_total': results['profit_abs'].sum() / starting_balance, 'profit_total_abs': results['profit_abs'].sum(), @@ -295,6 +296,7 @@ def generate_backtest_stats(btdata: Dict[str, DataFrame], 'pairlist': list(btdata.keys()), 'stake_amount': config['stake_amount'], 'stake_currency': config['stake_currency'], + 'stake_currency_decimals': decimals_per_coin(config['stake_currency']), 'starting_balance': starting_balance, 'dry_run_wallet': starting_balance, 'final_balance': content['final_balance'], @@ -334,8 +336,8 @@ def generate_backtest_stats(btdata: Dict[str, DataFrame], 'drawdown_end': drawdown_end, 'drawdown_end_ts': drawdown_end.timestamp() * 1000, - 'max_drawdown_low': low_val + starting_balance, - 'max_drawdown_high': high_val + starting_balance, + 'max_drawdown_low': low_val, + 'max_drawdown_high': high_val, }) csum_min, csum_max = calculate_csum(results, starting_balance) @@ -446,14 +448,16 @@ def text_table_add_metrics(strat_results: Dict) -> str: ('Max open trades', strat_results['max_open_trades']), ('', ''), # Empty line to improve readability ('Total trades', strat_results['total_trades']), - ('Starting capital', round_coin_value(strat_results['starting_balance'], + ('Starting balance', round_coin_value(strat_results['starting_balance'], strat_results['stake_currency'])), - ('End capital', round_coin_value(strat_results['final_balance'], - strat_results['stake_currency'])), + ('Final balance', round_coin_value(strat_results['final_balance'], + strat_results['stake_currency'])), ('Absolute profit ', round_coin_value(strat_results['profit_total_abs'], strat_results['stake_currency'])), - ('Total Profit %', f"{round(strat_results['profit_total'] * 100, 2)}%"), + ('Total profit %', f"{round(strat_results['profit_total'] * 100, 2)}%"), ('Trades per day', strat_results['trades_per_day']), + ('Avg. stake amount', round_coin_value(strat_results['avg_stake_amount'], + strat_results['stake_currency'])), ('Total trade volume', round_coin_value(strat_results['total_volume'], strat_results['stake_currency'])), @@ -474,18 +478,18 @@ def text_table_add_metrics(strat_results: Dict) -> str: ('Avg. Duration Loser', f"{strat_results['loser_holding_avg']}"), ('', ''), # Empty line to improve readability - ('Abs Profit Min', round_coin_value(strat_results['csum_min'], - strat_results['stake_currency'])), - ('Abs Profit Max', round_coin_value(strat_results['csum_max'], - strat_results['stake_currency'])), + ('Min balance', round_coin_value(strat_results['csum_min'], + strat_results['stake_currency'])), + ('Max balance', round_coin_value(strat_results['csum_max'], + strat_results['stake_currency'])), - ('Max Drawdown', f"{round(strat_results['max_drawdown'] * 100, 2)}%"), - ('Max Drawdown', round_coin_value(strat_results['max_drawdown_abs'], + ('Drawdown', f"{round(strat_results['max_drawdown'] * 100, 2)}%"), + ('Drawdown', round_coin_value(strat_results['max_drawdown_abs'], + strat_results['stake_currency'])), + ('Drawdown high', round_coin_value(strat_results['max_drawdown_high'], + strat_results['stake_currency'])), + ('Drawdown low', round_coin_value(strat_results['max_drawdown_low'], strat_results['stake_currency'])), - ('Max Drawdown high', round_coin_value(strat_results['max_drawdown_high'], - strat_results['stake_currency'])), - ('Max Drawdown low', round_coin_value(strat_results['max_drawdown_low'], - strat_results['stake_currency'])), ('Drawdown Start', strat_results['drawdown_start'].strftime(DATETIME_PRINT_FORMAT)), ('Drawdown End', strat_results['drawdown_end'].strftime(DATETIME_PRINT_FORMAT)), ('Market change', f"{round(strat_results['market_change'] * 100, 2)}%"), From 52acacbed5b43f5cec2f07af74336998c3e51523 Mon Sep 17 00:00:00 2001 From: Matthias Date: Sat, 20 Feb 2021 07:20:51 +0100 Subject: [PATCH 043/187] Check min-trade-stake in backtesting --- freqtrade/optimize/backtesting.py | 4 +++- tests/optimize/test_backtest_detail.py | 3 ++- tests/optimize/test_backtesting.py | 6 ++++++ 3 files changed, 11 insertions(+), 2 deletions(-) diff --git a/freqtrade/optimize/backtesting.py b/freqtrade/optimize/backtesting.py index f921f64c3..bd185234f 100644 --- a/freqtrade/optimize/backtesting.py +++ b/freqtrade/optimize/backtesting.py @@ -273,7 +273,9 @@ class Backtesting: pair, max_open_trades - open_trade_count, None) except DependencyException: stake_amount = 0 - if stake_amount: + min_stake_amount = self.exchange.get_min_pair_stake_amount(pair, row[OPEN_IDX], -0.05) + if stake_amount and stake_amount > min_stake_amount: + # print(f"{pair}, {stake_amount}") # Enter trade trade = Trade( pair=pair, diff --git a/tests/optimize/test_backtest_detail.py b/tests/optimize/test_backtest_detail.py index 4d6605b9f..a56e024f7 100644 --- a/tests/optimize/test_backtest_detail.py +++ b/tests/optimize/test_backtest_detail.py @@ -489,7 +489,8 @@ def test_backtest_results(default_conf, fee, mocker, caplog, data) -> None: default_conf["trailing_stop_positive_offset"] = data.trailing_stop_positive_offset default_conf["ask_strategy"] = {"use_sell_signal": data.use_sell_signal} - mocker.patch("freqtrade.exchange.Exchange.get_fee", MagicMock(return_value=0.0)) + mocker.patch("freqtrade.exchange.Exchange.get_fee", return_value=0.0) + mocker.patch("freqtrade.exchange.Exchange.get_min_pair_stake_amount", return_value=0.00001) patch_exchange(mocker) frame = _build_backtest_dataframe(data.data) backtesting = Backtesting(default_conf) diff --git a/tests/optimize/test_backtesting.py b/tests/optimize/test_backtesting.py index 8fba8724b..eda8aac9d 100644 --- a/tests/optimize/test_backtesting.py +++ b/tests/optimize/test_backtesting.py @@ -450,6 +450,7 @@ def test_backtesting_pairlist_list(default_conf, mocker, caplog, testdatadir, ti def test_backtest_one(default_conf, fee, mocker, testdatadir) -> None: default_conf['ask_strategy']['use_sell_signal'] = False mocker.patch('freqtrade.exchange.Exchange.get_fee', fee) + mocker.patch("freqtrade.exchange.Exchange.get_min_pair_stake_amount", return_value=0.00001) patch_exchange(mocker) backtesting = Backtesting(default_conf) pair = 'UNITTEST/BTC' @@ -510,6 +511,7 @@ def test_backtest_one(default_conf, fee, mocker, testdatadir) -> None: def test_backtest_1min_timeframe(default_conf, fee, mocker, testdatadir) -> None: default_conf['ask_strategy']['use_sell_signal'] = False mocker.patch('freqtrade.exchange.Exchange.get_fee', fee) + mocker.patch("freqtrade.exchange.Exchange.get_min_pair_stake_amount", return_value=0.00001) patch_exchange(mocker) backtesting = Backtesting(default_conf) @@ -555,6 +557,7 @@ def test_backtest_pricecontours_protections(default_conf, fee, mocker, testdatad default_conf['enable_protections'] = True mocker.patch('freqtrade.exchange.Exchange.get_fee', fee) + mocker.patch("freqtrade.exchange.Exchange.get_min_pair_stake_amount", return_value=0.00001) tests = [ ['sine', 9], ['raise', 10], @@ -586,6 +589,7 @@ def test_backtest_pricecontours(default_conf, fee, mocker, testdatadir, default_conf['protections'] = protections default_conf['enable_protections'] = True + mocker.patch("freqtrade.exchange.Exchange.get_min_pair_stake_amount", return_value=0.00001) mocker.patch('freqtrade.exchange.Exchange.get_fee', fee) # While buy-signals are unrealistic, running backtesting # over and over again should not cause different results @@ -623,6 +627,7 @@ def test_backtest_only_sell(mocker, default_conf, testdatadir): def test_backtest_alternate_buy_sell(default_conf, fee, mocker, testdatadir): + mocker.patch("freqtrade.exchange.Exchange.get_min_pair_stake_amount", return_value=0.00001) mocker.patch('freqtrade.exchange.Exchange.get_fee', fee) backtest_conf = _make_backtest_conf(mocker, conf=default_conf, pair='UNITTEST/BTC', datadir=testdatadir) @@ -655,6 +660,7 @@ def test_backtest_multi_pair(default_conf, fee, mocker, tres, pair, testdatadir) dataframe['sell'] = np.where((dataframe.index + multi - 2) % multi == 0, 1, 0) return dataframe + mocker.patch("freqtrade.exchange.Exchange.get_min_pair_stake_amount", return_value=0.00001) mocker.patch('freqtrade.exchange.Exchange.get_fee', fee) patch_exchange(mocker) From 394a6bbf2a86c8c4990c1f38e566ebaaa2ea2561 Mon Sep 17 00:00:00 2001 From: Matthias Date: Sat, 20 Feb 2021 20:21:30 +0100 Subject: [PATCH 044/187] Fix some type errors --- freqtrade/optimize/backtesting.py | 4 ++-- freqtrade/optimize/optimize_reports.py | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/freqtrade/optimize/backtesting.py b/freqtrade/optimize/backtesting.py index bd185234f..7028a38cd 100644 --- a/freqtrade/optimize/backtesting.py +++ b/freqtrade/optimize/backtesting.py @@ -274,7 +274,7 @@ class Backtesting: except DependencyException: stake_amount = 0 min_stake_amount = self.exchange.get_min_pair_stake_amount(pair, row[OPEN_IDX], -0.05) - if stake_amount and stake_amount > min_stake_amount: + if stake_amount and (not min_stake_amount or stake_amount > min_stake_amount): # print(f"{pair}, {stake_amount}") # Enter trade trade = Trade( @@ -341,7 +341,7 @@ class Backtesting: indexes: Dict = {} tmp = start_date + timedelta(minutes=self.timeframe_min) - open_trades: Dict[str, List] = defaultdict(list) + open_trades: Dict[str, List[Trade]] = defaultdict(list) open_trade_count = 0 # Loop timerange and get candle for each pair at that point in time diff --git a/freqtrade/optimize/optimize_reports.py b/freqtrade/optimize/optimize_reports.py index cee0bb1ce..e7111f20c 100644 --- a/freqtrade/optimize/optimize_reports.py +++ b/freqtrade/optimize/optimize_reports.py @@ -479,9 +479,9 @@ def text_table_add_metrics(strat_results: Dict) -> str: ('', ''), # Empty line to improve readability ('Min balance', round_coin_value(strat_results['csum_min'], - strat_results['stake_currency'])), + strat_results['stake_currency'])), ('Max balance', round_coin_value(strat_results['csum_max'], - strat_results['stake_currency'])), + strat_results['stake_currency'])), ('Drawdown', f"{round(strat_results['max_drawdown'] * 100, 2)}%"), ('Drawdown', round_coin_value(strat_results['max_drawdown_abs'], From 03eb23a4ce7cdf54ffbc3595814ca2029f979262 Mon Sep 17 00:00:00 2001 From: Matthias Date: Sat, 20 Feb 2021 19:29:04 +0100 Subject: [PATCH 045/187] 2 levels of Trade models, one with and one without sqlalchemy Fixes a performance issue when backtesting with sqlalchemy, as that uses descriptors for all properties. --- freqtrade/optimize/backtesting.py | 11 +- freqtrade/persistence/__init__.py | 3 +- freqtrade/persistence/models.py | 244 ++++++++++++++++++++---------- 3 files changed, 170 insertions(+), 88 deletions(-) diff --git a/freqtrade/optimize/backtesting.py b/freqtrade/optimize/backtesting.py index 7028a38cd..322a3f00b 100644 --- a/freqtrade/optimize/backtesting.py +++ b/freqtrade/optimize/backtesting.py @@ -23,6 +23,7 @@ from freqtrade.mixins import LoggingMixin from freqtrade.optimize.optimize_reports import (generate_backtest_stats, show_backtest_results, store_backtest_stats) from freqtrade.persistence import PairLocks, Trade +from freqtrade.persistence.models import LocalTrade from freqtrade.plugins.pairlistmanager import PairListManager from freqtrade.plugins.protectionmanager import ProtectionManager from freqtrade.resolvers import ExchangeResolver, StrategyResolver @@ -267,7 +268,7 @@ class Backtesting: return None def _enter_trade(self, pair: str, row, max_open_trades: int, - open_trade_count: int) -> Optional[Trade]: + open_trade_count: int) -> Optional[LocalTrade]: try: stake_amount = self.wallets.get_trade_stake_amount( pair, max_open_trades - open_trade_count, None) @@ -277,7 +278,7 @@ class Backtesting: if stake_amount and (not min_stake_amount or stake_amount > min_stake_amount): # print(f"{pair}, {stake_amount}") # Enter trade - trade = Trade( + trade = LocalTrade( pair=pair, open_rate=row[OPEN_IDX], open_date=row[DATE_IDX], @@ -291,8 +292,8 @@ class Backtesting: return trade return None - def handle_left_open(self, open_trades: Dict[str, List[Trade]], - data: Dict[str, List[Tuple]]) -> List[Trade]: + def handle_left_open(self, open_trades: Dict[str, List[LocalTrade]], + data: Dict[str, List[Tuple]]) -> List[LocalTrade]: """ Handling of left open trades at the end of backtesting """ @@ -381,7 +382,7 @@ class Backtesting: open_trade_count += 1 # logger.debug(f"{pair} - Emulate creation of new trade: {trade}.") open_trades[pair].append(trade) - Trade.trades.append(trade) + LocalTrade.trades.append(trade) for trade in open_trades[pair]: # also check the buying candle for sell conditions. diff --git a/freqtrade/persistence/__init__.py b/freqtrade/persistence/__init__.py index 35f2bc406..d1fcac0ba 100644 --- a/freqtrade/persistence/__init__.py +++ b/freqtrade/persistence/__init__.py @@ -1,4 +1,5 @@ # flake8: noqa: F401 -from freqtrade.persistence.models import Order, Trade, clean_dry_run_db, cleanup_db, init_db +from freqtrade.persistence.models import (LocalTrade, Order, Trade, clean_dry_run_db, cleanup_db, + init_db) from freqtrade.persistence.pairlock_middleware import PairLocks diff --git a/freqtrade/persistence/models.py b/freqtrade/persistence/models.py index f72705c34..48ae8bb40 100644 --- a/freqtrade/persistence/models.py +++ b/freqtrade/persistence/models.py @@ -199,67 +199,67 @@ class Order(_DECL_BASE): return Order.query.filter(Order.ft_is_open.is_(True)).all() -class Trade(_DECL_BASE): +class LocalTrade(): """ Trade database model. - Also handles updating and querying trades + Used in backtesting - must be aligned to Trade model! + """ - __tablename__ = 'trades' - - use_db: bool = True + use_db: bool = False # Trades container for backtesting - trades: List['Trade'] = [] + trades: List['LocalTrade'] = [] - id = Column(Integer, primary_key=True) + id: int = 0 - orders = relationship("Order", order_by="Order.id", cascade="all, delete-orphan") + orders: List[Order] = [] - exchange = Column(String, nullable=False) - pair = Column(String, nullable=False, index=True) - is_open = Column(Boolean, nullable=False, default=True, index=True) - fee_open = Column(Float, nullable=False, default=0.0) - fee_open_cost = Column(Float, nullable=True) - fee_open_currency = Column(String, nullable=True) - fee_close = Column(Float, nullable=False, default=0.0) - fee_close_cost = Column(Float, nullable=True) - fee_close_currency = Column(String, nullable=True) - open_rate = Column(Float) - open_rate_requested = Column(Float) + exchange: str = '' + pair: str = '' + is_open: bool = True + fee_open: float = 0.0 + fee_open_cost: Optional[float] = None + fee_open_currency: str = '' + fee_close: float = 0.0 + fee_close_cost: Optional[float] = None + fee_close_currency: str = '' + open_rate: float + open_rate_requested: Optional[float] = None # open_trade_value - calculated via _calc_open_trade_value - open_trade_value = Column(Float) - close_rate = Column(Float) - close_rate_requested = Column(Float) - close_profit = Column(Float) - close_profit_abs = Column(Float) - stake_amount = Column(Float, nullable=False) - amount = Column(Float) - amount_requested = Column(Float) - open_date = Column(DateTime, nullable=False, default=datetime.utcnow) - close_date = Column(DateTime) - open_order_id = Column(String) + open_trade_value: float + close_rate: Optional[float] = None + close_rate_requested: Optional[float] = None + close_profit: Optional[float] = None + close_profit_abs: Optional[float] = None + stake_amount: float + amount: float + amount_requested: Optional[float] = None + open_date: datetime + close_date: Optional[datetime] = None + open_order_id: Optional[str] = None # absolute value of the stop loss - stop_loss = Column(Float, nullable=True, default=0.0) + stop_loss: float = 0.0 # percentage value of the stop loss - stop_loss_pct = Column(Float, nullable=True) + stop_loss_pct: float = 0.0 # absolute value of the initial stop loss - initial_stop_loss = Column(Float, nullable=True, default=0.0) + initial_stop_loss: float = 0.0 # percentage value of the initial stop loss - initial_stop_loss_pct = Column(Float, nullable=True) + initial_stop_loss_pct: float = 0.0 # stoploss order id which is on exchange - stoploss_order_id = Column(String, nullable=True, index=True) + stoploss_order_id: Optional[str] = None # last update time of the stoploss order on exchange - stoploss_last_update = Column(DateTime, nullable=True) + stoploss_last_update: Optional[datetime] = None # absolute value of the highest reached price - max_rate = Column(Float, nullable=True, default=0.0) + max_rate: float = 0.0 # Lowest price reached - min_rate = Column(Float, nullable=True) - sell_reason = Column(String, nullable=True) - sell_order_status = Column(String, nullable=True) - strategy = Column(String, nullable=True) - timeframe = Column(Integer, nullable=True) + min_rate: float = 0.0 + sell_reason: str = '' + sell_order_status: str = '' + strategy: str = '' + timeframe: Optional[int] = None def __init__(self, **kwargs): - super().__init__(**kwargs) + for key in kwargs: + setattr(self, key, kwargs[key]) self.recalc_open_trade_value() def __repr__(self): @@ -349,8 +349,7 @@ class Trade(_DECL_BASE): """ Resets all trades. Only active for backtesting mode. """ - if not Trade.use_db: - Trade.trades = [] + LocalTrade.trades = [] def adjust_min_max_rates(self, current_price: float) -> None: """ @@ -418,8 +417,8 @@ class Trade(_DECL_BASE): if order_type in ('market', 'limit') and order['side'] == 'buy': # Update open rate and actual amount - self.open_rate = Decimal(safe_value_fallback(order, 'average', 'price')) - self.amount = Decimal(safe_value_fallback(order, 'filled', 'amount')) + self.open_rate = float(safe_value_fallback(order, 'average', 'price')) + self.amount = float(safe_value_fallback(order, 'filled', 'amount')) self.recalc_open_trade_value() if self.is_open: logger.info(f'{order_type.upper()}_BUY has been fulfilled for {self}.') @@ -443,7 +442,7 @@ class Trade(_DECL_BASE): Sets close_rate to the given rate, calculates total profit and marks trade as closed """ - self.close_rate = Decimal(rate) + self.close_rate = rate self.close_profit = self.calc_profit_ratio() self.close_profit_abs = self.calc_profit() self.close_date = self.close_date or datetime.utcnow() @@ -488,14 +487,6 @@ class Trade(_DECL_BASE): def update_order(self, order: Dict) -> None: Order.update_orders(self.orders, order) - def delete(self) -> None: - - for order in self.orders: - Order.session.delete(order) - - Trade.session.delete(self) - Trade.session.flush() - def _calc_open_trade_value(self) -> float: """ Calculate the open_rate including open_fee. @@ -525,7 +516,7 @@ class Trade(_DECL_BASE): if rate is None and not self.close_rate: return 0.0 - sell_trade = Decimal(self.amount) * Decimal(rate or self.close_rate) + sell_trade = Decimal(self.amount) * Decimal(rate or self.close_rate) # type: ignore fees = sell_trade * Decimal(fee or self.fee_close) return float(sell_trade - fees) @@ -597,7 +588,7 @@ class Trade(_DECL_BASE): @staticmethod def get_trades_proxy(*, pair: str = None, is_open: bool = None, open_date: datetime = None, close_date: datetime = None, - ) -> List['Trade']: + ) -> List['LocalTrade']: """ Helper function to query Trades. Returns a List of trades, filtered on the parameters given. @@ -606,30 +597,19 @@ class Trade(_DECL_BASE): :return: unsorted List[Trade] """ - if Trade.use_db: - trade_filter = [] - if pair: - trade_filter.append(Trade.pair == pair) - if open_date: - trade_filter.append(Trade.open_date > open_date) - if close_date: - trade_filter.append(Trade.close_date > close_date) - if is_open is not None: - trade_filter.append(Trade.is_open.is_(is_open)) - return Trade.get_trades(trade_filter).all() - else: - # Offline mode - without database - sel_trades = [trade for trade in Trade.trades] - if pair: - sel_trades = [trade for trade in sel_trades if trade.pair == pair] - if open_date: - sel_trades = [trade for trade in sel_trades if trade.open_date > open_date] - if close_date: - sel_trades = [trade for trade in sel_trades if trade.close_date - and trade.close_date > close_date] - if is_open is not None: - sel_trades = [trade for trade in sel_trades if trade.is_open == is_open] - return sel_trades + + # Offline mode - without database + sel_trades = [trade for trade in LocalTrade.trades] + if pair: + sel_trades = [trade for trade in sel_trades if trade.pair == pair] + if open_date: + sel_trades = [trade for trade in sel_trades if trade.open_date > open_date] + if close_date: + sel_trades = [trade for trade in sel_trades if trade.close_date + and trade.close_date > close_date] + if is_open is not None: + sel_trades = [trade for trade in sel_trades if trade.is_open == is_open] + return sel_trades @staticmethod def get_open_trades() -> List[Any]: @@ -735,6 +715,106 @@ class Trade(_DECL_BASE): logger.info(f"New stoploss: {trade.stop_loss}.") +class Trade(_DECL_BASE, LocalTrade): + """ + Trade database model. + Also handles updating and querying trades + """ + __tablename__ = 'trades' + + use_db: bool = True + + id = Column(Integer, primary_key=True) + + orders = relationship("Order", order_by="Order.id", cascade="all, delete-orphan") + + exchange = Column(String, nullable=False) + pair = Column(String, nullable=False, index=True) + is_open = Column(Boolean, nullable=False, default=True, index=True) + fee_open = Column(Float, nullable=False, default=0.0) + fee_open_cost = Column(Float, nullable=True) + fee_open_currency = Column(String, nullable=True) + fee_close = Column(Float, nullable=False, default=0.0) + fee_close_cost = Column(Float, nullable=True) + fee_close_currency = Column(String, nullable=True) + open_rate = Column(Float) + open_rate_requested = Column(Float) + # open_trade_value - calculated via _calc_open_trade_value + open_trade_value = Column(Float) + close_rate = Column(Float) + close_rate_requested = Column(Float) + close_profit = Column(Float) + close_profit_abs = Column(Float) + stake_amount = Column(Float, nullable=False) + amount = Column(Float) + amount_requested = Column(Float) + open_date = Column(DateTime, nullable=False, default=datetime.utcnow) + close_date = Column(DateTime) + open_order_id = Column(String) + # absolute value of the stop loss + stop_loss = Column(Float, nullable=True, default=0.0) + # percentage value of the stop loss + stop_loss_pct = Column(Float, nullable=True) + # absolute value of the initial stop loss + initial_stop_loss = Column(Float, nullable=True, default=0.0) + # percentage value of the initial stop loss + initial_stop_loss_pct = Column(Float, nullable=True) + # stoploss order id which is on exchange + stoploss_order_id = Column(String, nullable=True, index=True) + # last update time of the stoploss order on exchange + stoploss_last_update = Column(DateTime, nullable=True) + # absolute value of the highest reached price + max_rate = Column(Float, nullable=True, default=0.0) + # Lowest price reached + min_rate = Column(Float, nullable=True) + sell_reason = Column(String, nullable=True) + sell_order_status = Column(String, nullable=True) + strategy = Column(String, nullable=True) + timeframe = Column(Integer, nullable=True) + + def __init__(self, **kwargs): + super().__init__(**kwargs) + self.recalc_open_trade_value() + + def delete(self) -> None: + + for order in self.orders: + Order.session.delete(order) + + Trade.session.delete(self) + Trade.session.flush() + + @staticmethod + def get_trades_proxy(*, pair: str = None, is_open: bool = None, + open_date: datetime = None, close_date: datetime = None, + ) -> List['LocalTrade']: + """ + Helper function to query Trades. + Returns a List of trades, filtered on the parameters given. + In live mode, converts the filter to a database query and returns all rows + In Backtest mode, uses filters on Trade.trades to get the result. + + :return: unsorted List[Trade] + """ + if Trade.use_db: + trade_filter = [] + if pair: + trade_filter.append(Trade.pair == pair) + if open_date: + trade_filter.append(Trade.open_date > open_date) + if close_date: + trade_filter.append(Trade.close_date > close_date) + if is_open is not None: + trade_filter.append(Trade.is_open.is_(is_open)) + return Trade.get_trades(trade_filter).all() + else: + return LocalTrade.get_trades_proxy( + pair=pair, is_open=is_open, + open_date=open_date, + close_date=close_date + ) + + class PairLock(_DECL_BASE): """ Pair Locks database model. From 53a57f2c81f05c6bf7f2ce3cf5bd5cc95d591464 Mon Sep 17 00:00:00 2001 From: Matthias Date: Sat, 20 Feb 2021 20:22:00 +0100 Subject: [PATCH 046/187] Change some types Fix types of new model object --- freqtrade/data/btanalysis.py | 4 ++-- freqtrade/optimize/backtesting.py | 12 ++++++------ freqtrade/plugins/protections/cooldown_period.py | 3 ++- freqtrade/plugins/protections/iprotection.py | 6 +++--- freqtrade/plugins/protections/low_profit_pairs.py | 2 +- freqtrade/plugins/protections/stoploss_guard.py | 2 +- tests/conftest.py | 4 ++-- tests/optimize/test_backtest_detail.py | 1 - 8 files changed, 17 insertions(+), 17 deletions(-) diff --git a/freqtrade/data/btanalysis.py b/freqtrade/data/btanalysis.py index 3adee8775..c98477f4e 100644 --- a/freqtrade/data/btanalysis.py +++ b/freqtrade/data/btanalysis.py @@ -10,7 +10,7 @@ import pandas as pd from freqtrade.constants import LAST_BT_RESULT_FN from freqtrade.misc import json_load -from freqtrade.persistence import Trade, init_db +from freqtrade.persistence import LocalTrade, Trade, init_db logger = logging.getLogger(__name__) @@ -224,7 +224,7 @@ def evaluate_result_multi(results: pd.DataFrame, timeframe: str, return df_final[df_final['open_trades'] > max_open_trades] -def trade_list_to_dataframe(trades: List[Trade]) -> pd.DataFrame: +def trade_list_to_dataframe(trades: List[LocalTrade]) -> pd.DataFrame: """ Convert list of Trade objects to pandas Dataframe :param trades: List of trade objects diff --git a/freqtrade/optimize/backtesting.py b/freqtrade/optimize/backtesting.py index 322a3f00b..aeafaffd3 100644 --- a/freqtrade/optimize/backtesting.py +++ b/freqtrade/optimize/backtesting.py @@ -211,7 +211,7 @@ class Backtesting: data[pair] = [x for x in df_analyzed.itertuples(index=False, name=None)] return data - def _get_close_rate(self, sell_row: Tuple, trade: Trade, sell: SellCheckTuple, + def _get_close_rate(self, sell_row: Tuple, trade: LocalTrade, sell: SellCheckTuple, trade_dur: int) -> float: """ Get close rate for backtesting result @@ -251,10 +251,10 @@ class Backtesting: else: return sell_row[OPEN_IDX] - def _get_sell_trade_entry(self, trade: Trade, sell_row: Tuple) -> Optional[Trade]: + def _get_sell_trade_entry(self, trade: LocalTrade, sell_row: Tuple) -> Optional[LocalTrade]: - sell = self.strategy.should_sell(trade, sell_row[OPEN_IDX], sell_row[DATE_IDX], - sell_row[BUY_IDX], sell_row[SELL_IDX], + sell = self.strategy.should_sell(trade, sell_row[OPEN_IDX], # type: ignore + sell_row[DATE_IDX], sell_row[BUY_IDX], sell_row[SELL_IDX], low=sell_row[LOW_IDX], high=sell_row[HIGH_IDX]) if sell.sell_flag: @@ -331,7 +331,7 @@ class Backtesting: :param enable_protections: Should protections be enabled? :return: DataFrame with trades (results of backtesting) """ - trades: List[Trade] = [] + trades: List[LocalTrade] = [] self.prepare_backtest(enable_protections) # Use dict of lists with data for performance @@ -342,7 +342,7 @@ class Backtesting: indexes: Dict = {} tmp = start_date + timedelta(minutes=self.timeframe_min) - open_trades: Dict[str, List[Trade]] = defaultdict(list) + open_trades: Dict[str, List[LocalTrade]] = defaultdict(list) open_trade_count = 0 # Loop timerange and get candle for each pair at that point in time diff --git a/freqtrade/plugins/protections/cooldown_period.py b/freqtrade/plugins/protections/cooldown_period.py index 2d7d7b4c7..f74f83885 100644 --- a/freqtrade/plugins/protections/cooldown_period.py +++ b/freqtrade/plugins/protections/cooldown_period.py @@ -44,7 +44,8 @@ class CooldownPeriod(IProtection): trades = Trade.get_trades_proxy(pair=pair, is_open=False, close_date=look_back_until) if trades: # Get latest trade - trade = sorted(trades, key=lambda t: t.close_date)[-1] + # Ignore type error as we know we only get closed trades. + trade = sorted(trades, key=lambda t: t.close_date)[-1] # type: ignore self.log_once(f"Cooldown for {pair} for {self.stop_duration_str}.", logger.info) until = self.calculate_lock_end([trade], self._stop_duration) diff --git a/freqtrade/plugins/protections/iprotection.py b/freqtrade/plugins/protections/iprotection.py index 684bf6cd3..d034beefc 100644 --- a/freqtrade/plugins/protections/iprotection.py +++ b/freqtrade/plugins/protections/iprotection.py @@ -7,7 +7,7 @@ from typing import Any, Dict, List, Optional, Tuple from freqtrade.exchange import timeframe_to_minutes from freqtrade.misc import plural from freqtrade.mixins import LoggingMixin -from freqtrade.persistence import Trade +from freqtrade.persistence import LocalTrade logger = logging.getLogger(__name__) @@ -93,11 +93,11 @@ class IProtection(LoggingMixin, ABC): """ @staticmethod - def calculate_lock_end(trades: List[Trade], stop_minutes: int) -> datetime: + def calculate_lock_end(trades: List[LocalTrade], stop_minutes: int) -> datetime: """ Get lock end time """ - max_date: datetime = max([trade.close_date for trade in trades]) + max_date: datetime = max([trade.close_date for trade in trades if trade.close_date]) # comming from Database, tzinfo is not set. if max_date.tzinfo is None: max_date = max_date.replace(tzinfo=timezone.utc) diff --git a/freqtrade/plugins/protections/low_profit_pairs.py b/freqtrade/plugins/protections/low_profit_pairs.py index 9d5ed35b4..7822ce73c 100644 --- a/freqtrade/plugins/protections/low_profit_pairs.py +++ b/freqtrade/plugins/protections/low_profit_pairs.py @@ -53,7 +53,7 @@ class LowProfitPairs(IProtection): # Not enough trades in the relevant period return False, None, None - profit = sum(trade.close_profit for trade in trades) + profit = sum(trade.close_profit for trade in trades if trade.close_profit) if profit < self._required_profit: self.log_once( f"Trading for {pair} stopped due to {profit:.2f} < {self._required_profit} " diff --git a/freqtrade/plugins/protections/stoploss_guard.py b/freqtrade/plugins/protections/stoploss_guard.py index 5a9b9ddd0..635c0be04 100644 --- a/freqtrade/plugins/protections/stoploss_guard.py +++ b/freqtrade/plugins/protections/stoploss_guard.py @@ -56,7 +56,7 @@ class StoplossGuard(IProtection): trades = [trade for trade in trades1 if (str(trade.sell_reason) in ( SellType.TRAILING_STOP_LOSS.value, SellType.STOP_LOSS.value, SellType.STOPLOSS_ON_EXCHANGE.value) - and trade.close_profit < 0)] + and trade.close_profit and trade.close_profit < 0)] if len(trades) < self._trade_limit: return False, None, None diff --git a/tests/conftest.py b/tests/conftest.py index 6e70603b1..793ba83b0 100644 --- a/tests/conftest.py +++ b/tests/conftest.py @@ -19,7 +19,7 @@ from freqtrade.data.converter import ohlcv_to_dataframe from freqtrade.edge import Edge, PairInfo from freqtrade.exchange import Exchange from freqtrade.freqtradebot import FreqtradeBot -from freqtrade.persistence import Trade, init_db +from freqtrade.persistence import LocalTrade, Trade, init_db from freqtrade.resolvers import ExchangeResolver from freqtrade.worker import Worker from tests.conftest_trades import (mock_trade_1, mock_trade_2, mock_trade_3, mock_trade_4, @@ -191,7 +191,7 @@ def create_mock_trades(fee, use_db: bool = True): if use_db: Trade.session.add(trade) else: - Trade.trades.append(trade) + LocalTrade.trades.append(trade) # Simulate dry_run entries trade = mock_trade_1(fee) diff --git a/tests/optimize/test_backtest_detail.py b/tests/optimize/test_backtest_detail.py index a56e024f7..0ba6f4a7f 100644 --- a/tests/optimize/test_backtest_detail.py +++ b/tests/optimize/test_backtest_detail.py @@ -1,6 +1,5 @@ # pragma pylint: disable=missing-docstring, W0212, line-too-long, C0103, C0330, unused-argument import logging -from unittest.mock import MagicMock import pytest From 60db6ccf454715aa9d8b2ba56e4676006e8fb1fd Mon Sep 17 00:00:00 2001 From: Matthias Date: Sat, 20 Feb 2021 20:07:00 +0100 Subject: [PATCH 047/187] Add test for subclassing --- freqtrade/persistence/models.py | 8 ++++---- tests/test_persistence.py | 26 +++++++++++++++++++++++++- 2 files changed, 29 insertions(+), 5 deletions(-) diff --git a/freqtrade/persistence/models.py b/freqtrade/persistence/models.py index 48ae8bb40..51a48c246 100644 --- a/freqtrade/persistence/models.py +++ b/freqtrade/persistence/models.py @@ -222,16 +222,16 @@ class LocalTrade(): fee_close: float = 0.0 fee_close_cost: Optional[float] = None fee_close_currency: str = '' - open_rate: float + open_rate: float = 0.0 open_rate_requested: Optional[float] = None # open_trade_value - calculated via _calc_open_trade_value - open_trade_value: float + open_trade_value: float = 0.0 close_rate: Optional[float] = None close_rate_requested: Optional[float] = None close_profit: Optional[float] = None close_profit_abs: Optional[float] = None - stake_amount: float - amount: float + stake_amount: float = 0.0 + amount: float = 0.0 amount_requested: Optional[float] = None open_date: datetime close_date: Optional[datetime] = None diff --git a/tests/test_persistence.py b/tests/test_persistence.py index 1fced3e16..18a377ca3 100644 --- a/tests/test_persistence.py +++ b/tests/test_persistence.py @@ -1,14 +1,16 @@ # pragma pylint: disable=missing-docstring, C0103 +from types import FunctionType import logging from unittest.mock import MagicMock import arrow import pytest from sqlalchemy import create_engine +from sqlalchemy.sql.schema import Column from freqtrade import constants from freqtrade.exceptions import DependencyException, OperationalException -from freqtrade.persistence import Order, Trade, clean_dry_run_db, init_db +from freqtrade.persistence import LocalTrade, Order, Trade, clean_dry_run_db, init_db from tests.conftest import create_mock_trades, log_has, log_has_re @@ -1176,3 +1178,25 @@ def test_select_order(fee): assert order.ft_order_side == 'stoploss' order = trades[4].select_order('sell', False) assert order is None + + +def test_Trade_object_idem(): + + assert issubclass(Trade, LocalTrade) + + trade = vars(Trade) + localtrade = vars(LocalTrade) + + # Parent (LocalTrade) should have the same attributes + for item in trade: + # Exclude private attributes and open_date (as it's not assigned a default) + if (not item.startswith('_') + and item not in ('delete', 'session', 'query', 'open_date')): + assert item in localtrade + + # Fails if only a column is added without corresponding parent field + for item in localtrade: + if (not item.startswith('__') + and item not in ('trades', ) + and type(getattr(LocalTrade, item)) not in (property, FunctionType)): + assert item in trade From fc256749af4a29ee30353a2ae4edc17f9b3a4021 Mon Sep 17 00:00:00 2001 From: Matthias Date: Mon, 22 Feb 2021 06:54:33 +0100 Subject: [PATCH 048/187] Add test for backtesting _enter_trade --- freqtrade/optimize/backtesting.py | 7 +++-- tests/data/test_btanalysis.py | 1 - tests/optimize/test_backtesting.py | 41 +++++++++++++++++++++++++++++- tests/test_persistence.py | 3 +-- 4 files changed, 44 insertions(+), 8 deletions(-) diff --git a/freqtrade/optimize/backtesting.py b/freqtrade/optimize/backtesting.py index aeafaffd3..9a4a3787a 100644 --- a/freqtrade/optimize/backtesting.py +++ b/freqtrade/optimize/backtesting.py @@ -22,8 +22,7 @@ from freqtrade.exchange import timeframe_to_minutes, timeframe_to_seconds from freqtrade.mixins import LoggingMixin from freqtrade.optimize.optimize_reports import (generate_backtest_stats, show_backtest_results, store_backtest_stats) -from freqtrade.persistence import PairLocks, Trade -from freqtrade.persistence.models import LocalTrade +from freqtrade.persistence import LocalTrade, PairLocks, Trade from freqtrade.plugins.pairlistmanager import PairListManager from freqtrade.plugins.protectionmanager import ProtectionManager from freqtrade.resolvers import ExchangeResolver, StrategyResolver @@ -267,13 +266,13 @@ class Backtesting: return None - def _enter_trade(self, pair: str, row, max_open_trades: int, + def _enter_trade(self, pair: str, row: List, max_open_trades: int, open_trade_count: int) -> Optional[LocalTrade]: try: stake_amount = self.wallets.get_trade_stake_amount( pair, max_open_trades - open_trade_count, None) except DependencyException: - stake_amount = 0 + return None min_stake_amount = self.exchange.get_min_pair_stake_amount(pair, row[OPEN_IDX], -0.05) if stake_amount and (not min_stake_amount or stake_amount > min_stake_amount): # print(f"{pair}, {stake_amount}") diff --git a/tests/data/test_btanalysis.py b/tests/data/test_btanalysis.py index 538c89a90..e42c13e18 100644 --- a/tests/data/test_btanalysis.py +++ b/tests/data/test_btanalysis.py @@ -301,7 +301,6 @@ def test_calculate_csum(testdatadir): assert csum_min1 == csum_min + 5 assert csum_max1 == csum_max + 5 - with pytest.raises(ValueError, match='Trade dataframe empty.'): csum_min, csum_max = calculate_csum(DataFrame()) diff --git a/tests/optimize/test_backtesting.py b/tests/optimize/test_backtesting.py index eda8aac9d..354b3f6b0 100644 --- a/tests/optimize/test_backtesting.py +++ b/tests/optimize/test_backtesting.py @@ -17,8 +17,9 @@ from freqtrade.data.btanalysis import BT_DATA_COLUMNS, evaluate_result_multi from freqtrade.data.converter import clean_ohlcv_dataframe from freqtrade.data.dataprovider import DataProvider from freqtrade.data.history import get_timerange -from freqtrade.exceptions import OperationalException +from freqtrade.exceptions import DependencyException, OperationalException from freqtrade.optimize.backtesting import Backtesting +from freqtrade.persistence import LocalTrade from freqtrade.resolvers import StrategyResolver from freqtrade.state import RunMode from freqtrade.strategy.interface import SellType @@ -447,6 +448,44 @@ def test_backtesting_pairlist_list(default_conf, mocker, caplog, testdatadir, ti Backtesting(default_conf) +def test_backtest__enter_trade(default_conf, fee, mocker, testdatadir) -> None: + default_conf['ask_strategy']['use_sell_signal'] = False + mocker.patch('freqtrade.exchange.Exchange.get_fee', fee) + mocker.patch("freqtrade.exchange.Exchange.get_min_pair_stake_amount", return_value=0.00001) + patch_exchange(mocker) + default_conf['stake_amount'] = 'unlimited' + backtesting = Backtesting(default_conf) + pair = 'UNITTEST/BTC' + row = [ + pd.Timestamp(year=2020, month=1, day=1, hour=5, minute=0), + 1, # Sell + 0.001, # Open + 0.0011, # Close + 0, # Sell + 0.00099, # Low + 0.0012, # High + ] + trade = backtesting._enter_trade(pair, row=row, max_open_trades=2, open_trade_count=0) + assert isinstance(trade, LocalTrade) + assert trade.stake_amount == 495 + + trade = backtesting._enter_trade(pair, row=row, max_open_trades=2, open_trade_count=2) + assert trade is None + + # Stake-amount too high! + mocker.patch("freqtrade.exchange.Exchange.get_min_pair_stake_amount", return_value=600.0) + + trade = backtesting._enter_trade(pair, row=row, max_open_trades=2, open_trade_count=0) + assert trade is None + + # Stake-amount too high! + mocker.patch("freqtrade.wallets.Wallets.get_trade_stake_amount", + side_effect=DependencyException) + + trade = backtesting._enter_trade(pair, row=row, max_open_trades=2, open_trade_count=0) + assert trade is None + + def test_backtest_one(default_conf, fee, mocker, testdatadir) -> None: default_conf['ask_strategy']['use_sell_signal'] = False mocker.patch('freqtrade.exchange.Exchange.get_fee', fee) diff --git a/tests/test_persistence.py b/tests/test_persistence.py index 18a377ca3..1a8124b00 100644 --- a/tests/test_persistence.py +++ b/tests/test_persistence.py @@ -1,12 +1,11 @@ # pragma pylint: disable=missing-docstring, C0103 -from types import FunctionType import logging +from types import FunctionType from unittest.mock import MagicMock import arrow import pytest from sqlalchemy import create_engine -from sqlalchemy.sql.schema import Column from freqtrade import constants from freqtrade.exceptions import DependencyException, OperationalException From d3fb473e578e0f1ea5b7275d7023e6f8088d2583 Mon Sep 17 00:00:00 2001 From: Matthias Date: Wed, 24 Feb 2021 20:21:50 +0100 Subject: [PATCH 049/187] Improve backtesting documentation --- docs/backtesting.md | 86 ++++++++++++++++++++++++++------------------- 1 file changed, 50 insertions(+), 36 deletions(-) diff --git a/docs/backtesting.md b/docs/backtesting.md index bac12dae0..9fa9025d8 100644 --- a/docs/backtesting.md +++ b/docs/backtesting.md @@ -95,8 +95,7 @@ Strategy arguments: ## Test your strategy with Backtesting Now you have good Buy and Sell strategies and some historic data, you want to test it against -real data. This is what we call -[backtesting](https://en.wikipedia.org/wiki/Backtesting). +real data. This is what we call [backtesting](https://en.wikipedia.org/wiki/Backtesting). Backtesting will use the crypto-currencies (pairs) from your config file and load historical candle (OHCLV) data from `user_data/data/` by default. If no data is available for the exchange / pair / timeframe combination, backtesting will ask you to download them first using `freqtrade download-data`. @@ -104,6 +103,8 @@ For details on downloading, please refer to the [Data Downloading](data-download The result of backtesting will confirm if your bot has better odds of making a profit than a loss. +All profit calculations include fees, and freqtrade will use the exchange's default fees for the calculation. + !!! Warning "Using dynamic pairlists for backtesting" Using dynamic pairlists is possible, however it relies on the current market conditions - which will not reflect the historic status of the pairlist. Also, when using pairlists other than StaticPairlist, reproducability of backtesting-results cannot be guaranteed. @@ -111,38 +112,46 @@ The result of backtesting will confirm if your bot has better odds of making a p To achieve reproducible results, best generate a pairlist via the [`test-pairlist`](utils.md#test-pairlist) command and use that as static pairlist. -### Run a backtesting against the currencies listed in your config file +### Example backtesting commands -#### With 5 min candle (OHLCV) data (per default) +With 5 min candle (OHLCV) data (per default) ```bash -freqtrade backtesting +freqtrade backtesting --strategy AwesomeStrategy ``` -#### With 1 min candle (OHLCV) data +Where `--strategy AwesomeStrategy` / `-s AwesomeStrategy` refers to the class name of the strategy, which is within a python file in the `user_data/strategies` directory. + +--- + +With 1 min candle (OHLCV) data ```bash -freqtrade backtesting --timeframe 1m +freqtrade backtesting --strategy AwesomeStrategy --timeframe 1m ``` -#### Using a different on-disk historical candle (OHLCV) data source +--- + +Providing a custom starting balance of 1000 (in stake currency) + +```bash +freqtrade backtesting --strategy AwesomeStrategy --dry-run-wallet 1000 +``` + +--- + +Using a different on-disk historical candle (OHLCV) data source Assume you downloaded the history data from the Bittrex exchange and kept it in the `user_data/data/bittrex-20180101` directory. You can then use this data for backtesting as follows: ```bash -freqtrade --datadir user_data/data/bittrex-20180101 backtesting +freqtrade backtesting --strategy AwesomeStrategy --datadir user_data/data/bittrex-20180101 ``` -#### With a (custom) strategy file +--- -```bash -freqtrade backtesting -s SampleStrategy -``` - -Where `-s SampleStrategy` refers to the class name within the strategy file `sample_strategy.py` found in the `freqtrade/user_data/strategies` directory. - -#### Comparing multiple Strategies +Comparing multiple Strategies ```bash freqtrade backtesting --strategy-list SampleStrategy1 AwesomeStrategy --timeframe 5m @@ -150,23 +159,29 @@ freqtrade backtesting --strategy-list SampleStrategy1 AwesomeStrategy --timefram Where `SampleStrategy1` and `AwesomeStrategy` refer to class names of strategies. -#### Exporting trades to file +--- + +Exporting trades to file ```bash -freqtrade backtesting --export trades --config config.json --strategy SampleStrategy +freqtrade backtesting --strategy backtesting --export trades --config config.json ``` The exported trades can be used for [further analysis](#further-backtest-result-analysis), or can be used by the plotting script `plot_dataframe.py` in the scripts directory. -#### Exporting trades to file specifying a custom filename +--- + +Exporting trades to file specifying a custom filename ```bash -freqtrade backtesting --export trades --export-filename=backtest_samplestrategy.json +freqtrade backtesting --strategy backtesting --export trades --export-filename=backtest_samplestrategy.json ``` Please also read about the [strategy startup period](strategy-customization.md#strategy-startup-period). -#### Supplying custom fee value +--- + +Supplying custom fee value Sometimes your account has certain fee rebates (fee reductions starting with a certain account size or monthly volume), which are not visible to ccxt. To account for this in backtesting, you can use the `--fee` command line option to supply this value to backtesting. @@ -181,26 +196,26 @@ freqtrade backtesting --fee 0.001 !!! Note Only supply this option (or the corresponding configuration parameter) if you want to experiment with different fee values. By default, Backtesting fetches the default fee from the exchange pair/market info. -#### Running backtest with smaller testset by using timerange +--- -Use the `--timerange` argument to change how much of the testset you want to use. +Running backtest with smaller test-set by using timerange +Use the `--timerange` argument to change how much of the test-set you want to use. -For example, running backtesting with the `--timerange=20190501-` option will use all available data starting with May 1st, 2019 from your inputdata. +For example, running backtesting with the `--timerange=20190501-` option will use all available data starting with May 1st, 2019 from your input data. ```bash freqtrade backtesting --timerange=20190501- ``` -You can also specify particular dates or a range span indexed by start and stop. +You can also specify particular date ranges. The full timerange specification: -- Use tickframes till 2018/01/31: `--timerange=-20180131` -- Use tickframes since 2018/01/31: `--timerange=20180131-` -- Use tickframes since 2018/01/31 till 2018/03/01 : `--timerange=20180131-20180301` -- Use tickframes between POSIX timestamps 1527595200 1527618600: - `--timerange=1527595200-1527618600` +- Use data until 2018/01/31: `--timerange=-20180131` +- Use data since 2018/01/31: `--timerange=20180131-` +- Use data since 2018/01/31 till 2018/03/01 : `--timerange=20180131-20180301` +- Use data between POSIX / epoch timestamps 1527595200 1527618600: `--timerange=1527595200-1527618600` ## Understand the backtesting result @@ -296,9 +311,9 @@ here: The bot has made `429` trades for an average duration of `4:12:00`, with a performance of `76.20%` (profit), that means it has earned a total of `0.00762792 BTC` starting with a capital of 0.01 BTC. -The column `avg profit %` shows the average profit for all trades made while the column `cum profit %` sums up all the profits/losses. -The column `tot profit %` shows instead the total profit % in relation to allocated capital (`max_open_trades * stake_amount`). -In the above results we have `max_open_trades=2` and `stake_amount=0.005` in config so `tot_profit %` will be `(76.20/100) * (0.005 * 2) =~ 0.00762792 BTC`. +The column `Avg Profit %` shows the average profit for all trades made while the column `Cum Profit %` sums up all the profits/losses. +The column `Tot Profit %` shows instead the total profit % in relation to allocated capital (`max_open_trades * stake_amount`). +In the above results we have `max_open_trades=2` and `stake_amount=0.005` in config so `Tot Profit %` will be `(76.20/100) * (0.005 * 2) =~ 0.00762792 BTC`. Your strategy performance is influenced by your buy strategy, your sell strategy, and also by the `minimal_roi` and `stop_loss` you have set. @@ -452,6 +467,5 @@ Detailed output for all strategies one after the other will be available, so mak ## Next step -Great, your strategy is profitable. What if the bot can give your the -optimal parameters to use for your strategy? +Great, your strategy is profitable. What if the bot can give your the optimal parameters to use for your strategy? Your next step is to learn [how to find optimal parameters with Hyperopt](hyperopt.md) From 86f9409fd293604e03408e89beb460078768d103 Mon Sep 17 00:00:00 2001 From: Matthias Date: Thu, 25 Feb 2021 20:14:33 +0100 Subject: [PATCH 050/187] fix --stake-amount parameter --- freqtrade/commands/cli_options.py | 1 - freqtrade/configuration/configuration.py | 7 +++++++ tests/test_configuration.py | 5 ++++- 3 files changed, 11 insertions(+), 2 deletions(-) diff --git a/freqtrade/commands/cli_options.py b/freqtrade/commands/cli_options.py index 90ebb5e6a..3b27237da 100644 --- a/freqtrade/commands/cli_options.py +++ b/freqtrade/commands/cli_options.py @@ -133,7 +133,6 @@ AVAILABLE_CLI_OPTIONS = { "stake_amount": Arg( '--stake-amount', help='Override the value of the `stake_amount` configuration setting.', - type=float, ), # Backtesting "position_stacking": Arg( diff --git a/freqtrade/configuration/configuration.py b/freqtrade/configuration/configuration.py index 6295d01d4..88447e490 100644 --- a/freqtrade/configuration/configuration.py +++ b/freqtrade/configuration/configuration.py @@ -229,6 +229,13 @@ class Configuration: elif config['runmode'] in NON_UTIL_MODES: logger.info('Using max_open_trades: %s ...', config.get('max_open_trades')) + if self.args.get('stake_amount', None): + # Convert explicitly to float to support CLI argument for both unlimited and value + try: + self.args['stake_amount'] = float(self.args['stake_amount']) + except ValueError: + pass + self._args_to_config(config, argname='stake_amount', logstring='Parameter --stake-amount detected, ' 'overriding stake_amount to: {} ...') diff --git a/tests/test_configuration.py b/tests/test_configuration.py index 94c3e24f6..6b3df392b 100644 --- a/tests/test_configuration.py +++ b/tests/test_configuration.py @@ -430,7 +430,8 @@ def test_setup_configuration_with_arguments(mocker, default_conf, caplog) -> Non '--enable-position-stacking', '--disable-max-market-positions', '--timerange', ':100', - '--export', '/bar/foo' + '--export', '/bar/foo', + '--stake-amount', 'unlimited' ] args = Arguments(arglist).get_parsed_arg() @@ -463,6 +464,8 @@ def test_setup_configuration_with_arguments(mocker, default_conf, caplog) -> Non assert 'export' in config assert log_has('Parameter --export detected: {} ...'.format(config['export']), caplog) + assert 'stake_amount' in config + assert config['stake_amount'] == 'unlimited' def test_setup_configuration_with_stratlist(mocker, default_conf, caplog) -> None: From 98f3142b30e2067b4ead4e3dec51848d56a9c0cf Mon Sep 17 00:00:00 2001 From: Matthias Date: Fri, 26 Feb 2021 19:48:06 +0100 Subject: [PATCH 051/187] Improve handling of backtesting params --- freqtrade/commands/cli_options.py | 2 +- freqtrade/commands/optimize_commands.py | 11 ++++++++--- freqtrade/configuration/configuration.py | 6 +++--- freqtrade/optimize/backtesting.py | 2 +- tests/optimize/test_backtesting.py | 17 +++++++++++++---- tests/optimize/test_hyperopt.py | 17 +++++++++++++---- 6 files changed, 39 insertions(+), 16 deletions(-) diff --git a/freqtrade/commands/cli_options.py b/freqtrade/commands/cli_options.py index 3b27237da..15c13cec9 100644 --- a/freqtrade/commands/cli_options.py +++ b/freqtrade/commands/cli_options.py @@ -111,7 +111,7 @@ AVAILABLE_CLI_OPTIONS = { action='store_true', ), "dry_run_wallet": Arg( - '--dry-run-wallet', + '--dry-run-wallet', '--starting-balance', help='Starting balance, used for backtesting / hyperopt and dry-runs.', type=float, ), diff --git a/freqtrade/commands/optimize_commands.py b/freqtrade/commands/optimize_commands.py index bf36972c4..130743f68 100644 --- a/freqtrade/commands/optimize_commands.py +++ b/freqtrade/commands/optimize_commands.py @@ -4,6 +4,7 @@ from typing import Any, Dict from freqtrade import constants from freqtrade.configuration import setup_utils_configuration from freqtrade.exceptions import OperationalException +from freqtrade.misc import round_coin_value from freqtrade.state import RunMode @@ -22,9 +23,13 @@ def setup_optimize_configuration(args: Dict[str, Any], method: RunMode) -> Dict[ RunMode.BACKTEST: 'backtesting', RunMode.HYPEROPT: 'hyperoptimization', } - if (method in no_unlimited_runmodes.keys() and - config['stake_amount'] != constants.UNLIMITED_STAKE_AMOUNT and - config['max_open_trades'] != float('inf')): + if method in no_unlimited_runmodes.keys(): + if (config['stake_amount'] != constants.UNLIMITED_STAKE_AMOUNT + and config['stake_amount'] > config['dry_run_wallet']): + wallet = round_coin_value(config['dry_run_wallet'], config['stake_currency']) + stake = round_coin_value(config['stake_amount'], config['stake_currency']) + raise OperationalException(f"Starting balance ({wallet}) " + f"is smaller than stake_amount {stake}.") pass # config['dry_run_wallet'] = config['stake_amount'] * \ # config['max_open_trades'] * (2 - config['tradable_balance_ratio']) diff --git a/freqtrade/configuration/configuration.py b/freqtrade/configuration/configuration.py index 88447e490..a40a4fd83 100644 --- a/freqtrade/configuration/configuration.py +++ b/freqtrade/configuration/configuration.py @@ -214,9 +214,6 @@ class Configuration: self._args_to_config( config, argname='enable_protections', logstring='Parameter --enable-protections detected, enabling Protections. ...') - # Setting max_open_trades to infinite if -1 - if config.get('max_open_trades') == -1: - config['max_open_trades'] = float('inf') if 'use_max_market_positions' in self.args and not self.args["use_max_market_positions"]: config.update({'use_max_market_positions': False}) @@ -228,6 +225,9 @@ class Configuration: 'overriding max_open_trades to: %s ...', config.get('max_open_trades')) elif config['runmode'] in NON_UTIL_MODES: logger.info('Using max_open_trades: %s ...', config.get('max_open_trades')) + # Setting max_open_trades to infinite if -1 + if config.get('max_open_trades') == -1: + config['max_open_trades'] = float('inf') if self.args.get('stake_amount', None): # Convert explicitly to float to support CLI argument for both unlimited and value diff --git a/freqtrade/optimize/backtesting.py b/freqtrade/optimize/backtesting.py index 9a4a3787a..13ffc1d25 100644 --- a/freqtrade/optimize/backtesting.py +++ b/freqtrade/optimize/backtesting.py @@ -128,7 +128,7 @@ class Backtesting: PairLocks.use_db = True Trade.use_db = True - def _set_strategy(self, strategy): + def _set_strategy(self, strategy: IStrategy): """ Load strategy into backtesting """ diff --git a/tests/optimize/test_backtesting.py b/tests/optimize/test_backtesting.py index 354b3f6b0..4bbfe8a78 100644 --- a/tests/optimize/test_backtesting.py +++ b/tests/optimize/test_backtesting.py @@ -9,7 +9,6 @@ import pandas as pd import pytest from arrow import Arrow -from freqtrade import constants from freqtrade.commands.optimize_commands import setup_optimize_configuration, start_backtesting from freqtrade.configuration import TimeRange from freqtrade.data import history @@ -232,8 +231,7 @@ def test_setup_bt_configuration_with_arguments(mocker, default_conf, caplog) -> assert log_has('Parameter --fee detected, setting fee to: {} ...'.format(config['fee']), caplog) -def test_setup_optimize_configuration_unlimited_stake_amount(mocker, default_conf, caplog) -> None: - default_conf['stake_amount'] = constants.UNLIMITED_STAKE_AMOUNT +def test_setup_optimize_configuration_stake_amount(mocker, default_conf, caplog) -> None: patched_configuration_load_config_file(mocker, default_conf) @@ -241,12 +239,23 @@ def test_setup_optimize_configuration_unlimited_stake_amount(mocker, default_con 'backtesting', '--config', 'config.json', '--strategy', 'DefaultStrategy', + '--stake-amount', '1', + '--starting-balance', '2' ] - # TODO: does this test still make sense? conf = setup_optimize_configuration(get_args(args), RunMode.BACKTEST) assert isinstance(conf, dict) + args = [ + 'backtesting', + '--config', 'config.json', + '--strategy', 'DefaultStrategy', + '--stake-amount', '1', + '--starting-balance', '0.5' + ] + with pytest.raises(OperationalException, match=r"Starting balance .* smaller .*"): + setup_optimize_configuration(get_args(args), RunMode.BACKTEST) + def test_start(mocker, fee, default_conf, caplog) -> None: start_mock = MagicMock() diff --git a/tests/optimize/test_hyperopt.py b/tests/optimize/test_hyperopt.py index 88a4cea2d..9ebdad2b5 100644 --- a/tests/optimize/test_hyperopt.py +++ b/tests/optimize/test_hyperopt.py @@ -12,7 +12,6 @@ import pytest from arrow import Arrow from filelock import Timeout -from freqtrade import constants from freqtrade.commands.optimize_commands import setup_optimize_configuration, start_hyperopt from freqtrade.data.history import load_data from freqtrade.exceptions import OperationalException @@ -130,8 +129,7 @@ def test_setup_hyperopt_configuration_with_arguments(mocker, default_conf, caplo assert log_has('Parameter --print-all detected ...', caplog) -def test_setup_hyperopt_configuration_unlimited_stake_amount(mocker, default_conf) -> None: - default_conf['stake_amount'] = constants.UNLIMITED_STAKE_AMOUNT +def test_setup_hyperopt_configuration_stake_amount(mocker, default_conf) -> None: patched_configuration_load_config_file(mocker, default_conf) @@ -139,11 +137,22 @@ def test_setup_hyperopt_configuration_unlimited_stake_amount(mocker, default_con 'hyperopt', '--config', 'config.json', '--hyperopt', 'DefaultHyperOpt', + '--stake-amount', '1', + '--starting-balance', '2' ] - # TODO: does this test still make sense? conf = setup_optimize_configuration(get_args(args), RunMode.HYPEROPT) assert isinstance(conf, dict) + args = [ + 'hyperopt', + '--config', 'config.json', + '--strategy', 'DefaultStrategy', + '--stake-amount', '1', + '--starting-balance', '0.5' + ] + with pytest.raises(OperationalException, match=r"Starting balance .* smaller .*"): + setup_optimize_configuration(get_args(args), RunMode.HYPEROPT) + def test_hyperoptresolver(mocker, default_conf, caplog) -> None: patched_configuration_load_config_file(mocker, default_conf) From f5bb5f56f1aeefe13075f418d3ea15f24969fdbb Mon Sep 17 00:00:00 2001 From: Matthias Date: Fri, 26 Feb 2021 19:53:29 +0100 Subject: [PATCH 052/187] Update documentation with backtesting compounding possibilities --- docs/backtesting.md | 15 ++++++++++++--- docs/bot-usage.md | 2 +- docs/configuration.md | 7 ++++--- docs/hyperopt.md | 2 +- freqtrade/commands/optimize_commands.py | 6 ------ 5 files changed, 18 insertions(+), 14 deletions(-) diff --git a/docs/backtesting.md b/docs/backtesting.md index 9fa9025d8..96911763e 100644 --- a/docs/backtesting.md +++ b/docs/backtesting.md @@ -49,7 +49,7 @@ optional arguments: Enable protections for backtesting.Will slow backtesting down by a considerable amount, but will include configured protections - --dry-run-wallet DRY_RUN_WALLET + --dry-run-wallet DRY_RUN_WALLET, --starting-balance DRY_RUN_WALLET Starting balance, used for backtesting / hyperopt and dry-runs. --strategy-list STRATEGY_LIST [STRATEGY_LIST ...] @@ -108,10 +108,19 @@ All profit calculations include fees, and freqtrade will use the exchange's defa !!! Warning "Using dynamic pairlists for backtesting" Using dynamic pairlists is possible, however it relies on the current market conditions - which will not reflect the historic status of the pairlist. Also, when using pairlists other than StaticPairlist, reproducability of backtesting-results cannot be guaranteed. - Please read the [pairlists documentation](plugins.md#pairlists) for more information. - + Please read the [pairlists documentation](plugins.md#pairlists) for more information. To achieve reproducible results, best generate a pairlist via the [`test-pairlist`](utils.md#test-pairlist) command and use that as static pairlist. +### Starting balance + +Backtesting will require a starting balance, which can be given as `--dry-run-wallet ` or `--starting-balance ` command line argument, or via `dry_run_wallet` configuration setting. +This amount must be higher than `stake_amount`, otherwise the bot will not be able to simulate any trade. + +### Dynamic stake amount + +Backtesting supports [dynamic stake amount](configuration.md#dynamic-stake-amount) by configuring `stake_amount` as `"unlimited"`, which will split the starting balance into `max_open_trades` pieces. +Profits from early trades will result in subsequent higher stake amounts, resulting in compounding of profits over the backtesting period. + ### Example backtesting commands With 5 min candle (OHLCV) data (per default) diff --git a/docs/bot-usage.md b/docs/bot-usage.md index 4ff6168a0..b65220722 100644 --- a/docs/bot-usage.md +++ b/docs/bot-usage.md @@ -67,7 +67,7 @@ optional arguments: --sd-notify Notify systemd service manager. --dry-run Enforce dry-run for trading (removes Exchange secrets and simulates trades). - --dry-run-wallet DRY_RUN_WALLET + --dry-run-wallet DRY_RUN_WALLET, --starting-balance DRY_RUN_WALLET Starting balance, used for backtesting / hyperopt and dry-runs. diff --git a/docs/configuration.md b/docs/configuration.md index 663d9c5b2..2cc22d6ec 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -218,11 +218,12 @@ To allow the bot to trade all the available `stake_currency` in your account (mi "tradable_balance_ratio": 0.99, ``` -!!! Note - This configuration will allow increasing / decreasing stakes depending on the performance of the bot (lower stake if bot is loosing, higher stakes if the bot has a winning record, since higher balances are available). +!!! Tip "Compounding profits" + This configuration will allow increasing / decreasing stakes depending on the performance of the bot (lower stake if bot is loosing, higher stakes if the bot has a winning record, since higher balances are available), and will result in profit compounding. !!! Note "When using Dry-Run Mode" - When using `"stake_amount" : "unlimited",` in combination with Dry-Run, the balance will be simulated starting with a stake of `dry_run_wallet` which will evolve over time. It is therefore important to set `dry_run_wallet` to a sensible value (like 0.05 or 0.01 for BTC and 1000 or 100 for USDT, for example), otherwise it may simulate trades with 100 BTC (or more) or 0.05 USDT (or less) at once - which may not correspond to your real available balance or is less than the exchange minimal limit for the order amount for the stake currency. + When using `"stake_amount" : "unlimited",` in combination with Dry-Run, Backtesting or Hyperopt, the balance will be simulated starting with a stake of `dry_run_wallet` which will evolve over time. + It is therefore important to set `dry_run_wallet` to a sensible value (like 0.05 or 0.01 for BTC and 1000 or 100 for USDT, for example), otherwise it may simulate trades with 100 BTC (or more) or 0.05 USDT (or less) at once - which may not correspond to your real available balance or is less than the exchange minimal limit for the order amount for the stake currency. --8<-- "includes/pricing.md" diff --git a/docs/hyperopt.md b/docs/hyperopt.md index ee3d75d0b..d6959b457 100644 --- a/docs/hyperopt.md +++ b/docs/hyperopt.md @@ -83,7 +83,7 @@ optional arguments: Enable protections for backtesting.Will slow backtesting down by a considerable amount, but will include configured protections - --dry-run-wallet DRY_RUN_WALLET + --dry-run-wallet DRY_RUN_WALLET, --starting-balance DRY_RUN_WALLET Starting balance, used for backtesting / hyperopt and dry-runs. -e INT, --epochs INT Specify number of epochs (default: 100). diff --git a/freqtrade/commands/optimize_commands.py b/freqtrade/commands/optimize_commands.py index 130743f68..6323bc2b1 100644 --- a/freqtrade/commands/optimize_commands.py +++ b/freqtrade/commands/optimize_commands.py @@ -30,12 +30,6 @@ def setup_optimize_configuration(args: Dict[str, Any], method: RunMode) -> Dict[ stake = round_coin_value(config['stake_amount'], config['stake_currency']) raise OperationalException(f"Starting balance ({wallet}) " f"is smaller than stake_amount {stake}.") - pass - # config['dry_run_wallet'] = config['stake_amount'] * \ - # config['max_open_trades'] * (2 - config['tradable_balance_ratio']) - - # logger.warning(f"Changing dry-run-wallet to {config['dry_run_wallet']} " - # "(max_open_trades * stake_amount).") return config From fb489c11c921b77bf6f029c59ecb71f4d8712486 Mon Sep 17 00:00:00 2001 From: Matthias Date: Sat, 27 Feb 2021 10:07:02 +0100 Subject: [PATCH 053/187] Improve test-coverage of pairlocks --- tests/plugins/test_pairlocks.py | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/tests/plugins/test_pairlocks.py b/tests/plugins/test_pairlocks.py index dfcbff0ed..fce3a8cd1 100644 --- a/tests/plugins/test_pairlocks.py +++ b/tests/plugins/test_pairlocks.py @@ -73,9 +73,13 @@ def test_PairLocks(use_db): assert PairLocks.is_pair_locked('XRP/USDT', lock_time + timedelta(minutes=-50)) if use_db: - assert len(PairLock.query.all()) > 0 + locks = PairLocks.get_all_locks() + locks_db = PairLock.query.all() + assert len(locks) == len(locks_db) + assert len(locks_db) > 0 else: # Nothing was pushed to the database + assert len(PairLocks.get_all_locks()) > 0 assert len(PairLock.query.all()) == 0 # Reset use-db variable PairLocks.reset_locks() From f65092459a39ccd7238550a9518f989bab41feb7 Mon Sep 17 00:00:00 2001 From: Matthias Date: Sat, 27 Feb 2021 10:14:25 +0100 Subject: [PATCH 054/187] Fix optimize_reports test --- tests/optimize/test_optimize_reports.py | 32 ++++++++++++++++++------- 1 file changed, 23 insertions(+), 9 deletions(-) diff --git a/tests/optimize/test_optimize_reports.py b/tests/optimize/test_optimize_reports.py index ca6a4ab01..8119c732b 100644 --- a/tests/optimize/test_optimize_reports.py +++ b/tests/optimize/test_optimize_reports.py @@ -102,6 +102,7 @@ def test_generate_backtest_stats(default_conf, testdatadir): # Above sample had no loosing trade assert strat_stats['max_drawdown'] == 0.0 + # Retry with losing trade results = {'DefStrat': { 'results': pd.DataFrame( {"pair": ["UNITTEST/BTC", "UNITTEST/BTC", "UNITTEST/BTC", "UNITTEST/BTC"], @@ -118,18 +119,31 @@ def test_generate_backtest_stats(default_conf, testdatadir): "open_rate": [0.002543, 0.003003, 0.003089, 0.003214], "close_rate": [0.002546, 0.003014, 0.0032903, 0.003217], "trade_duration": [123, 34, 31, 14], - "open_at_end": [False, False, False, True], - "sell_reason": [SellType.ROI, SellType.STOP_LOSS, - SellType.ROI, SellType.FORCE_SELL] + "is_open": [False, False, False, True], + "stake_amount": [0.01, 0.01, 0.01, 0.01], + "sell_reason": [SellType.ROI, SellType.ROI, + SellType.STOP_LOSS, SellType.FORCE_SELL] }), - 'config': default_conf} + 'config': default_conf, + 'locks': [], + 'final_balance': 1000.02, + 'backtest_start_time': Arrow.utcnow().int_timestamp, + 'backtest_end_time': Arrow.utcnow().int_timestamp, + } } - assert strat_stats['max_drawdown'] == 0.0 - assert strat_stats['drawdown_start'] == datetime(1970, 1, 1, tzinfo=timezone.utc) - assert strat_stats['drawdown_end'] == datetime(1970, 1, 1, tzinfo=timezone.utc) - assert strat_stats['drawdown_end_ts'] == 0 - assert strat_stats['drawdown_start_ts'] == 0 + stats = generate_backtest_stats(btdata, results, min_date, max_date) + assert isinstance(stats, dict) + assert 'strategy' in stats + assert 'DefStrat' in stats['strategy'] + assert 'strategy_comparison' in stats + strat_stats = stats['strategy']['DefStrat'] + + assert strat_stats['max_drawdown'] == 0.013803 + assert strat_stats['drawdown_start'] == datetime(2017, 11, 14, 22, 10, tzinfo=timezone.utc) + assert strat_stats['drawdown_end'] == datetime(2017, 11, 14, 22, 43, tzinfo=timezone.utc) + assert strat_stats['drawdown_end_ts'] == 1510699380000 + assert strat_stats['drawdown_start_ts'] == 1510697400000 assert strat_stats['pairlist'] == ['UNITTEST/BTC'] # Test storing stats From 324b9dbdff126f53470919398081b2374d30c8b3 Mon Sep 17 00:00:00 2001 From: Matthias Date: Sat, 27 Feb 2021 10:31:21 +0100 Subject: [PATCH 055/187] Simplify wallet code --- freqtrade/optimize/backtesting.py | 3 +-- freqtrade/wallets.py | 7 +++---- 2 files changed, 4 insertions(+), 6 deletions(-) diff --git a/freqtrade/optimize/backtesting.py b/freqtrade/optimize/backtesting.py index 13ffc1d25..b9ae096e2 100644 --- a/freqtrade/optimize/backtesting.py +++ b/freqtrade/optimize/backtesting.py @@ -115,8 +115,7 @@ class Backtesting: if self.config.get('enable_protections', False): self.protections = ProtectionManager(self.config) - self.wallets = Wallets(self.config, self.exchange) - self.wallets._log = False + self.wallets = Wallets(self.config, self.exchange, log=False) # Get maximum required startup period self.required_startup = max([strat.startup_candle_count for strat in self.strategylist]) diff --git a/freqtrade/wallets.py b/freqtrade/wallets.py index c2085641e..553f7c61d 100644 --- a/freqtrade/wallets.py +++ b/freqtrade/wallets.py @@ -27,15 +27,14 @@ class Wallet(NamedTuple): class Wallets: - def __init__(self, config: dict, exchange: Exchange, skip_update: bool = False) -> None: + def __init__(self, config: dict, exchange: Exchange, log: bool = True) -> None: self._config = config - self._log = True + self._log = log self._exchange = exchange self._wallets: Dict[str, Wallet] = {} self.start_cap = config['dry_run_wallet'] self._last_wallet_refresh = 0 - if not skip_update: - self.update() + self.update() def get_free(self, currency: str) -> float: balance = self._wallets.get(currency) From 6018a0534367bff3895778a50cec945d03d1a0a5 Mon Sep 17 00:00:00 2001 From: Matthias Date: Sat, 27 Feb 2021 10:45:22 +0100 Subject: [PATCH 056/187] Improve backtest documentation --- docs/backtesting.md | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/docs/backtesting.md b/docs/backtesting.md index 96911763e..29ddb494b 100644 --- a/docs/backtesting.md +++ b/docs/backtesting.md @@ -108,12 +108,13 @@ All profit calculations include fees, and freqtrade will use the exchange's defa !!! Warning "Using dynamic pairlists for backtesting" Using dynamic pairlists is possible, however it relies on the current market conditions - which will not reflect the historic status of the pairlist. Also, when using pairlists other than StaticPairlist, reproducability of backtesting-results cannot be guaranteed. - Please read the [pairlists documentation](plugins.md#pairlists) for more information. + Please read the [pairlists documentation](plugins.md#pairlists) for more information. + To achieve reproducible results, best generate a pairlist via the [`test-pairlist`](utils.md#test-pairlist) command and use that as static pairlist. ### Starting balance -Backtesting will require a starting balance, which can be given as `--dry-run-wallet ` or `--starting-balance ` command line argument, or via `dry_run_wallet` configuration setting. +Backtesting will require a starting balance, which can be provided as `--dry-run-wallet ` or `--starting-balance ` command line argument, or via `dry_run_wallet` configuration setting. This amount must be higher than `stake_amount`, otherwise the bot will not be able to simulate any trade. ### Dynamic stake amount @@ -281,7 +282,7 @@ A backtesting result will look like that: | Absolute profit | 0.00762792 BTC | | Total profit % | 76.2% | | Trades per day | 3.575 | -| Avg. stake amount | 0.001 | +| Avg. stake amount | 0.001 BTC | | Total trade volume | 0.429 BTC | | | | | Best Pair | LSK/BTC 26.26% | @@ -368,7 +369,7 @@ It contains some useful key metrics about performance of your strategy on backte | Absolute profit | 0.00762792 BTC | | Total profit % | 76.2% | | Trades per day | 3.575 | -| Avg. stake amount | 0.001 | +| Avg. stake amount | 0.001 BTC | | Total trade volume | 0.429 BTC | | | | | Best Pair | LSK/BTC 26.26% | @@ -398,7 +399,7 @@ It contains some useful key metrics about performance of your strategy on backte - `Max open trades`: Setting of `max_open_trades` (or `--max-open-trades`) - or number of pairs in the pairlist (whatever is lower). - `Total trades`: Identical to the total trades of the backtest output table. - `Starting balance`: Start balance - as given by dry-run-wallet (config or command line). -- `End balance`: Final balance - starting balance + absolute profit. +- `Final balance`: Final balance - starting balance + absolute profit. - `Absolute profit`: Profit made in stake currency. - `Total profit %`: Total profit. Aligned to the `TOTAL` row's `Tot Profit %` from the first table. Calculated as `(End capital − Starting capital) / Starting capital`. - `Trades per day`: Total trades divided by the backtesting duration in days (this will give you information about how many trades to expect from the strategy). From b2e9295d7f86688e40278ebe253c81b7b6a6450e Mon Sep 17 00:00:00 2001 From: Matthias Date: Sat, 27 Feb 2021 19:57:42 +0100 Subject: [PATCH 057/187] Small stylistic fixes --- freqtrade/optimize/backtesting.py | 1 - freqtrade/persistence/models.py | 7 ++++--- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/freqtrade/optimize/backtesting.py b/freqtrade/optimize/backtesting.py index b9ae096e2..1b6d2e89c 100644 --- a/freqtrade/optimize/backtesting.py +++ b/freqtrade/optimize/backtesting.py @@ -274,7 +274,6 @@ class Backtesting: return None min_stake_amount = self.exchange.get_min_pair_stake_amount(pair, row[OPEN_IDX], -0.05) if stake_amount and (not min_stake_amount or stake_amount > min_stake_amount): - # print(f"{pair}, {stake_amount}") # Enter trade trade = LocalTrade( pair=pair, diff --git a/freqtrade/persistence/models.py b/freqtrade/persistence/models.py index 51a48c246..3a6474696 100644 --- a/freqtrade/persistence/models.py +++ b/freqtrade/persistence/models.py @@ -652,9 +652,8 @@ class LocalTrade(): in stake currency """ if Trade.use_db: - total_open_stake_amount = Trade.session.query(func.sum(Trade.stake_amount))\ - .filter(Trade.is_open.is_(True))\ - .scalar() + total_open_stake_amount = Trade.session.query( + func.sum(Trade.stake_amount)).filter(Trade.is_open.is_(True)).scalar() else: total_open_stake_amount = sum( t.stake_amount for t in Trade.get_trades_proxy(is_open=True)) @@ -719,6 +718,8 @@ class Trade(_DECL_BASE, LocalTrade): """ Trade database model. Also handles updating and querying trades + + Note: Fields must be aligned with LocalTrade class """ __tablename__ = 'trades' From d9d5617432cc991a2de976f8e84cb107913ea0d9 Mon Sep 17 00:00:00 2001 From: Matthias Date: Sat, 27 Feb 2021 20:26:13 +0100 Subject: [PATCH 058/187] UPdate backtesting doc for total profit calc --- docs/backtesting.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/backtesting.md b/docs/backtesting.md index 29ddb494b..2e91b6e74 100644 --- a/docs/backtesting.md +++ b/docs/backtesting.md @@ -322,8 +322,8 @@ The bot has made `429` trades for an average duration of `4:12:00`, with a perfo earned a total of `0.00762792 BTC` starting with a capital of 0.01 BTC. The column `Avg Profit %` shows the average profit for all trades made while the column `Cum Profit %` sums up all the profits/losses. -The column `Tot Profit %` shows instead the total profit % in relation to allocated capital (`max_open_trades * stake_amount`). -In the above results we have `max_open_trades=2` and `stake_amount=0.005` in config so `Tot Profit %` will be `(76.20/100) * (0.005 * 2) =~ 0.00762792 BTC`. +The column `Tot Profit %` shows instead the total profit % in relation to the starting balance. +In the above results, we have a starting balance of 0.01 BTC and an absolute profit of 0.00762792 BTC - so the `Tot Profit %` will be `(0.00762792 / 0.01) * 100 ~= 76.2%`. Your strategy performance is influenced by your buy strategy, your sell strategy, and also by the `minimal_roi` and `stop_loss` you have set. From e791ff60423949ce8bc466370fa223ada47dc6c1 Mon Sep 17 00:00:00 2001 From: JoeSchr Date: Sat, 27 Feb 2021 23:28:26 +0100 Subject: [PATCH 059/187] Fix: custom_stoploss returns typo Afaik it should return -0.07 for 7% instead of -0.7. As a side note, really interesting would also be an example for greater than 100% profits. especially when trailing stoploss, like * Once profit is > 200% - stoploss will be set to 150%. I assume it could be as simple as ```py if current_profit > 2: return (-1.50 + current_profit) ```` to achieve it But I'm not quite confident, if the bot can handle stuff smaller than `-1`, since `1` and `-1` seem to have some special meaning and are often used to disable stoploss etc. --- docs/strategy-advanced.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/strategy-advanced.md b/docs/strategy-advanced.md index c051e2232..dcd340fd1 100644 --- a/docs/strategy-advanced.md +++ b/docs/strategy-advanced.md @@ -176,7 +176,7 @@ class AwesomeStrategy(IStrategy): if current_profit > 0.25: return (-0.15 + current_profit) if current_profit > 0.20: - return (-0.7 + current_profit) + return (-0.07 + current_profit) return 1 ``` From 05f057fe727769a186c8a84b43b2dcc98544d6c3 Mon Sep 17 00:00:00 2001 From: Matthias Date: Sun, 28 Feb 2021 08:47:56 +0100 Subject: [PATCH 060/187] Stringify favicon path potentially closes #4459 --- freqtrade/rpc/api_server/web_ui.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/freqtrade/rpc/api_server/web_ui.py b/freqtrade/rpc/api_server/web_ui.py index 6d7e77953..13d22a63e 100644 --- a/freqtrade/rpc/api_server/web_ui.py +++ b/freqtrade/rpc/api_server/web_ui.py @@ -10,7 +10,7 @@ router_ui = APIRouter() @router_ui.get('/favicon.ico', include_in_schema=False) async def favicon(): - return FileResponse(Path(__file__).parent / 'ui/favicon.ico') + return FileResponse(str(Path(__file__).parent / 'ui/favicon.ico')) @router_ui.get('/{rest_of_path:path}', include_in_schema=False) From 0895407811c9e9926c2939b30cb303d75b5a2bca Mon Sep 17 00:00:00 2001 From: Florian Reitmeir Date: Wed, 17 Feb 2021 23:09:39 +0100 Subject: [PATCH 061/187] add balance_dust_level parameter to make telegram less chatty --- freqtrade/constants.py | 3 ++- freqtrade/rpc/telegram.py | 6 ++++-- 2 files changed, 6 insertions(+), 3 deletions(-) diff --git a/freqtrade/constants.py b/freqtrade/constants.py index 802ddc2b1..51f178806 100644 --- a/freqtrade/constants.py +++ b/freqtrade/constants.py @@ -243,7 +243,8 @@ CONF_SCHEMA = { } } }, - 'required': ['enabled', 'token', 'chat_id'] + 'required': ['enabled', 'token', 'chat_id'], + 'balance_dust_level': 0.0001 }, 'webhook': { 'type': 'object', diff --git a/freqtrade/rpc/telegram.py b/freqtrade/rpc/telegram.py index 88019601c..ad3a00292 100644 --- a/freqtrade/rpc/telegram.py +++ b/freqtrade/rpc/telegram.py @@ -487,6 +487,8 @@ class Telegram(RPCHandler): result = self._rpc._rpc_balance(self._config['stake_currency'], self._config.get('fiat_display_currency', '')) + balance_dust_level = self._config['telegram'].get('balance_dust_level', 0.0001 ) + output = '' if self._config['dry_run']: output += ( @@ -496,7 +498,7 @@ class Telegram(RPCHandler): f"`{self._config['dry_run_wallet']}` {self._config['stake_currency']}.\n" ) for curr in result['currencies']: - if curr['est_stake'] > 0.0001: + if curr['est_stake'] > balance_dust_level: curr_output = ( f"*{curr['currency']}:*\n" f"\t`Available: {curr['free']:.8f}`\n" @@ -505,7 +507,7 @@ class Telegram(RPCHandler): f"\t`Est. {curr['stake']}: " f"{round_coin_value(curr['est_stake'], curr['stake'], False)}`\n") else: - curr_output = f"*{curr['currency']}:* not showing <1$ amount \n" + curr_output = f"*{curr['currency']}:* not showing <{balance_dust_level} {curr['stake']} amount \n" # Handle overflowing messsage length if len(output + curr_output) >= MAX_TELEGRAM_MESSAGE_LENGTH: From 9cb37409fda6b0d9235ec7069489fbd062f0b873 Mon Sep 17 00:00:00 2001 From: Matthias Date: Sun, 28 Feb 2021 09:56:29 +0100 Subject: [PATCH 062/187] Explicitly convert starting-balance to float --- freqtrade/optimize/optimize_reports.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/freqtrade/optimize/optimize_reports.py b/freqtrade/optimize/optimize_reports.py index e7111f20c..0de0c16a0 100644 --- a/freqtrade/optimize/optimize_reports.py +++ b/freqtrade/optimize/optimize_reports.py @@ -277,7 +277,7 @@ def generate_backtest_stats(btdata: Dict[str, DataFrame], 'sell_reason_summary': sell_reason_stats, 'left_open_trades': left_open_results, 'total_trades': len(results), - 'total_volume': results['stake_amount'].sum(), + 'total_volume': float(results['stake_amount'].sum()), 'avg_stake_amount': results['stake_amount'].mean(), 'profit_mean': results['profit_ratio'].mean() if len(results) > 0 else 0, 'profit_total': results['profit_abs'].sum() / starting_balance, From a13dc3cdde3d5fba51538a9ce303824bc06df574 Mon Sep 17 00:00:00 2001 From: Matthias Date: Sun, 28 Feb 2021 09:03:27 +0100 Subject: [PATCH 063/187] Use sensible defaults for balance_dust_level --- docs/configuration.md | 1 + docs/telegram-usage.md | 5 ++++- freqtrade/constants.py | 7 ++++++- freqtrade/rpc/telegram.py | 8 ++++++-- tests/rpc/test_rpc_telegram.py | 2 +- 5 files changed, 18 insertions(+), 5 deletions(-) diff --git a/docs/configuration.md b/docs/configuration.md index 0163e1671..99a5fea04 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -97,6 +97,7 @@ Mandatory parameters are marked as **Required**, which means that they are requi | `telegram.enabled` | Enable the usage of Telegram.
**Datatype:** Boolean | `telegram.token` | Your Telegram bot token. Only required if `telegram.enabled` is `true`.
**Keep it in secret, do not disclose publicly.**
**Datatype:** String | `telegram.chat_id` | Your personal Telegram account id. Only required if `telegram.enabled` is `true`.
**Keep it in secret, do not disclose publicly.**
**Datatype:** String +| `telegram.balance_dust_level` | Dust-level (in stake currency) - currencies with a balance below this will not be shown by `/balance`.
**Datatype:** float | `webhook.enabled` | Enable usage of Webhook notifications
**Datatype:** Boolean | `webhook.url` | URL for the webhook. Only required if `webhook.enabled` is `true`. See the [webhook documentation](webhook-config.md) for more details.
**Datatype:** String | `webhook.webhookbuy` | Payload to send on buy. Only required if `webhook.enabled` is `true`. See the [webhook documentation](webhook-config.md) for more details.
**Datatype:** String diff --git a/docs/telegram-usage.md b/docs/telegram-usage.md index 57f2e98bd..d4a6fb118 100644 --- a/docs/telegram-usage.md +++ b/docs/telegram-usage.md @@ -83,10 +83,13 @@ Example configuration showing the different settings: "sell": "on", "buy_cancel": "silent", "sell_cancel": "on" - } + }, + "balance_dust_level": 0.01 }, ``` +`balance_dust_level` will define what the `/balance` command takes as "dust" - Currencies with a balance below this will be shown. + ## Create a custom keyboard (command shortcut buttons) Telegram allows us to create a custom keyboard with buttons for commands. diff --git a/freqtrade/constants.py b/freqtrade/constants.py index 51f178806..c03bff0ad 100644 --- a/freqtrade/constants.py +++ b/freqtrade/constants.py @@ -54,6 +54,11 @@ DECIMALS_PER_COIN = { 'ETH': 5, } +DUST_PER_COIN = { + 'BTC': 0.0001, + 'ETH': 0.01 +} + # Soure files with destination directories within user-directory USER_DATA_FILES = { @@ -230,6 +235,7 @@ CONF_SCHEMA = { 'enabled': {'type': 'boolean'}, 'token': {'type': 'string'}, 'chat_id': {'type': 'string'}, + 'balance_dust_level': {'type': 'number', 'minimum': 0.0}, 'notification_settings': { 'type': 'object', 'properties': { @@ -244,7 +250,6 @@ CONF_SCHEMA = { } }, 'required': ['enabled', 'token', 'chat_id'], - 'balance_dust_level': 0.0001 }, 'webhook': { 'type': 'object', diff --git a/freqtrade/rpc/telegram.py b/freqtrade/rpc/telegram.py index ad3a00292..9d05ae142 100644 --- a/freqtrade/rpc/telegram.py +++ b/freqtrade/rpc/telegram.py @@ -17,6 +17,7 @@ from telegram.ext import CallbackContext, CommandHandler, Updater from telegram.utils.helpers import escape_markdown from freqtrade.__init__ import __version__ +from freqtrade.constants import DUST_PER_COIN from freqtrade.exceptions import OperationalException from freqtrade.misc import round_coin_value from freqtrade.rpc import RPC, RPCException, RPCHandler, RPCMessageType @@ -487,7 +488,9 @@ class Telegram(RPCHandler): result = self._rpc._rpc_balance(self._config['stake_currency'], self._config.get('fiat_display_currency', '')) - balance_dust_level = self._config['telegram'].get('balance_dust_level', 0.0001 ) + balance_dust_level = self._config['telegram'].get('balance_dust_level', 0.0) + if not balance_dust_level: + balance_dust_level = DUST_PER_COIN.get(self._config['stake_currency'], 1.0) output = '' if self._config['dry_run']: @@ -507,7 +510,8 @@ class Telegram(RPCHandler): f"\t`Est. {curr['stake']}: " f"{round_coin_value(curr['est_stake'], curr['stake'], False)}`\n") else: - curr_output = f"*{curr['currency']}:* not showing <{balance_dust_level} {curr['stake']} amount \n" + curr_output = (f"*{curr['currency']}:* not showing <{balance_dust_level} " + f"{curr['stake']} amount \n") # Handle overflowing messsage length if len(output + curr_output) >= MAX_TELEGRAM_MESSAGE_LENGTH: diff --git a/tests/rpc/test_rpc_telegram.py b/tests/rpc/test_rpc_telegram.py index f065bb4c5..922aa2de8 100644 --- a/tests/rpc/test_rpc_telegram.py +++ b/tests/rpc/test_rpc_telegram.py @@ -520,7 +520,7 @@ def test_telegram_balance_handle(default_conf, update, mocker, rpc_balance, tick assert 'Balance:' in result assert 'Est. BTC:' in result assert 'BTC: 12.00000000' in result - assert '*XRP:* not showing <1$ amount' in result + assert '*XRP:* not showing <0.0001 BTC amount' in result def test_balance_handle_empty_response(default_conf, update, mocker) -> None: From 94cab4ed71adf24974eaa7badf2aea836fe90a61 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Mon, 22 Feb 2021 05:27:47 +0000 Subject: [PATCH 064/187] Bump mypy from 0.790 to 0.812 Bumps [mypy](https://github.com/python/mypy) from 0.790 to 0.812. - [Release notes](https://github.com/python/mypy/releases) - [Commits](https://github.com/python/mypy/compare/v0.790...v0.812) Signed-off-by: dependabot[bot] --- requirements-dev.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/requirements-dev.txt b/requirements-dev.txt index fa0ead603..6ca1a4d9c 100644 --- a/requirements-dev.txt +++ b/requirements-dev.txt @@ -7,7 +7,7 @@ coveralls==3.0.0 flake8==3.8.4 flake8-type-annotations==0.1.0 flake8-tidy-imports==4.2.1 -mypy==0.790 +mypy==0.812 pytest==6.2.2 pytest-asyncio==0.14.0 pytest-cov==2.11.1 From aba034ff40ed0ff06904ffb7bb50a91576c4ab73 Mon Sep 17 00:00:00 2001 From: Matthias Date: Sun, 28 Feb 2021 10:56:51 +0100 Subject: [PATCH 065/187] Fix mypy problem after mypy 0.800 upgrade --- freqtrade/rpc/api_server/api_schemas.py | 8 +++++--- freqtrade/rpc/api_server/api_v1.py | 6 +++--- 2 files changed, 8 insertions(+), 6 deletions(-) diff --git a/freqtrade/rpc/api_server/api_schemas.py b/freqtrade/rpc/api_server/api_schemas.py index 050540cc6..2738e5368 100644 --- a/freqtrade/rpc/api_server/api_schemas.py +++ b/freqtrade/rpc/api_server/api_schemas.py @@ -1,5 +1,5 @@ from datetime import date, datetime -from typing import Any, Dict, List, Optional, TypeVar, Union +from typing import Any, Dict, List, Optional, Union from pydantic import BaseModel @@ -205,7 +205,8 @@ class TradeResponse(BaseModel): trades_count: int -ForceBuyResponse = TypeVar('ForceBuyResponse', TradeSchema, StatusMsg) +class ForceBuyResponse(BaseModel): + __root__: Union[TradeSchema, StatusMsg] class LockModel(BaseModel): @@ -267,7 +268,8 @@ class PlotConfig_(BaseModel): subplots: Optional[Dict[str, Any]] -PlotConfig = TypeVar('PlotConfig', PlotConfig_, Dict) +class PlotConfig(BaseModel): + __root__: Union[PlotConfig_, Dict] class StrategyListResponse(BaseModel): diff --git a/freqtrade/rpc/api_server/api_v1.py b/freqtrade/rpc/api_server/api_v1.py index 3588f2196..546b93afb 100644 --- a/freqtrade/rpc/api_server/api_v1.py +++ b/freqtrade/rpc/api_server/api_v1.py @@ -111,9 +111,9 @@ def forcebuy(payload: ForceBuyPayload, rpc: RPC = Depends(get_rpc)): trade = rpc._rpc_forcebuy(payload.pair, payload.price) if trade: - return trade.to_json() + return {'__root__': trade.to_json()} else: - return {"status": f"Error buying pair {payload.pair}."} + return {'__root__': {"status": f"Error buying pair {payload.pair}."}} @router.post('/forcesell', response_model=ResultMsg, tags=['trading']) @@ -183,7 +183,7 @@ def pair_history(pair: str, timeframe: str, timerange: str, strategy: str, @router.get('/plot_config', response_model=PlotConfig, tags=['candle data']) def plot_config(rpc: RPC = Depends(get_rpc)): - return rpc._rpc_plot_config() + return {'__root__': rpc._rpc_plot_config()} @router.get('/strategies', response_model=StrategyListResponse, tags=['strategy']) From 00747a3bc353bbb5fcc2f1acc463da6117899c9c Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Mon, 1 Mar 2021 05:31:43 +0000 Subject: [PATCH 066/187] Bump mkdocs-material from 6.2.8 to 7.0.3 Bumps [mkdocs-material](https://github.com/squidfunk/mkdocs-material) from 6.2.8 to 7.0.3. - [Release notes](https://github.com/squidfunk/mkdocs-material/releases) - [Changelog](https://github.com/squidfunk/mkdocs-material/blob/master/docs/changelog.md) - [Commits](https://github.com/squidfunk/mkdocs-material/compare/6.2.8...7.0.3) Signed-off-by: dependabot[bot] --- docs/requirements-docs.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/requirements-docs.txt b/docs/requirements-docs.txt index 94b2fca39..73ae3ad29 100644 --- a/docs/requirements-docs.txt +++ b/docs/requirements-docs.txt @@ -1,3 +1,3 @@ -mkdocs-material==6.2.8 +mkdocs-material==7.0.3 mdx_truly_sane_lists==1.2 pymdown-extensions==8.1.1 From d0fd3c289caa49c6ab41c576b1fa98772d735e68 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Mon, 1 Mar 2021 05:32:14 +0000 Subject: [PATCH 067/187] Bump ccxt from 1.42.19 to 1.42.47 Bumps [ccxt](https://github.com/ccxt/ccxt) from 1.42.19 to 1.42.47. - [Release notes](https://github.com/ccxt/ccxt/releases) - [Changelog](https://github.com/ccxt/ccxt/blob/master/doc/exchanges-by-country.rst) - [Commits](https://github.com/ccxt/ccxt/compare/1.42.19...1.42.47) Signed-off-by: dependabot[bot] --- requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/requirements.txt b/requirements.txt index d17070e34..1cd7d74df 100644 --- a/requirements.txt +++ b/requirements.txt @@ -1,7 +1,7 @@ numpy==1.20.1 pandas==1.2.2 -ccxt==1.42.19 +ccxt==1.42.47 # Pin cryptography for now due to rust build errors with piwheels cryptography==3.4.6 aiohttp==3.7.4 From 4537a48988e4a8f6a2703ad4276d7dd564fe5545 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Mon, 1 Mar 2021 05:32:17 +0000 Subject: [PATCH 068/187] Bump arrow from 0.17.0 to 1.0.2 Bumps [arrow](https://github.com/arrow-py/arrow) from 0.17.0 to 1.0.2. - [Release notes](https://github.com/arrow-py/arrow/releases) - [Changelog](https://github.com/arrow-py/arrow/blob/master/CHANGELOG.rst) - [Commits](https://github.com/arrow-py/arrow/compare/0.17.0...1.0.2) Signed-off-by: dependabot[bot] --- requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/requirements.txt b/requirements.txt index d17070e34..1508e2abe 100644 --- a/requirements.txt +++ b/requirements.txt @@ -7,7 +7,7 @@ cryptography==3.4.6 aiohttp==3.7.4 SQLAlchemy==1.3.23 python-telegram-bot==13.3 -arrow==0.17.0 +arrow==1.0.2 cachetools==4.2.1 requests==2.25.1 urllib3==1.26.3 From bba9b9e819aef4a199c71ed1693eba13c218ce75 Mon Sep 17 00:00:00 2001 From: Matthias Date: Mon, 1 Mar 2021 07:08:44 +0100 Subject: [PATCH 069/187] Don't use __root__ directly for api response --- freqtrade/rpc/api_server/api_v1.py | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/freqtrade/rpc/api_server/api_v1.py b/freqtrade/rpc/api_server/api_v1.py index 546b93afb..90e3a612f 100644 --- a/freqtrade/rpc/api_server/api_v1.py +++ b/freqtrade/rpc/api_server/api_v1.py @@ -111,9 +111,9 @@ def forcebuy(payload: ForceBuyPayload, rpc: RPC = Depends(get_rpc)): trade = rpc._rpc_forcebuy(payload.pair, payload.price) if trade: - return {'__root__': trade.to_json()} + return ForceBuyResponse.parse_obj(trade.to_json()) else: - return {'__root__': {"status": f"Error buying pair {payload.pair}."}} + return ForceBuyResponse.parse_obj({"status": f"Error buying pair {payload.pair}."}) @router.post('/forcesell', response_model=ResultMsg, tags=['trading']) @@ -183,7 +183,7 @@ def pair_history(pair: str, timeframe: str, timerange: str, strategy: str, @router.get('/plot_config', response_model=PlotConfig, tags=['candle data']) def plot_config(rpc: RPC = Depends(get_rpc)): - return {'__root__': rpc._rpc_plot_config()} + return PlotConfig.parse_obj(rpc._rpc_plot_config()) @router.get('/strategies', response_model=StrategyListResponse, tags=['strategy']) From 3d65ba2dcbb72f8e96e6ea4e3df65cc34eed5035 Mon Sep 17 00:00:00 2001 From: Matthias Date: Mon, 1 Mar 2021 07:51:33 +0100 Subject: [PATCH 070/187] Add rpc method to delete locks --- freqtrade/persistence/models.py | 1 + freqtrade/rpc/rpc.py | 24 ++++++++++++++++++++++-- tests/rpc/test_rpc.py | 21 ++++++++++++++++++++- 3 files changed, 43 insertions(+), 3 deletions(-) diff --git a/freqtrade/persistence/models.py b/freqtrade/persistence/models.py index dff59819c..3c9a10fb7 100644 --- a/freqtrade/persistence/models.py +++ b/freqtrade/persistence/models.py @@ -765,6 +765,7 @@ class PairLock(_DECL_BASE): def to_json(self) -> Dict[str, Any]: return { + 'id': self.id, 'pair': self.pair, 'lock_time': self.lock_time.strftime(DATETIME_PRINT_FORMAT), 'lock_timestamp': int(self.lock_time.replace(tzinfo=timezone.utc).timestamp() * 1000), diff --git a/freqtrade/rpc/rpc.py b/freqtrade/rpc/rpc.py index 7549c38be..37a2dc1e5 100644 --- a/freqtrade/rpc/rpc.py +++ b/freqtrade/rpc/rpc.py @@ -3,7 +3,7 @@ This module contains class to define a RPC communications """ import logging from abc import abstractmethod -from datetime import date, datetime, timedelta +from datetime import date, datetime, timedelta, timezone from enum import Enum from math import isnan from typing import Any, Dict, List, Optional, Tuple, Union @@ -20,6 +20,7 @@ from freqtrade.exchange import timeframe_to_minutes, timeframe_to_msecs from freqtrade.loggers import bufferHandler from freqtrade.misc import shorten_date from freqtrade.persistence import PairLocks, Trade +from freqtrade.persistence.models import PairLock from freqtrade.plugins.pairlist.pairlist_helpers import expand_pairlist from freqtrade.rpc.fiat_convert import CryptoToFiatConverter from freqtrade.state import State @@ -663,7 +664,7 @@ class RPC: } def _rpc_locks(self) -> Dict[str, Any]: - """ Returns the current locks""" + """ Returns the current locks """ locks = PairLocks.get_pair_locks(None) return { @@ -671,6 +672,25 @@ class RPC: 'locks': [lock.to_json() for lock in locks] } + def _rpc_delete_lock(self, lockid: Optional[int] = None, + pair: Optional[str] = None) -> Dict[str, Any]: + """ Delete specific lock(s) """ + locks = [] + + if pair: + locks = PairLocks.get_pair_locks(pair) + if lockid: + locks = PairLock.query.filter(PairLock.id == lockid).all() + + for lock in locks: + lock.active = False + lock.lock_end_time = datetime.now(timezone.utc) + + # session is always the same + PairLock.session.flush() + + return self._rpc_locks() + def _rpc_whitelist(self) -> Dict: """ Returns the currently active whitelist""" res = {'method': self._freqtrade.pairlists.name_list, diff --git a/tests/rpc/test_rpc.py b/tests/rpc/test_rpc.py index 60d9950aa..f745be506 100644 --- a/tests/rpc/test_rpc.py +++ b/tests/rpc/test_rpc.py @@ -1,7 +1,8 @@ # pragma pylint: disable=missing-docstring, C0103 # pragma pylint: disable=invalid-sequence-index, invalid-name, too-many-arguments -from datetime import datetime +from datetime import datetime, timedelta, timezone +from freqtrade.persistence.pairlock_middleware import PairLocks from unittest.mock import ANY, MagicMock, PropertyMock import pytest @@ -911,6 +912,24 @@ def test_rpcforcebuy_disabled(mocker, default_conf) -> None: rpc._rpc_forcebuy(pair, None) +@pytest.mark.usefixtures("init_persistence") +def test_rpc_delete_lock(mocker, default_conf): + freqtradebot = get_patched_freqtradebot(mocker, default_conf) + rpc = RPC(freqtradebot) + pair = 'ETH/BTC' + + PairLocks.lock_pair(pair, datetime.now(timezone.utc) + timedelta(minutes=4)) + PairLocks.lock_pair(pair, datetime.now(timezone.utc) + timedelta(minutes=5)) + PairLocks.lock_pair(pair, datetime.now(timezone.utc) + timedelta(minutes=10)) + locks = rpc._rpc_locks() + assert locks['lock_count'] == 3 + locks1 = rpc._rpc_delete_lock(lockid=locks['locks'][0]['id']) + assert locks1['lock_count'] == 2 + + locks2 = rpc._rpc_delete_lock(pair=pair) + assert locks2['lock_count'] == 0 + + def test_rpc_whitelist(mocker, default_conf) -> None: mocker.patch('freqtrade.rpc.telegram.Telegram', MagicMock()) From 2083cf6ddf859ade1a9fb5e4c6f89ea5d4ad5c99 Mon Sep 17 00:00:00 2001 From: Matthias Date: Mon, 1 Mar 2021 08:57:57 +0100 Subject: [PATCH 071/187] Fix mypy errors introduced by Arrow update --- freqtrade/optimize/backtesting.py | 16 +++++++--------- 1 file changed, 7 insertions(+), 9 deletions(-) diff --git a/freqtrade/optimize/backtesting.py b/freqtrade/optimize/backtesting.py index 3186313e1..25ec3299d 100644 --- a/freqtrade/optimize/backtesting.py +++ b/freqtrade/optimize/backtesting.py @@ -443,16 +443,14 @@ class Backtesting: data, timerange = self.load_bt_data() - min_date = None - max_date = None for strat in self.strategylist: min_date, max_date = self.backtest_one_strategy(strat, data, timerange) + if len(self.strategylist) > 0: + stats = generate_backtest_stats(data, self.all_results, + min_date=min_date, max_date=max_date) - stats = generate_backtest_stats(data, self.all_results, - min_date=min_date, max_date=max_date) + if self.config.get('export', False): + store_backtest_stats(self.config['exportfilename'], stats) - if self.config.get('export', False): - store_backtest_stats(self.config['exportfilename'], stats) - - # Show backtest results - show_backtest_results(self.config, stats) + # Show backtest results + show_backtest_results(self.config, stats) From 64ef7becc70b5a5078298fea3a1438835286091f Mon Sep 17 00:00:00 2001 From: Matthias Date: Mon, 1 Mar 2021 09:35:35 +0100 Subject: [PATCH 072/187] Update styles to work with new mkdocs version --- docs/index.md | 4 ---- docs/partials/header.html | 47 ++++++++++++++++++++++++++++----------- mkdocs.yml | 46 ++++++++++++++++++++------------------ 3 files changed, 58 insertions(+), 39 deletions(-) diff --git a/docs/index.md b/docs/index.md index 9d1a1532e..61f2276c3 100644 --- a/docs/index.md +++ b/docs/index.md @@ -5,12 +5,8 @@ Star - Fork - Download - -Follow @freqtrade ## Introduction diff --git a/docs/partials/header.html b/docs/partials/header.html index f5243225b..22132bc96 100644 --- a/docs/partials/header.html +++ b/docs/partials/header.html @@ -6,22 +6,22 @@ This file was automatically generated - do not edit {% set site_url = site_url ~ "/index.html" %} {% endif %}
- - - - + + +
diff --git a/mkdocs.yml b/mkdocs.yml index ca52627cb..2520ca929 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -1,4 +1,5 @@ site_name: Freqtrade +repo_url: https://github.com/freqtrade/freqtrade nav: - Home: index.md - Quickstart with Docker: docker_quickstart.md @@ -13,8 +14,8 @@ nav: - Start the bot: bot-usage.md - Control the bot: - Telegram: telegram-usage.md - - Web Hook: webhook-config.md - REST API & FreqUI: rest-api.md + - Web Hook: webhook-config.md - Data Downloading: data-download.md - Backtesting: backtesting.md - Hyperopt: hyperopt.md @@ -50,24 +51,25 @@ extra_javascript: - https://polyfill.io/v3/polyfill.min.js?features=es6 - https://cdn.jsdelivr.net/npm/mathjax@3/es5/tex-mml-chtml.js markdown_extensions: - - admonition - - footnotes - - codehilite: - guess_lang: false - - toc: - permalink: true - - pymdownx.arithmatex: - generic: true - - pymdownx.details - - pymdownx.inlinehilite - - pymdownx.magiclink - - pymdownx.pathconverter - - pymdownx.smartsymbols - - pymdownx.snippets: - base_path: docs - check_paths: true - - pymdownx.tabbed - - pymdownx.superfences - - pymdownx.tasklist: - custom_checkbox: true - - mdx_truly_sane_lists + - attr_list + - admonition + - footnotes + - codehilite: + guess_lang: false + - toc: + permalink: true + - pymdownx.arithmatex: + generic: true + - pymdownx.details + - pymdownx.inlinehilite + - pymdownx.magiclink + - pymdownx.pathconverter + - pymdownx.smartsymbols + - pymdownx.snippets: + base_path: docs + check_paths: true + - pymdownx.tabbed + - pymdownx.superfences + - pymdownx.tasklist: + custom_checkbox: true + - mdx_truly_sane_lists From 4e5136405788d5f0a593083bfbb9a8b31187bc62 Mon Sep 17 00:00:00 2001 From: Matthias Date: Mon, 1 Mar 2021 19:12:02 +0100 Subject: [PATCH 073/187] Add warning about sandboxes closes #4468 --- docs/sandbox-testing.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/docs/sandbox-testing.md b/docs/sandbox-testing.md index 9c14412de..5f572eba8 100644 --- a/docs/sandbox-testing.md +++ b/docs/sandbox-testing.md @@ -6,6 +6,10 @@ With some configuration, freqtrade (in combination with ccxt) provides access to This document is an overview to configure Freqtrade to be used with sandboxes. This can be useful to developers and trader alike. +!!! Warning + Sandboxes usually have very low volume, and either a very wide spread, or no orders available at all. + Therefore, sandboxes will usually not do a good job of showing you how a strategy would work in real trading. + ## Exchanges known to have a sandbox / testnet * [binance](https://testnet.binance.vision/) From 6640156ac70f36ff328c0fa2a05ed0ed1d5d8970 Mon Sep 17 00:00:00 2001 From: Matthias Date: Mon, 1 Mar 2021 19:50:39 +0100 Subject: [PATCH 074/187] Support deleting locks via API --- docs/rest-api.md | 9 +++++++++ freqtrade/rpc/api_server/api_schemas.py | 6 ++++++ freqtrade/rpc/api_server/api_v1.py | 14 ++++++++++++-- scripts/rest_client.py | 8 ++++++++ tests/rpc/test_rpc_apiserver.py | 10 ++++++++++ 5 files changed, 45 insertions(+), 2 deletions(-) diff --git a/docs/rest-api.md b/docs/rest-api.md index e2b94f080..c41c3f24c 100644 --- a/docs/rest-api.md +++ b/docs/rest-api.md @@ -131,6 +131,7 @@ python3 scripts/rest_client.py --config rest_config.json [optional par | `status` | Lists all open trades. | `count` | Displays number of trades used and available. | `locks` | Displays currently locked pairs. +| `delete_lock ` | Deletes (disables) the lock by id. | `profit` | Display a summary of your profit/loss from close trades and some stats about your performance. | `forcesell ` | Instantly sells the given trade (Ignoring `minimum_roi`). | `forcesell all` | Instantly sells all open trades (Ignoring `minimum_roi`). @@ -182,6 +183,11 @@ count daily Return the amount of open trades. +delete_lock + Delete (disable) lock from the database. + + :param lock_id: ID for the lock to delete + delete_trade Delete trade from the database. Tries to close open orders. Requires manual handling of this asset on the exchange. @@ -202,6 +208,9 @@ forcesell :param tradeid: Id of the trade (can be received via status command) +locks + Return current locks + logs Show latest logs. diff --git a/freqtrade/rpc/api_server/api_schemas.py b/freqtrade/rpc/api_server/api_schemas.py index 2738e5368..244c5540a 100644 --- a/freqtrade/rpc/api_server/api_schemas.py +++ b/freqtrade/rpc/api_server/api_schemas.py @@ -210,6 +210,7 @@ class ForceBuyResponse(BaseModel): class LockModel(BaseModel): + id: int active: bool lock_end_time: str lock_end_timestamp: int @@ -224,6 +225,11 @@ class Locks(BaseModel): locks: List[LockModel] +class DeleteLockRequest(BaseModel): + pair: Optional[str] + lockid: Optional[int] + + class Logs(BaseModel): log_count: int logs: List[List] diff --git a/freqtrade/rpc/api_server/api_v1.py b/freqtrade/rpc/api_server/api_v1.py index 90e3a612f..7f1179a0b 100644 --- a/freqtrade/rpc/api_server/api_v1.py +++ b/freqtrade/rpc/api_server/api_v1.py @@ -11,7 +11,7 @@ from freqtrade.data.history import get_datahandler from freqtrade.exceptions import OperationalException from freqtrade.rpc import RPC from freqtrade.rpc.api_server.api_schemas import (AvailablePairs, Balances, BlacklistPayload, - BlacklistResponse, Count, Daily, DeleteTrade, + BlacklistResponse, Count, Daily, DeleteLockRequest, DeleteTrade, ForceBuyPayload, ForceBuyResponse, ForceSellPayload, Locks, Logs, OpenTradeSchema, PairHistory, PerformanceEntry, Ping, PlotConfig, @@ -136,11 +136,21 @@ def whitelist(rpc: RPC = Depends(get_rpc)): return rpc._rpc_whitelist() -@router.get('/locks', response_model=Locks, tags=['info']) +@router.get('/locks', response_model=Locks, tags=['info', 'locks']) def locks(rpc: RPC = Depends(get_rpc)): return rpc._rpc_locks() +@router.delete('/locks/{lockid}', response_model=Locks, tags=['info', 'locks']) +def delete_lock(lockid: int, rpc: RPC = Depends(get_rpc)): + return rpc._rpc_delete_lock(lockid=lockid) + + +@router.post('/locks/delete', response_model=Locks, tags=['info', 'locks']) +def delete_lock_pair(payload: DeleteLockRequest, rpc: RPC = Depends(get_rpc)): + return rpc._rpc_delete_lock(lockid=payload.lockid, pair=payload.pair) + + @router.get('/logs', response_model=Logs, tags=['info']) def logs(limit: Optional[int] = None, rpc: RPC = Depends(get_rpc)): return rpc._rpc_get_logs(limit) diff --git a/scripts/rest_client.py b/scripts/rest_client.py index b6e66cfa4..90d2e24d4 100755 --- a/scripts/rest_client.py +++ b/scripts/rest_client.py @@ -118,6 +118,14 @@ class FtRestClient(): """ return self._get("locks") + def delete_lock(self, lock_id): + """Delete (disable) lock from the database. + + :param lock_id: ID for the lock to delete + :return: json object + """ + return self._delete("locks/{}".format(lock_id)) + def daily(self, days=None): """Return the amount of open trades. diff --git a/tests/rpc/test_rpc_apiserver.py b/tests/rpc/test_rpc_apiserver.py index d7d69d0ae..56a496de2 100644 --- a/tests/rpc/test_rpc_apiserver.py +++ b/tests/rpc/test_rpc_apiserver.py @@ -418,6 +418,16 @@ def test_api_locks(botclient): assert 'randreason' in (rc.json()['locks'][0]['reason'], rc.json()['locks'][1]['reason']) assert 'deadbeef' in (rc.json()['locks'][0]['reason'], rc.json()['locks'][1]['reason']) + # Test deletions + rc = client_delete(client, f"{BASE_URI}/locks/1") + assert_response(rc) + assert rc.json()['lock_count'] == 1 + + rc = client_post(client, f"{BASE_URI}/locks/delete", + data='{"pair": "XRP/BTC"}') + assert_response(rc) + assert rc.json()['lock_count'] == 0 + def test_api_show_config(botclient, mocker): ftbot, client = botclient From 007ac7abb53a377af8aadd90c8be01ed71e9ce35 Mon Sep 17 00:00:00 2001 From: Matthias Date: Mon, 1 Mar 2021 20:08:49 +0100 Subject: [PATCH 075/187] Add telegram pair unlocking --- docs/telegram-usage.md | 1 + freqtrade/rpc/api_server/api_v1.py | 15 ++++++++------- freqtrade/rpc/telegram.py | 29 +++++++++++++++++++++++++++-- tests/rpc/test_rpc.py | 2 +- tests/rpc/test_rpc_telegram.py | 13 ++++++++++++- 5 files changed, 49 insertions(+), 11 deletions(-) diff --git a/docs/telegram-usage.md b/docs/telegram-usage.md index d4a6fb118..833fae1fe 100644 --- a/docs/telegram-usage.md +++ b/docs/telegram-usage.md @@ -146,6 +146,7 @@ official commands. You can ask at any moment for help with `/help`. | `/delete ` | Delete a specific trade from the Database. Tries to close open orders. Requires manual handling of this trade on the exchange. | `/count` | Displays number of trades used and available | `/locks` | Show currently locked pairs. +| `/unlock ` | Remove the lock for this pair (or for this lock id). | `/profit` | Display a summary of your profit/loss from close trades and some stats about your performance | `/forcesell ` | Instantly sells the given trade (Ignoring `minimum_roi`). | `/forcesell all` | Instantly sells all open trades (Ignoring `minimum_roi`). diff --git a/freqtrade/rpc/api_server/api_v1.py b/freqtrade/rpc/api_server/api_v1.py index 7f1179a0b..b983402e9 100644 --- a/freqtrade/rpc/api_server/api_v1.py +++ b/freqtrade/rpc/api_server/api_v1.py @@ -11,13 +11,14 @@ from freqtrade.data.history import get_datahandler from freqtrade.exceptions import OperationalException from freqtrade.rpc import RPC from freqtrade.rpc.api_server.api_schemas import (AvailablePairs, Balances, BlacklistPayload, - BlacklistResponse, Count, Daily, DeleteLockRequest, DeleteTrade, - ForceBuyPayload, ForceBuyResponse, - ForceSellPayload, Locks, Logs, OpenTradeSchema, - PairHistory, PerformanceEntry, Ping, PlotConfig, - Profit, ResultMsg, ShowConfig, Stats, StatusMsg, - StrategyListResponse, StrategyResponse, - TradeResponse, Version, WhitelistResponse) + BlacklistResponse, Count, Daily, + DeleteLockRequest, DeleteTrade, ForceBuyPayload, + ForceBuyResponse, ForceSellPayload, Locks, Logs, + OpenTradeSchema, PairHistory, PerformanceEntry, + Ping, PlotConfig, Profit, ResultMsg, ShowConfig, + Stats, StatusMsg, StrategyListResponse, + StrategyResponse, TradeResponse, Version, + WhitelistResponse) from freqtrade.rpc.api_server.deps import get_config, get_rpc, get_rpc_optional from freqtrade.rpc.rpc import RPCException diff --git a/freqtrade/rpc/telegram.py b/freqtrade/rpc/telegram.py index 9d05ae142..fc9676a49 100644 --- a/freqtrade/rpc/telegram.py +++ b/freqtrade/rpc/telegram.py @@ -6,6 +6,7 @@ This module manage Telegram communication import json import logging from datetime import timedelta +from html import escape from itertools import chain from typing import Any, Callable, Dict, List, Union @@ -144,6 +145,7 @@ class Telegram(RPCHandler): CommandHandler('daily', self._daily), CommandHandler('count', self._count), CommandHandler('locks', self._locks), + CommandHandler(['unlock', 'delete_locks'], self._delete_locks), CommandHandler(['reload_config', 'reload_conf'], self._reload_config), CommandHandler(['show_config', 'show_conf'], self._show_config), CommandHandler('stopbuy', self._stopbuy), @@ -722,17 +724,39 @@ class Telegram(RPCHandler): try: locks = self._rpc._rpc_locks() message = tabulate([[ + lock['id'], lock['pair'], lock['lock_end_time'], lock['reason']] for lock in locks['locks']], - headers=['Pair', 'Until', 'Reason'], + headers=['ID', 'Pair', 'Until', 'Reason'], tablefmt='simple') - message = "
{}
".format(message) + message = f"
{escape(message)}
" logger.debug(message) self._send_msg(message, parse_mode=ParseMode.HTML) except RPCException as e: self._send_msg(str(e)) + @authorized_only + def _delete_locks(self, update: Update, context: CallbackContext) -> None: + """ + Handler for /delete_locks. + Returns the currently active locks + """ + try: + arg = context.args[0] if context.args and len(context.args) > 0 else None + lockid = None + pair = None + if arg: + try: + lockid = int(arg) + except ValueError: + pair = arg + + self._rpc._rpc_delete_lock(lockid=lockid, pair=pair) + self._locks(update, context) + except RPCException as e: + self._send_msg(str(e)) + @authorized_only def _whitelist(self, update: Update, context: CallbackContext) -> None: """ @@ -850,6 +874,7 @@ class Telegram(RPCHandler): "Avg. holding durationsfor buys and sells.`\n" "*/count:* `Show number of active trades compared to allowed number of trades`\n" "*/locks:* `Show currently locked pairs`\n" + "*/unlock :* `Unlock this Pair (or this lock id if it's numeric)`\n" "*/balance:* `Show account balance per currency`\n" "*/stopbuy:* `Stops buying, but handles open trades gracefully` \n" "*/reload_config:* `Reload configuration file` \n" diff --git a/tests/rpc/test_rpc.py b/tests/rpc/test_rpc.py index f745be506..a22accab5 100644 --- a/tests/rpc/test_rpc.py +++ b/tests/rpc/test_rpc.py @@ -2,7 +2,6 @@ # pragma pylint: disable=invalid-sequence-index, invalid-name, too-many-arguments from datetime import datetime, timedelta, timezone -from freqtrade.persistence.pairlock_middleware import PairLocks from unittest.mock import ANY, MagicMock, PropertyMock import pytest @@ -11,6 +10,7 @@ from numpy import isnan from freqtrade.edge import PairInfo from freqtrade.exceptions import ExchangeError, InvalidOrderException, TemporaryError from freqtrade.persistence import Trade +from freqtrade.persistence.pairlock_middleware import PairLocks from freqtrade.rpc import RPC, RPCException from freqtrade.rpc.fiat_convert import CryptoToFiatConverter from freqtrade.state import State diff --git a/tests/rpc/test_rpc_telegram.py b/tests/rpc/test_rpc_telegram.py index 922aa2de8..0d86c578a 100644 --- a/tests/rpc/test_rpc_telegram.py +++ b/tests/rpc/test_rpc_telegram.py @@ -92,7 +92,8 @@ def test_telegram_init(default_conf, mocker, caplog) -> None: message_str = ("rpc.telegram is listening for following commands: [['status'], ['profit'], " "['balance'], ['start'], ['stop'], ['forcesell'], ['forcebuy'], ['trades'], " "['delete'], ['performance'], ['stats'], ['daily'], ['count'], ['locks'], " - "['reload_config', 'reload_conf'], ['show_config', 'show_conf'], ['stopbuy'], " + "['unlock', 'delete_locks'], ['reload_config', 'reload_conf'], " + "['show_config', 'show_conf'], ['stopbuy'], " "['whitelist'], ['blacklist'], ['logs'], ['edge'], ['help'], ['version']" "]") @@ -981,6 +982,16 @@ def test_telegram_lock_handle(default_conf, update, ticker, fee, mocker) -> None assert 'deadbeef' in msg_mock.call_args_list[0][0][0] assert 'randreason' in msg_mock.call_args_list[0][0][0] + context = MagicMock() + context.args = ['XRP/BTC'] + msg_mock.reset_mock() + telegram._delete_locks(update=update, context=context) + + assert 'ETH/BTC' in msg_mock.call_args_list[0][0][0] + assert 'randreason' in msg_mock.call_args_list[0][0][0] + assert 'XRP/BTC' not in msg_mock.call_args_list[0][0][0] + assert 'deadbeef' not in msg_mock.call_args_list[0][0][0] + def test_whitelist_static(default_conf, update, mocker) -> None: From 4bb6a27745df744442f10db7f0e4fd30a76d89b2 Mon Sep 17 00:00:00 2001 From: Matthias Date: Tue, 2 Mar 2021 06:59:58 +0100 Subject: [PATCH 076/187] Don't catch errors that can't happen --- freqtrade/rpc/telegram.py | 48 +++++++++++++++++---------------------- 1 file changed, 21 insertions(+), 27 deletions(-) diff --git a/freqtrade/rpc/telegram.py b/freqtrade/rpc/telegram.py index fc9676a49..168ae0e6a 100644 --- a/freqtrade/rpc/telegram.py +++ b/freqtrade/rpc/telegram.py @@ -721,20 +721,17 @@ class Telegram(RPCHandler): Handler for /locks. Returns the currently active locks """ - try: - locks = self._rpc._rpc_locks() - message = tabulate([[ - lock['id'], - lock['pair'], - lock['lock_end_time'], - lock['reason']] for lock in locks['locks']], - headers=['ID', 'Pair', 'Until', 'Reason'], - tablefmt='simple') - message = f"
{escape(message)}
" - logger.debug(message) - self._send_msg(message, parse_mode=ParseMode.HTML) - except RPCException as e: - self._send_msg(str(e)) + locks = self._rpc._rpc_locks() + message = tabulate([[ + lock['id'], + lock['pair'], + lock['lock_end_time'], + lock['reason']] for lock in locks['locks']], + headers=['ID', 'Pair', 'Until', 'Reason'], + tablefmt='simple') + message = f"
{escape(message)}
" + logger.debug(message) + self._send_msg(message, parse_mode=ParseMode.HTML) @authorized_only def _delete_locks(self, update: Update, context: CallbackContext) -> None: @@ -742,20 +739,17 @@ class Telegram(RPCHandler): Handler for /delete_locks. Returns the currently active locks """ - try: - arg = context.args[0] if context.args and len(context.args) > 0 else None - lockid = None - pair = None - if arg: - try: - lockid = int(arg) - except ValueError: - pair = arg + arg = context.args[0] if context.args and len(context.args) > 0 else None + lockid = None + pair = None + if arg: + try: + lockid = int(arg) + except ValueError: + pair = arg - self._rpc._rpc_delete_lock(lockid=lockid, pair=pair) - self._locks(update, context) - except RPCException as e: - self._send_msg(str(e)) + self._rpc._rpc_delete_lock(lockid=lockid, pair=pair) + self._locks(update, context) @authorized_only def _whitelist(self, update: Update, context: CallbackContext) -> None: From 7c35d107abfbba0152baa04f8c3ad2c4f3607502 Mon Sep 17 00:00:00 2001 From: av1nxsh <79896600+av1nxsh@users.noreply.github.com> Date: Tue, 2 Mar 2021 14:24:00 +0530 Subject: [PATCH 077/187] rest_client.py first --- scripts/rest_client.py | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/scripts/rest_client.py b/scripts/rest_client.py index b6e66cfa4..eb084f400 100755 --- a/scripts/rest_client.py +++ b/scripts/rest_client.py @@ -75,7 +75,7 @@ class FtRestClient(): :return: json object """ return self._post("start") - + def stop(self): """Stop the bot. Use `start` to restart. @@ -174,6 +174,14 @@ class FtRestClient(): """ return self._get("show_config") + def ping(self): + """simple ping""" + + if self.show_config()['state']=="running": + return {"status": "pong"} + else: + return{"status": "not_running"} + def logs(self, limit=None): """Show latest logs. From 4fe2e542b4d9d63c17ca7f3490f25c7b823b7c7a Mon Sep 17 00:00:00 2001 From: av1nxsh <79896600+av1nxsh@users.noreply.github.com> Date: Tue, 2 Mar 2021 14:25:37 +0530 Subject: [PATCH 078/187] rest_client.py removing tab --- scripts/rest_client.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scripts/rest_client.py b/scripts/rest_client.py index eb084f400..183a906df 100755 --- a/scripts/rest_client.py +++ b/scripts/rest_client.py @@ -75,7 +75,7 @@ class FtRestClient(): :return: json object """ return self._post("start") - + def stop(self): """Stop the bot. Use `start` to restart. From 82bf65f696af779bfbe537e813de6f2492b76c10 Mon Sep 17 00:00:00 2001 From: av1nxsh <79896600+av1nxsh@users.noreply.github.com> Date: Tue, 2 Mar 2021 14:49:33 +0530 Subject: [PATCH 079/187] rest_client.py flake8 issues --- scripts/rest_client.py | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/scripts/rest_client.py b/scripts/rest_client.py index 183a906df..6aba92e7a 100755 --- a/scripts/rest_client.py +++ b/scripts/rest_client.py @@ -176,12 +176,11 @@ class FtRestClient(): def ping(self): """simple ping""" - if self.show_config()['state']=="running": return {"status": "pong"} else: - return{"status": "not_running"} - + return {"status": "not_running"} + def logs(self, limit=None): """Show latest logs. From 95c635091ef1e7f16ec5fce669d67b26af4a7abc Mon Sep 17 00:00:00 2001 From: av1nxsh <79896600+av1nxsh@users.noreply.github.com> Date: Tue, 2 Mar 2021 14:57:05 +0530 Subject: [PATCH 080/187] rest_client.py fixed operator --- scripts/rest_client.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scripts/rest_client.py b/scripts/rest_client.py index 6aba92e7a..39da0a406 100755 --- a/scripts/rest_client.py +++ b/scripts/rest_client.py @@ -176,7 +176,7 @@ class FtRestClient(): def ping(self): """simple ping""" - if self.show_config()['state']=="running": + if self.show_config()['state'] == "running": return {"status": "pong"} else: return {"status": "not_running"} From 218d22ed528da1bcc22b6971c4ff4f4ec847eae6 Mon Sep 17 00:00:00 2001 From: av1nxsh <79896600+av1nxsh@users.noreply.github.com> Date: Tue, 2 Mar 2021 15:45:16 +0530 Subject: [PATCH 081/187] rest_client.py updated for connection error case --- scripts/rest_client.py | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/scripts/rest_client.py b/scripts/rest_client.py index 39da0a406..a7d7705fe 100755 --- a/scripts/rest_client.py +++ b/scripts/rest_client.py @@ -176,7 +176,9 @@ class FtRestClient(): def ping(self): """simple ping""" - if self.show_config()['state'] == "running": + if not self.show_config(): + return {"status": "not_running"} + elif self.show_config()['state'] == "running": return {"status": "pong"} else: return {"status": "not_running"} From a85e656e8d5320db721216365205c16160756d5f Mon Sep 17 00:00:00 2001 From: av1nxsh <79896600+av1nxsh@users.noreply.github.com> Date: Tue, 2 Mar 2021 16:16:20 +0530 Subject: [PATCH 082/187] rest_client.py optimised with var 'configstatus' --- scripts/rest_client.py | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/scripts/rest_client.py b/scripts/rest_client.py index a7d7705fe..ecf961ddd 100755 --- a/scripts/rest_client.py +++ b/scripts/rest_client.py @@ -176,9 +176,10 @@ class FtRestClient(): def ping(self): """simple ping""" - if not self.show_config(): + configstatus = self.show_config() + if not configstatus: return {"status": "not_running"} - elif self.show_config()['state'] == "running": + elif configstatus['state'] == "running": return {"status": "pong"} else: return {"status": "not_running"} From 55a315be14c00072983bd15e5e6c1ec21ce430c3 Mon Sep 17 00:00:00 2001 From: Joe Schr Date: Tue, 2 Mar 2021 13:34:14 +0100 Subject: [PATCH 083/187] fix: avg_stake_amount should not be `NaN` if df is empty --- freqtrade/optimize/optimize_reports.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/freqtrade/optimize/optimize_reports.py b/freqtrade/optimize/optimize_reports.py index 0de0c16a0..47ddfc9fc 100644 --- a/freqtrade/optimize/optimize_reports.py +++ b/freqtrade/optimize/optimize_reports.py @@ -278,7 +278,7 @@ def generate_backtest_stats(btdata: Dict[str, DataFrame], 'left_open_trades': left_open_results, 'total_trades': len(results), 'total_volume': float(results['stake_amount'].sum()), - 'avg_stake_amount': results['stake_amount'].mean(), + 'avg_stake_amount': results['stake_amount'].mean() if len(results) > 0 else 0, 'profit_mean': results['profit_ratio'].mean() if len(results) > 0 else 0, 'profit_total': results['profit_abs'].sum() / starting_balance, 'profit_total_abs': results['profit_abs'].sum(), From 078b77d41be278208b9bc4fb5ddfe224aa562e68 Mon Sep 17 00:00:00 2001 From: Matthias Date: Tue, 2 Mar 2021 16:12:22 +0100 Subject: [PATCH 084/187] Fix crash when using unlimited stake and no trades are made --- freqtrade/optimize/optimize_reports.py | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/freqtrade/optimize/optimize_reports.py b/freqtrade/optimize/optimize_reports.py index 47ddfc9fc..52ae09ad1 100644 --- a/freqtrade/optimize/optimize_reports.py +++ b/freqtrade/optimize/optimize_reports.py @@ -8,7 +8,7 @@ from numpy import int64 from pandas import DataFrame from tabulate import tabulate -from freqtrade.constants import DATETIME_PRINT_FORMAT, LAST_BT_RESULT_FN +from freqtrade.constants import DATETIME_PRINT_FORMAT, LAST_BT_RESULT_FN, UNLIMITED_STAKE_AMOUNT from freqtrade.data.btanalysis import (calculate_csum, calculate_market_change, calculate_max_drawdown) from freqtrade.misc import decimals_per_coin, file_dump_json, round_coin_value @@ -499,8 +499,10 @@ def text_table_add_metrics(strat_results: Dict) -> str: else: start_balance = round_coin_value(strat_results['starting_balance'], strat_results['stake_currency']) - stake_amount = round_coin_value(strat_results['stake_amount'], - strat_results['stake_currency']) + stake_amount = round_coin_value( + strat_results['stake_amount'], strat_results['stake_currency'] + ) if strat_results['stake_amount'] != UNLIMITED_STAKE_AMOUNT else 'unlimited' + message = ("No trades made. " f"Your starting balance was {start_balance}, " f"and your stake was {stake_amount}." From 0968ecc1af8c965e407c45ef038a36a5c888d1e4 Mon Sep 17 00:00:00 2001 From: raoulus Date: Thu, 4 Mar 2021 17:27:04 +0100 Subject: [PATCH 085/187] added "Median profit" column to hyperopt -> export-csv --- freqtrade/optimize/hyperopt.py | 4 ++-- tests/commands/test_commands.py | 2 +- tests/conftest.py | 24 ++++++++++++------------ 3 files changed, 15 insertions(+), 15 deletions(-) diff --git a/freqtrade/optimize/hyperopt.py b/freqtrade/optimize/hyperopt.py index eee0f13b3..c46d0da48 100644 --- a/freqtrade/optimize/hyperopt.py +++ b/freqtrade/optimize/hyperopt.py @@ -419,13 +419,13 @@ class Hyperopt: trials['Stake currency'] = config['stake_currency'] base_metrics = ['Best', 'current_epoch', 'results_metrics.trade_count', - 'results_metrics.avg_profit', 'results_metrics.total_profit', + 'results_metrics.avg_profit', 'results_metrics.median_profit', 'results_metrics.total_profit', 'Stake currency', 'results_metrics.profit', 'results_metrics.duration', 'loss', 'is_initial_point', 'is_best'] param_metrics = [("params_dict."+param) for param in results[0]['params_dict'].keys()] trials = trials[base_metrics + param_metrics] - base_columns = ['Best', 'Epoch', 'Trades', 'Avg profit', 'Total profit', 'Stake currency', + base_columns = ['Best', 'Epoch', 'Trades', 'Avg profit', 'Median profit', 'Total profit', 'Stake currency', 'Profit', 'Avg duration', 'Objective', 'is_initial_point', 'is_best'] param_columns = list(results[0]['params_dict'].keys()) trials.columns = base_columns + param_columns diff --git a/tests/commands/test_commands.py b/tests/commands/test_commands.py index c81909025..d5e76eeb6 100644 --- a/tests/commands/test_commands.py +++ b/tests/commands/test_commands.py @@ -1145,7 +1145,7 @@ def test_hyperopt_list(mocker, capsys, caplog, hyperopt_results): captured = capsys.readouterr() log_has("CSV file created: test_file.csv", caplog) f = Path("test_file.csv") - assert 'Best,1,2,-1.25%,-0.00125625,,-2.51,"3,930.0 m",0.43662' in f.read_text() + assert 'Best,1,2,-1.25%,-1.2222,-0.00125625,,-2.51,"3,930.0 m",0.43662' in f.read_text() assert f.is_file() f.unlink() diff --git a/tests/conftest.py b/tests/conftest.py index 61899dd53..9834aa36c 100644 --- a/tests/conftest.py +++ b/tests/conftest.py @@ -1766,7 +1766,7 @@ def hyperopt_results(): 'params_dict': { 'mfi-value': 15, 'fastd-value': 20, 'adx-value': 25, 'rsi-value': 28, 'mfi-enabled': False, 'fastd-enabled': True, 'adx-enabled': True, 'rsi-enabled': True, 'trigger': 'macd_cross_signal', 'sell-mfi-value': 88, 'sell-fastd-value': 97, 'sell-adx-value': 51, 'sell-rsi-value': 67, 'sell-mfi-enabled': False, 'sell-fastd-enabled': False, 'sell-adx-enabled': True, 'sell-rsi-enabled': True, 'sell-trigger': 'sell-bb_upper', 'roi_t1': 1190, 'roi_t2': 541, 'roi_t3': 408, 'roi_p1': 0.026035863879169705, 'roi_p2': 0.12508730043628782, 'roi_p3': 0.27766427921605896, 'stoploss': -0.2562930402099556}, # noqa: E501 'params_details': {'buy': {'mfi-value': 15, 'fastd-value': 20, 'adx-value': 25, 'rsi-value': 28, 'mfi-enabled': False, 'fastd-enabled': True, 'adx-enabled': True, 'rsi-enabled': True, 'trigger': 'macd_cross_signal'}, 'sell': {'sell-mfi-value': 88, 'sell-fastd-value': 97, 'sell-adx-value': 51, 'sell-rsi-value': 67, 'sell-mfi-enabled': False, 'sell-fastd-enabled': False, 'sell-adx-enabled': True, 'sell-rsi-enabled': True, 'sell-trigger': 'sell-bb_upper'}, 'roi': {0: 0.4287874435315165, 408: 0.15112316431545753, 949: 0.026035863879169705, 2139: 0}, 'stoploss': {'stoploss': -0.2562930402099556}}, # noqa: E501 - 'results_metrics': {'trade_count': 2, 'avg_profit': -1.254995, 'total_profit': -0.00125625, 'profit': -2.50999, 'duration': 3930.0}, # noqa: E501 + 'results_metrics': {'trade_count': 2, 'avg_profit': -1.254995, 'median_profit': -1.2222, 'total_profit': -0.00125625, 'profit': -2.50999, 'duration': 3930.0}, # noqa: E501 'results_explanation': ' 2 trades. Avg profit -1.25%. Total profit -0.00125625 BTC ( -2.51Σ%). Avg duration 3930.0 min.', # noqa: E501 'total_profit': -0.00125625, 'current_epoch': 1, @@ -1781,7 +1781,7 @@ def hyperopt_results(): 'sell': {'sell-mfi-value': 96, 'sell-fastd-value': 68, 'sell-adx-value': 63, 'sell-rsi-value': 81, 'sell-mfi-enabled': False, 'sell-fastd-enabled': True, 'sell-adx-enabled': True, 'sell-rsi-enabled': True, 'sell-trigger': 'sell-sar_reversal'}, # noqa: E501 'roi': {0: 0.4449309386008759, 140: 0.11955965746663, 823: 0.06403981740598495, 1157: 0}, # noqa: E501 'stoploss': {'stoploss': -0.338070047333259}}, - 'results_metrics': {'trade_count': 1, 'avg_profit': 0.12357, 'total_profit': 6.185e-05, 'profit': 0.12357, 'duration': 1200.0}, # noqa: E501 + 'results_metrics': {'trade_count': 1, 'avg_profit': 0.12357, 'median_profit': -1.2222, 'total_profit': 6.185e-05, 'profit': 0.12357, 'duration': 1200.0}, # noqa: E501 'results_explanation': ' 1 trades. Avg profit 0.12%. Total profit 0.00006185 BTC ( 0.12Σ%). Avg duration 1200.0 min.', # noqa: E501 'total_profit': 6.185e-05, 'current_epoch': 2, @@ -1791,7 +1791,7 @@ def hyperopt_results(): 'loss': 14.241196856510731, 'params_dict': {'mfi-value': 25, 'fastd-value': 16, 'adx-value': 29, 'rsi-value': 20, 'mfi-enabled': False, 'fastd-enabled': False, 'adx-enabled': False, 'rsi-enabled': False, 'trigger': 'macd_cross_signal', 'sell-mfi-value': 98, 'sell-fastd-value': 72, 'sell-adx-value': 51, 'sell-rsi-value': 82, 'sell-mfi-enabled': True, 'sell-fastd-enabled': True, 'sell-adx-enabled': True, 'sell-rsi-enabled': True, 'sell-trigger': 'sell-macd_cross_signal', 'roi_t1': 889, 'roi_t2': 533, 'roi_t3': 263, 'roi_p1': 0.04759065393663096, 'roi_p2': 0.1488819964638463, 'roi_p3': 0.4102801822104605, 'stoploss': -0.05394588767607611}, # noqa: E501 'params_details': {'buy': {'mfi-value': 25, 'fastd-value': 16, 'adx-value': 29, 'rsi-value': 20, 'mfi-enabled': False, 'fastd-enabled': False, 'adx-enabled': False, 'rsi-enabled': False, 'trigger': 'macd_cross_signal'}, 'sell': {'sell-mfi-value': 98, 'sell-fastd-value': 72, 'sell-adx-value': 51, 'sell-rsi-value': 82, 'sell-mfi-enabled': True, 'sell-fastd-enabled': True, 'sell-adx-enabled': True, 'sell-rsi-enabled': True, 'sell-trigger': 'sell-macd_cross_signal'}, 'roi': {0: 0.6067528326109377, 263: 0.19647265040047726, 796: 0.04759065393663096, 1685: 0}, 'stoploss': {'stoploss': -0.05394588767607611}}, # noqa: E501 - 'results_metrics': {'trade_count': 621, 'avg_profit': -0.43883302093397747, 'total_profit': -0.13639474, 'profit': -272.515306, 'duration': 1691.207729468599}, # noqa: E501 + 'results_metrics': {'trade_count': 621, 'avg_profit': -0.43883302093397747, 'median_profit': -1.2222, 'total_profit': -0.13639474, 'profit': -272.515306, 'duration': 1691.207729468599}, # noqa: E501 'results_explanation': ' 621 trades. Avg profit -0.44%. Total profit -0.13639474 BTC (-272.52Σ%). Avg duration 1691.2 min.', # noqa: E501 'total_profit': -0.13639474, 'current_epoch': 3, @@ -1801,14 +1801,14 @@ def hyperopt_results(): 'loss': 100000, 'params_dict': {'mfi-value': 13, 'fastd-value': 35, 'adx-value': 39, 'rsi-value': 29, 'mfi-enabled': True, 'fastd-enabled': False, 'adx-enabled': False, 'rsi-enabled': True, 'trigger': 'macd_cross_signal', 'sell-mfi-value': 87, 'sell-fastd-value': 54, 'sell-adx-value': 63, 'sell-rsi-value': 93, 'sell-mfi-enabled': False, 'sell-fastd-enabled': True, 'sell-adx-enabled': True, 'sell-rsi-enabled': True, 'sell-trigger': 'sell-bb_upper', 'roi_t1': 1402, 'roi_t2': 676, 'roi_t3': 215, 'roi_p1': 0.06264755784937427, 'roi_p2': 0.14258587851894644, 'roi_p3': 0.20671291201040828, 'stoploss': -0.11818343570194478}, # noqa: E501 'params_details': {'buy': {'mfi-value': 13, 'fastd-value': 35, 'adx-value': 39, 'rsi-value': 29, 'mfi-enabled': True, 'fastd-enabled': False, 'adx-enabled': False, 'rsi-enabled': True, 'trigger': 'macd_cross_signal'}, 'sell': {'sell-mfi-value': 87, 'sell-fastd-value': 54, 'sell-adx-value': 63, 'sell-rsi-value': 93, 'sell-mfi-enabled': False, 'sell-fastd-enabled': True, 'sell-adx-enabled': True, 'sell-rsi-enabled': True, 'sell-trigger': 'sell-bb_upper'}, 'roi': {0: 0.411946348378729, 215: 0.2052334363683207, 891: 0.06264755784937427, 2293: 0}, 'stoploss': {'stoploss': -0.11818343570194478}}, # noqa: E501 - 'results_metrics': {'trade_count': 0, 'avg_profit': None, 'total_profit': 0, 'profit': 0.0, 'duration': None}, # noqa: E501 + 'results_metrics': {'trade_count': 0, 'avg_profit': None, 'median_profit': None, 'total_profit': 0, 'profit': 0.0, 'duration': None}, # noqa: E501 'results_explanation': ' 0 trades. Avg profit nan%. Total profit 0.00000000 BTC ( 0.00Σ%). Avg duration nan min.', # noqa: E501 'total_profit': 0, 'current_epoch': 4, 'is_initial_point': True, 'is_best': False }, { 'loss': 0.22195522184191518, 'params_dict': {'mfi-value': 17, 'fastd-value': 21, 'adx-value': 38, 'rsi-value': 33, 'mfi-enabled': True, 'fastd-enabled': False, 'adx-enabled': True, 'rsi-enabled': False, 'trigger': 'macd_cross_signal', 'sell-mfi-value': 87, 'sell-fastd-value': 82, 'sell-adx-value': 78, 'sell-rsi-value': 69, 'sell-mfi-enabled': True, 'sell-fastd-enabled': False, 'sell-adx-enabled': True, 'sell-rsi-enabled': False, 'sell-trigger': 'sell-macd_cross_signal', 'roi_t1': 1269, 'roi_t2': 601, 'roi_t3': 444, 'roi_p1': 0.07280999507931168, 'roi_p2': 0.08946698095898986, 'roi_p3': 0.1454876733325284, 'stoploss': -0.18181041180901014}, # noqa: E501 'params_details': {'buy': {'mfi-value': 17, 'fastd-value': 21, 'adx-value': 38, 'rsi-value': 33, 'mfi-enabled': True, 'fastd-enabled': False, 'adx-enabled': True, 'rsi-enabled': False, 'trigger': 'macd_cross_signal'}, 'sell': {'sell-mfi-value': 87, 'sell-fastd-value': 82, 'sell-adx-value': 78, 'sell-rsi-value': 69, 'sell-mfi-enabled': True, 'sell-fastd-enabled': False, 'sell-adx-enabled': True, 'sell-rsi-enabled': False, 'sell-trigger': 'sell-macd_cross_signal'}, 'roi': {0: 0.3077646493708299, 444: 0.16227697603830155, 1045: 0.07280999507931168, 2314: 0}, 'stoploss': {'stoploss': -0.18181041180901014}}, # noqa: E501 - 'results_metrics': {'trade_count': 14, 'avg_profit': -0.3539515, 'total_profit': -0.002480140000000001, 'profit': -4.955321, 'duration': 3402.8571428571427}, # noqa: E501 + 'results_metrics': {'trade_count': 14, 'avg_profit': -0.3539515, 'median_profit': -1.2222, 'total_profit': -0.002480140000000001, 'profit': -4.955321, 'duration': 3402.8571428571427}, # noqa: E501 'results_explanation': ' 14 trades. Avg profit -0.35%. Total profit -0.00248014 BTC ( -4.96Σ%). Avg duration 3402.9 min.', # noqa: E501 'total_profit': -0.002480140000000001, 'current_epoch': 5, @@ -1818,7 +1818,7 @@ def hyperopt_results(): 'loss': 0.545315889154162, 'params_dict': {'mfi-value': 22, 'fastd-value': 43, 'adx-value': 46, 'rsi-value': 20, 'mfi-enabled': False, 'fastd-enabled': False, 'adx-enabled': True, 'rsi-enabled': True, 'trigger': 'bb_lower', 'sell-mfi-value': 87, 'sell-fastd-value': 65, 'sell-adx-value': 94, 'sell-rsi-value': 63, 'sell-mfi-enabled': False, 'sell-fastd-enabled': True, 'sell-adx-enabled': True, 'sell-rsi-enabled': True, 'sell-trigger': 'sell-macd_cross_signal', 'roi_t1': 319, 'roi_t2': 556, 'roi_t3': 216, 'roi_p1': 0.06251955472249589, 'roi_p2': 0.11659519602202795, 'roi_p3': 0.0953744132197762, 'stoploss': -0.024551752215582423}, # noqa: E501 'params_details': {'buy': {'mfi-value': 22, 'fastd-value': 43, 'adx-value': 46, 'rsi-value': 20, 'mfi-enabled': False, 'fastd-enabled': False, 'adx-enabled': True, 'rsi-enabled': True, 'trigger': 'bb_lower'}, 'sell': {'sell-mfi-value': 87, 'sell-fastd-value': 65, 'sell-adx-value': 94, 'sell-rsi-value': 63, 'sell-mfi-enabled': False, 'sell-fastd-enabled': True, 'sell-adx-enabled': True, 'sell-rsi-enabled': True, 'sell-trigger': 'sell-macd_cross_signal'}, 'roi': {0: 0.2744891639643, 216: 0.17911475074452382, 772: 0.06251955472249589, 1091: 0}, 'stoploss': {'stoploss': -0.024551752215582423}}, # noqa: E501 - 'results_metrics': {'trade_count': 39, 'avg_profit': -0.21400679487179478, 'total_profit': -0.0041773, 'profit': -8.346264999999997, 'duration': 636.9230769230769}, # noqa: E501 + 'results_metrics': {'trade_count': 39, 'avg_profit': -0.21400679487179478, 'median_profit': -1.2222, 'total_profit': -0.0041773, 'profit': -8.346264999999997, 'duration': 636.9230769230769}, # noqa: E501 'results_explanation': ' 39 trades. Avg profit -0.21%. Total profit -0.00417730 BTC ( -8.35Σ%). Avg duration 636.9 min.', # noqa: E501 'total_profit': -0.0041773, 'current_epoch': 6, @@ -1830,7 +1830,7 @@ def hyperopt_results(): 'params_details': { 'buy': {'mfi-value': 13, 'fastd-value': 41, 'adx-value': 21, 'rsi-value': 29, 'mfi-enabled': False, 'fastd-enabled': True, 'adx-enabled': False, 'rsi-enabled': False, 'trigger': 'bb_lower'}, 'sell': {'sell-mfi-value': 99, 'sell-fastd-value': 60, 'sell-adx-value': 81, 'sell-rsi-value': 69, 'sell-mfi-enabled': True, 'sell-fastd-enabled': True, 'sell-adx-enabled': True, 'sell-rsi-enabled': False, 'sell-trigger': 'sell-macd_cross_signal'}, 'roi': {0: 0.4837436938134452, 145: 0.10853310701097472, 765: 0.0586919200378493, 1536: 0}, # noqa: E501 'stoploss': {'stoploss': -0.14613268022709905}}, # noqa: E501 - 'results_metrics': {'trade_count': 318, 'avg_profit': -0.39833954716981146, 'total_profit': -0.06339929, 'profit': -126.67197600000004, 'duration': 3140.377358490566}, # noqa: E501 + 'results_metrics': {'trade_count': 318, 'avg_profit': -0.39833954716981146, 'median_profit': -1.2222, 'total_profit': -0.06339929, 'profit': -126.67197600000004, 'duration': 3140.377358490566}, # noqa: E501 'results_explanation': ' 318 trades. Avg profit -0.40%. Total profit -0.06339929 BTC (-126.67Σ%). Avg duration 3140.4 min.', # noqa: E501 'total_profit': -0.06339929, 'current_epoch': 7, @@ -1840,7 +1840,7 @@ def hyperopt_results(): 'loss': 20.0, # noqa: E501 'params_dict': {'mfi-value': 24, 'fastd-value': 43, 'adx-value': 33, 'rsi-value': 20, 'mfi-enabled': False, 'fastd-enabled': True, 'adx-enabled': True, 'rsi-enabled': True, 'trigger': 'sar_reversal', 'sell-mfi-value': 89, 'sell-fastd-value': 74, 'sell-adx-value': 70, 'sell-rsi-value': 70, 'sell-mfi-enabled': False, 'sell-fastd-enabled': False, 'sell-adx-enabled': False, 'sell-rsi-enabled': True, 'sell-trigger': 'sell-sar_reversal', 'roi_t1': 1149, 'roi_t2': 375, 'roi_t3': 289, 'roi_p1': 0.05571820757172588, 'roi_p2': 0.0606240398618907, 'roi_p3': 0.1729012220156157, 'stoploss': -0.1588514289110401}, # noqa: E501 'params_details': {'buy': {'mfi-value': 24, 'fastd-value': 43, 'adx-value': 33, 'rsi-value': 20, 'mfi-enabled': False, 'fastd-enabled': True, 'adx-enabled': True, 'rsi-enabled': True, 'trigger': 'sar_reversal'}, 'sell': {'sell-mfi-value': 89, 'sell-fastd-value': 74, 'sell-adx-value': 70, 'sell-rsi-value': 70, 'sell-mfi-enabled': False, 'sell-fastd-enabled': False, 'sell-adx-enabled': False, 'sell-rsi-enabled': True, 'sell-trigger': 'sell-sar_reversal'}, 'roi': {0: 0.2892434694492323, 289: 0.11634224743361658, 664: 0.05571820757172588, 1813: 0}, 'stoploss': {'stoploss': -0.1588514289110401}}, # noqa: E501 - 'results_metrics': {'trade_count': 1, 'avg_profit': 0.0, 'total_profit': 0.0, 'profit': 0.0, 'duration': 5340.0}, # noqa: E501 + 'results_metrics': {'trade_count': 1, 'avg_profit': 0.0, 'median_profit': 0.0, 'total_profit': 0.0, 'profit': 0.0, 'duration': 5340.0}, # noqa: E501 'results_explanation': ' 1 trades. Avg profit 0.00%. Total profit 0.00000000 BTC ( 0.00Σ%). Avg duration 5340.0 min.', # noqa: E501 'total_profit': 0.0, 'current_epoch': 8, @@ -1850,7 +1850,7 @@ def hyperopt_results(): 'loss': 2.4731817780991223, 'params_dict': {'mfi-value': 22, 'fastd-value': 20, 'adx-value': 29, 'rsi-value': 40, 'mfi-enabled': False, 'fastd-enabled': False, 'adx-enabled': False, 'rsi-enabled': False, 'trigger': 'sar_reversal', 'sell-mfi-value': 97, 'sell-fastd-value': 65, 'sell-adx-value': 81, 'sell-rsi-value': 64, 'sell-mfi-enabled': True, 'sell-fastd-enabled': True, 'sell-adx-enabled': True, 'sell-rsi-enabled': True, 'sell-trigger': 'sell-bb_upper', 'roi_t1': 1012, 'roi_t2': 584, 'roi_t3': 422, 'roi_p1': 0.036764323603472565, 'roi_p2': 0.10335480573205287, 'roi_p3': 0.10322347377503042, 'stoploss': -0.2780610808108503}, # noqa: E501 'params_details': {'buy': {'mfi-value': 22, 'fastd-value': 20, 'adx-value': 29, 'rsi-value': 40, 'mfi-enabled': False, 'fastd-enabled': False, 'adx-enabled': False, 'rsi-enabled': False, 'trigger': 'sar_reversal'}, 'sell': {'sell-mfi-value': 97, 'sell-fastd-value': 65, 'sell-adx-value': 81, 'sell-rsi-value': 64, 'sell-mfi-enabled': True, 'sell-fastd-enabled': True, 'sell-adx-enabled': True, 'sell-rsi-enabled': True, 'sell-trigger': 'sell-bb_upper'}, 'roi': {0: 0.2433426031105559, 422: 0.14011912933552545, 1006: 0.036764323603472565, 2018: 0}, 'stoploss': {'stoploss': -0.2780610808108503}}, # noqa: E501 - 'results_metrics': {'trade_count': 229, 'avg_profit': -0.38433433624454144, 'total_profit': -0.044050070000000004, 'profit': -88.01256299999999, 'duration': 6505.676855895196}, # noqa: E501 + 'results_metrics': {'trade_count': 229, 'avg_profit': -0.38433433624454144, 'median_profit': -1.2222, 'total_profit': -0.044050070000000004, 'profit': -88.01256299999999, 'duration': 6505.676855895196}, # noqa: E501 'results_explanation': ' 229 trades. Avg profit -0.38%. Total profit -0.04405007 BTC ( -88.01Σ%). Avg duration 6505.7 min.', # noqa: E501 'total_profit': -0.044050070000000004, # noqa: E501 'current_epoch': 9, @@ -1860,7 +1860,7 @@ def hyperopt_results(): 'loss': -0.2604606005845212, # noqa: E501 'params_dict': {'mfi-value': 23, 'fastd-value': 24, 'adx-value': 22, 'rsi-value': 24, 'mfi-enabled': False, 'fastd-enabled': False, 'adx-enabled': False, 'rsi-enabled': True, 'trigger': 'macd_cross_signal', 'sell-mfi-value': 97, 'sell-fastd-value': 70, 'sell-adx-value': 64, 'sell-rsi-value': 80, 'sell-mfi-enabled': False, 'sell-fastd-enabled': True, 'sell-adx-enabled': True, 'sell-rsi-enabled': True, 'sell-trigger': 'sell-sar_reversal', 'roi_t1': 792, 'roi_t2': 464, 'roi_t3': 215, 'roi_p1': 0.04594053535385903, 'roi_p2': 0.09623192684243963, 'roi_p3': 0.04428219070850663, 'stoploss': -0.16992287161634415}, # noqa: E501 'params_details': {'buy': {'mfi-value': 23, 'fastd-value': 24, 'adx-value': 22, 'rsi-value': 24, 'mfi-enabled': False, 'fastd-enabled': False, 'adx-enabled': False, 'rsi-enabled': True, 'trigger': 'macd_cross_signal'}, 'sell': {'sell-mfi-value': 97, 'sell-fastd-value': 70, 'sell-adx-value': 64, 'sell-rsi-value': 80, 'sell-mfi-enabled': False, 'sell-fastd-enabled': True, 'sell-adx-enabled': True, 'sell-rsi-enabled': True, 'sell-trigger': 'sell-sar_reversal'}, 'roi': {0: 0.18645465290480528, 215: 0.14217246219629864, 679: 0.04594053535385903, 1471: 0}, 'stoploss': {'stoploss': -0.16992287161634415}}, # noqa: E501 - 'results_metrics': {'trade_count': 4, 'avg_profit': 0.1080385, 'total_profit': 0.00021629, 'profit': 0.432154, 'duration': 2850.0}, # noqa: E501 + 'results_metrics': {'trade_count': 4, 'avg_profit': 0.1080385, 'median_profit': -1.2222, 'total_profit': 0.00021629, 'profit': 0.432154, 'duration': 2850.0}, # noqa: E501 'results_explanation': ' 4 trades. Avg profit 0.11%. Total profit 0.00021629 BTC ( 0.43Σ%). Avg duration 2850.0 min.', # noqa: E501 'total_profit': 0.00021629, 'current_epoch': 10, @@ -1870,7 +1870,7 @@ def hyperopt_results(): 'loss': 4.876465945994304, # noqa: E501 'params_dict': {'mfi-value': 20, 'fastd-value': 32, 'adx-value': 49, 'rsi-value': 23, 'mfi-enabled': True, 'fastd-enabled': True, 'adx-enabled': False, 'rsi-enabled': False, 'trigger': 'bb_lower', 'sell-mfi-value': 75, 'sell-fastd-value': 56, 'sell-adx-value': 61, 'sell-rsi-value': 62, 'sell-mfi-enabled': False, 'sell-fastd-enabled': False, 'sell-adx-enabled': True, 'sell-rsi-enabled': True, 'sell-trigger': 'sell-macd_cross_signal', 'roi_t1': 579, 'roi_t2': 614, 'roi_t3': 273, 'roi_p1': 0.05307643172744114, 'roi_p2': 0.1352282078262871, 'roi_p3': 0.1913307406325751, 'stoploss': -0.25728526022513887}, # noqa: E501 'params_details': {'buy': {'mfi-value': 20, 'fastd-value': 32, 'adx-value': 49, 'rsi-value': 23, 'mfi-enabled': True, 'fastd-enabled': True, 'adx-enabled': False, 'rsi-enabled': False, 'trigger': 'bb_lower'}, 'sell': {'sell-mfi-value': 75, 'sell-fastd-value': 56, 'sell-adx-value': 61, 'sell-rsi-value': 62, 'sell-mfi-enabled': False, 'sell-fastd-enabled': False, 'sell-adx-enabled': True, 'sell-rsi-enabled': True, 'sell-trigger': 'sell-macd_cross_signal'}, 'roi': {0: 0.3796353801863034, 273: 0.18830463955372825, 887: 0.05307643172744114, 1466: 0}, 'stoploss': {'stoploss': -0.25728526022513887}}, # noqa: E501 - 'results_metrics': {'trade_count': 117, 'avg_profit': -1.2698609145299145, 'total_profit': -0.07436117, 'profit': -148.573727, 'duration': 4282.5641025641025}, # noqa: E501 + 'results_metrics': {'trade_count': 117, 'avg_profit': -1.2698609145299145, 'median_profit': -1.2222, 'total_profit': -0.07436117, 'profit': -148.573727, 'duration': 4282.5641025641025}, # noqa: E501 'results_explanation': ' 117 trades. Avg profit -1.27%. Total profit -0.07436117 BTC (-148.57Σ%). Avg duration 4282.6 min.', # noqa: E501 'total_profit': -0.07436117, 'current_epoch': 11, @@ -1880,7 +1880,7 @@ def hyperopt_results(): 'loss': 100000, 'params_dict': {'mfi-value': 10, 'fastd-value': 36, 'adx-value': 31, 'rsi-value': 22, 'mfi-enabled': True, 'fastd-enabled': True, 'adx-enabled': True, 'rsi-enabled': False, 'trigger': 'sar_reversal', 'sell-mfi-value': 80, 'sell-fastd-value': 71, 'sell-adx-value': 60, 'sell-rsi-value': 85, 'sell-mfi-enabled': False, 'sell-fastd-enabled': False, 'sell-adx-enabled': True, 'sell-rsi-enabled': True, 'sell-trigger': 'sell-bb_upper', 'roi_t1': 1156, 'roi_t2': 581, 'roi_t3': 408, 'roi_p1': 0.06860454019988212, 'roi_p2': 0.12473718444931989, 'roi_p3': 0.2896360635226823, 'stoploss': -0.30889015124682806}, # noqa: E501 'params_details': {'buy': {'mfi-value': 10, 'fastd-value': 36, 'adx-value': 31, 'rsi-value': 22, 'mfi-enabled': True, 'fastd-enabled': True, 'adx-enabled': True, 'rsi-enabled': False, 'trigger': 'sar_reversal'}, 'sell': {'sell-mfi-value': 80, 'sell-fastd-value': 71, 'sell-adx-value': 60, 'sell-rsi-value': 85, 'sell-mfi-enabled': False, 'sell-fastd-enabled': False, 'sell-adx-enabled': True, 'sell-rsi-enabled': True, 'sell-trigger': 'sell-bb_upper'}, 'roi': {0: 0.4829777881718843, 408: 0.19334172464920202, 989: 0.06860454019988212, 2145: 0}, 'stoploss': {'stoploss': -0.30889015124682806}}, # noqa: E501 - 'results_metrics': {'trade_count': 0, 'avg_profit': None, 'total_profit': 0, 'profit': 0.0, 'duration': None}, # noqa: E501 + 'results_metrics': {'trade_count': 0, 'avg_profit': None, 'median_profit': None, 'total_profit': 0, 'profit': 0.0, 'duration': None}, # noqa: E501 'results_explanation': ' 0 trades. Avg profit nan%. Total profit 0.00000000 BTC ( 0.00Σ%). Avg duration nan min.', # noqa: E501 'total_profit': 0, 'current_epoch': 12, From d5993db064bcbd5fd1c83163e68d3e26a6af31c2 Mon Sep 17 00:00:00 2001 From: Joe Schr Date: Wed, 3 Mar 2021 14:59:55 +0100 Subject: [PATCH 086/187] refactor(docs/strategy-customization): change variable name for better readability `cust_info` -> `custom_info` --- docs/strategy-customization.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/docs/strategy-customization.md b/docs/strategy-customization.md index fdc95a3c1..6eaafa15c 100644 --- a/docs/strategy-customization.md +++ b/docs/strategy-customization.md @@ -311,18 +311,18 @@ The name of the variable can be chosen at will, but should be prefixed with `cus ```python class AwesomeStrategy(IStrategy): # Create custom dictionary - cust_info = {} + custom_info = {} def populate_indicators(self, dataframe: DataFrame, metadata: dict) -> DataFrame: # Check if the entry already exists - if not metadata["pair"] in self.cust_info: + if not metadata["pair"] in self.custom_info: # Create empty entry for this pair - self.cust_info[metadata["pair"]] = {} + self.custom_info[metadata["pair"]] = {} - if "crosstime" in self.cust_info[metadata["pair"]]: - self.cust_info[metadata["pair"]]["crosstime"] += 1 + if "crosstime" in self.custom_info[metadata["pair"]]: + self.custom_info[metadata["pair"]]["crosstime"] += 1 else: - self.cust_info[metadata["pair"]]["crosstime"] = 1 + self.custom_info[metadata["pair"]]["crosstime"] = 1 ``` !!! Warning From 5cf3194fab8b026d204f8992aa0ad76c18c8bcb5 Mon Sep 17 00:00:00 2001 From: Joe Schr Date: Wed, 3 Mar 2021 15:03:44 +0100 Subject: [PATCH 087/187] chore(docs/strategy-customization): clean up left over trailing whitespaces --- docs/strategy-advanced.md | 2 +- docs/strategy-customization.md | 6 +++--- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/strategy-advanced.md b/docs/strategy-advanced.md index dcd340fd1..2fe29d431 100644 --- a/docs/strategy-advanced.md +++ b/docs/strategy-advanced.md @@ -142,7 +142,7 @@ class AwesomeStrategy(IStrategy): return -1 # return a value bigger than the inital stoploss to keep using the inital stoploss # After reaching the desired offset, allow the stoploss to trail by half the profit - desired_stoploss = current_profit / 2 + desired_stoploss = current_profit / 2 # Use a minimum of 2.5% and a maximum of 5% return max(min(desired_stoploss, 0.05), 0.025) diff --git a/docs/strategy-customization.md b/docs/strategy-customization.md index 6eaafa15c..983a5f60a 100644 --- a/docs/strategy-customization.md +++ b/docs/strategy-customization.md @@ -399,7 +399,7 @@ if self.dp: ### *current_whitelist()* -Imagine you've developed a strategy that trades the `5m` timeframe using signals generated from a `1d` timeframe on the top 10 volume pairs by volume. +Imagine you've developed a strategy that trades the `5m` timeframe using signals generated from a `1d` timeframe on the top 10 volume pairs by volume. The strategy might look something like this: @@ -418,7 +418,7 @@ This is where calling `self.dp.current_whitelist()` comes in handy. pairs = self.dp.current_whitelist() # Assign tf to each pair so they can be downloaded and cached for strategy. informative_pairs = [(pair, '1d') for pair in pairs] - return informative_pairs + return informative_pairs ``` ### *get_pair_dataframe(pair, timeframe)* @@ -583,7 +583,7 @@ All columns of the informative dataframe will be available on the returning data ``` python 'date', 'open', 'high', 'low', 'close', 'rsi' # from the original dataframe - 'date_1h', 'open_1h', 'high_1h', 'low_1h', 'close_1h', 'rsi_1h' # from the informative dataframe + 'date_1h', 'open_1h', 'high_1h', 'low_1h', 'close_1h', 'rsi_1h' # from the informative dataframe ``` ??? Example "Custom implementation" From cc4e84bb7009330031dfd0aadc8b2db9a1d13b49 Mon Sep 17 00:00:00 2001 From: Joe Schr Date: Wed, 3 Mar 2021 15:04:18 +0100 Subject: [PATCH 088/187] feature(docs/strategy-customization): add example how to store indicator with DatetimeIndex into custom_info --- docs/strategy-customization.md | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/docs/strategy-customization.md b/docs/strategy-customization.md index 983a5f60a..0b09e073f 100644 --- a/docs/strategy-customization.md +++ b/docs/strategy-customization.md @@ -333,6 +333,22 @@ class AwesomeStrategy(IStrategy): *** +#### Storing custom information using DatetimeIndex from `dataframe` + +Imagine you need to store an indicator like `ATR` or `RSI` into `custom_info`. To use this in a meaningful way, you will not only need the raw data of the indicator, but probably also need to keep the right timestamps. + +class AwesomeStrategy(IStrategy): + # Create custom dictionary + custom_info = {} + + def populate_indicators(self, dataframe: DataFrame, metadata: dict) -> DataFrame: + # add indicator mapped to correct DatetimeIndex to custom_info + # using "ATR" here as example + self.custom_info[metadata['pair']] = dataframe[['date', 'atr']].copy().set_index('date') + return dataframe + +See: (custom_stoploss example)[WIP] for how to access it + ## Additional data (informative_pairs) ### Get data for non-tradeable pairs From c5900bbd384e3d3c22a5bc717dfddb47dc13c6a4 Mon Sep 17 00:00:00 2001 From: Joe Schr Date: Wed, 3 Mar 2021 15:16:27 +0100 Subject: [PATCH 089/187] feature(docs/strategy-customization): add example "Custom stoploss using an indicator from dataframe" --- docs/strategy-advanced.md | 29 +++++++++++++++++++++++++++++ 1 file changed, 29 insertions(+) diff --git a/docs/strategy-advanced.md b/docs/strategy-advanced.md index 2fe29d431..0cd4d1be0 100644 --- a/docs/strategy-advanced.md +++ b/docs/strategy-advanced.md @@ -179,6 +179,35 @@ class AwesomeStrategy(IStrategy): return (-0.07 + current_profit) return 1 ``` +#### Custom stoploss using an indicator from dataframe example + +Imagine you want to use `custom_stoploss()` to use a trailing indicator like e.g. "ATR" + +See: (Storing custom information using DatetimeIndex from `dataframe` +)[WIP] on how to store the indicator into `custom_info` + +``` python +from freqtrade.persistence import Trade + +class AwesomeStrategy(IStrategy): + + # ... populate_* methods + + use_custom_stoploss = True + + def custom_stoploss(self, pair: str, trade: 'Trade', current_time: datetime, + current_rate: float, current_profit: float, **kwargs) -> float: + + result = 1 + if self.custom_info[pair] is not None and trade is not None: + atr = self.custom_info[pair].loc[current_time]['atr'] + if (atr is not None): + # new stoploss relative to current_rate + new_stoploss = (current_rate-atr)/current_rate + # turn into relative negative offset required by `custom_stoploss` return implementation + result = new_stoploss - 1 + return result +``` --- From 32f35fcd904f44b8a96e23d726da1e5f1e5484b3 Mon Sep 17 00:00:00 2001 From: Joe Schr Date: Wed, 3 Mar 2021 21:26:21 +0100 Subject: [PATCH 090/187] fix(docs/strategy-customization): "custom_stoploss indicator" example need to check for RUN_MODE --- docs/strategy-advanced.md | 18 +++++++++++++++--- 1 file changed, 15 insertions(+), 3 deletions(-) diff --git a/docs/strategy-advanced.md b/docs/strategy-advanced.md index 0cd4d1be0..8c730b3df 100644 --- a/docs/strategy-advanced.md +++ b/docs/strategy-advanced.md @@ -188,6 +188,7 @@ See: (Storing custom information using DatetimeIndex from `dataframe` ``` python from freqtrade.persistence import Trade +from freqtrade.state import RunMode class AwesomeStrategy(IStrategy): @@ -200,12 +201,23 @@ class AwesomeStrategy(IStrategy): result = 1 if self.custom_info[pair] is not None and trade is not None: - atr = self.custom_info[pair].loc[current_time]['atr'] - if (atr is not None): + # using current_time directly (like below) will only work in backtesting. + # so check "runmode" to make sure that it's only used in backtesting + if(self.dp.runmode == RunMode.BACKTEST): + relative_sl = self.custom_info[pair].loc[current_time]['atr] + # in live / dry-run, it'll be really the current time + else: + # but we can just use the last entry to get the current value + relative_sl = self.custom_info[pair]['atr].iloc[ -1 ] + + if (relative_sl is not None): + print("Custom SL: {}".format(relative_sl)) # new stoploss relative to current_rate - new_stoploss = (current_rate-atr)/current_rate + new_stoploss = (current_rate-relative_sl)/current_rate # turn into relative negative offset required by `custom_stoploss` return implementation result = new_stoploss - 1 + + print("Result: {}".format(result)) return result ``` From d05acc30fa2819fe9ca03c224b62d3b46cae86fe Mon Sep 17 00:00:00 2001 From: Joe Schr Date: Wed, 3 Mar 2021 22:10:08 +0100 Subject: [PATCH 091/187] fix(docs/strategy-customization): remove superflous `prints` from example code --- docs/strategy-advanced.md | 2 -- 1 file changed, 2 deletions(-) diff --git a/docs/strategy-advanced.md b/docs/strategy-advanced.md index 8c730b3df..504b7270e 100644 --- a/docs/strategy-advanced.md +++ b/docs/strategy-advanced.md @@ -211,13 +211,11 @@ class AwesomeStrategy(IStrategy): relative_sl = self.custom_info[pair]['atr].iloc[ -1 ] if (relative_sl is not None): - print("Custom SL: {}".format(relative_sl)) # new stoploss relative to current_rate new_stoploss = (current_rate-relative_sl)/current_rate # turn into relative negative offset required by `custom_stoploss` return implementation result = new_stoploss - 1 - print("Result: {}".format(result)) return result ``` From b52698197b469d7739076b2c538f7c216f27cc8e Mon Sep 17 00:00:00 2001 From: Joe Schr Date: Thu, 4 Mar 2021 14:44:51 +0100 Subject: [PATCH 092/187] refactor(docs/strategy-advanced): extract "Storing information" section from `strategy-customization.md` --- docs/strategy-advanced.md | 47 ++++++++++++++++++++++++++++++++++ docs/strategy-customization.md | 47 ---------------------------------- 2 files changed, 47 insertions(+), 47 deletions(-) diff --git a/docs/strategy-advanced.md b/docs/strategy-advanced.md index 504b7270e..2cd411078 100644 --- a/docs/strategy-advanced.md +++ b/docs/strategy-advanced.md @@ -11,6 +11,53 @@ If you're just getting started, please be familiar with the methods described in !!! Tip You can get a strategy template containing all below methods by running `freqtrade new-strategy --strategy MyAwesomeStrategy --template advanced` +## Storing information + +Storing information can be accomplished by creating a new dictionary within the strategy class. + +The name of the variable can be chosen at will, but should be prefixed with `cust_` to avoid naming collisions with predefined strategy variables. + +```python +class AwesomeStrategy(IStrategy): + # Create custom dictionary + custom_info = {} + + def populate_indicators(self, dataframe: DataFrame, metadata: dict) -> DataFrame: + # Check if the entry already exists + if not metadata["pair"] in self.custom_info: + # Create empty entry for this pair + self.custom_info[metadata["pair"]] = {} + + if "crosstime" in self.custom_info[metadata["pair"]]: + self.custom_info[metadata["pair"]]["crosstime"] += 1 + else: + self.custom_info[metadata["pair"]]["crosstime"] = 1 +``` + +!!! Warning + The data is not persisted after a bot-restart (or config-reload). Also, the amount of data should be kept smallish (no DataFrames and such), otherwise the bot will start to consume a lot of memory and eventually run out of memory and crash. + +!!! Note + If the data is pair-specific, make sure to use pair as one of the keys in the dictionary. + +*** + +### Storing custom information using DatetimeIndex from `dataframe` + +Imagine you need to store an indicator like `ATR` or `RSI` into `custom_info`. To use this in a meaningful way, you will not only need the raw data of the indicator, but probably also need to keep the right timestamps. + +class AwesomeStrategy(IStrategy): + # Create custom dictionary + custom_info = {} + + def populate_indicators(self, dataframe: DataFrame, metadata: dict) -> DataFrame: + # add indicator mapped to correct DatetimeIndex to custom_info + # using "ATR" here as example + self.custom_info[metadata['pair']] = dataframe[['date', 'atr']].copy().set_index('date') + return dataframe + +See `custom_stoploss` examples below on how to access the saved dataframe columns + ## Custom stoploss A stoploss can only ever move upwards - so if you set it to an absolute profit of 2%, you can never move it below this price. diff --git a/docs/strategy-customization.md b/docs/strategy-customization.md index 0b09e073f..a66be013e 100644 --- a/docs/strategy-customization.md +++ b/docs/strategy-customization.md @@ -302,53 +302,6 @@ Currently this is `pair`, which can be accessed using `metadata['pair']` - and w The Metadata-dict should not be modified and does not persist information across multiple calls. Instead, have a look at the section [Storing information](#Storing-information) -### Storing information - -Storing information can be accomplished by creating a new dictionary within the strategy class. - -The name of the variable can be chosen at will, but should be prefixed with `cust_` to avoid naming collisions with predefined strategy variables. - -```python -class AwesomeStrategy(IStrategy): - # Create custom dictionary - custom_info = {} - - def populate_indicators(self, dataframe: DataFrame, metadata: dict) -> DataFrame: - # Check if the entry already exists - if not metadata["pair"] in self.custom_info: - # Create empty entry for this pair - self.custom_info[metadata["pair"]] = {} - - if "crosstime" in self.custom_info[metadata["pair"]]: - self.custom_info[metadata["pair"]]["crosstime"] += 1 - else: - self.custom_info[metadata["pair"]]["crosstime"] = 1 -``` - -!!! Warning - The data is not persisted after a bot-restart (or config-reload). Also, the amount of data should be kept smallish (no DataFrames and such), otherwise the bot will start to consume a lot of memory and eventually run out of memory and crash. - -!!! Note - If the data is pair-specific, make sure to use pair as one of the keys in the dictionary. - -*** - -#### Storing custom information using DatetimeIndex from `dataframe` - -Imagine you need to store an indicator like `ATR` or `RSI` into `custom_info`. To use this in a meaningful way, you will not only need the raw data of the indicator, but probably also need to keep the right timestamps. - -class AwesomeStrategy(IStrategy): - # Create custom dictionary - custom_info = {} - - def populate_indicators(self, dataframe: DataFrame, metadata: dict) -> DataFrame: - # add indicator mapped to correct DatetimeIndex to custom_info - # using "ATR" here as example - self.custom_info[metadata['pair']] = dataframe[['date', 'atr']].copy().set_index('date') - return dataframe - -See: (custom_stoploss example)[WIP] for how to access it - ## Additional data (informative_pairs) ### Get data for non-tradeable pairs From 4064f856d1b47d68dbd1d40aa09cbb47c342720c Mon Sep 17 00:00:00 2001 From: Joe Schr Date: Thu, 4 Mar 2021 14:50:04 +0100 Subject: [PATCH 093/187] fix(docs/strategy-customization): add "hyperopt" to runmode check for custom_info in custom_stoploss example --- docs/strategy-advanced.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/strategy-advanced.md b/docs/strategy-advanced.md index 2cd411078..f230a2371 100644 --- a/docs/strategy-advanced.md +++ b/docs/strategy-advanced.md @@ -249,8 +249,8 @@ class AwesomeStrategy(IStrategy): result = 1 if self.custom_info[pair] is not None and trade is not None: # using current_time directly (like below) will only work in backtesting. - # so check "runmode" to make sure that it's only used in backtesting - if(self.dp.runmode == RunMode.BACKTEST): + # so check "runmode" to make sure that it's only used in backtesting/hyperopt + if self.dp and self.dp.runmode.value in ('backtest', 'hyperopt'): relative_sl = self.custom_info[pair].loc[current_time]['atr] # in live / dry-run, it'll be really the current time else: From 1a02a146a1e1023fba4f040e88eec3d9b8ace32a Mon Sep 17 00:00:00 2001 From: Joe Schr Date: Thu, 4 Mar 2021 14:59:08 +0100 Subject: [PATCH 094/187] feature(docs/strategy-advanced/custom_info-storage/example): add ATR column calculation --- docs/strategy-advanced.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/docs/strategy-advanced.md b/docs/strategy-advanced.md index f230a2371..81ba24a67 100644 --- a/docs/strategy-advanced.md +++ b/docs/strategy-advanced.md @@ -46,15 +46,19 @@ class AwesomeStrategy(IStrategy): Imagine you need to store an indicator like `ATR` or `RSI` into `custom_info`. To use this in a meaningful way, you will not only need the raw data of the indicator, but probably also need to keep the right timestamps. +```python +import talib.abstract as ta class AwesomeStrategy(IStrategy): # Create custom dictionary custom_info = {} def populate_indicators(self, dataframe: DataFrame, metadata: dict) -> DataFrame: - # add indicator mapped to correct DatetimeIndex to custom_info # using "ATR" here as example + dataframe['atr'] = ta.ATR(dataframe) + # add indicator mapped to correct DatetimeIndex to custom_info self.custom_info[metadata['pair']] = dataframe[['date', 'atr']].copy().set_index('date') return dataframe +``` See `custom_stoploss` examples below on how to access the saved dataframe columns From 22a558e33120a9e16a29c53b4f00ee268e30d7fe Mon Sep 17 00:00:00 2001 From: Joe Schr Date: Thu, 4 Mar 2021 15:01:21 +0100 Subject: [PATCH 095/187] fix(docs/strategy-advanced): fix link to custom_info storage --- docs/strategy-advanced.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/docs/strategy-advanced.md b/docs/strategy-advanced.md index 81ba24a67..d685662eb 100644 --- a/docs/strategy-advanced.md +++ b/docs/strategy-advanced.md @@ -234,8 +234,7 @@ class AwesomeStrategy(IStrategy): Imagine you want to use `custom_stoploss()` to use a trailing indicator like e.g. "ATR" -See: (Storing custom information using DatetimeIndex from `dataframe` -)[WIP] on how to store the indicator into `custom_info` +See: "Storing custom information using DatetimeIndex from `dataframe`" example above) on how to store the indicator into `custom_info` ``` python from freqtrade.persistence import Trade From a6ef354a5fc5c289aee14ace1881055471370a10 Mon Sep 17 00:00:00 2001 From: Joe Schr Date: Thu, 4 Mar 2021 18:52:23 +0100 Subject: [PATCH 096/187] fix(docs/strategy-advanced): use `get_analyzed_dataframe()` instead of `custom_info.iloc` --- docs/strategy-advanced.md | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/docs/strategy-advanced.md b/docs/strategy-advanced.md index d685662eb..42ffaa423 100644 --- a/docs/strategy-advanced.md +++ b/docs/strategy-advanced.md @@ -257,8 +257,11 @@ class AwesomeStrategy(IStrategy): relative_sl = self.custom_info[pair].loc[current_time]['atr] # in live / dry-run, it'll be really the current time else: + # but we can just use the last entry from an already analyzed dataframe instead + dataframe, last_updated = self.dp.get_analyzed_dataframe(pair=pair, + timeframe=self.timeframe) # but we can just use the last entry to get the current value - relative_sl = self.custom_info[pair]['atr].iloc[ -1 ] + relative_sl = dataframe['atr'].iat[-1] if (relative_sl is not None): # new stoploss relative to current_rate From c56b9cd75172ef43d3de33814fd79c3c1f1a3ec1 Mon Sep 17 00:00:00 2001 From: Joe Schr Date: Thu, 4 Mar 2021 18:50:48 +0100 Subject: [PATCH 097/187] fix(docs/strategy-advanced): add warnings --- docs/strategy-advanced.md | 16 +++++++++++++++- 1 file changed, 15 insertions(+), 1 deletion(-) diff --git a/docs/strategy-advanced.md b/docs/strategy-advanced.md index 42ffaa423..4cc1adb49 100644 --- a/docs/strategy-advanced.md +++ b/docs/strategy-advanced.md @@ -60,6 +60,12 @@ class AwesomeStrategy(IStrategy): return dataframe ``` +!!! Warning + The data is not persisted after a bot-restart (or config-reload). Also, the amount of data should be kept smallish (no DataFrames and such), otherwise the bot will start to consume a lot of memory and eventually run out of memory and crash. + +!!! Note + If the data is pair-specific, make sure to use pair as one of the keys in the dictionary. + See `custom_stoploss` examples below on how to access the saved dataframe columns ## Custom stoploss @@ -236,6 +242,11 @@ Imagine you want to use `custom_stoploss()` to use a trailing indicator like e.g See: "Storing custom information using DatetimeIndex from `dataframe`" example above) on how to store the indicator into `custom_info` +!!! Warning + only use .iat[-1] in live mode, not in backtesting/hyperopt + otherwise you will look into the future + see: https://www.freqtrade.io/en/latest/strategy-customization/#common-mistakes-when-developing-strategies + ``` python from freqtrade.persistence import Trade from freqtrade.state import RunMode @@ -260,7 +271,10 @@ class AwesomeStrategy(IStrategy): # but we can just use the last entry from an already analyzed dataframe instead dataframe, last_updated = self.dp.get_analyzed_dataframe(pair=pair, timeframe=self.timeframe) - # but we can just use the last entry to get the current value + # WARNING + # only use .iat[-1] in live mode, not in backtesting/hyperopt + # otherwise you will look into the future + # see: https://www.freqtrade.io/en/latest/strategy-customization/#common-mistakes-when-developing-strategies relative_sl = dataframe['atr'].iat[-1] if (relative_sl is not None): From 900deb663a282b5e234993e6b9b36ccc5a4df487 Mon Sep 17 00:00:00 2001 From: Joe Schr Date: Thu, 4 Mar 2021 19:58:43 +0100 Subject: [PATCH 098/187] fix(docs/strategy-advanced/custom_stoploss/example): check if "pair" exists in "custom_info" before requesting --- docs/strategy-advanced.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/strategy-advanced.md b/docs/strategy-advanced.md index 4cc1adb49..c166f87fb 100644 --- a/docs/strategy-advanced.md +++ b/docs/strategy-advanced.md @@ -261,7 +261,7 @@ class AwesomeStrategy(IStrategy): current_rate: float, current_profit: float, **kwargs) -> float: result = 1 - if self.custom_info[pair] is not None and trade is not None: + if self.custom_info and pair in self.custom_info and trade: # using current_time directly (like below) will only work in backtesting. # so check "runmode" to make sure that it's only used in backtesting/hyperopt if self.dp and self.dp.runmode.value in ('backtest', 'hyperopt'): From 1304918a29de43e34b207b21a6f9b3b92bd79023 Mon Sep 17 00:00:00 2001 From: Joe Schr Date: Thu, 4 Mar 2021 19:59:57 +0100 Subject: [PATCH 099/187] fix(docs/strategy-advanced/custom_info-storage/example): only add to "custom_info" in backtesting and hyperopt --- docs/strategy-advanced.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/docs/strategy-advanced.md b/docs/strategy-advanced.md index c166f87fb..5c7ae83bc 100644 --- a/docs/strategy-advanced.md +++ b/docs/strategy-advanced.md @@ -55,8 +55,9 @@ class AwesomeStrategy(IStrategy): def populate_indicators(self, dataframe: DataFrame, metadata: dict) -> DataFrame: # using "ATR" here as example dataframe['atr'] = ta.ATR(dataframe) - # add indicator mapped to correct DatetimeIndex to custom_info - self.custom_info[metadata['pair']] = dataframe[['date', 'atr']].copy().set_index('date') + if self.dp.runmode.value in ('backtest', 'hyperopt'): + # add indicator mapped to correct DatetimeIndex to custom_info + self.custom_info[metadata['pair']] = dataframe[['date', 'atr']].copy().set_index('date') return dataframe ``` From 161a4656d5769ac3942fc2f1ec96d971bf224494 Mon Sep 17 00:00:00 2001 From: JoeSchr Date: Thu, 4 Mar 2021 20:05:21 +0100 Subject: [PATCH 100/187] Update docs/strategy-advanced.md Co-authored-by: Matthias --- docs/strategy-advanced.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/strategy-advanced.md b/docs/strategy-advanced.md index 5c7ae83bc..56061365e 100644 --- a/docs/strategy-advanced.md +++ b/docs/strategy-advanced.md @@ -246,7 +246,7 @@ See: "Storing custom information using DatetimeIndex from `dataframe`" example a !!! Warning only use .iat[-1] in live mode, not in backtesting/hyperopt otherwise you will look into the future - see: https://www.freqtrade.io/en/latest/strategy-customization/#common-mistakes-when-developing-strategies + see [Common mistakes when developing strategies](strategy-customization.md#common-mistakes-when-developing-strategies) for more info. ``` python from freqtrade.persistence import Trade From dfeafc22044b169c99c39436ac9679ea31211afc Mon Sep 17 00:00:00 2001 From: JoeSchr Date: Thu, 4 Mar 2021 20:05:27 +0100 Subject: [PATCH 101/187] Update docs/strategy-customization.md Co-authored-by: Matthias --- docs/strategy-customization.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/strategy-customization.md b/docs/strategy-customization.md index a66be013e..aebc51509 100644 --- a/docs/strategy-customization.md +++ b/docs/strategy-customization.md @@ -300,7 +300,7 @@ The metadata-dict (available for `populate_buy_trend`, `populate_sell_trend`, `p Currently this is `pair`, which can be accessed using `metadata['pair']` - and will return a pair in the format `XRP/BTC`. The Metadata-dict should not be modified and does not persist information across multiple calls. -Instead, have a look at the section [Storing information](#Storing-information) +Instead, have a look at the section [Storing information](strategy-advanced.md#Storing-information) ## Additional data (informative_pairs) From bc05d03126aa7a9622b2fa75552c1e026c17d0f6 Mon Sep 17 00:00:00 2001 From: Matthias Date: Fri, 5 Mar 2021 19:21:09 +0100 Subject: [PATCH 102/187] Make best / worst day absolute --- docs/backtesting.md | 10 +++++----- freqtrade/optimize/optimize_reports.py | 19 ++++++++++++++----- 2 files changed, 19 insertions(+), 10 deletions(-) diff --git a/docs/backtesting.md b/docs/backtesting.md index 2e91b6e74..d02c59f05 100644 --- a/docs/backtesting.md +++ b/docs/backtesting.md @@ -289,8 +289,8 @@ A backtesting result will look like that: | Worst Pair | ZEC/BTC -10.18% | | Best Trade | LSK/BTC 4.25% | | Worst Trade | ZEC/BTC -10.25% | -| Best day | 25.27% | -| Worst day | -30.67% | +| Best day | 0.00076 BTC | +| Worst day | -0.00036 BTC | | Days win/draw/lose | 12 / 82 / 25 | | Avg. Duration Winners | 4:23:00 | | Avg. Duration Loser | 6:55:00 | @@ -376,8 +376,8 @@ It contains some useful key metrics about performance of your strategy on backte | Worst Pair | ZEC/BTC -10.18% | | Best Trade | LSK/BTC 4.25% | | Worst Trade | ZEC/BTC -10.25% | -| Best day | 25.27% | -| Worst day | -30.67% | +| Best day | 0.00076 BTC | +| Worst day | -0.00036 BTC | | Days win/draw/lose | 12 / 82 / 25 | | Avg. Duration Winners | 4:23:00 | | Avg. Duration Loser | 6:55:00 | @@ -406,7 +406,7 @@ It contains some useful key metrics about performance of your strategy on backte - `Avg. stake amount`: Average stake amount, either `stake_amount` or the average when using dynamic stake amount. - `Total trade volume`: Volume generated on the exchange to reach the above profit. - `Best Pair` / `Worst Pair`: Best and worst performing pair, and it's corresponding `Cum Profit %`. -- `Best Trade` / `Worst Trade`: Biggest winning trade and biggest losing trade +- `Best Trade` / `Worst Trade`: Biggest single winning trade and biggest single losing trade. - `Best day` / `Worst day`: Best and worst day based on daily profit. - `Days win/draw/lose`: Winning / Losing days (draws are usually days without closed trade). - `Avg. Duration Winners` / `Avg. Duration Loser`: Average durations for winning and losing trades. diff --git a/freqtrade/optimize/optimize_reports.py b/freqtrade/optimize/optimize_reports.py index 52ae09ad1..099976aa9 100644 --- a/freqtrade/optimize/optimize_reports.py +++ b/freqtrade/optimize/optimize_reports.py @@ -196,13 +196,18 @@ def generate_daily_stats(results: DataFrame) -> Dict[str, Any]: return { 'backtest_best_day': 0, 'backtest_worst_day': 0, + 'backtest_best_day_abs': 0, + 'backtest_worst_day_abs': 0, 'winning_days': 0, 'draw_days': 0, 'losing_days': 0, 'winner_holding_avg': timedelta(), 'loser_holding_avg': timedelta(), } - daily_profit = results.resample('1d', on='close_date')['profit_ratio'].sum() + daily_profit_rel = results.resample('1d', on='close_date')['profit_ratio'].sum() + daily_profit = results.resample('1d', on='close_date')['profit_abs'].sum().round(10) + worst_rel = min(daily_profit_rel) + best_rel = max(daily_profit_rel) worst = min(daily_profit) best = max(daily_profit) winning_days = sum(daily_profit > 0) @@ -213,8 +218,10 @@ def generate_daily_stats(results: DataFrame) -> Dict[str, Any]: losing_trades = results.loc[results['profit_ratio'] < 0] return { - 'backtest_best_day': best, - 'backtest_worst_day': worst, + 'backtest_best_day': best_rel, + 'backtest_worst_day': worst_rel, + 'backtest_best_day_abs': best, + 'backtest_worst_day_abs': worst, 'winning_days': winning_days, 'draw_days': draw_days, 'losing_days': losing_days, @@ -470,8 +477,10 @@ def text_table_add_metrics(strat_results: Dict) -> str: ('Worst trade', f"{worst_trade['pair']} " f"{round(worst_trade['profit_ratio'] * 100, 2)}%"), - ('Best day', f"{round(strat_results['backtest_best_day'] * 100, 2)}%"), - ('Worst day', f"{round(strat_results['backtest_worst_day'] * 100, 2)}%"), + ('Best day', round_coin_value(strat_results['backtest_best_day_abs'], + strat_results['stake_currency'])), + ('Worst day', round_coin_value(strat_results['backtest_worst_day_abs'], + strat_results['stake_currency'])), ('Days win/draw/lose', f"{strat_results['winning_days']} / " f"{strat_results['draw_days']} / {strat_results['losing_days']}"), ('Avg. Duration Winners', f"{strat_results['winner_holding_avg']}"), From 731ab5d2a775ff5f10d345e24066ac0398cf597b Mon Sep 17 00:00:00 2001 From: Matthias Date: Fri, 5 Mar 2021 19:22:57 +0100 Subject: [PATCH 103/187] Fix too long line errors --- freqtrade/optimize/hyperopt.py | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/freqtrade/optimize/hyperopt.py b/freqtrade/optimize/hyperopt.py index c46d0da48..955f97f33 100644 --- a/freqtrade/optimize/hyperopt.py +++ b/freqtrade/optimize/hyperopt.py @@ -419,14 +419,16 @@ class Hyperopt: trials['Stake currency'] = config['stake_currency'] base_metrics = ['Best', 'current_epoch', 'results_metrics.trade_count', - 'results_metrics.avg_profit', 'results_metrics.median_profit', 'results_metrics.total_profit', + 'results_metrics.avg_profit', 'results_metrics.median_profit', + 'results_metrics.total_profit', 'Stake currency', 'results_metrics.profit', 'results_metrics.duration', 'loss', 'is_initial_point', 'is_best'] param_metrics = [("params_dict."+param) for param in results[0]['params_dict'].keys()] trials = trials[base_metrics + param_metrics] - base_columns = ['Best', 'Epoch', 'Trades', 'Avg profit', 'Median profit', 'Total profit', 'Stake currency', - 'Profit', 'Avg duration', 'Objective', 'is_initial_point', 'is_best'] + base_columns = ['Best', 'Epoch', 'Trades', 'Avg profit', 'Median profit', 'Total profit', + 'Stake currency', 'Profit', 'Avg duration', 'Objective', + 'is_initial_point', 'is_best'] param_columns = list(results[0]['params_dict'].keys()) trials.columns = base_columns + param_columns From 345f7404e990451e4e2024b20df9d9f0a975cdf0 Mon Sep 17 00:00:00 2001 From: Patrick Weber Date: Fri, 5 Mar 2021 12:56:11 -0600 Subject: [PATCH 104/187] Add strategy name to HyperOpt results filename This just extends the HyperOpt result filename by adding the strategy name. This allows analysis of HyperOpt results folder with no additional necessary context. An alternative idea would be to expand the result dict, but the additional static copies are non value added. --- freqtrade/optimize/hyperopt.py | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/freqtrade/optimize/hyperopt.py b/freqtrade/optimize/hyperopt.py index 955f97f33..66e11cf68 100644 --- a/freqtrade/optimize/hyperopt.py +++ b/freqtrade/optimize/hyperopt.py @@ -77,8 +77,9 @@ class Hyperopt: self.custom_hyperoptloss = HyperOptLossResolver.load_hyperoptloss(self.config) self.calculate_loss = self.custom_hyperoptloss.hyperopt_loss_function time_now = datetime.now().strftime("%Y-%m-%d_%H-%M-%S") + strategy = str(self.config['strategy']) self.results_file = (self.config['user_data_dir'] / - 'hyperopt_results' / f'hyperopt_results_{time_now}.pickle') + 'hyperopt_results' / f'strategy_{strategy}_' f'hyperopt_results_{time_now}.pickle') self.data_pickle_file = (self.config['user_data_dir'] / 'hyperopt_results' / 'hyperopt_tickerdata.pkl') self.total_epochs = config.get('epochs', 0) From 5196306407a7d0c764d961d43e19c07ccefc4161 Mon Sep 17 00:00:00 2001 From: Matthias Date: Fri, 5 Mar 2021 20:03:49 +0100 Subject: [PATCH 105/187] Remove deprecated profit return value --- freqtrade/rpc/api_server/api_schemas.py | 2 -- freqtrade/rpc/rpc.py | 2 -- tests/rpc/test_rpc.py | 8 ++++---- tests/rpc/test_rpc_apiserver.py | 2 -- 4 files changed, 4 insertions(+), 10 deletions(-) diff --git a/freqtrade/rpc/api_server/api_schemas.py b/freqtrade/rpc/api_server/api_schemas.py index 244c5540a..32a1c8597 100644 --- a/freqtrade/rpc/api_server/api_schemas.py +++ b/freqtrade/rpc/api_server/api_schemas.py @@ -62,14 +62,12 @@ class PerformanceEntry(BaseModel): class Profit(BaseModel): profit_closed_coin: float - profit_closed_percent: float profit_closed_percent_mean: float profit_closed_ratio_mean: float profit_closed_percent_sum: float profit_closed_ratio_sum: float profit_closed_fiat: float profit_all_coin: float - profit_all_percent: float profit_all_percent_mean: float profit_all_ratio_mean: float profit_all_percent_sum: float diff --git a/freqtrade/rpc/rpc.py b/freqtrade/rpc/rpc.py index 37a2dc1e5..fa830486e 100644 --- a/freqtrade/rpc/rpc.py +++ b/freqtrade/rpc/rpc.py @@ -402,14 +402,12 @@ class RPC: num = float(len(durations) or 1) return { 'profit_closed_coin': profit_closed_coin_sum, - 'profit_closed_percent': round(profit_closed_ratio_mean * 100, 2), # DEPRECATED 'profit_closed_percent_mean': round(profit_closed_ratio_mean * 100, 2), 'profit_closed_ratio_mean': profit_closed_ratio_mean, 'profit_closed_percent_sum': round(profit_closed_ratio_sum * 100, 2), 'profit_closed_ratio_sum': profit_closed_ratio_sum, 'profit_closed_fiat': profit_closed_fiat, 'profit_all_coin': profit_all_coin_sum, - 'profit_all_percent': round(profit_all_ratio_mean * 100, 2), # DEPRECATED 'profit_all_percent_mean': round(profit_all_ratio_mean * 100, 2), 'profit_all_ratio_mean': profit_all_ratio_mean, 'profit_all_percent_sum': round(profit_all_ratio_sum * 100, 2), diff --git a/tests/rpc/test_rpc.py b/tests/rpc/test_rpc.py index a22accab5..b11470711 100644 --- a/tests/rpc/test_rpc.py +++ b/tests/rpc/test_rpc.py @@ -413,10 +413,10 @@ def test_rpc_trade_statistics(default_conf, ticker, ticker_sell_up, fee, stats = rpc._rpc_trade_statistics(stake_currency, fiat_display_currency) assert prec_satoshi(stats['profit_closed_coin'], 6.217e-05) - assert prec_satoshi(stats['profit_closed_percent'], 6.2) + assert prec_satoshi(stats['profit_closed_percent_mean'], 6.2) assert prec_satoshi(stats['profit_closed_fiat'], 0.93255) assert prec_satoshi(stats['profit_all_coin'], 5.802e-05) - assert prec_satoshi(stats['profit_all_percent'], 2.89) + assert prec_satoshi(stats['profit_all_percent_mean'], 2.89) assert prec_satoshi(stats['profit_all_fiat'], 0.8703) assert stats['trade_count'] == 2 assert stats['first_trade_date'] == 'just now' @@ -482,10 +482,10 @@ def test_rpc_trade_statistics_closed(mocker, default_conf, ticker, fee, stats = rpc._rpc_trade_statistics(stake_currency, fiat_display_currency) assert prec_satoshi(stats['profit_closed_coin'], 0) - assert prec_satoshi(stats['profit_closed_percent'], 0) + assert prec_satoshi(stats['profit_closed_percent_mean'], 0) assert prec_satoshi(stats['profit_closed_fiat'], 0) assert prec_satoshi(stats['profit_all_coin'], 0) - assert prec_satoshi(stats['profit_all_percent'], 0) + assert prec_satoshi(stats['profit_all_percent_mean'], 0) assert prec_satoshi(stats['profit_all_fiat'], 0) assert stats['trade_count'] == 1 assert stats['first_trade_date'] == 'just now' diff --git a/tests/rpc/test_rpc_apiserver.py b/tests/rpc/test_rpc_apiserver.py index 56a496de2..8590e0d21 100644 --- a/tests/rpc/test_rpc_apiserver.py +++ b/tests/rpc/test_rpc_apiserver.py @@ -624,14 +624,12 @@ def test_api_profit(botclient, mocker, ticker, fee, markets, limit_buy_order, li 'latest_trade_timestamp': ANY, 'profit_all_coin': 6.217e-05, 'profit_all_fiat': 0.76748865, - 'profit_all_percent': 6.2, 'profit_all_percent_mean': 6.2, 'profit_all_ratio_mean': 0.06201058, 'profit_all_percent_sum': 6.2, 'profit_all_ratio_sum': 0.06201058, 'profit_closed_coin': 6.217e-05, 'profit_closed_fiat': 0.76748865, - 'profit_closed_percent': 6.2, 'profit_closed_ratio_mean': 0.06201058, 'profit_closed_percent_mean': 6.2, 'profit_closed_ratio_sum': 0.06201058, From 45322220107ea447b8a0d7626cc6bb1bdd388bdb Mon Sep 17 00:00:00 2001 From: Patrick Weber Date: Fri, 5 Mar 2021 13:16:49 -0600 Subject: [PATCH 106/187] Fixed line length in HyperOpt for new name Fixed line length errors and multiple f strings to facilitate strategy being added in the name --- freqtrade/optimize/hyperopt.py | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/freqtrade/optimize/hyperopt.py b/freqtrade/optimize/hyperopt.py index 66e11cf68..9001a3657 100644 --- a/freqtrade/optimize/hyperopt.py +++ b/freqtrade/optimize/hyperopt.py @@ -79,7 +79,8 @@ class Hyperopt: time_now = datetime.now().strftime("%Y-%m-%d_%H-%M-%S") strategy = str(self.config['strategy']) self.results_file = (self.config['user_data_dir'] / - 'hyperopt_results' / f'strategy_{strategy}_' f'hyperopt_results_{time_now}.pickle') + 'hyperopt_results' / + f'strategy_{strategy}_hyperopt_results_{time_now}.pickle') self.data_pickle_file = (self.config['user_data_dir'] / 'hyperopt_results' / 'hyperopt_tickerdata.pkl') self.total_epochs = config.get('epochs', 0) From a405d578da0a6eaa6e0e1e5f794672771b094ad3 Mon Sep 17 00:00:00 2001 From: Matthias Date: Fri, 5 Mar 2021 20:22:04 +0100 Subject: [PATCH 107/187] Introduce forcebuy ordertype to allow specifiying a different ordertype for forcebuy / forcesells --- config_full.json.example | 1 + docs/configuration.md | 6 ++++-- docs/stoploss.md | 4 ++++ freqtrade/constants.py | 2 ++ freqtrade/freqtradebot.py | 7 ++++++- freqtrade/rpc/rpc.py | 2 +- 6 files changed, 18 insertions(+), 4 deletions(-) diff --git a/config_full.json.example b/config_full.json.example index 9a613c0a1..8366774c4 100644 --- a/config_full.json.example +++ b/config_full.json.example @@ -50,6 +50,7 @@ "sell": "limit", "emergencysell": "market", "forcesell": "market", + "forcebuy": "market", "stoploss": "market", "stoploss_on_exchange": false, "stoploss_on_exchange_interval": 60 diff --git a/docs/configuration.md b/docs/configuration.md index 99a5fea04..83ffbbff9 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -278,7 +278,7 @@ For example, if your strategy is using a 1h timeframe, and you only want to buy ### Understand order_types -The `order_types` configuration parameter maps actions (`buy`, `sell`, `stoploss`, `emergencysell`, `forcesell`) to order-types (`market`, `limit`, ...) as well as configures stoploss to be on the exchange and defines stoploss on exchange update interval in seconds. +The `order_types` configuration parameter maps actions (`buy`, `sell`, `stoploss`, `emergencysell`, `forcesell`, `forcebuy`) to order-types (`market`, `limit`, ...) as well as configures stoploss to be on the exchange and defines stoploss on exchange update interval in seconds. This allows to buy using limit orders, sell using limit-orders, and create stoplosses using market orders. It also allows to set the @@ -290,7 +290,7 @@ the buy order is fulfilled. If this is configured, the following 4 values (`buy`, `sell`, `stoploss` and `stoploss_on_exchange`) need to be present, otherwise the bot will fail to start. -For information on (`emergencysell`,`forcesell`, `stoploss_on_exchange`,`stoploss_on_exchange_interval`,`stoploss_on_exchange_limit_ratio`) please see stop loss documentation [stop loss on exchange](stoploss.md) +For information on (`emergencysell`,`forcesell`, `forcebuy`, `stoploss_on_exchange`,`stoploss_on_exchange_interval`,`stoploss_on_exchange_limit_ratio`) please see stop loss documentation [stop loss on exchange](stoploss.md) Syntax for Strategy: @@ -299,6 +299,7 @@ order_types = { "buy": "limit", "sell": "limit", "emergencysell": "market", + "forcebuy": "market", "forcesell": "market", "stoploss": "market", "stoploss_on_exchange": False, @@ -314,6 +315,7 @@ Configuration: "buy": "limit", "sell": "limit", "emergencysell": "market", + "forcebuy": "market", "forcesell": "market", "stoploss": "market", "stoploss_on_exchange": false, diff --git a/docs/stoploss.md b/docs/stoploss.md index 4a4391655..ae191f639 100644 --- a/docs/stoploss.md +++ b/docs/stoploss.md @@ -55,6 +55,10 @@ This same logic will reapply a stoploss order on the exchange should you cancel `forcesell` is an optional value, which defaults to the same value as `sell` and is used when sending a `/forcesell` command from Telegram or from the Rest API. +### forcebuy + +`forcebuy` is an optional value, which defaults to the same value as `buy` and is used when sending a `/forcebuy` command from Telegram or from the Rest API. + ### emergencysell `emergencysell` is an optional value, which defaults to `market` and is used when creating stop loss on exchange orders fails. diff --git a/freqtrade/constants.py b/freqtrade/constants.py index c03bff0ad..06eaad4f9 100644 --- a/freqtrade/constants.py +++ b/freqtrade/constants.py @@ -179,6 +179,8 @@ CONF_SCHEMA = { 'properties': { 'buy': {'type': 'string', 'enum': ORDERTYPE_POSSIBILITIES}, 'sell': {'type': 'string', 'enum': ORDERTYPE_POSSIBILITIES}, + 'forcesell': {'type': 'string', 'enum': ORDERTYPE_POSSIBILITIES}, + 'forcebuy': {'type': 'string', 'enum': ORDERTYPE_POSSIBILITIES}, 'emergencysell': {'type': 'string', 'enum': ORDERTYPE_POSSIBILITIES}, 'stoploss': {'type': 'string', 'enum': ORDERTYPE_POSSIBILITIES}, 'stoploss_on_exchange': {'type': 'boolean'}, diff --git a/freqtrade/freqtradebot.py b/freqtrade/freqtradebot.py index 2f64f3dac..f605d61c4 100644 --- a/freqtrade/freqtradebot.py +++ b/freqtrade/freqtradebot.py @@ -520,7 +520,8 @@ class FreqtradeBot(LoggingMixin): logger.info(f"Bids to asks delta for {pair} does not satisfy condition.") return False - def execute_buy(self, pair: str, stake_amount: float, price: Optional[float] = None) -> bool: + def execute_buy(self, pair: str, stake_amount: float, price: Optional[float] = None, + forcebuy: bool = False) -> bool: """ Executes a limit buy for the given pair :param pair: pair for which we want to create a LIMIT_BUY @@ -548,6 +549,10 @@ class FreqtradeBot(LoggingMixin): amount = stake_amount / buy_limit_requested order_type = self.strategy.order_types['buy'] + if forcebuy: + # Forcebuy can define a different ordertype + order_type = self.strategy.order_types.get('forcebuy', order_type) + if not strategy_safe_wrapper(self.strategy.confirm_trade_entry, default_retval=True)( pair=pair, order_type=order_type, amount=amount, rate=buy_limit_requested, time_in_force=time_in_force): diff --git a/freqtrade/rpc/rpc.py b/freqtrade/rpc/rpc.py index fa830486e..61e22234d 100644 --- a/freqtrade/rpc/rpc.py +++ b/freqtrade/rpc/rpc.py @@ -593,7 +593,7 @@ class RPC: pair, self._freqtrade.get_free_open_trades()) # execute buy - if self._freqtrade.execute_buy(pair, stakeamount, price): + if self._freqtrade.execute_buy(pair, stakeamount, price, forcebuy=True): trade = Trade.get_trades([Trade.is_open.is_(True), Trade.pair == pair]).first() return trade else: From 03b89e7f7829ebeb24f79df45f021360af580ce7 Mon Sep 17 00:00:00 2001 From: Th0masL Date: Sat, 6 Mar 2021 00:04:12 +0200 Subject: [PATCH 108/187] Add trade_id in Telegram messages --- freqtrade/rpc/telegram.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/freqtrade/rpc/telegram.py b/freqtrade/rpc/telegram.py index 168ae0e6a..fb93caee2 100644 --- a/freqtrade/rpc/telegram.py +++ b/freqtrade/rpc/telegram.py @@ -193,7 +193,7 @@ class Telegram(RPCHandler): else: msg['stake_amount_fiat'] = 0 - message = (f"\N{LARGE BLUE CIRCLE} *{msg['exchange']}:* Buying {msg['pair']}\n" + message = (f"\N{LARGE BLUE CIRCLE} *{msg['exchange']}:* Buying {msg['pair']} (#{msg['trade_id']})\n" f"*Amount:* `{msg['amount']:.8f}`\n" f"*Open Rate:* `{msg['limit']:.8f}`\n" f"*Current Rate:* `{msg['current_rate']:.8f}`\n" @@ -216,7 +216,7 @@ class Telegram(RPCHandler): msg['emoji'] = self._get_sell_emoji(msg) - message = ("{emoji} *{exchange}:* Selling {pair}\n" + message = ("{emoji} *{exchange}:* Selling {pair} (#{trade_id})\n" "*Amount:* `{amount:.8f}`\n" "*Open Rate:* `{open_rate:.8f}`\n" "*Current Rate:* `{current_rate:.8f}`\n" From 2472f52874661c79566adbd08469ddd131df9388 Mon Sep 17 00:00:00 2001 From: Th0masL Date: Sat, 6 Mar 2021 01:07:37 +0200 Subject: [PATCH 109/187] Add trade_id to tests --- tests/rpc/test_rpc_telegram.py | 15 ++++++++++----- 1 file changed, 10 insertions(+), 5 deletions(-) diff --git a/tests/rpc/test_rpc_telegram.py b/tests/rpc/test_rpc_telegram.py index 0d86c578a..86b978f3b 100644 --- a/tests/rpc/test_rpc_telegram.py +++ b/tests/rpc/test_rpc_telegram.py @@ -1196,6 +1196,7 @@ def test_send_msg_buy_notification(default_conf, mocker, caplog) -> None: msg = { 'type': RPCMessageType.BUY_NOTIFICATION, + 'trade_id': 1, 'exchange': 'Bittrex', 'pair': 'ETH/BTC', 'limit': 1.099e-05, @@ -1212,7 +1213,7 @@ def test_send_msg_buy_notification(default_conf, mocker, caplog) -> None: telegram.send_msg(msg) assert msg_mock.call_args[0][0] \ - == '\N{LARGE BLUE CIRCLE} *Bittrex:* Buying ETH/BTC\n' \ + == '\N{LARGE BLUE CIRCLE} *Bittrex:* Buying ETH/BTC (#1)\n' \ '*Amount:* `1333.33333333`\n' \ '*Open Rate:* `0.00001099`\n' \ '*Current Rate:* `0.00001099`\n' \ @@ -1256,6 +1257,7 @@ def test_send_msg_sell_notification(default_conf, mocker) -> None: telegram._rpc._fiat_converter.convert_amount = lambda a, b, c: -24.812 telegram.send_msg({ 'type': RPCMessageType.SELL_NOTIFICATION, + 'trade_id': 1, 'exchange': 'Binance', 'pair': 'KEY/ETH', 'gain': 'loss', @@ -1273,7 +1275,7 @@ def test_send_msg_sell_notification(default_conf, mocker) -> None: 'close_date': arrow.utcnow(), }) assert msg_mock.call_args[0][0] \ - == ('\N{WARNING SIGN} *Binance:* Selling KEY/ETH\n' + == ('\N{WARNING SIGN} *Binance:* Selling KEY/ETH (#1)\n' '*Amount:* `1333.33333333`\n' '*Open Rate:* `0.00007500`\n' '*Current Rate:* `0.00003201`\n' @@ -1285,6 +1287,7 @@ def test_send_msg_sell_notification(default_conf, mocker) -> None: msg_mock.reset_mock() telegram.send_msg({ 'type': RPCMessageType.SELL_NOTIFICATION, + 'trade_id': 1, 'exchange': 'Binance', 'pair': 'KEY/ETH', 'gain': 'loss', @@ -1301,7 +1304,7 @@ def test_send_msg_sell_notification(default_conf, mocker) -> None: 'close_date': arrow.utcnow(), }) assert msg_mock.call_args[0][0] \ - == ('\N{WARNING SIGN} *Binance:* Selling KEY/ETH\n' + == ('\N{WARNING SIGN} *Binance:* Selling KEY/ETH (#1)\n' '*Amount:* `1333.33333333`\n' '*Open Rate:* `0.00007500`\n' '*Current Rate:* `0.00003201`\n' @@ -1384,6 +1387,7 @@ def test_send_msg_buy_notification_no_fiat(default_conf, mocker) -> None: telegram.send_msg({ 'type': RPCMessageType.BUY_NOTIFICATION, + 'trade_id': 1, 'exchange': 'Bittrex', 'pair': 'ETH/BTC', 'limit': 1.099e-05, @@ -1396,7 +1400,7 @@ def test_send_msg_buy_notification_no_fiat(default_conf, mocker) -> None: 'amount': 1333.3333333333335, 'open_date': arrow.utcnow().shift(hours=-1) }) - assert msg_mock.call_args[0][0] == ('\N{LARGE BLUE CIRCLE} *Bittrex:* Buying ETH/BTC\n' + assert msg_mock.call_args[0][0] == ('\N{LARGE BLUE CIRCLE} *Bittrex:* Buying ETH/BTC (#1)\n' '*Amount:* `1333.33333333`\n' '*Open Rate:* `0.00001099`\n' '*Current Rate:* `0.00001099`\n' @@ -1409,6 +1413,7 @@ def test_send_msg_sell_notification_no_fiat(default_conf, mocker) -> None: telegram.send_msg({ 'type': RPCMessageType.SELL_NOTIFICATION, + 'trade_id': 1, 'exchange': 'Binance', 'pair': 'KEY/ETH', 'gain': 'loss', @@ -1425,7 +1430,7 @@ def test_send_msg_sell_notification_no_fiat(default_conf, mocker) -> None: 'open_date': arrow.utcnow().shift(hours=-2, minutes=-35, seconds=-3), 'close_date': arrow.utcnow(), }) - assert msg_mock.call_args[0][0] == ('\N{WARNING SIGN} *Binance:* Selling KEY/ETH\n' + assert msg_mock.call_args[0][0] == ('\N{WARNING SIGN} *Binance:* Selling KEY/ETH (#1)\n' '*Amount:* `1333.33333333`\n' '*Open Rate:* `0.00007500`\n' '*Current Rate:* `0.00003201`\n' From ad0e60b5b662b20be9b2aa4cf799e4e387809aa9 Mon Sep 17 00:00:00 2001 From: Th0masL Date: Sat, 6 Mar 2021 15:07:47 +0200 Subject: [PATCH 110/187] Add trade_id to Cancel messages and reduced lines length --- freqtrade/rpc/telegram.py | 8 +++++--- tests/rpc/test_rpc_telegram.py | 13 +++++++++---- 2 files changed, 14 insertions(+), 7 deletions(-) diff --git a/freqtrade/rpc/telegram.py b/freqtrade/rpc/telegram.py index fb93caee2..037e40983 100644 --- a/freqtrade/rpc/telegram.py +++ b/freqtrade/rpc/telegram.py @@ -193,7 +193,8 @@ class Telegram(RPCHandler): else: msg['stake_amount_fiat'] = 0 - message = (f"\N{LARGE BLUE CIRCLE} *{msg['exchange']}:* Buying {msg['pair']} (#{msg['trade_id']})\n" + message = (f"\N{LARGE BLUE CIRCLE} *{msg['exchange']}:* Buying {msg['pair']}" + f" (#{msg['trade_id']})\n" f"*Amount:* `{msg['amount']:.8f}`\n" f"*Open Rate:* `{msg['limit']:.8f}`\n" f"*Current Rate:* `{msg['current_rate']:.8f}`\n" @@ -205,7 +206,8 @@ class Telegram(RPCHandler): elif msg['type'] == RPCMessageType.BUY_CANCEL_NOTIFICATION: message = ("\N{WARNING SIGN} *{exchange}:* " - "Cancelling open buy Order for {pair}. Reason: {reason}.".format(**msg)) + "Cancelling open buy Order for {pair} (#{trade_id}). " + "Reason: {reason}.".format(**msg)) elif msg['type'] == RPCMessageType.SELL_NOTIFICATION: msg['amount'] = round(msg['amount'], 8) @@ -236,7 +238,7 @@ class Telegram(RPCHandler): elif msg['type'] == RPCMessageType.SELL_CANCEL_NOTIFICATION: message = ("\N{WARNING SIGN} *{exchange}:* Cancelling Open Sell Order " - "for {pair}. Reason: {reason}").format(**msg) + "for {pair} (#{trade_id}). Reason: {reason}").format(**msg) elif msg['type'] == RPCMessageType.STATUS_NOTIFICATION: message = '*Status:* `{status}`'.format(**msg) diff --git a/tests/rpc/test_rpc_telegram.py b/tests/rpc/test_rpc_telegram.py index 86b978f3b..25b7e35cf 100644 --- a/tests/rpc/test_rpc_telegram.py +++ b/tests/rpc/test_rpc_telegram.py @@ -1241,12 +1241,14 @@ def test_send_msg_buy_cancel_notification(default_conf, mocker) -> None: telegram.send_msg({ 'type': RPCMessageType.BUY_CANCEL_NOTIFICATION, + 'trade_id': 1, 'exchange': 'Bittrex', 'pair': 'ETH/BTC', 'reason': CANCEL_REASON['TIMEOUT'] }) assert (msg_mock.call_args[0][0] == '\N{WARNING SIGN} *Bittrex:* ' - 'Cancelling open buy Order for ETH/BTC. Reason: cancelled due to timeout.') + 'Cancelling open buy Order for ETH/BTC (#1). ' + 'Reason: cancelled due to timeout.') def test_send_msg_sell_notification(default_conf, mocker) -> None: @@ -1324,23 +1326,26 @@ def test_send_msg_sell_cancel_notification(default_conf, mocker) -> None: telegram._rpc._fiat_converter.convert_amount = lambda a, b, c: -24.812 telegram.send_msg({ 'type': RPCMessageType.SELL_CANCEL_NOTIFICATION, + 'trade_id': 1, 'exchange': 'Binance', 'pair': 'KEY/ETH', 'reason': 'Cancelled on exchange' }) assert msg_mock.call_args[0][0] \ - == ('\N{WARNING SIGN} *Binance:* Cancelling Open Sell Order for KEY/ETH. ' - 'Reason: Cancelled on exchange') + == ('\N{WARNING SIGN} *Binance:* Cancelling Open Sell Order for KEY/ETH (#1).' + ' Reason: Cancelled on exchange') msg_mock.reset_mock() telegram.send_msg({ 'type': RPCMessageType.SELL_CANCEL_NOTIFICATION, + 'trade_id': 1, 'exchange': 'Binance', 'pair': 'KEY/ETH', 'reason': 'timeout' }) assert msg_mock.call_args[0][0] \ - == ('\N{WARNING SIGN} *Binance:* Cancelling Open Sell Order for KEY/ETH. Reason: timeout') + == ('\N{WARNING SIGN} *Binance:* Cancelling Open Sell Order for KEY/ETH (#1).' + ' Reason: timeout') # Reset singleton function to avoid random breaks telegram._rpc._fiat_converter.convert_amount = old_convamount From 02d7dc47802fe1ea80442d64f0410756e3415075 Mon Sep 17 00:00:00 2001 From: Matthias Date: Sat, 6 Mar 2021 19:55:02 +0100 Subject: [PATCH 111/187] Increase cache size to be large enough to hold all pairs closes #4483 --- freqtrade/plugins/pairlist/rangestabilityfilter.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/freqtrade/plugins/pairlist/rangestabilityfilter.py b/freqtrade/plugins/pairlist/rangestabilityfilter.py index db51a9c77..a1430a223 100644 --- a/freqtrade/plugins/pairlist/rangestabilityfilter.py +++ b/freqtrade/plugins/pairlist/rangestabilityfilter.py @@ -28,7 +28,7 @@ class RangeStabilityFilter(IPairList): self._min_rate_of_change = pairlistconfig.get('min_rate_of_change', 0.01) self._refresh_period = pairlistconfig.get('refresh_period', 1440) - self._pair_cache: TTLCache = TTLCache(maxsize=100, ttl=self._refresh_period) + self._pair_cache: TTLCache = TTLCache(maxsize=1000, ttl=self._refresh_period) if self._days < 1: raise OperationalException("RangeStabilityFilter requires lookback_days to be >= 1") From 0b81b58d287cad3ffbd628ec221abd44f53b41ee Mon Sep 17 00:00:00 2001 From: Matthias Date: Sun, 7 Mar 2021 11:28:54 +0100 Subject: [PATCH 112/187] Use pandas.values.tolist instead of itertuples speeds up backtesting closes #4494 --- freqtrade/optimize/backtesting.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/freqtrade/optimize/backtesting.py b/freqtrade/optimize/backtesting.py index 1b6d2e89c..bb90fedce 100644 --- a/freqtrade/optimize/backtesting.py +++ b/freqtrade/optimize/backtesting.py @@ -206,7 +206,7 @@ class Backtesting: # Convert from Pandas to list for performance reasons # (Looping Pandas is slow.) - data[pair] = [x for x in df_analyzed.itertuples(index=False, name=None)] + data[pair] = df_analyzed.values.tolist() return data def _get_close_rate(self, sell_row: Tuple, trade: LocalTrade, sell: SellCheckTuple, From 46965b1a2c9807c6f4521c1c3f9c9755f965725d Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Mon, 8 Mar 2021 05:28:11 +0000 Subject: [PATCH 113/187] Bump coveralls from 3.0.0 to 3.0.1 Bumps [coveralls](https://github.com/TheKevJames/coveralls-python) from 3.0.0 to 3.0.1. - [Release notes](https://github.com/TheKevJames/coveralls-python/releases) - [Changelog](https://github.com/TheKevJames/coveralls-python/blob/master/CHANGELOG.md) - [Commits](https://github.com/TheKevJames/coveralls-python/compare/3.0.0...3.0.1) Signed-off-by: dependabot[bot] --- requirements-dev.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/requirements-dev.txt b/requirements-dev.txt index 6ca1a4d9c..68b1dd53f 100644 --- a/requirements-dev.txt +++ b/requirements-dev.txt @@ -3,7 +3,7 @@ -r requirements-plot.txt -r requirements-hyperopt.txt -coveralls==3.0.0 +coveralls==3.0.1 flake8==3.8.4 flake8-type-annotations==0.1.0 flake8-tidy-imports==4.2.1 From 1f314f7d45e732db9a49d01ba827e3490f8d207e Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Mon, 8 Mar 2021 05:28:15 +0000 Subject: [PATCH 114/187] Bump ccxt from 1.42.47 to 1.42.66 Bumps [ccxt](https://github.com/ccxt/ccxt) from 1.42.47 to 1.42.66. - [Release notes](https://github.com/ccxt/ccxt/releases) - [Changelog](https://github.com/ccxt/ccxt/blob/master/doc/exchanges-by-country.rst) - [Commits](https://github.com/ccxt/ccxt/compare/1.42.47...1.42.66) Signed-off-by: dependabot[bot] --- requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/requirements.txt b/requirements.txt index ed5d24be1..45a378d6f 100644 --- a/requirements.txt +++ b/requirements.txt @@ -1,7 +1,7 @@ numpy==1.20.1 pandas==1.2.2 -ccxt==1.42.47 +ccxt==1.42.66 # Pin cryptography for now due to rust build errors with piwheels cryptography==3.4.6 aiohttp==3.7.4 From a2b9236082379d17157d05a9585360a52718ccdd Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Mon, 8 Mar 2021 05:28:23 +0000 Subject: [PATCH 115/187] Bump arrow from 1.0.2 to 1.0.3 Bumps [arrow](https://github.com/arrow-py/arrow) from 1.0.2 to 1.0.3. - [Release notes](https://github.com/arrow-py/arrow/releases) - [Changelog](https://github.com/arrow-py/arrow/blob/master/CHANGELOG.rst) - [Commits](https://github.com/arrow-py/arrow/compare/1.0.2...1.0.3) Signed-off-by: dependabot[bot] --- requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/requirements.txt b/requirements.txt index ed5d24be1..24c2d2560 100644 --- a/requirements.txt +++ b/requirements.txt @@ -7,7 +7,7 @@ cryptography==3.4.6 aiohttp==3.7.4 SQLAlchemy==1.3.23 python-telegram-bot==13.3 -arrow==1.0.2 +arrow==1.0.3 cachetools==4.2.1 requests==2.25.1 urllib3==1.26.3 From a9c114d30196f9d9c372bc5f77ca5fbd07970a96 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Mon, 8 Mar 2021 05:28:27 +0000 Subject: [PATCH 116/187] Bump mkdocs-material from 7.0.3 to 7.0.5 Bumps [mkdocs-material](https://github.com/squidfunk/mkdocs-material) from 7.0.3 to 7.0.5. - [Release notes](https://github.com/squidfunk/mkdocs-material/releases) - [Changelog](https://github.com/squidfunk/mkdocs-material/blob/master/docs/changelog.md) - [Commits](https://github.com/squidfunk/mkdocs-material/compare/7.0.3...7.0.5) Signed-off-by: dependabot[bot] --- docs/requirements-docs.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/requirements-docs.txt b/docs/requirements-docs.txt index 73ae3ad29..22c09ff69 100644 --- a/docs/requirements-docs.txt +++ b/docs/requirements-docs.txt @@ -1,3 +1,3 @@ -mkdocs-material==7.0.3 +mkdocs-material==7.0.5 mdx_truly_sane_lists==1.2 pymdown-extensions==8.1.1 From 7950acf6d4883092c8e68fab6e78a5cb104a2f9d Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Mon, 8 Mar 2021 08:53:20 +0000 Subject: [PATCH 117/187] Bump pandas from 1.2.2 to 1.2.3 Bumps [pandas](https://github.com/pandas-dev/pandas) from 1.2.2 to 1.2.3. - [Release notes](https://github.com/pandas-dev/pandas/releases) - [Changelog](https://github.com/pandas-dev/pandas/blob/master/RELEASE.md) - [Commits](https://github.com/pandas-dev/pandas/compare/v1.2.2...v1.2.3) Signed-off-by: dependabot[bot] --- requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/requirements.txt b/requirements.txt index e1a8d26f9..3228cc89a 100644 --- a/requirements.txt +++ b/requirements.txt @@ -1,5 +1,5 @@ numpy==1.20.1 -pandas==1.2.2 +pandas==1.2.3 ccxt==1.42.66 # Pin cryptography for now due to rust build errors with piwheels From 25c9e89956931c660d3656148f728ae66099cb40 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Mon, 8 Mar 2021 09:15:30 +0000 Subject: [PATCH 118/187] Bump aiohttp from 3.7.4 to 3.7.4.post0 Bumps [aiohttp](https://github.com/aio-libs/aiohttp) from 3.7.4 to 3.7.4.post0. - [Release notes](https://github.com/aio-libs/aiohttp/releases) - [Changelog](https://github.com/aio-libs/aiohttp/blob/master/CHANGES.rst) - [Commits](https://github.com/aio-libs/aiohttp/compare/v3.7.4...v3.7.4.post0) Signed-off-by: dependabot[bot] --- requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/requirements.txt b/requirements.txt index 3228cc89a..f62c8ff52 100644 --- a/requirements.txt +++ b/requirements.txt @@ -4,7 +4,7 @@ pandas==1.2.3 ccxt==1.42.66 # Pin cryptography for now due to rust build errors with piwheels cryptography==3.4.6 -aiohttp==3.7.4 +aiohttp==3.7.4.post0 SQLAlchemy==1.3.23 python-telegram-bot==13.3 arrow==1.0.3 From 4b550dab17d8121dd79545f6d809f38e67f0a5f3 Mon Sep 17 00:00:00 2001 From: Matthias Date: Mon, 8 Mar 2021 19:40:29 +0100 Subject: [PATCH 119/187] Always reset fake-databases Otherwise results may stick around for the next strategy --- freqtrade/optimize/backtesting.py | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/freqtrade/optimize/backtesting.py b/freqtrade/optimize/backtesting.py index bb90fedce..aa289dc2b 100644 --- a/freqtrade/optimize/backtesting.py +++ b/freqtrade/optimize/backtesting.py @@ -174,10 +174,8 @@ class Backtesting: PairLocks.use_db = False PairLocks.timeframe = self.config['timeframe'] Trade.use_db = False - if enable_protections: - # Reset persisted data - used for protections only - PairLocks.reset_locks() - Trade.reset_trades() + PairLocks.reset_locks() + Trade.reset_trades() def _get_ohlcv_as_lists(self, processed: Dict[str, DataFrame]) -> Dict[str, Tuple]: """ From 37e60061692e43e9094638de77328a65b2ffc541 Mon Sep 17 00:00:00 2001 From: Th0masL Date: Mon, 8 Mar 2021 23:21:56 +0200 Subject: [PATCH 120/187] Fix order_by in trades command --- freqtrade/rpc/rpc.py | 5 +++-- freqtrade/rpc/telegram.py | 8 ++++---- tests/rpc/test_rpc_telegram.py | 4 +++- 3 files changed, 10 insertions(+), 7 deletions(-) diff --git a/freqtrade/rpc/rpc.py b/freqtrade/rpc/rpc.py index 61e22234d..62f1c2592 100644 --- a/freqtrade/rpc/rpc.py +++ b/freqtrade/rpc/rpc.py @@ -289,9 +289,10 @@ class RPC: """ Returns the X last trades """ if limit > 0: trades = Trade.get_trades([Trade.is_open.is_(False)]).order_by( - Trade.id.desc()).limit(limit) + Trade.close_date.desc()).limit(limit) else: - trades = Trade.get_trades([Trade.is_open.is_(False)]).order_by(Trade.id.desc()).all() + trades = Trade.get_trades([Trade.is_open.is_(False)]).order_by( + Trade.close_date.desc()).all() output = [trade.to_json() for trade in trades] diff --git a/freqtrade/rpc/telegram.py b/freqtrade/rpc/telegram.py index 037e40983..759d40197 100644 --- a/freqtrade/rpc/telegram.py +++ b/freqtrade/rpc/telegram.py @@ -637,13 +637,13 @@ class Telegram(RPCHandler): nrecent ) trades_tab = tabulate( - [[arrow.get(trade['open_date']).humanize(), - trade['pair'], + [[arrow.get(trade['close_date']).humanize(), + trade['pair'] + " (#" + str(trade['trade_id']) + ")", f"{(100 * trade['close_profit']):.2f}% ({trade['close_profit_abs']})"] for trade in trades['trades']], headers=[ - 'Open Date', - 'Pair', + 'Close Date', + 'Pair (ID)', f'Profit ({stake_cur})', ], tablefmt='simple') diff --git a/tests/rpc/test_rpc_telegram.py b/tests/rpc/test_rpc_telegram.py index 25b7e35cf..924490821 100644 --- a/tests/rpc/test_rpc_telegram.py +++ b/tests/rpc/test_rpc_telegram.py @@ -1128,8 +1128,10 @@ def test_telegram_trades(mocker, update, default_conf, fee): msg_mock.call_count == 1 assert "2 recent trades:" in msg_mock.call_args_list[0][0][0] assert "Profit (" in msg_mock.call_args_list[0][0][0] - assert "Open Date" in msg_mock.call_args_list[0][0][0] + assert "Close Date" in msg_mock.call_args_list[0][0][0] assert "
" in msg_mock.call_args_list[0][0][0]
+    assert bool(re.search("just now[ ]*XRP\\/BTC \\(#3\\)  1.00% \\(None\\)",
+                msg_mock.call_args_list[0][0][0]))
 
 
 def test_telegram_delete_trade(mocker, update, default_conf, fee):

From a1902f226d682ae795ef31cf154eebec62042039 Mon Sep 17 00:00:00 2001
From: Matthias 
Date: Tue, 9 Mar 2021 19:29:00 +0100
Subject: [PATCH 121/187] Make trade-close sequence clear for mock trades

---
 tests/conftest_trades.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/tests/conftest_trades.py b/tests/conftest_trades.py
index fa9910b8d..df9929db7 100644
--- a/tests/conftest_trades.py
+++ b/tests/conftest_trades.py
@@ -88,7 +88,7 @@ def mock_trade_2(fee):
         timeframe=5,
         sell_reason='sell_signal',
         open_date=datetime.now(tz=timezone.utc) - timedelta(minutes=20),
-        close_date=datetime.now(tz=timezone.utc),
+        close_date=datetime.now(tz=timezone.utc) - timedelta(minutes=2),
     )
     o = Order.parse_from_ccxt_object(mock_order_2(), 'ETC/BTC', 'buy')
     trade.orders.append(o)

From 99583bbd0ca9525aca9968740b4eda5f9e3da9a8 Mon Sep 17 00:00:00 2001
From: Matthias 
Date: Tue, 9 Mar 2021 20:21:08 +0100
Subject: [PATCH 122/187] Fix problem with FTX

 where cancelled orders are "cancelled", not "canceled"
---
 freqtrade/exchange/exchange.py  | 3 ++-
 freqtrade/freqtradebot.py       | 4 ++--
 freqtrade/persistence/models.py | 2 +-
 3 files changed, 5 insertions(+), 4 deletions(-)

diff --git a/freqtrade/exchange/exchange.py b/freqtrade/exchange/exchange.py
index 617cd6c26..e457b78de 100644
--- a/freqtrade/exchange/exchange.py
+++ b/freqtrade/exchange/exchange.py
@@ -1053,7 +1053,8 @@ class Exchange:
         :param order: Order dict as returned from fetch_order()
         :return: True if order has been cancelled without being filled, False otherwise.
         """
-        return order.get('status') in ('closed', 'canceled') and order.get('filled') == 0.0
+        return (order.get('status') in ('closed', 'canceled', 'cancelled')
+                and order.get('filled') == 0.0)
 
     @retrier
     def cancel_order(self, order_id: str, pair: str) -> Dict:
diff --git a/freqtrade/freqtradebot.py b/freqtrade/freqtradebot.py
index f605d61c4..27c8bd48a 100644
--- a/freqtrade/freqtradebot.py
+++ b/freqtrade/freqtradebot.py
@@ -1023,13 +1023,13 @@ class FreqtradeBot(LoggingMixin):
         was_trade_fully_canceled = False
 
         # Cancelled orders may have the status of 'canceled' or 'closed'
-        if order['status'] not in ('canceled', 'closed'):
+        if order['status'] not in ('cancelled', 'canceled', 'closed'):
             corder = self.exchange.cancel_order_with_result(trade.open_order_id, trade.pair,
                                                             trade.amount)
             # Avoid race condition where the order could not be cancelled coz its already filled.
             # Simply bailing here is the only safe way - as this order will then be
             # handled in the next iteration.
-            if corder.get('status') not in ('canceled', 'closed'):
+            if corder.get('status') not in ('cancelled', 'canceled', 'closed'):
                 logger.warning(f"Order {trade.open_order_id} for {trade.pair} not cancelled.")
                 return False
         else:
diff --git a/freqtrade/persistence/models.py b/freqtrade/persistence/models.py
index 3c9a10fb7..a4702fdf5 100644
--- a/freqtrade/persistence/models.py
+++ b/freqtrade/persistence/models.py
@@ -425,7 +425,7 @@ class Trade(_DECL_BASE):
             self.close_rate_requested = self.stop_loss
             if self.is_open:
                 logger.info(f'{order_type.upper()} is hit for {self}.')
-            self.close(order['average'])
+            self.close(safe_value_fallback(order, 'average', 'price'))
         else:
             raise ValueError(f'Unknown order type: {order_type}')
         cleanup_db()

From 60f6b998d3cc27599b3e95fa2da63f573d3727b8 Mon Sep 17 00:00:00 2001
From: Matthias 
Date: Wed, 10 Mar 2021 09:27:03 +0100
Subject: [PATCH 123/187] Update logo with smiling one

---
 docs/images/logo.png | Bin 12030 -> 11025 bytes
 1 file changed, 0 insertions(+), 0 deletions(-)

diff --git a/docs/images/logo.png b/docs/images/logo.png
index c7138e84b7e6f06a523e1d1a27b21691c6793288..8a7ffdd7091e40e0713c71967fa2b4bc32cad64c 100644
GIT binary patch
literal 11025
zcmZWvbySpn6Mg9J25A(eQ9wkx8>AcQ4rM8+r8}gN?(PO9MY_9}?gnZ2p7-x>&)GP8
zcF%8jo|!v$?wv3dC21^la&!Oyuw-Quixw)}hJJ>p#8atV@JASiFI}soU02)A6LR8&7{W!zJ
zi%|0ZMOZL?V3;9r8Ks|WIG0ug7Y`1BX|MFKmi1)a52d#92*)6C?K`x5gYNLamPnff
zXZiYz0WP6r^tMoU!8$5!ph$OOY#;+eyy=_x)PZ4$_sgC_dMd=_X86kNUH)2?)^GP8
zl_jMo-&`u%V#6&1Xl;h!rCd-vfW4EuU
zOTi|PQt8kvHfF*U7&J~yM&_YfHJ1f^AsX;>OKvjEkv>Oe>Q0W0RdHJH4zie3($LUw
z*4NjkWfO)fi+@8w7YsIlk^Iulo3NraRM*nd1L{?LoEaG0aHWdHR-l}WL8mc2!C0RN
z2ng~npFMSTf3gb-K4BKH{w`;0L;f7d!`wBubO3vL;_p#Y@CAvneX?`n~y;MSQOQTgJe|rmkXT$&EPunK`l4Yq|NW$HKybG-)sw>NXOmAmXvsp7D_gEDgRW$7`Fg
z6^e(eb&K(u5&G6()h38)=q2E#`(6rlv248OjsPHTM>|-{E-oh&`=bt!(6Z+zst~KA
zuK;XNHxZKQWH^bCau;-T!$)|Cu_C5%TQ|9+K`t&YQ+MhAKV*t-(UXhVd6Lt`I%4t}
z*kJ&s4;|?EDz3_2j6|JgKo8t9Dhazq(((5#Xvk6VNFDJhj1|3A@u}fDYtpf-VvzuCnqBzz2If^1Xx0BYJRu|0Vkq#L9?F0}wE=REjG!AC|$%MZ=Xh$iq$
zsIZ+%?*#O&Exb^S(yu4WEiLx;_R%V6B_EKGkdpIqbCtsC(*RCrgnc&djso15fHTI9
z{Nmzb=IQQ%kQE2FRo|B@%G$2aI!
zXeLb&kBp4WvgtKWJI+_bhA)_X#zKYHCdRdmpARvNUG
zqTo<@;V0|`;^EW<6kD+o4`rEMpzwnd@18l^Kf
zKVw~gQmnen8H60Uwr%YPF4(!Qem|`6z#n4d&p5fr5f6#1zbGv&jRX#m2HoGx_?maCyn%#GGpNxZHW8
z+rshdtlP(LEm`!VGl}XIEImgM4n_nF0p*C**qvx9{3pF6MgkItXTN^^LcX76zW{OZh!<8xn7pggc)+GGT6Oqey`27S-3g$n-}
zct$@_$*7_=ir|ER>Fo;=W
z1scxet^r~ZGH-KpCg|l^U0#uLmJ+3gR+S-}OEhx|coCJ)631#;C-xx)-==H6ZvM4t
z=Ldsp4~uqx&~s&n){A_S^;>#Z;rBwh&ieLkiyXnMaZm8)-5;1e2~XEmzAJ7!!>RI~
zHz#c5+*acbIjy5v!fX3#8Oh0v&6|+Rkq^|%jrL31-QC@T!Y_AY51o_~J0qXIU+tOJ
zKag^n{8L=wH2Id*z`A
z>YYGhg2ib19YRIG(&A!qn+T53Elj9=t-i65Ty#P3`_>QX@ZXopsqeP=3ZgY5db@4s
zz1O^NCZx`Ii{g#W$7g3PkOZa5Ira@(_?5A{2j*sm=;qLRI9-?%!@m<-g8}fio>O=e
zAK-eNq~{Q}{m^Dub&3RIUTY{UEL3vvV{mX?ao76*B4IzTGIWMHn*bAF7NxRbs`x60sJXJr`>LA-hFx>080f0Oc!ci
zM#et5HcfT)1R2yYM_XH4gI35s_V|;%V(_=(Y0G=Ri;|xo99Oxj#&*csy)b*ya~0wA
zi`7f{=$(P%Y~xlRv1ssBX1(3$;_L+a1@ldavsr)ryo>HZorJCi1qk$8>B1
zPt3v2&S>qjy}eDgd^!4)CxTayc27!bdHr3j>cH^u6)B#co}P!Wy>7)UlcuJtt1Hj8
zs=ky|@KZsu;h-Q0Zb8?Fb6E&;y!MOreyhG|Lf-r{jx!({oG8l6FEu%=
zw8nw`T5~kx(+9WV6t2(N<%5h0DNeF`hLclMCG@#7HrFW^3h8{GnMOs^RRqy9a-z$#
zeC|s^l$?*|Yl`kJdV71f!=uXenj8uw%m2LvQHwHX=0`#T;a(vEoXEe^HDAd+!$!M>
z?uS=Eto>4>$DS2fH{P<@l@!|te?-t8erOb2{6bpnFP#dbpj`33Ss9o!a9bGw(j;cf
zb*8MsPrp(~`<$yR`HL+G7qXnlvG^KUAAaIuG{gVIf9%1
zAI93cx)>j@J_-I@9M9L7PJ;(xc(G*}rvxLQU>BockxNW5L$YUVO1f=ow6$+uA|lII
z{)z}61rep=K(Xz3DNY4T)R_mEQ$C$E)3XT+ua(RlB9U-8XlHrTugN{UYBG7TR3Si^LFer_ozCwH~k7ZZ~lXH+($DlPA5eErPtF=4N+Wz13T+P1RU
z)zr{%E~IZ1NCZelkrT}=FAG?mZ}jfw=H{}WZ5xvo?wqup3hChJ64V
z!|?s}p`*~(jllE5G>3b*$5y4d*jOiv-f$w4!^14mG`!<
zd~zhEG-am}Drt_2;IKn_9^(IcDnCb-uVeObPJ0yhtXm@+bCMRj;;95`2vr*XKH!?8
z4@CUc*-DK26WKkcJTWox@m>g9QhR&jxM?jK!k2O_<{y6x*HuZ0!R^I
zuYt7y=ME~!8wFgR;<%{u%|4VXPwGOr3g90!qEnD`5bubc7B9Vt*pQL*!3!vooL{?xvVsi~=mz2MgkiBu8Fh|b?fzx}64L}~Smn}?Nc
z*i%KyIQ68TsXULk{`4ZbUdft`+5ko3*LrRE=#{UHT_%V@Le8oKU!ghuZ^6;T#N_Ot
ztO{8_x`}780ZLac^bWG8^6PFVO_b&3>1IuOtzF&dRUBSwU2p--4XJUWvl^^rH?v1g
zS@5;iS4vjxYT&))XhH!io;}ncNzPv!9=ZmQ5+4LZkba$4ZfSpM*@W;9ydM*wAMy`3
zCcRj+tr}Pmh)YXj*-Ae?uiQfwzD4r6O5c+pX>b;R&UJZcKVv}kT5~5|U2`!4XL@~~
zFR9}a61JMM{~a$Sd)=(K)i&9VWeSRQ-+$3^36}VdaErs=R7aPRSQvoO(|-ekr2jZG*Ns$FNHm@4w5)|O43
z{|>WL=d9`S8z)Oef%<`*E#Bi)T-ogb2&ReHA>!P8k~H=8^|4j$_u<2tf@#Z;DMhO8
zwAfgYzma6(V2>8(by${RCAN9Q(s93@(HcU$_nV7+hxmvNnM>CM-ur*Heg%}A^!XA;
zA~$noY2I9H4+ZeHPNHh?&8MWKhz&Yd1E`rNiu|S`F`>?QqL`a4k*aTv7Rk${>fc9e
zHpvz6+w!Ibe9Cu0Na8T^5rN$~;#^!Nqq9jZNC9zP^tV0NFkkU@L{7Nnn$SwGIxz)M*mi8Fk&(RZZ3hQrXw%Ndru60Ua_kb!&wNCbb4CUWYNsT7
zukDYW8L+b%>RH^(TFtO0rg2)g_xv0wLI|8g$1g>lL4}adA(iKaUMNt5Sl1YP!0KfD
z)0f4svU1s2S0ssY1tu*|)2IL!zHcLuzR}&J16#AzVvpp-6HX^Wtb=Je3dvNbH@`OA
zs00_2bVpy_<&rTH=QvwT>yzV&2R33J7op*T?(iE@4uvzVoAMhJwu#F;$h|LG%x{nQO^(sL~ZD
zTP6LGqZmcj0(Z|0Zl(-p=zBP~x=MYx^Cq*OHcaRNN7Vg_sycTvtaeGWXg6_vz@
zlV_+-C>Osl3VPKuhU0HmM969AI|6+D+((`sZiLT9Kb>xAX=w1sUCutEUVZoOI;`sO`D4Z1ILMl<4mQz6Znu+Vappw1Ql2!=d(4XkBeTl{uz}GO
zWY-~gB{urCJDGI8BWf#@sJr@cXIlkl|rb
zhjOxZa&m*u$4gCFhs0p>cp!B`YZeSiMEI?p*qTqZ8j=Oy(j;xNqFQbZDoaA+pF
zO5U>+xKlh^9EPbq&ibpZuAw-TB{;fs5EPT{Jr>r7ZQ;Dpflaf=@wd92u6lrx?;w?<
z-S&I>x8$fSEh&kHg!E5|=xq7mUvkz(f&s>Tks#KUpE1VXw@x>oQPYEYiMQ&kXc?Pf
z5|#wjYVdafve<5+<21NJbEEi+=AQ>)cO4@cWla%)haO
za<$2v);SRRA>RF>vHKXK%hN)Z!5*)r<%dAWT};2D{)byPCyW+iC-jXNn6)DeLm0uW
zF83xOGt(!#u`$U)W9Ar?Ksd2p@tqYXc-s0Vx!v7i8k|}C%h7nHcW~U2Y$K-2(qwgB
zEMT&_7$^W|IH*gatwrK6Esd5-rm@rKj($fAsIQ?yr{Z48`sCYXIHldUuEGs0n&o_i
zHtUzkO&~w<^xNGL6jS_nPS#2Q=Y#ajA?f2rxhI{;3tb|0+O>dadNjiCZ8``QA$^ec
z#_bf`D<5v~uQTu>SY|W~b?fvfaJEF}>oE{IfxEHBRkisLpqH8yHjU+!^_vp+Gpn?O
zL^|`^w_%FP$~f+QMHLkzK0Cu?YCb+b^0KmF4Xv#|c}OwwqN1Wek`@{hV+rzt1tIVI
z;$R%=N=hLwcLhyN%}A*{w0S3bP@y^j2gaI@8A^^%MIS1YpHs3l1fxd(%6;k_u@2fp
zagdG}(|*tcI)aKeqoM$xlf;th&WCR1^F`IUT@ZH@LT;kyB4Y}w+n<|myMWRZR{DFnWt
z-=C;2w%YXajr8?V?q9i53Sc=7{;HglDeqytzvK>dZQs!n@X&F}tblV++Voy|uy^Z%
zuAOz$8~8q5FFk(bOi4@A`}Cy~nztvAg)U_?G&(Azsjq)@$({<*_Nl2U-&mDC9hvt#
z#$DdV25R*o9v1>i`vgMwP@!xujL}aRo%>JTWW|pgy8=9XHL&VrPW
z!u0n?*;rU?Tb}O^3o2W;mHvVJk-DO!q~>nY0vz^!owOdp*LuQaJ>izo
z9YznlN3<|r)W_XgnY%`GejDXS2+h)l1qoom^tjkJ2DDHd@!@enn|1DJolg`$Pepa`3l;3~nwt
zBbQxve}7+^f}ETorWQG%t6a5&U7l$PS*h!E5_=?&+l3U)l4_45EStiqjS<6t>!RPY
zdo^&BQQ=i$U2=@2w)ZzF(G>5q%
z3L!?@6Gv_Iz!}3M#w@RI5;lDf^VRG4d=)lu9^d
zy4(^c&Y*f}4I?2ab#zcHtB=Dwh+b3rTRn1PW;h;TG#%=mosvwwdiAR_-tE25Rrn*`
z*mZP5YHEw3O-Q9~!vWo1ba$(1?bA)iOQIRx-&)qRpFicC)z{_3zUrdDr>taJa`C-x
zx7*DZL&{R)z6l8Q+PzhTT{(K0`BgIGW9AQWDnx-gjqLugW5<(bW!@WXr!vh?Yc7d$
z8falsJoxedX{&Kk^A{HvQmEAHT>ccYo|t*#ceKs^GuQ|({S7Ml3hhW~R-kySf#Iy16B%VU5I|dT6vesy}
z4Z$xN*I)&7gkAOWNIF7*Uj{f>N3=gG&O1#ieZSMW_;-YSvZbcIP
z^haaTzbhz2WWqj=*f)oPvDhSY{~cb=uM#Dt=Dz`q7U76QoH)fkdEKm%Z7#chpfvGFMCgyI!F(=v)ReYdSoLI(T^
zA5z0|e}8|Eg#Yu}cUyVYMr4v%jqq`Ma5t`)-ur4oI@fuow)Wk7GtJK)7>D3|wYJ?o
z$f=-aZ(mXDH~M&Y*~=@?%AQ+`(|>|kma*HZGxrPqL?AlC8CHmxhQRayw}=#MJo<9t
z)C@`MRbL7#Kgft!c#)Q53v#rYxD~9e4xt;vFyV?0+V~3ib0@2dWsc#Dkv!_ZG#QTIy$&0n*!ag85aYN
zO!fHT!QdDY2!$
zxwtg+UcZ2<)jj@GKEdH^1;OPh^mCpRn!Pa&R=!UQ8l!Q+$+~i;K*r{xV)YEF)be|oyoveqjj)9epl4T%kDz^?T3+hzeO!VfKKcnBS<*NXPSam9n~~CU!)4_<=hCs4zqRn!Q6R=WqRf
zUHXL@jI&ht)Z3q*Y;K7+-W3N9VJF3kq|k2oPk|pgF8Bo=_-kr^D}w|qND3R5-NN^n
zo8M0&4Xa|n;-+Hc!$Jp$OHF5@F--yOuM)Qv)VDDz0cI@f>
z2mt7D|Lp=Ou}DM)q0yG>W1VLrW0G^rFrkNyX;)~*rKWlWo9pO6(%-~eW!rEWVFdLZNJupQ0$HY|0%gBs3$4TW)aAI!C+obWk9KeB`1Hz)#=KbKBOdSAK32Zpz
z3aAs`rVq!ZYhL*o7dM$vv6=@CDH-ueNv<_HIXR{pLM{i>k)Z4GD1xD^MC>Ye4BXl0OoL#O|gECbf{h
zladK1X<(*-AYa|EgJ?4<3!tB4z~}j|92WQC{rmS{w!dqssHkubT5-F2iJpK6AmSD~
z4<_!U=3OO!P^N)B^m<5+5AD!nF$ff^rE;i2ujYU(xFo5BvP(d%ik*Ywv^s$t#6S*2
zPDhOrb@Tt+(9)vQju%5KL8q>@)&NmHG?wcw`-NIbh#{Ey8m}Fl{lyg&6#6XNT3Qx@
z0mJAT)4p(r6%dKaWMSV*I$Tf!obWXd0zP@&R_|Fp=Y&ul@Gb>&)6vn{pkR`YbKw({
zkf78%Up=h77*4xC^#7n2JMiBSIXE3S3z|RT<1nkyMqH^jYWddd->DRd4{U+^9Q->j
z+?+mQLQY|7_y@Y_qPq|*EUXG#q2ohi?K~PvP@rHr4KV2N@vc$n
z!xfVOwVTe62N0y5ym13(gdA4|Vv@5P8fN}-My^p|gVY$~Pi9}KMp>Z00e4=Sw?~gT
z3Ih5oBYuGt$5yqL@B5f
zKp^!`!4wWE!cSLI-j2V{rEoCH`al8HQ>>rCcd5l}0RQt`;>+W%aL}SRup(nm6#a;n
z@B?a`5&?GWASrw>&&){wuEky>0;%9ErBs8*qSAi55h14X%Titz)c&n(wn!
zMjXVa%uG36-^tZGO6jGe39;Ni8yPuTl1kY`GPdJ+q0s&*3iVax^Lcb%h@};Gy<1|}
zQY}$WB#w|g!w^TM1v6F51qpwN*(z`rR`c;Zx`aSo`C$rap{NMnCg#8b`*d%Qa-0%PRfmDVPFWBZ*{zwy!wfaq1qOB$G04X;1!NseIXb>1sb1OG;~pixj9go0RW&s=9KR?c
zAZX9Hi=#EX1$VC#jo^(P(XpVIjJUXXno?1`DfT{iMfMHFPzT1wszA0Y9`X^i8&_C|
zbgK-1>*meaY&*rv0Q~T&)1Pao4V?CIYMW2Li^!ElXkZt>yHLTLWzB(lStwQHFf5hyPrRQzBy-q
z{sDS*_sz|GI_i3U`QSbR0SUW4=iTJLjg1XCfQQbU+7Ew=jzIw$s$&2y<9rqu4=8?~
zp)8t`O=36D&DSg&3L3y&KT*uGs;;h13L;6W%-tKNBn{eaq)=3-VU_Ug
z{Jhma4Zw~!?UKXyPX4o*+1AGzet1Q2!($j6iy}b3-wV4<$I#G_1P|}4rkJ2IVg(5y
zLlI-gnRAU;ieW}k;O`lR>em-G$Nc9A_Pu+c0QzCGx|-W!1c~2(-psw{@ZB!|F5!SKVUR6+de0^v?eFtt>btkf)}nv!->u@LsQ~WbZy84
z+|Sm{hmwfE$noHd!B56Cntt;9ceMRlK56Uvc`t=P7-L3agK7c4
zB7w+{q`@UroL@w)|AY~6fi>p+{rh))aBuAtgn-aLeC(p68u5${bSepSgKQI60XR>w9EMMi)UOK)XH=$D1*0E;
ze%IJ4j7eg^)c=Bkup6ETN{0K9nP8)y2x_;~gFc7+5okwz%D2%f{0`l~j_b6#L@;KZuHj5C8xG

literal 12030
zcmY*<1ymeCvn~X8cMb0DED$8PyE|-fw_w5DgIfsB;_mM5?h=B#+ui*4zWd(Wb7rTf
zr>CS(bywF{6``ysg@Q+eVU{C&!
zu8FCGo2w87#Xm;>`}=P>U9BwtFDHAK|4|F9AnU(3tn4gotp7_J{3+mHuYi)Xl?9md
zKl;M#fd7pA|N8!i4}kR_^Z!Sf|JL+Byb#+BF_CRMw|y{SrYTB?)v92M(rz(qGJe8i&K+Ai1lUMP&5D
z2I7?0P|>7lv;8u~PAIy*|9IPI?^xmBY-{RxId2opo$~hZ7)?*-1Og|s_z$BG5Aw?J
zJ#S~En$~QpI?f?W9G+|l_#C%8Mzt0qU5xo*WqZ0yi*p3Z)QT02lm{hC1|ADjbggHs
zinH%Qs@sQNF39T9kXsLeB)+#Zsh>)uqjhrO&`1HDyRiEZ&dBBi-535JFMA_;9C?i?
zR!?UR{(@XK240=SP!?v}TOaQa*4cs{@5B%Xm>|}a?QD-to#InTE}0${h1sbLx4ObY
zi^n7-;TAn=90)V|QU9lTH6sOu?w!cfKGsLe
zVb0CNrniHcaPq}!!wCTR)yp(CK$eZoY%rRc&Vj1i_hBQ|SBuN%{ps-CzU|C%oR!c2
zWo>J*sz8XO!}c(Cvq$&-;-B1uJdfcum@;y#9rYgj2~sq>LE--IH@R;?hp)U50a@Bf
zd?UnARaOU2ONQQdlv&AIhAa40!!Ss=u5~nj5DXQ|anO5?K~?Vo6^~G80Au;@XAMiy
zSCRTI6Wl-@teaqBsAS{q{5WA<$eKAZW2(n#?nUiB|CR!Z%c2O@5
zRy`T->so%;EPTGdzk=+kgAyQ8_3Zimm#aVTv{)|Jn?M5>CJuY>E
z#U@%Uc;391pG2#)S!$hp@WUBdJrXij^AL(sw0wxfWBnkx*M9N(2|=?i)(7fdkaiK4
zxZcvvS+RUV;2LH5B1@adaXO=6gujO8WSG~u%i<*|43X`JZ3B|*a>q=yuE%D(7vn^(
zi2oC7*Q;}Bo_H9cn?Yfn49#d{g3QUI7A!}GmODvne@d00F);MnBL
zV)@=+jNne@*6W{*O#9X&BpwVLZkm40%H}G+JCIA`=J%7Mt+#Uz+D*TgvzHqT*9t?B
zi?l)zWVvF~`vI#zKk?Z|(_dE^32o8$6y~P&>xd|p*1y62Zzo1J+$gm-u_9&GPSdeH|*!wD&e`EVMjvh{cf6+oSswgGsp{!VA$?fY_n_k4S_
zmp66nd%$RP#YQ2&A1B46-#ncu=uye*@<$%+5O?$CqU&O(Zb6)tCuiXaRYdUPdW==%
zPf31kBC|HB00S)fyoEb>s^N}|iVlc->G0o8SPv-G3O(6)>*1B9%Cx>r+GnUu=jGg~xhNE1yR05HyJv-B}+^D)d}mM6w+_f7OR7
zcq)M91nabppuA0R^7W_H9fjE)@0QPvdSJ)XpG>sB(tC@pzB_-fwFNdX(Jf!`4lq9!
z^&AxPQ(KGs9A4HRcd4|0n*aF|28SZMTh;ac9W7Px?jY4@Gnd0cb_0Rk1}cR~XZNCM
z%TF;dWXc!9^T@d#mShff&6Y=fZkN;QGM2jRV_F=|qA{M*M~GHq*R66oqNlD0_~dU|~!OTQ2kK$9gT#jP*l={Lxc!p)@K
zO$nSZeLNlJdi=aU*xaCxI&fv|vH@X)FStqJJeR%$bA2tTgV97@r=J{
zL?6lgUu$qu=y!t9gfFkgRbL^X$8KqRlgH#^feOZmPWh5C94R<}$E&!T3EZ@AQvoWc
z(KnlDR{2ikWPXn(C);~LvO%^?z@V^*$hehA9811{lW-qwplsk)hqs4Ov2M{1`WO-8
zZU{zzV$)#5VU2ZJam?FZ8}%xJ0*zBVXh
zunGO4SmbR4(W5^cW6*h>>KijwYTRzqLdJtf*Nef(Wep2kN(=a8QgLTKK*IUbksPhY
zJj#PDK2hE}C-Qz3H^C^{FZ^<9#vHpomtn~mrrx6qi_<4fh=(e**~eN9x3jd%35Pat
z_y`3XdsWigHg3n)!ck-q<8#>q_h4bk%gewR&v0l%M-68i6T14`HNmsyDMjX6AC~JE
zM^qvtFCMLf!A!7+6S0`gt-zyN)Ghjhz7;(z)SqesCKtp;=@o>howCh-1!^-^5Di|f
zG7>?%Qm4>)-0@u5`_;2ovItcSLsTsxIdyW|kEL`f_L>x04`4ung6{yv)7JD?G;MzO
z^OR}7>_x)z3UpSjs)I48Bi)bs_zh>nMAJx-!|Un47sE?op?`@c{J1T>x&2jk8?ybe
zibt*Svckc4Oj}V_aL0Lf&JJUYhq*bO8i5)aw#W4dEgmYix}xl&
zz;VNYhM<9HD{Nz}wKt#RQ~*2cgNqPsv^y4IB(T8*<>(95$l%yfk+u6wLi-LLbBy?Z)E{ZZ7PFwnC+2+NBizc
zVA<)ZzMoyV__9#thZBKhs4VT|$2NMntCq~N1^w7Lo6kKQ7QnpxXr}jmkGb_ZBAI$J
z+(^%dkMp%EEQ^C!|9B(g2THLiKV%_53mm1i66%D<%<8kJDMQQ
zI{X1uraak{ltlHS8)~fW;o)WLqw^st_qBfodb~|Yu`J8WHsbPdGtTdpQqP%P*v^g4
z()L$amNrTdb;2C@40=$bB6pb`dKyA^)oW`FaK#1GKvt3);`Ver?2!Q#39!+
z`J8?uxrQl@MJ%wl?+wL0ykDnwd7h?_v;Z8y`@}6@^=1EsB?`s2ayubxB>=Wf)&O*F
zz5&_<9k4St=i3&c$88Sqlx!$A=Y1c5;;VCURGfQXpDBs_=kw~7veIoc7CB|TInRDx
z&O0=jh!&|-eU9QRRm-#}T+SyX#jQl@Oja7&#b~qGx8&lk3m>V*umv>8*9+eZ?*+I`
zQSv{wU*HIaE&ob}k!!ZUaQA61l*c`ivGs;l*=uoBhEiz81Lu#MKi*%2Vnt)%rzkh@
z(Kq{PGe1eSe(($|F;?bzYCg8&S&paG_!8@^UjDk=$+O+fyVbv>A(=|*`apxymi5$L
z`i@WfJ&8u)+LqQ?@S^=j&Hpu@1vDZ)g6|35M+(-h)LVx!Swmth5m>%nGc{98Vg4Lw
z+HTqi4$aU}!SIn02BjHzRKUTy6|q?0MpQRTS2)ss;a>Y3g1MnWPK})%EaR$C^<(I4
zE#YI7ffDV&6hg3MLb)z!&}D?vwuO^XLYED5{FtR9t1*4__F`*mi3I8(Sk_V#+fO+3
zt9Jf6`{-B*+a-7bnTYV{Q9mJ|v;Pz?AOD53_5?b2{z$X)rB6yF9HrJ|LJ-aXJ8@GZ
zbTR*TZ3bAjmSxP1*;KjDfvOQ2oh1bl2C^g#qc)si-|EEQKMH%lvR>ApF3a>;1h9{>
zj_AqxoLQUW@{CEBtX*d)pu+eGrEdXQ(CgZq$c|>06Z0ey>koAp
z2Ow}iZMa&*p9My#hT~L#+}ZO(5j0Wxc8fR=80=ykF-{bCH(cW{B{bVY%nO55g04}h
zQKV1&<4A>Y6M>&CCvbAx^_>QACXWlc1DC$oJSX}rMN~8~{smqG{npUy5D6%5>#LfN^CV@BY9vg`Y6uO9?#IxvL}q{u6dT0L
zilVg=?a6Tk=IasYW10BTDW+C;y4vtd2_@wm=iBQ~|ZA|jHK5@nzHPai3
zd{bxkTR9^X`gH{8(s4DqYBl}JocLp9v7bkFy4L@+X2>vFD^n0Z$zpeb;-kWsh+SMb
zH8Pt+7G5(}(Pl94vL?E=0kTiH%smVo`^~PtgQs%JpqX0(6j6%eEc%n0*kzo3@R~Tx
zlKy-Rm8&{28xErB$u7&e>{anCTSdfYI+$#rW+bZ*hKfk5WtEPPMT*GW<_jQaj6OCd
zrE-&qW%l50l5g2IUKOrwCgKhptzNn9TJ%B^HMKI9?DFoYtF2)V!Jb%>XkY5LLBP0*
zZNK4iOT_mh*FiJWKix$1@2MSk>NZGn!nwaj?YSKraFV{(Cm9o^Yr74^>yKnUSMs$_
zL~l3QL-w-p$UbNI7U^N7F#u|WV1q-
zqOwt@RnGd#gm>&LVuchVTYeqSXFiN>maF%&rFvdIaNaIBD!z6c*Vf6y;Bm^9mmua*
z5OZ0~sfTki29w!A46lt^xYNHBNR-3*iLN4&W+Evtf!GFcTeh7|2uGchs2iTI+
z9}!ZOPDe_Zpl#&$WZ}Q%`@ntA{g;>1e&k&<;iS^HA45t$T!|;sq4P;Ct)E!^xLfFc
z*$gW19=a>7X)7K^QKLv7!9|Jx;Ad1Xj`P{n%kUIWurv~;JN^qRE~}ybx_aJld$57=
zm5_d;?$l**>W=-PCtmyAlF=3spQG^(d=khF@jOr)1#weV{$Rfo&Ptp@2KP57aFU4+
zWQ?VGIh{Bx0WIlC3UxLDnYC^?%F(^AYH-fF(}6|)lh-kX{_3auo4`UKJRr4?J7pzA
zUvjBQDOdP{>W41Hw3;@QQ1GEjvi?xwS3m5^Vj=M~A1?L+*mkEq%cgB$Q(Ef8Vt#$s
z{UVvJ=A|}mVfe6?(D&QFA4s}E%g3NASJ>k!(eD+0>FjNy*c{$5^%V$PdX5vq1?qsh
zsz(=G{m(^NzZbZXe3m0LmX)O8^@*`)Dnrr`??zNRe*@S~`}%NB(pPp#;0tXa@cmW1
z`){_@A0o*+Qo43Sl!aw%FL%$5hA$qSJF3-*qj;HNXv0U(f#X=$s%jnjG|p1QJlBgL
z1uqb8aaHhL
zvX%uuG#Bp$y9ck+BG+Q2OR+fdimSuS_>rwquk>(+824?^3-^l!=Q~Buu_LB5p9BE#C(mZ+yAax;2wb?x;{U@voD|J$HkjT3;dtPOtJwd
zC`NYm%X-~B!Bv2
zxLgmph=*ptCu*rmo57ofXM-{9)&i+s+~HC`Xcdyb9U%rERQ8cNRhT}8-;WB12Aq2L
zT}2_1)-b*6K4;@h<$D}=!RUJ+Txa+>?X=|Qonn@Iy<}c%VGp&OQ(uiXCp_s863p-#
z)3yMNy!tVdNsh*FGz?8l8dP|sq>(C(si$GfsW($4zo&j7saY^MC7YCV^0`BtQKPe`
zps~~J8w1RS7)tX}8A$s))*nvtG54>96fGF-TC`w2oO^agZc=+kjcurx+Ly&bN$&Y+
z6^4AUU5-&{MfH{43l5(`i@eG8TPJK-HCO254ET%QcAn?VQupv7+}F@tR<)fA7&Y1E
z>!?2-@U@~>xpGY-rXso=KhM<^wc8QUennV)m2J1v+2r@V*=g}%slDX)xlF+0q&a}wpAxoY61HFS{psVQ
z{z&mSI!kkqkXPX8k^{ozdk3I=NoZti>y%WYBnxqj2#itL?HzDP8aDcgJV0dmo0`Wu
zY42(DJDyU2K9kdVB}op#whV1HSLma3m;z^YscpsXEhKxUAFhMes$gZ*t4hA))B1?6
zu!SWZg2;u-^P!6;dXJv&%TJcyCNv^H4kMvvbL_XRgnv++bvwIH@i`KA*oE{qCxm^K
z_tF9QU)tXor5t~Q`nZQ07}|l{r^<}SGvi_2Nf_L#^kFbFHZOTv&5b^+8u}wUoEIb4
zH0nWl7T3i3irThkMy&0)XC{y_@x|K$sN$P-`WQy~{o9}WvNK)TmYIVlfEKYf%oL<_
zf#0Ba212z&FflrTfuPez>iVdJfb+ChBE(4a^E_$1~gC_+s=~pVp;Z0kQEM1!7VW@Zf
zV$7M#Wje}r%w!oFcT@AM^N<%RU5q`9pkaY+u_ki7D=JcwxugyS#{+t}KyF#RX831k
zHGpCh
z{``i3L{BDIZv&b-XOk-3YU8r(nBzZb!`&iO*2%5`2Y%05nGqd
zNMK_}6swpcfDe)mo85M~`eyIos`d3uIi;VHC~a=fS*K@umU6990#YitoaSh*OI`R(
zNvl4rD!>6ca6vK-hrPk!STM7Jz~yjKp)8935>GL4ohw${{;ZQDi}oGayBQ8tdz*q~
zC*jIVkwcc08+_X7_PVPhq?F0fk|bg1sRx2F|Bb1^B-ZhK44QbAfM*(8z&=3x*n#)V
zR8ASDObqIG#$o{-{zLJw>CL?Ymh_Ves5Q8#*%JV_N4odc&{xeR__89ux92o==~fJm
zv;=)dj4a?%7E%~16fiSo#4v`EjmR+qL#CChWTjSE}b6k=oi
zfBd%0f!Mb9hoLD89)AI2euC4w0hoW)wRjAqP{ANG&Cf5EHTzS0&TCrfy+e88D;qk^
z+!zKkS}+WcNu?1LF6epKdXZjsQ_rey^Kz}+)AxmX9_5OjX{2h#_
zYLUy1NQu_h2@?{_X!$8R`Pe4<@T(>j`L{rBeWWjc%I;E=N-%b_9p%oX9AF^7e^!3r
zA&2Bc!#j(Rff;g4^p|{w=y@9#K(9bp{V}&ROvC)6&)Q3~xm_j-4fpQYk@S!=`=s9}
zc*qf#T!oXUpI~PAc&Td3rN~*orITWk67v<&i!vNxbIwxBsu9l1u8a8T*~isl2G*x_
zioazF)512lVzS;7z6b*EzOdZA##&}FW3>pJ5plz|yE5O7NPG<|a<{Tr5DpDxs%S^7
zy*i}*M0S5PC}uh;;>8kdyyxQpiA(c5u%Mc0VreTA9eMXHDeY5dt@$znLQxdXF@eah
zrLU{W@3D9yWGTf|f$<<2=8JDFQmdF@
z1~DF2hiw;~-a2){oEAHOc`HcTz!3Y-dVuUwEU{HDA;6}=fTof(JSg%oIfc4-A1svGEpLoWshqO|2I_}F+#Ie%$j50zW
z(Y_81po_Uu{B@mZT#s*o$Olg4P>(4dEae_>&W1u5R`{Fr`AqB=LPuY7)IjU3vD~A<
zJR4Be#WS&~y?6P*8ZX4^2SgDhdYw%@7*ge3$djL|p`)UW!)7kWVe&O5Gvf
z87YE!&xos9kITrW1K|U|44Wb!xWns)28Q9kNs}syVa7P%@JL;Uj9G0{DNhl`Ge5U7
zQzPeZPMM03FKB27VWe78W@O2E*-!4oa!HpXU+XeDyk<)W5
zJe+UnB?g#6?2P?>yoAKSFgK0jtVxRq(=>WW08mQB0w}ZKigEPB_irMi_K~_c@pXol
z!?y0NGI2EV?e#<`)32V=I3K?5*4Fcg&3h*l$r0NSiYj#iBb!z%?!*{o|g#^l?cuvj?J&^_Y>nwaI3w|
z@T#O6j#Vm$R*eV`;VKJvAWtII$GBt)xf_UQ4g94U;GfqVdm;ZC7puK@>4uCKM(y>h
z0EN8X2GpusQ0eFwk;(=$#Z~#&0atOf_h9X$jtaI~CH;BeszI%wP@I(c{xuIANK
zbK4euj)A}=2dW;aes&_QFOvKw$|7yc@=)Js&_
z+;^xDd%U5GW~ap%3-YW9WH@_935#RQaCvjYpjEMf9o*a@MdD{1-vo+HF#5?gzmveM
z%gJZ(!IzM^x)}!oYE>i9=GJLW%$1(<@4Z`ms?@7!iW}|>6Qi6Qc6a{Cn7=<
zDI#6}d;#6dl~OZHzaN)DVf|c_cBmxRIz=C&y9g7j9
zUKFREg1+M(mYa~%j&Lrks>=XG9-~)NCZ;Pk)+OjPtdLoAS>;0Qc~>dv!tr>{qU{Rz
zeWR<-?+;5MR(=-#G=OQd#-8Bb`_o3+(l2gL9Vo0v;mf%GDcqJV5yp3Bx2=+VH$TB%
z{4?rjZ*~PV(F~TbAd5@3Blp(wXza%JzdVG#k4hGrdX|ZU0{dlmlp}hZTs`JIJ&ukuo{O6kPjl#|
zar1U8!T56i+A(?x`bh9*Vw`8i4PUi1O?^-rI1}$Ft=O(4l6-LO4<(EnIF#tGvU#=b
zpM9{sQE^neT}`t{pO+V|;Rm6;8%@FZ$Tbz>vy++*Qv5cl$^4dN7H?|Fl^r>C0O;Nl
zjLOrsvf!>RdDI0Zhhjj2kDen2R}L&0*YlyZ)IzjdL-)VK-jKk!|yBiim&vg7Yk8YyQo*h
zAN+!}W3zGF2BmV>pdz4rEnoi)Y#5L+gCxJ*wNfRy%xaO9feZYFMZfgbR#bSciV8Wr
z+IA$w^VM5tZYMvrfoAbDSS1`I^;$wUSQ3ZUPpcA@`|b%eVh663%eZqWK?D8KXqP#Y
zS)Lz_ML`Zt#M8R9JpcHNb6~ubv&r(gA#~B?VARv@uBt&JQjrR0w>ygP#ekY4^uu?8
zcTAx!hb(g4vz2Xg*fR4LId0KKU3YsXTz~y@eF@aE}qJY#24nnesT|gd%fZnyBT%
zr61l(XmF$<0J<+@zQtkeg*Ebv!(6xn1NO$;FHVc!S1EFR%dDeT<;wFHOc%8oVrbn)
zuwo$A_*A8O)VXqk{6y_sIU2H@Ymja{T$)BGe?S@Qi3jIZyx4X
z+;c_vAX&f(-J%onMj+gEF;>Vi{)}=zBHA4p52@w~S?5P~2{D`T56+Iw2!U1i^*kpTAj9&72h@E2EnM={*T$Ssj}WhaRo<|uTeL4mY)>Y{0$EoCl<96=*vqFkUs!2Fg&jqK-xQh%-04nlmm(9&
zvI@@U6p;%gKMI%kwcQ!OGrDb_T1YwV3AFvOeB#f;vzX--hy#_zz&b`hC1uW*fl-`p
zHa4%LrL{T=j6lmY4O1uvT`@H!KR#BK_?DBFDkKbv>KUn+dmvTNr-eo18?#kWgD_TF
zPd`r>8s_}AJE-eVb-s~fSP14Y<~*P@1eMr@xBB#ZelGbVwV`Ko!boeiCu8H0x3dgk
zqO4`UJJ3JsxI;+mR=QM~dBQVBi0)VLOaT;z{%Y>
z-!w;2tj1XfHO_{~At)1<%pGxQF{f&Q=P5J9AN6kbH9R}DVTXT1u+k3t$Y4f!1xi+E%!~mS>X=0qR6hml}y@
z_z7|H3qw<1R+pbz^Kqh7DUpe6^#GiDHNq`f=AjI|F>>_1Ry>|MdkcQZ6j5$#q`_!f
z9SYufxZ#yv-#P>toi})saS^DZ;H99(YX1`Vv&k=UI?ItwynL!~#M7=JVF)r5XFm%i
zvJWq4F)k`R%8&LY~^uak;@5G
zG`*C^NJ}tpy8hA>4ME3?gjR5Ww|)w1YTA{wld|;X@ACSFg#3$eIX7K0C2dg@11+8V
zFN;o9s#7pZO$c2`i|5a$6}PI)7nYF3&6#GlG5a-
z#dIl_Xmdr14|V;qy%F);-CM>aBK1eaP2@tfvAg>m+bV1$nRvJ3zccwAbyLEE3z)bk
zn?o%D7UYQu`k*q*f~DU~EFqevElU=kCJ8i;=*)n#Fx(T?cah>&YfH@AI?qEjx5x&w
z`bdC7Hc&9+tyAOpi*jB*AwX~
z!wy4~aE>#)DP{}1&eg!-hBYAn6jx?h?jx!vW~zkBOQtS`V3+|vEO~)_7KBhaQ$xFG!yIQX{(YH!sZ~kDOAqe31m!fJH;xGk~VK1qGQm=wBGez-h|@T}i(o
z8ygkeJCwsP$AE5MtXNe2)9!=x?=kmdj5g~ZCZhZm5lX0Bf_Z@HawUVwOU`YQG!A7W-v(O#b7HT+Q^^-V#$mbQE!n+X1dGRaWF2%~?2B(IkAooYUGjEsd2;kM-Muw<8~G)3qq}Tjr@%QFb=(`_roJ{O!!YX!3zduN3mq4c{%z?g=*4
zXslC!lxax!qONLkP@_+c
Date: Wed, 10 Mar 2021 10:39:38 +0100
Subject: [PATCH 124/187] Fix random test failure

---
 tests/rpc/test_rpc_telegram.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/tests/rpc/test_rpc_telegram.py b/tests/rpc/test_rpc_telegram.py
index 924490821..27babb1b7 100644
--- a/tests/rpc/test_rpc_telegram.py
+++ b/tests/rpc/test_rpc_telegram.py
@@ -1130,7 +1130,7 @@ def test_telegram_trades(mocker, update, default_conf, fee):
     assert "Profit (" in msg_mock.call_args_list[0][0][0]
     assert "Close Date" in msg_mock.call_args_list[0][0][0]
     assert "
" in msg_mock.call_args_list[0][0][0]
-    assert bool(re.search("just now[ ]*XRP\\/BTC \\(#3\\)  1.00% \\(None\\)",
+    assert bool(re.search(r"just now[ ]*XRP\/BTC \(#3\)  1.00% \(",
                 msg_mock.call_args_list[0][0][0]))
 
 

From ef9977fc1e8d0cafb5de9fd57c93103e2bb9eb37 Mon Sep 17 00:00:00 2001
From: Matthias 
Date: Wed, 10 Mar 2021 10:43:44 +0100
Subject: [PATCH 125/187] Make stake_amount + stake_currency mandatory for
 backtesting

---
 docs/configuration.md                        |  6 ++----
 freqtrade/configuration/config_validation.py |  2 ++
 freqtrade/constants.py                       | 10 ++++++++++
 3 files changed, 14 insertions(+), 4 deletions(-)

diff --git a/docs/configuration.md b/docs/configuration.md
index 823b4bc20..20b26ec13 100644
--- a/docs/configuration.md
+++ b/docs/configuration.md
@@ -40,8 +40,8 @@ Mandatory parameters are marked as **Required**, which means that they are requi
 |  Parameter | Description |
 |------------|-------------|
 | `max_open_trades` | **Required.** Number of open trades your bot is allowed to have. Only one open trade per pair is possible, so the length of your pairlist is another limitation which can apply. If -1 then it is ignored (i.e. potentially unlimited open trades, limited by the pairlist). [More information below](#configuring-amount-per-trade).
 **Datatype:** Positive integer or -1.
-| `stake_currency` | **Required.** Crypto-currency used for trading. [Strategy Override](#parameters-in-the-strategy). 
 **Datatype:** String
-| `stake_amount` | **Required.** Amount of crypto-currency your bot will use for each trade. Set it to `"unlimited"` to allow the bot to use all available balance. [More information below](#configuring-amount-per-trade). [Strategy Override](#parameters-in-the-strategy). 
 **Datatype:** Positive float or `"unlimited"`.
+| `stake_currency` | **Required.** Crypto-currency used for trading. 
 **Datatype:** String
+| `stake_amount` | **Required.** Amount of crypto-currency your bot will use for each trade. Set it to `"unlimited"` to allow the bot to use all available balance. [More information below](#configuring-amount-per-trade). 
 **Datatype:** Positive float or `"unlimited"`.
 | `tradable_balance_ratio` | Ratio of the total account balance the bot is allowed to trade. [More information below](#configuring-amount-per-trade). 
*Defaults to `0.99` 99%).*
 **Datatype:** Positive float between `0.1` and `1.0`.
 | `amend_last_stake_amount` | Use reduced last stake amount if necessary. [More information below](#configuring-amount-per-trade). 
*Defaults to `false`.* 
 **Datatype:** Boolean
 | `last_stake_amount_min_ratio` | Defines minimum stake amount that has to be left and executed. Applies only to the last stake amount when it's amended to a reduced value (i.e. if `amend_last_stake_amount` is set to `true`). [More information below](#configuring-amount-per-trade). 
*Defaults to `0.5`.* 
 **Datatype:** Float (as ratio)
@@ -142,8 +142,6 @@ Values set in the configuration file always overwrite values set in the strategy
 * `process_only_new_candles`
 * `order_types`
 * `order_time_in_force`
-* `stake_currency`
-* `stake_amount`
 * `unfilledtimeout`
 * `disable_dataframe_checks`
 * `protections`
diff --git a/freqtrade/configuration/config_validation.py b/freqtrade/configuration/config_validation.py
index 187b2e3c7..df9f16f3e 100644
--- a/freqtrade/configuration/config_validation.py
+++ b/freqtrade/configuration/config_validation.py
@@ -47,6 +47,8 @@ def validate_config_schema(conf: Dict[str, Any]) -> Dict[str, Any]:
     conf_schema = deepcopy(constants.CONF_SCHEMA)
     if conf.get('runmode', RunMode.OTHER) in (RunMode.DRY_RUN, RunMode.LIVE):
         conf_schema['required'] = constants.SCHEMA_TRADE_REQUIRED
+    elif conf.get('runmode', RunMode.OTHER) in (RunMode.BACKTEST, RunMode.HYPEROPT):
+        conf_schema['required'] = constants.SCHEMA_BACKTEST_REQUIRED
     else:
         conf_schema['required'] = constants.SCHEMA_MINIMAL_REQUIRED
     try:
diff --git a/freqtrade/constants.py b/freqtrade/constants.py
index 06eaad4f9..f25f6653d 100644
--- a/freqtrade/constants.py
+++ b/freqtrade/constants.py
@@ -378,6 +378,16 @@ SCHEMA_TRADE_REQUIRED = [
     'dataformat_trades',
 ]
 
+SCHEMA_BACKTEST_REQUIRED = [
+    'exchange',
+    'max_open_trades',
+    'stake_currency',
+    'stake_amount',
+    'dry_run_wallet',
+    'dataformat_ohlcv',
+    'dataformat_trades',
+]
+
 SCHEMA_MINIMAL_REQUIRED = [
     'exchange',
     'dry_run',

From 425cd7adba609b810ad0a0486925c4585af42b76 Mon Sep 17 00:00:00 2001
From: Jackson Law <178053+jlaw@users.noreply.github.com>
Date: Fri, 12 Mar 2021 16:16:03 -0800
Subject: [PATCH 126/187] Create event loop manually if uvloop is available

asyncio.get_event_loop() does not call new_event_loop()
if current_thread() != main_thread()
---
 freqtrade/rpc/api_server/uvicorn_threaded.py | 27 +++++++++++++++++---
 1 file changed, 24 insertions(+), 3 deletions(-)

diff --git a/freqtrade/rpc/api_server/uvicorn_threaded.py b/freqtrade/rpc/api_server/uvicorn_threaded.py
index 1554a8e52..2f72cb74c 100644
--- a/freqtrade/rpc/api_server/uvicorn_threaded.py
+++ b/freqtrade/rpc/api_server/uvicorn_threaded.py
@@ -8,12 +8,33 @@ import uvicorn
 class UvicornServer(uvicorn.Server):
     """
     Multithreaded server - as found in https://github.com/encode/uvicorn/issues/742
+
+    Removed install_signal_handlers() override based on changes from this commit:
+        https://github.com/encode/uvicorn/commit/ce2ef45a9109df8eae038c0ec323eb63d644cbc6
+
+    Cannot rely on asyncio.get_event_loop() to create new event loop because of this check:
+        https://github.com/python/cpython/blob/4d7f11e05731f67fd2c07ec2972c6cb9861d52be/Lib/asyncio/events.py#L638
+
+    Fix by overriding run() and forcing creation of new event loop if uvloop is available
     """
-    def install_signal_handlers(self):
+
+    def run(self, sockets=None):
+        import asyncio
+
         """
-        In the parent implementation, this starts the thread, therefore we must patch it away here.
+        Parent implementation calls self.config.setup_event_loop(),
+            but we need to create uvloop event loop manually
         """
-        pass
+        try:
+            import uvloop  # noqa
+        except ImportError:  # pragma: no cover
+            from uvicorn.loops.asyncio import asyncio_setup
+            asyncio_setup()
+        else:
+            asyncio.set_event_loop(uvloop.new_event_loop())
+
+        loop = asyncio.get_event_loop()
+        loop.run_until_complete(self.serve(sockets=sockets))
 
     @contextlib.contextmanager
     def run_in_thread(self):

From d1acc8092cef3bd2a2035642640a7cab92429dcd Mon Sep 17 00:00:00 2001
From: Matthias 
Date: Sat, 13 Mar 2021 10:16:32 +0100
Subject: [PATCH 127/187] Improve backtest performance

---
 freqtrade/optimize/backtesting.py |  4 +++-
 freqtrade/persistence/models.py   | 32 ++++++++++++++++++++++++++++---
 freqtrade/wallets.py              | 11 ++++++++---
 tests/conftest.py                 |  2 +-
 tests/test_persistence.py         |  2 +-
 5 files changed, 42 insertions(+), 9 deletions(-)

diff --git a/freqtrade/optimize/backtesting.py b/freqtrade/optimize/backtesting.py
index 575ad486a..f2cf0d0dc 100644
--- a/freqtrade/optimize/backtesting.py
+++ b/freqtrade/optimize/backtesting.py
@@ -377,7 +377,7 @@ class Backtesting:
                         open_trade_count += 1
                         # logger.debug(f"{pair} - Emulate creation of new trade: {trade}.")
                         open_trades[pair].append(trade)
-                        LocalTrade.trades.append(trade)
+                        LocalTrade.add_bt_trade(trade)
 
                 for trade in open_trades[pair]:
                     # also check the buying candle for sell conditions.
@@ -387,6 +387,8 @@ class Backtesting:
                         # logger.debug(f"{pair} - Backtesting sell {trade}")
                         open_trade_count -= 1
                         open_trades[pair].remove(trade)
+
+                        LocalTrade.close_bt_trade(trade)
                         trades.append(trade_entry)
                         if enable_protections:
                             self.protections.stop_per_pair(pair, row[DATE_IDX])
diff --git a/freqtrade/persistence/models.py b/freqtrade/persistence/models.py
index ab714ae8b..41a5a99ff 100644
--- a/freqtrade/persistence/models.py
+++ b/freqtrade/persistence/models.py
@@ -208,6 +208,8 @@ class LocalTrade():
     use_db: bool = False
     # Trades container for backtesting
     trades: List['LocalTrade'] = []
+    trades_open: List['LocalTrade'] = []
+    total_profit: float = 0
 
     id: int = 0
 
@@ -350,6 +352,8 @@ class LocalTrade():
         Resets all trades. Only active for backtesting mode.
         """
         LocalTrade.trades = []
+        LocalTrade.trades_open = []
+        LocalTrade.total_profit = 0
 
     def adjust_min_max_rates(self, current_price: float) -> None:
         """
@@ -599,7 +603,17 @@ class LocalTrade():
         """
 
         # Offline mode - without database
-        sel_trades = [trade for trade in LocalTrade.trades]
+        if is_open is not None:
+            if is_open:
+                sel_trades = LocalTrade.trades_open
+            else:
+                sel_trades = LocalTrade.trades
+
+        else:
+            # Not used during backtesting, but might be used by a strategy
+            sel_trades = [trade for trade in LocalTrade.trades + LocalTrade.trades_open
+                          if trade.is_open == is_open]
+
         if pair:
             sel_trades = [trade for trade in sel_trades if trade.pair == pair]
         if open_date:
@@ -607,10 +621,22 @@ class LocalTrade():
         if close_date:
             sel_trades = [trade for trade in sel_trades if trade.close_date
                           and trade.close_date > close_date]
-        if is_open is not None:
-            sel_trades = [trade for trade in sel_trades if trade.is_open == is_open]
+
         return sel_trades
 
+    @staticmethod
+    def close_bt_trade(trade):
+        LocalTrade.trades_open.remove(trade)
+        LocalTrade.trades.append(trade)
+        LocalTrade.total_profit += trade.close_profit_abs
+
+    @staticmethod
+    def add_bt_trade(trade):
+        if trade.is_open:
+            LocalTrade.trades_open.append(trade)
+        else:
+            LocalTrade.trades.append(trade)
+
     @staticmethod
     def get_open_trades() -> List[Any]:
         """
diff --git a/freqtrade/wallets.py b/freqtrade/wallets.py
index 553f7c61d..575fe1b67 100644
--- a/freqtrade/wallets.py
+++ b/freqtrade/wallets.py
@@ -10,7 +10,7 @@ import arrow
 from freqtrade.constants import UNLIMITED_STAKE_AMOUNT
 from freqtrade.exceptions import DependencyException
 from freqtrade.exchange import Exchange
-from freqtrade.persistence import Trade
+from freqtrade.persistence import LocalTrade, Trade
 from freqtrade.state import RunMode
 
 
@@ -66,9 +66,14 @@ class Wallets:
         """
         # Recreate _wallets to reset closed trade balances
         _wallets = {}
-        closed_trades = Trade.get_trades_proxy(is_open=False)
         open_trades = Trade.get_trades_proxy(is_open=True)
-        tot_profit = sum([trade.close_profit_abs for trade in closed_trades])
+        # If not backtesting...
+        # TODO: potentially remove the ._log workaround to determine backtest mode.
+        if self._log:
+            closed_trades = Trade.get_trades_proxy(is_open=False)
+            tot_profit = sum([trade.close_profit_abs for trade in closed_trades])
+        else:
+            tot_profit = LocalTrade.total_profit
         tot_in_trades = sum([trade.stake_amount for trade in open_trades])
 
         current_stake = self.start_cap + tot_profit - tot_in_trades
diff --git a/tests/conftest.py b/tests/conftest.py
index 498d65b0a..801ffad2f 100644
--- a/tests/conftest.py
+++ b/tests/conftest.py
@@ -191,7 +191,7 @@ def create_mock_trades(fee, use_db: bool = True):
         if use_db:
             Trade.session.add(trade)
         else:
-            LocalTrade.trades.append(trade)
+            LocalTrade.add_bt_trade(trade)
 
     # Simulate dry_run entries
     trade = mock_trade_1(fee)
diff --git a/tests/test_persistence.py b/tests/test_persistence.py
index 1a8124b00..8c89c98ed 100644
--- a/tests/test_persistence.py
+++ b/tests/test_persistence.py
@@ -1196,6 +1196,6 @@ def test_Trade_object_idem():
     # Fails if only a column is added without corresponding parent field
     for item in localtrade:
         if (not item.startswith('__')
-                and item not in ('trades', )
+                and item not in ('trades', 'trades_open', 'total_profit')
                 and type(getattr(LocalTrade, item)) not in (property, FunctionType)):
             assert item in trade

From 5e872273d1217e43e15ce91317f9ef1f68f36277 Mon Sep 17 00:00:00 2001
From: Rokas Kupstys <19151258+rokups@users.noreply.github.com>
Date: Sat, 13 Mar 2021 09:45:16 +0200
Subject: [PATCH 128/187] Provide access to strategy instance from hyperopt
 class.

---
 freqtrade/optimize/hyperopt.py           | 1 +
 freqtrade/optimize/hyperopt_interface.py | 2 ++
 2 files changed, 3 insertions(+)

diff --git a/freqtrade/optimize/hyperopt.py b/freqtrade/optimize/hyperopt.py
index 16a39d7d6..6b5bc171b 100644
--- a/freqtrade/optimize/hyperopt.py
+++ b/freqtrade/optimize/hyperopt.py
@@ -73,6 +73,7 @@ class Hyperopt:
         self.backtesting = Backtesting(self.config)
 
         self.custom_hyperopt = HyperOptResolver.load_hyperopt(self.config)
+        self.custom_hyperopt.__class__.strategy = self.backtesting.strategy
 
         self.custom_hyperoptloss = HyperOptLossResolver.load_hyperoptloss(self.config)
         self.calculate_loss = self.custom_hyperoptloss.hyperopt_loss_function
diff --git a/freqtrade/optimize/hyperopt_interface.py b/freqtrade/optimize/hyperopt_interface.py
index b8c44ed59..561fb8e11 100644
--- a/freqtrade/optimize/hyperopt_interface.py
+++ b/freqtrade/optimize/hyperopt_interface.py
@@ -12,6 +12,7 @@ from skopt.space import Categorical, Dimension, Integer, Real
 from freqtrade.exceptions import OperationalException
 from freqtrade.exchange import timeframe_to_minutes
 from freqtrade.misc import round_dict
+from freqtrade.strategy import IStrategy
 
 
 logger = logging.getLogger(__name__)
@@ -34,6 +35,7 @@ class IHyperOpt(ABC):
     """
     ticker_interval: str  # DEPRECATED
     timeframe: str
+    strategy: IStrategy
 
     def __init__(self, config: dict) -> None:
         self.config = config

From 0320c8dc92d4e4acb4f61310f202ba6db2be840a Mon Sep 17 00:00:00 2001
From: Matthias 
Date: Sat, 13 Mar 2021 15:46:20 +0100
Subject: [PATCH 129/187] Improve tests for trades_proxy

---
 freqtrade/persistence/models.py |  3 +--
 tests/conftest_trades.py        |  4 ++++
 tests/test_persistence.py       | 22 ++++++++++++++++++++++
 3 files changed, 27 insertions(+), 2 deletions(-)

diff --git a/freqtrade/persistence/models.py b/freqtrade/persistence/models.py
index 41a5a99ff..ed8a2259b 100644
--- a/freqtrade/persistence/models.py
+++ b/freqtrade/persistence/models.py
@@ -611,8 +611,7 @@ class LocalTrade():
 
         else:
             # Not used during backtesting, but might be used by a strategy
-            sel_trades = [trade for trade in LocalTrade.trades + LocalTrade.trades_open
-                          if trade.is_open == is_open]
+            sel_trades = [trade for trade in LocalTrade.trades + LocalTrade.trades_open]
 
         if pair:
             sel_trades = [trade for trade in sel_trades if trade.pair == pair]
diff --git a/tests/conftest_trades.py b/tests/conftest_trades.py
index 1d775830d..8e4be9165 100644
--- a/tests/conftest_trades.py
+++ b/tests/conftest_trades.py
@@ -29,6 +29,7 @@ def mock_trade_1(fee):
         fee_open=fee.return_value,
         fee_close=fee.return_value,
         is_open=True,
+        open_date=datetime.now(tz=timezone.utc) - timedelta(minutes=17),
         open_rate=0.123,
         exchange='bittrex',
         open_order_id='dry_run_buy_12345',
@@ -183,6 +184,7 @@ def mock_trade_4(fee):
         amount_requested=124.0,
         fee_open=fee.return_value,
         fee_close=fee.return_value,
+        open_date=datetime.now(tz=timezone.utc) - timedelta(minutes=14),
         is_open=True,
         open_rate=0.123,
         exchange='bittrex',
@@ -234,6 +236,7 @@ def mock_trade_5(fee):
         amount_requested=124.0,
         fee_open=fee.return_value,
         fee_close=fee.return_value,
+        open_date=datetime.now(tz=timezone.utc) - timedelta(minutes=12),
         is_open=True,
         open_rate=0.123,
         exchange='bittrex',
@@ -284,6 +287,7 @@ def mock_trade_6(fee):
         stake_amount=0.001,
         amount=2.0,
         amount_requested=2.0,
+        open_date=datetime.now(tz=timezone.utc) - timedelta(minutes=5),
         fee_open=fee.return_value,
         fee_close=fee.return_value,
         is_open=True,
diff --git a/tests/test_persistence.py b/tests/test_persistence.py
index 8c89c98ed..ab900cbb8 100644
--- a/tests/test_persistence.py
+++ b/tests/test_persistence.py
@@ -1,5 +1,6 @@
 # pragma pylint: disable=missing-docstring, C0103
 import logging
+from datetime import datetime, timedelta, timezone
 from types import FunctionType
 from unittest.mock import MagicMock
 
@@ -1044,6 +1045,7 @@ def test_fee_updated(fee):
 def test_total_open_trades_stakes(fee, use_db):
 
     Trade.use_db = use_db
+    Trade.reset_trades()
     res = Trade.total_open_trades_stakes()
     assert res == 0
     create_mock_trades(fee, use_db)
@@ -1053,6 +1055,26 @@ def test_total_open_trades_stakes(fee, use_db):
     Trade.use_db = True
 
 
+@pytest.mark.usefixtures("init_persistence")
+@pytest.mark.parametrize('use_db', [True, False])
+def test_get_trades_proxy(fee, use_db):
+    Trade.use_db = use_db
+    Trade.reset_trades()
+    create_mock_trades(fee, use_db)
+    trades = Trade.get_trades_proxy()
+    assert len(trades) == 6
+
+    assert isinstance(trades[0], Trade)
+
+    assert len(Trade.get_trades_proxy(is_open=True)) == 4
+    assert len(Trade.get_trades_proxy(is_open=False)) == 2
+    opendate = datetime.now(tz=timezone.utc) - timedelta(minutes=15)
+
+    assert len(Trade.get_trades_proxy(open_date=opendate)) == 3
+
+    Trade.use_db = True
+
+
 @pytest.mark.usefixtures("init_persistence")
 def test_get_overall_performance(fee):
 

From 6389e86ed68c42425c456fe3c8373436be78f71c Mon Sep 17 00:00:00 2001
From: Matthias 
Date: Sat, 13 Mar 2021 15:36:34 +0100
Subject: [PATCH 130/187] Add test for uvloop fix

---
 tests/conftest.py               | 12 +++++++++--
 tests/exchange/test_exchange.py | 10 +--------
 tests/rpc/test_rpc_apiserver.py | 38 ++++++++++++++++++++++++++++++---
 3 files changed, 46 insertions(+), 14 deletions(-)

diff --git a/tests/conftest.py b/tests/conftest.py
index 498d65b0a..75632e4c7 100644
--- a/tests/conftest.py
+++ b/tests/conftest.py
@@ -6,7 +6,7 @@ from copy import deepcopy
 from datetime import datetime
 from functools import reduce
 from pathlib import Path
-from unittest.mock import MagicMock, PropertyMock
+from unittest.mock import MagicMock, Mock, PropertyMock
 
 import arrow
 import numpy as np
@@ -64,6 +64,14 @@ def get_args(args):
     return Arguments(args).get_parsed_arg()
 
 
+# Source: https://stackoverflow.com/questions/29881236/how-to-mock-asyncio-coroutines
+def get_mock_coro(return_value):
+    async def mock_coro(*args, **kwargs):
+        return return_value
+
+    return Mock(wraps=mock_coro)
+
+
 def patched_configuration_load_config_file(mocker, config) -> None:
     mocker.patch(
         'freqtrade.configuration.configuration.load_config_file',
@@ -1736,7 +1744,7 @@ def import_fails() -> None:
     realimport = builtins.__import__
 
     def mockedimport(name, *args, **kwargs):
-        if name in ["filelock", 'systemd.journal']:
+        if name in ["filelock", 'systemd.journal', 'uvloop']:
             raise ImportError(f"No module named '{name}'")
         return realimport(name, *args, **kwargs)
 
diff --git a/tests/exchange/test_exchange.py b/tests/exchange/test_exchange.py
index 75db2de26..3fd566fa3 100644
--- a/tests/exchange/test_exchange.py
+++ b/tests/exchange/test_exchange.py
@@ -18,21 +18,13 @@ from freqtrade.exchange.exchange import (market_is_active, timeframe_to_minutes,
                                          timeframe_to_next_date, timeframe_to_prev_date,
                                          timeframe_to_seconds)
 from freqtrade.resolvers.exchange_resolver import ExchangeResolver
-from tests.conftest import get_patched_exchange, log_has, log_has_re
+from tests.conftest import get_mock_coro, get_patched_exchange, log_has, log_has_re
 
 
 # Make sure to always keep one exchange here which is NOT subclassed!!
 EXCHANGES = ['bittrex', 'binance', 'kraken', 'ftx']
 
 
-# Source: https://stackoverflow.com/questions/29881236/how-to-mock-asyncio-coroutines
-def get_mock_coro(return_value):
-    async def mock_coro(*args, **kwargs):
-        return return_value
-
-    return Mock(wraps=mock_coro)
-
-
 def ccxt_exceptionhandlers(mocker, default_conf, api_mock, exchange_name,
                            fun, mock_ccxt_fun, retries=API_RETRY_COUNT + 1, **kwargs):
 
diff --git a/tests/rpc/test_rpc_apiserver.py b/tests/rpc/test_rpc_apiserver.py
index 8590e0d21..01492b4f2 100644
--- a/tests/rpc/test_rpc_apiserver.py
+++ b/tests/rpc/test_rpc_apiserver.py
@@ -23,8 +23,8 @@ from freqtrade.rpc.api_server import ApiServer
 from freqtrade.rpc.api_server.api_auth import create_token, get_user_from_token
 from freqtrade.rpc.api_server.uvicorn_threaded import UvicornServer
 from freqtrade.state import RunMode, State
-from tests.conftest import (create_mock_trades, get_patched_freqtradebot, log_has, log_has_re,
-                            patch_get_signal)
+from tests.conftest import (create_mock_trades, get_mock_coro, get_patched_freqtradebot, log_has,
+                            log_has_re, patch_get_signal)
 
 
 BASE_URI = "/api/v1"
@@ -230,7 +230,7 @@ def test_api__init__(default_conf, mocker):
     assert apiserver._config == default_conf
 
 
-def test_api_UvicornServer(default_conf, mocker):
+def test_api_UvicornServer(mocker):
     thread_mock = mocker.patch('freqtrade.rpc.api_server.uvicorn_threaded.threading.Thread')
     s = UvicornServer(uvicorn.Config(MagicMock(), port=8080, host='127.0.0.1'))
     assert thread_mock.call_count == 0
@@ -248,6 +248,38 @@ def test_api_UvicornServer(default_conf, mocker):
     assert s.should_exit is True
 
 
+def test_api_UvicornServer_run(mocker):
+    serve_mock = mocker.patch('freqtrade.rpc.api_server.uvicorn_threaded.UvicornServer.serve',
+                              get_mock_coro(None))
+    s = UvicornServer(uvicorn.Config(MagicMock(), port=8080, host='127.0.0.1'))
+    assert serve_mock.call_count == 0
+
+    s.install_signal_handlers()
+    # Original implementation starts a thread - make sure that's not the case
+    assert serve_mock.call_count == 0
+
+    # Fake started to avoid sleeping forever
+    s.started = True
+    s.run()
+    assert serve_mock.call_count == 1
+
+
+def test_api_UvicornServer_run_no_uvloop(mocker, import_fails):
+    serve_mock = mocker.patch('freqtrade.rpc.api_server.uvicorn_threaded.UvicornServer.serve',
+                              get_mock_coro(None))
+    s = UvicornServer(uvicorn.Config(MagicMock(), port=8080, host='127.0.0.1'))
+    assert serve_mock.call_count == 0
+
+    s.install_signal_handlers()
+    # Original implementation starts a thread - make sure that's not the case
+    assert serve_mock.call_count == 0
+
+    # Fake started to avoid sleeping forever
+    s.started = True
+    s.run()
+    assert serve_mock.call_count == 1
+
+
 def test_api_run(default_conf, mocker, caplog):
     default_conf.update({"api_server": {"enabled": True,
                                         "listen_ip_address": "127.0.0.1",

From eb4f05eb23efa08be363a374054439b0a3ccd07e Mon Sep 17 00:00:00 2001
From: Matthias 
Date: Sat, 13 Mar 2021 16:30:47 +0100
Subject: [PATCH 131/187] Add documentation for hyperopt.strategy availability

---
 docs/advanced-hyperopt.md | 47 ++++++++++++++++++++++++++++++++++++++-
 docs/hyperopt.md          |  2 +-
 2 files changed, 47 insertions(+), 2 deletions(-)

diff --git a/docs/advanced-hyperopt.md b/docs/advanced-hyperopt.md
index d2237b3e8..bdaafb936 100644
--- a/docs/advanced-hyperopt.md
+++ b/docs/advanced-hyperopt.md
@@ -6,7 +6,7 @@ class.
 
 ## Derived hyperopt classes
 
-Custom hyperop classes can be derived in the same way [it can be done for strategies](strategy-customization.md#derived-strategies).
+Custom hyperopt classes can be derived in the same way [it can be done for strategies](strategy-customization.md#derived-strategies).
 
 Applying to hyperoptimization, as an example, you may override how dimensions are defined in your optimization hyperspace:
 
@@ -32,6 +32,51 @@ or
 $ freqtrade hyperopt --hyperopt MyAwesomeHyperOpt2 --hyperopt-loss SharpeHyperOptLossDaily --strategy MyAwesomeStrategy ...
 ```
 
+## Sharing methods with your strategy
+
+Hyperopt classes provide access to the Strategy via the `strategy` class attribute.
+This can be a great way to reduce code duplication if used correctly, but will also complicate usage for inexperienced users.
+
+``` python
+from pandas import DataFrame
+from freqtrade.strategy.interface import IStrategy
+import freqtrade.vendor.qtpylib.indicators as qtpylib
+
+class MyAwesomeStrategy(IStrategy):
+
+    buy_params = {
+        'rsi-value': 30,
+        'adx-value': 35,
+    }
+
+    def populate_buy_trend(self, dataframe: DataFrame, metadata: dict) -> DataFrame:
+        return self.buy_strategy_generator(self.buy_params, dataframe, metadata)
+
+    @staticmethod
+    def buy_strategy_generator(params, dataframe: DataFrame, metadata: dict) -> DataFrame:
+        dataframe.loc[
+            (
+                qtpylib.crossed_above(dataframe['rsi'], params['rsi-value']) &
+                dataframe['adx'] > params['adx-value']) &
+                dataframe['volume'] > 0
+            )
+            , 'buy'] = 1
+        return dataframe
+
+class MyAwesomeHyperOpt(IHyperOpt):
+    ...
+    @staticmethod
+    def buy_strategy_generator(params: Dict[str, Any]) -> Callable:
+        """
+        Define the buy strategy parameters to be used by Hyperopt.
+        """
+        def populate_buy_trend(dataframe: DataFrame, metadata: dict) -> DataFrame:
+            # Call strategy's buy strategy generator
+            return self.StrategyClass.buy_strategy_generator(params, dataframe, metadata)
+
+        return populate_buy_trend
+```
+
 ## Creating and using a custom loss function
 
 To use a custom loss function class, make sure that the function `hyperopt_loss_function` is defined in your custom hyperopt loss class.
diff --git a/docs/hyperopt.md b/docs/hyperopt.md
index d6959b457..69bc57d1a 100644
--- a/docs/hyperopt.md
+++ b/docs/hyperopt.md
@@ -283,7 +283,7 @@ So let's write the buy strategy using these values:
         """
         Define the buy strategy parameters to be used by Hyperopt.
         """
-        def populate_buy_trend(dataframe: DataFrame) -> DataFrame:
+        def populate_buy_trend(dataframe: DataFrame, metadata: dict) -> DataFrame:
             conditions = []
             # GUARDS AND TRENDS
             if 'adx-enabled' in params and params['adx-enabled']:

From 618bae23a64930c3b4cb84fb1298e8a0e7b4c40d Mon Sep 17 00:00:00 2001
From: Jackson Law <178053+jlaw@users.noreply.github.com>
Date: Sat, 13 Mar 2021 11:14:36 -0800
Subject: [PATCH 132/187] fix: Use now() to match timezone of download data

---
 tests/commands/test_commands.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/tests/commands/test_commands.py b/tests/commands/test_commands.py
index d5e76eeb6..27875ac94 100644
--- a/tests/commands/test_commands.py
+++ b/tests/commands/test_commands.py
@@ -706,7 +706,7 @@ def test_download_data_timerange(mocker, caplog, markets):
     start_download_data(get_args(args))
     assert dl_mock.call_count == 1
     # 20days ago
-    days_ago = arrow.get(arrow.utcnow().shift(days=-20).date()).int_timestamp
+    days_ago = arrow.get(arrow.now().shift(days=-20).date()).int_timestamp
     assert dl_mock.call_args_list[0][1]['timerange'].startts == days_ago
 
     dl_mock.reset_mock()

From b57c150654235086164f1199584ce990aa38f854 Mon Sep 17 00:00:00 2001
From: Matthias 
Date: Sun, 14 Mar 2021 09:48:40 +0100
Subject: [PATCH 133/187] Final balance should include forcesold pairs

---
 freqtrade/optimize/backtesting.py | 1 +
 1 file changed, 1 insertion(+)

diff --git a/freqtrade/optimize/backtesting.py b/freqtrade/optimize/backtesting.py
index f2cf0d0dc..0b884dae5 100644
--- a/freqtrade/optimize/backtesting.py
+++ b/freqtrade/optimize/backtesting.py
@@ -301,6 +301,7 @@ class Backtesting:
                     trade.close_date = sell_row[DATE_IDX]
                     trade.sell_reason = SellType.FORCE_SELL.value
                     trade.close(sell_row[OPEN_IDX], show_msg=False)
+                    LocalTrade.close_bt_trade(trade)
                     # Deepcopy object to have wallets update correctly
                     trade1 = deepcopy(trade)
                     trade1.is_open = True

From 7a63f8cc319d9543b9041caa5be9497e6fb01d4d Mon Sep 17 00:00:00 2001
From: Matthias 
Date: Sun, 14 Mar 2021 13:25:08 +0100
Subject: [PATCH 134/187] Fix hdf5 support on raspberry

---
 Dockerfile.armhf | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/Dockerfile.armhf b/Dockerfile.armhf
index f938ec457..eecd9fdc0 100644
--- a/Dockerfile.armhf
+++ b/Dockerfile.armhf
@@ -41,7 +41,9 @@ COPY --from=python-deps /root/.local /root/.local
 
 # Install and execute
 COPY . /freqtrade/
-RUN pip install -e . --no-cache-dir \
+RUN apt-get install -y libhdf5-serial-dev \
+  && apt-get clean \
+  && pip install -e . --no-cache-dir \
   && freqtrade install-ui
 
 ENTRYPOINT ["freqtrade"]

From e92441643157652467eed377a43d057d32c18708 Mon Sep 17 00:00:00 2001
From: Brook Miles 
Date: Sun, 14 Mar 2021 22:02:53 +0900
Subject: [PATCH 135/187] correct math used in examples and clarify some
 terminology regarding custom stoploss functions

---
 docs/strategy-advanced.md | 64 +++++++++++++++++++++++++++------------
 1 file changed, 44 insertions(+), 20 deletions(-)

diff --git a/docs/strategy-advanced.md b/docs/strategy-advanced.md
index 56061365e..cda988acd 100644
--- a/docs/strategy-advanced.md
+++ b/docs/strategy-advanced.md
@@ -71,12 +71,13 @@ See `custom_stoploss` examples below on how to access the saved dataframe column
 
 ## Custom stoploss
 
-A stoploss can only ever move upwards - so if you set it to an absolute profit of 2%, you can never move it below this price.
-Also, the traditional `stoploss` value serves as an absolute lower level and will be instated as the initial stoploss.
+The stoploss price can only ever move upwards - if the stoploss value returned from `custom_stoploss` would result in a lower stoploss price than was previously set, it will be ignored. The traditional `stoploss` value serves as an absolute lower level and will be instated as the initial stoploss.
 
 The usage of the custom stoploss method must be enabled by setting `use_custom_stoploss=True` on the strategy object.
-The method must return a stoploss value (float / number) with a relative ratio below the current price.
-E.g. `current_profit = 0.05` (5% profit) - stoploss returns `0.02` - then you "locked in" a profit of 3% (`0.05 - 0.02 = 0.03`).
+The method must return a stoploss value (float / number) as a percentage of the current price.
+E.g. If the `current_rate` is 200 USD, then returning `0.02` will set the stoploss price 2% lower, at 196 USD.
+
+The absolute value of the return value is used (the sign is ignored), so returning `0.05` or `-0.05` have the same result, a stoploss 5% below the current price.
 
 To simulate a regular trailing stoploss of 4% (trailing 4% behind the maximum reached price) you would use the following very simple method:
 
@@ -177,16 +178,33 @@ class AwesomeStrategy(IStrategy):
         return -0.15
 ```
 
+
+#### Calculating stoploss relative to open price
+
+Stoploss values returned from `custom_stoploss` always specify a percentage relative to `current_rate`.  In order to set a stoploss relative to the *open* price, we need to use `current_profit` to calculate what percentage relative to the `current_rate` will give you the same result as if the percentage was specified from the open price.
+
+This can be calculated as:
+
+``` python
+def stoploss_from_open(open_relative_stop: float, current_profit: float) -> float:
+    return 1-((1+open_relative_stop)/(1+current_profit))
+
+```
+
+For example, say our open price was $100, and `current_price` is $121 (`current_profit` will be `0.21`).  If we want a stop price at 7% above the open price we can call `stoploss_from_open(0.07, 0.21)` which will return `0.1157024793`.  11.57% below $121 is $107, which is the same as 7% above $100.
+
 #### Trailing stoploss with positive offset
 
 Use the initial stoploss until the profit is above 4%, then use a trailing stoploss of 50% of the current profit with a minimum of 2.5% and a maximum of 5%.
 
-Please note that the stoploss can only increase, values lower than the current stoploss are ignored.
 
 ``` python
 from datetime import datetime, timedelta
 from freqtrade.persistence import Trade
 
+def stoploss_from_open(open_relative_stop: float, current_profit: float) -> float:
+    return 1-((1+open_relative_stop)/(1+current_profit))
+
 class AwesomeStrategy(IStrategy):
 
     # ... populate_* methods
@@ -197,28 +215,32 @@ class AwesomeStrategy(IStrategy):
                         current_rate: float, current_profit: float, **kwargs) -> float:
 
         if current_profit < 0.04:
-            return -1 # return a value bigger than the inital stoploss to keep using the inital stoploss
+            return 1 # return a value bigger than the inital stoploss to keep using the inital stoploss
 
         # After reaching the desired offset, allow the stoploss to trail by half the profit
-        desired_stoploss = current_profit / 2
-
         # Use a minimum of 2.5% and a maximum of 5%
-        return max(min(desired_stoploss, 0.05), 0.025)
+        desired_stop_from_open = max(min(current_profit / 2, 0.05), 0.025)
+
+        return stoploss_from_open(desired_stop_from_open, current_profit)
 ```
 
-#### Absolute stoploss
+#### Stepped stoploss
 
-The below example sets absolute profit levels based on the current profit.
+Instead of continuously trailing behind the current price, this example sets fixed stoploss price levels based on the current profit.
 
 * Use the regular stoploss until 20% profit is reached
-* Once profit is > 40%, stoploss will be at 25%, locking in at least 25% of the profit.
-* Once profit is > 25% - stoploss will be 15%.
-* Once profit is > 20% - stoploss will be set to 7%.
+* Once profit is > 20% - set stoploss to 7% above open price.
+* Once profit is > 25% - set stoploss to 15% above open price.
+* Once profit is > 40% - set stoploss to 25% above open price.
+
 
 ``` python
 from datetime import datetime
 from freqtrade.persistence import Trade
 
+def stoploss_from_open(open_relative_stop: float, current_profit: float) -> float:
+    return 1-((1+open_relative_stop)/(1+current_profit))
+
 class AwesomeStrategy(IStrategy):
 
     # ... populate_* methods
@@ -228,13 +250,15 @@ class AwesomeStrategy(IStrategy):
     def custom_stoploss(self, pair: str, trade: 'Trade', current_time: datetime,
                         current_rate: float, current_profit: float, **kwargs) -> float:
 
-        # Calculate as `-desired_stop_from_open + current_profit` to get the distance between current_profit and initial price
+        # evaluate highest to lowest, so that highest possible stop is used
         if current_profit > 0.40:
-            return (-0.25 + current_profit)
-        if current_profit > 0.25:
-            return (-0.15 + current_profit)
-        if current_profit > 0.20:
-            return (-0.07 + current_profit)
+            return stoploss_from_open(0.25, current_profit)
+        elif current_profit > 0.25:
+            return stoploss_from_open(0.15, current_profit)
+        elif current_profit > 0.20:
+            return stoploss_from_open(0.07, current_profit)
+
+        # return maximum stoploss value, keeping current stoploss price unchanged
         return 1
 ```
 #### Custom stoploss using an indicator from dataframe example

From 0b35c0571fd1d0ac5beadd645c2c87958331e61a Mon Sep 17 00:00:00 2001
From: Matthias 
Date: Sun, 14 Mar 2021 19:37:30 +0100
Subject: [PATCH 136/187] Allow custom fee to be used during dry-run

closes #3696
---
 docs/configuration.md           | 1 +
 freqtrade/commands/arguments.py | 2 +-
 freqtrade/exchange/exchange.py  | 2 ++
 tests/exchange/test_exchange.py | 8 ++++++++
 4 files changed, 12 insertions(+), 1 deletion(-)

diff --git a/docs/configuration.md b/docs/configuration.md
index 20b26ec13..2e8edca2e 100644
--- a/docs/configuration.md
+++ b/docs/configuration.md
@@ -58,6 +58,7 @@ Mandatory parameters are marked as **Required**, which means that they are requi
 | `trailing_stop_positive` | Changes stoploss once profit has been reached. More details in the [stoploss documentation](stoploss.md#trailing-stop-loss-custom-positive-loss). [Strategy Override](#parameters-in-the-strategy). 
 **Datatype:** Float
 | `trailing_stop_positive_offset` | Offset on when to apply `trailing_stop_positive`. Percentage value which should be positive. More details in the [stoploss documentation](stoploss.md#trailing-stop-loss-only-once-the-trade-has-reached-a-certain-offset). [Strategy Override](#parameters-in-the-strategy). 
*Defaults to `0.0` (no offset).* 
 **Datatype:** Float
 | `trailing_only_offset_is_reached` | Only apply trailing stoploss when the offset is reached. [stoploss documentation](stoploss.md). [Strategy Override](#parameters-in-the-strategy). 
*Defaults to `false`.*  
 **Datatype:** Boolean
+| `fee` | Fee used during backtesting / dry-runs. Should normally not be configured, which has freqtrade fall back to the exchange default fee. Set as ratio (e.g. 0.001 = 0.1%). Fee is applied twice for each trade, once when buying, once when selling. 
 **Datatype:** Float (as ratio)
 | `unfilledtimeout.buy` | **Required.** How long (in minutes) the bot will wait for an unfilled buy order to complete, after which the order will be cancelled and repeated at current (new) price, as long as there is a signal. [Strategy Override](#parameters-in-the-strategy).
 **Datatype:** Integer
 | `unfilledtimeout.sell` | **Required.** How long (in minutes) the bot will wait for an unfilled sell order to complete, after which the order will be cancelled and repeated at current (new) price, as long as there is a signal. [Strategy Override](#parameters-in-the-strategy).
 **Datatype:** Integer
 | `bid_strategy.price_side` | Select the side of the spread the bot should look at to get the buy rate. [More information below](#buy-price-side).
 *Defaults to `bid`.* 
 **Datatype:** String (either `ask` or `bid`).
diff --git a/freqtrade/commands/arguments.py b/freqtrade/commands/arguments.py
index 88cec7b3e..9468a7f7d 100644
--- a/freqtrade/commands/arguments.py
+++ b/freqtrade/commands/arguments.py
@@ -14,7 +14,7 @@ ARGS_COMMON = ["verbosity", "logfile", "version", "config", "datadir", "user_dat
 
 ARGS_STRATEGY = ["strategy", "strategy_path"]
 
-ARGS_TRADE = ["db_url", "sd_notify", "dry_run", "dry_run_wallet", ]
+ARGS_TRADE = ["db_url", "sd_notify", "dry_run", "dry_run_wallet", "fee"]
 
 ARGS_COMMON_OPTIMIZE = ["timeframe", "timerange", "dataformat_ohlcv",
                         "max_open_trades", "stake_amount", "fee"]
diff --git a/freqtrade/exchange/exchange.py b/freqtrade/exchange/exchange.py
index 138be3537..fdb34eb41 100644
--- a/freqtrade/exchange/exchange.py
+++ b/freqtrade/exchange/exchange.py
@@ -1232,6 +1232,8 @@ class Exchange:
     def get_fee(self, symbol: str, type: str = '', side: str = '', amount: float = 1,
                 price: float = 1, taker_or_maker: str = 'maker') -> float:
         try:
+            if self._config['dry_run'] and self._config.get('fee', None) is not None:
+                return self._config['fee']
             # validate that markets are loaded before trying to get fee
             if self._api.markets is None or len(self._api.markets) == 0:
                 self._api.load_markets()
diff --git a/tests/exchange/test_exchange.py b/tests/exchange/test_exchange.py
index 3fd566fa3..8a8c95a62 100644
--- a/tests/exchange/test_exchange.py
+++ b/tests/exchange/test_exchange.py
@@ -2268,12 +2268,20 @@ def test_get_fee(default_conf, mocker, exchange_name):
         'cost': 0.05
     })
     exchange = get_patched_exchange(mocker, default_conf, api_mock, id=exchange_name)
+    exchange._config.pop('fee', None)
 
     assert exchange.get_fee('ETH/BTC') == 0.025
+    assert api_mock.calculate_fee.call_count == 1
 
     ccxt_exceptionhandlers(mocker, default_conf, api_mock, exchange_name,
                            'get_fee', 'calculate_fee', symbol="ETH/BTC")
 
+    api_mock.calculate_fee.reset_mock()
+    exchange._config['fee'] = 0.001
+
+    assert exchange.get_fee('ETH/BTC') == 0.001
+    assert api_mock.calculate_fee.call_count == 0
+
 
 def test_stoploss_order_unsupported_exchange(default_conf, mocker):
     exchange = get_patched_exchange(mocker, default_conf, id='bittrex')

From b191663a7ec512c68607a40408ae9e07f0d5275d Mon Sep 17 00:00:00 2001
From: Matthias 
Date: Sun, 14 Mar 2021 19:49:46 +0100
Subject: [PATCH 137/187] Adapt hyperopt templates to be better aligned

closes #3027
---
 freqtrade/templates/base_hyperopt.py.j2       | 23 ++---
 freqtrade/templates/sample_hyperopt.py        | 94 ++++++------------
 .../templates/sample_hyperopt_advanced.py     | 97 ++++++-------------
 3 files changed, 72 insertions(+), 142 deletions(-)

diff --git a/freqtrade/templates/base_hyperopt.py.j2 b/freqtrade/templates/base_hyperopt.py.j2
index ec787cbb6..f6ca1477a 100644
--- a/freqtrade/templates/base_hyperopt.py.j2
+++ b/freqtrade/templates/base_hyperopt.py.j2
@@ -39,6 +39,15 @@ class {{ hyperopt }}(IHyperOpt):
     https://github.com/freqtrade/freqtrade/blob/develop/freqtrade/templates/sample_hyperopt_advanced.py.
     """
 
+    @staticmethod
+    def indicator_space() -> List[Dimension]:
+        """
+        Define your Hyperopt space for searching buy strategy parameters.
+        """
+        return [
+            {{ buy_space | indent(12) }}
+        ]
+
     @staticmethod
     def buy_strategy_generator(params: Dict[str, Any]) -> Callable:
         """
@@ -79,12 +88,12 @@ class {{ hyperopt }}(IHyperOpt):
         return populate_buy_trend
 
     @staticmethod
-    def indicator_space() -> List[Dimension]:
+    def sell_indicator_space() -> List[Dimension]:
         """
-        Define your Hyperopt space for searching buy strategy parameters.
+        Define your Hyperopt space for searching sell strategy parameters.
         """
         return [
-            {{ buy_space | indent(12) }}
+            {{ sell_space | indent(12) }}
         ]
 
     @staticmethod
@@ -126,11 +135,3 @@ class {{ hyperopt }}(IHyperOpt):
 
         return populate_sell_trend
 
-    @staticmethod
-    def sell_indicator_space() -> List[Dimension]:
-        """
-        Define your Hyperopt space for searching sell strategy parameters.
-        """
-        return [
-            {{ sell_space | indent(12) }}
-        ]
diff --git a/freqtrade/templates/sample_hyperopt.py b/freqtrade/templates/sample_hyperopt.py
index 10743e911..ed1af7718 100644
--- a/freqtrade/templates/sample_hyperopt.py
+++ b/freqtrade/templates/sample_hyperopt.py
@@ -45,6 +45,23 @@ class SampleHyperOpt(IHyperOpt):
     https://github.com/freqtrade/freqtrade/blob/develop/freqtrade/templates/sample_hyperopt_advanced.py.
     """
 
+    @staticmethod
+    def indicator_space() -> List[Dimension]:
+        """
+        Define your Hyperopt space for searching buy strategy parameters.
+        """
+        return [
+            Integer(10, 25, name='mfi-value'),
+            Integer(15, 45, name='fastd-value'),
+            Integer(20, 50, name='adx-value'),
+            Integer(20, 40, name='rsi-value'),
+            Categorical([True, False], name='mfi-enabled'),
+            Categorical([True, False], name='fastd-enabled'),
+            Categorical([True, False], name='adx-enabled'),
+            Categorical([True, False], name='rsi-enabled'),
+            Categorical(['bb_lower', 'macd_cross_signal', 'sar_reversal'], name='trigger')
+        ]
+
     @staticmethod
     def buy_strategy_generator(params: Dict[str, Any]) -> Callable:
         """
@@ -92,20 +109,22 @@ class SampleHyperOpt(IHyperOpt):
         return populate_buy_trend
 
     @staticmethod
-    def indicator_space() -> List[Dimension]:
+    def sell_indicator_space() -> List[Dimension]:
         """
-        Define your Hyperopt space for searching buy strategy parameters.
+        Define your Hyperopt space for searching sell strategy parameters.
         """
         return [
-            Integer(10, 25, name='mfi-value'),
-            Integer(15, 45, name='fastd-value'),
-            Integer(20, 50, name='adx-value'),
-            Integer(20, 40, name='rsi-value'),
-            Categorical([True, False], name='mfi-enabled'),
-            Categorical([True, False], name='fastd-enabled'),
-            Categorical([True, False], name='adx-enabled'),
-            Categorical([True, False], name='rsi-enabled'),
-            Categorical(['bb_lower', 'macd_cross_signal', 'sar_reversal'], name='trigger')
+            Integer(75, 100, name='sell-mfi-value'),
+            Integer(50, 100, name='sell-fastd-value'),
+            Integer(50, 100, name='sell-adx-value'),
+            Integer(60, 100, name='sell-rsi-value'),
+            Categorical([True, False], name='sell-mfi-enabled'),
+            Categorical([True, False], name='sell-fastd-enabled'),
+            Categorical([True, False], name='sell-adx-enabled'),
+            Categorical([True, False], name='sell-rsi-enabled'),
+            Categorical(['sell-bb_upper',
+                         'sell-macd_cross_signal',
+                         'sell-sar_reversal'], name='sell-trigger')
         ]
 
     @staticmethod
@@ -153,56 +172,3 @@ class SampleHyperOpt(IHyperOpt):
             return dataframe
 
         return populate_sell_trend
-
-    @staticmethod
-    def sell_indicator_space() -> List[Dimension]:
-        """
-        Define your Hyperopt space for searching sell strategy parameters.
-        """
-        return [
-            Integer(75, 100, name='sell-mfi-value'),
-            Integer(50, 100, name='sell-fastd-value'),
-            Integer(50, 100, name='sell-adx-value'),
-            Integer(60, 100, name='sell-rsi-value'),
-            Categorical([True, False], name='sell-mfi-enabled'),
-            Categorical([True, False], name='sell-fastd-enabled'),
-            Categorical([True, False], name='sell-adx-enabled'),
-            Categorical([True, False], name='sell-rsi-enabled'),
-            Categorical(['sell-bb_upper',
-                         'sell-macd_cross_signal',
-                         'sell-sar_reversal'], name='sell-trigger')
-        ]
-
-    def populate_buy_trend(self, dataframe: DataFrame, metadata: dict) -> DataFrame:
-        """
-        Based on TA indicators. Should be a copy of same method from strategy.
-        Must align to populate_indicators in this file.
-        Only used when --spaces does not include buy space.
-        """
-        dataframe.loc[
-            (
-                (dataframe['close'] < dataframe['bb_lowerband']) &
-                (dataframe['mfi'] < 16) &
-                (dataframe['adx'] > 25) &
-                (dataframe['rsi'] < 21)
-            ),
-            'buy'] = 1
-
-        return dataframe
-
-    def populate_sell_trend(self, dataframe: DataFrame, metadata: dict) -> DataFrame:
-        """
-        Based on TA indicators. Should be a copy of same method from strategy.
-        Must align to populate_indicators in this file.
-        Only used when --spaces does not include sell space.
-        """
-        dataframe.loc[
-            (
-                (qtpylib.crossed_above(
-                    dataframe['macdsignal'], dataframe['macd']
-                )) &
-                (dataframe['fastd'] > 54)
-            ),
-            'sell'] = 1
-
-        return dataframe
diff --git a/freqtrade/templates/sample_hyperopt_advanced.py b/freqtrade/templates/sample_hyperopt_advanced.py
index 52e397466..7736570f7 100644
--- a/freqtrade/templates/sample_hyperopt_advanced.py
+++ b/freqtrade/templates/sample_hyperopt_advanced.py
@@ -60,6 +60,23 @@ class AdvancedSampleHyperOpt(IHyperOpt):
         dataframe['sar'] = ta.SAR(dataframe)
         return dataframe
 
+    @staticmethod
+    def indicator_space() -> List[Dimension]:
+        """
+        Define your Hyperopt space for searching buy strategy parameters.
+        """
+        return [
+            Integer(10, 25, name='mfi-value'),
+            Integer(15, 45, name='fastd-value'),
+            Integer(20, 50, name='adx-value'),
+            Integer(20, 40, name='rsi-value'),
+            Categorical([True, False], name='mfi-enabled'),
+            Categorical([True, False], name='fastd-enabled'),
+            Categorical([True, False], name='adx-enabled'),
+            Categorical([True, False], name='rsi-enabled'),
+            Categorical(['bb_lower', 'macd_cross_signal', 'sar_reversal'], name='trigger')
+        ]
+
     @staticmethod
     def buy_strategy_generator(params: Dict[str, Any]) -> Callable:
         """
@@ -106,20 +123,22 @@ class AdvancedSampleHyperOpt(IHyperOpt):
         return populate_buy_trend
 
     @staticmethod
-    def indicator_space() -> List[Dimension]:
+    def sell_indicator_space() -> List[Dimension]:
         """
-        Define your Hyperopt space for searching strategy parameters
+        Define your Hyperopt space for searching sell strategy parameters.
         """
         return [
-            Integer(10, 25, name='mfi-value'),
-            Integer(15, 45, name='fastd-value'),
-            Integer(20, 50, name='adx-value'),
-            Integer(20, 40, name='rsi-value'),
-            Categorical([True, False], name='mfi-enabled'),
-            Categorical([True, False], name='fastd-enabled'),
-            Categorical([True, False], name='adx-enabled'),
-            Categorical([True, False], name='rsi-enabled'),
-            Categorical(['bb_lower', 'macd_cross_signal', 'sar_reversal'], name='trigger')
+            Integer(75, 100, name='sell-mfi-value'),
+            Integer(50, 100, name='sell-fastd-value'),
+            Integer(50, 100, name='sell-adx-value'),
+            Integer(60, 100, name='sell-rsi-value'),
+            Categorical([True, False], name='sell-mfi-enabled'),
+            Categorical([True, False], name='sell-fastd-enabled'),
+            Categorical([True, False], name='sell-adx-enabled'),
+            Categorical([True, False], name='sell-rsi-enabled'),
+            Categorical(['sell-bb_upper',
+                         'sell-macd_cross_signal',
+                         'sell-sar_reversal'], name='sell-trigger')
         ]
 
     @staticmethod
@@ -168,25 +187,6 @@ class AdvancedSampleHyperOpt(IHyperOpt):
 
         return populate_sell_trend
 
-    @staticmethod
-    def sell_indicator_space() -> List[Dimension]:
-        """
-        Define your Hyperopt space for searching sell strategy parameters
-        """
-        return [
-            Integer(75, 100, name='sell-mfi-value'),
-            Integer(50, 100, name='sell-fastd-value'),
-            Integer(50, 100, name='sell-adx-value'),
-            Integer(60, 100, name='sell-rsi-value'),
-            Categorical([True, False], name='sell-mfi-enabled'),
-            Categorical([True, False], name='sell-fastd-enabled'),
-            Categorical([True, False], name='sell-adx-enabled'),
-            Categorical([True, False], name='sell-rsi-enabled'),
-            Categorical(['sell-bb_upper',
-                         'sell-macd_cross_signal',
-                         'sell-sar_reversal'], name='sell-trigger')
-        ]
-
     @staticmethod
     def generate_roi_table(params: Dict) -> Dict[int, float]:
         """
@@ -267,40 +267,3 @@ class AdvancedSampleHyperOpt(IHyperOpt):
 
             Categorical([True, False], name='trailing_only_offset_is_reached'),
         ]
-
-    def populate_buy_trend(self, dataframe: DataFrame, metadata: dict) -> DataFrame:
-        """
-        Based on TA indicators.
-        Can be a copy of the corresponding method from the strategy,
-        or will be loaded from the strategy.
-        Must align to populate_indicators used (either from this File, or from the strategy)
-        Only used when --spaces does not include buy
-        """
-        dataframe.loc[
-            (
-                (dataframe['close'] < dataframe['bb_lowerband']) &
-                (dataframe['mfi'] < 16) &
-                (dataframe['adx'] > 25) &
-                (dataframe['rsi'] < 21)
-            ),
-            'buy'] = 1
-
-        return dataframe
-
-    def populate_sell_trend(self, dataframe: DataFrame, metadata: dict) -> DataFrame:
-        """
-        Based on TA indicators.
-        Can be a copy of the corresponding method from the strategy,
-        or will be loaded from the strategy.
-        Must align to populate_indicators used (either from this File, or from the strategy)
-        Only used when --spaces does not include sell
-        """
-        dataframe.loc[
-            (
-                (qtpylib.crossed_above(
-                    dataframe['macdsignal'], dataframe['macd']
-                )) &
-                (dataframe['fastd'] > 54)
-            ),
-            'sell'] = 1
-        return dataframe

From 09872d8e42cd6a2ccccb228bad037c0aa90def42 Mon Sep 17 00:00:00 2001
From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com>
Date: Mon, 15 Mar 2021 05:27:18 +0000
Subject: [PATCH 138/187] Bump flake8 from 3.8.4 to 3.9.0

Bumps [flake8](https://gitlab.com/pycqa/flake8) from 3.8.4 to 3.9.0.
- [Release notes](https://gitlab.com/pycqa/flake8/tags)
- [Commits](https://gitlab.com/pycqa/flake8/compare/3.8.4...3.9.0)

Signed-off-by: dependabot[bot] 
---
 requirements-dev.txt | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/requirements-dev.txt b/requirements-dev.txt
index 68b1dd53f..4f0ea7706 100644
--- a/requirements-dev.txt
+++ b/requirements-dev.txt
@@ -4,7 +4,7 @@
 -r requirements-hyperopt.txt
 
 coveralls==3.0.1
-flake8==3.8.4
+flake8==3.9.0
 flake8-type-annotations==0.1.0
 flake8-tidy-imports==4.2.1
 mypy==0.812

From 22c34faca388ee38887813710825e2283c7651ac Mon Sep 17 00:00:00 2001
From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com>
Date: Mon, 15 Mar 2021 05:28:02 +0000
Subject: [PATCH 139/187] Bump mkdocs-material from 7.0.5 to 7.0.6

Bumps [mkdocs-material](https://github.com/squidfunk/mkdocs-material) from 7.0.5 to 7.0.6.
- [Release notes](https://github.com/squidfunk/mkdocs-material/releases)
- [Changelog](https://github.com/squidfunk/mkdocs-material/blob/master/docs/changelog.md)
- [Commits](https://github.com/squidfunk/mkdocs-material/compare/7.0.5...7.0.6)

Signed-off-by: dependabot[bot] 
---
 docs/requirements-docs.txt | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/docs/requirements-docs.txt b/docs/requirements-docs.txt
index 22c09ff69..0068dd5d2 100644
--- a/docs/requirements-docs.txt
+++ b/docs/requirements-docs.txt
@@ -1,3 +1,3 @@
-mkdocs-material==7.0.5
+mkdocs-material==7.0.6
 mdx_truly_sane_lists==1.2
 pymdown-extensions==8.1.1

From 1173d8971a57f4005a48b4067110fd985d90e726 Mon Sep 17 00:00:00 2001
From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com>
Date: Mon, 15 Mar 2021 05:28:06 +0000
Subject: [PATCH 140/187] Bump prompt-toolkit from 3.0.16 to 3.0.17

Bumps [prompt-toolkit](https://github.com/prompt-toolkit/python-prompt-toolkit) from 3.0.16 to 3.0.17.
- [Release notes](https://github.com/prompt-toolkit/python-prompt-toolkit/releases)
- [Changelog](https://github.com/prompt-toolkit/python-prompt-toolkit/blob/master/CHANGELOG)
- [Commits](https://github.com/prompt-toolkit/python-prompt-toolkit/commits)

Signed-off-by: dependabot[bot] 
---
 requirements.txt | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/requirements.txt b/requirements.txt
index f62c8ff52..1cbfadb17 100644
--- a/requirements.txt
+++ b/requirements.txt
@@ -39,4 +39,4 @@ aiofiles==0.6.0
 colorama==0.4.4
 # Building config files interactively
 questionary==1.9.0
-prompt-toolkit==3.0.16
+prompt-toolkit==3.0.17

From a209b0a392415c3b2b8382f097f4982c10497fe7 Mon Sep 17 00:00:00 2001
From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com>
Date: Mon, 15 Mar 2021 05:28:18 +0000
Subject: [PATCH 141/187] Bump python-telegram-bot from 13.3 to 13.4.1

Bumps [python-telegram-bot](https://github.com/python-telegram-bot/python-telegram-bot) from 13.3 to 13.4.1.
- [Release notes](https://github.com/python-telegram-bot/python-telegram-bot/releases)
- [Changelog](https://github.com/python-telegram-bot/python-telegram-bot/blob/master/CHANGES.rst)
- [Commits](https://github.com/python-telegram-bot/python-telegram-bot/compare/v13.3...v13.4.1)

Signed-off-by: dependabot[bot] 
---
 requirements.txt | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/requirements.txt b/requirements.txt
index f62c8ff52..77d22dbf3 100644
--- a/requirements.txt
+++ b/requirements.txt
@@ -6,7 +6,7 @@ ccxt==1.42.66
 cryptography==3.4.6
 aiohttp==3.7.4.post0
 SQLAlchemy==1.3.23
-python-telegram-bot==13.3
+python-telegram-bot==13.4.1
 arrow==1.0.3
 cachetools==4.2.1
 requests==2.25.1

From b6c29bebb07bbab854f1086c75df7b8ad16d6ab3 Mon Sep 17 00:00:00 2001
From: Matthias 
Date: Mon, 15 Mar 2021 06:56:48 +0100
Subject: [PATCH 142/187] Update slack action

---
 .github/workflows/ci.yml | 12 ++++++------
 1 file changed, 6 insertions(+), 6 deletions(-)

diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index 3f294347a..61ecaa522 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -102,7 +102,7 @@ jobs:
         mypy freqtrade scripts
 
     - name: Slack Notification
-      uses: homoluctus/slatify@v1.8.0
+      uses: lazy-actions/slatify@v3.0.0
       if: failure() && ( github.event_name != 'pull_request' || github.event.pull_request.head.repo.fork == false)
       with:
         type: ${{ job.status }}
@@ -194,7 +194,7 @@ jobs:
         mypy freqtrade scripts
 
     - name: Slack Notification
-      uses: homoluctus/slatify@v1.8.0
+      uses: lazy-actions/slatify@v3.0.0
       if: failure() && ( github.event_name != 'pull_request' || github.event.pull_request.head.repo.fork == false)
       with:
         type: ${{ job.status }}
@@ -257,7 +257,7 @@ jobs:
         mypy freqtrade scripts
 
     - name: Slack Notification
-      uses: homoluctus/slatify@v1.8.0
+      uses: lazy-actions/slatify@v3.0.0
       if: failure() && ( github.event_name != 'pull_request' || github.event.pull_request.head.repo.fork == false)
       with:
         type: ${{ job.status }}
@@ -288,7 +288,7 @@ jobs:
         mkdocs build
 
     - name: Slack Notification
-      uses: homoluctus/slatify@v1.8.0
+      uses: lazy-actions/slatify@v3.0.0
       if: failure() && ( github.event_name != 'pull_request' || github.event.pull_request.head.repo.fork == false)
       with:
         type: ${{ job.status }}
@@ -311,7 +311,7 @@ jobs:
     runs-on: ubuntu-20.04
     steps:
     - name: Slack Notification
-      uses: homoluctus/slatify@v1.8.0
+      uses: lazy-actions/slatify@v3.0.0
       if: always() && ( github.event_name != 'pull_request' || github.event.pull_request.head.repo.fork == false)
       with:
         type: ${{ job.status }}
@@ -398,7 +398,7 @@ jobs:
 
 
     - name: Slack Notification
-      uses: homoluctus/slatify@v1.8.0
+      uses: lazy-actions/slatify@v3.0.0
       if: always() && ( github.event_name != 'pull_request' || github.event.pull_request.head.repo.fork == false)
       with:
         type: ${{ job.status }}

From 8f26935259f20fadf33d3b928fa5f6a1dde10a43 Mon Sep 17 00:00:00 2001
From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com>
Date: Mon, 15 Mar 2021 09:26:02 +0000
Subject: [PATCH 143/187] Bump ccxt from 1.42.66 to 1.43.27

Bumps [ccxt](https://github.com/ccxt/ccxt) from 1.42.66 to 1.43.27.
- [Release notes](https://github.com/ccxt/ccxt/releases)
- [Changelog](https://github.com/ccxt/ccxt/blob/master/doc/exchanges-by-country.rst)
- [Commits](https://github.com/ccxt/ccxt/compare/1.42.66...1.43.27)

Signed-off-by: dependabot[bot] 
---
 requirements.txt | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/requirements.txt b/requirements.txt
index 77d22dbf3..090cdc588 100644
--- a/requirements.txt
+++ b/requirements.txt
@@ -1,7 +1,7 @@
 numpy==1.20.1
 pandas==1.2.3
 
-ccxt==1.42.66
+ccxt==1.43.27
 # Pin cryptography for now due to rust build errors with piwheels
 cryptography==3.4.6
 aiohttp==3.7.4.post0

From 79d4585dadf14e7e3749cabb498ef8cfe47f99eb Mon Sep 17 00:00:00 2001
From: Matthias 
Date: Tue, 16 Mar 2021 19:24:03 +0100
Subject: [PATCH 144/187] Add check to ensure close_profit_abs is filled on
 closed trades

Technically, this should not be possible, but #4554 shows it is.

closes #4554
---
 freqtrade/wallets.py      |  3 ++-
 tests/test_persistence.py | 10 ++++++++--
 2 files changed, 10 insertions(+), 3 deletions(-)

diff --git a/freqtrade/wallets.py b/freqtrade/wallets.py
index 575fe1b67..f4432e932 100644
--- a/freqtrade/wallets.py
+++ b/freqtrade/wallets.py
@@ -71,7 +71,8 @@ class Wallets:
         # TODO: potentially remove the ._log workaround to determine backtest mode.
         if self._log:
             closed_trades = Trade.get_trades_proxy(is_open=False)
-            tot_profit = sum([trade.close_profit_abs for trade in closed_trades])
+            tot_profit = sum(
+                [trade.close_profit_abs for trade in closed_trades if trade.close_profit_abs])
         else:
             tot_profit = LocalTrade.total_profit
         tot_in_trades = sum([trade.stake_amount for trade in open_trades])
diff --git a/tests/test_persistence.py b/tests/test_persistence.py
index ab900cbb8..1820250a5 100644
--- a/tests/test_persistence.py
+++ b/tests/test_persistence.py
@@ -1066,8 +1066,14 @@ def test_get_trades_proxy(fee, use_db):
 
     assert isinstance(trades[0], Trade)
 
-    assert len(Trade.get_trades_proxy(is_open=True)) == 4
-    assert len(Trade.get_trades_proxy(is_open=False)) == 2
+    trades = Trade.get_trades_proxy(is_open=True)
+    assert len(trades) == 4
+    assert trades[0].is_open
+    trades = Trade.get_trades_proxy(is_open=False)
+
+    assert len(trades) == 2
+    assert not trades[0].is_open
+
     opendate = datetime.now(tz=timezone.utc) - timedelta(minutes=15)
 
     assert len(Trade.get_trades_proxy(open_date=opendate)) == 3

From aee2591490b442a731c8efb039658e8230958cd5 Mon Sep 17 00:00:00 2001
From: Brook Miles 
Date: Wed, 17 Mar 2021 17:58:23 +0900
Subject: [PATCH 145/187] add stoploss_from_open() as a strategy_helper

---
 docs/strategy-customization.md        | 36 +++++++++++++++++++++++++++
 freqtrade/strategy/__init__.py        |  1 +
 freqtrade/strategy/strategy_helper.py | 18 ++++++++++++++
 3 files changed, 55 insertions(+)

diff --git a/docs/strategy-customization.md b/docs/strategy-customization.md
index a1708a481..bf086bc0a 100644
--- a/docs/strategy-customization.md
+++ b/docs/strategy-customization.md
@@ -587,6 +587,42 @@ All columns of the informative dataframe will be available on the returning data
 
 ***
 
+### *stoploss_from_open()*
+
+Stoploss values returned from `custom_stoploss` must specify a percentage relative to `current_rate`, but sometimes you may want to specify a stoploss relative to the open price instead. `stoploss_from_open()` is a helper function to calculate a stoploss value that can be returned from `custom_stoploss` which will be equivalent to the desired percentage above the open price.
+
+??? Example "Returning a stoploss relative to the open price from the custom stoploss function"
+
+    Say the open price was $100, and `current_price` is $121 (`current_profit` will be `0.21`).  
+
+    If we want a stop price at 7% above the open price we can call `stoploss_from_open(0.07, current_profit)` which will return `0.1157024793`.  11.57% below $121 is $107, which is the same as 7% above $100.
+
+
+    ``` python
+
+    from freqtrade.strategy import IStrategy, stoploss_from_open
+    from datetime import datetime
+    from freqtrade.persistence import Trade
+
+    class AwesomeStrategy(IStrategy):
+
+        # ... populate_* methods
+
+        use_custom_stoploss = True
+
+        def custom_stoploss(self, pair: str, trade: 'Trade', current_time: datetime,
+                            current_rate: float, current_profit: float, **kwargs) -> float:
+
+            # once the profit has risin above 10%, keep the stoploss at 7% above the open price
+            if current_profit > 0.10:
+                return stoploss_from_open(0.07, current_profit)
+
+            return 1
+
+    ```
+
+
+
 ## Additional data (Wallets)
 
 The strategy provides access to the `Wallets` object. This contains the current balances on the exchange.
diff --git a/freqtrade/strategy/__init__.py b/freqtrade/strategy/__init__.py
index 662156ae9..3de90666e 100644
--- a/freqtrade/strategy/__init__.py
+++ b/freqtrade/strategy/__init__.py
@@ -3,3 +3,4 @@ from freqtrade.exchange import (timeframe_to_minutes, timeframe_to_msecs, timefr
                                 timeframe_to_prev_date, timeframe_to_seconds)
 from freqtrade.strategy.interface import IStrategy
 from freqtrade.strategy.strategy_helper import merge_informative_pair
+from freqtrade.strategy.strategy_helper import stoploss_from_open
diff --git a/freqtrade/strategy/strategy_helper.py b/freqtrade/strategy/strategy_helper.py
index d7b1327d9..f40fa285d 100644
--- a/freqtrade/strategy/strategy_helper.py
+++ b/freqtrade/strategy/strategy_helper.py
@@ -56,3 +56,21 @@ def merge_informative_pair(dataframe: pd.DataFrame, informative: pd.DataFrame,
         dataframe = dataframe.ffill()
 
     return dataframe
+
+
+def stoploss_from_open(open_relative_stop: float, current_profit: float) -> float:
+    """
+
+    Given the current profit, and a desired stop loss value relative to the open price,
+    return a stop loss value that is relative to the current price, and which can be
+    returned from `custom_stoploss`.
+
+    :param open_relative_stop: Desired stop loss value relative to open price
+    :param current_profit: The current profit percentage
+    :return: Stop loss value relative to current price
+    """
+
+    if current_profit == -1:
+        return 1
+
+    return 1-((1+open_relative_stop)/(1+current_profit))

From ce1ed76269370862f6f717c4d9ffe98b049d7caa Mon Sep 17 00:00:00 2001
From: Brook Miles 
Date: Wed, 17 Mar 2021 22:44:10 +0900
Subject: [PATCH 146/187] complete stoploss_from_open and associated test

---
 freqtrade/strategy/__init__.py          |  3 +-
 freqtrade/strategy/strategy_helper.py   | 15 ++++++++--
 tests/strategy/test_strategy_helpers.py | 39 ++++++++++++++++++++++++-
 3 files changed, 51 insertions(+), 6 deletions(-)

diff --git a/freqtrade/strategy/__init__.py b/freqtrade/strategy/__init__.py
index 3de90666e..85148b6ea 100644
--- a/freqtrade/strategy/__init__.py
+++ b/freqtrade/strategy/__init__.py
@@ -2,5 +2,4 @@
 from freqtrade.exchange import (timeframe_to_minutes, timeframe_to_msecs, timeframe_to_next_date,
                                 timeframe_to_prev_date, timeframe_to_seconds)
 from freqtrade.strategy.interface import IStrategy
-from freqtrade.strategy.strategy_helper import merge_informative_pair
-from freqtrade.strategy.strategy_helper import stoploss_from_open
+from freqtrade.strategy.strategy_helper import merge_informative_pair, stoploss_from_open
diff --git a/freqtrade/strategy/strategy_helper.py b/freqtrade/strategy/strategy_helper.py
index f40fa285d..22b6f0be5 100644
--- a/freqtrade/strategy/strategy_helper.py
+++ b/freqtrade/strategy/strategy_helper.py
@@ -65,12 +65,21 @@ def stoploss_from_open(open_relative_stop: float, current_profit: float) -> floa
     return a stop loss value that is relative to the current price, and which can be
     returned from `custom_stoploss`.
 
-    :param open_relative_stop: Desired stop loss value relative to open price
+    The requested stop can be positive for a stop above the open price, or negative for
+    a stop below the open price. The return value is always >= 0.
+
+    Returns 0 if the resulting stop price would be above the current price.
+
+    :param open_relative_stop: Desired stop loss percentage relative to open price
     :param current_profit: The current profit percentage
-    :return: Stop loss value relative to current price
+    :return: Positive stop loss value relative to current price
     """
 
+    # formula is undefined for current_profit -1, return maximum value
     if current_profit == -1:
         return 1
 
-    return 1-((1+open_relative_stop)/(1+current_profit))
+    stoploss = 1-((1+open_relative_stop)/(1+current_profit))
+
+    # negative stoploss values indicate the requested stop price is higher than the current price
+    return max(stoploss, 0.0)
diff --git a/tests/strategy/test_strategy_helpers.py b/tests/strategy/test_strategy_helpers.py
index 252288e2e..3b84fc254 100644
--- a/tests/strategy/test_strategy_helpers.py
+++ b/tests/strategy/test_strategy_helpers.py
@@ -1,8 +1,10 @@
+from math import isclose
+
 import numpy as np
 import pandas as pd
 import pytest
 
-from freqtrade.strategy import merge_informative_pair, timeframe_to_minutes
+from freqtrade.strategy import merge_informative_pair, stoploss_from_open, timeframe_to_minutes
 
 
 def generate_test_data(timeframe: str, size: int):
@@ -95,3 +97,38 @@ def test_merge_informative_pair_lower():
 
     with pytest.raises(ValueError, match=r"Tried to merge a faster timeframe .*"):
         merge_informative_pair(data, informative, '1h', '15m', ffill=True)
+
+
+def test_stoploss_from_open():
+    open_price_ranges = [
+        [0.01, 1.00, 30],
+        [1, 100, 30],
+        [100, 10000, 30],
+    ]
+    current_profit_range = [-0.99, 2, 30]
+    desired_stop_range = [-0.50, 0.50, 30]
+
+    for open_range in open_price_ranges:
+        for open_price in np.linspace(*open_range):
+            for desired_stop in np.linspace(*desired_stop_range):
+
+                # -1 is not a valid current_profit, should return 1
+                assert stoploss_from_open(desired_stop, -1) == 1
+
+                for current_profit in np.linspace(*current_profit_range):
+                    current_price = open_price * (1 + current_profit)
+                    expected_stop_price = open_price * (1 + desired_stop)
+
+                    stoploss = stoploss_from_open(desired_stop, current_profit)
+
+                    assert stoploss >= 0
+                    assert stoploss <= 1
+
+                    stop_price = current_price * (1 - stoploss)
+
+                    # there is no correct answer if the expected stop price is above
+                    # the current price
+                    if expected_stop_price > current_price:
+                        assert stoploss == 0
+                    else:
+                        assert isclose(stop_price, expected_stop_price, rel_tol=0.00001)

From 6597055a24f5592057f62ec330e72dda310a606c Mon Sep 17 00:00:00 2001
From: Matthias 
Date: Wed, 17 Mar 2021 19:36:11 +0100
Subject: [PATCH 147/187] Ensure ccxt tests run without dry-run

closes #4566
---
 tests/exchange/test_ccxt_compat.py | 1 +
 1 file changed, 1 insertion(+)

diff --git a/tests/exchange/test_ccxt_compat.py b/tests/exchange/test_ccxt_compat.py
index 03cb30d62..870e6cabd 100644
--- a/tests/exchange/test_ccxt_compat.py
+++ b/tests/exchange/test_ccxt_compat.py
@@ -44,6 +44,7 @@ EXCHANGES = {
 def exchange_conf():
     config = get_default_conf((Path(__file__).parent / "testdata").resolve())
     config['exchange']['pair_whitelist'] = []
+    config['dry_run'] = False
     return config
 
 

From b05de6d4687299f5012aec4eeebc0ed8dbebb173 Mon Sep 17 00:00:00 2001
From: Matthias 
Date: Wed, 17 Mar 2021 19:36:35 +0100
Subject: [PATCH 148/187] Move advanced exchange config to exchange page

---
 docs/configuration.md | 20 --------------------
 docs/exchanges.md     | 20 ++++++++++++++++++++
 2 files changed, 20 insertions(+), 20 deletions(-)

diff --git a/docs/configuration.md b/docs/configuration.md
index 2e8edca2e..ca1e03b0a 100644
--- a/docs/configuration.md
+++ b/docs/configuration.md
@@ -417,26 +417,6 @@ This configuration enables binance, as well as rate limiting to avoid bans from
     Optimal settings for rate limiting depend on the exchange and the size of the whitelist, so an ideal parameter will vary on many other settings.
     We try to provide sensible defaults per exchange where possible, if you encounter bans please make sure that `"enableRateLimit"` is enabled and increase the `"rateLimit"` parameter step by step.
 
-#### Advanced Freqtrade Exchange configuration
-
-Advanced options can be configured using the `_ft_has_params` setting, which will override Defaults and exchange-specific behaviours.
-
-Available options are listed in the exchange-class as `_ft_has_default`.
-
-For example, to test the order type `FOK` with Kraken, and modify candle limit to 200 (so you only get 200 candles per API call):
-
-```json
-"exchange": {
-    "name": "kraken",
-    "_ft_has_params": {
-        "order_time_in_force": ["gtc", "fok"],
-        "ohlcv_candle_limit": 200
-        }
-```
-
-!!! Warning
-    Please make sure to fully understand the impacts of these settings before modifying them.
-
 ### What values can be used for fiat_display_currency?
 
 The `fiat_display_currency` configuration parameter sets the base currency to use for the
diff --git a/docs/exchanges.md b/docs/exchanges.md
index 2e5bdfadd..4c7e44b06 100644
--- a/docs/exchanges.md
+++ b/docs/exchanges.md
@@ -118,3 +118,23 @@ Whether your exchange returns incomplete candles or not can be checked using [th
 Due to the danger of repainting, Freqtrade does not allow you to use this incomplete candle.
 
 However, if it is based on the need for the latest price for your strategy - then this requirement can be acquired using the [data provider](strategy-customization.md#possible-options-for-dataprovider) from within the strategy.
+
+### Advanced Freqtrade Exchange configuration
+
+Advanced options can be configured using the `_ft_has_params` setting, which will override Defaults and exchange-specific behavior.
+
+Available options are listed in the exchange-class as `_ft_has_default`.
+
+For example, to test the order type `FOK` with Kraken, and modify candle limit to 200 (so you only get 200 candles per API call):
+
+```json
+"exchange": {
+    "name": "kraken",
+    "_ft_has_params": {
+        "order_time_in_force": ["gtc", "fok"],
+        "ohlcv_candle_limit": 200
+        }
+```
+
+!!! Warning
+    Please make sure to fully understand the impacts of these settings before modifying them.

From 76ca3c219f9d7b7ccf8a2723267529acbf54f658 Mon Sep 17 00:00:00 2001
From: Matthias 
Date: Wed, 17 Mar 2021 20:43:51 +0100
Subject: [PATCH 149/187] extract result-printing from hyperopt class

---
 freqtrade/commands/hyperopt_commands.py |  21 +-
 freqtrade/optimize/hyperopt.py          | 290 +----------------------
 freqtrade/optimize/hyperopt_tools.py    | 294 ++++++++++++++++++++++++
 tests/commands/test_commands.py         |   4 +-
 tests/optimize/test_hyperopt.py         |  19 +-
 5 files changed, 324 insertions(+), 304 deletions(-)
 create mode 100644 freqtrade/optimize/hyperopt_tools.py

diff --git a/freqtrade/commands/hyperopt_commands.py b/freqtrade/commands/hyperopt_commands.py
index fd8f737f0..268e3eeef 100755
--- a/freqtrade/commands/hyperopt_commands.py
+++ b/freqtrade/commands/hyperopt_commands.py
@@ -17,7 +17,7 @@ def start_hyperopt_list(args: Dict[str, Any]) -> None:
     """
     List hyperopt epochs previously evaluated
     """
-    from freqtrade.optimize.hyperopt import Hyperopt
+    from freqtrade.optimize.hyperopt_tools import HyperoptTools
 
     config = setup_utils_configuration(args, RunMode.UTIL_NO_EXCHANGE)
 
@@ -47,7 +47,7 @@ def start_hyperopt_list(args: Dict[str, Any]) -> None:
         config.get('hyperoptexportfilename'))
 
     # Previous evaluations
-    epochs = Hyperopt.load_previous_results(results_file)
+    epochs = HyperoptTools.load_previous_results(results_file)
     total_epochs = len(epochs)
 
     epochs = hyperopt_filter_epochs(epochs, filteroptions)
@@ -57,18 +57,19 @@ def start_hyperopt_list(args: Dict[str, Any]) -> None:
 
     if not export_csv:
         try:
-            print(Hyperopt.get_result_table(config, epochs, total_epochs,
-                                            not filteroptions['only_best'], print_colorized, 0))
+            print(HyperoptTools.get_result_table(config, epochs, total_epochs,
+                                                 not filteroptions['only_best'],
+                                                 print_colorized, 0))
         except KeyboardInterrupt:
             print('User interrupted..')
 
     if epochs and not no_details:
         sorted_epochs = sorted(epochs, key=itemgetter('loss'))
         results = sorted_epochs[0]
-        Hyperopt.print_epoch_details(results, total_epochs, print_json, no_header)
+        HyperoptTools.print_epoch_details(results, total_epochs, print_json, no_header)
 
     if epochs and export_csv:
-        Hyperopt.export_csv_file(
+        HyperoptTools.export_csv_file(
             config, epochs, total_epochs, not filteroptions['only_best'], export_csv
         )
 
@@ -77,7 +78,7 @@ def start_hyperopt_show(args: Dict[str, Any]) -> None:
     """
     Show details of a hyperopt epoch previously evaluated
     """
-    from freqtrade.optimize.hyperopt import Hyperopt
+    from freqtrade.optimize.hyperopt_tools import HyperoptTools
 
     config = setup_utils_configuration(args, RunMode.UTIL_NO_EXCHANGE)
 
@@ -105,7 +106,7 @@ def start_hyperopt_show(args: Dict[str, Any]) -> None:
     }
 
     # Previous evaluations
-    epochs = Hyperopt.load_previous_results(results_file)
+    epochs = HyperoptTools.load_previous_results(results_file)
     total_epochs = len(epochs)
 
     epochs = hyperopt_filter_epochs(epochs, filteroptions)
@@ -124,8 +125,8 @@ def start_hyperopt_show(args: Dict[str, Any]) -> None:
 
     if epochs:
         val = epochs[n]
-        Hyperopt.print_epoch_details(val, total_epochs, print_json, no_header,
-                                     header_str="Epoch details")
+        HyperoptTools.print_epoch_details(val, total_epochs, print_json, no_header,
+                                          header_str="Epoch details")
 
 
 def hyperopt_filter_epochs(epochs: List, filteroptions: dict) -> List:
diff --git a/freqtrade/optimize/hyperopt.py b/freqtrade/optimize/hyperopt.py
index 6b5bc171b..03f34a511 100644
--- a/freqtrade/optimize/hyperopt.py
+++ b/freqtrade/optimize/hyperopt.py
@@ -4,36 +4,31 @@
 This module contains the hyperopt logic
 """
 
-import io
 import locale
 import logging
 import random
 import warnings
-from collections import OrderedDict
 from datetime import datetime
 from math import ceil
 from operator import itemgetter
 from pathlib import Path
-from pprint import pformat
 from typing import Any, Dict, List, Optional
 
 import progressbar
-import rapidjson
-import tabulate
 from colorama import Fore, Style
 from colorama import init as colorama_init
 from joblib import Parallel, cpu_count, delayed, dump, load, wrap_non_picklable_objects
-from pandas import DataFrame, isna, json_normalize
+from pandas import DataFrame
 
 from freqtrade.constants import DATETIME_PRINT_FORMAT, LAST_BT_RESULT_FN
 from freqtrade.data.converter import trim_dataframe
 from freqtrade.data.history import get_timerange
-from freqtrade.exceptions import OperationalException
-from freqtrade.misc import file_dump_json, plural, round_dict
+from freqtrade.misc import file_dump_json, plural
 from freqtrade.optimize.backtesting import Backtesting
 # Import IHyperOpt and IHyperOptLoss to allow unpickling classes from these modules
 from freqtrade.optimize.hyperopt_interface import IHyperOpt  # noqa: F401
 from freqtrade.optimize.hyperopt_loss_interface import IHyperOptLoss  # noqa: F401
+from freqtrade.optimize.hyperopt_tools import HyperoptTools
 from freqtrade.resolvers.hyperopt_resolver import HyperOptLossResolver, HyperOptResolver
 from freqtrade.strategy import IStrategy
 
@@ -169,15 +164,6 @@ class Hyperopt:
             file_dump_json(latest_filename, {'latest_hyperopt': str(self.results_file.name)},
                            log=False)
 
-    @staticmethod
-    def _read_results(results_file: Path) -> List:
-        """
-        Read hyperopt results from file
-        """
-        logger.info("Reading epochs from '%s'", results_file)
-        data = load(results_file)
-        return data
-
     def _get_params_details(self, params: Dict) -> Dict:
         """
         Return the params for each space
@@ -200,102 +186,16 @@ class Hyperopt:
 
         return result
 
-    @staticmethod
-    def print_epoch_details(results, total_epochs: int, print_json: bool,
-                            no_header: bool = False, header_str: str = None) -> None:
-        """
-        Display details of the hyperopt result
-        """
-        params = results.get('params_details', {})
-
-        # Default header string
-        if header_str is None:
-            header_str = "Best result"
-
-        if not no_header:
-            explanation_str = Hyperopt._format_explanation_string(results, total_epochs)
-            print(f"\n{header_str}:\n\n{explanation_str}\n")
-
-        if print_json:
-            result_dict: Dict = {}
-            for s in ['buy', 'sell', 'roi', 'stoploss', 'trailing']:
-                Hyperopt._params_update_for_json(result_dict, params, s)
-            print(rapidjson.dumps(result_dict, default=str, number_mode=rapidjson.NM_NATIVE))
-
-        else:
-            Hyperopt._params_pretty_print(params, 'buy', "Buy hyperspace params:")
-            Hyperopt._params_pretty_print(params, 'sell', "Sell hyperspace params:")
-            Hyperopt._params_pretty_print(params, 'roi', "ROI table:")
-            Hyperopt._params_pretty_print(params, 'stoploss', "Stoploss:")
-            Hyperopt._params_pretty_print(params, 'trailing', "Trailing stop:")
-
-    @staticmethod
-    def _params_update_for_json(result_dict, params, space: str) -> None:
-        if space in params:
-            space_params = Hyperopt._space_params(params, space)
-            if space in ['buy', 'sell']:
-                result_dict.setdefault('params', {}).update(space_params)
-            elif space == 'roi':
-                # TODO: get rid of OrderedDict when support for python 3.6 will be
-                # dropped (dicts keep the order as the language feature)
-
-                # Convert keys in min_roi dict to strings because
-                # rapidjson cannot dump dicts with integer keys...
-                # OrderedDict is used to keep the numeric order of the items
-                # in the dict.
-                result_dict['minimal_roi'] = OrderedDict(
-                    (str(k), v) for k, v in space_params.items()
-                )
-            else:  # 'stoploss', 'trailing'
-                result_dict.update(space_params)
-
-    @staticmethod
-    def _params_pretty_print(params, space: str, header: str) -> None:
-        if space in params:
-            space_params = Hyperopt._space_params(params, space, 5)
-            params_result = f"\n# {header}\n"
-            if space == 'stoploss':
-                params_result += f"stoploss = {space_params.get('stoploss')}"
-            elif space == 'roi':
-                # TODO: get rid of OrderedDict when support for python 3.6 will be
-                # dropped (dicts keep the order as the language feature)
-                minimal_roi_result = rapidjson.dumps(
-                    OrderedDict(
-                        (str(k), v) for k, v in space_params.items()
-                    ),
-                    default=str, indent=4, number_mode=rapidjson.NM_NATIVE)
-                params_result += f"minimal_roi = {minimal_roi_result}"
-            elif space == 'trailing':
-
-                for k, v in space_params.items():
-                    params_result += f'{k} = {v}\n'
-
-            else:
-                params_result += f"{space}_params = {pformat(space_params, indent=4)}"
-                params_result = params_result.replace("}", "\n}").replace("{", "{\n ")
-
-            params_result = params_result.replace("\n", "\n    ")
-            print(params_result)
-
-    @staticmethod
-    def _space_params(params, space: str, r: int = None) -> Dict:
-        d = params[space]
-        # Round floats to `r` digits after the decimal point if requested
-        return round_dict(d, r) if r else d
-
-    @staticmethod
-    def is_best_loss(results, current_best_loss: float) -> bool:
-        return results['loss'] < current_best_loss
-
     def print_results(self, results) -> None:
         """
         Log results if it is better than any previous evaluation
+        TODO: this should be moved to HyperoptTools too
         """
         is_best = results['is_best']
 
         if self.print_all or is_best:
             print(
-                self.get_result_table(
+                HyperoptTools.get_result_table(
                     self.config, results, self.total_epochs,
                     self.print_all, self.print_colorized,
                     self.hyperopt_table_header
@@ -303,166 +203,6 @@ class Hyperopt:
             )
             self.hyperopt_table_header = 2
 
-    @staticmethod
-    def _format_explanation_string(results, total_epochs) -> str:
-        return (("*" if results['is_initial_point'] else " ") +
-                f"{results['current_epoch']:5d}/{total_epochs}: " +
-                f"{results['results_explanation']} " +
-                f"Objective: {results['loss']:.5f}")
-
-    @staticmethod
-    def get_result_table(config: dict, results: list, total_epochs: int, highlight_best: bool,
-                         print_colorized: bool, remove_header: int) -> str:
-        """
-        Log result table
-        """
-        if not results:
-            return ''
-
-        tabulate.PRESERVE_WHITESPACE = True
-
-        trials = json_normalize(results, max_level=1)
-        trials['Best'] = ''
-        if 'results_metrics.winsdrawslosses' not in trials.columns:
-            # Ensure compatibility with older versions of hyperopt results
-            trials['results_metrics.winsdrawslosses'] = 'N/A'
-
-        trials = trials[['Best', 'current_epoch', 'results_metrics.trade_count',
-                         'results_metrics.winsdrawslosses',
-                         'results_metrics.avg_profit', 'results_metrics.total_profit',
-                         'results_metrics.profit', 'results_metrics.duration',
-                         'loss', 'is_initial_point', 'is_best']]
-        trials.columns = ['Best', 'Epoch', 'Trades', ' Win Draw Loss', 'Avg profit',
-                          'Total profit', 'Profit', 'Avg duration', 'Objective',
-                          'is_initial_point', 'is_best']
-        trials['is_profit'] = False
-        trials.loc[trials['is_initial_point'], 'Best'] = '*     '
-        trials.loc[trials['is_best'], 'Best'] = 'Best'
-        trials.loc[trials['is_initial_point'] & trials['is_best'], 'Best'] = '* Best'
-        trials.loc[trials['Total profit'] > 0, 'is_profit'] = True
-        trials['Trades'] = trials['Trades'].astype(str)
-
-        trials['Epoch'] = trials['Epoch'].apply(
-            lambda x: '{}/{}'.format(str(x).rjust(len(str(total_epochs)), ' '), total_epochs)
-        )
-        trials['Avg profit'] = trials['Avg profit'].apply(
-            lambda x: '{:,.2f}%'.format(x).rjust(7, ' ') if not isna(x) else "--".rjust(7, ' ')
-        )
-        trials['Avg duration'] = trials['Avg duration'].apply(
-            lambda x: '{:,.1f} m'.format(x).rjust(7, ' ') if not isna(x) else "--".rjust(7, ' ')
-        )
-        trials['Objective'] = trials['Objective'].apply(
-            lambda x: '{:,.5f}'.format(x).rjust(8, ' ') if x != 100000 else "N/A".rjust(8, ' ')
-        )
-
-        trials['Profit'] = trials.apply(
-            lambda x: '{:,.8f} {} {}'.format(
-                x['Total profit'], config['stake_currency'],
-                '({:,.2f}%)'.format(x['Profit']).rjust(10, ' ')
-            ).rjust(25+len(config['stake_currency']))
-            if x['Total profit'] != 0.0 else '--'.rjust(25+len(config['stake_currency'])),
-            axis=1
-        )
-        trials = trials.drop(columns=['Total profit'])
-
-        if print_colorized:
-            for i in range(len(trials)):
-                if trials.loc[i]['is_profit']:
-                    for j in range(len(trials.loc[i])-3):
-                        trials.iat[i, j] = "{}{}{}".format(Fore.GREEN,
-                                                           str(trials.loc[i][j]), Fore.RESET)
-                if trials.loc[i]['is_best'] and highlight_best:
-                    for j in range(len(trials.loc[i])-3):
-                        trials.iat[i, j] = "{}{}{}".format(Style.BRIGHT,
-                                                           str(trials.loc[i][j]), Style.RESET_ALL)
-
-        trials = trials.drop(columns=['is_initial_point', 'is_best', 'is_profit'])
-        if remove_header > 0:
-            table = tabulate.tabulate(
-                trials.to_dict(orient='list'), tablefmt='orgtbl',
-                headers='keys', stralign="right"
-            )
-
-            table = table.split("\n", remove_header)[remove_header]
-        elif remove_header < 0:
-            table = tabulate.tabulate(
-                trials.to_dict(orient='list'), tablefmt='psql',
-                headers='keys', stralign="right"
-            )
-            table = "\n".join(table.split("\n")[0:remove_header])
-        else:
-            table = tabulate.tabulate(
-                trials.to_dict(orient='list'), tablefmt='psql',
-                headers='keys', stralign="right"
-            )
-        return table
-
-    @staticmethod
-    def export_csv_file(config: dict, results: list, total_epochs: int, highlight_best: bool,
-                        csv_file: str) -> None:
-        """
-        Log result to csv-file
-        """
-        if not results:
-            return
-
-        # Verification for overwrite
-        if Path(csv_file).is_file():
-            logger.error(f"CSV file already exists: {csv_file}")
-            return
-
-        try:
-            io.open(csv_file, 'w+').close()
-        except IOError:
-            logger.error(f"Failed to create CSV file: {csv_file}")
-            return
-
-        trials = json_normalize(results, max_level=1)
-        trials['Best'] = ''
-        trials['Stake currency'] = config['stake_currency']
-
-        base_metrics = ['Best', 'current_epoch', 'results_metrics.trade_count',
-                        'results_metrics.avg_profit', 'results_metrics.median_profit',
-                        'results_metrics.total_profit',
-                        'Stake currency', 'results_metrics.profit', 'results_metrics.duration',
-                        'loss', 'is_initial_point', 'is_best']
-        param_metrics = [("params_dict."+param) for param in results[0]['params_dict'].keys()]
-        trials = trials[base_metrics + param_metrics]
-
-        base_columns = ['Best', 'Epoch', 'Trades', 'Avg profit', 'Median profit', 'Total profit',
-                        'Stake currency', 'Profit', 'Avg duration', 'Objective',
-                        'is_initial_point', 'is_best']
-        param_columns = list(results[0]['params_dict'].keys())
-        trials.columns = base_columns + param_columns
-
-        trials['is_profit'] = False
-        trials.loc[trials['is_initial_point'], 'Best'] = '*'
-        trials.loc[trials['is_best'], 'Best'] = 'Best'
-        trials.loc[trials['is_initial_point'] & trials['is_best'], 'Best'] = '* Best'
-        trials.loc[trials['Total profit'] > 0, 'is_profit'] = True
-        trials['Epoch'] = trials['Epoch'].astype(str)
-        trials['Trades'] = trials['Trades'].astype(str)
-
-        trials['Total profit'] = trials['Total profit'].apply(
-            lambda x: '{:,.8f}'.format(x) if x != 0.0 else ""
-        )
-        trials['Profit'] = trials['Profit'].apply(
-            lambda x: '{:,.2f}'.format(x) if not isna(x) else ""
-        )
-        trials['Avg profit'] = trials['Avg profit'].apply(
-            lambda x: '{:,.2f}%'.format(x) if not isna(x) else ""
-        )
-        trials['Avg duration'] = trials['Avg duration'].apply(
-            lambda x: '{:,.1f} m'.format(x) if not isna(x) else ""
-        )
-        trials['Objective'] = trials['Objective'].apply(
-            lambda x: '{:,.5f}'.format(x) if x != 100000 else ""
-        )
-
-        trials = trials.drop(columns=['is_initial_point', 'is_best', 'is_profit'])
-        trials.to_csv(csv_file, index=False, header=True, mode='w', encoding='UTF-8')
-        logger.info(f"CSV file created: {csv_file}")
-
     def has_space(self, space: str) -> bool:
         """
         Tell if the space value is contained in the configuration
@@ -626,22 +366,6 @@ class Hyperopt:
         return parallel(delayed(
                         wrap_non_picklable_objects(self.generate_optimizer))(v, i) for v in asked)
 
-    @staticmethod
-    def load_previous_results(results_file: Path) -> List:
-        """
-        Load data for epochs from the file if we have one
-        """
-        epochs: List = []
-        if results_file.is_file() and results_file.stat().st_size > 0:
-            epochs = Hyperopt._read_results(results_file)
-            # Detection of some old format, without 'is_best' field saved
-            if epochs[0].get('is_best') is None:
-                raise OperationalException(
-                    "The file with Hyperopt results is incompatible with this version "
-                    "of Freqtrade and cannot be loaded.")
-            logger.info(f"Loaded {len(epochs)} previous evaluations from disk.")
-        return epochs
-
     def _set_random_state(self, random_state: Optional[int]) -> int:
         return random_state or random.randint(1, 2**16 - 1)
 
@@ -734,7 +458,7 @@ class Hyperopt:
 
                             logger.debug(f"Optimizer epoch evaluated: {val}")
 
-                            is_best = self.is_best_loss(val, self.current_best_loss)
+                            is_best = HyperoptTools.is_best_loss(val, self.current_best_loss)
                             # This value is assigned here and not in the optimization method
                             # to keep proper order in the list of results. That's because
                             # evaluations can take different time. Here they are aligned in the
@@ -762,7 +486,7 @@ class Hyperopt:
         if self.epochs:
             sorted_epochs = sorted(self.epochs, key=itemgetter('loss'))
             best_epoch = sorted_epochs[0]
-            self.print_epoch_details(best_epoch, self.total_epochs, self.print_json)
+            HyperoptTools.print_epoch_details(best_epoch, self.total_epochs, self.print_json)
         else:
             # This is printed when Ctrl+C is pressed quickly, before first epochs have
             # a chance to be evaluated.
diff --git a/freqtrade/optimize/hyperopt_tools.py b/freqtrade/optimize/hyperopt_tools.py
new file mode 100644
index 000000000..d4c347f80
--- /dev/null
+++ b/freqtrade/optimize/hyperopt_tools.py
@@ -0,0 +1,294 @@
+
+import io
+import logging
+from collections import OrderedDict
+from pathlib import Path
+from pprint import pformat
+from typing import Dict, List
+
+import rapidjson
+import tabulate
+from colorama import Fore, Style
+from joblib import load
+from pandas import isna, json_normalize
+
+from freqtrade.exceptions import OperationalException
+from freqtrade.misc import round_dict
+
+
+logger = logging.getLogger(__name__)
+
+
+class HyperoptTools():
+
+    @staticmethod
+    def _read_results(results_file: Path) -> List:
+        """
+        Read hyperopt results from file
+        """
+        logger.info("Reading epochs from '%s'", results_file)
+        data = load(results_file)
+        return data
+
+    @staticmethod
+    def load_previous_results(results_file: Path) -> List:
+        """
+        Load data for epochs from the file if we have one
+        """
+        epochs: List = []
+        if results_file.is_file() and results_file.stat().st_size > 0:
+            epochs = HyperoptTools._read_results(results_file)
+            # Detection of some old format, without 'is_best' field saved
+            if epochs[0].get('is_best') is None:
+                raise OperationalException(
+                    "The file with HyperoptTools results is incompatible with this version "
+                    "of Freqtrade and cannot be loaded.")
+            logger.info(f"Loaded {len(epochs)} previous evaluations from disk.")
+        return epochs
+
+    @staticmethod
+    def print_epoch_details(results, total_epochs: int, print_json: bool,
+                            no_header: bool = False, header_str: str = None) -> None:
+        """
+        Display details of the hyperopt result
+        """
+        params = results.get('params_details', {})
+
+        # Default header string
+        if header_str is None:
+            header_str = "Best result"
+
+        if not no_header:
+            explanation_str = HyperoptTools._format_explanation_string(results, total_epochs)
+            print(f"\n{header_str}:\n\n{explanation_str}\n")
+
+        if print_json:
+            result_dict: Dict = {}
+            for s in ['buy', 'sell', 'roi', 'stoploss', 'trailing']:
+                HyperoptTools._params_update_for_json(result_dict, params, s)
+            print(rapidjson.dumps(result_dict, default=str, number_mode=rapidjson.NM_NATIVE))
+
+        else:
+            HyperoptTools._params_pretty_print(params, 'buy', "Buy hyperspace params:")
+            HyperoptTools._params_pretty_print(params, 'sell', "Sell hyperspace params:")
+            HyperoptTools._params_pretty_print(params, 'roi', "ROI table:")
+            HyperoptTools._params_pretty_print(params, 'stoploss', "Stoploss:")
+            HyperoptTools._params_pretty_print(params, 'trailing', "Trailing stop:")
+
+    @staticmethod
+    def _params_update_for_json(result_dict, params, space: str) -> None:
+        if space in params:
+            space_params = HyperoptTools._space_params(params, space)
+            if space in ['buy', 'sell']:
+                result_dict.setdefault('params', {}).update(space_params)
+            elif space == 'roi':
+                # TODO: get rid of OrderedDict when support for python 3.6 will be
+                # dropped (dicts keep the order as the language feature)
+
+                # Convert keys in min_roi dict to strings because
+                # rapidjson cannot dump dicts with integer keys...
+                # OrderedDict is used to keep the numeric order of the items
+                # in the dict.
+                result_dict['minimal_roi'] = OrderedDict(
+                    (str(k), v) for k, v in space_params.items()
+                )
+            else:  # 'stoploss', 'trailing'
+                result_dict.update(space_params)
+
+    @staticmethod
+    def _params_pretty_print(params, space: str, header: str) -> None:
+        if space in params:
+            space_params = HyperoptTools._space_params(params, space, 5)
+            params_result = f"\n# {header}\n"
+            if space == 'stoploss':
+                params_result += f"stoploss = {space_params.get('stoploss')}"
+            elif space == 'roi':
+                # TODO: get rid of OrderedDict when support for python 3.6 will be
+                # dropped (dicts keep the order as the language feature)
+                minimal_roi_result = rapidjson.dumps(
+                    OrderedDict(
+                        (str(k), v) for k, v in space_params.items()
+                    ),
+                    default=str, indent=4, number_mode=rapidjson.NM_NATIVE)
+                params_result += f"minimal_roi = {minimal_roi_result}"
+            elif space == 'trailing':
+
+                for k, v in space_params.items():
+                    params_result += f'{k} = {v}\n'
+
+            else:
+                params_result += f"{space}_params = {pformat(space_params, indent=4)}"
+                params_result = params_result.replace("}", "\n}").replace("{", "{\n ")
+
+            params_result = params_result.replace("\n", "\n    ")
+            print(params_result)
+
+    @staticmethod
+    def _space_params(params, space: str, r: int = None) -> Dict:
+        d = params[space]
+        # Round floats to `r` digits after the decimal point if requested
+        return round_dict(d, r) if r else d
+
+    @staticmethod
+    def is_best_loss(results, current_best_loss: float) -> bool:
+        return results['loss'] < current_best_loss
+
+    @staticmethod
+    def _format_explanation_string(results, total_epochs) -> str:
+        return (("*" if results['is_initial_point'] else " ") +
+                f"{results['current_epoch']:5d}/{total_epochs}: " +
+                f"{results['results_explanation']} " +
+                f"Objective: {results['loss']:.5f}")
+
+    @staticmethod
+    def get_result_table(config: dict, results: list, total_epochs: int, highlight_best: bool,
+                         print_colorized: bool, remove_header: int) -> str:
+        """
+        Log result table
+        """
+        if not results:
+            return ''
+
+        tabulate.PRESERVE_WHITESPACE = True
+
+        trials = json_normalize(results, max_level=1)
+        trials['Best'] = ''
+        if 'results_metrics.winsdrawslosses' not in trials.columns:
+            # Ensure compatibility with older versions of hyperopt results
+            trials['results_metrics.winsdrawslosses'] = 'N/A'
+
+        trials = trials[['Best', 'current_epoch', 'results_metrics.trade_count',
+                         'results_metrics.winsdrawslosses',
+                         'results_metrics.avg_profit', 'results_metrics.total_profit',
+                         'results_metrics.profit', 'results_metrics.duration',
+                         'loss', 'is_initial_point', 'is_best']]
+        trials.columns = ['Best', 'Epoch', 'Trades', ' Win Draw Loss', 'Avg profit',
+                          'Total profit', 'Profit', 'Avg duration', 'Objective',
+                          'is_initial_point', 'is_best']
+        trials['is_profit'] = False
+        trials.loc[trials['is_initial_point'], 'Best'] = '*     '
+        trials.loc[trials['is_best'], 'Best'] = 'Best'
+        trials.loc[trials['is_initial_point'] & trials['is_best'], 'Best'] = '* Best'
+        trials.loc[trials['Total profit'] > 0, 'is_profit'] = True
+        trials['Trades'] = trials['Trades'].astype(str)
+
+        trials['Epoch'] = trials['Epoch'].apply(
+            lambda x: '{}/{}'.format(str(x).rjust(len(str(total_epochs)), ' '), total_epochs)
+        )
+        trials['Avg profit'] = trials['Avg profit'].apply(
+            lambda x: '{:,.2f}%'.format(x).rjust(7, ' ') if not isna(x) else "--".rjust(7, ' ')
+        )
+        trials['Avg duration'] = trials['Avg duration'].apply(
+            lambda x: '{:,.1f} m'.format(x).rjust(7, ' ') if not isna(x) else "--".rjust(7, ' ')
+        )
+        trials['Objective'] = trials['Objective'].apply(
+            lambda x: '{:,.5f}'.format(x).rjust(8, ' ') if x != 100000 else "N/A".rjust(8, ' ')
+        )
+
+        trials['Profit'] = trials.apply(
+            lambda x: '{:,.8f} {} {}'.format(
+                x['Total profit'], config['stake_currency'],
+                '({:,.2f}%)'.format(x['Profit']).rjust(10, ' ')
+            ).rjust(25+len(config['stake_currency']))
+            if x['Total profit'] != 0.0 else '--'.rjust(25+len(config['stake_currency'])),
+            axis=1
+        )
+        trials = trials.drop(columns=['Total profit'])
+
+        if print_colorized:
+            for i in range(len(trials)):
+                if trials.loc[i]['is_profit']:
+                    for j in range(len(trials.loc[i])-3):
+                        trials.iat[i, j] = "{}{}{}".format(Fore.GREEN,
+                                                           str(trials.loc[i][j]), Fore.RESET)
+                if trials.loc[i]['is_best'] and highlight_best:
+                    for j in range(len(trials.loc[i])-3):
+                        trials.iat[i, j] = "{}{}{}".format(Style.BRIGHT,
+                                                           str(trials.loc[i][j]), Style.RESET_ALL)
+
+        trials = trials.drop(columns=['is_initial_point', 'is_best', 'is_profit'])
+        if remove_header > 0:
+            table = tabulate.tabulate(
+                trials.to_dict(orient='list'), tablefmt='orgtbl',
+                headers='keys', stralign="right"
+            )
+
+            table = table.split("\n", remove_header)[remove_header]
+        elif remove_header < 0:
+            table = tabulate.tabulate(
+                trials.to_dict(orient='list'), tablefmt='psql',
+                headers='keys', stralign="right"
+            )
+            table = "\n".join(table.split("\n")[0:remove_header])
+        else:
+            table = tabulate.tabulate(
+                trials.to_dict(orient='list'), tablefmt='psql',
+                headers='keys', stralign="right"
+            )
+        return table
+
+    @staticmethod
+    def export_csv_file(config: dict, results: list, total_epochs: int, highlight_best: bool,
+                        csv_file: str) -> None:
+        """
+        Log result to csv-file
+        """
+        if not results:
+            return
+
+        # Verification for overwrite
+        if Path(csv_file).is_file():
+            logger.error(f"CSV file already exists: {csv_file}")
+            return
+
+        try:
+            io.open(csv_file, 'w+').close()
+        except IOError:
+            logger.error(f"Failed to create CSV file: {csv_file}")
+            return
+
+        trials = json_normalize(results, max_level=1)
+        trials['Best'] = ''
+        trials['Stake currency'] = config['stake_currency']
+
+        base_metrics = ['Best', 'current_epoch', 'results_metrics.trade_count',
+                        'results_metrics.avg_profit', 'results_metrics.median_profit',
+                        'results_metrics.total_profit',
+                        'Stake currency', 'results_metrics.profit', 'results_metrics.duration',
+                        'loss', 'is_initial_point', 'is_best']
+        param_metrics = [("params_dict."+param) for param in results[0]['params_dict'].keys()]
+        trials = trials[base_metrics + param_metrics]
+
+        base_columns = ['Best', 'Epoch', 'Trades', 'Avg profit', 'Median profit', 'Total profit',
+                        'Stake currency', 'Profit', 'Avg duration', 'Objective',
+                        'is_initial_point', 'is_best']
+        param_columns = list(results[0]['params_dict'].keys())
+        trials.columns = base_columns + param_columns
+
+        trials['is_profit'] = False
+        trials.loc[trials['is_initial_point'], 'Best'] = '*'
+        trials.loc[trials['is_best'], 'Best'] = 'Best'
+        trials.loc[trials['is_initial_point'] & trials['is_best'], 'Best'] = '* Best'
+        trials.loc[trials['Total profit'] > 0, 'is_profit'] = True
+        trials['Epoch'] = trials['Epoch'].astype(str)
+        trials['Trades'] = trials['Trades'].astype(str)
+
+        trials['Total profit'] = trials['Total profit'].apply(
+            lambda x: '{:,.8f}'.format(x) if x != 0.0 else ""
+        )
+        trials['Profit'] = trials['Profit'].apply(
+            lambda x: '{:,.2f}'.format(x) if not isna(x) else ""
+        )
+        trials['Avg profit'] = trials['Avg profit'].apply(
+            lambda x: '{:,.2f}%'.format(x) if not isna(x) else ""
+        )
+        trials['Avg duration'] = trials['Avg duration'].apply(
+            lambda x: '{:,.1f} m'.format(x) if not isna(x) else ""
+        )
+        trials['Objective'] = trials['Objective'].apply(
+            lambda x: '{:,.5f}'.format(x) if x != 100000 else ""
+        )
+
+        trials = trials.drop(columns=['is_initial_point', 'is_best', 'is_profit'])
+        trials.to_csv(csv_file, index=False, header=True, mode='w', encoding='UTF-8')
+        logger.info(f"CSV file created: {csv_file}")
diff --git a/tests/commands/test_commands.py b/tests/commands/test_commands.py
index 27875ac94..e21ef4dd1 100644
--- a/tests/commands/test_commands.py
+++ b/tests/commands/test_commands.py
@@ -920,7 +920,7 @@ def test_start_test_pairlist(mocker, caplog, tickers, default_conf, capsys):
 
 def test_hyperopt_list(mocker, capsys, caplog, hyperopt_results):
     mocker.patch(
-        'freqtrade.optimize.hyperopt.Hyperopt.load_previous_results',
+        'freqtrade.optimize.hyperopt_tools.HyperoptTools.load_previous_results',
         MagicMock(return_value=hyperopt_results)
     )
 
@@ -1152,7 +1152,7 @@ def test_hyperopt_list(mocker, capsys, caplog, hyperopt_results):
 
 def test_hyperopt_show(mocker, capsys, hyperopt_results):
     mocker.patch(
-        'freqtrade.optimize.hyperopt.Hyperopt.load_previous_results',
+        'freqtrade.optimize.hyperopt_tools.HyperoptTools.load_previous_results',
         MagicMock(return_value=hyperopt_results)
     )
 
diff --git a/tests/optimize/test_hyperopt.py b/tests/optimize/test_hyperopt.py
index 9ebdad2b5..193d997db 100644
--- a/tests/optimize/test_hyperopt.py
+++ b/tests/optimize/test_hyperopt.py
@@ -16,6 +16,7 @@ from freqtrade.commands.optimize_commands import setup_optimize_configuration, s
 from freqtrade.data.history import load_data
 from freqtrade.exceptions import OperationalException
 from freqtrade.optimize.hyperopt import Hyperopt
+from freqtrade.optimize.hyperopt_tools import HyperoptTools
 from freqtrade.resolvers.hyperopt_resolver import HyperOptResolver
 from freqtrade.state import RunMode
 from tests.conftest import (get_args, log_has, log_has_re, patch_exchange,
@@ -336,9 +337,9 @@ def test_save_results_saves_epochs(mocker, hyperopt, testdatadir, caplog) -> Non
 
 def test_read_results_returns_epochs(mocker, hyperopt, testdatadir, caplog) -> None:
     epochs = create_results(mocker, hyperopt, testdatadir)
-    mock_load = mocker.patch('freqtrade.optimize.hyperopt.load', return_value=epochs)
+    mock_load = mocker.patch('freqtrade.optimize.hyperopt_tools.load', return_value=epochs)
     results_file = testdatadir / 'optimize' / 'ut_results.pickle'
-    hyperopt_epochs = hyperopt._read_results(results_file)
+    hyperopt_epochs = HyperoptTools._read_results(results_file)
     assert log_has(f"Reading epochs from '{results_file}'", caplog)
     assert hyperopt_epochs == epochs
     mock_load.assert_called_once()
@@ -346,7 +347,7 @@ def test_read_results_returns_epochs(mocker, hyperopt, testdatadir, caplog) -> N
 
 def test_load_previous_results(mocker, hyperopt, testdatadir, caplog) -> None:
     epochs = create_results(mocker, hyperopt, testdatadir)
-    mock_load = mocker.patch('freqtrade.optimize.hyperopt.load', return_value=epochs)
+    mock_load = mocker.patch('freqtrade.optimize.hyperopt_tools.load', return_value=epochs)
     mocker.patch.object(Path, 'is_file', MagicMock(return_value=True))
     statmock = MagicMock()
     statmock.st_size = 5
@@ -354,16 +355,16 @@ def test_load_previous_results(mocker, hyperopt, testdatadir, caplog) -> None:
 
     results_file = testdatadir / 'optimize' / 'ut_results.pickle'
 
-    hyperopt_epochs = hyperopt.load_previous_results(results_file)
+    hyperopt_epochs = HyperoptTools.load_previous_results(results_file)
 
     assert hyperopt_epochs == epochs
     mock_load.assert_called_once()
 
     del epochs[0]['is_best']
-    mock_load = mocker.patch('freqtrade.optimize.hyperopt.load', return_value=epochs)
+    mock_load = mocker.patch('freqtrade.optimize.hyperopt_tools.load', return_value=epochs)
 
     with pytest.raises(OperationalException):
-        hyperopt.load_previous_results(results_file)
+        HyperoptTools.load_previous_results(results_file)
 
 
 def test_roi_table_generation(hyperopt) -> None:
@@ -453,7 +454,7 @@ def test_format_results(hyperopt):
         'is_initial_point': True,
     }
 
-    result = hyperopt._format_explanation_string(results, 1)
+    result = HyperoptTools._format_explanation_string(results, 1)
     assert result.find(' 66.67%')
     assert result.find('Total profit 1.00000000 BTC')
     assert result.find('2.0000Σ %')
@@ -467,7 +468,7 @@ def test_format_results(hyperopt):
     df = pd.DataFrame.from_records(trades, columns=labels)
     results_metrics = hyperopt._calculate_results_metrics(df)
     results['total_profit'] = results_metrics['total_profit']
-    result = hyperopt._format_explanation_string(results, 1)
+    result = HyperoptTools._format_explanation_string(results, 1)
     assert result.find('Total profit 1.00000000 EUR')
 
 
@@ -1076,7 +1077,7 @@ def test_print_epoch_details(capsys):
         'is_best': True
     }
 
-    Hyperopt.print_epoch_details(test_result, 5, False, no_header=True)
+    HyperoptTools.print_epoch_details(test_result, 5, False, no_header=True)
     captured = capsys.readouterr()
     assert '# Trailing stop:' in captured.out
     # re.match(r"Pairs for .*", captured.out)

From 983c0ef118e5ee6a63d91478e051477300616fcf Mon Sep 17 00:00:00 2001
From: Brook Miles 
Date: Thu, 18 Mar 2021 09:47:03 +0900
Subject: [PATCH 150/187] update stoploss_from_open examples to use helper
 function

---
 docs/strategy-advanced.md | 22 +++++-----------------
 1 file changed, 5 insertions(+), 17 deletions(-)

diff --git a/docs/strategy-advanced.md b/docs/strategy-advanced.md
index cda988acd..ddf845fca 100644
--- a/docs/strategy-advanced.md
+++ b/docs/strategy-advanced.md
@@ -181,17 +181,9 @@ class AwesomeStrategy(IStrategy):
 
 #### Calculating stoploss relative to open price
 
-Stoploss values returned from `custom_stoploss` always specify a percentage relative to `current_rate`.  In order to set a stoploss relative to the *open* price, we need to use `current_profit` to calculate what percentage relative to the `current_rate` will give you the same result as if the percentage was specified from the open price.
+Stoploss values returned from `custom_stoploss()` always specify a percentage relative to `current_rate`.  In order to set a stoploss relative to the *open* price, we need to use `current_profit` to calculate what percentage relative to the `current_rate` will give you the same result as if the percentage was specified from the open price.
 
-This can be calculated as:
-
-``` python
-def stoploss_from_open(open_relative_stop: float, current_profit: float) -> float:
-    return 1-((1+open_relative_stop)/(1+current_profit))
-
-```
-
-For example, say our open price was $100, and `current_price` is $121 (`current_profit` will be `0.21`).  If we want a stop price at 7% above the open price we can call `stoploss_from_open(0.07, 0.21)` which will return `0.1157024793`.  11.57% below $121 is $107, which is the same as 7% above $100.
+The helper function [`stoploss_from_open()`](strategy-customization.md#stoploss_from_open) can be used to convert from an open price relative stop, to a current price relative stop which can be returned from `custom_stoploss()`.
 
 #### Trailing stoploss with positive offset
 
@@ -201,9 +193,7 @@ Use the initial stoploss until the profit is above 4%, then use a trailing stopl
 ``` python
 from datetime import datetime, timedelta
 from freqtrade.persistence import Trade
-
-def stoploss_from_open(open_relative_stop: float, current_profit: float) -> float:
-    return 1-((1+open_relative_stop)/(1+current_profit))
+from freqtrade.strategy import stoploss_from_open
 
 class AwesomeStrategy(IStrategy):
 
@@ -237,9 +227,7 @@ Instead of continuously trailing behind the current price, this example sets fix
 ``` python
 from datetime import datetime
 from freqtrade.persistence import Trade
-
-def stoploss_from_open(open_relative_stop: float, current_profit: float) -> float:
-    return 1-((1+open_relative_stop)/(1+current_profit))
+from freqtrade.strategy import stoploss_from_open
 
 class AwesomeStrategy(IStrategy):
 
@@ -290,7 +278,7 @@ class AwesomeStrategy(IStrategy):
             # using current_time directly (like below) will only work in backtesting.
             # so check "runmode" to make sure that it's only used in backtesting/hyperopt
             if self.dp and self.dp.runmode.value in ('backtest', 'hyperopt'):
-              relative_sl = self.custom_info[pair].loc[current_time]['atr]
+              relative_sl = self.custom_info[pair].loc[current_time]['atr']
             # in live / dry-run, it'll be really the current time
             else:
               # but we can just use the last entry from an already analyzed dataframe instead

From b6e9e74a8b3eb9c2efcf50350555783da6fe104a Mon Sep 17 00:00:00 2001
From: Matthias 
Date: Thu, 18 Mar 2021 06:46:08 +0100
Subject: [PATCH 151/187] Add link between stoploss_from_open and custom_stop
 documentation

---
 docs/strategy-advanced.md      | 4 +---
 docs/strategy-customization.md | 3 ++-
 2 files changed, 3 insertions(+), 4 deletions(-)

diff --git a/docs/strategy-advanced.md b/docs/strategy-advanced.md
index ddf845fca..4e8ecb67e 100644
--- a/docs/strategy-advanced.md
+++ b/docs/strategy-advanced.md
@@ -178,10 +178,9 @@ class AwesomeStrategy(IStrategy):
         return -0.15
 ```
 
-
 #### Calculating stoploss relative to open price
 
-Stoploss values returned from `custom_stoploss()` always specify a percentage relative to `current_rate`.  In order to set a stoploss relative to the *open* price, we need to use `current_profit` to calculate what percentage relative to the `current_rate` will give you the same result as if the percentage was specified from the open price.
+Stoploss values returned from `custom_stoploss()` always specify a percentage relative to `current_rate`. In order to set a stoploss relative to the *open* price, we need to use `current_profit` to calculate what percentage relative to the `current_rate` will give you the same result as if the percentage was specified from the open price.
 
 The helper function [`stoploss_from_open()`](strategy-customization.md#stoploss_from_open) can be used to convert from an open price relative stop, to a current price relative stop which can be returned from `custom_stoploss()`.
 
@@ -189,7 +188,6 @@ The helper function [`stoploss_from_open()`](strategy-customization.md#stoploss_
 
 Use the initial stoploss until the profit is above 4%, then use a trailing stoploss of 50% of the current profit with a minimum of 2.5% and a maximum of 5%.
 
-
 ``` python
 from datetime import datetime, timedelta
 from freqtrade.persistence import Trade
diff --git a/docs/strategy-customization.md b/docs/strategy-customization.md
index bf086bc0a..a00928a67 100644
--- a/docs/strategy-customization.md
+++ b/docs/strategy-customization.md
@@ -600,9 +600,9 @@ Stoploss values returned from `custom_stoploss` must specify a percentage relati
 
     ``` python
 
-    from freqtrade.strategy import IStrategy, stoploss_from_open
     from datetime import datetime
     from freqtrade.persistence import Trade
+    from freqtrade.strategy import IStrategy, stoploss_from_open
 
     class AwesomeStrategy(IStrategy):
 
@@ -621,6 +621,7 @@ Stoploss values returned from `custom_stoploss` must specify a percentage relati
 
     ```
 
+    Full examples can be found in the [Custom stoploss](strategy-advanced.md#custom-stoploss) section of the Documentation.
 
 
 ## Additional data (Wallets)

From bf14796d4ceb372545ab9f71844c82c6f6a97ed2 Mon Sep 17 00:00:00 2001
From: Brook Miles 
Date: Thu, 18 Mar 2021 21:50:54 +0900
Subject: [PATCH 152/187] revert "Trailing stoploss with positive offset"
 example as stoploss_from_open() wasn't adding value

---
 docs/strategy-advanced.md | 23 ++++++++++++-----------
 1 file changed, 12 insertions(+), 11 deletions(-)

diff --git a/docs/strategy-advanced.md b/docs/strategy-advanced.md
index 4e8ecb67e..962b750b5 100644
--- a/docs/strategy-advanced.md
+++ b/docs/strategy-advanced.md
@@ -178,20 +178,15 @@ class AwesomeStrategy(IStrategy):
         return -0.15
 ```
 
-#### Calculating stoploss relative to open price
-
-Stoploss values returned from `custom_stoploss()` always specify a percentage relative to `current_rate`. In order to set a stoploss relative to the *open* price, we need to use `current_profit` to calculate what percentage relative to the `current_rate` will give you the same result as if the percentage was specified from the open price.
-
-The helper function [`stoploss_from_open()`](strategy-customization.md#stoploss_from_open) can be used to convert from an open price relative stop, to a current price relative stop which can be returned from `custom_stoploss()`.
-
 #### Trailing stoploss with positive offset
 
 Use the initial stoploss until the profit is above 4%, then use a trailing stoploss of 50% of the current profit with a minimum of 2.5% and a maximum of 5%.
 
+Please note that the stoploss can only increase, values lower than the current stoploss are ignored.
+
 ``` python
 from datetime import datetime, timedelta
 from freqtrade.persistence import Trade
-from freqtrade.strategy import stoploss_from_open
 
 class AwesomeStrategy(IStrategy):
 
@@ -203,15 +198,21 @@ class AwesomeStrategy(IStrategy):
                         current_rate: float, current_profit: float, **kwargs) -> float:
 
         if current_profit < 0.04:
-            return 1 # return a value bigger than the inital stoploss to keep using the inital stoploss
+            return -1 # return a value bigger than the inital stoploss to keep using the inital stoploss
 
         # After reaching the desired offset, allow the stoploss to trail by half the profit
-        # Use a minimum of 2.5% and a maximum of 5%
-        desired_stop_from_open = max(min(current_profit / 2, 0.05), 0.025)
+        desired_stoploss = current_profit / 2
 
-        return stoploss_from_open(desired_stop_from_open, current_profit)
+        # Use a minimum of 2.5% and a maximum of 5%
+        return max(min(desired_stoploss, 0.05), 0.025
 ```
 
+#### Calculating stoploss relative to open price
+
+Stoploss values returned from `custom_stoploss()` always specify a percentage relative to `current_rate`. In order to set a stoploss relative to the *open* price, we need to use `current_profit` to calculate what percentage relative to the `current_rate` will give you the same result as if the percentage was specified from the open price.
+
+The helper function [`stoploss_from_open()`](strategy-customization.md#stoploss_from_open) can be used to convert from an open price relative stop, to a current price relative stop which can be returned from `custom_stoploss()`.
+
 #### Stepped stoploss
 
 Instead of continuously trailing behind the current price, this example sets fixed stoploss price levels based on the current profit.

From dd4d1d82d46341f7ee99b18b5f3eb7237051834e Mon Sep 17 00:00:00 2001
From: Matthias 
Date: Thu, 18 Mar 2021 14:19:33 +0100
Subject: [PATCH 153/187] Update docs/strategy-advanced.md

---
 docs/strategy-advanced.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/docs/strategy-advanced.md b/docs/strategy-advanced.md
index 962b750b5..801bc4731 100644
--- a/docs/strategy-advanced.md
+++ b/docs/strategy-advanced.md
@@ -204,7 +204,7 @@ class AwesomeStrategy(IStrategy):
         desired_stoploss = current_profit / 2
 
         # Use a minimum of 2.5% and a maximum of 5%
-        return max(min(desired_stoploss, 0.05), 0.025
+        return max(min(desired_stoploss, 0.05), 0.025)
 ```
 
 #### Calculating stoploss relative to open price

From 4d52732d30b8f55ec683de156c3ee87203398a08 Mon Sep 17 00:00:00 2001
From: Patrick Brunier 
Date: Thu, 18 Mar 2021 22:38:54 +0100
Subject: [PATCH 154/187] Added a small snippet to give users a descent error
 message,  when their start date is afer the stop date. Also updated the
 tests.

---
 freqtrade/configuration/timerange.py | 2 ++
 tests/test_timerange.py              | 3 +++
 2 files changed, 5 insertions(+)

diff --git a/freqtrade/configuration/timerange.py b/freqtrade/configuration/timerange.py
index 32bbd02a0..2075b38c6 100644
--- a/freqtrade/configuration/timerange.py
+++ b/freqtrade/configuration/timerange.py
@@ -103,5 +103,7 @@ class TimeRange:
                         stop = int(stops) // 1000
                     else:
                         stop = int(stops)
+                if start > stop > 0:
+                    raise Exception('Start date is after stop date for timerange "%s"' % text)
                 return TimeRange(stype[0], stype[1], start, stop)
         raise Exception('Incorrect syntax for timerange "%s"' % text)
diff --git a/tests/test_timerange.py b/tests/test_timerange.py
index 5c35535f0..cd10e219f 100644
--- a/tests/test_timerange.py
+++ b/tests/test_timerange.py
@@ -30,6 +30,9 @@ def test_parse_timerange_incorrect():
     with pytest.raises(Exception, match=r'Incorrect syntax.*'):
         TimeRange.parse_timerange('-')
 
+    with pytest.raises(Exception, match=r'Start date is after stop date for timerange.*'):
+        TimeRange.parse_timerange('20100523-20100522')
+
 
 def test_subtract_start():
     x = TimeRange('date', 'date', 1274486400, 1438214400)

From 0d5833ed9133ff629423143db9c810051f3abf45 Mon Sep 17 00:00:00 2001
From: Matthias 
Date: Fri, 19 Mar 2021 06:40:04 +0100
Subject: [PATCH 155/187] Use OperationalException for TimeRange errors

---
 freqtrade/configuration/timerange.py | 7 +++++--
 tests/test_timerange.py              | 6 ++++--
 2 files changed, 9 insertions(+), 4 deletions(-)

diff --git a/freqtrade/configuration/timerange.py b/freqtrade/configuration/timerange.py
index 2075b38c6..6072e296c 100644
--- a/freqtrade/configuration/timerange.py
+++ b/freqtrade/configuration/timerange.py
@@ -7,6 +7,8 @@ from typing import Optional
 
 import arrow
 
+from freqtrade.exceptions import OperationalException
+
 
 logger = logging.getLogger(__name__)
 
@@ -104,6 +106,7 @@ class TimeRange:
                     else:
                         stop = int(stops)
                 if start > stop > 0:
-                    raise Exception('Start date is after stop date for timerange "%s"' % text)
+                    raise OperationalException(
+                        f'Start date is after stop date for timerange "{text}"')
                 return TimeRange(stype[0], stype[1], start, stop)
-        raise Exception('Incorrect syntax for timerange "%s"' % text)
+        raise OperationalException(f'Incorrect syntax for timerange "{text}"')
diff --git a/tests/test_timerange.py b/tests/test_timerange.py
index cd10e219f..dcdaad09d 100644
--- a/tests/test_timerange.py
+++ b/tests/test_timerange.py
@@ -3,6 +3,7 @@ import arrow
 import pytest
 
 from freqtrade.configuration import TimeRange
+from freqtrade.exceptions import OperationalException
 
 
 def test_parse_timerange_incorrect():
@@ -27,10 +28,11 @@ def test_parse_timerange_incorrect():
     timerange = TimeRange.parse_timerange('-1231006505000')
     assert TimeRange(None, 'date', 0, 1231006505) == timerange
 
-    with pytest.raises(Exception, match=r'Incorrect syntax.*'):
+    with pytest.raises(OperationalException, match=r'Incorrect syntax.*'):
         TimeRange.parse_timerange('-')
 
-    with pytest.raises(Exception, match=r'Start date is after stop date for timerange.*'):
+    with pytest.raises(OperationalException,
+                       match=r'Start date is after stop date for timerange.*'):
         TimeRange.parse_timerange('20100523-20100522')
 
 

From c1f79922700cd51020cab850905ad6d46599a4f4 Mon Sep 17 00:00:00 2001
From: Maycon Maia Vitali 
Date: Fri, 19 Mar 2021 10:39:45 -0300
Subject: [PATCH 156/187] Added slash to fix a broken formatting

On the command table the pipe(|) broke the formatting.
---
 docs/telegram-usage.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/docs/telegram-usage.md b/docs/telegram-usage.md
index 833fae1fe..5ecdf8065 100644
--- a/docs/telegram-usage.md
+++ b/docs/telegram-usage.md
@@ -146,7 +146,7 @@ official commands. You can ask at any moment for help with `/help`.
 | `/delete ` | Delete a specific trade from the Database. Tries to close open orders. Requires manual handling of this trade on the exchange.
 | `/count` | Displays number of trades used and available
 | `/locks` | Show currently locked pairs.
-| `/unlock ` | Remove the lock for this pair (or for this lock id).
+| `/unlock ` | Remove the lock for this pair (or for this lock id).
 | `/profit` | Display a summary of your profit/loss from close trades and some stats about your performance
 | `/forcesell ` | Instantly sells the given trade  (Ignoring `minimum_roi`).
 | `/forcesell all` | Instantly sells all open trades (Ignoring `minimum_roi`).

From fb90901bb3408555331d24e1f46a0144e3afbb55 Mon Sep 17 00:00:00 2001
From: Matthias 
Date: Fri, 19 Mar 2021 20:12:12 +0100
Subject: [PATCH 157/187] Fix telegram table for both rendered and github
 markdown

---
 docs/telegram-usage.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/docs/telegram-usage.md b/docs/telegram-usage.md
index 5ecdf8065..377977892 100644
--- a/docs/telegram-usage.md
+++ b/docs/telegram-usage.md
@@ -146,7 +146,7 @@ official commands. You can ask at any moment for help with `/help`.
 | `/delete ` | Delete a specific trade from the Database. Tries to close open orders. Requires manual handling of this trade on the exchange.
 | `/count` | Displays number of trades used and available
 | `/locks` | Show currently locked pairs.
-| `/unlock ` | Remove the lock for this pair (or for this lock id).
+| `/unlock ` | Remove the lock for this pair (or for this lock id).
 | `/profit` | Display a summary of your profit/loss from close trades and some stats about your performance
 | `/forcesell ` | Instantly sells the given trade  (Ignoring `minimum_roi`).
 | `/forcesell all` | Instantly sells all open trades (Ignoring `minimum_roi`).

From 7ffe1fd36a230712a1f0eb19cc9005bf9564e231 Mon Sep 17 00:00:00 2001
From: Matthias 
Date: Sat, 20 Mar 2021 07:21:22 +0100
Subject: [PATCH 158/187] Fix calculation error for min-trade-stake

---
 docs/configuration.md           | 17 +++++++++++++++++
 freqtrade/exchange/exchange.py  |  8 ++++----
 tests/exchange/test_exchange.py | 18 +++++++++++++-----
 3 files changed, 34 insertions(+), 9 deletions(-)

diff --git a/docs/configuration.md b/docs/configuration.md
index ca1e03b0a..573cbfba2 100644
--- a/docs/configuration.md
+++ b/docs/configuration.md
@@ -156,6 +156,23 @@ Values set in the configuration file always overwrite values set in the strategy
 
 There are several methods to configure how much of the stake currency the bot will use to enter a trade. All methods respect the [available balance configuration](#available-balance) as explained below.
 
+#### Minimum trade stake
+
+The minimum stake amount will depend by exchange and pair, and is usually listed in the exchange support pages.
+Assuming the minimum tradable amount for XRP/USD is 20 XRP (given by the exchange), and the price is 0.4$.
+
+The minimum stake amount to buy this pair is therefore `20 * 0.6 ~= 12`.
+This exchange has also a limit on USD - where all orders must be > 10$ - which however does not apply in this case.
+
+To guarantee safe execution, freqtrade will not allow buying with a stake-amount of 10.1$, instead, it'll make sure that there's enough space to place a stoploss below the pair (+ an offset, defined by `amount_reserve_percent`, which defaults to 5%).
+
+With a stoploss of 10% - we'd therefore end up with a value of ~13.8$ (`12 * (1 + 0.05 + 0.1)`).
+
+To limit this calculation in case of large stoploss values, the calculated minimum stake-limit will never be more than 50% above the real limit.
+
+!!! Warning
+    Since the limits on exchanges are usually stable and are not updated often, some pairs can show pretty high minimum limits, simply because the price increased a lot since the last limit adjustment by the exchange.
+
 #### Available balance
 
 By default, the bot assumes that the `complete amount - 1%` is at it's disposal, and when using [dynamic stake amount](#dynamic-stake-amount), it will split the complete balance into `max_open_trades` buckets per trade.
diff --git a/freqtrade/exchange/exchange.py b/freqtrade/exchange/exchange.py
index fdb34eb41..6b8261afc 100644
--- a/freqtrade/exchange/exchange.py
+++ b/freqtrade/exchange/exchange.py
@@ -531,16 +531,16 @@ class Exchange:
             return None
 
         # reserve some percent defined in config (5% default) + stoploss
-        amount_reserve_percent = 1.0 - self._config.get('amount_reserve_percent',
+        amount_reserve_percent = 1.0 + self._config.get('amount_reserve_percent',
                                                         DEFAULT_AMOUNT_RESERVE_PERCENT)
-        amount_reserve_percent += stoploss
+        amount_reserve_percent += abs(stoploss)
         # it should not be more than 50%
-        amount_reserve_percent = max(amount_reserve_percent, 0.5)
+        amount_reserve_percent = max(min(amount_reserve_percent, 1.5), 1)
 
         # The value returned should satisfy both limits: for amount (base currency) and
         # for cost (quote, stake currency), so max() is used here.
         # See also #2575 at github.
-        return max(min_stake_amounts) / amount_reserve_percent
+        return max(min_stake_amounts) * amount_reserve_percent
 
     def dry_run_order(self, pair: str, ordertype: str, side: str, amount: float,
                       rate: float, params: Dict = {}) -> Dict[str, Any]:
diff --git a/tests/exchange/test_exchange.py b/tests/exchange/test_exchange.py
index 8a8c95a62..942ffd4ab 100644
--- a/tests/exchange/test_exchange.py
+++ b/tests/exchange/test_exchange.py
@@ -1,6 +1,7 @@
 import copy
 import logging
 from datetime import datetime, timedelta, timezone
+from math import isclose
 from random import randint
 from unittest.mock import MagicMock, Mock, PropertyMock, patch
 
@@ -370,7 +371,7 @@ def test_get_min_pair_stake_amount(mocker, default_conf) -> None:
         PropertyMock(return_value=markets)
     )
     result = exchange.get_min_pair_stake_amount('ETH/BTC', 1, stoploss)
-    assert result == 2 / 0.9
+    assert isclose(result, 2 * 1.1)
 
     # min amount is set
     markets["ETH/BTC"]["limits"] = {
@@ -382,7 +383,7 @@ def test_get_min_pair_stake_amount(mocker, default_conf) -> None:
         PropertyMock(return_value=markets)
     )
     result = exchange.get_min_pair_stake_amount('ETH/BTC', 2, stoploss)
-    assert result == 2 * 2 / 0.9
+    assert isclose(result, 2 * 2 * 1.1)
 
     # min amount and cost are set (cost is minimal)
     markets["ETH/BTC"]["limits"] = {
@@ -394,7 +395,7 @@ def test_get_min_pair_stake_amount(mocker, default_conf) -> None:
         PropertyMock(return_value=markets)
     )
     result = exchange.get_min_pair_stake_amount('ETH/BTC', 2, stoploss)
-    assert result == max(2, 2 * 2) / 0.9
+    assert isclose(result, max(2, 2 * 2) * 1.1)
 
     # min amount and cost are set (amount is minial)
     markets["ETH/BTC"]["limits"] = {
@@ -406,7 +407,14 @@ def test_get_min_pair_stake_amount(mocker, default_conf) -> None:
         PropertyMock(return_value=markets)
     )
     result = exchange.get_min_pair_stake_amount('ETH/BTC', 2, stoploss)
-    assert result == max(8, 2 * 2) / 0.9
+    assert isclose(result, max(8, 2 * 2) * 1.1)
+
+    result = exchange.get_min_pair_stake_amount('ETH/BTC', 2, -0.4)
+    assert isclose(result, max(8, 2 * 2) * 1.45)
+
+    # Really big stoploss
+    result = exchange.get_min_pair_stake_amount('ETH/BTC', 2, -1)
+    assert isclose(result, max(8, 2 * 2) * 1.5)
 
 
 def test_get_min_pair_stake_amount_real_data(mocker, default_conf) -> None:
@@ -424,7 +432,7 @@ def test_get_min_pair_stake_amount_real_data(mocker, default_conf) -> None:
         PropertyMock(return_value=markets)
     )
     result = exchange.get_min_pair_stake_amount('ETH/BTC', 0.020405, stoploss)
-    assert round(result, 8) == round(max(0.0001, 0.001 * 0.020405) / 0.9, 8)
+    assert round(result, 8) == round(max(0.0001, 0.001 * 0.020405) * 1.1, 8)
 
 
 def test_set_sandbox(default_conf, mocker):

From 69799532a67fc9732e851c71a500e223d4ffb589 Mon Sep 17 00:00:00 2001
From: Matthias 
Date: Sat, 20 Mar 2021 08:13:10 +0100
Subject: [PATCH 159/187] Document usage of open_date_utc

closes #4580
---
 docs/strategy-advanced.md | 18 +++++++++---------
 1 file changed, 9 insertions(+), 9 deletions(-)

diff --git a/docs/strategy-advanced.md b/docs/strategy-advanced.md
index 801bc4731..7fa824a5b 100644
--- a/docs/strategy-advanced.md
+++ b/docs/strategy-advanced.md
@@ -146,9 +146,9 @@ class AwesomeStrategy(IStrategy):
                         current_rate: float, current_profit: float, **kwargs) -> float:
 
         # Make sure you have the longest interval first - these conditions are evaluated from top to bottom.
-        if current_time - timedelta(minutes=120) > trade.open_date:
+        if current_time - timedelta(minutes=120) > trade.open_date_utc:
             return -0.05
-        elif current_time - timedelta(minutes=60) > trade.open_date:
+        elif current_time - timedelta(minutes=60) > trade.open_date_utc:
             return -0.10
         return 1
 ```
@@ -317,7 +317,7 @@ It applies a tight timeout for higher priced assets, while allowing more time to
 The function must return either `True` (cancel order) or `False` (keep order alive).
 
 ``` python
-from datetime import datetime, timedelta
+from datetime import datetime, timedelta, timezone
 from freqtrade.persistence import Trade
 
 class AwesomeStrategy(IStrategy):
@@ -331,21 +331,21 @@ class AwesomeStrategy(IStrategy):
     }
 
     def check_buy_timeout(self, pair: str, trade: 'Trade', order: dict, **kwargs) -> bool:
-        if trade.open_rate > 100 and trade.open_date < datetime.utcnow() - timedelta(minutes=5):
+        if trade.open_rate > 100 and trade.open_date_utc < datetime.now(timezone.utc) - timedelta(minutes=5):
             return True
-        elif trade.open_rate > 10 and trade.open_date < datetime.utcnow() - timedelta(minutes=3):
+        elif trade.open_rate > 10 and trade.open_date_utc < datetime.now(timezone.utc) - timedelta(minutes=3):
             return True
-        elif trade.open_rate < 1 and trade.open_date < datetime.utcnow() - timedelta(hours=24):
+        elif trade.open_rate < 1 and trade.open_date_utc < datetime.now(timezone.utc) - timedelta(hours=24):
            return True
         return False
 
 
     def check_sell_timeout(self, pair: str, trade: 'Trade', order: dict, **kwargs) -> bool:
-        if trade.open_rate > 100 and trade.open_date < datetime.utcnow() - timedelta(minutes=5):
+        if trade.open_rate > 100 and trade.open_date_utc < datetime.now(timezone.utc) - timedelta(minutes=5):
             return True
-        elif trade.open_rate > 10 and trade.open_date < datetime.utcnow() - timedelta(minutes=3):
+        elif trade.open_rate > 10 and trade.open_date_utc < datetime.now(timezone.utc) - timedelta(minutes=3):
             return True
-        elif trade.open_rate < 1 and trade.open_date < datetime.utcnow() - timedelta(hours=24):
+        elif trade.open_rate < 1 and trade.open_date_utc < datetime.now(timezone.utc) - timedelta(hours=24):
            return True
         return False
 ```

From 066dd72210889776c06726ee54bbd1ae798d1f20 Mon Sep 17 00:00:00 2001
From: Matthias 
Date: Sat, 20 Mar 2021 08:34:15 +0100
Subject: [PATCH 160/187] add orderbook structure documentation

---
 docs/strategy-customization.md | 20 ++++++++++++++++++++
 1 file changed, 20 insertions(+)

diff --git a/docs/strategy-customization.md b/docs/strategy-customization.md
index a00928a67..256b28990 100644
--- a/docs/strategy-customization.md
+++ b/docs/strategy-customization.md
@@ -436,6 +436,26 @@ if self.dp:
         dataframe['best_ask'] = ob['asks'][0][0]
 ```
 
+The orderbook structure is aligned with the order structure from [ccxt](https://github.com/ccxt/ccxt/wiki/Manual#order-book-structure), so the result will look as follows:
+
+``` js
+{
+    'bids': [
+        [ price, amount ], // [ float, float ]
+        [ price, amount ],
+        ...
+    ],
+    'asks': [
+        [ price, amount ],
+        [ price, amount ],
+        //...
+    ],
+    //...
+}
+```
+
+Therefore, using `ob['bids'][0][0]` as demonstrated above will result in using the best bid price. `ob['bids'][0][1]` would look at the amount at this orderbook position.
+
 !!! Warning "Warning about backtesting"
     The order book is not part of the historic data which means backtesting and hyperopt will not work correctly if this method is used, as the method will return uptodate values.
 

From fe7f3d9c37f70983fdb55459528378969c1f3d71 Mon Sep 17 00:00:00 2001
From: Matthias 
Date: Sat, 20 Mar 2021 11:48:39 +0100
Subject: [PATCH 161/187] Add price side validation for market orders

---
 freqtrade/configuration/config_validation.py | 14 +++++++++
 tests/test_configuration.py                  | 32 ++++++++++++++++++++
 2 files changed, 46 insertions(+)

diff --git a/freqtrade/configuration/config_validation.py b/freqtrade/configuration/config_validation.py
index df9f16f3e..b6029b6a5 100644
--- a/freqtrade/configuration/config_validation.py
+++ b/freqtrade/configuration/config_validation.py
@@ -74,6 +74,7 @@ def validate_config_consistency(conf: Dict[str, Any]) -> None:
 
     # validating trailing stoploss
     _validate_trailing_stoploss(conf)
+    _validate_price_config(conf)
     _validate_edge(conf)
     _validate_whitelist(conf)
     _validate_protections(conf)
@@ -95,6 +96,19 @@ def _validate_unlimited_amount(conf: Dict[str, Any]) -> None:
         raise OperationalException("`max_open_trades` and `stake_amount` cannot both be unlimited.")
 
 
+def _validate_price_config(conf: Dict[str, Any]) -> None:
+    """
+    When using market orders, price sides must be using the "other" side of the price
+    """
+    if (conf['order_types'].get('buy') == 'market'
+            and conf['bid_strategy'].get('price_side') != 'ask'):
+        raise OperationalException('Market buy orders require bid_strategy.price_side = "ask".')
+
+    if (conf['order_types'].get('sell') == 'market'
+            and conf['ask_strategy'].get('price_side') != 'bid'):
+        raise OperationalException('Market sell orders require ask_strategy.price_side = "bid".')
+
+
 def _validate_trailing_stoploss(conf: Dict[str, Any]) -> None:
 
     if conf.get('stoploss') == 0.0:
diff --git a/tests/test_configuration.py b/tests/test_configuration.py
index 6b3df392b..a0824e65c 100644
--- a/tests/test_configuration.py
+++ b/tests/test_configuration.py
@@ -790,6 +790,38 @@ def test_validate_max_open_trades(default_conf):
         validate_config_consistency(default_conf)
 
 
+def test_validate_price_side(default_conf):
+    default_conf['order_types'] = {
+        "buy": "limit",
+        "sell": "limit",
+        "stoploss": "limit",
+        "stoploss_on_exchange": False,
+    }
+    # Default should pass
+    validate_config_consistency(default_conf)
+
+    conf = deepcopy(default_conf)
+    conf['order_types']['buy'] = 'market'
+    with pytest.raises(OperationalException,
+                       match='Market buy orders require bid_strategy.price_side = "ask".'):
+        validate_config_consistency(conf)
+
+    conf = deepcopy(default_conf)
+    conf['order_types']['sell'] = 'market'
+    with pytest.raises(OperationalException,
+                       match='Market sell orders require ask_strategy.price_side = "bid".'):
+        validate_config_consistency(conf)
+
+    # Validate inversed case
+    conf = deepcopy(default_conf)
+    conf['order_types']['sell'] = 'market'
+    conf['order_types']['buy'] = 'market'
+    conf['ask_strategy']['price_side'] = 'bid'
+    conf['bid_strategy']['price_side'] = 'ask'
+
+    validate_config_consistency(conf)
+
+
 def test_validate_tsl(default_conf):
     default_conf['stoploss'] = 0.0
     with pytest.raises(OperationalException, match='The config stoploss needs to be different '

From 16a54b3616efa47bd394ddb660a00881d1fda989 Mon Sep 17 00:00:00 2001
From: Matthias 
Date: Sat, 20 Mar 2021 13:08:02 +0100
Subject: [PATCH 162/187] Don't require non-mandatory arguments

---
 freqtrade/configuration/config_validation.py | 8 ++++----
 tests/test_freqtradebot.py                   | 2 ++
 2 files changed, 6 insertions(+), 4 deletions(-)

diff --git a/freqtrade/configuration/config_validation.py b/freqtrade/configuration/config_validation.py
index b6029b6a5..c7e49f33d 100644
--- a/freqtrade/configuration/config_validation.py
+++ b/freqtrade/configuration/config_validation.py
@@ -100,12 +100,12 @@ def _validate_price_config(conf: Dict[str, Any]) -> None:
     """
     When using market orders, price sides must be using the "other" side of the price
     """
-    if (conf['order_types'].get('buy') == 'market'
-            and conf['bid_strategy'].get('price_side') != 'ask'):
+    if (conf.get('order_types', {}).get('buy') == 'market'
+            and conf.get('bid_strategy', {}).get('price_side') != 'ask'):
         raise OperationalException('Market buy orders require bid_strategy.price_side = "ask".')
 
-    if (conf['order_types'].get('sell') == 'market'
-            and conf['ask_strategy'].get('price_side') != 'bid'):
+    if (conf.get('order_types', {}).get('sell') == 'market'
+            and conf.get('ask_strategy', {}).get('price_side') != 'bid'):
         raise OperationalException('Market sell orders require ask_strategy.price_side = "bid".')
 
 
diff --git a/tests/test_freqtradebot.py b/tests/test_freqtradebot.py
index d7d2e19f6..5ef9960ab 100644
--- a/tests/test_freqtradebot.py
+++ b/tests/test_freqtradebot.py
@@ -94,6 +94,7 @@ def test_order_dict_dry_run(default_conf, mocker, caplog) -> None:
         'stoploss': 'limit',
         'stoploss_on_exchange': True,
     }
+    conf['bid_strategy']['price_side'] = 'ask'
 
     freqtrade = FreqtradeBot(conf)
     assert freqtrade.strategy.order_types['stoploss_on_exchange']
@@ -128,6 +129,7 @@ def test_order_dict_live(default_conf, mocker, caplog) -> None:
         'stoploss': 'limit',
         'stoploss_on_exchange': True,
     }
+    conf['bid_strategy']['price_side'] = 'ask'
 
     freqtrade = FreqtradeBot(conf)
     assert not log_has_re(".*stoploss_on_exchange .* dry-run", caplog)

From 73876b61b491b62f1337b9500eb5e926e53247e0 Mon Sep 17 00:00:00 2001
From: Matthias 
Date: Sat, 20 Mar 2021 13:33:49 +0100
Subject: [PATCH 163/187] Show potential errors when loading markets

---
 freqtrade/exchange/exchange.py  | 4 ++--
 tests/exchange/test_exchange.py | 2 +-
 2 files changed, 3 insertions(+), 3 deletions(-)

diff --git a/freqtrade/exchange/exchange.py b/freqtrade/exchange/exchange.py
index 6b8261afc..5b6e2b20d 100644
--- a/freqtrade/exchange/exchange.py
+++ b/freqtrade/exchange/exchange.py
@@ -311,8 +311,8 @@ class Exchange:
             self._markets = self._api.load_markets()
             self._load_async_markets()
             self._last_markets_refresh = arrow.utcnow().int_timestamp
-        except ccxt.BaseError as e:
-            logger.warning('Unable to initialize markets. Reason: %s', e)
+        except ccxt.BaseError:
+            logger.exception('Unable to initialize markets.')
 
     def reload_markets(self) -> None:
         """Reload markets both sync and async if refresh interval has passed """
diff --git a/tests/exchange/test_exchange.py b/tests/exchange/test_exchange.py
index 942ffd4ab..3439c7a09 100644
--- a/tests/exchange/test_exchange.py
+++ b/tests/exchange/test_exchange.py
@@ -498,7 +498,7 @@ def test__load_markets(default_conf, mocker, caplog):
     mocker.patch('freqtrade.exchange.Exchange._load_async_markets')
     mocker.patch('freqtrade.exchange.Exchange.validate_stakecurrency')
     Exchange(default_conf)
-    assert log_has('Unable to initialize markets. Reason: SomeError', caplog)
+    assert log_has('Unable to initialize markets.', caplog)
 
     expected_return = {'ETH/BTC': 'available'}
     api_mock = MagicMock()

From f4e71c1f145a47c3bd104da5f068b22df902642f Mon Sep 17 00:00:00 2001
From: Matthias 
Date: Sat, 20 Mar 2021 14:02:13 +0100
Subject: [PATCH 164/187] get_buy_rate tests should be sensible

---
 tests/test_freqtradebot.py | 24 ++++++++++++------------
 1 file changed, 12 insertions(+), 12 deletions(-)

diff --git a/tests/test_freqtradebot.py b/tests/test_freqtradebot.py
index 5ef9960ab..8f55a8fe6 100644
--- a/tests/test_freqtradebot.py
+++ b/tests/test_freqtradebot.py
@@ -839,17 +839,17 @@ def test_process_informative_pairs_added(default_conf, ticker, mocker) -> None:
     ('ask', 4, 5, None, 0.5, 4),  # last not available - uses ask
     ('ask', 4, 5, None, 1, 4),  # last not available - uses ask
     ('ask', 4, 5, None, 0, 4),  # last not available - uses ask
-    ('bid', 10, 20, 10, 0.0, 20),  # Full bid side
-    ('bid', 10, 20, 10, 1.0, 10),  # Full last side
-    ('bid', 10, 20, 10, 0.5, 15),  # Between bid and last
-    ('bid', 10, 20, 10, 0.7, 13),  # Between bid and last
-    ('bid', 10, 20, 10, 0.3, 17),  # Between bid and last
-    ('bid', 4, 5, 10, 1.0, 5),  # last bigger than bid
-    ('bid', 4, 5, 10, 0.5, 5),  # last bigger than bid
-    ('bid', 10, 20, None, 0.5, 20),  # last not available - uses bid
-    ('bid', 4, 5, None, 0.5, 5),  # last not available - uses bid
-    ('bid', 4, 5, None, 1, 5),  # last not available - uses bid
-    ('bid', 4, 5, None, 0, 5),  # last not available - uses bid
+    ('bid', 21, 20, 10, 0.0, 20),  # Full bid side
+    ('bid', 21, 20, 10, 1.0, 10),  # Full last side
+    ('bid', 21, 20, 10, 0.5, 15),  # Between bid and last
+    ('bid', 21, 20, 10, 0.7, 13),  # Between bid and last
+    ('bid', 21, 20, 10, 0.3, 17),  # Between bid and last
+    ('bid', 6, 5, 10, 1.0, 5),  # last bigger than bid
+    ('bid', 6, 5, 10, 0.5, 5),  # last bigger than bid
+    ('bid', 21, 20, None, 0.5, 20),  # last not available - uses bid
+    ('bid', 6, 5, None, 0.5, 5),  # last not available - uses bid
+    ('bid', 6, 5, None, 1, 5),  # last not available - uses bid
+    ('bid', 6, 5, None, 0, 5),  # last not available - uses bid
 ])
 def test_get_buy_rate(mocker, default_conf, caplog, side, ask, bid,
                       last, last_ab, expected) -> None:
@@ -858,7 +858,7 @@ def test_get_buy_rate(mocker, default_conf, caplog, side, ask, bid,
     default_conf['bid_strategy']['price_side'] = side
     freqtrade = get_patched_freqtradebot(mocker, default_conf)
     mocker.patch('freqtrade.exchange.Exchange.fetch_ticker',
-                 MagicMock(return_value={'ask': ask, 'last': last, 'bid': bid}))
+                 return_value={'ask': ask, 'last': last, 'bid': bid})
 
     assert freqtrade.get_buy_rate('ETH/BTC', True) == expected
     assert not log_has("Using cached buy rate for ETH/BTC.", caplog)

From 43d7f9ac67a16b10edb63b5e48a0bc23bc7e5bb5 Mon Sep 17 00:00:00 2001
From: Matthias 
Date: Sat, 20 Mar 2021 14:38:26 +0100
Subject: [PATCH 165/187] Add bid_last_balance parameter

to interpolate sell prices

closes #3270
---
 docs/configuration.md      |  3 ++-
 docs/includes/pricing.md   |  4 ++++
 freqtrade/constants.py     |  6 ++++++
 freqtrade/freqtradebot.py  | 10 ++++++++--
 tests/test_freqtradebot.py | 39 ++++++++++++++++++++++++--------------
 5 files changed, 45 insertions(+), 17 deletions(-)

diff --git a/docs/configuration.md b/docs/configuration.md
index 573cbfba2..eb3351b8f 100644
--- a/docs/configuration.md
+++ b/docs/configuration.md
@@ -62,12 +62,13 @@ Mandatory parameters are marked as **Required**, which means that they are requi
 | `unfilledtimeout.buy` | **Required.** How long (in minutes) the bot will wait for an unfilled buy order to complete, after which the order will be cancelled and repeated at current (new) price, as long as there is a signal. [Strategy Override](#parameters-in-the-strategy).
 **Datatype:** Integer
 | `unfilledtimeout.sell` | **Required.** How long (in minutes) the bot will wait for an unfilled sell order to complete, after which the order will be cancelled and repeated at current (new) price, as long as there is a signal. [Strategy Override](#parameters-in-the-strategy).
 **Datatype:** Integer
 | `bid_strategy.price_side` | Select the side of the spread the bot should look at to get the buy rate. [More information below](#buy-price-side).
 *Defaults to `bid`.* 
 **Datatype:** String (either `ask` or `bid`).
-| `bid_strategy.ask_last_balance` | **Required.** Set the bidding price. More information [below](#buy-price-without-orderbook-enabled).
+| `bid_strategy.ask_last_balance` | **Required.** Interpolate the bidding price. More information [below](#buy-price-without-orderbook-enabled).
 | `bid_strategy.use_order_book` | Enable buying using the rates in [Order Book Bids](#buy-price-with-orderbook-enabled). 
 **Datatype:** Boolean
 | `bid_strategy.order_book_top` | Bot will use the top N rate in Order Book Bids to buy. I.e. a value of 2 will allow the bot to pick the 2nd bid rate in [Order Book Bids](#buy-price-with-orderbook-enabled). 
*Defaults to `1`.*  
 **Datatype:** Positive Integer
 | `bid_strategy. check_depth_of_market.enabled` | Do not buy if the difference of buy orders and sell orders is met in Order Book. [Check market depth](#check-depth-of-market). 
*Defaults to `false`.* 
 **Datatype:** Boolean
 | `bid_strategy. check_depth_of_market.bids_to_ask_delta` | The difference ratio of buy orders and sell orders found in Order Book. A value below 1 means sell order size is greater, while value greater than 1 means buy order size is higher. [Check market depth](#check-depth-of-market) 
 *Defaults to `0`.*  
 **Datatype:** Float (as ratio)
 | `ask_strategy.price_side` | Select the side of the spread the bot should look at to get the sell rate. [More information below](#sell-price-side).
 *Defaults to `ask`.* 
 **Datatype:** String (either `ask` or `bid`).
+| `ask_strategy.bid_last_balance` | Interpolate the selling price. More information [below](#sell-price-without-orderbook-enabled).
 | `ask_strategy.use_order_book` | Enable selling of open trades using [Order Book Asks](#sell-price-with-orderbook-enabled). 
 **Datatype:** Boolean
 | `ask_strategy.order_book_min` | Bot will scan from the top min to max Order Book Asks searching for a profitable rate. 
*Defaults to `1`.* 
 **Datatype:** Positive Integer
 | `ask_strategy.order_book_max` | Bot will scan from the top min to max Order Book Asks searching for a profitable rate. 
*Defaults to `1`.* 
 **Datatype:** Positive Integer
diff --git a/docs/includes/pricing.md b/docs/includes/pricing.md
index d8a72cc58..bdf27eb20 100644
--- a/docs/includes/pricing.md
+++ b/docs/includes/pricing.md
@@ -103,6 +103,10 @@ A fixed slot (mirroring `bid_strategy.order_book_top`) can be defined by setting
 
 When not using orderbook (`ask_strategy.use_order_book=False`), the price at the `ask_strategy.price_side` side (defaults to `"ask"`) from the ticker will be used as the sell price.
 
+When not using orderbook (`ask_strategy.use_order_book=False`), Freqtrade uses the best `side` price from the ticker if it's below the `last` traded price from the ticker. Otherwise (when the `side` price is above the `last` price), it calculates a rate between `side` and `last` price.
+
+The `ask_strategy.bid_last_balance` configuration parameter controls this. A value of `0.0` will use `side` price, while `1.0` will use the last price and values between those interpolate between `side` and last price.
+
 ### Market order pricing
 
 When using market orders, prices should be configured to use the "correct" side of the orderbook to allow realistic pricing detection.
diff --git a/freqtrade/constants.py b/freqtrade/constants.py
index f25f6653d..3a2ed98e9 100644
--- a/freqtrade/constants.py
+++ b/freqtrade/constants.py
@@ -165,6 +165,12 @@ CONF_SCHEMA = {
             'type': 'object',
             'properties': {
                 'price_side': {'type': 'string', 'enum': ORDERBOOK_SIDES, 'default': 'ask'},
+                'bid_last_balance': {
+                    'type': 'number',
+                    'minimum': 0,
+                    'maximum': 1,
+                    'exclusiveMaximum': False,
+                },
                 'use_order_book': {'type': 'boolean'},
                 'order_book_min': {'type': 'integer', 'minimum': 1},
                 'order_book_max': {'type': 'integer', 'minimum': 1, 'maximum': 50},
diff --git a/freqtrade/freqtradebot.py b/freqtrade/freqtradebot.py
index c60d65f72..73f4c91be 100644
--- a/freqtrade/freqtradebot.py
+++ b/freqtrade/freqtradebot.py
@@ -432,7 +432,7 @@ class FreqtradeBot(LoggingMixin):
             ticker = self.exchange.fetch_ticker(pair)
             ticker_rate = ticker[bid_strategy['price_side']]
             if ticker['last'] and ticker_rate > ticker['last']:
-                balance = self.config['bid_strategy']['ask_last_balance']
+                balance = bid_strategy['ask_last_balance']
                 ticker_rate = ticker_rate + balance * (ticker['last'] - ticker_rate)
             used_rate = ticker_rate
 
@@ -745,7 +745,13 @@ class FreqtradeBot(LoggingMixin):
                 logger.warning("Sell Price at location from orderbook could not be determined.")
                 raise PricingError from e
         else:
-            rate = self.exchange.fetch_ticker(pair)[ask_strategy['price_side']]
+            ticker = self.exchange.fetch_ticker(pair)
+            ticker_rate = ticker[ask_strategy['price_side']]
+            if ticker['last'] and ticker_rate < ticker['last']:
+                balance = ask_strategy.get('bid_last_balance', 0.0)
+                ticker_rate = ticker_rate - balance * (ticker_rate - ticker['last'])
+            rate = ticker_rate
+
         if rate is None:
             raise PricingError(f"Sell-Rate for {pair} was empty.")
         self._sell_rate_cache[pair] = rate
diff --git a/tests/test_freqtradebot.py b/tests/test_freqtradebot.py
index 8f55a8fe6..c1a17164f 100644
--- a/tests/test_freqtradebot.py
+++ b/tests/test_freqtradebot.py
@@ -4112,22 +4112,33 @@ def test_order_book_ask_strategy(default_conf, limit_buy_order_open, limit_buy_o
     assert log_has('Sell Price at location 1 from orderbook could not be determined.', caplog)
 
 
-@pytest.mark.parametrize('side,ask,bid,expected', [
-    ('bid', 10.0, 11.0, 11.0),
-    ('bid', 10.0, 11.2, 11.2),
-    ('bid', 10.0, 11.0, 11.0),
-    ('bid', 9.8, 11.0, 11.0),
-    ('bid', 0.0001, 0.002, 0.002),
-    ('ask', 10.0, 11.0, 10.0),
-    ('ask', 10.11, 11.2, 10.11),
-    ('ask', 0.001, 0.002, 0.001),
-    ('ask', 0.006, 1.0, 0.006),
+@pytest.mark.parametrize('side,ask,bid,last,last_ab,expected', [
+    ('bid', 12.0, 11.0, 11.5, 0.0, 11.0),  # full bid side
+    ('bid', 12.0, 11.0, 11.5, 1.0, 11.5),  # full last side
+    ('bid', 12.0, 11.0, 11.5, 0.5, 11.25),  # between bid and lat
+    ('bid', 12.0, 11.2, 10.5, 0.0, 11.2),  # Last smaller than bid
+    ('bid', 12.0, 11.2, 10.5, 1.0, 11.2),  # Last smaller than bid - uses bid
+    ('bid', 12.0, 11.2, 10.5, 0.5, 11.2),  # Last smaller than bid - uses bid
+    ('bid', 0.003, 0.002, 0.005, 0.0, 0.002),
+    ('ask', 12.0, 11.0, 12.5, 0.0, 12.0),  # full ask side
+    ('ask', 12.0, 11.0, 12.5, 1.0, 12.5),  # full last side
+    ('ask', 12.0, 11.0, 12.5, 0.5, 12.25),  # between bid and lat
+    ('ask', 12.2, 11.2, 10.5, 0.0, 12.2),  # Last smaller than ask
+    ('ask', 12.0, 11.0, 10.5, 1.0, 12.0),  # Last smaller than ask - uses ask
+    ('ask', 12.0, 11.2, 10.5, 0.5, 12.0),  # Last smaller than ask - uses ask
+    ('ask', 10.0, 11.0, 11.0, 0.0, 10.0),
+    ('ask', 10.11, 11.2, 11.0, 0.0, 10.11),
+    ('ask', 0.001, 0.002, 11.0, 0.0, 0.001),
+    ('ask', 0.006, 1.0, 11.0, 0.0, 0.006),
 ])
-def test_get_sell_rate(default_conf, mocker, caplog, side, bid, ask, expected) -> None:
+def test_get_sell_rate(default_conf, mocker, caplog, side, bid, ask,
+                       last, last_ab, expected) -> None:
     caplog.set_level(logging.DEBUG)
 
     default_conf['ask_strategy']['price_side'] = side
-    mocker.patch('freqtrade.exchange.Exchange.fetch_ticker', return_value={'ask': ask, 'bid': bid})
+    default_conf['ask_strategy']['bid_last_balance'] = last_ab
+    mocker.patch('freqtrade.exchange.Exchange.fetch_ticker',
+                 return_value={'ask': ask, 'bid': bid, 'last': last})
     pair = "ETH/BTC"
 
     # Test regular mode
@@ -4186,7 +4197,7 @@ def test_get_sell_rate_exception(default_conf, mocker, caplog):
     default_conf['ask_strategy']['price_side'] = 'ask'
     pair = "ETH/BTC"
     mocker.patch('freqtrade.exchange.Exchange.fetch_ticker',
-                 return_value={'ask': None, 'bid': 0.12})
+                 return_value={'ask': None, 'bid': 0.12, 'last': None})
     ft = get_patched_freqtradebot(mocker, default_conf)
     with pytest.raises(PricingError, match=r"Sell-Rate for ETH/BTC was empty."):
         ft.get_sell_rate(pair, True)
@@ -4195,7 +4206,7 @@ def test_get_sell_rate_exception(default_conf, mocker, caplog):
     assert ft.get_sell_rate(pair, True) == 0.12
     # Reverse sides
     mocker.patch('freqtrade.exchange.Exchange.fetch_ticker',
-                 return_value={'ask': 0.13, 'bid': None})
+                 return_value={'ask': 0.13, 'bid': None, 'last': None})
     with pytest.raises(PricingError, match=r"Sell-Rate for ETH/BTC was empty."):
         ft.get_sell_rate(pair, True)
 

From e315a6a0da4e9e0232f3463f12cff7561a64c32d Mon Sep 17 00:00:00 2001
From: Matthias 
Date: Sat, 20 Mar 2021 14:58:51 +0100
Subject: [PATCH 166/187] assume "last" can miss from a ticker response

closes #4573
---
 freqtrade/plugins/pairlist/PriceFilter.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/freqtrade/plugins/pairlist/PriceFilter.py b/freqtrade/plugins/pairlist/PriceFilter.py
index 6558f196f..a0579b196 100644
--- a/freqtrade/plugins/pairlist/PriceFilter.py
+++ b/freqtrade/plugins/pairlist/PriceFilter.py
@@ -64,7 +64,7 @@ class PriceFilter(IPairList):
         :param ticker: ticker dict as returned from ccxt.load_markets()
         :return: True if the pair can stay, false if it should be removed
         """
-        if ticker['last'] is None or ticker['last'] == 0:
+        if ticker.get('last', None) is None or ticker.get('last') == 0:
             self.log_once(f"Removed {pair} from whitelist, because "
                           "ticker['last'] is empty (Usually no trade in the last 24h).",
                           logger.info)

From ac7a1305cbb78d588bbb1d0849370828b17219fd Mon Sep 17 00:00:00 2001
From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com>
Date: Mon, 22 Mar 2021 05:25:11 +0000
Subject: [PATCH 167/187] Bump ccxt from 1.43.27 to 1.43.89

Bumps [ccxt](https://github.com/ccxt/ccxt) from 1.43.27 to 1.43.89.
- [Release notes](https://github.com/ccxt/ccxt/releases)
- [Changelog](https://github.com/ccxt/ccxt/blob/master/doc/exchanges-by-country.rst)
- [Commits](https://github.com/ccxt/ccxt/compare/1.43.27...1.43.89)

Signed-off-by: dependabot[bot] 
---
 requirements.txt | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/requirements.txt b/requirements.txt
index 08f5b9078..7c8841e67 100644
--- a/requirements.txt
+++ b/requirements.txt
@@ -1,7 +1,7 @@
 numpy==1.20.1
 pandas==1.2.3
 
-ccxt==1.43.27
+ccxt==1.43.89
 # Pin cryptography for now due to rust build errors with piwheels
 cryptography==3.4.6
 aiohttp==3.7.4.post0

From 9612ba34ed0c97e97d681c4726c13bc2f5d69048 Mon Sep 17 00:00:00 2001
From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com>
Date: Mon, 22 Mar 2021 05:25:17 +0000
Subject: [PATCH 168/187] Bump urllib3 from 1.26.3 to 1.26.4

Bumps [urllib3](https://github.com/urllib3/urllib3) from 1.26.3 to 1.26.4.
- [Release notes](https://github.com/urllib3/urllib3/releases)
- [Changelog](https://github.com/urllib3/urllib3/blob/main/CHANGES.rst)
- [Commits](https://github.com/urllib3/urllib3/compare/1.26.3...1.26.4)

Signed-off-by: dependabot[bot] 
---
 requirements.txt | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/requirements.txt b/requirements.txt
index 08f5b9078..0554f326b 100644
--- a/requirements.txt
+++ b/requirements.txt
@@ -10,7 +10,7 @@ python-telegram-bot==13.4.1
 arrow==1.0.3
 cachetools==4.2.1
 requests==2.25.1
-urllib3==1.26.3
+urllib3==1.26.4
 wrapt==1.12.1
 jsonschema==3.2.0
 TA-Lib==0.4.19

From 09c7ee9e923b936cd8e236b7253c08ec4d58a309 Mon Sep 17 00:00:00 2001
From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com>
Date: Mon, 22 Mar 2021 05:25:28 +0000
Subject: [PATCH 169/187] Bump isort from 5.7.0 to 5.8.0

Bumps [isort](https://github.com/pycqa/isort) from 5.7.0 to 5.8.0.
- [Release notes](https://github.com/pycqa/isort/releases)
- [Changelog](https://github.com/PyCQA/isort/blob/develop/CHANGELOG.md)
- [Commits](https://github.com/pycqa/isort/compare/5.7.0...5.8.0)

Signed-off-by: dependabot[bot] 
---
 requirements-dev.txt | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/requirements-dev.txt b/requirements-dev.txt
index 4f0ea7706..02f7fbca8 100644
--- a/requirements-dev.txt
+++ b/requirements-dev.txt
@@ -13,7 +13,7 @@ pytest-asyncio==0.14.0
 pytest-cov==2.11.1
 pytest-mock==3.5.1
 pytest-random-order==1.0.4
-isort==5.7.0
+isort==5.8.0
 
 # Convert jupyter notebooks to markdown documents
 nbconvert==6.0.7

From ea3012e94d78c5119ba3b457a4799f2b714aafee Mon Sep 17 00:00:00 2001
From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com>
Date: Mon, 22 Mar 2021 05:25:35 +0000
Subject: [PATCH 170/187] Bump sqlalchemy from 1.3.23 to 1.4.2

Bumps [sqlalchemy](https://github.com/sqlalchemy/sqlalchemy) from 1.3.23 to 1.4.2.
- [Release notes](https://github.com/sqlalchemy/sqlalchemy/releases)
- [Changelog](https://github.com/sqlalchemy/sqlalchemy/blob/master/CHANGES)
- [Commits](https://github.com/sqlalchemy/sqlalchemy/commits)

Signed-off-by: dependabot[bot] 
---
 requirements.txt | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/requirements.txt b/requirements.txt
index 08f5b9078..c714a8f1a 100644
--- a/requirements.txt
+++ b/requirements.txt
@@ -5,7 +5,7 @@ ccxt==1.43.27
 # Pin cryptography for now due to rust build errors with piwheels
 cryptography==3.4.6
 aiohttp==3.7.4.post0
-SQLAlchemy==1.3.23
+SQLAlchemy==1.4.2
 python-telegram-bot==13.4.1
 arrow==1.0.3
 cachetools==4.2.1

From e39cff522d33e9a529045ec58610e87362a37328 Mon Sep 17 00:00:00 2001
From: Matthias 
Date: Mon, 22 Mar 2021 17:30:16 +0100
Subject: [PATCH 171/187] Remove duplicate dict keys in test

---
 tests/rpc/test_rpc_apiserver.py | 2 --
 1 file changed, 2 deletions(-)

diff --git a/tests/rpc/test_rpc_apiserver.py b/tests/rpc/test_rpc_apiserver.py
index 01492b4f2..5a0a04943 100644
--- a/tests/rpc/test_rpc_apiserver.py
+++ b/tests/rpc/test_rpc_apiserver.py
@@ -810,14 +810,12 @@ def test_api_status(botclient, mocker, ticker, fee, markets):
         'stoploss_entry_dist_ratio': -0.10448878,
         'trade_id': 1,
         'close_rate_requested': None,
-        'current_rate': 1.099e-05,
         'fee_close': 0.0025,
         'fee_close_cost': None,
         'fee_close_currency': None,
         'fee_open': 0.0025,
         'fee_open_cost': None,
         'fee_open_currency': None,
-        'open_date': ANY,
         'is_open': True,
         'max_rate': 1.099e-05,
         'min_rate': 1.098e-05,

From b7702a1e9f121764605144d63c2480a2b82b08cd Mon Sep 17 00:00:00 2001
From: Matthias 
Date: Mon, 22 Mar 2021 19:39:06 +0100
Subject: [PATCH 172/187] Improve tests to work with new sqlalchemy version

---
 tests/test_freqtradebot.py |  2 +-
 tests/test_persistence.py  | 27 +++++++++++++++------------
 2 files changed, 16 insertions(+), 13 deletions(-)

diff --git a/tests/test_freqtradebot.py b/tests/test_freqtradebot.py
index c1a17164f..486c31090 100644
--- a/tests/test_freqtradebot.py
+++ b/tests/test_freqtradebot.py
@@ -2798,7 +2798,7 @@ def test_execute_sell_sloe_cancel_exception(mocker, default_conf, ticker, fee, c
     mocker.patch('freqtrade.exchange.Exchange.cancel_stoploss_order',
                  side_effect=InvalidOrderException())
     mocker.patch('freqtrade.wallets.Wallets.get_free', MagicMock(return_value=300))
-    sellmock = MagicMock()
+    sellmock = MagicMock(return_value={'id': '12345555'})
     patch_exchange(mocker)
     mocker.patch.multiple(
         'freqtrade.exchange.Exchange',
diff --git a/tests/test_persistence.py b/tests/test_persistence.py
index 1820250a5..6a388327c 100644
--- a/tests/test_persistence.py
+++ b/tests/test_persistence.py
@@ -1,6 +1,7 @@
 # pragma pylint: disable=missing-docstring, C0103
 import logging
 from datetime import datetime, timedelta, timezone
+from pathlib import Path
 from types import FunctionType
 from unittest.mock import MagicMock
 
@@ -21,14 +22,15 @@ def test_init_create_session(default_conf):
     assert 'scoped_session' in type(Trade.session).__name__
 
 
-def test_init_custom_db_url(default_conf, mocker):
+def test_init_custom_db_url(default_conf, tmpdir):
     # Update path to a value other than default, but still in-memory
-    default_conf.update({'db_url': 'sqlite:///tmp/freqtrade2_test.sqlite'})
-    create_engine_mock = mocker.patch('freqtrade.persistence.models.create_engine', MagicMock())
+    filename = f"{tmpdir}/freqtrade2_test.sqlite"
+    assert not Path(filename).is_file()
+
+    default_conf.update({'db_url': f'sqlite:///{filename}'})
 
     init_db(default_conf['db_url'], default_conf['dry_run'])
-    assert create_engine_mock.call_count == 1
-    assert create_engine_mock.mock_calls[0][1][0] == 'sqlite:///tmp/freqtrade2_test.sqlite'
+    assert Path(filename).is_file()
 
 
 def test_init_invalid_db_url(default_conf):
@@ -49,15 +51,16 @@ def test_init_prod_db(default_conf, mocker):
     assert create_engine_mock.mock_calls[0][1][0] == 'sqlite:///tradesv3.sqlite'
 
 
-def test_init_dryrun_db(default_conf, mocker):
-    default_conf.update({'dry_run': True})
-    default_conf.update({'db_url': constants.DEFAULT_DB_DRYRUN_URL})
-
-    create_engine_mock = mocker.patch('freqtrade.persistence.models.create_engine', MagicMock())
+def test_init_dryrun_db(default_conf, tmpdir):
+    filename = f"{tmpdir}/freqtrade2_prod.sqlite"
+    assert not Path(filename).is_file()
+    default_conf.update({
+        'dry_run': True,
+        'db_url': f'sqlite:///{filename}'
+    })
 
     init_db(default_conf['db_url'], default_conf['dry_run'])
-    assert create_engine_mock.call_count == 1
-    assert create_engine_mock.mock_calls[0][1][0] == 'sqlite:///tradesv3.dryrun.sqlite'
+    assert Path(filename).is_file()
 
 
 @pytest.mark.usefixtures("init_persistence")

From 4e8999ade3e8e9d39a5e78d175b736ed6d6b7dc1 Mon Sep 17 00:00:00 2001
From: Erwin Hoeckx 
Date: Mon, 22 Mar 2021 20:40:11 +0100
Subject: [PATCH 173/187] Changed the code for status table a bit so that it
 splits up the trades per 50 trades, to make sure it can be sent regardless of
 number of trades

Signed-off-by: Erwin Hoeckx 
---
 freqtrade/rpc/telegram.py | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)

diff --git a/freqtrade/rpc/telegram.py b/freqtrade/rpc/telegram.py
index 7ec67e5d0..2d753db70 100644
--- a/freqtrade/rpc/telegram.py
+++ b/freqtrade/rpc/telegram.py
@@ -330,8 +330,12 @@ class Telegram(RPCHandler):
             statlist, head = self._rpc._rpc_status_table(
                 self._config['stake_currency'], self._config.get('fiat_display_currency', ''))
 
-            message = tabulate(statlist, headers=head, tablefmt='simple')
-            self._send_msg(f"
{message}
", parse_mode=ParseMode.HTML)
+            max_trades_per_msg = 50
+            for i in range(0, max(int(len(statlist) / max_trades_per_msg), 1)):
+                message = tabulate(statlist[i * max_trades_per_msg:(i + 1) * max_trades_per_msg],
+                                   headers=head,
+                                   tablefmt='simple')
+                self._send_msg(f"
{message}
", parse_mode=ParseMode.HTML)
         except RPCException as e:
             self._send_msg(str(e))
 

From 6856963aef37303b40fd29a0745e22ff312dc11c Mon Sep 17 00:00:00 2001
From: rextea 
Date: Tue, 23 Mar 2021 10:09:41 +0200
Subject: [PATCH 174/187] Add confirm_trade_exit and confirm_trade_entry to
 backtesting

---
 freqtrade/optimize/backtesting.py | 21 +++++++++++++++++++--
 1 file changed, 19 insertions(+), 2 deletions(-)

diff --git a/freqtrade/optimize/backtesting.py b/freqtrade/optimize/backtesting.py
index 0b884dae5..1dcf6428b 100644
--- a/freqtrade/optimize/backtesting.py
+++ b/freqtrade/optimize/backtesting.py
@@ -30,7 +30,6 @@ from freqtrade.strategy.interface import IStrategy, SellCheckTuple, SellType
 from freqtrade.strategy.strategy_wrapper import strategy_safe_wrapper
 from freqtrade.wallets import Wallets
 
-
 logger = logging.getLogger(__name__)
 
 # Indexes for backtest tuples
@@ -252,8 +251,17 @@ class Backtesting:
         sell = self.strategy.should_sell(trade, sell_row[OPEN_IDX],  # type: ignore
                                          sell_row[DATE_IDX], sell_row[BUY_IDX], sell_row[SELL_IDX],
                                          low=sell_row[LOW_IDX], high=sell_row[HIGH_IDX])
-        if sell.sell_flag:
 
+        time_in_force = self.strategy.order_time_in_force['sell']
+
+        # confirm_trade_exit
+        if not strategy_safe_wrapper(self.strategy.confirm_trade_exit, default_retval=False)(
+                pair=trade.pair, trade=trade, order_type='limit', amount=trade.amount, rate=sell_row[LOW_IDX],
+                time_in_force=time_in_force,
+                sell_reason=sell.sell_type.value):
+            return None
+
+        if sell.sell_flag:
             trade.close_date = sell_row[DATE_IDX]
             trade.sell_reason = sell.sell_type.value
             trade_dur = int((trade.close_date_utc - trade.open_date_utc).total_seconds() // 60)
@@ -271,6 +279,15 @@ class Backtesting:
         except DependencyException:
             return None
         min_stake_amount = self.exchange.get_min_pair_stake_amount(pair, row[OPEN_IDX], -0.05)
+
+        order_type = self.strategy.order_types['buy']
+        time_in_force = self.strategy.order_time_in_force['sell']
+        # confirm_trade_entry
+        if not strategy_safe_wrapper(self.strategy.confirm_trade_entry, default_retval=True)(
+                pair=pair, order_type=order_type, amount=stake_amount, rate=row[OPEN_IDX],
+                time_in_force=time_in_force):
+            return None
+
         if stake_amount and (not min_stake_amount or stake_amount > min_stake_amount):
             # Enter trade
             trade = LocalTrade(

From eb5d69dcd486305d3448caf3eab1152dbd9cf59f Mon Sep 17 00:00:00 2001
From: rextea 
Date: Tue, 23 Mar 2021 10:12:08 +0200
Subject: [PATCH 175/187] Add confirm_trade_exit and confirm_trade_entry to
 backtesting

---
 freqtrade/optimize/backtesting.py | 1 -
 1 file changed, 1 deletion(-)

diff --git a/freqtrade/optimize/backtesting.py b/freqtrade/optimize/backtesting.py
index 1dcf6428b..080e6b1a2 100644
--- a/freqtrade/optimize/backtesting.py
+++ b/freqtrade/optimize/backtesting.py
@@ -253,7 +253,6 @@ class Backtesting:
                                          low=sell_row[LOW_IDX], high=sell_row[HIGH_IDX])
 
         time_in_force = self.strategy.order_time_in_force['sell']
-
         # confirm_trade_exit
         if not strategy_safe_wrapper(self.strategy.confirm_trade_exit, default_retval=False)(
                 pair=trade.pair, trade=trade, order_type='limit', amount=trade.amount, rate=sell_row[LOW_IDX],

From dc4ea604dd6d1fc8bd6af7a68e5fba36f3e7c517 Mon Sep 17 00:00:00 2001
From: rextea 
Date: Tue, 23 Mar 2021 10:19:16 +0200
Subject: [PATCH 176/187] Add confirm_trade_exit and confirm_trade_entry to
 backtesting

---
 freqtrade/optimize/backtesting.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/freqtrade/optimize/backtesting.py b/freqtrade/optimize/backtesting.py
index 080e6b1a2..321b60ecd 100644
--- a/freqtrade/optimize/backtesting.py
+++ b/freqtrade/optimize/backtesting.py
@@ -254,7 +254,7 @@ class Backtesting:
 
         time_in_force = self.strategy.order_time_in_force['sell']
         # confirm_trade_exit
-        if not strategy_safe_wrapper(self.strategy.confirm_trade_exit, default_retval=False)(
+        if not strategy_safe_wrapper(self.strategy.confirm_trade_exit, default_retval=True)(
                 pair=trade.pair, trade=trade, order_type='limit', amount=trade.amount, rate=sell_row[LOW_IDX],
                 time_in_force=time_in_force,
                 sell_reason=sell.sell_type.value):

From f51f4b1817efd2451d4877478312f22d9acabefe Mon Sep 17 00:00:00 2001
From: rextea 
Date: Tue, 23 Mar 2021 10:35:46 +0200
Subject: [PATCH 177/187] Add confirm_trade_exit and confirm_trade_entry to
 backtesting

---
 freqtrade/optimize/backtesting.py | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/freqtrade/optimize/backtesting.py b/freqtrade/optimize/backtesting.py
index 321b60ecd..d858acc1a 100644
--- a/freqtrade/optimize/backtesting.py
+++ b/freqtrade/optimize/backtesting.py
@@ -255,7 +255,8 @@ class Backtesting:
         time_in_force = self.strategy.order_time_in_force['sell']
         # confirm_trade_exit
         if not strategy_safe_wrapper(self.strategy.confirm_trade_exit, default_retval=True)(
-                pair=trade.pair, trade=trade, order_type='limit', amount=trade.amount, rate=sell_row[LOW_IDX],
+                pair=trade.pair, trade=trade, order_type='limit', amount=trade.amount,
+                rate=sell_row[LOW_IDX],
                 time_in_force=time_in_force,
                 sell_reason=sell.sell_type.value):
             return None

From d5301b4d6316f822d387ae6d2947cec5a49e316f Mon Sep 17 00:00:00 2001
From: Matthias 
Date: Tue, 23 Mar 2021 10:53:09 +0100
Subject: [PATCH 178/187] RateLimit should be enabled by default

---
 config_full.json.example | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/config_full.json.example b/config_full.json.example
index 8366774c4..717797933 100644
--- a/config_full.json.example
+++ b/config_full.json.example
@@ -113,7 +113,7 @@
         "password": "",
         "ccxt_config": {"enableRateLimit": true},
         "ccxt_async_config": {
-            "enableRateLimit": false,
+            "enableRateLimit": true,
             "rateLimit": 500,
             "aiohttp_trust_env": false
         },

From c928cd38dc2e5a67fcf70d7a76d350f5b7549560 Mon Sep 17 00:00:00 2001
From: Erwin Hoeckx 
Date: Tue, 23 Mar 2021 16:45:42 +0100
Subject: [PATCH 179/187] Small bugfix to make sure it shows all the trades

Signed-off-by: Erwin Hoeckx 
---
 freqtrade/rpc/telegram.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/freqtrade/rpc/telegram.py b/freqtrade/rpc/telegram.py
index 2d753db70..b83cbf1a2 100644
--- a/freqtrade/rpc/telegram.py
+++ b/freqtrade/rpc/telegram.py
@@ -331,7 +331,7 @@ class Telegram(RPCHandler):
                 self._config['stake_currency'], self._config.get('fiat_display_currency', ''))
 
             max_trades_per_msg = 50
-            for i in range(0, max(int(len(statlist) / max_trades_per_msg), 1)):
+            for i in range(0, max(int(len(statlist) / max_trades_per_msg) + 1, 1)):
                 message = tabulate(statlist[i * max_trades_per_msg:(i + 1) * max_trades_per_msg],
                                    headers=head,
                                    tablefmt='simple')

From 65a9763fa587aa42823956cd4e207f08905e7906 Mon Sep 17 00:00:00 2001
From: Erwin Hoeckx 
Date: Tue, 23 Mar 2021 16:54:38 +0100
Subject: [PATCH 180/187] Fixed an issue when there were exactly 50 trades, it
 was sending an extra empty table

Signed-off-by: Erwin Hoeckx 
---
 freqtrade/rpc/telegram.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/freqtrade/rpc/telegram.py b/freqtrade/rpc/telegram.py
index b83cbf1a2..61a0188cb 100644
--- a/freqtrade/rpc/telegram.py
+++ b/freqtrade/rpc/telegram.py
@@ -331,7 +331,7 @@ class Telegram(RPCHandler):
                 self._config['stake_currency'], self._config.get('fiat_display_currency', ''))
 
             max_trades_per_msg = 50
-            for i in range(0, max(int(len(statlist) / max_trades_per_msg) + 1, 1)):
+            for i in range(0, max(int(len(statlist) / max_trades_per_msg + 0.99), 1)):
                 message = tabulate(statlist[i * max_trades_per_msg:(i + 1) * max_trades_per_msg],
                                    headers=head,
                                    tablefmt='simple')

From 2fd510e6e4e8a2108f2a64c6ccad72f83fb047d7 Mon Sep 17 00:00:00 2001
From: Erwin Hoeckx 
Date: Tue, 23 Mar 2021 21:52:46 +0100
Subject: [PATCH 181/187] Added comment with an example calculation

Signed-off-by: Erwin Hoeckx 
---
 freqtrade/rpc/telegram.py | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/freqtrade/rpc/telegram.py b/freqtrade/rpc/telegram.py
index 61a0188cb..2063a4f58 100644
--- a/freqtrade/rpc/telegram.py
+++ b/freqtrade/rpc/telegram.py
@@ -331,6 +331,11 @@ class Telegram(RPCHandler):
                 self._config['stake_currency'], self._config.get('fiat_display_currency', ''))
 
             max_trades_per_msg = 50
+            """
+            Calculate the number of messages of 50 trades per message
+            0.99 is used to make sure that there are no extra (empty) messages
+            As an example with 50 trades, there will be int(50/50 + 0.99) = 1 message
+            """
             for i in range(0, max(int(len(statlist) / max_trades_per_msg + 0.99), 1)):
                 message = tabulate(statlist[i * max_trades_per_msg:(i + 1) * max_trades_per_msg],
                                    headers=head,

From d795febf9243e667de70d06a8c2ff30e951f847a Mon Sep 17 00:00:00 2001
From: rextea 
Date: Wed, 24 Mar 2021 18:26:03 +0200
Subject: [PATCH 182/187] Add info to documantation

---
 docs/bot-basics.md | 1 +
 1 file changed, 1 insertion(+)

diff --git a/docs/bot-basics.md b/docs/bot-basics.md
index 13694c316..943af0362 100644
--- a/docs/bot-basics.md
+++ b/docs/bot-basics.md
@@ -53,6 +53,7 @@ This loop will be repeated again and again until the bot is stopped.
 * Calls `bot_loop_start()` once.
 * Calculate indicators (calls `populate_indicators()` once per pair).
 * Calculate buy / sell signals (calls `populate_buy_trend()` and `populate_sell_trend()` once per pair)
+* Confirm trade buy / sell (calls `confirm_trade_entry()` and `confirm_trade_exit()` if implemented in the strategy)
 * Loops per candle simulating entry and exit points.
 * Generate backtest report output
 

From ec15610bff2314e8f47620e410097106878ffd18 Mon Sep 17 00:00:00 2001
From: Matthias 
Date: Wed, 24 Mar 2021 19:21:07 +0100
Subject: [PATCH 183/187] Fix isort issue

---
 freqtrade/optimize/backtesting.py | 1 +
 1 file changed, 1 insertion(+)

diff --git a/freqtrade/optimize/backtesting.py b/freqtrade/optimize/backtesting.py
index d858acc1a..7de5b171c 100644
--- a/freqtrade/optimize/backtesting.py
+++ b/freqtrade/optimize/backtesting.py
@@ -30,6 +30,7 @@ from freqtrade.strategy.interface import IStrategy, SellCheckTuple, SellType
 from freqtrade.strategy.strategy_wrapper import strategy_safe_wrapper
 from freqtrade.wallets import Wallets
 
+
 logger = logging.getLogger(__name__)
 
 # Indexes for backtest tuples

From 0ca95aa0c2035b41c9835f2291020222f9ed690d Mon Sep 17 00:00:00 2001
From: rextea 
Date: Thu, 25 Mar 2021 10:25:25 +0200
Subject: [PATCH 184/187] Change rate to acctual close rate

---
 freqtrade/optimize/backtesting.py | 22 +++++++++++-----------
 1 file changed, 11 insertions(+), 11 deletions(-)

diff --git a/freqtrade/optimize/backtesting.py b/freqtrade/optimize/backtesting.py
index 7de5b171c..00b2f278d 100644
--- a/freqtrade/optimize/backtesting.py
+++ b/freqtrade/optimize/backtesting.py
@@ -30,7 +30,6 @@ from freqtrade.strategy.interface import IStrategy, SellCheckTuple, SellType
 from freqtrade.strategy.strategy_wrapper import strategy_safe_wrapper
 from freqtrade.wallets import Wallets
 
-
 logger = logging.getLogger(__name__)
 
 # Indexes for backtest tuples
@@ -253,20 +252,21 @@ class Backtesting:
                                          sell_row[DATE_IDX], sell_row[BUY_IDX], sell_row[SELL_IDX],
                                          low=sell_row[LOW_IDX], high=sell_row[HIGH_IDX])
 
-        time_in_force = self.strategy.order_time_in_force['sell']
-        # confirm_trade_exit
-        if not strategy_safe_wrapper(self.strategy.confirm_trade_exit, default_retval=True)(
-                pair=trade.pair, trade=trade, order_type='limit', amount=trade.amount,
-                rate=sell_row[LOW_IDX],
-                time_in_force=time_in_force,
-                sell_reason=sell.sell_type.value):
-            return None
-
         if sell.sell_flag:
             trade.close_date = sell_row[DATE_IDX]
             trade.sell_reason = sell.sell_type.value
             trade_dur = int((trade.close_date_utc - trade.open_date_utc).total_seconds() // 60)
             closerate = self._get_close_rate(sell_row, trade, sell, trade_dur)
+
+            # Confirm trade exit:
+            time_in_force = self.strategy.order_time_in_force['sell']
+            if not strategy_safe_wrapper(self.strategy.confirm_trade_exit, default_retval=True)(
+                    pair=trade.pair, trade=trade, order_type='limit', amount=trade.amount,
+                    rate=closerate,
+                    time_in_force=time_in_force,
+                    sell_reason=sell.sell_type.value):
+                return None
+
             trade.close(closerate, show_msg=False)
             return trade
 
@@ -283,7 +283,7 @@ class Backtesting:
 
         order_type = self.strategy.order_types['buy']
         time_in_force = self.strategy.order_time_in_force['sell']
-        # confirm_trade_entry
+        # Confirm trade entry:
         if not strategy_safe_wrapper(self.strategy.confirm_trade_entry, default_retval=True)(
                 pair=pair, order_type=order_type, amount=stake_amount, rate=row[OPEN_IDX],
                 time_in_force=time_in_force):

From 292ea8c1d03b0ecf100d9a1935f0718f4c316cae Mon Sep 17 00:00:00 2001
From: Matthias 
Date: Thu, 25 Mar 2021 09:34:33 +0100
Subject: [PATCH 185/187] Update backtesting.py

---
 freqtrade/optimize/backtesting.py | 1 +
 1 file changed, 1 insertion(+)

diff --git a/freqtrade/optimize/backtesting.py b/freqtrade/optimize/backtesting.py
index 00b2f278d..765e2844a 100644
--- a/freqtrade/optimize/backtesting.py
+++ b/freqtrade/optimize/backtesting.py
@@ -30,6 +30,7 @@ from freqtrade.strategy.interface import IStrategy, SellCheckTuple, SellType
 from freqtrade.strategy.strategy_wrapper import strategy_safe_wrapper
 from freqtrade.wallets import Wallets
 
+
 logger = logging.getLogger(__name__)
 
 # Indexes for backtest tuples

From 39bfe5e1a79263b18c10e5b8ca51c58705885e66 Mon Sep 17 00:00:00 2001
From: Masoud Azizi 
Date: Fri, 26 Mar 2021 22:50:27 +0430
Subject: [PATCH 186/187] Thee to the

---
 docs/hyperopt.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/docs/hyperopt.md b/docs/hyperopt.md
index 69bc57d1a..96c7354b9 100644
--- a/docs/hyperopt.md
+++ b/docs/hyperopt.md
@@ -165,7 +165,7 @@ Depending on the space you want to optimize, only some of the below are required
 * fill `sell_indicator_space` - for sell signal optimization
 
 !!! Note
-    `populate_indicators` needs to create all indicators any of thee spaces may use, otherwise hyperopt will not work.
+    `populate_indicators` needs to create all indicators any of the spaces may use, otherwise hyperopt will not work.
 
 Optional in hyperopt - can also be loaded from a strategy (recommended):
 

From 7fb34f7e25e1a9d08b3200fb8a0c7b57df773a52 Mon Sep 17 00:00:00 2001
From: Matthias 
Date: Sat, 27 Mar 2021 11:34:11 +0100
Subject: [PATCH 187/187] Version bump 2021.3

---
 freqtrade/__init__.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/freqtrade/__init__.py b/freqtrade/__init__.py
index 2205d284d..5e2a1f88e 100644
--- a/freqtrade/__init__.py
+++ b/freqtrade/__init__.py
@@ -1,5 +1,5 @@
 """ Freqtrade bot """
-__version__ = '2021.2'
+__version__ = '2021.3'
 
 if __version__ == 'develop':