Merge remote-tracking branch 'upstream/develop' into pairlists-shuffle

docs/configuration.md

@@ -81,14 +81,14 @@ Mandatory parameters are marked as **Required**, which means that they are requi
 | `exchange.key` | API key to use for the exchange. Only required when you are in production mode.<br>**Keep it in secret, do not disclose publicly.** <br> **Datatype:** String
 | `exchange.secret` | API secret to use for the exchange. Only required when you are in production mode.<br>**Keep it in secret, do not disclose publicly.** <br> **Datatype:** String
 | `exchange.password` | API password to use for the exchange. Only required when you are in production mode and for exchanges that use password for API requests.<br>**Keep it in secret, do not disclose publicly.** <br> **Datatype:** String
-| `exchange.pair_whitelist` | List of pairs to use by the bot for trading and to check for potential trades during backtesting. Not used by VolumePairList (see [below](#dynamic-pairlists)). <br> **Datatype:** List
+| `exchange.pair_whitelist` | List of pairs to use by the bot for trading and to check for potential trades during backtesting. Not used by VolumePairList (see [below](#pairlists-and-pairlist-handlers)). <br> **Datatype:** List
-| `exchange.pair_blacklist` | List of pairs the bot must absolutely avoid for trading and backtesting (see [below](#dynamic-pairlists)). <br> **Datatype:** List
+| `exchange.pair_blacklist` | List of pairs the bot must absolutely avoid for trading and backtesting (see [below](#pairlists-and-pairlist-handlers)). <br> **Datatype:** List
 | `exchange.ccxt_config` | Additional CCXT parameters passed to the regular ccxt instance. Parameters may differ from exchange to exchange and are documented in the [ccxt documentation](https://ccxt.readthedocs.io/en/latest/manual.html#instantiation) <br> **Datatype:** Dict
 | `exchange.ccxt_async_config` | Additional CCXT parameters passed to the async ccxt instance. Parameters may differ from exchange to exchange  and are documented in the [ccxt documentation](https://ccxt.readthedocs.io/en/latest/manual.html#instantiation) <br> **Datatype:** Dict
 | `exchange.markets_refresh_interval` | The interval in minutes in which markets are reloaded. <br>*Defaults to `60` minutes.* <br> **Datatype:** Positive Integer
 | `edge.*` | Please refer to [edge configuration document](edge.md) for detailed explanation.
 | `experimental.block_bad_exchanges` | Block exchanges known to not work with freqtrade. Leave on default unless you want to test if that exchange works now. <br>*Defaults to `true`.* <br> **Datatype:** Boolean
-| `pairlists` | Define one or more pairlists to be used. [More information below](#dynamic-pairlists). <br>*Defaults to `StaticPairList`.*  <br> **Datatype:** List of Dicts
+| `pairlists` | Define one or more pairlists to be used. [More information below](#pairlists-and-pairlist-handlers). <br>*Defaults to `StaticPairList`.*  <br> **Datatype:** List of Dicts
 | `telegram.enabled` | Enable the usage of Telegram. <br> **Datatype:** Boolean
 | `telegram.token` | Your Telegram bot token. Only required if `telegram.enabled` is `true`. <br>**Keep it in secret, do not disclose publicly.** <br> **Datatype:** String
 | `telegram.chat_id` | Your personal Telegram account id. Only required if `telegram.enabled` is `true`. <br>**Keep it in secret, do not disclose publicly.** <br> **Datatype:** String
@@ -549,18 +549,19 @@ A fixed slot (mirroring `bid_strategy.order_book_top`) can be defined by setting
 
 When not using orderbook (`ask_strategy.use_order_book=False`), the price at the `ask_strategy.price_side` side (defaults to `"ask"`) from the ticker will be used as the sell price.
 
-## Pairlists
+## Pairlists and Pairlist Handlers
 
-Pairlists define the list of pairs that the bot should trade.
+Pairlist Handlers define the list of pairs (pairlist) that the bot should trade. They are configured in the `pairlists` section of the configuration settings.
-There are [`StaticPairList`](#static-pair-list) and dynamic Whitelists available.
 
-[`PrecisionFilter`](#precision-filter) and [`PriceFilter`](#price-pair-filter) act as filters, removing low-value pairs.
+In your configuration, you can use Static Pairlist (defined by the [`StaticPairList`](#static-pair-list) Pairlist Handler) and Dynamic Pairlist (defined by the [`VolumePairList`](#volume-pair-list) Pairlist Handler).
 
-All pairlists can be chained, and a combination of all pairlists will become your new whitelist. Pairlists are executed in the sequence they are configured. You should always configure either `StaticPairList` or `DynamicPairList` as starting pairlists.
+Additionaly, [`PrecisionFilter`](#precision-filter), [`PriceFilter`](#price-pair-filter) and [`SpreadFilter`](#spread-pair-filter) act as Pairlist Filters, removing certain pairs.
 
-Inactive markets and blacklisted pairs are always removed from the resulting `pair_whitelist`.
+If multiple Pairlist Handlers are used, they are chained and a combination of all Pairlist Handlers forms the resulting pairlist the bot uses for trading and backtesting. Pairlist Handlers are executed in the sequence they are configured. You should always configure either `StaticPairList` or `VolumePairList` as the starting Pairlist Handler.
 
-### Available Pairlists
+Inactive markets are always removed from the resulting pairlist. Explicitly blacklisted pairs (those in the `pair_blacklist` configuration setting) are also always removed from the resulting pairlist.
+
+### Available Pairlist Handlers
 
 * [`StaticPairList`](#static-pair-list) (default, if not configured differently)
 * [`VolumePairList`](#volume-pair-list)
@@ -569,7 +570,7 @@ Inactive markets and blacklisted pairs are always removed from the resulting `pa
 * [`SpreadFilter`](#spread-filter)
 
 !!! Tip "Testing pairlists"
-    Pairlist configurations can be quite tricky to get right. Best use the [`test-pairlist`](utils.md#test-pairlist) subcommand to test your configuration quickly.
+    Pairlist configurations can be quite tricky to get right. Best use the [`test-pairlist`](utils.md#test-pairlist) utility subcommand to test your configuration quickly.
 
 #### Static Pair List
 
@@ -585,13 +586,15 @@ It uses configuration from `exchange.pair_whitelist` and `exchange.pair_blacklis
 
 #### Volume Pair List
 
-`VolumePairList` selects `number_assets` top pairs based on `sort_key`, which can only be `quoteVolume`.
+`VolumePairList` employs sorting/filtering of pairs by their trading volume. I selects `number_assets` top pairs with sorting based on the `sort_key` (which can only be `quoteVolume`).
 
-`VolumePairList` considers outputs of previous pairlists unless  it's the first configured pairlist, it does not consider `pair_whitelist`, but selects the top assets from all available markets (with matching stake-currency) on the exchange.
+When used in the chain of Pairlist Handlers in a non-leading position (after StaticPairList and other Pairlist Filters), `VolumePairList` considers outputs of previous Pairlist Handlers, adding its sorting/selection of the pairs by the trading volume.
 
-`refresh_period` allows setting the period (in seconds), at which the pairlist will be refreshed. Defaults to 1800s (30 minutes).
+When used on the leading position of the chain of Pairlist Handlers, it does not consider `pair_whitelist` configuration setting, but selects the top assets from all available markets (with matching stake-currency) on the exchange.
 
-`VolumePairList` is based on the ticker data, as reported by the ccxt library:
+The `refresh_period` setting allows to define the period (in seconds), at which the pairlist will be refreshed. Defaults to 1800s (30 minutes).
+
+`VolumePairList` is based on the ticker data from exchange, as reported by the ccxt library:
 
 * The `quoteVolume` is the amount of quote (stake) currency traded (bought or sold) in last 24 hours.
 
@@ -606,28 +609,32 @@ It uses configuration from `exchange.pair_whitelist` and `exchange.pair_blacklis
 
 #### PrecisionFilter
 
-Filters low-value coins which would not allow setting a stoploss.
+Filters low-value coins which would not allow setting stoplosses.
 
-#### Price Pair Filter
+#### PriceFilter
 
 The `PriceFilter` allows filtering of pairs by price.
-Currently, only `low_price_ratio` is implemented, where a raise of 1 price unit (pip) is below the `low_price_ratio` ratio.
+
+Currently, only `low_price_ratio` setting is implemented, where a raise of 1 price unit (pip) is below the `low_price_ratio` ratio.
 This option is disabled by default, and will only apply if set to <> 0.
 
 Calculation example:
+
 Min price precision is 8 decimals. If price is 0.00000011 - one step would be 0.00000012 - which is almost 10% higher than the previous value.
 
-These pairs are dangerous since it may be impossible to place the desired stoploss - and often result in high losses.
+These pairs are dangerous since it may be impossible to place the desired stoploss - and often result in high losses. Here is what the PriceFilters takes over.
 
 #### SpreadFilter
 
 Removes pairs that have a difference between asks and bids above the specified ratio (default `0.005`).
+
 Example:
+
 If `DOGE/BTC` maximum bid is 0.00000026 and minimum ask is 0.00000027 the ratio is calculated as: `1 - bid/ask ~= 0.037` which is `> 0.005`
 
-### Full Pairlist example
+### Full example of Pairlist Handlers
 
-The below example blacklists `BNB/BTC`, uses `VolumePairList` with `20` assets, sorting by `quoteVolume` and applies both [`PrecisionFilter`](#precision-filter) and [`PriceFilter`](#price-pair-filter), filtering all assets where 1 priceunit is > 1%.
+The below example blacklists `BNB/BTC`, uses `VolumePairList` with `20` assets, sorting pairs by `quoteVolume` and applies both [`PrecisionFilter`](#precision-filter) and [`PriceFilter`](#price-pair-filter), filtering all assets where 1 priceunit is > 1%.
 
 ```json
 "exchange": {

docs/requirements-docs.txt

@@ -1,2 +1,2 @@
-mkdocs-material==5.1.6
+mkdocs-material==5.1.7
 mdx_truly_sane_lists==1.2

freqtrade/commands/build_config_commands.py

@@ -163,7 +163,7 @@ def deploy_new_config(config_path: Path, selections: Dict[str, Any]) -> None:
             )
     except TemplateNotFound:
         selections['exchange'] = render_template(
-            templatefile=f"subtemplates/exchange_generic.j2",
+            templatefile="subtemplates/exchange_generic.j2",
             arguments=selections
         )
 

freqtrade/commands/cli_options.py

@@ -372,8 +372,8 @@ AVAILABLE_CLI_OPTIONS = {
     ),
     "timeframes": Arg(
         '-t', '--timeframes',
-        help=f'Specify which tickers to download. Space-separated list. '
+        help='Specify which tickers to download. Space-separated list. '
-        f'Default: `1m 5m`.',
+        'Default: `1m 5m`.',
         choices=['1m', '3m', '5m', '15m', '30m', '1h', '2h', '4h',
                  '6h', '8h', '12h', '1d', '3d', '1w'],
         default=['1m', '5m'],

freqtrade/commands/deploy_commands.py

@@ -51,7 +51,7 @@ def deploy_new_strategy(strategy_name: str, strategy_path: Path, subtemplate: st
     )
     additional_methods = render_template_with_fallback(
         templatefile=f"subtemplates/strategy_methods_{subtemplate}.j2",
-        templatefallbackfile=f"subtemplates/strategy_methods_empty.j2",
+        templatefallbackfile="subtemplates/strategy_methods_empty.j2",
     )
 
     strategy_text = render_template(templatefile='base_strategy.py.j2',

freqtrade/data/dataprovider.py

@@ -5,7 +5,7 @@ including ticker and orderbook data, live and historical candle (OHLCV) data
 Common Interface for bot and strategy to access data.
 """
 import logging
-from typing import Any, Dict, List, Optional, Tuple
+from typing import Any, Dict, List, Optional
 
 from pandas import DataFrame
 
@@ -13,6 +13,8 @@ from freqtrade.data.history import load_pair_history
 from freqtrade.exceptions import DependencyException, OperationalException
 from freqtrade.exchange import Exchange
 from freqtrade.state import RunMode
+from freqtrade.typing import ListPairsWithTimeframes
+
 
 logger = logging.getLogger(__name__)
 
@@ -25,8 +27,8 @@ class DataProvider:
         self._pairlists = pairlists
 
     def refresh(self,
-                pairlist: List[Tuple[str, str]],
+                pairlist: ListPairsWithTimeframes,
-                helping_pairs: List[Tuple[str, str]] = None) -> None:
+                helping_pairs: ListPairsWithTimeframes = None) -> None:
         """
         Refresh data, called with each cycle
         """
@@ -36,7 +38,7 @@ class DataProvider:
             self._exchange.refresh_latest_ohlcv(pairlist)
 
     @property
-    def available_pairs(self) -> List[Tuple[str, str]]:
+    def available_pairs(self) -> ListPairsWithTimeframes:
         """
         Return a list of tuples containing (pair, timeframe) for which data is currently cached.
         Should be whitelist + open trades.

freqtrade/exchange/common.py

@@ -1,6 +1,6 @@
 import logging
 
-from freqtrade.exceptions import DependencyException, TemporaryError
+from freqtrade.exceptions import TemporaryError
 
 logger = logging.getLogger(__name__)
 
@@ -93,7 +93,7 @@ def retrier_async(f):
         count = kwargs.pop('count', API_RETRY_COUNT)
         try:
             return await f(*args, **kwargs)
-        except (TemporaryError, DependencyException) as ex:
+        except TemporaryError as ex:
             logger.warning('%s() returned exception: "%s"', f.__name__, ex)
             if count > 0:
                 count -= 1
@@ -111,7 +111,7 @@ def retrier(f):
         count = kwargs.pop('count', API_RETRY_COUNT)
         try:
             return f(*args, **kwargs)
-        except (TemporaryError, DependencyException) as ex:
+        except TemporaryError as ex:
             logger.warning('%s() returned exception: "%s"', f.__name__, ex)
             if count > 0:
                 count -= 1

freqtrade/exchange/exchange.py

@@ -23,6 +23,7 @@ from freqtrade.exceptions import (DependencyException, InvalidOrderException,
                                   OperationalException, TemporaryError)
 from freqtrade.exchange.common import BAD_EXCHANGES, retrier, retrier_async
 from freqtrade.misc import deep_merge_dicts, safe_value_fallback
+from freqtrade.typing import ListPairsWithTimeframes
 
 CcxtModuleType = Any
 
@@ -366,8 +367,7 @@ class Exchange:
                 f"Invalid timeframe '{timeframe}'. This exchange supports: {self.timeframes}")
 
         if timeframe and timeframe_to_minutes(timeframe) < 1:
-            raise OperationalException(
+            raise OperationalException("Timeframes < 1m are currently not supported by Freqtrade.")
-                f"Timeframes < 1m are currently not supported by Freqtrade.")
 
     def validate_ordertypes(self, order_types: Dict) -> None:
         """
@@ -676,7 +676,7 @@ class Exchange:
         logger.info("Downloaded data for %s with length %s.", pair, len(data))
         return data
 
-    def refresh_latest_ohlcv(self, pair_list: List[Tuple[str, str]]) -> List[Tuple[str, List]]:
+    def refresh_latest_ohlcv(self, pair_list: ListPairsWithTimeframes) -> List[Tuple[str, List]]:
         """
         Refresh in-memory OHLCV asynchronously and set `_klines` with the result
         Loops asynchronously over pair_list and downloads all pairs async (semi-parallel).

freqtrade/exchange/kraken.py

@@ -7,7 +7,7 @@ import ccxt
 from freqtrade.exceptions import (DependencyException, InvalidOrderException,
                                   OperationalException, TemporaryError)
 from freqtrade.exchange import Exchange
-from freqtrade.exchange.exchange import retrier
+from freqtrade.exchange.common import retrier
 
 logger = logging.getLogger(__name__)
 

freqtrade/freqtradebot.py

@@ -7,7 +7,7 @@ import traceback
 from datetime import datetime
 from math import isclose
 from threading import Lock
-from typing import Any, Dict, List, Optional, Tuple
+from typing import Any, Dict, List, Optional
 
 import arrow
 from cachetools import TTLCache
@@ -84,7 +84,7 @@ class FreqtradeBot:
         self.edge = Edge(self.config, self.exchange, self.strategy) if \
             self.config.get('edge', {}).get('enabled', False) else None
 
-        self.active_pair_whitelist = self._refresh_whitelist()
+        self.active_pair_whitelist = self._refresh_active_whitelist()
 
         # Set initial bot state from config
         initial_state = self.config.get('initial_state')
@@ -145,10 +145,10 @@ class FreqtradeBot:
         # Query trades from persistence layer
         trades = Trade.get_open_trades()
 
-        self.active_pair_whitelist = self._refresh_whitelist(trades)
+        self.active_pair_whitelist = self._refresh_active_whitelist(trades)
 
         # Refreshing candles
-        self.dataprovider.refresh(self._create_pair_whitelist(self.active_pair_whitelist),
+        self.dataprovider.refresh(self.pairlists.create_pair_list(self.active_pair_whitelist),
                                   self.strategy.informative_pairs())
 
         with self._sell_lock:
@@ -175,9 +175,10 @@ class FreqtradeBot:
         if self.config['cancel_open_orders_on_exit']:
             self.cancel_all_open_orders()
 
-    def _refresh_whitelist(self, trades: List[Trade] = []) -> List[str]:
+    def _refresh_active_whitelist(self, trades: List[Trade] = []) -> List[str]:
         """
-        Refresh whitelist from pairlist or edge and extend it with trades.
+        Refresh active whitelist from pairlist or edge and extend it with
+        pairs that have open trades.
         """
         # Refresh whitelist
         self.pairlists.refresh_pairlist()
@@ -194,12 +195,6 @@ class FreqtradeBot:
             _whitelist.extend([trade.pair for trade in trades if trade.pair not in _whitelist])
         return _whitelist
 
-    def _create_pair_whitelist(self, pairs: List[str]) -> List[Tuple[str, str]]:
-        """
-        Create pair-whitelist tuple with (pair, ticker_interval)
-        """
-        return [(pair, self.config['ticker_interval']) for pair in pairs]
-
     def get_free_open_trades(self):
         """
         Return the number of free open trades slots or 0 if

freqtrade/pairlist/VolumePairList.py

@@ -25,7 +25,7 @@ class VolumePairList(IPairList):
 
         if 'number_assets' not in self._pairlistconfig:
             raise OperationalException(
-                f'`number_assets` not specified. Please check your configuration '
+                '`number_assets` not specified. Please check your configuration '
                 'for "pairlist.config.number_assets"')
 
         self._stake_currency = config['stake_currency']

freqtrade/pairlist/pairlistmanager.py

@@ -10,6 +10,7 @@ from cachetools import TTLCache, cached
 from freqtrade.exceptions import OperationalException
 from freqtrade.pairlist.IPairList import IPairList
 from freqtrade.resolvers import PairListResolver
+from freqtrade.typing import ListPairsWithTimeframes
 
 
 logger = logging.getLogger(__name__)
@@ -107,3 +108,9 @@ class PairListManager():
                     pairlist.remove(p)
 
         return pairlist
+
+    def create_pair_list(self, pairs: List[str], timeframe: str = None) -> ListPairsWithTimeframes:
+        """
+        Create list of pair tuples with (pair, ticker_interval)
+        """
+        return [(pair, timeframe or self._config['ticker_interval']) for pair in pairs]

freqtrade/rpc/rpc.py

@@ -545,5 +545,5 @@ class RPC:
     def _rpc_edge(self) -> List[Dict[str, Any]]:
         """ Returns information related to Edge """
         if not self._freqtrade.edge:
-            raise RPCException(f'Edge is not enabled.')
+            raise RPCException('Edge is not enabled.')
         return self._freqtrade.edge.accepted_pairs()

freqtrade/rpc/telegram.py

@@ -234,7 +234,7 @@ class Telegram(RPC):
                         lines.append("*Open Order:* `{open_order}`")
 
                 # Filter empty lines using list-comprehension
-                messages.append("\n".join([l for l in lines if l]).format(**r))
+                messages.append("\n".join([line for line in lines if line]).format(**r))
 
             for msg in messages:
                 self._send_msg(msg)
@@ -289,7 +289,7 @@ class Telegram(RPC):
                     'Day',
                     f'Profit {stake_cur}',
                     f'Profit {fiat_disp_cur}',
-                    f'Trades',
+                    'Trades',
                 ],
                 tablefmt='simple')
             message = f'<b>Daily Profit over the last {timescale} days</b>:\n<pre>{stats_tab}</pre>'

freqtrade/strategy/interface.py

@@ -7,7 +7,7 @@ import warnings
 from abc import ABC, abstractmethod
 from datetime import datetime, timezone
 from enum import Enum
-from typing import Dict, List, NamedTuple, Optional, Tuple
+from typing import Dict, NamedTuple, Optional, Tuple
 
 import arrow
 from pandas import DataFrame
@@ -17,8 +17,10 @@ from freqtrade.exceptions import StrategyError
 from freqtrade.exchange import timeframe_to_minutes
 from freqtrade.persistence import Trade
 from freqtrade.strategy.strategy_wrapper import strategy_safe_wrapper
+from freqtrade.typing import ListPairsWithTimeframes
 from freqtrade.wallets import Wallets
 
+
 logger = logging.getLogger(__name__)
 
 
@@ -185,7 +187,7 @@ class IStrategy(ABC):
         """
         return False
 
-    def informative_pairs(self) -> List[Tuple[str, str]]:
+    def informative_pairs(self) -> ListPairsWithTimeframes:
         """
         Define additional, informative pair/interval combinations to be cached from the exchange.
         These pair/interval combinations are non-tradeable, unless they are part

freqtrade/typing.py

@@ -0,0 +1,8 @@
+"""
+Common Freqtrade types
+"""
+
+from typing import List, Tuple
+
+# List of pairs with their timeframes
+ListPairsWithTimeframes = List[Tuple[str, str]]

freqtrade/worker.py

@@ -37,9 +37,7 @@ class Worker:
         self._heartbeat_msg: float = 0
 
         # Tell systemd that we completed initialization phase
-        if self._sd_notify:
+        self._notify("READY=1")
-            logger.debug("sd_notify: READY=1")
-            self._sd_notify.notify("READY=1")
 
     def _init(self, reconfig: bool) -> None:
         """
@@ -60,6 +58,15 @@ class Worker:
         self._sd_notify = sdnotify.SystemdNotifier() if \
             self._config.get('internals', {}).get('sd_notify', False) else None
 
+    def _notify(self, message: str) -> None:
+        """
+        Removes the need to verify in all occurances if sd_notify is enabled
+        :param message: Message to send to systemd if it's enabled.
+        """
+        if self._sd_notify:
+            logger.debug(f"sd_notify: {message}")
+            self._sd_notify.notify(message)
+
     def run(self) -> None:
         state = None
         while True:
@@ -89,17 +96,13 @@ class Worker:
 
         if state == State.STOPPED:
             # Ping systemd watchdog before sleeping in the stopped state
-            if self._sd_notify:
+            self._notify("WATCHDOG=1\nSTATUS=State: STOPPED.")
-                logger.debug("sd_notify: WATCHDOG=1\\nSTATUS=State: STOPPED.")
-                self._sd_notify.notify("WATCHDOG=1\nSTATUS=State: STOPPED.")
 
             self._throttle(func=self._process_stopped, throttle_secs=self._throttle_secs)
 
         elif state == State.RUNNING:
             # Ping systemd watchdog before throttling
-            if self._sd_notify:
+            self._notify("WATCHDOG=1\nSTATUS=State: RUNNING.")
-                logger.debug("sd_notify: WATCHDOG=1\\nSTATUS=State: RUNNING.")
-                self._sd_notify.notify("WATCHDOG=1\nSTATUS=State: RUNNING.")
 
             self._throttle(func=self._process_running, throttle_secs=self._throttle_secs)
 
@@ -154,9 +157,7 @@ class Worker:
         replaces it with the new instance
         """
         # Tell systemd that we initiated reconfiguration
-        if self._sd_notify:
+        self._notify("RELOADING=1")
-            logger.debug("sd_notify: RELOADING=1")
-            self._sd_notify.notify("RELOADING=1")
 
         # Clean up current freqtrade modules
         self.freqtrade.cleanup()
@@ -167,15 +168,11 @@ class Worker:
         self.freqtrade.notify_status('config reloaded')
 
         # Tell systemd that we completed reconfiguration
-        if self._sd_notify:
+        self._notify("READY=1")
-            logger.debug("sd_notify: READY=1")
-            self._sd_notify.notify("READY=1")
 
     def exit(self) -> None:
         # Tell systemd that we are exiting now
-        if self._sd_notify:
+        self._notify("STOPPING=1")
-            logger.debug("sd_notify: STOPPING=1")
-            self._sd_notify.notify("STOPPING=1")
 
         if self.freqtrade:
             self.freqtrade.notify_status('process died')

requirements-common.txt

@@ -1,7 +1,7 @@
 # requirements without requirements installable via conda
 # mainly used for Raspberry pi installs
-ccxt==1.27.49
+ccxt==1.27.91
-SQLAlchemy==1.3.16
+SQLAlchemy==1.3.17
 python-telegram-bot==12.7
 arrow==0.15.6
 cachetools==4.1.0

requirements-dev.txt

@@ -4,7 +4,7 @@
 -r requirements-hyperopt.txt
 
 coveralls==2.0.0
-flake8==3.7.9
+flake8==3.8.1
 flake8-type-annotations==0.1.0
 flake8-tidy-imports==4.1.0
 mypy==0.770

requirements-hyperopt.txt

@@ -6,5 +6,5 @@ scipy==1.4.1
 scikit-learn==0.22.2.post1
 scikit-optimize==0.7.4
 filelock==3.0.12
-joblib==0.14.1
+joblib==0.15.1
 progressbar2==3.51.3

tests/conftest.py

@@ -92,7 +92,7 @@ def patch_wallet(mocker, free=999.9) -> None:
 
 
 def patch_whitelist(mocker, conf) -> None:
-    mocker.patch('freqtrade.freqtradebot.FreqtradeBot._refresh_whitelist',
+    mocker.patch('freqtrade.freqtradebot.FreqtradeBot._refresh_active_whitelist',
                  MagicMock(return_value=conf['exchange']['pair_whitelist']))
 
 
@@ -1717,7 +1717,8 @@ def hyperopt_results():
             'loss': 20.0,
             'params_dict': {
                 'mfi-value': 17, 'fastd-value': 38, 'adx-value': 48, 'rsi-value': 22, 'mfi-enabled': True, 'fastd-enabled': False, 'adx-enabled': True, 'rsi-enabled': True, 'trigger': 'macd_cross_signal', 'sell-mfi-value': 96, 'sell-fastd-value': 68, 'sell-adx-value': 63, 'sell-rsi-value': 81, 'sell-mfi-enabled': False, 'sell-fastd-enabled': True, 'sell-adx-enabled': True, 'sell-rsi-enabled': True, 'sell-trigger': 'sell-sar_reversal', 'roi_t1': 334, 'roi_t2': 683, 'roi_t3': 140, 'roi_p1': 0.06403981740598495, 'roi_p2': 0.055519840060645045, 'roi_p3': 0.3253712811342459, 'stoploss': -0.338070047333259},  # noqa: E501
-            'params_details': {'buy': {'mfi-value': 17, 'fastd-value': 38, 'adx-value': 48, 'rsi-value': 22, 'mfi-enabled': True, 'fastd-enabled': False, 'adx-enabled': True, 'rsi-enabled': True, 'trigger': 'macd_cross_signal'},  # noqa: E501
+            'params_details': {
+                'buy': {'mfi-value': 17, 'fastd-value': 38, 'adx-value': 48, 'rsi-value': 22, 'mfi-enabled': True, 'fastd-enabled': False, 'adx-enabled': True, 'rsi-enabled': True, 'trigger': 'macd_cross_signal'},  # noqa: E501
                 'sell': {'sell-mfi-value': 96, 'sell-fastd-value': 68, 'sell-adx-value': 63, 'sell-rsi-value': 81, 'sell-mfi-enabled': False, 'sell-fastd-enabled': True, 'sell-adx-enabled': True, 'sell-rsi-enabled': True, 'sell-trigger': 'sell-sar_reversal'},  # noqa: E501
                 'roi': {0: 0.4449309386008759, 140: 0.11955965746663, 823: 0.06403981740598495, 1157: 0},  # noqa: E501
                 'stoploss': {'stoploss': -0.338070047333259}},
@@ -1767,7 +1768,8 @@ def hyperopt_results():
         }, {
             'loss': 4.713497421432944,
             'params_dict': {'mfi-value': 13, 'fastd-value': 41, 'adx-value': 21, 'rsi-value': 29, 'mfi-enabled': False, 'fastd-enabled': True, 'adx-enabled': False, 'rsi-enabled': False, 'trigger': 'bb_lower', 'sell-mfi-value': 99, 'sell-fastd-value': 60, 'sell-adx-value': 81, 'sell-rsi-value': 69, 'sell-mfi-enabled': True, 'sell-fastd-enabled': True, 'sell-adx-enabled': True, 'sell-rsi-enabled': False, 'sell-trigger': 'sell-macd_cross_signal', 'roi_t1': 771, 'roi_t2': 620, 'roi_t3': 145, 'roi_p1': 0.0586919200378493, 'roi_p2': 0.04984118697312542, 'roi_p3': 0.37521058680247044, 'stoploss': -0.14613268022709905},  # noqa: E501
-            'params_details': {'buy': {'mfi-value': 13, 'fastd-value': 41, 'adx-value': 21, 'rsi-value': 29, 'mfi-enabled': False, 'fastd-enabled': True, 'adx-enabled': False, 'rsi-enabled': False, 'trigger': 'bb_lower'}, 'sell': {'sell-mfi-value': 99, 'sell-fastd-value': 60, 'sell-adx-value': 81, 'sell-rsi-value': 69, 'sell-mfi-enabled': True, 'sell-fastd-enabled': True, 'sell-adx-enabled': True, 'sell-rsi-enabled': False, 'sell-trigger': 'sell-macd_cross_signal'}, 'roi': {0: 0.4837436938134452, 145: 0.10853310701097472, 765: 0.0586919200378493, 1536: 0},  # noqa: E501
+            'params_details': {
+                'buy': {'mfi-value': 13, 'fastd-value': 41, 'adx-value': 21, 'rsi-value': 29, 'mfi-enabled': False, 'fastd-enabled': True, 'adx-enabled': False, 'rsi-enabled': False, 'trigger': 'bb_lower'}, 'sell': {'sell-mfi-value': 99, 'sell-fastd-value': 60, 'sell-adx-value': 81, 'sell-rsi-value': 69, 'sell-mfi-enabled': True, 'sell-fastd-enabled': True, 'sell-adx-enabled': True, 'sell-rsi-enabled': False, 'sell-trigger': 'sell-macd_cross_signal'}, 'roi': {0: 0.4837436938134452, 145: 0.10853310701097472, 765: 0.0586919200378493, 1536: 0},  # noqa: E501
                 'stoploss': {'stoploss': -0.14613268022709905}},  # noqa: E501
             'results_metrics': {'trade_count': 318, 'avg_profit': -0.39833954716981146, 'total_profit': -0.06339929, 'profit': -126.67197600000004, 'duration': 3140.377358490566},  # noqa: E501
             'results_explanation': '   318 trades. Avg profit  -0.40%. Total profit -0.06339929 BTC (-126.67Σ%). Avg duration 3140.4 min.',  # noqa: E501

tests/exchange/test_exchange.py

@@ -517,9 +517,9 @@ def test_validate_pairs_restricted(default_conf, mocker, caplog):
     mocker.patch('freqtrade.exchange.Exchange.validate_stakecurrency')
 
     Exchange(default_conf)
-    assert log_has(f"Pair XRP/BTC is restricted for some users on this exchange."
+    assert log_has("Pair XRP/BTC is restricted for some users on this exchange."
-                   f"Please check if you are impacted by this restriction "
+                   "Please check if you are impacted by this restriction "
-                   f"on the exchange and eventually remove XRP/BTC from your whitelist.", caplog)
+                   "on the exchange and eventually remove XRP/BTC from your whitelist.", caplog)
 
 
 def test_validate_pairs_stakecompatibility(default_conf, mocker, caplog):

tests/optimize/test_hyperopt.py

@@ -820,7 +820,7 @@ def test_continue_hyperopt(mocker, default_conf, caplog):
     Hyperopt(default_conf)
 
     assert unlinkmock.call_count == 0
-    assert log_has(f"Continuing on previous hyperopt results.", caplog)
+    assert log_has("Continuing on previous hyperopt results.", caplog)
 
 
 def test_print_json_spaces_all(mocker, default_conf, caplog, capsys) -> None:

tests/rpc/test_rpc.py

@@ -83,7 +83,7 @@ def test_rpc_trade_status(default_conf, ticker, fee, mocker) -> None:
     } == results[0]
 
     mocker.patch('freqtrade.freqtradebot.FreqtradeBot.get_sell_rate',
-                 MagicMock(side_effect=DependencyException(f"Pair 'ETH/BTC' not available")))
+                 MagicMock(side_effect=DependencyException("Pair 'ETH/BTC' not available")))
     results = rpc._rpc_trade_status()
     assert isnan(results[0]['current_profit'])
     assert isnan(results[0]['current_rate'])
@@ -167,7 +167,7 @@ def test_rpc_status_table(default_conf, ticker, fee, mocker) -> None:
     assert '-0.41% (-0.06)' == result[0][3]
 
     mocker.patch('freqtrade.freqtradebot.FreqtradeBot.get_sell_rate',
-                 MagicMock(side_effect=DependencyException(f"Pair 'ETH/BTC' not available")))
+                 MagicMock(side_effect=DependencyException("Pair 'ETH/BTC' not available")))
     result, headers = rpc._rpc_status_table(default_conf['stake_currency'], 'USD')
     assert 'instantly' == result[0][2]
     assert 'ETH/BTC' in result[0][1]
@@ -319,7 +319,7 @@ def test_rpc_trade_statistics(default_conf, ticker, ticker_sell_up, fee,
 
     # Test non-available pair
     mocker.patch('freqtrade.freqtradebot.FreqtradeBot.get_sell_rate',
-                 MagicMock(side_effect=DependencyException(f"Pair 'ETH/BTC' not available")))
+                 MagicMock(side_effect=DependencyException("Pair 'ETH/BTC' not available")))
     stats = rpc._rpc_trade_statistics(stake_currency, fiat_display_currency)
     assert stats['trade_count'] == 2
     assert stats['first_trade_date'] == 'just now'

tests/test_configuration.py

@@ -73,7 +73,7 @@ def test_load_config_file_error(default_conf, mocker, caplog) -> None:
     mocker.patch('freqtrade.configuration.load_config.open', mocker.mock_open(read_data=filedata))
     mocker.patch.object(Path, "read_text", MagicMock(return_value=filedata))
 
-    with pytest.raises(OperationalException, match=f".*Please verify the following segment.*"):
+    with pytest.raises(OperationalException, match=r".*Please verify the following segment.*"):
         load_config_file('somefile')
 
 

tests/test_freqtradebot.py

@@ -3153,10 +3153,8 @@ def test_trailing_stop_loss(default_conf, limit_buy_order, fee, caplog, mocker)
     caplog.set_level(logging.DEBUG)
     # Sell as trailing-stop is reached
     assert freqtrade.handle_trade(trade) is True
-    assert log_has(
+    assert log_has("ETH/BTC - HIT STOP: current price at 0.000012, stoploss is 0.000015, "
-        f"ETH/BTC - HIT STOP: current price at 0.000012, "
+                   "initial stoploss was at 0.000010, trade opened at 0.000011", caplog)
-        f"stoploss is 0.000015, "
-        f"initial stoploss was at 0.000010, trade opened at 0.000011", caplog)
     assert trade.sell_reason == SellType.TRAILING_STOP_LOSS.value
 
 
@@ -3199,8 +3197,8 @@ def test_trailing_stop_loss_positive(default_conf, limit_buy_order, fee,
                  }))
     # stop-loss not reached, adjusted stoploss
     assert freqtrade.handle_trade(trade) is False
-    assert log_has(f"ETH/BTC - Using positive stoploss: 0.01 offset: 0 profit: 0.2666%", caplog)
+    assert log_has("ETH/BTC - Using positive stoploss: 0.01 offset: 0 profit: 0.2666%", caplog)
-    assert log_has(f"ETH/BTC - Adjusting stoploss...", caplog)
+    assert log_has("ETH/BTC - Adjusting stoploss...", caplog)
     assert trade.stop_loss == 0.0000138501
 
     mocker.patch('freqtrade.exchange.Exchange.fetch_ticker',
@@ -3256,9 +3254,8 @@ def test_trailing_stop_loss_offset(default_conf, limit_buy_order, fee,
                  }))
     # stop-loss not reached, adjusted stoploss
     assert freqtrade.handle_trade(trade) is False
-    assert log_has(f"ETH/BTC - Using positive stoploss: 0.01 offset: 0.011 profit: 0.2666%",
+    assert log_has("ETH/BTC - Using positive stoploss: 0.01 offset: 0.011 profit: 0.2666%", caplog)
-                   caplog)
+    assert log_has("ETH/BTC - Adjusting stoploss...", caplog)
-    assert log_has(f"ETH/BTC - Adjusting stoploss...", caplog)
     assert trade.stop_loss == 0.0000138501
 
     mocker.patch('freqtrade.exchange.Exchange.fetch_ticker',
@@ -3322,7 +3319,7 @@ def test_tsl_only_offset_reached(default_conf, limit_buy_order, fee,
     # stop-loss should not be adjusted as offset is not reached yet
     assert freqtrade.handle_trade(trade) is False
 
-    assert not log_has(f"ETH/BTC - Adjusting stoploss...", caplog)
+    assert not log_has("ETH/BTC - Adjusting stoploss...", caplog)
     assert trade.stop_loss == 0.0000098910
 
     # price rises above the offset (rises 12% when the offset is 5.5%)
@@ -3334,9 +3331,8 @@ def test_tsl_only_offset_reached(default_conf, limit_buy_order, fee,
                  }))
 
     assert freqtrade.handle_trade(trade) is False
-    assert log_has(f"ETH/BTC - Using positive stoploss: 0.05 offset: 0.055 profit: 0.1218%",
+    assert log_has("ETH/BTC - Using positive stoploss: 0.05 offset: 0.055 profit: 0.1218%", caplog)
-                   caplog)
+    assert log_has("ETH/BTC - Adjusting stoploss...", caplog)
-    assert log_has(f"ETH/BTC - Adjusting stoploss...", caplog)
     assert trade.stop_loss == 0.0000117705