diff --git a/docs/developer.md b/docs/developer.md index 34b2f1ba5..036109d5b 100644 --- a/docs/developer.md +++ b/docs/developer.md @@ -92,13 +92,13 @@ docker-compose exec freqtrade_develop /bin/bash You have a great idea for a new pair selection algorithm you would like to try out? Great. Hopefully you also want to contribute this back upstream. -Whatever your motivations are - This should get you off the ground in trying to develop a new Pairlist provider. +Whatever your motivations are - This should get you off the ground in trying to develop a new Pairlist Handler. -First of all, have a look at the [VolumePairList](https://github.com/freqtrade/freqtrade/blob/develop/freqtrade/pairlist/VolumePairList.py) provider, and best copy this file with a name of your new Pairlist Provider. +First of all, have a look at the [VolumePairList](https://github.com/freqtrade/freqtrade/blob/develop/freqtrade/pairlist/VolumePairList.py) Handler, and best copy this file with a name of your new Pairlist Handler. -This is a simple provider, which however serves as a good example on how to start developing. +This is a simple Handler, which however serves as a good example on how to start developing. -Next, modify the classname of the provider (ideally align this with the Filename). +Next, modify the classname of the Handler (ideally align this with the module filename). The base-class provides an instance of the exchange (`self._exchange`) the pairlist manager (`self._pairlistmanager`), as well as the main configuration (`self._config`), the pairlist dedicated configuration (`self._pairlistconfig`) and the absolute position within the list of pairlists. @@ -114,28 +114,44 @@ Now, let's step through the methods which require actions: #### Pairlist configuration -Configuration for PairListProvider is done in the bot configuration file in the element `"pairlist"`. -This Pairlist-object may contain configurations with additional configurations for the configured pairlist. -By convention, `"number_assets"` is used to specify the maximum number of pairs to keep in the whitelist. Please follow this to ensure a consistent user experience. +Configuration for the chain of Pairlist Handlers is done in the bot configuration file in the element `"pairlists"`, an array of configuration parameters for each Pairlist Handlers in the chain. -Additional elements can be configured as needed. `VolumePairList` uses `"sort_key"` to specify the sorting value - however feel free to specify whatever is necessary for your great algorithm to be successfull and dynamic. +By convention, `"number_assets"` is used to specify the maximum number of pairs to keep in the pairlist. Please follow this to ensure a consistent user experience. + +Additional parameters can be configured as needed. For instance, `VolumePairList` uses `"sort_key"` to specify the sorting value - however feel free to specify whatever is necessary for your great algorithm to be successfull and dynamic. #### short_desc Returns a description used for Telegram messages. -This should contain the name of the Provider, as well as a short description containing the number of assets. Please follow the format `"PairlistName - top/bottom X pairs"`. + +This should contain the name of the Pairlist Handler, as well as a short description containing the number of assets. Please follow the format `"PairlistName - top/bottom X pairs"`. + +#### gen_pairlist + +Override this method if the Pairlist Handler can be used as the leading Pairlist Handler in the chain, defining the initial pairlist which is then handled by all Pairlist Handlers in the chain. Examples are `StaticPairList` and `VolumePairList`. + +This is called with each iteration of the bot (only if the Pairlist Handler is at the first location) - so consider implementing caching for compute/network heavy calculations. + +It must return the resulting pairlist (which may then be passed into the chain of Pairlist Handlers). + +Validations are optional, the parent class exposes a `_verify_blacklist(pairlist)` and `_whitelist_for_active_markets(pairlist)` to do default filtering. Use this if you limit your result to a certain number of pairs - so the endresult is not shorter than expected. #### filter_pairlist -Override this method and run all calculations needed in this method. +This method is called for each Pairlist Handler in the chain by the pairlist manager. + This is called with each iteration of the bot - so consider implementing caching for compute/network heavy calculations. It get's passed a pairlist (which can be the result of previous pairlists) as well as `tickers`, a pre-fetched version of `get_tickers()`. -It must return the resulting pairlist (which may then be passed into the next pairlist filter). +The default implementation in the base class simply calls the `_validate_pair()` method for each pair in the pairlist, but you may override it. So you should either implement the `_validate_pair()` in your Pairlist Handler or override `filter_pairlist()` to do something else. + +If overridden, it must return the resulting pairlist (which may then be passed into the next Pairlist Handler in the chain). Validations are optional, the parent class exposes a `_verify_blacklist(pairlist)` and `_whitelist_for_active_markets(pairlist)` to do default filters. Use this if you limit your result to a certain number of pairs - so the endresult is not shorter than expected. +In `VolumePairList`, this implements different methods of sorting, does early validation so only the expected number of pairs is returned. + ##### sample ``` python @@ -145,11 +161,6 @@ Validations are optional, the parent class exposes a `_verify_blacklist(pairlist return pairs ``` -#### _gen_pair_whitelist - -This is a simple method used by `VolumePairList` - however serves as a good example. -In VolumePairList, this implements different methods of sorting, does early validation so only the expected number of pairs is returned. - ## Implement a new Exchange (WIP) !!! Note diff --git a/freqtrade/pairlist/IPairList.py b/freqtrade/pairlist/IPairList.py index e49ad1561..f48a7dcfd 100644 --- a/freqtrade/pairlist/IPairList.py +++ b/freqtrade/pairlist/IPairList.py @@ -8,6 +8,7 @@ from typing import Any, Dict, List from cachetools import TTLCache, cached +from freqtrade.exceptions import OperationalException from freqtrade.exchange import market_is_active @@ -90,6 +91,24 @@ class IPairList(ABC): """ raise NotImplementedError() + def gen_pairlist(self, cached_pairlist: List[str], tickers: Dict) -> List[str]: + """ + Generate the pairlist. + + This method is called once by the pairlistmanager in the refresh_pairlist() + method to supply the starting pairlist for the chain of the Pairlist Handlers. + Pairlist Filters (those Pairlist Handlers that cannot be used at the first + position in the chain) shall not override this base implementation -- + it will raise the exception if a Pairlist Handler is used at the first + position in the chain. + + :param cached_pairlist: Previously generated pairlist (cached) + :param tickers: Tickers (from exchange.get_tickers()). + :return: List of pairs + """ + raise OperationalException("This Pairlist Handler should not be used " + "at the first position in the list of Pairlist Handlers.") + def filter_pairlist(self, pairlist: List[str], tickers: Dict) -> List[str]: """ Filters and sorts pairlist and returns the whitelist again. diff --git a/freqtrade/pairlist/StaticPairList.py b/freqtrade/pairlist/StaticPairList.py index 07e559168..b5c1bc767 100644 --- a/freqtrade/pairlist/StaticPairList.py +++ b/freqtrade/pairlist/StaticPairList.py @@ -4,8 +4,9 @@ Static Pair List provider Provides pair white list as it configured in config """ import logging -from typing import Dict, List +from typing import Any, Dict, List +from freqtrade.exceptions import OperationalException from freqtrade.pairlist.IPairList import IPairList @@ -14,6 +15,15 @@ logger = logging.getLogger(__name__) class StaticPairList(IPairList): + def __init__(self, exchange, pairlistmanager, + config: Dict[str, Any], pairlistconfig: Dict[str, Any], + pairlist_pos: int) -> None: + super().__init__(exchange, pairlistmanager, config, pairlistconfig, pairlist_pos) + + if self._pairlist_pos != 0: + raise OperationalException(f"{self.name} can only be used in the first position " + "in the list of Pairlist Handlers.") + @property def needstickers(self) -> bool: """ @@ -30,6 +40,15 @@ class StaticPairList(IPairList): """ return f"{self.name}" + def gen_pairlist(self, cached_pairlist: List[str], tickers: Dict) -> List[str]: + """ + Generate the pairlist + :param cached_pairlist: Previously generated pairlist (cached) + :param tickers: Tickers (from exchange.get_tickers()). + :return: List of pairs + """ + return self._whitelist_for_active_markets(self._config['exchange']['pair_whitelist']) + def filter_pairlist(self, pairlist: List[str], tickers: Dict) -> List[str]: """ Filters and sorts pairlist and returns the whitelist again. @@ -38,4 +57,4 @@ class StaticPairList(IPairList): :param tickers: Tickers (from exchange.get_tickers()). May be cached. :return: new whitelist """ - return self._whitelist_for_active_markets(self._config['exchange']['pair_whitelist']) + return pairlist diff --git a/freqtrade/pairlist/VolumePairList.py b/freqtrade/pairlist/VolumePairList.py index 6f39ae6d6..d32be3dc9 100644 --- a/freqtrade/pairlist/VolumePairList.py +++ b/freqtrade/pairlist/VolumePairList.py @@ -68,6 +68,31 @@ class VolumePairList(IPairList): """ return f"{self.name} - top {self._pairlistconfig['number_assets']} volume pairs." + def gen_pairlist(self, cached_pairlist: List[str], tickers: Dict) -> List[str]: + """ + Generate the pairlist + :param cached_pairlist: Previously generated pairlist (cached) + :param tickers: Tickers (from exchange.get_tickers()). + :return: List of pairs + """ + # Generate dynamic whitelist + # Must always run if this pairlist is not the first in the list. + if self._last_refresh + self.refresh_period < datetime.now().timestamp(): + self._last_refresh = int(datetime.now().timestamp()) + + # Use fresh pairlist + # Check if pair quote currency equals to the stake currency. + filtered_tickers = [ + v for k, v in tickers.items() + if (self._exchange.get_pair_quote_currency(k) == self._stake_currency + and v[self._sort_key] is not None)] + pairlist = [s['symbol'] for s in filtered_tickers] + else: + # Use the cached pairlist if it's not time yet to refresh + pairlist = cached_pairlist + + return pairlist + def filter_pairlist(self, pairlist: List[str], tickers: Dict) -> List[str]: """ Filters and sorts pairlist and returns the whitelist again. @@ -76,37 +101,8 @@ class VolumePairList(IPairList): :param tickers: Tickers (from exchange.get_tickers()). May be cached. :return: new whitelist """ - # Generate dynamic whitelist - # Must always run if this pairlist is not the first in the list. - if (self._pairlist_pos != 0 or - (self._last_refresh + self.refresh_period < datetime.now().timestamp())): - - self._last_refresh = int(datetime.now().timestamp()) - pairs = self._gen_pair_whitelist(pairlist, tickers) - else: - pairs = pairlist - - self.log_on_refresh(logger.info, f"Searching {self._number_pairs} pairs: {pairs}") - - return pairs - - def _gen_pair_whitelist(self, pairlist: List[str], tickers: Dict) -> List[str]: - """ - Updates the whitelist with with a dynamically generated list - :param pairlist: pairlist to filter or sort - :param tickers: Tickers (from exchange.get_tickers()). - :return: List of pairs - """ - if self._pairlist_pos == 0: - # If VolumePairList is the first in the list, use fresh pairlist - # Check if pair quote currency equals to the stake currency. - filtered_tickers = [ - v for k, v in tickers.items() - if (self._exchange.get_pair_quote_currency(k) == self._stake_currency - and v[self._sort_key] is not None)] - else: - # If other pairlist is in front, use the incoming pairlist. - filtered_tickers = [v for k, v in tickers.items() if k in pairlist] + # Use the incoming pairlist. + filtered_tickers = [v for k, v in tickers.items() if k in pairlist] if self._min_value > 0: filtered_tickers = [ @@ -120,4 +116,6 @@ class VolumePairList(IPairList): # Limit pairlist to the requested number of pairs pairs = pairs[:self._number_pairs] + self.log_on_refresh(logger.info, f"Searching {self._number_pairs} pairs: {pairs}") + return pairs diff --git a/freqtrade/pairlist/pairlistmanager.py b/freqtrade/pairlist/pairlistmanager.py index 98878bcb0..f532f2cd0 100644 --- a/freqtrade/pairlist/pairlistmanager.py +++ b/freqtrade/pairlist/pairlistmanager.py @@ -87,6 +87,9 @@ class PairListManager(): # Adjust whitelist if filters are using tickers pairlist = self._prepare_whitelist(self._whitelist.copy(), tickers) + # Generate the pairlist with first Pairlist Handler in the chain + pairlist = self._pairlist_handlers[0].gen_pairlist(self._whitelist, tickers) + # Process all Pairlist Handlers in the chain for pairlist_handler in self._pairlist_handlers: pairlist = pairlist_handler.filter_pairlist(pairlist, tickers) diff --git a/tests/pairlist/test_pairlist.py b/tests/pairlist/test_pairlist.py index e9e688b78..421f06911 100644 --- a/tests/pairlist/test_pairlist.py +++ b/tests/pairlist/test_pairlist.py @@ -19,7 +19,8 @@ def whitelist_conf(default_conf): 'TKN/BTC', 'TRST/BTC', 'SWT/BTC', - 'BCC/BTC' + 'BCC/BTC', + 'HOT/BTC', ] default_conf['exchange']['pair_blacklist'] = [ 'BLK/BTC' @@ -226,10 +227,12 @@ def test_VolumePairList_refresh_empty(mocker, markets_empty, whitelist_conf): "ETH", []), # Precisionfilter and quote volume ([{"method": "VolumePairList", "number_assets": 5, "sort_key": "quoteVolume"}, - {"method": "PrecisionFilter"}], "BTC", ['ETH/BTC', 'TKN/BTC', 'LTC/BTC', 'XRP/BTC']), + {"method": "PrecisionFilter"}], + "BTC", ['ETH/BTC', 'TKN/BTC', 'LTC/BTC', 'XRP/BTC']), # Precisionfilter bid ([{"method": "VolumePairList", "number_assets": 5, "sort_key": "bidVolume"}, - {"method": "PrecisionFilter"}], "BTC", ['FUEL/BTC', 'XRP/BTC', 'LTC/BTC', 'TKN/BTC']), + {"method": "PrecisionFilter"}], + "BTC", ['FUEL/BTC', 'XRP/BTC', 'LTC/BTC', 'TKN/BTC']), # PriceFilter and VolumePairList ([{"method": "VolumePairList", "number_assets": 5, "sort_key": "quoteVolume"}, {"method": "PriceFilter", "low_price_ratio": 0.03}], @@ -249,11 +252,11 @@ def test_VolumePairList_refresh_empty(mocker, markets_empty, whitelist_conf): "BTC", ['ETH/BTC', 'TKN/BTC', 'LTC/BTC']), # StaticPairlist only ([{"method": "StaticPairList"}], - "BTC", ['ETH/BTC', 'TKN/BTC']), + "BTC", ['ETH/BTC', 'TKN/BTC', 'HOT/BTC']), # Static Pairlist before VolumePairList - sorting changes ([{"method": "StaticPairList"}, {"method": "VolumePairList", "number_assets": 5, "sort_key": "bidVolume"}], - "BTC", ['TKN/BTC', 'ETH/BTC']), + "BTC", ['HOT/BTC', 'TKN/BTC', 'ETH/BTC']), # SpreadFilter ([{"method": "VolumePairList", "number_assets": 5, "sort_key": "quoteVolume"}, {"method": "SpreadFilter", "max_spread_ratio": 0.005}], @@ -269,7 +272,39 @@ def test_VolumePairList_refresh_empty(mocker, markets_empty, whitelist_conf): # ShuffleFilter, no seed ([{"method": "VolumePairList", "number_assets": 5, "sort_key": "quoteVolume"}, {"method": "ShuffleFilter"}], - "USDT", 3), + "USDT", 3), # whitelist_result is integer -- check only lenght of randomized pairlist + # PrecisionFilter after StaticPairList + ([{"method": "StaticPairList"}, + {"method": "PrecisionFilter"}], + "BTC", ['ETH/BTC', 'TKN/BTC']), + # PrecisionFilter only + ([{"method": "PrecisionFilter"}], + "BTC", 'filter_at_the_beginning'), # OperationalException expected + # PriceFilter after StaticPairList + ([{"method": "StaticPairList"}, + {"method": "PriceFilter", "low_price_ratio": 0.02}], + "BTC", ['ETH/BTC', 'TKN/BTC']), + # PriceFilter only + ([{"method": "PriceFilter", "low_price_ratio": 0.02}], + "BTC", 'filter_at_the_beginning'), # OperationalException expected + # ShuffleFilter after StaticPairList + ([{"method": "StaticPairList"}, + {"method": "ShuffleFilter", "seed": 42}], + "BTC", ['TKN/BTC', 'ETH/BTC', 'HOT/BTC']), + # ShuffleFilter only + ([{"method": "ShuffleFilter", "seed": 42}], + "BTC", 'filter_at_the_beginning'), # OperationalException expected + # SpreadFilter after StaticPairList + ([{"method": "StaticPairList"}, + {"method": "SpreadFilter", "max_spread_ratio": 0.005}], + "BTC", ['ETH/BTC', 'TKN/BTC']), + # SpreadFilter only + ([{"method": "SpreadFilter", "max_spread_ratio": 0.005}], + "BTC", 'filter_at_the_beginning'), # OperationalException expected + # Static Pairlist after VolumePairList, on a non-first position + ([{"method": "VolumePairList", "number_assets": 5, "sort_key": "bidVolume"}, + {"method": "StaticPairList"}], + "BTC", 'static_in_the_middle'), ]) def test_VolumePairList_whitelist_gen(mocker, whitelist_conf, shitcoinmarkets, tickers, pairlists, base_currency, whitelist_result, @@ -278,39 +313,53 @@ def test_VolumePairList_whitelist_gen(mocker, whitelist_conf, shitcoinmarkets, t whitelist_conf['stake_currency'] = base_currency mocker.patch('freqtrade.exchange.Exchange.exchange_has', MagicMock(return_value=True)) - freqtrade = get_patched_freqtradebot(mocker, whitelist_conf) + if whitelist_result == 'static_in_the_middle': + with pytest.raises(OperationalException, + match=r"StaticPairList can only be used in the first position " + r"in the list of Pairlist Handlers."): + freqtrade = get_patched_freqtradebot(mocker, whitelist_conf) + return + + freqtrade = get_patched_freqtradebot(mocker, whitelist_conf) mocker.patch.multiple('freqtrade.exchange.Exchange', get_tickers=tickers, markets=PropertyMock(return_value=shitcoinmarkets), ) - freqtrade.pairlists.refresh_pairlist() - whitelist = freqtrade.pairlists.whitelist - - assert isinstance(whitelist, list) - - # Verify length of pairlist matches (used for ShuffleFilter without seed) - if type(whitelist_result) is list: - assert whitelist == whitelist_result + # Set whitelist_result to None if pairlist is invalid and should produce exception + if whitelist_result == 'filter_at_the_beginning': + with pytest.raises(OperationalException, + match=r"This Pairlist Handler should not be used at the first position " + r"in the list of Pairlist Handlers."): + freqtrade.pairlists.refresh_pairlist() else: - len(whitelist) == whitelist_result + freqtrade.pairlists.refresh_pairlist() + whitelist = freqtrade.pairlists.whitelist - for pairlist in pairlists: - if pairlist['method'] == 'PrecisionFilter' and whitelist_result: - assert log_has_re(r'^Removed .* from whitelist, because stop price .* ' - r'would be <= stop limit.*', caplog) - if pairlist['method'] == 'PriceFilter' and whitelist_result: - assert (log_has_re(r'^Removed .* from whitelist, because 1 unit is .*%$', caplog) or - log_has_re(r"^Removed .* from whitelist, because ticker\['last'\] is empty.*", - caplog)) - if pairlist['method'] == 'VolumePairList': - logmsg = ("DEPRECATED: using any key other than quoteVolume for " - "VolumePairList is deprecated.") - if pairlist['sort_key'] != 'quoteVolume': - assert log_has(logmsg, caplog) - else: - assert not log_has(logmsg, caplog) + assert isinstance(whitelist, list) + + # Verify length of pairlist matches (used for ShuffleFilter without seed) + if type(whitelist_result) is list: + assert whitelist == whitelist_result + else: + len(whitelist) == whitelist_result + + for pairlist in pairlists: + if pairlist['method'] == 'PrecisionFilter' and whitelist_result: + assert log_has_re(r'^Removed .* from whitelist, because stop price .* ' + r'would be <= stop limit.*', caplog) + if pairlist['method'] == 'PriceFilter' and whitelist_result: + assert (log_has_re(r'^Removed .* from whitelist, because 1 unit is .*%$', caplog) or + log_has_re(r"^Removed .* from whitelist, because ticker\['last'\] " + r"is empty.*", caplog)) + if pairlist['method'] == 'VolumePairList': + logmsg = ("DEPRECATED: using any key other than quoteVolume for " + "VolumePairList is deprecated.") + if pairlist['sort_key'] != 'quoteVolume': + assert log_has(logmsg, caplog) + else: + assert not log_has(logmsg, caplog) def test_gen_pair_whitelist_not_supported(mocker, default_conf, tickers) -> None: