adelorenzo/stable
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Merge pull request #3358 from hroff-1902/refactor_generate_pairlist

Browse Source 
Split the pairlist generation logic and filtering
This commit is contained in:
Matthias 2020-05-29 14:16:15 +02:00 committed by GitHub
commit 9f8b21de4a
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
6 changed files with 179 additions and 80 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

43
docs/developer.md
View File

@ -92,13 +92,13 @@ docker-compose exec freqtrade_develop /bin/bash
You have a great idea for a new pair selection algorithm you would like to try out? Great.
Hopefully you also want to contribute this back upstream.
Whatever your motivations are - This should get you off the ground in trying to develop a new Pairlist provider.
Whatever your motivations are - This should get you off the ground in trying to develop a new Pairlist Handler.
First of all, have a look at the [VolumePairList](https://github.com/freqtrade/freqtrade/blob/develop/freqtrade/pairlist/VolumePairList.py) provider, and best copy this file with a name of your new Pairlist Provider.
First of all, have a look at the [VolumePairList](https://github.com/freqtrade/freqtrade/blob/develop/freqtrade/pairlist/VolumePairList.py) Handler, and best copy this file with a name of your new Pairlist Handler.
This is a simple provider, which however serves as a good example on how to start developing.
This is a simple Handler, which however serves as a good example on how to start developing.
Next, modify the classname of the provider (ideally align this with the Filename).
Next, modify the classname of the Handler (ideally align this with the module filename).
The base-class provides an instance of the exchange (`self._exchange`) the pairlist manager (`self._pairlistmanager`), as well as the main configuration (`self._config`), the pairlist dedicated configuration (`self._pairlistconfig`) and the absolute position within the list of pairlists.
@ -114,28 +114,44 @@ Now, let's step through the methods which require actions:
#### Pairlist configuration
Configuration for PairListProvider is done in the bot configuration file in the element `"pairlist"`.
This Pairlist-object may contain configurations with additional configurations for the configured pairlist.
By convention, `"number_assets"` is used to specify the maximum number of pairs to keep in the whitelist. Please follow this to ensure a consistent user experience.
Configuration for the chain of Pairlist Handlers is done in the bot configuration file in the element `"pairlists"`, an array of configuration parameters for each Pairlist Handlers in the chain.
Additional elements can be configured as needed. `VolumePairList` uses `"sort_key"` to specify the sorting value - however feel free to specify whatever is necessary for your great algorithm to be successfull and dynamic.
By convention, `"number_assets"` is used to specify the maximum number of pairs to keep in the pairlist. Please follow this to ensure a consistent user experience.
Additional parameters can be configured as needed. For instance, `VolumePairList` uses `"sort_key"` to specify the sorting value - however feel free to specify whatever is necessary for your great algorithm to be successfull and dynamic.
#### short_desc
Returns a description used for Telegram messages.
This should contain the name of the Provider, as well as a short description containing the number of assets. Please follow the format `"PairlistName - top/bottom X pairs"`.
This should contain the name of the Pairlist Handler, as well as a short description containing the number of assets. Please follow the format `"PairlistName - top/bottom X pairs"`.
#### gen_pairlist
Override this method if the Pairlist Handler can be used as the leading Pairlist Handler in the chain, defining the initial pairlist which is then handled by all Pairlist Handlers in the chain. Examples are `StaticPairList` and `VolumePairList`.
This is called with each iteration of the bot (only if the Pairlist Handler is at the first location) - so consider implementing caching for compute/network heavy calculations.
It must return the resulting pairlist (which may then be passed into the chain of Pairlist Handlers).
Validations are optional, the parent class exposes a `_verify_blacklist(pairlist)` and `_whitelist_for_active_markets(pairlist)` to do default filtering. Use this if you limit your result to a certain number of pairs - so the endresult is not shorter than expected.
#### filter_pairlist
Override this method and run all calculations needed in this method.
This method is called for each Pairlist Handler in the chain by the pairlist manager.
This is called with each iteration of the bot - so consider implementing caching for compute/network heavy calculations.
It get's passed a pairlist (which can be the result of previous pairlists) as well as `tickers`, a pre-fetched version of `get_tickers()`.
It must return the resulting pairlist (which may then be passed into the next pairlist filter).
The default implementation in the base class simply calls the `_validate_pair()` method for each pair in the pairlist, but you may override it. So you should either implement the `_validate_pair()` in your Pairlist Handler or override `filter_pairlist()` to do something else.
If overridden, it must return the resulting pairlist (which may then be passed into the next Pairlist Handler in the chain).
Validations are optional, the parent class exposes a `_verify_blacklist(pairlist)` and `_whitelist_for_active_markets(pairlist)` to do default filters. Use this if you limit your result to a certain number of pairs - so the endresult is not shorter than expected.
In `VolumePairList`, this implements different methods of sorting, does early validation so only the expected number of pairs is returned.
##### sample
``` python
@ -145,11 +161,6 @@ Validations are optional, the parent class exposes a `_verify_blacklist(pairlist
return pairs
```
#### _gen_pair_whitelist
This is a simple method used by `VolumePairList` - however serves as a good example.
In VolumePairList, this implements different methods of sorting, does early validation so only the expected number of pairs is returned.
## Implement a new Exchange (WIP)
!!! Note

19
freqtrade/pairlist/IPairList.py
View File

@ -8,6 +8,7 @@ from typing import Any, Dict, List
from cachetools import TTLCache, cached
from freqtrade.exceptions import OperationalException
from freqtrade.exchange import market_is_active
@ -90,6 +91,24 @@ class IPairList(ABC):
"""
raise NotImplementedError()
def gen_pairlist(self, cached_pairlist: List[str], tickers: Dict) -> List[str]:
"""
Generate the pairlist.
This method is called once by the pairlistmanager in the refresh_pairlist()
method to supply the starting pairlist for the chain of the Pairlist Handlers.
Pairlist Filters (those Pairlist Handlers that cannot be used at the first
position in the chain) shall not override this base implementation --
it will raise the exception if a Pairlist Handler is used at the first
position in the chain.
:param cached_pairlist: Previously generated pairlist (cached)
:param tickers: Tickers (from exchange.get_tickers()).
:return: List of pairs
"""
raise OperationalException("This Pairlist Handler should not be used "
"at the first position in the list of Pairlist Handlers.")
def filter_pairlist(self, pairlist: List[str], tickers: Dict) -> List[str]:
"""
Filters and sorts pairlist and returns the whitelist again.

23
freqtrade/pairlist/StaticPairList.py
View File

@ -4,8 +4,9 @@ Static Pair List provider
Provides pair white list as it configured in config
"""
import logging
from typing import Dict, List
from typing import Any, Dict, List
from freqtrade.exceptions import OperationalException
from freqtrade.pairlist.IPairList import IPairList
@ -14,6 +15,15 @@ logger = logging.getLogger(__name__)
class StaticPairList(IPairList):
def __init__(self, exchange, pairlistmanager,
config: Dict[str, Any], pairlistconfig: Dict[str, Any],
pairlist_pos: int) -> None:
super().__init__(exchange, pairlistmanager, config, pairlistconfig, pairlist_pos)
if self._pairlist_pos != 0:
raise OperationalException(f"{self.name} can only be used in the first position "
"in the list of Pairlist Handlers.")
@property
def needstickers(self) -> bool:
"""
@ -30,6 +40,15 @@ class StaticPairList(IPairList):
"""
return f"{self.name}"
def gen_pairlist(self, cached_pairlist: List[str], tickers: Dict) -> List[str]:
"""
Generate the pairlist
:param cached_pairlist: Previously generated pairlist (cached)
:param tickers: Tickers (from exchange.get_tickers()).
:return: List of pairs
"""
return self._whitelist_for_active_markets(self._config['exchange']['pair_whitelist'])
def filter_pairlist(self, pairlist: List[str], tickers: Dict) -> List[str]:
"""
Filters and sorts pairlist and returns the whitelist again.
@ -38,4 +57,4 @@ class StaticPairList(IPairList):
:param tickers: Tickers (from exchange.get_tickers()). May be cached.
:return: new whitelist
"""
return self._whitelist_for_active_markets(self._config['exchange']['pair_whitelist'])
return pairlist

60
freqtrade/pairlist/VolumePairList.py
View File

@ -68,6 +68,31 @@ class VolumePairList(IPairList):
"""
return f"{self.name} - top {self._pairlistconfig['number_assets']} volume pairs."
def gen_pairlist(self, cached_pairlist: List[str], tickers: Dict) -> List[str]:
"""
Generate the pairlist
:param cached_pairlist: Previously generated pairlist (cached)
:param tickers: Tickers (from exchange.get_tickers()).
:return: List of pairs
"""
# Generate dynamic whitelist
# Must always run if this pairlist is not the first in the list.
if self._last_refresh + self.refresh_period < datetime.now().timestamp():
self._last_refresh = int(datetime.now().timestamp())
# Use fresh pairlist
# Check if pair quote currency equals to the stake currency.
filtered_tickers = [
v for k, v in tickers.items()
if (self._exchange.get_pair_quote_currency(k) == self._stake_currency
and v[self._sort_key] is not None)]
pairlist = [s['symbol'] for s in filtered_tickers]
else:
# Use the cached pairlist if it's not time yet to refresh
pairlist = cached_pairlist
return pairlist
def filter_pairlist(self, pairlist: List[str], tickers: Dict) -> List[str]:
"""
Filters and sorts pairlist and returns the whitelist again.
@ -76,37 +101,8 @@ class VolumePairList(IPairList):
:param tickers: Tickers (from exchange.get_tickers()). May be cached.
:return: new whitelist
"""
# Generate dynamic whitelist
# Must always run if this pairlist is not the first in the list.
if (self._pairlist_pos != 0 or
(self._last_refresh + self.refresh_period < datetime.now().timestamp())):
self._last_refresh = int(datetime.now().timestamp())
pairs = self._gen_pair_whitelist(pairlist, tickers)
else:
pairs = pairlist
self.log_on_refresh(logger.info, f"Searching {self._number_pairs} pairs: {pairs}")
return pairs
def _gen_pair_whitelist(self, pairlist: List[str], tickers: Dict) -> List[str]:
"""
Updates the whitelist with with a dynamically generated list
:param pairlist: pairlist to filter or sort
:param tickers: Tickers (from exchange.get_tickers()).
:return: List of pairs
"""
if self._pairlist_pos == 0:
# If VolumePairList is the first in the list, use fresh pairlist
# Check if pair quote currency equals to the stake currency.
filtered_tickers = [
v for k, v in tickers.items()
if (self._exchange.get_pair_quote_currency(k) == self._stake_currency
and v[self._sort_key] is not None)]
else:
# If other pairlist is in front, use the incoming pairlist.
filtered_tickers = [v for k, v in tickers.items() if k in pairlist]
# Use the incoming pairlist.
filtered_tickers = [v for k, v in tickers.items() if k in pairlist]
if self._min_value > 0:
filtered_tickers = [
@ -120,4 +116,6 @@ class VolumePairList(IPairList):
# Limit pairlist to the requested number of pairs
pairs = pairs[:self._number_pairs]
self.log_on_refresh(logger.info, f"Searching {self._number_pairs} pairs: {pairs}")
return pairs

3
freqtrade/pairlist/pairlistmanager.py
View File

@ -87,6 +87,9 @@ class PairListManager():
# Adjust whitelist if filters are using tickers
pairlist = self._prepare_whitelist(self._whitelist.copy(), tickers)
# Generate the pairlist with first Pairlist Handler in the chain
pairlist = self._pairlist_handlers[0].gen_pairlist(self._whitelist, tickers)
# Process all Pairlist Handlers in the chain
for pairlist_handler in self._pairlist_handlers:
pairlist = pairlist_handler.filter_pairlist(pairlist, tickers)

111
tests/pairlist/test_pairlist.py
View File

@ -19,7 +19,8 @@ def whitelist_conf(default_conf):
'TKN/BTC',
'TRST/BTC',
'SWT/BTC',
'BCC/BTC'
'BCC/BTC',
'HOT/BTC',
]
default_conf['exchange']['pair_blacklist'] = [
'BLK/BTC'
@ -226,10 +227,12 @@ def test_VolumePairList_refresh_empty(mocker, markets_empty, whitelist_conf):
"ETH", []),
# Precisionfilter and quote volume
([{"method": "VolumePairList", "number_assets": 5, "sort_key": "quoteVolume"},
{"method": "PrecisionFilter"}], "BTC", ['ETH/BTC', 'TKN/BTC', 'LTC/BTC', 'XRP/BTC']),
{"method": "PrecisionFilter"}],
"BTC", ['ETH/BTC', 'TKN/BTC', 'LTC/BTC', 'XRP/BTC']),
# Precisionfilter bid
([{"method": "VolumePairList", "number_assets": 5, "sort_key": "bidVolume"},
{"method": "PrecisionFilter"}], "BTC", ['FUEL/BTC', 'XRP/BTC', 'LTC/BTC', 'TKN/BTC']),
{"method": "PrecisionFilter"}],
"BTC", ['FUEL/BTC', 'XRP/BTC', 'LTC/BTC', 'TKN/BTC']),
# PriceFilter and VolumePairList
([{"method": "VolumePairList", "number_assets": 5, "sort_key": "quoteVolume"},
{"method": "PriceFilter", "low_price_ratio": 0.03}],
@ -249,11 +252,11 @@ def test_VolumePairList_refresh_empty(mocker, markets_empty, whitelist_conf):
"BTC", ['ETH/BTC', 'TKN/BTC', 'LTC/BTC']),
# StaticPairlist only
([{"method": "StaticPairList"}],
"BTC", ['ETH/BTC', 'TKN/BTC']),
"BTC", ['ETH/BTC', 'TKN/BTC', 'HOT/BTC']),
# Static Pairlist before VolumePairList - sorting changes
([{"method": "StaticPairList"},
{"method": "VolumePairList", "number_assets": 5, "sort_key": "bidVolume"}],
"BTC", ['TKN/BTC', 'ETH/BTC']),
"BTC", ['HOT/BTC', 'TKN/BTC', 'ETH/BTC']),
# SpreadFilter
([{"method": "VolumePairList", "number_assets": 5, "sort_key": "quoteVolume"},
{"method": "SpreadFilter", "max_spread_ratio": 0.005}],
@ -269,7 +272,39 @@ def test_VolumePairList_refresh_empty(mocker, markets_empty, whitelist_conf):
# ShuffleFilter, no seed
([{"method": "VolumePairList", "number_assets": 5, "sort_key": "quoteVolume"},
{"method": "ShuffleFilter"}],
"USDT", 3),
"USDT", 3), # whitelist_result is integer -- check only lenght of randomized pairlist
# PrecisionFilter after StaticPairList
([{"method": "StaticPairList"},
{"method": "PrecisionFilter"}],
"BTC", ['ETH/BTC', 'TKN/BTC']),
# PrecisionFilter only
([{"method": "PrecisionFilter"}],
"BTC", 'filter_at_the_beginning'), # OperationalException expected
# PriceFilter after StaticPairList
([{"method": "StaticPairList"},
{"method": "PriceFilter", "low_price_ratio": 0.02}],
"BTC", ['ETH/BTC', 'TKN/BTC']),
# PriceFilter only
([{"method": "PriceFilter", "low_price_ratio": 0.02}],
"BTC", 'filter_at_the_beginning'), # OperationalException expected
# ShuffleFilter after StaticPairList
([{"method": "StaticPairList"},
{"method": "ShuffleFilter", "seed": 42}],
"BTC", ['TKN/BTC', 'ETH/BTC', 'HOT/BTC']),
# ShuffleFilter only
([{"method": "ShuffleFilter", "seed": 42}],
"BTC", 'filter_at_the_beginning'), # OperationalException expected
# SpreadFilter after StaticPairList
([{"method": "StaticPairList"},
{"method": "SpreadFilter", "max_spread_ratio": 0.005}],
"BTC", ['ETH/BTC', 'TKN/BTC']),
# SpreadFilter only
([{"method": "SpreadFilter", "max_spread_ratio": 0.005}],
"BTC", 'filter_at_the_beginning'), # OperationalException expected
# Static Pairlist after VolumePairList, on a non-first position
([{"method": "VolumePairList", "number_assets": 5, "sort_key": "bidVolume"},
{"method": "StaticPairList"}],
"BTC", 'static_in_the_middle'),
])
def test_VolumePairList_whitelist_gen(mocker, whitelist_conf, shitcoinmarkets, tickers,
pairlists, base_currency, whitelist_result,
@ -278,39 +313,53 @@ def test_VolumePairList_whitelist_gen(mocker, whitelist_conf, shitcoinmarkets, t
whitelist_conf['stake_currency'] = base_currency
mocker.patch('freqtrade.exchange.Exchange.exchange_has', MagicMock(return_value=True))
freqtrade = get_patched_freqtradebot(mocker, whitelist_conf)
if whitelist_result == 'static_in_the_middle':
with pytest.raises(OperationalException,
match=r"StaticPairList can only be used in the first position "
r"in the list of Pairlist Handlers."):
freqtrade = get_patched_freqtradebot(mocker, whitelist_conf)
return
freqtrade = get_patched_freqtradebot(mocker, whitelist_conf)
mocker.patch.multiple('freqtrade.exchange.Exchange',
get_tickers=tickers,
markets=PropertyMock(return_value=shitcoinmarkets),
)
freqtrade.pairlists.refresh_pairlist()
whitelist = freqtrade.pairlists.whitelist
assert isinstance(whitelist, list)
# Verify length of pairlist matches (used for ShuffleFilter without seed)
if type(whitelist_result) is list:
assert whitelist == whitelist_result
# Set whitelist_result to None if pairlist is invalid and should produce exception
if whitelist_result == 'filter_at_the_beginning':
with pytest.raises(OperationalException,
match=r"This Pairlist Handler should not be used at the first position "
r"in the list of Pairlist Handlers."):
freqtrade.pairlists.refresh_pairlist()
else:
len(whitelist) == whitelist_result
freqtrade.pairlists.refresh_pairlist()
whitelist = freqtrade.pairlists.whitelist
for pairlist in pairlists:
if pairlist['method'] == 'PrecisionFilter' and whitelist_result:
assert log_has_re(r'^Removed .* from whitelist, because stop price .* '
r'would be <= stop limit.*', caplog)
if pairlist['method'] == 'PriceFilter' and whitelist_result:
assert (log_has_re(r'^Removed .* from whitelist, because 1 unit is .*%$', caplog) or
log_has_re(r"^Removed .* from whitelist, because ticker\['last'\] is empty.*",
caplog))
if pairlist['method'] == 'VolumePairList':
logmsg = ("DEPRECATED: using any key other than quoteVolume for "
"VolumePairList is deprecated.")
if pairlist['sort_key'] != 'quoteVolume':
assert log_has(logmsg, caplog)
else:
assert not log_has(logmsg, caplog)
assert isinstance(whitelist, list)
# Verify length of pairlist matches (used for ShuffleFilter without seed)
if type(whitelist_result) is list:
assert whitelist == whitelist_result
else:
len(whitelist) == whitelist_result
for pairlist in pairlists:
if pairlist['method'] == 'PrecisionFilter' and whitelist_result:
assert log_has_re(r'^Removed .* from whitelist, because stop price .* '
r'would be <= stop limit.*', caplog)
if pairlist['method'] == 'PriceFilter' and whitelist_result:
assert (log_has_re(r'^Removed .* from whitelist, because 1 unit is .*%$', caplog) or
log_has_re(r"^Removed .* from whitelist, because ticker\['last'\] "
r"is empty.*", caplog))
if pairlist['method'] == 'VolumePairList':
logmsg = ("DEPRECATED: using any key other than quoteVolume for "
"VolumePairList is deprecated.")
if pairlist['sort_key'] != 'quoteVolume':
assert log_has(logmsg, caplog)
else:
assert not log_has(logmsg, caplog)
def test_gen_pair_whitelist_not_supported(mocker, default_conf, tickers) -> None: