diff --git a/docs/assets/binance_futures_settings.png b/docs/assets/binance_futures_settings.png new file mode 100644 index 000000000..a3f7a2c70 Binary files /dev/null and b/docs/assets/binance_futures_settings.png differ diff --git a/docs/exchanges.md b/docs/exchanges.md index a9ba16c64..980d102b2 100644 --- a/docs/exchanges.md +++ b/docs/exchanges.md @@ -60,11 +60,18 @@ Binance supports [time_in_force](configuration.md#understand-order_time_in_force Binance supports `stoploss_on_exchange` and uses `stop-loss-limit` orders. It provides great advantages, so we recommend to benefit from it by enabling stoploss on exchange. On futures, Binance supports both `stop-limit` as well as `stop-market` orders. You can use either `"limit"` or `"market"` in the `order_types.stoploss` configuration setting to decide which type to use. -### Binance Blacklist +### Binance Blacklist recommendation For Binance, it is suggested to add `"BNB/"` to your blacklist to avoid issues, unless you are willing to maintain enough extra `BNB` on the account or unless you're willing to disable using `BNB` for fees. Binance accounts may use `BNB` for fees, and if a trade happens to be on `BNB`, further trades may consume this position and make the initial BNB trade unsellable as the expected amount is not there anymore. +### Binance sites + +Binance has been split into 2, and users must use the correct ccxt exchange ID for their exchange, otherwise API keys are not recognized. + +* [binance.com](https://www.binance.com/) - International users. Use exchange id: `binance`. +* [binance.us](https://www.binance.us/) - US based users. Use exchange id: `binanceus`. + ### Binance Futures Binance has specific (unfortunately complex) [Futures Trading Quantitative Rules](https://www.binance.com/en/support/faq/4f462ebe6ff445d4a170be7d9e897272) which need to be followed, and which prohibit a too low stake-amount (among others) for too many orders. @@ -87,12 +94,14 @@ When trading on Binance Futures market, orderbook must be used because there is }, ``` -### Binance sites +#### Binance futures settings -Binance has been split into 2, and users must use the correct ccxt exchange ID for their exchange, otherwise API keys are not recognized. +Users will also have to have the futures-setting "Position Mode" set to "One-way Mode", and "Asset Mode" set to "Single-Asset Mode". +These settings will be checked on startup, and freqtrade will show an error if this setting is wrong. -* [binance.com](https://www.binance.com/) - International users. Use exchange id: `binance`. -* [binance.us](https://www.binance.us/) - US based users. Use exchange id: `binanceus`. +![Binance futures settings](assets/binance_futures_settings.png) + +Freqtrade will not attempt to change these settings. ## Kraken diff --git a/docs/includes/pairlists.md b/docs/includes/pairlists.md index 0f55c1b79..7dff75a02 100644 --- a/docs/includes/pairlists.md +++ b/docs/includes/pairlists.md @@ -22,6 +22,7 @@ You may also use something like `.*DOWN/BTC` or `.*UP/BTC` to exclude leveraged * [`StaticPairList`](#static-pair-list) (default, if not configured differently) * [`VolumePairList`](#volume-pair-list) +* [`ProducerPairList`](#producerpairlist) * [`AgeFilter`](#agefilter) * [`OffsetFilter`](#offsetfilter) * [`PerformanceFilter`](#performancefilter) @@ -84,7 +85,7 @@ Filtering instances (not the first position in the list) will not apply any cach You can define a minimum volume with `min_value` - which will filter out pairs with a volume lower than the specified value in the specified timerange. -### VolumePairList Advanced mode +##### VolumePairList Advanced mode `VolumePairList` can also operate in an advanced mode to build volume over a given timerange of specified candle size. It utilizes exchange historical candle data, builds a typical price (calculated by (open+high+low)/3) and multiplies the typical price with every candle's volume. The sum is the `quoteVolume` over the given range. This allows different scenarios, for a more smoothened volume, when using longer ranges with larger candle sizes, or the opposite when using a short range with small candles. @@ -146,6 +147,32 @@ More sophisticated approach can be used, by using `lookback_timeframe` for candl !!! Note `VolumePairList` does not support backtesting mode. +#### ProducerPairList + +With `ProducerPairList`, you can reuse the pairlist from a [Producer](producer-consumer.md) without explicitly defining the pairlist on each consumer. + +[Consumer mode](producer-consumer.md) is required for this pairlist to work. + +The pairlist will perform a check on active pairs against the current exchange configuration to avoid attempting to trade on invalid markets. + +You can limit the length of the pairlist with the optional parameter `number_assets`. Using `"number_assets"=0` or omitting this key will result in the reuse of all producer pairs valid for the current setup. + +```json +"pairlists": [ + { + "method": "ProducerPairList", + "number_assets": 5, + "producer_name": "default", + } +], +``` + + +!!! Tip "Combining pairlists" + This pairlist can be combined with all other pairlists and filters for further pairlist reduction, and can also act as an "additional" pairlist, on top of already defined pairs. + `ProducerPairList` can also be used multiple times in sequence, combining the pairs from multiple producers. + Obviously in complex such configurations, the Producer may not provide data for all pairs, so the strategy must be fit for this. + #### AgeFilter Removes pairs that have been listed on the exchange for less than `min_days_listed` days (defaults to `10`) or more than `max_days_listed` days (defaults `None` mean infinity). diff --git a/docs/strategy-callbacks.md b/docs/strategy-callbacks.md index 0b8403414..ea10fc472 100644 --- a/docs/strategy-callbacks.md +++ b/docs/strategy-callbacks.md @@ -643,7 +643,7 @@ This callback is **not** called when there is an open order (either buy or sell) Additional Buys are ignored once you have reached the maximum amount of extra buys that you have set on `max_entry_position_adjustment`, but the callback is called anyway looking for partial exits. -Position adjustments will always be applied in the direction of the trade, so a positive value will always increase your position (negative values will decrease your position), no matter if it's a long or short trade. Modifications to leverage are not possible. +Position adjustments will always be applied in the direction of the trade, so a positive value will always increase your position (negative values will decrease your position), no matter if it's a long or short trade. Modifications to leverage are not possible, and the stake-amount is assumed to be before applying leverage. !!! Note "About stake size" Using fixed stake size means it will be the amount used for the first order, just like without position adjustment. diff --git a/docs/updating.md b/docs/updating.md index 8dc7279a4..893bc846e 100644 --- a/docs/updating.md +++ b/docs/updating.md @@ -37,3 +37,12 @@ pip install -e . # Ensure freqUI is at the latest version freqtrade install-ui ``` + +### Problems updating + +Update-problems usually come missing dependencies (you didn't follow the above instructions) - or from updated dependencies, which fail to install (for example TA-lib). +Please refer to the corresponding installation sections (common problems linked below) + +Common problems and their solutions: + +* [ta-lib update on windows](windows_installation.md#2-install-ta-lib) diff --git a/docs/windows_installation.md b/docs/windows_installation.md index 9fbbf8250..5cfae8c10 100644 --- a/docs/windows_installation.md +++ b/docs/windows_installation.md @@ -34,7 +34,7 @@ python -m venv .env .env\Scripts\activate.ps1 # optionally install ta-lib from wheel # Eventually adjust the below filename to match the downloaded wheel -pip install --find-links build_helpers\ TA-Lib +pip install --find-links build_helpers\ TA-Lib -U pip install -r requirements.txt pip install -e . freqtrade diff --git a/freqtrade/__init__.py b/freqtrade/__init__.py index 77c305c66..1e62266a8 100644 --- a/freqtrade/__init__.py +++ b/freqtrade/__init__.py @@ -1,5 +1,5 @@ """ Freqtrade bot """ -__version__ = '2022.9.dev' +__version__ = '2022.10.dev' if 'dev' in __version__: try: diff --git a/freqtrade/constants.py b/freqtrade/constants.py index acab8489c..e0eb5e288 100644 --- a/freqtrade/constants.py +++ b/freqtrade/constants.py @@ -31,7 +31,7 @@ HYPEROPT_LOSS_BUILTIN = ['ShortTradeDurHyperOptLoss', 'OnlyProfitHyperOptLoss', 'CalmarHyperOptLoss', 'MaxDrawDownHyperOptLoss', 'MaxDrawDownRelativeHyperOptLoss', 'ProfitDrawDownHyperOptLoss'] -AVAILABLE_PAIRLISTS = ['StaticPairList', 'VolumePairList', +AVAILABLE_PAIRLISTS = ['StaticPairList', 'VolumePairList', 'ProducerPairList', 'AgeFilter', 'OffsetFilter', 'PerformanceFilter', 'PrecisionFilter', 'PriceFilter', 'RangeStabilityFilter', 'ShuffleFilter', 'SpreadFilter', 'VolatilityFilter'] diff --git a/freqtrade/data/converter.py b/freqtrade/data/converter.py index 67461973f..98ed15489 100644 --- a/freqtrade/data/converter.py +++ b/freqtrade/data/converter.py @@ -47,8 +47,7 @@ def ohlcv_to_dataframe(ohlcv: list, timeframe: str, pair: str, *, def clean_ohlcv_dataframe(data: DataFrame, timeframe: str, pair: str, *, - fill_missing: bool = True, - drop_incomplete: bool = True) -> DataFrame: + fill_missing: bool, drop_incomplete: bool) -> DataFrame: """ Cleanse a OHLCV dataframe by * Grouping it by date (removes duplicate tics) diff --git a/freqtrade/data/history/history_utils.py b/freqtrade/data/history/history_utils.py index 6a6e29429..93534e919 100644 --- a/freqtrade/data/history/history_utils.py +++ b/freqtrade/data/history/history_utils.py @@ -26,7 +26,7 @@ def load_pair_history(pair: str, datadir: Path, *, timerange: Optional[TimeRange] = None, fill_up_missing: bool = True, - drop_incomplete: bool = True, + drop_incomplete: bool = False, startup_candles: int = 0, data_format: str = None, data_handler: IDataHandler = None, diff --git a/freqtrade/data/history/idatahandler.py b/freqtrade/data/history/idatahandler.py index c2d92fc4f..80e29f4c0 100644 --- a/freqtrade/data/history/idatahandler.py +++ b/freqtrade/data/history/idatahandler.py @@ -275,7 +275,7 @@ class IDataHandler(ABC): candle_type: CandleType, *, timerange: Optional[TimeRange] = None, fill_missing: bool = True, - drop_incomplete: bool = True, + drop_incomplete: bool = False, startup_candles: int = 0, warn_no_data: bool = True, ) -> DataFrame: diff --git a/freqtrade/exchange/binance.py b/freqtrade/exchange/binance.py index f9fb4a8b1..a0d4b2d82 100644 --- a/freqtrade/exchange/binance.py +++ b/freqtrade/exchange/binance.py @@ -68,6 +68,37 @@ class Binance(Exchange): tickers = deep_merge_dicts(bidsasks, tickers, allow_null_overrides=False) return tickers + @retrier + def additional_exchange_init(self) -> None: + """ + Additional exchange initialization logic. + .api will be available at this point. + Must be overridden in child methods if required. + """ + try: + if self.trading_mode == TradingMode.FUTURES and not self._config['dry_run']: + position_side = self._api.fapiPrivateGetPositionsideDual() + self._log_exchange_response('position_side_setting', position_side) + assets_margin = self._api.fapiPrivateGetMultiAssetsMargin() + self._log_exchange_response('multi_asset_margin', assets_margin) + msg = "" + if position_side.get('dualSidePosition') is True: + msg += ( + "\nHedge Mode is not supported by freqtrade. " + "Please change 'Position Mode' on your binance futures account.") + if assets_margin.get('multiAssetsMargin') is True: + msg += ("\nMulti-Asset Mode is not supported by freqtrade. " + "Please change 'Asset Mode' on your binance futures account.") + if msg: + raise OperationalException(msg) + except ccxt.DDoSProtection as e: + raise DDosProtection(e) from e + except (ccxt.NetworkError, ccxt.ExchangeError) as e: + raise TemporaryError( + f'Could not set leverage due to {e.__class__.__name__}. Message: {e}') from e + except ccxt.BaseError as e: + raise OperationalException(e) from e + @retrier def _set_leverage( self, diff --git a/freqtrade/exchange/okx.py b/freqtrade/exchange/okx.py index 2db5fb6a9..6792c2cba 100644 --- a/freqtrade/exchange/okx.py +++ b/freqtrade/exchange/okx.py @@ -78,7 +78,8 @@ class Okx(Exchange): raise DDosProtection(e) from e except (ccxt.NetworkError, ccxt.ExchangeError) as e: raise TemporaryError( - f'Could not set leverage due to {e.__class__.__name__}. Message: {e}') from e + f'Error in additional_exchange_init due to {e.__class__.__name__}. Message: {e}' + ) from e except ccxt.BaseError as e: raise OperationalException(e) from e diff --git a/freqtrade/freqai/data_kitchen.py b/freqtrade/freqai/data_kitchen.py index 5cf9b2f03..6263934c8 100644 --- a/freqtrade/freqai/data_kitchen.py +++ b/freqtrade/freqai/data_kitchen.py @@ -204,7 +204,7 @@ class FreqaiDataKitchen: filtered_df = unfiltered_df.filter(training_feature_list, axis=1) filtered_df = filtered_df.replace([np.inf, -np.inf], np.nan) - drop_index = pd.isnull(filtered_df).any(1) # get the rows that have NaNs, + drop_index = pd.isnull(filtered_df).any(axis=1) # get the rows that have NaNs, drop_index = drop_index.replace(True, 1).replace(False, 0) # pep8 requirement. if (training_filter): const_cols = list((filtered_df.nunique() == 1).loc[lambda x: x].index) @@ -215,7 +215,7 @@ class FreqaiDataKitchen: # about removing any row with NaNs # if labels has multiple columns (user wants to train multiple modelEs), we detect here labels = unfiltered_df.filter(label_list, axis=1) - drop_index_labels = pd.isnull(labels).any(1) + drop_index_labels = pd.isnull(labels).any(axis=1) drop_index_labels = drop_index_labels.replace(True, 1).replace(False, 0) dates = unfiltered_df['date'] filtered_df = filtered_df[ @@ -243,7 +243,7 @@ class FreqaiDataKitchen: else: # we are backtesting so we need to preserve row number to send back to strategy, # so now we use do_predict to avoid any prediction based on a NaN - drop_index = pd.isnull(filtered_df).any(1) + drop_index = pd.isnull(filtered_df).any(axis=1) self.data["filter_drop_index_prediction"] = drop_index filtered_df.fillna(0, inplace=True) # replacing all NaNs with zeros to avoid issues in 'prediction', but any prediction @@ -802,7 +802,7 @@ class FreqaiDataKitchen: :, :no_prev_pts ] distances = distances.replace([np.inf, -np.inf], np.nan) - drop_index = pd.isnull(distances).any(1) + drop_index = pd.isnull(distances).any(axis=1) distances = distances[drop_index == 0] inliers = pd.DataFrame(index=distances.index) diff --git a/freqtrade/freqtradebot.py b/freqtrade/freqtradebot.py index 72b88a82f..387bae534 100644 --- a/freqtrade/freqtradebot.py +++ b/freqtrade/freqtradebot.py @@ -82,7 +82,10 @@ class FreqtradeBot(LoggingMixin): # Keep this at the end of this initialization method. self.rpc: RPCManager = RPCManager(self) - self.dataprovider = DataProvider(self.config, self.exchange, self.pairlists, self.rpc) + self.dataprovider = DataProvider(self.config, self.exchange, rpc=self.rpc) + self.pairlists = PairListManager(self.exchange, self.config, self.dataprovider) + + self.dataprovider.add_pairlisthandler(self.pairlists) # Attach Dataprovider to strategy instance self.strategy.dp = self.dataprovider @@ -597,7 +600,7 @@ class FreqtradeBot(LoggingMixin): # We should decrease our position amount = self.exchange.amount_to_contract_precision( trade.pair, - abs(float(FtPrecise(stake_amount) / FtPrecise(current_exit_rate)))) + abs(float(FtPrecise(stake_amount * trade.leverage) / FtPrecise(current_exit_rate)))) if amount > trade.amount: # This is currently ineffective as remaining would become < min tradable # Fixing this would require checking for 0.0 there - @@ -1340,11 +1343,12 @@ class FreqtradeBot(LoggingMixin): replacing: Optional[bool] = False ) -> bool: """ - Buy cancel - cancel order + entry cancel - cancel order :param replacing: Replacing order - prevent trade deletion. - :return: True if order was fully cancelled + :return: True if trade was fully cancelled """ was_trade_fully_canceled = False + side = trade.entry_side.capitalize() # Cancelled orders may have the status of 'canceled' or 'closed' if order['status'] not in constants.NON_OPEN_EXCHANGE_STATES: @@ -1371,7 +1375,6 @@ class FreqtradeBot(LoggingMixin): corder = order reason = constants.CANCEL_REASON['CANCELLED_ON_EXCHANGE'] - side = trade.entry_side.capitalize() logger.info('%s order %s for %s.', side, reason, trade) # Using filled to determine the filled amount @@ -1385,24 +1388,13 @@ class FreqtradeBot(LoggingMixin): was_trade_fully_canceled = True reason += f", {constants.CANCEL_REASON['FULLY_CANCELLED']}" else: - # FIXME TODO: This could possibly reworked to not duplicate the code 15 lines below. self.update_trade_state(trade, trade.open_order_id, corder) - trade.open_order_id = None logger.info(f'{side} Order timeout for {trade}.') else: - # if trade is partially complete, edit the stake details for the trade - # and close the order - # cancel_order may not contain the full order dict, so we need to fallback - # to the order dict acquired before cancelling. - # we need to fall back to the values from order if corder does not contain these keys. - trade.amount = filled_amount - # * Check edge cases, we don't want to make leverage > 1.0 if we don't have to - # * (for leverage modes which aren't isolated futures) - - trade.stake_amount = trade.amount * trade.open_rate / trade.leverage + # update_trade_state (and subsequently recalc_trade_from_orders) will handle updates + # to the trade object self.update_trade_state(trade, trade.open_order_id, corder) - trade.open_order_id = None logger.info(f'Partial {trade.entry_side} order timeout for {trade}.') reason += f", {constants.CANCEL_REASON['PARTIALLY_FILLED']}" @@ -1439,8 +1431,6 @@ class FreqtradeBot(LoggingMixin): trade.close_rate_requested = None trade.close_profit = None trade.close_profit_abs = None - trade.close_date = None - trade.is_open = True trade.open_order_id = None trade.exit_reason = None cancelled = True @@ -1700,11 +1690,6 @@ class FreqtradeBot(LoggingMixin): 'stake_amount': trade.stake_amount, } - if 'fiat_display_currency' in self.config: - msg.update({ - 'fiat_currency': self.config['fiat_display_currency'], - }) - # Send the message self.rpc.send_msg(msg) diff --git a/freqtrade/optimize/backtesting.py b/freqtrade/optimize/backtesting.py index e942bdfeb..376c2de7c 100644 --- a/freqtrade/optimize/backtesting.py +++ b/freqtrade/optimize/backtesting.py @@ -110,7 +110,7 @@ class Backtesting: self.timeframe = str(self.config.get('timeframe')) self.timeframe_min = timeframe_to_minutes(self.timeframe) self.init_backtest_detail() - self.pairlists = PairListManager(self.exchange, self.config) + self.pairlists = PairListManager(self.exchange, self.config, self.dataprovider) if 'VolumePairList' in self.pairlists.name_list: raise OperationalException("VolumePairList not allowed for backtesting. " "Please use StaticPairList instead.") @@ -540,7 +540,7 @@ class Backtesting: if stake_amount is not None and stake_amount < 0.0: amount = amount_to_contract_precision( - abs(stake_amount) / current_rate, trade.amount_precision, + abs(stake_amount * trade.leverage) / current_rate, trade.amount_precision, self.precision_mode, trade.contract_size) if amount == 0.0: return trade diff --git a/freqtrade/plugins/pairlist/ProducerPairList.py b/freqtrade/plugins/pairlist/ProducerPairList.py new file mode 100644 index 000000000..50b674e60 --- /dev/null +++ b/freqtrade/plugins/pairlist/ProducerPairList.py @@ -0,0 +1,90 @@ +""" +External Pair List provider + +Provides pair list from Leader data +""" +import logging +from typing import Any, Dict, List, Optional + +from freqtrade.exceptions import OperationalException +from freqtrade.plugins.pairlist.IPairList import IPairList + + +logger = logging.getLogger(__name__) + + +class ProducerPairList(IPairList): + """ + PairList plugin for use with external_message_consumer. + Will use pairs given from leader data. + + Usage: + "pairlists": [ + { + "method": "ProducerPairList", + "number_assets": 5, + "producer_name": "default", + } + ], + """ + + def __init__(self, exchange, pairlistmanager, + config: Dict[str, Any], pairlistconfig: Dict[str, Any], + pairlist_pos: int) -> None: + super().__init__(exchange, pairlistmanager, config, pairlistconfig, pairlist_pos) + + self._num_assets: int = self._pairlistconfig.get('number_assets', 0) + self._producer_name = self._pairlistconfig.get('producer_name', 'default') + if not config.get('external_message_consumer', {}).get('enabled'): + raise OperationalException( + "ProducerPairList requires external_message_consumer to be enabled.") + + @property + def needstickers(self) -> bool: + """ + Boolean property defining if tickers are necessary. + If no Pairlist requires tickers, an empty Dict is passed + as tickers argument to filter_pairlist + """ + return False + + def short_desc(self) -> str: + """ + Short whitelist method description - used for startup-messages + -> Please overwrite in subclasses + """ + return f"{self.name} - {self._producer_name}" + + def _filter_pairlist(self, pairlist: Optional[List[str]]): + upstream_pairlist = self._pairlistmanager._dataprovider.get_producer_pairs( + self._producer_name) + + if pairlist is None: + pairlist = self._pairlistmanager._dataprovider.get_producer_pairs(self._producer_name) + + pairs = list(dict.fromkeys(pairlist + upstream_pairlist)) + if self._num_assets: + pairs = pairs[:self._num_assets] + + return pairs + + def gen_pairlist(self, tickers: Dict) -> List[str]: + """ + Generate the pairlist + :param tickers: Tickers (from exchange.get_tickers()). May be cached. + :return: List of pairs + """ + pairs = self._filter_pairlist(None) + self.log_once(f"Received pairs: {pairs}", logger.debug) + pairs = self._whitelist_for_active_markets(self.verify_whitelist(pairs, logger.info)) + return pairs + + def filter_pairlist(self, pairlist: List[str], tickers: Dict) -> List[str]: + """ + Filters and sorts pairlist and returns the whitelist again. + Called on each bot iteration - please use internal caching if necessary + :param pairlist: pairlist to filter or sort + :param tickers: Tickers (from exchange.get_tickers()). May be cached. + :return: new whitelist + """ + return self._filter_pairlist(pairlist) diff --git a/freqtrade/plugins/pairlist/VolumePairList.py b/freqtrade/plugins/pairlist/VolumePairList.py index 9dcada291..b290f76aa 100644 --- a/freqtrade/plugins/pairlist/VolumePairList.py +++ b/freqtrade/plugins/pairlist/VolumePairList.py @@ -232,6 +232,4 @@ class VolumePairList(IPairList): # Limit pairlist to the requested number of pairs pairs = pairs[:self._number_pairs] - self.log_once(f"Searching {self._number_pairs} pairs: {pairs}", logger.info) - return pairs diff --git a/freqtrade/plugins/pairlistmanager.py b/freqtrade/plugins/pairlistmanager.py index e01abb297..5ed319e93 100644 --- a/freqtrade/plugins/pairlistmanager.py +++ b/freqtrade/plugins/pairlistmanager.py @@ -3,11 +3,12 @@ PairList manager class """ import logging from functools import partial -from typing import Dict, List +from typing import Dict, List, Optional from cachetools import TTLCache, cached from freqtrade.constants import Config, ListPairsWithTimeframes +from freqtrade.data.dataprovider import DataProvider from freqtrade.enums import CandleType from freqtrade.exceptions import OperationalException from freqtrade.mixins import LoggingMixin @@ -21,13 +22,14 @@ logger = logging.getLogger(__name__) class PairListManager(LoggingMixin): - def __init__(self, exchange, config: Config) -> None: + def __init__(self, exchange, config: Config, dataprovider: DataProvider = None) -> None: self._exchange = exchange self._config = config self._whitelist = self._config['exchange'].get('pair_whitelist') self._blacklist = self._config['exchange'].get('pair_blacklist', []) self._pairlist_handlers: List[IPairList] = [] self._tickers_needed = False + self._dataprovider: Optional[DataProvider] = dataprovider for pairlist_handler_config in self._config.get('pairlists', []): pairlist_handler = PairListResolver.load_pairlist( pairlist_handler_config['method'], @@ -96,6 +98,8 @@ class PairListManager(LoggingMixin): # to ensure blacklist is respected. pairlist = self.verify_blacklist(pairlist, logger.warning) + self.log_once(f"Whitelist with {len(pairlist)} pairs: {pairlist}", logger.info) + self._whitelist = pairlist def verify_blacklist(self, pairlist: List[str], logmethod) -> List[str]: diff --git a/freqtrade/rpc/discord.py b/freqtrade/rpc/discord.py index 9efe6f427..c48508300 100644 --- a/freqtrade/rpc/discord.py +++ b/freqtrade/rpc/discord.py @@ -30,9 +30,9 @@ class Discord(Webhook): pass def send_msg(self, msg) -> None: - logger.info(f"Sending discord message: {msg}") if msg['type'].value in self.config['discord']: + logger.info(f"Sending discord message: {msg}") msg['strategy'] = self.strategy msg['timeframe'] = self.timeframe diff --git a/freqtrade/rpc/external_message_consumer.py b/freqtrade/rpc/external_message_consumer.py index dcfe1d109..f5ba4b490 100644 --- a/freqtrade/rpc/external_message_consumer.py +++ b/freqtrade/rpc/external_message_consumer.py @@ -284,7 +284,7 @@ class ExternalMessageConsumer: logger.error(f"Empty message received from `{producer_name}`") return - logger.info(f"Received message of type `{producer_message.type}` from `{producer_name}`") + logger.debug(f"Received message of type `{producer_message.type}` from `{producer_name}`") message_handler = self._message_handlers.get(producer_message.type) diff --git a/freqtrade/rpc/rpc_manager.py b/freqtrade/rpc/rpc_manager.py index e286487ff..e3b31d225 100644 --- a/freqtrade/rpc/rpc_manager.py +++ b/freqtrade/rpc/rpc_manager.py @@ -67,7 +67,7 @@ class RPCManager: 'status': 'stopping bot' } """ - if msg.get('type') is not RPCMessageType.ANALYZED_DF: + if msg.get('type') not in (RPCMessageType.ANALYZED_DF, RPCMessageType.WHITELIST): logger.info('Sending rpc message: %s', msg) if 'pair' in msg: msg.update({ diff --git a/freqtrade/rpc/webhook.py b/freqtrade/rpc/webhook.py index 6109e80bc..bb3b3922f 100644 --- a/freqtrade/rpc/webhook.py +++ b/freqtrade/rpc/webhook.py @@ -61,6 +61,14 @@ class Webhook(RPCHandler): RPCMessageType.STARTUP, RPCMessageType.WARNING): valuedict = whconfig.get('webhookstatus') + elif msg['type'] in ( + RPCMessageType.PROTECTION_TRIGGER, + RPCMessageType.PROTECTION_TRIGGER_GLOBAL, + RPCMessageType.WHITELIST, + RPCMessageType.ANALYZED_DF, + RPCMessageType.STRATEGY_MSG): + # Don't fail for non-implemented types + return else: raise NotImplementedError('Unknown message type: {}'.format(msg['type'])) if not valuedict: diff --git a/setup.py b/setup.py index 0581081fa..d3f9ea7c0 100644 --- a/setup.py +++ b/setup.py @@ -72,7 +72,7 @@ setup( 'pandas', 'tables', 'blosc', - 'joblib', + 'joblib>=1.2.0', 'pyarrow; platform_machine != "armv7l"', 'fastapi', 'uvicorn', diff --git a/tests/conftest.py b/tests/conftest.py index 51b1b03e3..a9eeb481e 100644 --- a/tests/conftest.py +++ b/tests/conftest.py @@ -200,6 +200,8 @@ def patch_freqtradebot(mocker, config) -> None: mocker.patch('freqtrade.freqtradebot.RPCManager._init', MagicMock()) mocker.patch('freqtrade.freqtradebot.RPCManager.send_msg', MagicMock()) patch_whitelist(mocker, config) + mocker.patch('freqtrade.freqtradebot.ExternalMessageConsumer') + mocker.patch('freqtrade.configuration.config_validation._validate_consumers') def get_patched_freqtradebot(mocker, config) -> FreqtradeBot: diff --git a/tests/data/test_btanalysis.py b/tests/data/test_btanalysis.py index dab76d0cb..ec7b457ea 100644 --- a/tests/data/test_btanalysis.py +++ b/tests/data/test_btanalysis.py @@ -235,7 +235,7 @@ def test_calculate_market_change(testdatadir): data = load_data(datadir=testdatadir, pairs=pairs, timeframe='5m') result = calculate_market_change(data) assert isinstance(result, float) - assert pytest.approx(result) == 0.00955514 + assert pytest.approx(result) == 0.01100002 def test_combine_dataframes_with_mean(testdatadir): diff --git a/tests/data/test_datahandler.py b/tests/data/test_datahandler.py index 8e1b0050a..5d6d60f84 100644 --- a/tests/data/test_datahandler.py +++ b/tests/data/test_datahandler.py @@ -139,10 +139,10 @@ def test_jsondatahandler_ohlcv_purge(mocker, testdatadir): def test_jsondatahandler_ohlcv_load(testdatadir, caplog): dh = JsonDataHandler(testdatadir) df = dh.ohlcv_load('XRP/ETH', '5m', 'spot') - assert len(df) == 711 + assert len(df) == 712 df_mark = dh.ohlcv_load('UNITTEST/USDT', '1h', candle_type="mark") - assert len(df_mark) == 99 + assert len(df_mark) == 100 df_no_mark = dh.ohlcv_load('UNITTEST/USDT', '1h', 'spot') assert len(df_no_mark) == 0 diff --git a/tests/data/test_entryexitanalysis.py b/tests/data/test_entryexitanalysis.py index 09fbe9957..588220465 100755 --- a/tests/data/test_entryexitanalysis.py +++ b/tests/data/test_entryexitanalysis.py @@ -124,8 +124,8 @@ def test_backtest_analysis_nomock(default_conf, mocker, caplog, testdatadir, tmp assert '0' in captured.out assert '0.01616' in captured.out assert '34.049' in captured.out - assert '0.104104' in captured.out - assert '47.0996' in captured.out + assert '0.104411' in captured.out + assert '52.8292' in captured.out # test group 1 args = get_args(base_args + ['--analysis-groups', "1"]) diff --git a/tests/data/test_history.py b/tests/data/test_history.py index 5642442b2..e7e3d4063 100644 --- a/tests/data/test_history.py +++ b/tests/data/test_history.py @@ -377,8 +377,8 @@ def test_load_partial_missing(testdatadir, caplog) -> None: td = ((end - start).total_seconds() // 60 // 5) + 1 assert td != len(data['UNITTEST/BTC']) - # Shift endtime with +5 - as last candle is dropped (partial candle) - end_real = arrow.get(data['UNITTEST/BTC'].iloc[-1, 0]).shift(minutes=5) + # Shift endtime with +5 + end_real = arrow.get(data['UNITTEST/BTC'].iloc[-1, 0]) assert log_has(f'UNITTEST/BTC, spot, 5m, ' f'data ends at {end_real.strftime(DATETIME_PRINT_FORMAT)}', caplog) @@ -447,7 +447,7 @@ def test_get_timerange(default_conf, mocker, testdatadir) -> None: ) min_date, max_date = get_timerange(data) assert min_date.isoformat() == '2017-11-04T23:02:00+00:00' - assert max_date.isoformat() == '2017-11-14T22:58:00+00:00' + assert max_date.isoformat() == '2017-11-14T22:59:00+00:00' def test_validate_backtest_data_warn(default_conf, mocker, caplog, testdatadir) -> None: @@ -470,7 +470,7 @@ def test_validate_backtest_data_warn(default_conf, mocker, caplog, testdatadir) min_date, max_date, timeframe_to_minutes('1m')) assert len(caplog.record_tuples) == 1 assert log_has( - "UNITTEST/BTC has missing frames: expected 14396, got 13680, that's 716 missing values", + "UNITTEST/BTC has missing frames: expected 14397, got 13681, that's 716 missing values", caplog) diff --git a/tests/exchange/test_binance.py b/tests/exchange/test_binance.py index e9f4dfa8a..ef5cb1240 100644 --- a/tests/exchange/test_binance.py +++ b/tests/exchange/test_binance.py @@ -501,6 +501,24 @@ def test_fill_leverage_tiers_binance_dryrun(default_conf, mocker, leverage_tiers assert len(v) == len(value) +def test_additional_exchange_init_binance(default_conf, mocker): + api_mock = MagicMock() + api_mock.fapiPrivateGetPositionsideDual = MagicMock(return_value={"dualSidePosition": True}) + api_mock.fapiPrivateGetMultiAssetsMargin = MagicMock(return_value={"multiAssetsMargin": True}) + default_conf['dry_run'] = False + default_conf['trading_mode'] = TradingMode.FUTURES + default_conf['margin_mode'] = MarginMode.ISOLATED + with pytest.raises(OperationalException, + match=r"Hedge Mode is not supported.*\nMulti-Asset Mode is not supported.*"): + get_patched_exchange(mocker, default_conf, id="binance", api_mock=api_mock) + api_mock.fapiPrivateGetPositionsideDual = MagicMock(return_value={"dualSidePosition": False}) + api_mock.fapiPrivateGetMultiAssetsMargin = MagicMock(return_value={"multiAssetsMargin": False}) + exchange = get_patched_exchange(mocker, default_conf, id="binance", api_mock=api_mock) + assert exchange + ccxt_exceptionhandlers(mocker, default_conf, api_mock, 'binance', + "additional_exchange_init", "fapiPrivateGetPositionsideDual") + + def test__set_leverage_binance(mocker, default_conf): api_mock = MagicMock() diff --git a/tests/exchange/test_ccxt_compat.py b/tests/exchange/test_ccxt_compat.py index 82be6196a..6798cd2f7 100644 --- a/tests/exchange/test_ccxt_compat.py +++ b/tests/exchange/test_ccxt_compat.py @@ -137,6 +137,7 @@ def exchange_futures(request, exchange_conf, class_mocker): 'freqtrade.exchange.binance.Binance.fill_leverage_tiers') class_mocker.patch('freqtrade.exchange.exchange.Exchange.fetch_trading_fees') class_mocker.patch('freqtrade.exchange.okx.Okx.additional_exchange_init') + class_mocker.patch('freqtrade.exchange.binance.Binance.additional_exchange_init') class_mocker.patch('freqtrade.exchange.exchange.Exchange.load_cached_leverage_tiers', return_value=None) class_mocker.patch('freqtrade.exchange.exchange.Exchange.cache_leverage_tiers') diff --git a/tests/optimize/test_backtesting.py b/tests/optimize/test_backtesting.py index bd87b2b42..907e97fb7 100644 --- a/tests/optimize/test_backtesting.py +++ b/tests/optimize/test_backtesting.py @@ -80,7 +80,7 @@ def load_data_test(what, testdatadir): data.loc[:, 'close'] = np.sin(data.index * hz) / 1000 + base return {'UNITTEST/BTC': clean_ohlcv_dataframe(data, timeframe='1m', pair='UNITTEST/BTC', - fill_missing=True)} + fill_missing=True, drop_incomplete=True)} # FIX: fixturize this? @@ -323,7 +323,7 @@ def test_data_to_dataframe_bt(default_conf, mocker, testdatadir) -> None: backtesting = Backtesting(default_conf) backtesting._set_strategy(backtesting.strategylist[0]) processed = backtesting.strategy.advise_all_indicators(data) - assert len(processed['UNITTEST/BTC']) == 102 + assert len(processed['UNITTEST/BTC']) == 103 # Load strategy to compare the result between Backtesting function and strategy are the same strategy = StrategyResolver.load_strategy(default_conf) @@ -1165,9 +1165,9 @@ def test_backtest_start_timerange(default_conf, mocker, caplog, testdatadir): 'Parameter --timerange detected: 1510694220-1510700340 ...', f'Using data directory: {testdatadir} ...', 'Loading data from 2017-11-14 20:57:00 ' - 'up to 2017-11-14 22:58:00 (0 days).', + 'up to 2017-11-14 22:59:00 (0 days).', 'Backtesting with data from 2017-11-14 21:17:00 ' - 'up to 2017-11-14 22:58:00 (0 days).', + 'up to 2017-11-14 22:59:00 (0 days).', 'Parameter --enable-position-stacking detected ...' ] @@ -1244,9 +1244,9 @@ def test_backtest_start_multi_strat(default_conf, mocker, caplog, testdatadir): 'Parameter --timerange detected: 1510694220-1510700340 ...', f'Using data directory: {testdatadir} ...', 'Loading data from 2017-11-14 20:57:00 ' - 'up to 2017-11-14 22:58:00 (0 days).', + 'up to 2017-11-14 22:59:00 (0 days).', 'Backtesting with data from 2017-11-14 21:17:00 ' - 'up to 2017-11-14 22:58:00 (0 days).', + 'up to 2017-11-14 22:59:00 (0 days).', 'Parameter --enable-position-stacking detected ...', f'Running backtesting for Strategy {CURRENT_TEST_STRATEGY}', 'Running backtesting for Strategy StrategyTestV2', @@ -1355,9 +1355,9 @@ def test_backtest_start_multi_strat_nomock(default_conf, mocker, caplog, testdat 'Parameter --timerange detected: 1510694220-1510700340 ...', f'Using data directory: {testdatadir} ...', 'Loading data from 2017-11-14 20:57:00 ' - 'up to 2017-11-14 22:58:00 (0 days).', + 'up to 2017-11-14 22:59:00 (0 days).', 'Backtesting with data from 2017-11-14 21:17:00 ' - 'up to 2017-11-14 22:58:00 (0 days).', + 'up to 2017-11-14 22:59:00 (0 days).', 'Parameter --enable-position-stacking detected ...', f'Running backtesting for Strategy {CURRENT_TEST_STRATEGY}', 'Running backtesting for Strategy StrategyTestV2', @@ -1371,7 +1371,7 @@ def test_backtest_start_multi_strat_nomock(default_conf, mocker, caplog, testdat assert 'EXIT REASON STATS' in captured.out assert 'DAY BREAKDOWN' in captured.out assert 'LEFT OPEN TRADES REPORT' in captured.out - assert '2017-11-14 21:17:00 -> 2017-11-14 22:58:00 | Max open trades : 1' in captured.out + assert '2017-11-14 21:17:00 -> 2017-11-14 22:59:00 | Max open trades : 1' in captured.out assert 'STRATEGY SUMMARY' in captured.out @@ -1503,9 +1503,9 @@ def test_backtest_start_nomock_futures(default_conf_usdt, mocker, 'Parameter -i/--timeframe detected ... Using timeframe: 1h ...', f'Using data directory: {testdatadir} ...', 'Loading data from 2021-11-17 01:00:00 ' - 'up to 2021-11-21 03:00:00 (4 days).', + 'up to 2021-11-21 04:00:00 (4 days).', 'Backtesting with data from 2021-11-17 21:00:00 ' - 'up to 2021-11-21 03:00:00 (3 days).', + 'up to 2021-11-21 04:00:00 (3 days).', 'XRP/USDT, funding_rate, 8h, data starts at 2021-11-18 00:00:00', 'XRP/USDT, mark, 8h, data starts at 2021-11-18 00:00:00', f'Running backtesting for Strategy {CURRENT_TEST_STRATEGY}', @@ -1616,9 +1616,9 @@ def test_backtest_start_multi_strat_nomock_detail(default_conf, mocker, 'Parameter --timeframe-detail detected, using 1m for intra-candle backtesting ...', f'Using data directory: {testdatadir} ...', 'Loading data from 2019-10-11 00:00:00 ' - 'up to 2019-10-13 11:10:00 (2 days).', + 'up to 2019-10-13 11:15:00 (2 days).', 'Backtesting with data from 2019-10-11 01:40:00 ' - 'up to 2019-10-13 11:10:00 (2 days).', + 'up to 2019-10-13 11:15:00 (2 days).', f'Running backtesting for Strategy {CURRENT_TEST_STRATEGY}', ] @@ -1719,7 +1719,7 @@ def test_backtest_start_multi_strat_caching(default_conf, mocker, caplog, testda 'Parameter --timerange detected: 1510694220-1510700340 ...', f'Using data directory: {testdatadir} ...', 'Loading data from 2017-11-14 20:57:00 ' - 'up to 2017-11-14 22:58:00 (0 days).', + 'up to 2017-11-14 22:59:00 (0 days).', 'Parameter --enable-position-stacking detected ...', ] @@ -1732,7 +1732,7 @@ def test_backtest_start_multi_strat_caching(default_conf, mocker, caplog, testda 'Running backtesting for Strategy StrategyTestV2', 'Running backtesting for Strategy StrategyTestV3', 'Ignoring max_open_trades (--disable-max-market-positions was used) ...', - 'Backtesting with data from 2017-11-14 21:17:00 up to 2017-11-14 22:58:00 (0 days).', + 'Backtesting with data from 2017-11-14 21:17:00 up to 2017-11-14 22:59:00 (0 days).', ] elif run_id == '2' and min_backtest_date < start_time: assert backtestmock.call_count == 0 @@ -1745,7 +1745,7 @@ def test_backtest_start_multi_strat_caching(default_conf, mocker, caplog, testda 'Reusing result of previous backtest for StrategyTestV2', 'Running backtesting for Strategy StrategyTestV3', 'Ignoring max_open_trades (--disable-max-market-positions was used) ...', - 'Backtesting with data from 2017-11-14 21:17:00 up to 2017-11-14 22:58:00 (0 days).', + 'Backtesting with data from 2017-11-14 21:17:00 up to 2017-11-14 22:59:00 (0 days).', ] assert backtestmock.call_count == 1 diff --git a/tests/optimize/test_backtesting_adjust_position.py b/tests/optimize/test_backtesting_adjust_position.py index 71f8cdcea..99c160a40 100644 --- a/tests/optimize/test_backtesting_adjust_position.py +++ b/tests/optimize/test_backtesting_adjust_position.py @@ -93,11 +93,16 @@ def test_backtest_position_adjustment(default_conf, fee, mocker, testdatadir) -> t["close_rate"], 6) < round(ln.iloc[0]["high"], 6)) -def test_backtest_position_adjustment_detailed(default_conf, fee, mocker) -> None: +@pytest.mark.parametrize('leverage', [ + 1, 2 +]) +def test_backtest_position_adjustment_detailed(default_conf, fee, mocker, leverage) -> None: default_conf['use_exit_signal'] = False mocker.patch('freqtrade.exchange.Exchange.get_fee', fee) mocker.patch("freqtrade.exchange.Exchange.get_min_pair_stake_amount", return_value=10) mocker.patch("freqtrade.exchange.Exchange.get_max_pair_stake_amount", return_value=float('inf')) + mocker.patch("freqtrade.exchange.Exchange.get_max_leverage", return_value=10) + patch_exchange(mocker) default_conf.update({ "stake_amount": 100.0, @@ -105,6 +110,7 @@ def test_backtest_position_adjustment_detailed(default_conf, fee, mocker) -> Non "strategy": "StrategyTestV3" }) backtesting = Backtesting(default_conf) + backtesting._can_short = True backtesting._set_strategy(backtesting.strategylist[0]) pair = 'XRP/USDT' row = [ @@ -120,18 +126,19 @@ def test_backtest_position_adjustment_detailed(default_conf, fee, mocker) -> Non '', # enter_tag '', # exit_tag ] + backtesting.strategy.leverage = MagicMock(return_value=leverage) trade = backtesting._enter_trade(pair, row=row, direction='long') trade.orders[0].close_bt_order(row[0], trade) assert trade assert pytest.approx(trade.stake_amount) == 100.0 - assert pytest.approx(trade.amount) == 47.61904762 + assert pytest.approx(trade.amount) == 47.61904762 * leverage assert len(trade.orders) == 1 backtesting.strategy.adjust_trade_position = MagicMock(return_value=None) trade = backtesting._get_adjust_trade_entry_for_candle(trade, row) assert trade assert pytest.approx(trade.stake_amount) == 100.0 - assert pytest.approx(trade.amount) == 47.61904762 + assert pytest.approx(trade.amount) == 47.61904762 * leverage assert len(trade.orders) == 1 # Increase position by 100 backtesting.strategy.adjust_trade_position = MagicMock(return_value=100) @@ -140,7 +147,7 @@ def test_backtest_position_adjustment_detailed(default_conf, fee, mocker) -> Non assert trade assert pytest.approx(trade.stake_amount) == 200.0 - assert pytest.approx(trade.amount) == 95.23809524 + assert pytest.approx(trade.amount) == 95.23809524 * leverage assert len(trade.orders) == 2 # Reduce by more than amount - no change to trade. @@ -150,7 +157,7 @@ def test_backtest_position_adjustment_detailed(default_conf, fee, mocker) -> Non assert trade assert pytest.approx(trade.stake_amount) == 200.0 - assert pytest.approx(trade.amount) == 95.23809524 + assert pytest.approx(trade.amount) == 95.23809524 * leverage assert len(trade.orders) == 2 assert trade.nr_of_successful_entries == 2 @@ -160,7 +167,7 @@ def test_backtest_position_adjustment_detailed(default_conf, fee, mocker) -> Non assert trade assert pytest.approx(trade.stake_amount) == 100.0 - assert pytest.approx(trade.amount) == 47.61904762 + assert pytest.approx(trade.amount) == 47.61904762 * leverage assert len(trade.orders) == 3 assert trade.nr_of_successful_entries == 2 assert trade.nr_of_successful_exits == 1 @@ -171,7 +178,7 @@ def test_backtest_position_adjustment_detailed(default_conf, fee, mocker) -> Non assert trade assert pytest.approx(trade.stake_amount) == 100.0 - assert pytest.approx(trade.amount) == 47.61904762 + assert pytest.approx(trade.amount) == 47.61904762 * leverage assert len(trade.orders) == 3 assert trade.nr_of_successful_entries == 2 assert trade.nr_of_successful_exits == 1 diff --git a/tests/plugins/test_pairlist.py b/tests/plugins/test_pairlist.py index d6f074edb..f0b983063 100644 --- a/tests/plugins/test_pairlist.py +++ b/tests/plugins/test_pairlist.py @@ -9,6 +9,7 @@ import pytest import time_machine from freqtrade.constants import AVAILABLE_PAIRLISTS +from freqtrade.data.dataprovider import DataProvider from freqtrade.enums import CandleType, RunMode from freqtrade.exceptions import OperationalException from freqtrade.persistence import Trade @@ -40,6 +41,12 @@ def whitelist_conf(default_conf): "sort_key": "quoteVolume", }, ] + default_conf.update({ + "external_message_consumer": { + "enabled": True, + "producers": [], + } + }) return default_conf @@ -126,7 +133,7 @@ def test_log_cached(mocker, static_pl_conf, markets, tickers): def test_load_pairlist_noexist(mocker, markets, default_conf): freqtrade = get_patched_freqtradebot(mocker, default_conf) mocker.patch('freqtrade.exchange.Exchange.markets', PropertyMock(return_value=markets)) - plm = PairListManager(freqtrade.exchange, default_conf) + plm = PairListManager(freqtrade.exchange, default_conf, MagicMock()) with pytest.raises(OperationalException, match=r"Impossible to load Pairlist 'NonexistingPairList'. " r"This class does not exist or contains Python code errors."): @@ -137,7 +144,7 @@ def test_load_pairlist_noexist(mocker, markets, default_conf): def test_load_pairlist_verify_multi(mocker, markets_static, default_conf): freqtrade = get_patched_freqtradebot(mocker, default_conf) mocker.patch('freqtrade.exchange.Exchange.markets', PropertyMock(return_value=markets_static)) - plm = PairListManager(freqtrade.exchange, default_conf) + plm = PairListManager(freqtrade.exchange, default_conf, MagicMock()) # Call different versions one after the other, should always consider what was passed in # and have no side-effects (therefore the same check multiple times) assert plm.verify_whitelist(['ETH/BTC', 'XRP/BTC', ], print) == ['ETH/BTC', 'XRP/BTC'] @@ -269,7 +276,7 @@ def test_refresh_pairlist_dynamic(mocker, shitcoinmarkets, tickers, whitelist_co with pytest.raises(OperationalException, match=r'`number_assets` not specified. Please check your configuration ' r'for "pairlist.config.number_assets"'): - PairListManager(freqtrade.exchange, whitelist_conf) + PairListManager(freqtrade.exchange, whitelist_conf, MagicMock()) def test_refresh_pairlist_dynamic_2(mocker, shitcoinmarkets, tickers, whitelist_conf_2): @@ -694,7 +701,7 @@ def test_PrecisionFilter_error(mocker, whitelist_conf) -> None: with pytest.raises(OperationalException, match=r"PrecisionFilter can only work with stoploss defined\..*"): - PairListManager(MagicMock, whitelist_conf) + PairListManager(MagicMock, whitelist_conf, MagicMock()) def test_PerformanceFilter_error(mocker, whitelist_conf, caplog) -> None: @@ -703,7 +710,7 @@ def test_PerformanceFilter_error(mocker, whitelist_conf, caplog) -> None: del Trade.query mocker.patch('freqtrade.exchange.Exchange.exchange_has', MagicMock(return_value=True)) exchange = get_patched_exchange(mocker, whitelist_conf) - pm = PairListManager(exchange, whitelist_conf) + pm = PairListManager(exchange, whitelist_conf, MagicMock()) pm.refresh_pairlist() assert log_has("PerformanceFilter is not available in this mode.", caplog) @@ -1167,6 +1174,10 @@ def test_spreadfilter_invalid_data(mocker, default_conf, markets, tickers, caplo "[{'OffsetFilter': 'OffsetFilter - Taking 10 Pairs, starting from 5.'}]", None ), + ({"method": "ProducerPairList"}, + "[{'ProducerPairList': 'ProducerPairList - default'}]", + None + ), ]) def test_pricefilter_desc(mocker, whitelist_conf, markets, pairlistconfig, desc_expected, exception_expected): @@ -1341,3 +1352,77 @@ def test_expand_pairlist_keep_invalid(wildcardlist, pairs, expected): expand_pairlist(wildcardlist, pairs, keep_invalid=True) else: assert sorted(expand_pairlist(wildcardlist, pairs, keep_invalid=True)) == sorted(expected) + + +def test_ProducerPairlist_no_emc(mocker, whitelist_conf): + mocker.patch('freqtrade.exchange.Exchange.exchange_has', MagicMock(return_value=True)) + + whitelist_conf['pairlists'] = [ + { + "method": "ProducerPairList", + "number_assets": 10, + "producer_name": "hello_world", + } + ] + del whitelist_conf['external_message_consumer'] + + with pytest.raises(OperationalException, + match=r"ProducerPairList requires external_message_consumer to be enabled."): + get_patched_freqtradebot(mocker, whitelist_conf) + + +def test_ProducerPairlist(mocker, whitelist_conf, markets): + mocker.patch('freqtrade.exchange.Exchange.exchange_has', MagicMock(return_value=True)) + mocker.patch.multiple('freqtrade.exchange.Exchange', + markets=PropertyMock(return_value=markets), + exchange_has=MagicMock(return_value=True), + ) + whitelist_conf['pairlists'] = [ + { + "method": "ProducerPairList", + "number_assets": 2, + "producer_name": "hello_world", + } + ] + whitelist_conf.update({ + "external_message_consumer": { + "enabled": True, + "producers": [ + { + "name": "hello_world", + "host": "null", + "port": 9891, + "ws_token": "dummy", + } + ] + } + }) + + exchange = get_patched_exchange(mocker, whitelist_conf) + dp = DataProvider(whitelist_conf, exchange, None) + pairs = ['ETH/BTC', 'LTC/BTC', 'XRP/BTC'] + # different producer + dp._set_producer_pairs(pairs + ['MEEP/USDT'], 'default') + pm = PairListManager(exchange, whitelist_conf, dp) + pm.refresh_pairlist() + assert pm.whitelist == [] + # proper producer + dp._set_producer_pairs(pairs, 'hello_world') + pm.refresh_pairlist() + + # Pairlist reduced to 2 + assert pm.whitelist == pairs[:2] + assert len(pm.whitelist) == 2 + whitelist_conf['exchange']['pair_whitelist'] = ['TKN/BTC'] + + whitelist_conf['pairlists'] = [ + {"method": "StaticPairList"}, + { + "method": "ProducerPairList", + "producer_name": "hello_world", + } + ] + pm = PairListManager(exchange, whitelist_conf, dp) + pm.refresh_pairlist() + assert len(pm.whitelist) == 4 + assert pm.whitelist == ['TKN/BTC'] + pairs diff --git a/tests/rpc/test_rpc_emc.py b/tests/rpc/test_rpc_emc.py index 2649c5460..28adc66b9 100644 --- a/tests/rpc/test_rpc_emc.py +++ b/tests/rpc/test_rpc_emc.py @@ -276,6 +276,8 @@ async def test_emc_create_connection_error(default_conf, caplog, mocker): async def test_emc_receive_messages_valid(default_conf, caplog, mocker): + caplog.set_level(logging.DEBUG) + default_conf.update({ "external_message_consumer": { "enabled": True, diff --git a/tests/rpc/test_rpc_webhook.py b/tests/rpc/test_rpc_webhook.py index 4d65b4966..3bbb85d54 100644 --- a/tests/rpc/test_rpc_webhook.py +++ b/tests/rpc/test_rpc_webhook.py @@ -365,6 +365,14 @@ def test_exception_send_msg(default_conf, mocker, caplog): with pytest.raises(NotImplementedError): webhook.send_msg(msg) + # Test no failure for not implemented but known messagetypes + for e in RPCMessageType: + msg = { + 'type': e, + 'status': 'whatever' + } + webhook.send_msg(msg) + def test__send_msg(default_conf, mocker, caplog): default_conf["webhook"] = get_webhook_dict() diff --git a/tests/strategy/test_interface.py b/tests/strategy/test_interface.py index 070e78b1d..294021c83 100644 --- a/tests/strategy/test_interface.py +++ b/tests/strategy/test_interface.py @@ -288,7 +288,7 @@ def test_advise_all_indicators(default_conf, testdatadir) -> None: data = load_data(testdatadir, '1m', ['UNITTEST/BTC'], timerange=timerange, fill_up_missing=True) processed = strategy.advise_all_indicators(data) - assert len(processed['UNITTEST/BTC']) == 102 # partial candle was removed + assert len(processed['UNITTEST/BTC']) == 103 def test_populate_any_indicators(default_conf, testdatadir) -> None: @@ -300,7 +300,7 @@ def test_populate_any_indicators(default_conf, testdatadir) -> None: processed = strategy.populate_any_indicators('UNITTEST/BTC', data, '5m') assert processed == data assert id(processed) == id(data) - assert len(processed['UNITTEST/BTC']) == 102 # partial candle was removed + assert len(processed['UNITTEST/BTC']) == 103 def test_freqai_not_initialized(default_conf) -> None: diff --git a/tests/test_freqtradebot.py b/tests/test_freqtradebot.py index 5fe4d4011..0f1a05ab4 100644 --- a/tests/test_freqtradebot.py +++ b/tests/test_freqtradebot.py @@ -2661,6 +2661,7 @@ def test_manage_open_orders_exit_usercustom( rpc_mock = patch_RPCManager(mocker) cancel_order_mock = MagicMock() patch_exchange(mocker) + mocker.patch('freqtrade.exchange.Exchange.get_min_pair_stake_amount', return_value=0.0) et_mock = mocker.patch('freqtrade.freqtradebot.FreqtradeBot.execute_trade_exit') mocker.patch.multiple( 'freqtrade.exchange.Exchange', @@ -2673,7 +2674,6 @@ def test_manage_open_orders_exit_usercustom( open_trade_usdt.open_date = arrow.utcnow().shift(hours=-5).datetime open_trade_usdt.close_date = arrow.utcnow().shift(minutes=-601).datetime open_trade_usdt.close_profit_abs = 0.001 - open_trade_usdt.is_open = False Trade.query.session.add(open_trade_usdt) Trade.commit() @@ -2687,7 +2687,6 @@ def test_manage_open_orders_exit_usercustom( freqtrade.manage_open_orders() assert cancel_order_mock.call_count == 0 assert rpc_mock.call_count == 1 - assert open_trade_usdt.is_open is False assert freqtrade.strategy.check_exit_timeout.call_count == 1 assert freqtrade.strategy.check_entry_timeout.call_count == 0 @@ -2697,7 +2696,6 @@ def test_manage_open_orders_exit_usercustom( freqtrade.manage_open_orders() assert cancel_order_mock.call_count == 0 assert rpc_mock.call_count == 1 - assert open_trade_usdt.is_open is False assert freqtrade.strategy.check_exit_timeout.call_count == 1 assert freqtrade.strategy.check_entry_timeout.call_count == 0 @@ -2707,7 +2705,6 @@ def test_manage_open_orders_exit_usercustom( freqtrade.manage_open_orders() assert cancel_order_mock.call_count == 1 assert rpc_mock.call_count == 2 - assert open_trade_usdt.is_open is True assert freqtrade.strategy.check_exit_timeout.call_count == 1 assert freqtrade.strategy.check_entry_timeout.call_count == 0 @@ -2748,14 +2745,14 @@ def test_manage_open_orders_exit( 'freqtrade.exchange.Exchange', fetch_ticker=ticker_usdt, fetch_order=MagicMock(return_value=limit_sell_order_old), - cancel_order=cancel_order_mock + cancel_order=cancel_order_mock, + get_min_pair_stake_amount=MagicMock(return_value=0), ) freqtrade = FreqtradeBot(default_conf_usdt) open_trade_usdt.open_date = arrow.utcnow().shift(hours=-5).datetime open_trade_usdt.close_date = arrow.utcnow().shift(minutes=-601).datetime open_trade_usdt.close_profit_abs = 0.001 - open_trade_usdt.is_open = False open_trade_usdt.is_short = is_short Trade.query.session.add(open_trade_usdt) @@ -2796,7 +2793,6 @@ def test_check_handle_cancelled_exit( open_trade_usdt.open_date = arrow.utcnow().shift(hours=-5).datetime open_trade_usdt.close_date = arrow.utcnow().shift(minutes=-601).datetime - open_trade_usdt.is_open = False open_trade_usdt.is_short = is_short Trade.query.session.add(open_trade_usdt) @@ -3004,6 +3000,7 @@ def test_handle_cancel_enter(mocker, caplog, default_conf_usdt, limit_order, is_ trade.open_rate = 200 trade.is_short = False trade.entry_side = "buy" + trade.amount = 100 l_order['filled'] = 0.0 l_order['status'] = 'open' trade.nr_of_successful_entries = 0 @@ -3092,6 +3089,7 @@ def test_handle_cancel_enter_corder_empty(mocker, default_conf_usdt, limit_order trade.entry_side = "buy" trade.open_order_id = "open_order_noop" trade.nr_of_successful_entries = 0 + trade.amount = 100 l_order['filled'] = 0.0 l_order['status'] = 'open' reason = CANCEL_REASON['TIMEOUT'] @@ -3121,20 +3119,21 @@ def test_handle_cancel_exit_limit(mocker, default_conf_usdt, fee) -> None: amount=2, exchange='binance', open_rate=0.245441, - open_order_id="123456", + open_order_id="sell_123456", open_date=arrow.utcnow().shift(days=-2).datetime, fee_open=fee.return_value, fee_close=fee.return_value, close_rate=0.555, close_date=arrow.utcnow().datetime, exit_reason="sell_reason_whatever", + stake_amount=0.245441 * 2, ) trade.orders = [ - Order( + Order( ft_order_side='buy', ft_pair=trade.pair, - ft_is_open=True, - order_id='123456', + ft_is_open=False, + order_id='buy_123456', status="closed", symbol=trade.pair, order_type="market", @@ -3147,15 +3146,33 @@ def test_handle_cancel_exit_limit(mocker, default_conf_usdt, fee) -> None: order_date=trade.open_date, order_filled_date=trade.open_date, ), + Order( + ft_order_side='sell', + ft_pair=trade.pair, + ft_is_open=True, + order_id='sell_123456', + status="open", + symbol=trade.pair, + order_type="limit", + side="sell", + price=trade.open_rate, + average=trade.open_rate, + filled=0.0, + remaining=trade.amount, + cost=trade.open_rate * trade.amount, + order_date=trade.open_date, + order_filled_date=trade.open_date, + ), ] - order = {'id': "123456", + order = {'id': "sell_123456", 'remaining': 1, 'amount': 1, 'status': "open"} reason = CANCEL_REASON['TIMEOUT'] + send_msg_mock.reset_mock() assert freqtrade.handle_cancel_exit(trade, order, reason) assert cancel_order_mock.call_count == 1 - assert send_msg_mock.call_count == 2 + assert send_msg_mock.call_count == 1 assert trade.close_rate is None assert trade.exit_reason is None @@ -3181,8 +3198,9 @@ def test_handle_cancel_exit_limit(mocker, default_conf_usdt, fee) -> None: def test_handle_cancel_exit_cancel_exception(mocker, default_conf_usdt) -> None: patch_RPCManager(mocker) patch_exchange(mocker) - mocker.patch( - 'freqtrade.exchange.Exchange.cancel_order_with_result', side_effect=InvalidOrderException()) + mocker.patch('freqtrade.exchange.Exchange.get_min_pair_stake_amount', return_value=0.0) + mocker.patch('freqtrade.exchange.Exchange.cancel_order_with_result', + side_effect=InvalidOrderException()) freqtrade = FreqtradeBot(default_conf_usdt) diff --git a/tests/test_integration.py b/tests/test_integration.py index a7b4fbdd3..a848de5d3 100644 --- a/tests/test_integration.py +++ b/tests/test_integration.py @@ -2,7 +2,7 @@ from unittest.mock import MagicMock import pytest -from freqtrade.enums import ExitCheckTuple, ExitType +from freqtrade.enums import ExitCheckTuple, ExitType, TradingMode from freqtrade.persistence import Trade from freqtrade.persistence.models import Order from freqtrade.rpc.rpc import RPC @@ -455,10 +455,12 @@ def test_dca_order_adjust(default_conf_usdt, ticker_usdt, fee, mocker) -> None: assert pytest.approx(trade.orders[-1].amount) == 61.538461232 -def test_dca_exiting(default_conf_usdt, ticker_usdt, fee, mocker, caplog) -> None: +@pytest.mark.parametrize('leverage', [1, 2]) +def test_dca_exiting(default_conf_usdt, ticker_usdt, fee, mocker, caplog, leverage) -> None: default_conf_usdt['position_adjustment_enable'] = True freqtrade = get_patched_freqtradebot(mocker, default_conf_usdt) + freqtrade.trading_mode = TradingMode.FUTURES mocker.patch.multiple( 'freqtrade.exchange.Exchange', fetch_ticker=ticker_usdt, @@ -467,15 +469,17 @@ def test_dca_exiting(default_conf_usdt, ticker_usdt, fee, mocker, caplog) -> Non price_to_precision=lambda s, x, y: y, get_min_pair_stake_amount=MagicMock(return_value=10), ) + mocker.patch("freqtrade.exchange.Exchange.get_max_leverage", return_value=10) patch_get_signal(freqtrade) + freqtrade.strategy.leverage = MagicMock(return_value=leverage) freqtrade.enter_positions() assert len(Trade.get_trades().all()) == 1 trade = Trade.get_trades().first() assert len(trade.orders) == 1 assert pytest.approx(trade.stake_amount) == 60 - assert pytest.approx(trade.amount) == 30.0 + assert pytest.approx(trade.amount) == 30.0 * leverage assert trade.open_rate == 2.0 # Too small size @@ -484,8 +488,9 @@ def test_dca_exiting(default_conf_usdt, ticker_usdt, fee, mocker, caplog) -> Non trade = Trade.get_trades().first() assert len(trade.orders) == 1 assert pytest.approx(trade.stake_amount) == 60 - assert pytest.approx(trade.amount) == 30.0 - assert log_has_re("Remaining amount of 1.6.* would be smaller than the minimum of 10.", caplog) + assert pytest.approx(trade.amount) == 30.0 * leverage + assert log_has_re( + r"Remaining amount of \d\.\d+.* would be smaller than the minimum of 10.", caplog) freqtrade.strategy.adjust_trade_position = MagicMock(return_value=-20) @@ -494,7 +499,7 @@ def test_dca_exiting(default_conf_usdt, ticker_usdt, fee, mocker, caplog) -> Non assert len(trade.orders) == 2 assert trade.orders[-1].ft_order_side == 'sell' assert pytest.approx(trade.stake_amount) == 40.198 - assert pytest.approx(trade.amount) == 20.099 + assert pytest.approx(trade.amount) == 20.099 * leverage assert trade.open_rate == 2.0 assert trade.is_open caplog.clear()