diff --git a/docs/bot-usage.md b/docs/bot-usage.md index 5ec390d5c..55988985a 100644 --- a/docs/bot-usage.md +++ b/docs/bot-usage.md @@ -79,7 +79,7 @@ prevent unintended disclosure of sensitive private data when you publish example of your configuration in the project issues or in the Internet. See more details on this technique with examples in the documentation page on -[configuration](bot-configuration.md). +[configuration](configuration.md). ### How to use **--strategy**? diff --git a/docs/hyperopt.md b/docs/hyperopt.md index e25f35c35..b4e42de16 100644 --- a/docs/hyperopt.md +++ b/docs/hyperopt.md @@ -62,7 +62,7 @@ If you have updated the buy strategy, ie. changed the contents of #### Sell optimization -Similar to the buy-signal above, sell-signals can also be optimized. +Similar to the buy-signal above, sell-signals can also be optimized. Place the corresponding settings into the following methods * Inside `sell_indicator_space()` - the parameters hyperopt shall be optimizing. @@ -163,7 +163,7 @@ running at least several thousand evaluations. The `--spaces all` flag determines that all possible parameters should be optimized. Possibilities are listed below. !!! Warning -When switching parameters or changing configuration options, the file `user_data/hyperopt_results.pickle` should be removed. It's used to be able to continue interrupted calculations, but does not detect changes to settings or the hyperopt file. + When switching parameters or changing configuration options, the file `user_data/hyperopt_results.pickle` should be removed. It's used to be able to continue interrupted calculations, but does not detect changes to settings or the hyperopt file. ### Execute Hyperopt with Different Ticker-Data Source diff --git a/docs/sql_cheatsheet.md b/docs/sql_cheatsheet.md index 54f9b8213..f41520bd9 100644 --- a/docs/sql_cheatsheet.md +++ b/docs/sql_cheatsheet.md @@ -1,5 +1,5 @@ # SQL Helper -This page constains some help if you want to edit your sqlite db. +This page contains some help if you want to edit your sqlite db. ## Install sqlite3 **Ubuntu/Debian installation** @@ -66,11 +66,11 @@ SELECT * FROM trades; ## Fix trade still open after a manual sell on the exchange !!! Warning - Manually selling on the exchange should not be done by default, since the bot does not detect this and will try to sell anyway. - /foresell should accomplish the same thing. + Manually selling a pair on the exchange will not be detected by the bot and it will try to sell anyway. Whenever possible, forcesell should be used to accomplish the same thing. + It is strongly advised to backup your database file before making any manual changes. !!! Note - This should not be necessary after /forcesell, as forcesell orders are closed automatically by the bot on the next iteration. + This should not be necessary after /forcesell, as forcesell orders are closed automatically by the bot on the next iteration. ```sql UPDATE trades diff --git a/docs/telegram-usage.md b/docs/telegram-usage.md index 1ca61e54a..9d6877318 100644 --- a/docs/telegram-usage.md +++ b/docs/telegram-usage.md @@ -1,13 +1,13 @@ # Telegram usage -This page explains how to command your bot with Telegram. - ## Prerequisite + To control your bot with Telegram, you need first to [set up a Telegram bot](installation.md) and add your Telegram API keys into your config file. ## Telegram commands + Per default, the Telegram bot shows predefined commands. Some commands are only available by sending them to the bot. The table below list the official commands. You can ask at any moment for help with `/help`. @@ -65,16 +65,14 @@ Once all positions are sold, run `/stop` to completely stop the bot. For each open trade, the bot will send you the following message. -> **Trade ID:** `123` -> **Current Pair:** CVC/BTC -> **Open Since:** `1 days ago` -> **Amount:** `26.64180098` -> **Open Rate:** `0.00007489` -> **Close Rate:** `None` -> **Current Rate:** `0.00007489` -> **Close Profit:** `None` -> **Current Profit:** `12.95%` -> **Open Order:** `None` +> **Trade ID:** `123` `(since 1 days ago)` +> **Current Pair:** CVC/BTC +> **Open Since:** `1 days ago` +> **Amount:** `26.64180098` +> **Open Rate:** `0.00007489` +> **Current Rate:** `0.00007489` +> **Current Profit:** `12.95%` +> **Stoploss:** `0.00007389 (-0.02%)` ### /status table @@ -99,18 +97,18 @@ current max Return a summary of your profit/loss and performance. -> **ROI:** Close trades -> ∙ `0.00485701 BTC (258.45%)` -> ∙ `62.968 USD` -> **ROI:** All trades -> ∙ `0.00255280 BTC (143.43%)` -> ∙ `33.095 EUR` -> -> **Total Trade Count:** `138` -> **First Trade opened:** `3 days ago` -> **Latest Trade opened:** `2 minutes ago` -> **Avg. Duration:** `2:33:45` -> **Best Performing:** `PAY/BTC: 50.23%` +> **ROI:** Close trades +> ∙ `0.00485701 BTC (258.45%)` +> ∙ `62.968 USD` +> **ROI:** All trades +> ∙ `0.00255280 BTC (143.43%)` +> ∙ `33.095 EUR` +> +> **Total Trade Count:** `138` +> **First Trade opened:** `3 days ago` +> **Latest Trade opened:** `2 minutes ago` +> **Avg. Duration:** `2:33:45` +> **Best Performing:** `PAY/BTC: 50.23%` ### /forcesell @@ -128,26 +126,26 @@ Note that for this to work, `forcebuy_enable` needs to be set to true. Return the performance of each crypto-currency the bot has sold. > Performance: -> 1. `RCN/BTC 57.77%` -> 2. `PAY/BTC 56.91%` -> 3. `VIB/BTC 47.07%` -> 4. `SALT/BTC 30.24%` -> 5. `STORJ/BTC 27.24%` -> ... +> 1. `RCN/BTC 57.77%` +> 2. `PAY/BTC 56.91%` +> 3. `VIB/BTC 47.07%` +> 4. `SALT/BTC 30.24%` +> 5. `STORJ/BTC 27.24%` +> ... ### /balance Return the balance of all crypto-currency your have on the exchange. -> **Currency:** BTC -> **Available:** 3.05890234 -> **Balance:** 3.05890234 -> **Pending:** 0.0 +> **Currency:** BTC +> **Available:** 3.05890234 +> **Balance:** 3.05890234 +> **Pending:** 0.0 -> **Currency:** CVC -> **Available:** 86.64180098 -> **Balance:** 86.64180098 -> **Pending:** 0.0 +> **Currency:** CVC +> **Available:** 86.64180098 +> **Balance:** 86.64180098 +> **Pending:** 0.0 ### /daily diff --git a/docs/webhook-config.md b/docs/webhook-config.md index 2b5365e32..811b57f9b 100644 --- a/docs/webhook-config.md +++ b/docs/webhook-config.md @@ -1,7 +1,5 @@ # Webhook usage -This page explains how to configure your bot to talk to webhooks. - ## Configuration Enable webhooks by adding a webhook-section to your configuration file, and setting `webhook.enabled` to `true`. @@ -39,32 +37,30 @@ Different payloads can be configured for different events. Not all fields are ne The fields in `webhook.webhookbuy` are filled when the bot executes a buy. Parameters are filled using string.format. Possible parameters are: -* exchange -* pair -* limit -* stake_amount -* stake_amount_fiat -* stake_currency -* fiat_currency +* `exchange` +* `pair` +* `limit` +* `stake_amount` +* `stake_currency` +* `fiat_currency` ### Webhooksell The fields in `webhook.webhooksell` are filled when the bot sells a trade. Parameters are filled using string.format. Possible parameters are: -* exchange -* pair -* gain -* limit -* amount -* open_rate -* current_rate -* profit_amount -* profit_percent -* profit_fiat -* stake_currency -* fiat_currency -* sell_reason +* `exchange` +* `pair` +* `gain` +* `limit` +* `amount` +* `open_rate` +* `current_rate` +* `profit_amount` +* `profit_percent` +* `stake_currency` +* `fiat_currency` +* `sell_reason` ### Webhookstatus diff --git a/freqtrade/edge/__init__.py b/freqtrade/edge/__init__.py index b4dfa5624..4801c6cb3 100644 --- a/freqtrade/edge/__init__.py +++ b/freqtrade/edge/__init__.py @@ -203,6 +203,22 @@ class Edge(): return self._final_pairs + def accepted_pairs(self) -> list: + """ + return a list of accepted pairs along with their winrate, expectancy and stoploss + """ + final = [] + for pair, info in self._cached_pairs.items(): + if info.expectancy > float(self.edge_config.get('minimum_expectancy', 0.2)) and \ + info.winrate > float(self.edge_config.get('minimum_winrate', 0.60)): + final.append({ + 'Pair': pair, + 'Winrate': info.winrate, + 'Expectancy': info.expectancy, + 'Stoploss': info.stoploss, + }) + return final + def _fill_calculable_fields(self, result: DataFrame) -> DataFrame: """ The result frame contains a number of columns that are calculable diff --git a/freqtrade/freqtradebot.py b/freqtrade/freqtradebot.py index 3bab1758d..346bcfce3 100644 --- a/freqtrade/freqtradebot.py +++ b/freqtrade/freqtradebot.py @@ -4,14 +4,12 @@ Freqtrade is the main module of this bot. It contains the class Freqtrade() import copy import logging -import time import traceback from datetime import datetime -from typing import Any, Callable, Dict, List, Optional, Tuple +from typing import Any, Dict, List, Optional, Tuple import arrow from requests.exceptions import RequestException -import sdnotify from freqtrade import (DependencyException, OperationalException, InvalidOrderException, TemporaryError, __version__, constants, persistence) @@ -42,20 +40,14 @@ class FreqtradeBot(object): to get the config dict. """ - logger.info( - 'Starting freqtrade %s', - __version__, - ) + logger.info('Starting freqtrade %s', __version__) - # Init bot states + # Init bot state self.state = State.STOPPED # Init objects self.config = config - self._sd_notify = sdnotify.SystemdNotifier() if \ - self.config.get('internals', {}).get('sd_notify', False) else None - self.strategy: IStrategy = StrategyResolver(self.config).strategy self.rpc: RPCManager = RPCManager(self) @@ -79,29 +71,12 @@ class FreqtradeBot(object): self.config.get('edge', {}).get('enabled', False) else None self.active_pair_whitelist: List[str] = self.config['exchange']['pair_whitelist'] - self._init_modules() - - # Tell the systemd that we completed initialization phase - if self._sd_notify: - logger.debug("sd_notify: READY=1") - self._sd_notify.notify("READY=1") - - def _init_modules(self) -> None: - """ - Initializes all modules and updates the config - :return: None - """ - # Initialize all modules persistence.init(self.config) - # Set initial application state + # Set initial bot state from config initial_state = self.config.get('initial_state') - - if initial_state: - self.state = State[initial_state.upper()] - else: - self.state = State.STOPPED + self.state = State[initial_state.upper()] if initial_state else State.STOPPED def cleanup(self) -> None: """ @@ -113,130 +88,50 @@ class FreqtradeBot(object): self.rpc.cleanup() persistence.cleanup() - def stopping(self) -> None: - # Tell systemd that we are exiting now - if self._sd_notify: - logger.debug("sd_notify: STOPPING=1") - self._sd_notify.notify("STOPPING=1") - - def reconfigure(self) -> None: - # Tell systemd that we initiated reconfiguring - if self._sd_notify: - logger.debug("sd_notify: RELOADING=1") - self._sd_notify.notify("RELOADING=1") - - def worker(self, old_state: State = None) -> State: - """ - Trading routine that must be run at each loop - :param old_state: the previous service state from the previous call - :return: current service state - """ - # Log state transition - state = self.state - if state != old_state: - self.rpc.send_msg({ - 'type': RPCMessageType.STATUS_NOTIFICATION, - 'status': f'{state.name.lower()}' - }) - logger.info('Changing state to: %s', state.name) - if state == State.RUNNING: - self.rpc.startup_messages(self.config, self.pairlists) - - throttle_secs = self.config.get('internals', {}).get( - 'process_throttle_secs', - constants.PROCESS_THROTTLE_SECS - ) - - if state == State.STOPPED: - # Ping systemd watchdog before sleeping in the stopped state - if self._sd_notify: - logger.debug("sd_notify: WATCHDOG=1\\nSTATUS=State: STOPPED.") - self._sd_notify.notify("WATCHDOG=1\nSTATUS=State: STOPPED.") - - time.sleep(throttle_secs) - - elif state == State.RUNNING: - # Ping systemd watchdog before throttling - if self._sd_notify: - logger.debug("sd_notify: WATCHDOG=1\\nSTATUS=State: RUNNING.") - self._sd_notify.notify("WATCHDOG=1\nSTATUS=State: RUNNING.") - - self._throttle(func=self._process, min_secs=throttle_secs) - - return state - - def _throttle(self, func: Callable[..., Any], min_secs: float, *args, **kwargs) -> Any: - """ - Throttles the given callable that it - takes at least `min_secs` to finish execution. - :param func: Any callable - :param min_secs: minimum execution time in seconds - :return: Any - """ - start = time.time() - result = func(*args, **kwargs) - end = time.time() - duration = max(min_secs - (end - start), 0.0) - logger.debug('Throttling %s for %.2f seconds', func.__name__, duration) - time.sleep(duration) - return result - - def _process(self) -> bool: + def process(self) -> bool: """ Queries the persistence layer for open trades and handles them, otherwise a new trade is created. :return: True if one or more trades has been created or closed, False otherwise """ state_changed = False - try: - # Check whether markets have to be reloaded - self.exchange._reload_markets() - # Refresh whitelist - self.pairlists.refresh_pairlist() - self.active_pair_whitelist = self.pairlists.whitelist + # Check whether markets have to be reloaded + self.exchange._reload_markets() - # Calculating Edge positioning - if self.edge: - self.edge.calculate() - self.active_pair_whitelist = self.edge.adjust(self.active_pair_whitelist) + # Refresh whitelist + self.pairlists.refresh_pairlist() + self.active_pair_whitelist = self.pairlists.whitelist - # Query trades from persistence layer - trades = Trade.get_open_trades() + # Calculating Edge positioning + if self.edge: + self.edge.calculate() + self.active_pair_whitelist = self.edge.adjust(self.active_pair_whitelist) - # Extend active-pair whitelist with pairs from open trades - # It ensures that tickers are downloaded for open trades - self._extend_whitelist_with_trades(self.active_pair_whitelist, trades) + # Query trades from persistence layer + trades = Trade.get_open_trades() - # Refreshing candles - self.dataprovider.refresh(self._create_pair_whitelist(self.active_pair_whitelist), - self.strategy.informative_pairs()) + # Extend active-pair whitelist with pairs from open trades + # It ensures that tickers are downloaded for open trades + self._extend_whitelist_with_trades(self.active_pair_whitelist, trades) - # First process current opened trades - for trade in trades: - state_changed |= self.process_maybe_execute_sell(trade) + # Refreshing candles + self.dataprovider.refresh(self._create_pair_whitelist(self.active_pair_whitelist), + self.strategy.informative_pairs()) - # Then looking for buy opportunities - if len(trades) < self.config['max_open_trades']: - state_changed = self.process_maybe_execute_buy() + # First process current opened trades + for trade in trades: + state_changed |= self.process_maybe_execute_sell(trade) - if 'unfilledtimeout' in self.config: - # Check and handle any timed out open orders - self.check_handle_timedout() - Trade.session.flush() + # Then looking for buy opportunities + if len(trades) < self.config['max_open_trades']: + state_changed = self.process_maybe_execute_buy() + + if 'unfilledtimeout' in self.config: + # Check and handle any timed out open orders + self.check_handle_timedout() + Trade.session.flush() - except TemporaryError as error: - logger.warning(f"Error: {error}, retrying in {constants.RETRY_TIMEOUT} seconds...") - time.sleep(constants.RETRY_TIMEOUT) - except OperationalException: - tb = traceback.format_exc() - hint = 'Issue `/start` if you think it is safe to restart.' - self.rpc.send_msg({ - 'type': RPCMessageType.STATUS_NOTIFICATION, - 'status': f'OperationalException:\n```\n{tb}```{hint}' - }) - logger.exception('OperationalException. Stopping trader ...') - self.state = State.STOPPED return state_changed def _extend_whitelist_with_trades(self, whitelist: List[str], trades: List[Any]): @@ -356,6 +251,10 @@ class FreqtradeBot(object): interval = self.strategy.ticker_interval whitelist = copy.deepcopy(self.active_pair_whitelist) + if not whitelist: + logger.warning("Whitelist is empty.") + return False + # Remove currently opened and latest pairs from whitelist for trade in Trade.get_open_trades(): if trade.pair in whitelist: @@ -363,7 +262,8 @@ class FreqtradeBot(object): logger.debug('Ignoring %s in pair whitelist', trade.pair) if not whitelist: - raise DependencyException('No currency pairs in whitelist') + logger.info("No currency pair in whitelist, but checking to sell open trades.") + return False # running get_signal on historical data fetched for _pair in whitelist: diff --git a/freqtrade/main.py b/freqtrade/main.py index c41d54f0e..877e2921d 100755 --- a/freqtrade/main.py +++ b/freqtrade/main.py @@ -10,10 +10,9 @@ from typing import List from freqtrade import OperationalException from freqtrade.arguments import Arguments -from freqtrade.configuration import Configuration, set_loggers -from freqtrade.freqtradebot import FreqtradeBot -from freqtrade.state import State -from freqtrade.rpc import RPCMessageType +from freqtrade.configuration import set_loggers +from freqtrade.worker import Worker + logger = logging.getLogger('freqtrade') @@ -27,7 +26,7 @@ def main(sysargv: List[str]) -> None: sysargv, 'Free, open source crypto trading bot' ) - args = arguments.get_parsed_arg() + args: Namespace = arguments.get_parsed_arg() # A subcommand has been issued. # Means if Backtesting or Hyperopt have been called we exit the bot @@ -35,20 +34,12 @@ def main(sysargv: List[str]) -> None: args.func(args) return - freqtrade = None + worker = None return_code = 1 try: - # Load and validate configuration - config = Configuration(args, None).get_config() - - # Init the bot - freqtrade = FreqtradeBot(config) - - state = None - while True: - state = freqtrade.worker(old_state=state) - if state == State.RELOAD_CONF: - freqtrade = reconfigure(freqtrade, args) + # Load and run worker + worker = Worker(args) + worker.run() except KeyboardInterrupt: logger.info('SIGINT received, aborting ...') @@ -59,34 +50,11 @@ def main(sysargv: List[str]) -> None: except BaseException: logger.exception('Fatal exception!') finally: - if freqtrade: - freqtrade.stopping() - freqtrade.rpc.send_msg({ - 'type': RPCMessageType.STATUS_NOTIFICATION, - 'status': 'process died' - }) - freqtrade.cleanup() + if worker: + worker.exit() sys.exit(return_code) -def reconfigure(freqtrade: FreqtradeBot, args: Namespace) -> FreqtradeBot: - """ - Cleans up current instance, reloads the configuration and returns the new instance - """ - freqtrade.reconfigure() - - # Clean up current modules - freqtrade.cleanup() - - # Create new instance - freqtrade = FreqtradeBot(Configuration(args, None).get_config()) - freqtrade.rpc.send_msg({ - 'type': RPCMessageType.STATUS_NOTIFICATION, - 'status': 'config reloaded' - }) - return freqtrade - - if __name__ == '__main__': set_loggers() main(sys.argv[1:]) diff --git a/freqtrade/optimize/backtesting.py b/freqtrade/optimize/backtesting.py index 293511fc0..a7a3f2ba4 100644 --- a/freqtrade/optimize/backtesting.py +++ b/freqtrade/optimize/backtesting.py @@ -210,6 +210,32 @@ class Backtesting(object): logger.info('Dumping backtest results to %s', recordfilename) file_dump_json(recordfilename, records) + def _get_ticker_list(self, processed) -> Dict[str, DataFrame]: + """ + Helper function to convert a processed tickerlist into a list for performance reasons. + + Used by backtest() - so keep this optimized for performance. + """ + headers = ['date', 'buy', 'open', 'close', 'sell', 'low', 'high'] + ticker: Dict = {} + # Create ticker dict + for pair, pair_data in processed.items(): + pair_data['buy'], pair_data['sell'] = 0, 0 # cleanup from previous run + + ticker_data = self.advise_sell( + self.advise_buy(pair_data, {'pair': pair}), {'pair': pair})[headers].copy() + + # to avoid using data from future, we buy/sell with signal from previous candle + ticker_data.loc[:, 'buy'] = ticker_data['buy'].shift(1) + ticker_data.loc[:, 'sell'] = ticker_data['sell'].shift(1) + + ticker_data.drop(ticker_data.head(1).index, inplace=True) + + # Convert from Pandas to list for performance reasons + # (Looping Pandas is slow.) + ticker[pair] = [x for x in ticker_data.itertuples()] + return ticker + def _get_sell_trade_entry( self, pair: str, buy_row: DataFrame, partial_ticker: List, trade_count_lock: Dict, args: Dict) -> Optional[BacktestResult]: @@ -304,7 +330,6 @@ class Backtesting(object): position_stacking: do we allow position stacking? (default: False) :return: DataFrame """ - headers = ['date', 'buy', 'open', 'close', 'sell', 'low', 'high'] processed = args['processed'] max_open_trades = args.get('max_open_trades', 0) position_stacking = args.get('position_stacking', False) @@ -312,54 +337,50 @@ class Backtesting(object): end_date = args['end_date'] trades = [] trade_count_lock: Dict = {} - ticker: Dict = {} - pairs = [] - # Create ticker dict - for pair, pair_data in processed.items(): - pair_data['buy'], pair_data['sell'] = 0, 0 # cleanup from previous run - ticker_data = self.advise_sell( - self.advise_buy(pair_data, {'pair': pair}), {'pair': pair})[headers].copy() - - # to avoid using data from future, we buy/sell with signal from previous candle - ticker_data.loc[:, 'buy'] = ticker_data['buy'].shift(1) - ticker_data.loc[:, 'sell'] = ticker_data['sell'].shift(1) - - ticker_data.drop(ticker_data.head(1).index, inplace=True) - - # Convert from Pandas to list for performance reasons - # (Looping Pandas is slow.) - ticker[pair] = [x for x in ticker_data.itertuples()] - pairs.append(pair) + # Dict of ticker-lists for performance (looping lists is a lot faster than dataframes) + ticker: Dict = self._get_ticker_list(processed) lock_pair_until: Dict = {} + # Indexes per pair, so some pairs are allowed to have a missing start. + indexes: Dict = {} tmp = start_date + timedelta(minutes=self.ticker_interval_mins) - index = 0 - # Loop timerange and test per pair + + # Loop timerange and get candle for each pair at that point in time while tmp < end_date: - # print(f"time: {tmp}") + for i, pair in enumerate(ticker): + if pair not in indexes: + indexes[pair] = 0 + try: - row = ticker[pair][index] + row = ticker[pair][indexes[pair]] except IndexError: - # missing Data for one pair ... + # missing Data for one pair at the end. # Warnings for this are shown by `validate_backtest_data` continue + # Waits until the time-counter reaches the start of the data for this pair. + if row.date > tmp.datetime: + continue + + indexes[pair] += 1 + if row.buy == 0 or row.sell == 1: continue # skip rows where no buy signal or that would immediately sell off - if not position_stacking: - if pair in lock_pair_until and row.date <= lock_pair_until[pair]: - continue + if (not position_stacking and pair in lock_pair_until + and row.date <= lock_pair_until[pair]): + # without positionstacking, we can only have one open trade per pair. + continue + if max_open_trades > 0: # Check if max_open_trades has already been reached for the given date if not trade_count_lock.get(row.date, 0) < max_open_trades: continue - trade_count_lock[row.date] = trade_count_lock.get(row.date, 0) + 1 - trade_entry = self._get_sell_trade_entry(pair, row, ticker[pair][index + 1:], + trade_entry = self._get_sell_trade_entry(pair, row, ticker[pair][indexes[pair]:], trade_count_lock, args) if trade_entry: @@ -367,11 +388,10 @@ class Backtesting(object): trades.append(trade_entry) else: # Set lock_pair_until to end of testing period if trade could not be closed - # This happens only if the buy-signal was with the last candle - lock_pair_until[pair] = end_date + lock_pair_until[pair] = end_date.datetime + # Move time one configured time_interval ahead. tmp += timedelta(minutes=self.ticker_interval_mins) - index += 1 return DataFrame.from_records(trades, columns=BacktestResult._fields) def start(self) -> None: diff --git a/freqtrade/rpc/rpc.py b/freqtrade/rpc/rpc.py index 79db08fd3..88ade0c27 100644 --- a/freqtrade/rpc/rpc.py +++ b/freqtrade/rpc/rpc.py @@ -103,16 +103,23 @@ class RPC(object): results.append(dict( trade_id=trade.id, pair=trade.pair, + base_currency=self._freqtrade.config['stake_currency'], date=arrow.get(trade.open_date), open_rate=trade.open_rate, close_rate=trade.close_rate, current_rate=current_rate, amount=round(trade.amount, 8), + stake_amount=round(trade.stake_amount, 8), close_profit=fmt_close_profit, current_profit=round(current_profit * 100, 2), stop_loss=trade.stop_loss, + stop_loss_pct=(trade.stop_loss_pct * 100) + if trade.stop_loss_pct else None, + initial_stop_loss=trade.initial_stop_loss, + initial_stop_loss_pct=(trade.initial_stop_loss_pct * 100) + if trade.initial_stop_loss_pct else None, open_order='({} {} rem={:.8f})'.format( - order['type'], order['side'], order['remaining'] + order['type'], order['side'], order['remaining'] ) if order else None, )) return results @@ -481,13 +488,4 @@ class RPC(object): """ Returns information related to Edge """ if not self._freqtrade.edge: raise RPCException(f'Edge is not enabled.') - - return [ - { - 'Pair': k, - 'Winrate': v.winrate, - 'Expectancy': v.expectancy, - 'Stoploss': v.stoploss, - } - for k, v in self._freqtrade.edge._cached_pairs.items() - ] + return self._freqtrade.edge.accepted_pairs() diff --git a/freqtrade/rpc/telegram.py b/freqtrade/rpc/telegram.py index 7b36e8a1f..e17f73502 100644 --- a/freqtrade/rpc/telegram.py +++ b/freqtrade/rpc/telegram.py @@ -197,16 +197,25 @@ class Telegram(RPC): messages = [] for r in results: lines = [ - "*Trade ID:* `{trade_id}` (since `{date}`)", + "*Trade ID:* `{trade_id}` `(since {date})`", "*Current Pair:* {pair}", - "*Amount:* `{amount}`", + "*Amount:* `{amount} ({stake_amount} {base_currency})`", "*Open Rate:* `{open_rate:.8f}`", "*Close Rate:* `{close_rate}`" if r['close_rate'] else "", "*Current Rate:* `{current_rate:.8f}`", "*Close Profit:* `{close_profit}`" if r['close_profit'] else "", "*Current Profit:* `{current_profit:.2f}%`", - "*Stoploss:* `{stop_loss:.8f}`", - "*Open Order:* `{open_order}`" if r['open_order'] else "", + + # Adding initial stoploss only if it is different from stoploss + "*Initial Stoploss:* `{initial_stop_loss:.8f}` " + + ("`({initial_stop_loss_pct:.2f}%)`" if r['initial_stop_loss_pct'] else "") + if r['stop_loss'] != r['initial_stop_loss'] else "", + + # Adding stoploss and stoploss percentage only if it is not None + "*Stoploss:* `{stop_loss:.8f}` " + + ("`({stop_loss_pct:.2f}%)`" if r['stop_loss_pct'] else ""), + + "*Open Order:* `{open_order}`" if r['open_order'] else "" ] messages.append("\n".join(filter(None, lines)).format(**r)) @@ -502,7 +511,6 @@ class Telegram(RPC): """ try: edge_pairs = self._rpc_edge() - print(edge_pairs) edge_pairs_tab = tabulate(edge_pairs, headers='keys', tablefmt='simple') message = f'Edge only validated following pairs:\n
{edge_pairs_tab}
' self._send_msg(message, bot=bot, parse_mode=ParseMode.HTML) diff --git a/freqtrade/tests/conftest.py b/freqtrade/tests/conftest.py index cba754199..0bff1d5e9 100644 --- a/freqtrade/tests/conftest.py +++ b/freqtrade/tests/conftest.py @@ -16,6 +16,7 @@ from freqtrade.edge import Edge, PairInfo from freqtrade.exchange import Exchange from freqtrade.freqtradebot import FreqtradeBot from freqtrade.resolvers import ExchangeResolver +from freqtrade.worker import Worker logging.getLogger('').setLevel(logging.INFO) @@ -87,7 +88,7 @@ def get_patched_edge(mocker, config) -> Edge: # Functions for recurrent object patching -def get_patched_freqtradebot(mocker, config) -> FreqtradeBot: +def patch_freqtradebot(mocker, config) -> None: """ This function patch _init_modules() to not call dependencies :param mocker: a Mocker object to apply patches @@ -100,9 +101,17 @@ def get_patched_freqtradebot(mocker, config) -> FreqtradeBot: mocker.patch('freqtrade.freqtradebot.RPCManager._init', MagicMock()) mocker.patch('freqtrade.freqtradebot.RPCManager.send_msg', MagicMock()) + +def get_patched_freqtradebot(mocker, config) -> FreqtradeBot: + patch_freqtradebot(mocker, config) return FreqtradeBot(config) +def get_patched_worker(mocker, config) -> Worker: + patch_freqtradebot(mocker, config) + return Worker(args=None, config=config) + + @pytest.fixture(autouse=True) def patch_coinmarketcap(mocker) -> None: """ diff --git a/freqtrade/tests/edge/test_edge.py b/freqtrade/tests/edge/test_edge.py index c1c1b49cd..af8674188 100644 --- a/freqtrade/tests/edge/test_edge.py +++ b/freqtrade/tests/edge/test_edge.py @@ -122,8 +122,8 @@ def test_edge_results(edge_conf, mocker, caplog, data) -> None: for c, trade in enumerate(data.trades): res = results.iloc[c] assert res.exit_type == trade.sell_reason - assert res.open_time == _get_frame_time_from_offset(trade.open_tick) - assert res.close_time == _get_frame_time_from_offset(trade.close_tick) + assert arrow.get(res.open_time) == _get_frame_time_from_offset(trade.open_tick) + assert arrow.get(res.close_time) == _get_frame_time_from_offset(trade.close_tick) def test_adjust(mocker, edge_conf): diff --git a/freqtrade/tests/optimize/__init__.py b/freqtrade/tests/optimize/__init__.py index 075938a61..bbcd0e45e 100644 --- a/freqtrade/tests/optimize/__init__.py +++ b/freqtrade/tests/optimize/__init__.py @@ -33,7 +33,7 @@ class BTContainer(NamedTuple): def _get_frame_time_from_offset(offset): return ticker_start_time.shift(minutes=(offset * TICKER_INTERVAL_MINUTES[tests_ticker_interval]) - ).datetime.replace(tzinfo=None) + ).datetime def _build_backtest_dataframe(ticker_with_signals): diff --git a/freqtrade/tests/optimize/test_backtesting.py b/freqtrade/tests/optimize/test_backtesting.py index d0b21b8f4..0596cffb5 100644 --- a/freqtrade/tests/optimize/test_backtesting.py +++ b/freqtrade/tests/optimize/test_backtesting.py @@ -685,25 +685,32 @@ def test_backtest_alternate_buy_sell(default_conf, fee, mocker): assert len(results.loc[results.open_at_end]) == 0 -def test_backtest_multi_pair(default_conf, fee, mocker): +@pytest.mark.parametrize("pair", ['ADA/BTC', 'LTC/BTC']) +@pytest.mark.parametrize("tres", [0, 20, 30]) +def test_backtest_multi_pair(default_conf, fee, mocker, tres, pair): def _trend_alternate_hold(dataframe=None, metadata=None): """ - Buy every 8th candle - sell every other 8th -2 (hold on to pairs a bit) + Buy every xth candle - sell every other xth -2 (hold on to pairs a bit) """ - multi = 8 + if metadata['pair'] in('ETH/BTC', 'LTC/BTC'): + multi = 20 + else: + multi = 18 dataframe['buy'] = np.where(dataframe.index % multi == 0, 1, 0) dataframe['sell'] = np.where((dataframe.index + multi - 2) % multi == 0, 1, 0) - if metadata['pair'] in('ETH/BTC', 'LTC/BTC'): - dataframe['buy'] = dataframe['buy'].shift(-4) - dataframe['sell'] = dataframe['sell'].shift(-4) return dataframe mocker.patch('freqtrade.exchange.Exchange.get_fee', fee) patch_exchange(mocker) + pairs = ['ADA/BTC', 'DASH/BTC', 'ETH/BTC', 'LTC/BTC', 'NXT/BTC'] data = history.load_data(datadir=None, ticker_interval='5m', pairs=pairs) + # Only use 500 lines to increase performance data = trim_dictlist(data, -500) + + # Remove data for one pair from the beginning of the data + data[pair] = data[pair][tres:] # We need to enable sell-signal - otherwise it sells on ROI!! default_conf['experimental'] = {"use_sell_signal": True} default_conf['ticker_interval'] = '5m' diff --git a/freqtrade/tests/rpc/test_rpc.py b/freqtrade/tests/rpc/test_rpc.py index 627768ec2..8b10a1314 100644 --- a/freqtrade/tests/rpc/test_rpc.py +++ b/freqtrade/tests/rpc/test_rpc.py @@ -51,14 +51,19 @@ def test_rpc_trade_status(default_conf, ticker, fee, markets, mocker) -> None: assert { 'trade_id': 1, 'pair': 'ETH/BTC', + 'base_currency': 'BTC', 'date': ANY, 'open_rate': 1.099e-05, 'close_rate': None, 'current_rate': 1.098e-05, 'amount': 90.99181074, + 'stake_amount': 0.001, 'close_profit': None, 'current_profit': -0.59, 'stop_loss': 0.0, + 'initial_stop_loss': 0.0, + 'initial_stop_loss_pct': None, + 'stop_loss_pct': None, 'open_order': '(limit buy rem=0.00000000)' } == results[0] @@ -72,14 +77,19 @@ def test_rpc_trade_status(default_conf, ticker, fee, markets, mocker) -> None: assert { 'trade_id': 1, 'pair': 'ETH/BTC', + 'base_currency': 'BTC', 'date': ANY, 'open_rate': 1.099e-05, 'close_rate': None, 'current_rate': ANY, 'amount': 90.99181074, + 'stake_amount': 0.001, 'close_profit': None, 'current_profit': ANY, 'stop_loss': 0.0, + 'initial_stop_loss': 0.0, + 'initial_stop_loss_pct': None, + 'stop_loss_pct': None, 'open_order': '(limit buy rem=0.00000000)' } == results[0] diff --git a/freqtrade/tests/rpc/test_rpc_telegram.py b/freqtrade/tests/rpc/test_rpc_telegram.py index fb2d71d4f..8f43d7ed0 100644 --- a/freqtrade/tests/rpc/test_rpc_telegram.py +++ b/freqtrade/tests/rpc/test_rpc_telegram.py @@ -189,14 +189,19 @@ def test_status(default_conf, update, mocker, fee, ticker, markets) -> None: _rpc_trade_status=MagicMock(return_value=[{ 'trade_id': 1, 'pair': 'ETH/BTC', + 'base_currency': 'BTC', 'date': arrow.utcnow(), 'open_rate': 1.099e-05, 'close_rate': None, 'current_rate': 1.098e-05, 'amount': 90.99181074, + 'stake_amount': 90.99181074, 'close_profit': None, 'current_profit': -0.59, + 'initial_stop_loss': 1.098e-05, 'stop_loss': 1.099e-05, + 'initial_stop_loss_pct': -0.05, + 'stop_loss_pct': -0.01, 'open_order': '(limit buy rem=0.00000000)' }]), _status_table=status_table, diff --git a/freqtrade/tests/test_freqtradebot.py b/freqtrade/tests/test_freqtradebot.py index 9687fe903..103c0777e 100644 --- a/freqtrade/tests/test_freqtradebot.py +++ b/freqtrade/tests/test_freqtradebot.py @@ -21,12 +21,13 @@ from freqtrade.state import State from freqtrade.strategy.interface import SellCheckTuple, SellType from freqtrade.tests.conftest import (log_has, log_has_re, patch_edge, patch_exchange, patch_wallet) +from freqtrade.worker import Worker # Functions for recurrent object patching -def get_patched_freqtradebot(mocker, config) -> FreqtradeBot: +def patch_freqtradebot(mocker, config) -> None: """ - This function patch _init_modules() to not call dependencies + This function patches _init_modules() to not call dependencies :param mocker: a Mocker object to apply patches :param config: Config to pass to the bot :return: None @@ -35,9 +36,29 @@ def get_patched_freqtradebot(mocker, config) -> FreqtradeBot: mocker.patch('freqtrade.freqtradebot.persistence.init', MagicMock()) patch_exchange(mocker) + +def get_patched_freqtradebot(mocker, config) -> FreqtradeBot: + """ + This function patches _init_modules() to not call dependencies + :param mocker: a Mocker object to apply patches + :param config: Config to pass to the bot + :return: FreqtradeBot + """ + patch_freqtradebot(mocker, config) return FreqtradeBot(config) +def get_patched_worker(mocker, config) -> Worker: + """ + This function patches _init_modules() to not call dependencies + :param mocker: a Mocker object to apply patches + :param config: Config to pass to the bot + :return: Worker + """ + patch_freqtradebot(mocker, config) + return Worker(args=None, config=config) + + def patch_get_signal(freqtrade: FreqtradeBot, value=(True, False)) -> None: """ :param mocker: mocker to patch IStrategy class @@ -61,7 +82,7 @@ def patch_RPCManager(mocker) -> MagicMock: # Unit tests -def test_freqtradebot(mocker, default_conf, markets) -> None: +def test_freqtradebot_state(mocker, default_conf, markets) -> None: mocker.patch('freqtrade.exchange.Exchange.markets', PropertyMock(return_value=markets)) freqtrade = get_patched_freqtradebot(mocker, default_conf) assert freqtrade.state is State.RUNNING @@ -71,6 +92,16 @@ def test_freqtradebot(mocker, default_conf, markets) -> None: assert freqtrade.state is State.STOPPED +def test_worker_state(mocker, default_conf, markets) -> None: + mocker.patch('freqtrade.exchange.Exchange.markets', PropertyMock(return_value=markets)) + worker = get_patched_worker(mocker, default_conf) + assert worker.state is State.RUNNING + + default_conf.pop('initial_state') + worker = Worker(args=None, config=default_conf) + assert worker.state is State.STOPPED + + def test_cleanup(mocker, default_conf, caplog) -> None: mock_cleanup = MagicMock() mocker.patch('freqtrade.persistence.cleanup', mock_cleanup) @@ -82,28 +113,28 @@ def test_cleanup(mocker, default_conf, caplog) -> None: def test_worker_running(mocker, default_conf, caplog) -> None: mock_throttle = MagicMock() - mocker.patch('freqtrade.freqtradebot.FreqtradeBot._throttle', mock_throttle) + mocker.patch('freqtrade.worker.Worker._throttle', mock_throttle) - freqtrade = get_patched_freqtradebot(mocker, default_conf) + worker = get_patched_worker(mocker, default_conf) - state = freqtrade.worker(old_state=None) + state = worker._worker(old_state=None) assert state is State.RUNNING assert log_has('Changing state to: RUNNING', caplog.record_tuples) assert mock_throttle.call_count == 1 # Check strategy is loaded, and received a dataprovider object - assert freqtrade.strategy - assert freqtrade.strategy.dp - assert isinstance(freqtrade.strategy.dp, DataProvider) + assert worker.freqtrade.strategy + assert worker.freqtrade.strategy.dp + assert isinstance(worker.freqtrade.strategy.dp, DataProvider) def test_worker_stopped(mocker, default_conf, caplog) -> None: mock_throttle = MagicMock() - mocker.patch('freqtrade.freqtradebot.FreqtradeBot._throttle', mock_throttle) + mocker.patch('freqtrade.worker.Worker._throttle', mock_throttle) mock_sleep = mocker.patch('time.sleep', return_value=None) - freqtrade = get_patched_freqtradebot(mocker, default_conf) - freqtrade.state = State.STOPPED - state = freqtrade.worker(old_state=State.RUNNING) + worker = get_patched_worker(mocker, default_conf) + worker.state = State.STOPPED + state = worker._worker(old_state=State.RUNNING) assert state is State.STOPPED assert log_has('Changing state to: STOPPED', caplog.record_tuples) assert mock_throttle.call_count == 0 @@ -115,17 +146,17 @@ def test_throttle(mocker, default_conf, caplog) -> None: return 42 caplog.set_level(logging.DEBUG) - freqtrade = get_patched_freqtradebot(mocker, default_conf) + worker = get_patched_worker(mocker, default_conf) start = time.time() - result = freqtrade._throttle(throttled_func, min_secs=0.1) + result = worker._throttle(throttled_func, min_secs=0.1) end = time.time() assert result == 42 assert end - start > 0.1 assert log_has('Throttling throttled_func for 0.10 seconds', caplog.record_tuples) - result = freqtrade._throttle(throttled_func, min_secs=-1) + result = worker._throttle(throttled_func, min_secs=-1) assert result == 42 @@ -133,12 +164,12 @@ def test_throttle_with_assets(mocker, default_conf) -> None: def throttled_func(nb_assets=-1): return nb_assets - freqtrade = get_patched_freqtradebot(mocker, default_conf) + worker = get_patched_worker(mocker, default_conf) - result = freqtrade._throttle(throttled_func, min_secs=0.1, nb_assets=666) + result = worker._throttle(throttled_func, min_secs=0.1, nb_assets=666) assert result == 666 - result = freqtrade._throttle(throttled_func, min_secs=0.1) + result = worker._throttle(throttled_func, min_secs=0.1) assert result == -1 @@ -224,7 +255,7 @@ def test_edge_called_in_process(mocker, edge_conf) -> None: freqtrade = FreqtradeBot(edge_conf) freqtrade.pairlists._validate_whitelist = _refresh_whitelist patch_get_signal(freqtrade) - freqtrade._process() + freqtrade.process() assert freqtrade.active_pair_whitelist == ['NEO/BTC', 'LTC/BTC'] @@ -551,8 +582,7 @@ def test_create_trade_too_small_stake_amount(default_conf, ticker, limit_buy_ord freqtrade = FreqtradeBot(default_conf) patch_get_signal(freqtrade) - result = freqtrade.create_trade() - assert result is False + assert not freqtrade.create_trade() def test_create_trade_limit_reached(default_conf, ticker, limit_buy_order, @@ -573,11 +603,12 @@ def test_create_trade_limit_reached(default_conf, ticker, limit_buy_order, freqtrade = FreqtradeBot(default_conf) patch_get_signal(freqtrade) - assert freqtrade.create_trade() is False + assert not freqtrade.create_trade() assert freqtrade._get_trade_stake_amount('ETH/BTC') is None -def test_create_trade_no_pairs(default_conf, ticker, limit_buy_order, fee, markets, mocker) -> None: +def test_create_trade_no_pairs_let(default_conf, ticker, limit_buy_order, fee, + markets, mocker, caplog) -> None: patch_RPCManager(mocker) patch_exchange(mocker) mocker.patch.multiple( @@ -589,18 +620,17 @@ def test_create_trade_no_pairs(default_conf, ticker, limit_buy_order, fee, marke ) default_conf['exchange']['pair_whitelist'] = ["ETH/BTC"] - default_conf['exchange']['pair_blacklist'] = ["ETH/BTC"] freqtrade = FreqtradeBot(default_conf) patch_get_signal(freqtrade) - freqtrade.create_trade() - - with pytest.raises(DependencyException, match=r'.*No currency pairs in whitelist.*'): - freqtrade.create_trade() + assert freqtrade.create_trade() + assert not freqtrade.create_trade() + assert log_has("No currency pair in whitelist, but checking to sell open trades.", + caplog.record_tuples) -def test_create_trade_no_pairs_after_blacklist(default_conf, ticker, - limit_buy_order, fee, markets, mocker) -> None: +def test_create_trade_no_pairs_in_whitelist(default_conf, ticker, limit_buy_order, fee, + markets, mocker, caplog) -> None: patch_RPCManager(mocker) patch_exchange(mocker) mocker.patch.multiple( @@ -610,15 +640,12 @@ def test_create_trade_no_pairs_after_blacklist(default_conf, ticker, get_fee=fee, markets=PropertyMock(return_value=markets) ) - default_conf['exchange']['pair_whitelist'] = ["ETH/BTC"] - default_conf['exchange']['pair_blacklist'] = ["ETH/BTC"] + default_conf['exchange']['pair_whitelist'] = [] freqtrade = FreqtradeBot(default_conf) patch_get_signal(freqtrade) - freqtrade.create_trade() - - with pytest.raises(DependencyException, match=r'.*No currency pairs in whitelist.*'): - freqtrade.create_trade() + assert not freqtrade.create_trade() + assert log_has("Whitelist is empty.", caplog.record_tuples) def test_create_trade_no_signal(default_conf, fee, mocker) -> None: @@ -658,7 +685,7 @@ def test_process_trade_creation(default_conf, ticker, limit_buy_order, trades = Trade.query.filter(Trade.is_open.is_(True)).all() assert not trades - result = freqtrade._process() + result = freqtrade.process() assert result is True trades = Trade.query.filter(Trade.is_open.is_(True)).all() @@ -689,10 +716,10 @@ def test_process_exchange_failures(default_conf, ticker, markets, mocker) -> Non ) sleep_mock = mocker.patch('time.sleep', side_effect=lambda _: None) - freqtrade = FreqtradeBot(default_conf) - patch_get_signal(freqtrade) + worker = Worker(args=None, config=default_conf) + patch_get_signal(worker.freqtrade) - result = freqtrade._process() + result = worker._process() assert result is False assert sleep_mock.has_calls() @@ -706,14 +733,14 @@ def test_process_operational_exception(default_conf, ticker, markets, mocker) -> markets=PropertyMock(return_value=markets), buy=MagicMock(side_effect=OperationalException) ) - freqtrade = FreqtradeBot(default_conf) - patch_get_signal(freqtrade) + worker = Worker(args=None, config=default_conf) + patch_get_signal(worker.freqtrade) - assert freqtrade.state == State.RUNNING + assert worker.state == State.RUNNING - result = freqtrade._process() + result = worker._process() assert result is False - assert freqtrade.state == State.STOPPED + assert worker.state == State.STOPPED assert 'OperationalException' in msg_mock.call_args_list[-1][0][0]['status'] @@ -734,18 +761,18 @@ def test_process_trade_handling( trades = Trade.query.filter(Trade.is_open.is_(True)).all() assert not trades - result = freqtrade._process() + result = freqtrade.process() assert result is True trades = Trade.query.filter(Trade.is_open.is_(True)).all() assert len(trades) == 1 - result = freqtrade._process() + result = freqtrade.process() assert result is False def test_process_trade_no_whitelist_pair( default_conf, ticker, limit_buy_order, markets, fee, mocker) -> None: - """ Test _process with trade not in pair list """ + """ Test process with trade not in pair list """ patch_RPCManager(mocker) patch_exchange(mocker) mocker.patch.multiple( @@ -782,7 +809,7 @@ def test_process_trade_no_whitelist_pair( )) assert pair not in freqtrade.active_pair_whitelist - result = freqtrade._process() + result = freqtrade.process() assert pair in freqtrade.active_pair_whitelist # Make sure each pair is only in the list once assert len(freqtrade.active_pair_whitelist) == len(set(freqtrade.active_pair_whitelist)) @@ -812,7 +839,7 @@ def test_process_informative_pairs_added(default_conf, ticker, markets, mocker) freqtrade.strategy.informative_pairs = inf_pairs # patch_get_signal(freqtrade) - freqtrade._process() + freqtrade.process() assert inf_pairs.call_count == 1 assert refresh_mock.call_count == 1 assert ("BTC/ETH", "1m") in refresh_mock.call_args[0][0] @@ -3083,5 +3110,5 @@ def test_startup_messages(default_conf, mocker): 'config': {'number_assets': 20} } mocker.patch('freqtrade.exchange.Exchange.exchange_has', MagicMock(return_value=True)) - freqtrade = get_patched_freqtradebot(mocker, default_conf) - assert freqtrade.state is State.RUNNING + worker = get_patched_worker(mocker, default_conf) + assert worker.state is State.RUNNING diff --git a/freqtrade/tests/test_main.py b/freqtrade/tests/test_main.py index 51c95a4a9..fc5d2e378 100644 --- a/freqtrade/tests/test_main.py +++ b/freqtrade/tests/test_main.py @@ -7,8 +7,8 @@ import pytest from freqtrade import OperationalException from freqtrade.arguments import Arguments -from freqtrade.freqtradebot import FreqtradeBot -from freqtrade.main import main, reconfigure +from freqtrade.worker import Worker +from freqtrade.main import main from freqtrade.state import State from freqtrade.tests.conftest import log_has, patch_exchange @@ -43,17 +43,14 @@ def test_main_start_hyperopt(mocker) -> None: def test_main_fatal_exception(mocker, default_conf, caplog) -> None: patch_exchange(mocker) - mocker.patch.multiple( - 'freqtrade.freqtradebot.FreqtradeBot', - _init_modules=MagicMock(), - worker=MagicMock(side_effect=Exception), - cleanup=MagicMock(), - ) + mocker.patch('freqtrade.freqtradebot.FreqtradeBot.cleanup', MagicMock()) + mocker.patch('freqtrade.worker.Worker._worker', MagicMock(side_effect=Exception)) mocker.patch( 'freqtrade.configuration.Configuration._load_config_file', lambda *args, **kwargs: default_conf ) mocker.patch('freqtrade.freqtradebot.RPCManager', MagicMock()) + mocker.patch('freqtrade.freqtradebot.persistence.init', MagicMock()) args = ['-c', 'config.json.example'] @@ -66,17 +63,14 @@ def test_main_fatal_exception(mocker, default_conf, caplog) -> None: def test_main_keyboard_interrupt(mocker, default_conf, caplog) -> None: patch_exchange(mocker) - mocker.patch.multiple( - 'freqtrade.freqtradebot.FreqtradeBot', - _init_modules=MagicMock(), - worker=MagicMock(side_effect=KeyboardInterrupt), - cleanup=MagicMock(), - ) + mocker.patch('freqtrade.freqtradebot.FreqtradeBot.cleanup', MagicMock()) + mocker.patch('freqtrade.worker.Worker._worker', MagicMock(side_effect=KeyboardInterrupt)) mocker.patch( 'freqtrade.configuration.Configuration._load_config_file', lambda *args, **kwargs: default_conf ) mocker.patch('freqtrade.freqtradebot.RPCManager', MagicMock()) + mocker.patch('freqtrade.freqtradebot.persistence.init', MagicMock()) args = ['-c', 'config.json.example'] @@ -89,17 +83,17 @@ def test_main_keyboard_interrupt(mocker, default_conf, caplog) -> None: def test_main_operational_exception(mocker, default_conf, caplog) -> None: patch_exchange(mocker) - mocker.patch.multiple( - 'freqtrade.freqtradebot.FreqtradeBot', - _init_modules=MagicMock(), - worker=MagicMock(side_effect=OperationalException('Oh snap!')), - cleanup=MagicMock(), + mocker.patch('freqtrade.freqtradebot.FreqtradeBot.cleanup', MagicMock()) + mocker.patch( + 'freqtrade.worker.Worker._worker', + MagicMock(side_effect=OperationalException('Oh snap!')) ) mocker.patch( 'freqtrade.configuration.Configuration._load_config_file', lambda *args, **kwargs: default_conf ) mocker.patch('freqtrade.freqtradebot.RPCManager', MagicMock()) + mocker.patch('freqtrade.freqtradebot.persistence.init', MagicMock()) args = ['-c', 'config.json.example'] @@ -112,21 +106,18 @@ def test_main_operational_exception(mocker, default_conf, caplog) -> None: def test_main_reload_conf(mocker, default_conf, caplog) -> None: patch_exchange(mocker) - mocker.patch.multiple( - 'freqtrade.freqtradebot.FreqtradeBot', - _init_modules=MagicMock(), - worker=MagicMock(return_value=State.RELOAD_CONF), - cleanup=MagicMock(), - ) + mocker.patch('freqtrade.freqtradebot.FreqtradeBot.cleanup', MagicMock()) + mocker.patch('freqtrade.worker.Worker._worker', MagicMock(return_value=State.RELOAD_CONF)) mocker.patch( 'freqtrade.configuration.Configuration._load_config_file', lambda *args, **kwargs: default_conf ) mocker.patch('freqtrade.freqtradebot.RPCManager', MagicMock()) + mocker.patch('freqtrade.freqtradebot.persistence.init', MagicMock()) # Raise exception as side effect to avoid endless loop reconfigure_mock = mocker.patch( - 'freqtrade.main.reconfigure', MagicMock(side_effect=Exception) + 'freqtrade.main.Worker._reconfigure', MagicMock(side_effect=Exception) ) with pytest.raises(SystemExit): @@ -138,19 +129,21 @@ def test_main_reload_conf(mocker, default_conf, caplog) -> None: def test_reconfigure(mocker, default_conf) -> None: patch_exchange(mocker) - mocker.patch.multiple( - 'freqtrade.freqtradebot.FreqtradeBot', - _init_modules=MagicMock(), - worker=MagicMock(side_effect=OperationalException('Oh snap!')), - cleanup=MagicMock(), + mocker.patch('freqtrade.freqtradebot.FreqtradeBot.cleanup', MagicMock()) + mocker.patch( + 'freqtrade.worker.Worker._worker', + MagicMock(side_effect=OperationalException('Oh snap!')) ) mocker.patch( 'freqtrade.configuration.Configuration._load_config_file', lambda *args, **kwargs: default_conf ) mocker.patch('freqtrade.freqtradebot.RPCManager', MagicMock()) + mocker.patch('freqtrade.freqtradebot.persistence.init', MagicMock()) - freqtrade = FreqtradeBot(default_conf) + args = Arguments(['-c', 'config.json.example'], '').get_parsed_arg() + worker = Worker(args=args, config=default_conf) + freqtrade = worker.freqtrade # Renew mock to return modified data conf = deepcopy(default_conf) @@ -160,11 +153,10 @@ def test_reconfigure(mocker, default_conf) -> None: lambda *args, **kwargs: conf ) + worker._config = conf # reconfigure should return a new instance - freqtrade2 = reconfigure( - freqtrade, - Arguments(['-c', 'config.json.example'], '').get_parsed_arg() - ) + worker._reconfigure() + freqtrade2 = worker.freqtrade # Verify we have a new instance with the new config assert freqtrade is not freqtrade2 diff --git a/freqtrade/worker.py b/freqtrade/worker.py new file mode 100755 index 000000000..c7afe5c97 --- /dev/null +++ b/freqtrade/worker.py @@ -0,0 +1,188 @@ +""" +Main Freqtrade worker class. +""" +import logging +import time +import traceback +from argparse import Namespace +from typing import Any, Callable, Optional +import sdnotify + +from freqtrade import (constants, OperationalException, TemporaryError, + __version__) +from freqtrade.configuration import Configuration +from freqtrade.freqtradebot import FreqtradeBot +from freqtrade.state import State +from freqtrade.rpc import RPCMessageType + + +logger = logging.getLogger(__name__) + + +class Worker(object): + """ + Freqtradebot worker class + """ + + def __init__(self, args: Namespace, config=None) -> None: + """ + Init all variables and objects the bot needs to work + """ + logger.info('Starting worker %s', __version__) + + self._args = args + self._config = config + self._init(False) + + # Tell systemd that we completed initialization phase + if self._sd_notify: + logger.debug("sd_notify: READY=1") + self._sd_notify.notify("READY=1") + + def _init(self, reconfig: bool): + """ + Also called from the _reconfigure() method (with reconfig=True). + """ + if reconfig or self._config is None: + # Load configuration + self._config = Configuration(self._args, None).get_config() + + # Init the instance of the bot + self.freqtrade = FreqtradeBot(self._config) + + self._throttle_secs = self._config.get('internals', {}).get( + 'process_throttle_secs', + constants.PROCESS_THROTTLE_SECS + ) + + self._sd_notify = sdnotify.SystemdNotifier() if \ + self._config.get('internals', {}).get('sd_notify', False) else None + + @property + def state(self) -> State: + return self.freqtrade.state + + @state.setter + def state(self, value: State): + self.freqtrade.state = value + + def run(self): + state = None + while True: + state = self._worker(old_state=state) + if state == State.RELOAD_CONF: + self.freqtrade = self._reconfigure() + + def _worker(self, old_state: State, throttle_secs: Optional[float] = None) -> State: + """ + Trading routine that must be run at each loop + :param old_state: the previous service state from the previous call + :return: current service state + """ + state = self.freqtrade.state + if throttle_secs is None: + throttle_secs = self._throttle_secs + + # Log state transition + if state != old_state: + self.freqtrade.rpc.send_msg({ + 'type': RPCMessageType.STATUS_NOTIFICATION, + 'status': f'{state.name.lower()}' + }) + logger.info('Changing state to: %s', state.name) + if state == State.RUNNING: + self.freqtrade.rpc.startup_messages(self._config, self.freqtrade.pairlists) + + if state == State.STOPPED: + # Ping systemd watchdog before sleeping in the stopped state + if self._sd_notify: + logger.debug("sd_notify: WATCHDOG=1\\nSTATUS=State: STOPPED.") + self._sd_notify.notify("WATCHDOG=1\nSTATUS=State: STOPPED.") + + time.sleep(throttle_secs) + + elif state == State.RUNNING: + # Ping systemd watchdog before throttling + if self._sd_notify: + logger.debug("sd_notify: WATCHDOG=1\\nSTATUS=State: RUNNING.") + self._sd_notify.notify("WATCHDOG=1\nSTATUS=State: RUNNING.") + + self._throttle(func=self._process, min_secs=throttle_secs) + + return state + + def _throttle(self, func: Callable[..., Any], min_secs: float, *args, **kwargs) -> Any: + """ + Throttles the given callable that it + takes at least `min_secs` to finish execution. + :param func: Any callable + :param min_secs: minimum execution time in seconds + :return: Any + """ + start = time.time() + result = func(*args, **kwargs) + end = time.time() + duration = max(min_secs - (end - start), 0.0) + logger.debug('Throttling %s for %.2f seconds', func.__name__, duration) + time.sleep(duration) + return result + + def _process(self) -> bool: + state_changed = False + try: + state_changed = self.freqtrade.process() + except TemporaryError as error: + logger.warning(f"Error: {error}, retrying in {constants.RETRY_TIMEOUT} seconds...") + time.sleep(constants.RETRY_TIMEOUT) + except OperationalException: + tb = traceback.format_exc() + hint = 'Issue `/start` if you think it is safe to restart.' + self.freqtrade.rpc.send_msg({ + 'type': RPCMessageType.STATUS_NOTIFICATION, + 'status': f'OperationalException:\n```\n{tb}```{hint}' + }) + logger.exception('OperationalException. Stopping trader ...') + self.freqtrade.state = State.STOPPED + # TODO: The return value of _process() is not used apart tests + # and should (could) be eliminated later. See PR #1689. +# state_changed = True + return state_changed + + def _reconfigure(self): + """ + Cleans up current freqtradebot instance, reloads the configuration and + replaces it with the new instance + """ + # Tell systemd that we initiated reconfiguration + if self._sd_notify: + logger.debug("sd_notify: RELOADING=1") + self._sd_notify.notify("RELOADING=1") + + # Clean up current freqtrade modules + self.freqtrade.cleanup() + + # Load and validate config and create new instance of the bot + self._init(True) + + self.freqtrade.rpc.send_msg({ + 'type': RPCMessageType.STATUS_NOTIFICATION, + 'status': 'config reloaded' + }) + + # Tell systemd that we completed reconfiguration + if self._sd_notify: + logger.debug("sd_notify: READY=1") + self._sd_notify.notify("READY=1") + + def exit(self): + # Tell systemd that we are exiting now + if self._sd_notify: + logger.debug("sd_notify: STOPPING=1") + self._sd_notify.notify("STOPPING=1") + + if self.freqtrade: + self.freqtrade.rpc.send_msg({ + 'type': RPCMessageType.STATUS_NOTIFICATION, + 'status': 'process died' + }) + self.freqtrade.cleanup() diff --git a/mkdocs.yml b/mkdocs.yml index 9a6fec851..ecac265c1 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -3,11 +3,12 @@ nav: - About: index.md - Installation: installation.md - Configuration: configuration.md - - Start the bot: bot-usage.md - - Stoploss: stoploss.md - Custom Strategy: bot-optimization.md - - Telegram: telegram-usage.md - - Web Hook: webhook-config.md + - Stoploss: stoploss.md + - Start the bot: bot-usage.md + - Control the bot: + - Telegram: telegram-usage.md + - Web Hook: webhook-config.md - Backtesting: backtesting.md - Hyperopt: hyperopt.md - Edge positioning: edge.md diff --git a/requirements-dev.txt b/requirements-dev.txt index 69082587a..c59923ed2 100644 --- a/requirements-dev.txt +++ b/requirements-dev.txt @@ -9,4 +9,4 @@ pytest-mock==1.10.3 pytest-asyncio==0.10.0 pytest-cov==2.6.1 coveralls==1.7.0 -mypy==0.670 +mypy==0.700 diff --git a/requirements.txt b/requirements.txt index 83d77b693..eb5d564fa 100644 --- a/requirements.txt +++ b/requirements.txt @@ -1,5 +1,5 @@ -ccxt==1.18.425 -SQLAlchemy==1.3.1 +ccxt==1.18.432 +SQLAlchemy==1.3.2 python-telegram-bot==11.1.0 arrow==0.13.1 cachetools==3.1.0