conflict with develop resolved

docs/bot-usage.md

@@ -79,7 +79,7 @@ prevent unintended disclosure of sensitive private data when you publish example
 of your configuration in the project issues or in the Internet.
 
 See more details on this technique with examples in the documentation page on
-[configuration](bot-configuration.md).
+[configuration](configuration.md).
 
 ### How to use **--strategy**?
 

docs/hyperopt.md

@@ -163,7 +163,7 @@ running at least several thousand evaluations.
 The `--spaces all` flag determines that all possible parameters should be optimized. Possibilities are listed below.
 
 !!! Warning
-When switching parameters or changing configuration options, the file `user_data/hyperopt_results.pickle` should be removed. It's used to be able to continue interrupted calculations, but does not detect changes to settings or the hyperopt file.
+    When switching parameters or changing configuration options, the file `user_data/hyperopt_results.pickle` should be removed. It's used to be able to continue interrupted calculations, but does not detect changes to settings or the hyperopt file.
 
 ### Execute Hyperopt with Different Ticker-Data Source
 

docs/sql_cheatsheet.md

@@ -1,5 +1,5 @@
 # SQL Helper
-This page constains some help if you want to edit your sqlite db.
+This page contains some help if you want to edit your sqlite db.
 
 ## Install sqlite3
 **Ubuntu/Debian installation**
@@ -66,11 +66,11 @@ SELECT * FROM trades;
 ## Fix trade still open after a manual sell on the exchange
 
 !!! Warning
-  Manually selling on the exchange should not be done by default, since the bot does not detect this and will try to sell anyway.
+  	Manually selling a pair on the exchange will not be detected by the bot and it will try to sell anyway. Whenever possible, forcesell <tradeid> should be used to accomplish the same thing.  
-  /foresell <tradeid> should accomplish the same thing.
+		It is strongly advised to backup your database file before making any manual changes.
 
 !!! Note
-  This should not be necessary after /forcesell, as forcesell orders are closed automatically by the bot on the next iteration.
+  	This should not be necessary after /forcesell, as forcesell orders are closed automatically by the bot on the next iteration.
 
 ```sql
 UPDATE trades

docs/telegram-usage.md

@@ -1,13 +1,13 @@
 # Telegram usage
 
-This page explains how to command your bot with Telegram.
-
 ## Prerequisite
+
 To control your bot with Telegram, you need first to
 [set up a Telegram bot](installation.md)
 and add your Telegram API keys into your config file.
 
 ## Telegram commands
+
 Per default, the Telegram bot shows predefined commands. Some commands
 are only available by sending them to the bot. The table below list the
 official commands. You can ask at any moment for help with `/help`.
@@ -65,16 +65,14 @@ Once all positions are sold, run `/stop` to completely stop the bot.
 
 For each open trade, the bot will send you the following message.
 
-> **Trade ID:** `123`
+> **Trade ID:** `123` `(since 1 days ago)`  
 > **Current Pair:** CVC/BTC  
 > **Open Since:** `1 days ago`  
 > **Amount:** `26.64180098`  
 > **Open Rate:** `0.00007489`  
-> **Close Rate:** `None`
 > **Current Rate:** `0.00007489`  
-> **Close Profit:** `None`
 > **Current Profit:** `12.95%`  
-> **Open Order:** `None`
+> **Stoploss:** `0.00007389 (-0.02%)`  
 
 ### /status table
 

docs/webhook-config.md

@@ -1,7 +1,5 @@
 # Webhook usage
 
-This page explains how to configure your bot to talk to webhooks.
-
 ## Configuration
 
 Enable webhooks by adding a webhook-section to your configuration file, and setting `webhook.enabled` to `true`.
@@ -39,32 +37,30 @@ Different payloads can be configured for different events. Not all fields are ne
 The fields in `webhook.webhookbuy` are filled when the bot executes a buy. Parameters are filled using string.format.
 Possible parameters are:
 
-* exchange
+* `exchange`
-* pair
+* `pair`
-* limit
+* `limit`
-* stake_amount
+* `stake_amount`
-* stake_amount_fiat
+* `stake_currency`
-* stake_currency
+* `fiat_currency`
-* fiat_currency
 
 ### Webhooksell
 
 The fields in `webhook.webhooksell` are filled when the bot sells a trade. Parameters are filled using string.format.
 Possible parameters are:
 
-* exchange
+* `exchange`
-* pair
+* `pair`
-* gain
+* `gain`
-* limit
+* `limit`
-* amount
+* `amount`
-* open_rate
+* `open_rate`
-* current_rate
+* `current_rate`
-* profit_amount
+* `profit_amount`
-* profit_percent
+* `profit_percent`
-* profit_fiat
+* `stake_currency`
-* stake_currency
+* `fiat_currency`
-* fiat_currency
+* `sell_reason`
-* sell_reason
 
 ### Webhookstatus
 

freqtrade/edge/__init__.py

@@ -203,6 +203,22 @@ class Edge():
 
         return self._final_pairs
 
+    def accepted_pairs(self) -> list:
+        """
+        return a list of accepted pairs along with their winrate, expectancy and stoploss
+        """
+        final = []
+        for pair, info in self._cached_pairs.items():
+            if info.expectancy > float(self.edge_config.get('minimum_expectancy', 0.2)) and \
+                 info.winrate > float(self.edge_config.get('minimum_winrate', 0.60)):
+                final.append({
+                    'Pair': pair,
+                    'Winrate': info.winrate,
+                    'Expectancy': info.expectancy,
+                    'Stoploss': info.stoploss,
+                })
+        return final
+
     def _fill_calculable_fields(self, result: DataFrame) -> DataFrame:
         """
         The result frame contains a number of columns that are calculable

freqtrade/freqtradebot.py

@@ -4,14 +4,12 @@ Freqtrade is the main module of this bot. It contains the class Freqtrade()
 
 import copy
 import logging
-import time
 import traceback
 from datetime import datetime
-from typing import Any, Callable, Dict, List, Optional, Tuple
+from typing import Any, Dict, List, Optional, Tuple
 
 import arrow
 from requests.exceptions import RequestException
-import sdnotify
 
 from freqtrade import (DependencyException, OperationalException, InvalidOrderException,
                        TemporaryError, __version__, constants, persistence)
@@ -42,20 +40,14 @@ class FreqtradeBot(object):
         to get the config dict.
         """
 
-        logger.info(
+        logger.info('Starting freqtrade %s', __version__)
-            'Starting freqtrade %s',
-            __version__,
-        )
 
-        # Init bot states
+        # Init bot state
         self.state = State.STOPPED
 
         # Init objects
         self.config = config
 
-        self._sd_notify = sdnotify.SystemdNotifier() if \
-            self.config.get('internals', {}).get('sd_notify', False) else None
-
         self.strategy: IStrategy = StrategyResolver(self.config).strategy
 
         self.rpc: RPCManager = RPCManager(self)
@@ -79,29 +71,12 @@ class FreqtradeBot(object):
             self.config.get('edge', {}).get('enabled', False) else None
 
         self.active_pair_whitelist: List[str] = self.config['exchange']['pair_whitelist']
-        self._init_modules()
-
-        # Tell the systemd that we completed initialization phase
-        if self._sd_notify:
-            logger.debug("sd_notify: READY=1")
-            self._sd_notify.notify("READY=1")
-
-    def _init_modules(self) -> None:
-        """
-        Initializes all modules and updates the config
-        :return: None
-        """
-        # Initialize all modules
 
         persistence.init(self.config)
 
-        # Set initial application state
+        # Set initial bot state from config
         initial_state = self.config.get('initial_state')
-
+        self.state = State[initial_state.upper()] if initial_state else State.STOPPED
-        if initial_state:
-            self.state = State[initial_state.upper()]
-        else:
-            self.state = State.STOPPED
 
     def cleanup(self) -> None:
         """
@@ -113,130 +88,50 @@ class FreqtradeBot(object):
         self.rpc.cleanup()
         persistence.cleanup()
 
-    def stopping(self) -> None:
+    def process(self) -> bool:
-        # Tell systemd that we are exiting now
-        if self._sd_notify:
-            logger.debug("sd_notify: STOPPING=1")
-            self._sd_notify.notify("STOPPING=1")
-
-    def reconfigure(self) -> None:
-        # Tell systemd that we initiated reconfiguring
-        if self._sd_notify:
-            logger.debug("sd_notify: RELOADING=1")
-            self._sd_notify.notify("RELOADING=1")
-
-    def worker(self, old_state: State = None) -> State:
-        """
-        Trading routine that must be run at each loop
-        :param old_state: the previous service state from the previous call
-        :return: current service state
-        """
-        # Log state transition
-        state = self.state
-        if state != old_state:
-            self.rpc.send_msg({
-                'type': RPCMessageType.STATUS_NOTIFICATION,
-                'status': f'{state.name.lower()}'
-            })
-            logger.info('Changing state to: %s', state.name)
-            if state == State.RUNNING:
-                self.rpc.startup_messages(self.config, self.pairlists)
-
-        throttle_secs = self.config.get('internals', {}).get(
-            'process_throttle_secs',
-            constants.PROCESS_THROTTLE_SECS
-        )
-
-        if state == State.STOPPED:
-            # Ping systemd watchdog before sleeping in the stopped state
-            if self._sd_notify:
-                logger.debug("sd_notify: WATCHDOG=1\\nSTATUS=State: STOPPED.")
-                self._sd_notify.notify("WATCHDOG=1\nSTATUS=State: STOPPED.")
-
-            time.sleep(throttle_secs)
-
-        elif state == State.RUNNING:
-            # Ping systemd watchdog before throttling
-            if self._sd_notify:
-                logger.debug("sd_notify: WATCHDOG=1\\nSTATUS=State: RUNNING.")
-                self._sd_notify.notify("WATCHDOG=1\nSTATUS=State: RUNNING.")
-
-            self._throttle(func=self._process, min_secs=throttle_secs)
-
-        return state
-
-    def _throttle(self, func: Callable[..., Any], min_secs: float, *args, **kwargs) -> Any:
-        """
-        Throttles the given callable that it
-        takes at least `min_secs` to finish execution.
-        :param func: Any callable
-        :param min_secs: minimum execution time in seconds
-        :return: Any
-        """
-        start = time.time()
-        result = func(*args, **kwargs)
-        end = time.time()
-        duration = max(min_secs - (end - start), 0.0)
-        logger.debug('Throttling %s for %.2f seconds', func.__name__, duration)
-        time.sleep(duration)
-        return result
-
-    def _process(self) -> bool:
         """
         Queries the persistence layer for open trades and handles them,
         otherwise a new trade is created.
         :return: True if one or more trades has been created or closed, False otherwise
         """
         state_changed = False
-        try:
-            # Check whether markets have to be reloaded
-            self.exchange._reload_markets()
 
-            # Refresh whitelist
+        # Check whether markets have to be reloaded
-            self.pairlists.refresh_pairlist()
+        self.exchange._reload_markets()
-            self.active_pair_whitelist = self.pairlists.whitelist
 
-            # Calculating Edge positioning
+        # Refresh whitelist
-            if self.edge:
+        self.pairlists.refresh_pairlist()
-                self.edge.calculate()
+        self.active_pair_whitelist = self.pairlists.whitelist
-                self.active_pair_whitelist = self.edge.adjust(self.active_pair_whitelist)
 
-            # Query trades from persistence layer
+        # Calculating Edge positioning
-            trades = Trade.get_open_trades()
+        if self.edge:
+            self.edge.calculate()
+            self.active_pair_whitelist = self.edge.adjust(self.active_pair_whitelist)
 
-            # Extend active-pair whitelist with pairs from open trades
+        # Query trades from persistence layer
-            # It ensures that tickers are downloaded for open trades
+        trades = Trade.get_open_trades()
-            self._extend_whitelist_with_trades(self.active_pair_whitelist, trades)
 
-            # Refreshing candles
+        # Extend active-pair whitelist with pairs from open trades
-            self.dataprovider.refresh(self._create_pair_whitelist(self.active_pair_whitelist),
+        # It ensures that tickers are downloaded for open trades
-                                      self.strategy.informative_pairs())
+        self._extend_whitelist_with_trades(self.active_pair_whitelist, trades)
 
-            # First process current opened trades
+        # Refreshing candles
-            for trade in trades:
+        self.dataprovider.refresh(self._create_pair_whitelist(self.active_pair_whitelist),
-                state_changed |= self.process_maybe_execute_sell(trade)
+                                  self.strategy.informative_pairs())
 
-            # Then looking for buy opportunities
+        # First process current opened trades
-            if len(trades) < self.config['max_open_trades']:
+        for trade in trades:
-                state_changed = self.process_maybe_execute_buy()
+            state_changed |= self.process_maybe_execute_sell(trade)
 
-            if 'unfilledtimeout' in self.config:
+        # Then looking for buy opportunities
-                # Check and handle any timed out open orders
+        if len(trades) < self.config['max_open_trades']:
-                self.check_handle_timedout()
+            state_changed = self.process_maybe_execute_buy()
-                Trade.session.flush()
+
+        if 'unfilledtimeout' in self.config:
+            # Check and handle any timed out open orders
+            self.check_handle_timedout()
+            Trade.session.flush()
 
-        except TemporaryError as error:
-            logger.warning(f"Error: {error}, retrying in {constants.RETRY_TIMEOUT} seconds...")
-            time.sleep(constants.RETRY_TIMEOUT)
-        except OperationalException:
-            tb = traceback.format_exc()
-            hint = 'Issue `/start` if you think it is safe to restart.'
-            self.rpc.send_msg({
-                'type': RPCMessageType.STATUS_NOTIFICATION,
-                'status': f'OperationalException:\n```\n{tb}```{hint}'
-            })
-            logger.exception('OperationalException. Stopping trader ...')
-            self.state = State.STOPPED
         return state_changed
 
     def _extend_whitelist_with_trades(self, whitelist: List[str], trades: List[Any]):
@@ -356,6 +251,10 @@ class FreqtradeBot(object):
         interval = self.strategy.ticker_interval
         whitelist = copy.deepcopy(self.active_pair_whitelist)
 
+        if not whitelist:
+            logger.warning("Whitelist is empty.")
+            return False
+
         # Remove currently opened and latest pairs from whitelist
         for trade in Trade.get_open_trades():
             if trade.pair in whitelist:
@@ -363,7 +262,8 @@ class FreqtradeBot(object):
                 logger.debug('Ignoring %s in pair whitelist', trade.pair)
 
         if not whitelist:
-            raise DependencyException('No currency pairs in whitelist')
+            logger.info("No currency pair in whitelist, but checking to sell open trades.")
+            return False
 
         # running get_signal on historical data fetched
         for _pair in whitelist:

freqtrade/main.py

@@ -10,10 +10,9 @@ from typing import List
 
 from freqtrade import OperationalException
 from freqtrade.arguments import Arguments
-from freqtrade.configuration import Configuration, set_loggers
+from freqtrade.configuration import set_loggers
-from freqtrade.freqtradebot import FreqtradeBot
+from freqtrade.worker import Worker
-from freqtrade.state import State
+
-from freqtrade.rpc import RPCMessageType
 
 logger = logging.getLogger('freqtrade')
 
@@ -27,7 +26,7 @@ def main(sysargv: List[str]) -> None:
         sysargv,
         'Free, open source crypto trading bot'
     )
-    args = arguments.get_parsed_arg()
+    args: Namespace = arguments.get_parsed_arg()
 
     # A subcommand has been issued.
     # Means if Backtesting or Hyperopt have been called we exit the bot
@@ -35,20 +34,12 @@ def main(sysargv: List[str]) -> None:
         args.func(args)
         return
 
-    freqtrade = None
+    worker = None
     return_code = 1
     try:
-        # Load and validate configuration
+        # Load and run worker
-        config = Configuration(args, None).get_config()
+        worker = Worker(args)
-
+        worker.run()
-        # Init the bot
-        freqtrade = FreqtradeBot(config)
-
-        state = None
-        while True:
-            state = freqtrade.worker(old_state=state)
-            if state == State.RELOAD_CONF:
-                freqtrade = reconfigure(freqtrade, args)
 
     except KeyboardInterrupt:
         logger.info('SIGINT received, aborting ...')
@@ -59,34 +50,11 @@ def main(sysargv: List[str]) -> None:
     except BaseException:
         logger.exception('Fatal exception!')
     finally:
-        if freqtrade:
+        if worker:
-            freqtrade.stopping()
+            worker.exit()
-            freqtrade.rpc.send_msg({
-                'type': RPCMessageType.STATUS_NOTIFICATION,
-                'status': 'process died'
-            })
-            freqtrade.cleanup()
         sys.exit(return_code)
 
 
-def reconfigure(freqtrade: FreqtradeBot, args: Namespace) -> FreqtradeBot:
-    """
-    Cleans up current instance, reloads the configuration and returns the new instance
-    """
-    freqtrade.reconfigure()
-
-    # Clean up current modules
-    freqtrade.cleanup()
-
-    # Create new instance
-    freqtrade = FreqtradeBot(Configuration(args, None).get_config())
-    freqtrade.rpc.send_msg({
-        'type': RPCMessageType.STATUS_NOTIFICATION,
-        'status': 'config reloaded'
-    })
-    return freqtrade
-
-
 if __name__ == '__main__':
     set_loggers()
     main(sys.argv[1:])

freqtrade/optimize/backtesting.py

@@ -210,6 +210,32 @@ class Backtesting(object):
             logger.info('Dumping backtest results to %s', recordfilename)
             file_dump_json(recordfilename, records)
 
+    def _get_ticker_list(self, processed) -> Dict[str, DataFrame]:
+        """
+        Helper function to convert a processed tickerlist into a list for performance reasons.
+
+        Used by backtest() - so keep this optimized for performance.
+        """
+        headers = ['date', 'buy', 'open', 'close', 'sell', 'low', 'high']
+        ticker: Dict = {}
+        # Create ticker dict
+        for pair, pair_data in processed.items():
+            pair_data['buy'], pair_data['sell'] = 0, 0  # cleanup from previous run
+
+            ticker_data = self.advise_sell(
+                self.advise_buy(pair_data, {'pair': pair}), {'pair': pair})[headers].copy()
+
+            # to avoid using data from future, we buy/sell with signal from previous candle
+            ticker_data.loc[:, 'buy'] = ticker_data['buy'].shift(1)
+            ticker_data.loc[:, 'sell'] = ticker_data['sell'].shift(1)
+
+            ticker_data.drop(ticker_data.head(1).index, inplace=True)
+
+            # Convert from Pandas to list for performance reasons
+            # (Looping Pandas is slow.)
+            ticker[pair] = [x for x in ticker_data.itertuples()]
+        return ticker
+
     def _get_sell_trade_entry(
             self, pair: str, buy_row: DataFrame,
             partial_ticker: List, trade_count_lock: Dict, args: Dict) -> Optional[BacktestResult]:
@@ -304,7 +330,6 @@ class Backtesting(object):
             position_stacking: do we allow position stacking? (default: False)
         :return: DataFrame
         """
-        headers = ['date', 'buy', 'open', 'close', 'sell', 'low', 'high']
         processed = args['processed']
         max_open_trades = args.get('max_open_trades', 0)
         position_stacking = args.get('position_stacking', False)
@@ -312,54 +337,50 @@ class Backtesting(object):
         end_date = args['end_date']
         trades = []
         trade_count_lock: Dict = {}
-        ticker: Dict = {}
-        pairs = []
-        # Create ticker dict
-        for pair, pair_data in processed.items():
-            pair_data['buy'], pair_data['sell'] = 0, 0  # cleanup from previous run
 
-            ticker_data = self.advise_sell(
+        # Dict of ticker-lists for performance (looping lists is a lot faster than dataframes)
-                self.advise_buy(pair_data, {'pair': pair}), {'pair': pair})[headers].copy()
+        ticker: Dict = self._get_ticker_list(processed)
-
-            # to avoid using data from future, we buy/sell with signal from previous candle
-            ticker_data.loc[:, 'buy'] = ticker_data['buy'].shift(1)
-            ticker_data.loc[:, 'sell'] = ticker_data['sell'].shift(1)
-
-            ticker_data.drop(ticker_data.head(1).index, inplace=True)
-
-            # Convert from Pandas to list for performance reasons
-            # (Looping Pandas is slow.)
-            ticker[pair] = [x for x in ticker_data.itertuples()]
-            pairs.append(pair)
 
         lock_pair_until: Dict = {}
+        # Indexes per pair, so some pairs are allowed to have a missing start.
+        indexes: Dict = {}
         tmp = start_date + timedelta(minutes=self.ticker_interval_mins)
-        index = 0
+
-        # Loop timerange and test per pair
+        # Loop timerange and get candle for each pair at that point in time
         while tmp < end_date:
-            # print(f"time: {tmp}")
+
             for i, pair in enumerate(ticker):
+                if pair not in indexes:
+                    indexes[pair] = 0
+
                 try:
-                    row = ticker[pair][index]
+                    row = ticker[pair][indexes[pair]]
                 except IndexError:
-                    # missing Data for one pair ...
+                    # missing Data for one pair at the end.
                     # Warnings for this are shown by `validate_backtest_data`
                     continue
 
+                # Waits until the time-counter reaches the start of the data for this pair.
+                if row.date > tmp.datetime:
+                    continue
+
+                indexes[pair] += 1
+
                 if row.buy == 0 or row.sell == 1:
                     continue  # skip rows where no buy signal or that would immediately sell off
 
-                if not position_stacking:
+                if (not position_stacking and pair in lock_pair_until
-                    if pair in lock_pair_until and row.date <= lock_pair_until[pair]:
+                        and row.date <= lock_pair_until[pair]):
-                        continue
+                    # without positionstacking, we can only have one open trade per pair.
+                    continue
+
                 if max_open_trades > 0:
                     # Check if max_open_trades has already been reached for the given date
                     if not trade_count_lock.get(row.date, 0) < max_open_trades:
                         continue
-
                     trade_count_lock[row.date] = trade_count_lock.get(row.date, 0) + 1
 
-                trade_entry = self._get_sell_trade_entry(pair, row, ticker[pair][index + 1:],
+                trade_entry = self._get_sell_trade_entry(pair, row, ticker[pair][indexes[pair]:],
                                                          trade_count_lock, args)
 
                 if trade_entry:
@@ -367,11 +388,10 @@ class Backtesting(object):
                     trades.append(trade_entry)
                 else:
                     # Set lock_pair_until to end of testing period if trade could not be closed
-                    # This happens only if the buy-signal was with the last candle
+                    lock_pair_until[pair] = end_date.datetime
-                    lock_pair_until[pair] = end_date
 
+            # Move time one configured time_interval ahead.
             tmp += timedelta(minutes=self.ticker_interval_mins)
-            index += 1
         return DataFrame.from_records(trades, columns=BacktestResult._fields)
 
     def start(self) -> None:

freqtrade/rpc/rpc.py

@@ -103,16 +103,23 @@ class RPC(object):
                 results.append(dict(
                     trade_id=trade.id,
                     pair=trade.pair,
+                    base_currency=self._freqtrade.config['stake_currency'],
                     date=arrow.get(trade.open_date),
                     open_rate=trade.open_rate,
                     close_rate=trade.close_rate,
                     current_rate=current_rate,
                     amount=round(trade.amount, 8),
+                    stake_amount=round(trade.stake_amount, 8),
                     close_profit=fmt_close_profit,
                     current_profit=round(current_profit * 100, 2),
                     stop_loss=trade.stop_loss,
+                    stop_loss_pct=(trade.stop_loss_pct * 100)
+                    if trade.stop_loss_pct else None,
+                    initial_stop_loss=trade.initial_stop_loss,
+                    initial_stop_loss_pct=(trade.initial_stop_loss_pct * 100)
+                    if trade.initial_stop_loss_pct else None,
                     open_order='({} {} rem={:.8f})'.format(
-                      order['type'], order['side'], order['remaining']
+                        order['type'], order['side'], order['remaining']
                     ) if order else None,
                 ))
             return results
@@ -481,13 +488,4 @@ class RPC(object):
         """ Returns information related to Edge """
         if not self._freqtrade.edge:
             raise RPCException(f'Edge is not enabled.')
-
+        return self._freqtrade.edge.accepted_pairs()
-        return [
-            {
-                'Pair': k,
-                'Winrate': v.winrate,
-                'Expectancy': v.expectancy,
-                'Stoploss': v.stoploss,
-            }
-            for k, v in self._freqtrade.edge._cached_pairs.items()
-        ]

freqtrade/rpc/telegram.py

@@ -197,16 +197,25 @@ class Telegram(RPC):
             messages = []
             for r in results:
                 lines = [
-                    "*Trade ID:* `{trade_id}` (since `{date}`)",
+                    "*Trade ID:* `{trade_id}` `(since {date})`",
                     "*Current Pair:* {pair}",
-                    "*Amount:* `{amount}`",
+                    "*Amount:* `{amount} ({stake_amount} {base_currency})`",
                     "*Open Rate:* `{open_rate:.8f}`",
                     "*Close Rate:* `{close_rate}`" if r['close_rate'] else "",
                     "*Current Rate:* `{current_rate:.8f}`",
                     "*Close Profit:* `{close_profit}`" if r['close_profit'] else "",
                     "*Current Profit:* `{current_profit:.2f}%`",
-                    "*Stoploss:* `{stop_loss:.8f}`",
+
-                    "*Open Order:* `{open_order}`" if r['open_order'] else "",
+                    # Adding initial stoploss only if it is different from stoploss
+                    "*Initial Stoploss:* `{initial_stop_loss:.8f}` " +
+                    ("`({initial_stop_loss_pct:.2f}%)`" if r['initial_stop_loss_pct'] else "")
+                    if r['stop_loss'] != r['initial_stop_loss'] else "",
+
+                    # Adding stoploss and stoploss percentage only if it is not None
+                    "*Stoploss:* `{stop_loss:.8f}` " +
+                    ("`({stop_loss_pct:.2f}%)`" if r['stop_loss_pct'] else ""),
+
+                    "*Open Order:* `{open_order}`" if r['open_order'] else ""
                 ]
                 messages.append("\n".join(filter(None, lines)).format(**r))
 
@@ -502,7 +511,6 @@ class Telegram(RPC):
         """
         try:
             edge_pairs = self._rpc_edge()
-            print(edge_pairs)
             edge_pairs_tab = tabulate(edge_pairs, headers='keys', tablefmt='simple')
             message = f'<b>Edge only validated following pairs:</b>\n<pre>{edge_pairs_tab}</pre>'
             self._send_msg(message, bot=bot, parse_mode=ParseMode.HTML)

freqtrade/tests/conftest.py

@@ -16,6 +16,7 @@ from freqtrade.edge import Edge, PairInfo
 from freqtrade.exchange import Exchange
 from freqtrade.freqtradebot import FreqtradeBot
 from freqtrade.resolvers import ExchangeResolver
+from freqtrade.worker import Worker
 
 logging.getLogger('').setLevel(logging.INFO)
 
@@ -87,7 +88,7 @@ def get_patched_edge(mocker, config) -> Edge:
 # Functions for recurrent object patching
 
 
-def get_patched_freqtradebot(mocker, config) -> FreqtradeBot:
+def patch_freqtradebot(mocker, config) -> None:
     """
     This function patch _init_modules() to not call dependencies
     :param mocker: a Mocker object to apply patches
@@ -100,9 +101,17 @@ def get_patched_freqtradebot(mocker, config) -> FreqtradeBot:
     mocker.patch('freqtrade.freqtradebot.RPCManager._init', MagicMock())
     mocker.patch('freqtrade.freqtradebot.RPCManager.send_msg', MagicMock())
 
+
+def get_patched_freqtradebot(mocker, config) -> FreqtradeBot:
+    patch_freqtradebot(mocker, config)
     return FreqtradeBot(config)
 
 
+def get_patched_worker(mocker, config) -> Worker:
+    patch_freqtradebot(mocker, config)
+    return Worker(args=None, config=config)
+
+
 @pytest.fixture(autouse=True)
 def patch_coinmarketcap(mocker) -> None:
     """

freqtrade/tests/edge/test_edge.py

@@ -122,8 +122,8 @@ def test_edge_results(edge_conf, mocker, caplog, data) -> None:
     for c, trade in enumerate(data.trades):
         res = results.iloc[c]
         assert res.exit_type == trade.sell_reason
-        assert res.open_time == _get_frame_time_from_offset(trade.open_tick)
+        assert arrow.get(res.open_time) == _get_frame_time_from_offset(trade.open_tick)
-        assert res.close_time == _get_frame_time_from_offset(trade.close_tick)
+        assert arrow.get(res.close_time) == _get_frame_time_from_offset(trade.close_tick)
 
 
 def test_adjust(mocker, edge_conf):

freqtrade/tests/optimize/__init__.py

@@ -33,7 +33,7 @@ class BTContainer(NamedTuple):
 
 def _get_frame_time_from_offset(offset):
     return ticker_start_time.shift(minutes=(offset * TICKER_INTERVAL_MINUTES[tests_ticker_interval])
-                                   ).datetime.replace(tzinfo=None)
+                                   ).datetime
 
 
 def _build_backtest_dataframe(ticker_with_signals):

freqtrade/tests/optimize/test_backtesting.py

@@ -685,25 +685,32 @@ def test_backtest_alternate_buy_sell(default_conf, fee, mocker):
     assert len(results.loc[results.open_at_end]) == 0
 
 
-def test_backtest_multi_pair(default_conf, fee, mocker):
+@pytest.mark.parametrize("pair", ['ADA/BTC', 'LTC/BTC'])
+@pytest.mark.parametrize("tres", [0, 20, 30])
+def test_backtest_multi_pair(default_conf, fee, mocker, tres, pair):
 
     def _trend_alternate_hold(dataframe=None, metadata=None):
         """
-        Buy every 8th candle - sell every other 8th -2 (hold on to pairs a bit)
+        Buy every xth candle - sell every other xth -2 (hold on to pairs a bit)
         """
-        multi = 8
+        if metadata['pair'] in('ETH/BTC', 'LTC/BTC'):
+            multi = 20
+        else:
+            multi = 18
         dataframe['buy'] = np.where(dataframe.index % multi == 0, 1, 0)
         dataframe['sell'] = np.where((dataframe.index + multi - 2) % multi == 0, 1, 0)
-        if metadata['pair'] in('ETH/BTC', 'LTC/BTC'):
-            dataframe['buy'] = dataframe['buy'].shift(-4)
-            dataframe['sell'] = dataframe['sell'].shift(-4)
         return dataframe
 
     mocker.patch('freqtrade.exchange.Exchange.get_fee', fee)
     patch_exchange(mocker)
+
     pairs = ['ADA/BTC', 'DASH/BTC', 'ETH/BTC', 'LTC/BTC', 'NXT/BTC']
     data = history.load_data(datadir=None, ticker_interval='5m', pairs=pairs)
+    # Only use 500 lines to increase performance
     data = trim_dictlist(data, -500)
+
+    # Remove data for one pair from the beginning of the data
+    data[pair] = data[pair][tres:]
     # We need to enable sell-signal - otherwise it sells on ROI!!
     default_conf['experimental'] = {"use_sell_signal": True}
     default_conf['ticker_interval'] = '5m'

freqtrade/tests/rpc/test_rpc.py

@@ -51,14 +51,19 @@ def test_rpc_trade_status(default_conf, ticker, fee, markets, mocker) -> None:
     assert {
         'trade_id': 1,
         'pair': 'ETH/BTC',
+        'base_currency': 'BTC',
         'date': ANY,
         'open_rate': 1.099e-05,
         'close_rate': None,
         'current_rate': 1.098e-05,
         'amount': 90.99181074,
+        'stake_amount': 0.001,
         'close_profit': None,
         'current_profit': -0.59,
         'stop_loss': 0.0,
+        'initial_stop_loss': 0.0,
+        'initial_stop_loss_pct': None,
+        'stop_loss_pct': None,
         'open_order': '(limit buy rem=0.00000000)'
     } == results[0]
 
@@ -72,14 +77,19 @@ def test_rpc_trade_status(default_conf, ticker, fee, markets, mocker) -> None:
     assert {
         'trade_id': 1,
         'pair': 'ETH/BTC',
+        'base_currency': 'BTC',
         'date': ANY,
         'open_rate': 1.099e-05,
         'close_rate': None,
         'current_rate': ANY,
         'amount': 90.99181074,
+        'stake_amount': 0.001,
         'close_profit': None,
         'current_profit': ANY,
         'stop_loss': 0.0,
+        'initial_stop_loss': 0.0,
+        'initial_stop_loss_pct': None,
+        'stop_loss_pct': None,
         'open_order': '(limit buy rem=0.00000000)'
     } == results[0]
 

freqtrade/tests/rpc/test_rpc_telegram.py

@@ -189,14 +189,19 @@ def test_status(default_conf, update, mocker, fee, ticker, markets) -> None:
         _rpc_trade_status=MagicMock(return_value=[{
             'trade_id': 1,
             'pair': 'ETH/BTC',
+            'base_currency': 'BTC',
             'date': arrow.utcnow(),
             'open_rate': 1.099e-05,
             'close_rate': None,
             'current_rate': 1.098e-05,
             'amount': 90.99181074,
+            'stake_amount': 90.99181074,
             'close_profit': None,
             'current_profit': -0.59,
+            'initial_stop_loss': 1.098e-05,
             'stop_loss': 1.099e-05,
+            'initial_stop_loss_pct': -0.05,
+            'stop_loss_pct': -0.01,
             'open_order': '(limit buy rem=0.00000000)'
         }]),
         _status_table=status_table,

freqtrade/tests/test_freqtradebot.py

@@ -21,12 +21,13 @@ from freqtrade.state import State
 from freqtrade.strategy.interface import SellCheckTuple, SellType
 from freqtrade.tests.conftest import (log_has, log_has_re, patch_edge,
                                       patch_exchange, patch_wallet)
+from freqtrade.worker import Worker
 
 
 # Functions for recurrent object patching
-def get_patched_freqtradebot(mocker, config) -> FreqtradeBot:
+def patch_freqtradebot(mocker, config) -> None:
     """
-    This function patch _init_modules() to not call dependencies
+    This function patches _init_modules() to not call dependencies
     :param mocker: a Mocker object to apply patches
     :param config: Config to pass to the bot
     :return: None
@@ -35,9 +36,29 @@ def get_patched_freqtradebot(mocker, config) -> FreqtradeBot:
     mocker.patch('freqtrade.freqtradebot.persistence.init', MagicMock())
     patch_exchange(mocker)
 
+
+def get_patched_freqtradebot(mocker, config) -> FreqtradeBot:
+    """
+    This function patches _init_modules() to not call dependencies
+    :param mocker: a Mocker object to apply patches
+    :param config: Config to pass to the bot
+    :return: FreqtradeBot
+    """
+    patch_freqtradebot(mocker, config)
     return FreqtradeBot(config)
 
 
+def get_patched_worker(mocker, config) -> Worker:
+    """
+    This function patches _init_modules() to not call dependencies
+    :param mocker: a Mocker object to apply patches
+    :param config: Config to pass to the bot
+    :return: Worker
+    """
+    patch_freqtradebot(mocker, config)
+    return Worker(args=None, config=config)
+
+
 def patch_get_signal(freqtrade: FreqtradeBot, value=(True, False)) -> None:
     """
     :param mocker: mocker to patch IStrategy class
@@ -61,7 +82,7 @@ def patch_RPCManager(mocker) -> MagicMock:
 
 # Unit tests
 
-def test_freqtradebot(mocker, default_conf, markets) -> None:
+def test_freqtradebot_state(mocker, default_conf, markets) -> None:
     mocker.patch('freqtrade.exchange.Exchange.markets', PropertyMock(return_value=markets))
     freqtrade = get_patched_freqtradebot(mocker, default_conf)
     assert freqtrade.state is State.RUNNING
@@ -71,6 +92,16 @@ def test_freqtradebot(mocker, default_conf, markets) -> None:
     assert freqtrade.state is State.STOPPED
 
 
+def test_worker_state(mocker, default_conf, markets) -> None:
+    mocker.patch('freqtrade.exchange.Exchange.markets', PropertyMock(return_value=markets))
+    worker = get_patched_worker(mocker, default_conf)
+    assert worker.state is State.RUNNING
+
+    default_conf.pop('initial_state')
+    worker = Worker(args=None, config=default_conf)
+    assert worker.state is State.STOPPED
+
+
 def test_cleanup(mocker, default_conf, caplog) -> None:
     mock_cleanup = MagicMock()
     mocker.patch('freqtrade.persistence.cleanup', mock_cleanup)
@@ -82,28 +113,28 @@ def test_cleanup(mocker, default_conf, caplog) -> None:
 
 def test_worker_running(mocker, default_conf, caplog) -> None:
     mock_throttle = MagicMock()
-    mocker.patch('freqtrade.freqtradebot.FreqtradeBot._throttle', mock_throttle)
+    mocker.patch('freqtrade.worker.Worker._throttle', mock_throttle)
 
-    freqtrade = get_patched_freqtradebot(mocker, default_conf)
+    worker = get_patched_worker(mocker, default_conf)
 
-    state = freqtrade.worker(old_state=None)
+    state = worker._worker(old_state=None)
     assert state is State.RUNNING
     assert log_has('Changing state to: RUNNING', caplog.record_tuples)
     assert mock_throttle.call_count == 1
     # Check strategy is loaded, and received a dataprovider object
-    assert freqtrade.strategy
+    assert worker.freqtrade.strategy
-    assert freqtrade.strategy.dp
+    assert worker.freqtrade.strategy.dp
-    assert isinstance(freqtrade.strategy.dp, DataProvider)
+    assert isinstance(worker.freqtrade.strategy.dp, DataProvider)
 
 
 def test_worker_stopped(mocker, default_conf, caplog) -> None:
     mock_throttle = MagicMock()
-    mocker.patch('freqtrade.freqtradebot.FreqtradeBot._throttle', mock_throttle)
+    mocker.patch('freqtrade.worker.Worker._throttle', mock_throttle)
     mock_sleep = mocker.patch('time.sleep', return_value=None)
 
-    freqtrade = get_patched_freqtradebot(mocker, default_conf)
+    worker = get_patched_worker(mocker, default_conf)
-    freqtrade.state = State.STOPPED
+    worker.state = State.STOPPED
-    state = freqtrade.worker(old_state=State.RUNNING)
+    state = worker._worker(old_state=State.RUNNING)
     assert state is State.STOPPED
     assert log_has('Changing state to: STOPPED', caplog.record_tuples)
     assert mock_throttle.call_count == 0
@@ -115,17 +146,17 @@ def test_throttle(mocker, default_conf, caplog) -> None:
         return 42
 
     caplog.set_level(logging.DEBUG)
-    freqtrade = get_patched_freqtradebot(mocker, default_conf)
+    worker = get_patched_worker(mocker, default_conf)
 
     start = time.time()
-    result = freqtrade._throttle(throttled_func, min_secs=0.1)
+    result = worker._throttle(throttled_func, min_secs=0.1)
     end = time.time()
 
     assert result == 42
     assert end - start > 0.1
     assert log_has('Throttling throttled_func for 0.10 seconds', caplog.record_tuples)
 
-    result = freqtrade._throttle(throttled_func, min_secs=-1)
+    result = worker._throttle(throttled_func, min_secs=-1)
     assert result == 42
 
 
@@ -133,12 +164,12 @@ def test_throttle_with_assets(mocker, default_conf) -> None:
     def throttled_func(nb_assets=-1):
         return nb_assets
 
-    freqtrade = get_patched_freqtradebot(mocker, default_conf)
+    worker = get_patched_worker(mocker, default_conf)
 
-    result = freqtrade._throttle(throttled_func, min_secs=0.1, nb_assets=666)
+    result = worker._throttle(throttled_func, min_secs=0.1, nb_assets=666)
     assert result == 666
 
-    result = freqtrade._throttle(throttled_func, min_secs=0.1)
+    result = worker._throttle(throttled_func, min_secs=0.1)
     assert result == -1
 
 
@@ -224,7 +255,7 @@ def test_edge_called_in_process(mocker, edge_conf) -> None:
     freqtrade = FreqtradeBot(edge_conf)
     freqtrade.pairlists._validate_whitelist = _refresh_whitelist
     patch_get_signal(freqtrade)
-    freqtrade._process()
+    freqtrade.process()
     assert freqtrade.active_pair_whitelist == ['NEO/BTC', 'LTC/BTC']
 
 
@@ -551,8 +582,7 @@ def test_create_trade_too_small_stake_amount(default_conf, ticker, limit_buy_ord
     freqtrade = FreqtradeBot(default_conf)
     patch_get_signal(freqtrade)
 
-    result = freqtrade.create_trade()
+    assert not freqtrade.create_trade()
-    assert result is False
 
 
 def test_create_trade_limit_reached(default_conf, ticker, limit_buy_order,
@@ -573,11 +603,12 @@ def test_create_trade_limit_reached(default_conf, ticker, limit_buy_order,
     freqtrade = FreqtradeBot(default_conf)
     patch_get_signal(freqtrade)
 
-    assert freqtrade.create_trade() is False
+    assert not freqtrade.create_trade()
     assert freqtrade._get_trade_stake_amount('ETH/BTC') is None
 
 
-def test_create_trade_no_pairs(default_conf, ticker, limit_buy_order, fee, markets, mocker) -> None:
+def test_create_trade_no_pairs_let(default_conf, ticker, limit_buy_order, fee,
+                                   markets, mocker, caplog) -> None:
     patch_RPCManager(mocker)
     patch_exchange(mocker)
     mocker.patch.multiple(
@@ -589,18 +620,17 @@ def test_create_trade_no_pairs(default_conf, ticker, limit_buy_order, fee, marke
     )
 
     default_conf['exchange']['pair_whitelist'] = ["ETH/BTC"]
-    default_conf['exchange']['pair_blacklist'] = ["ETH/BTC"]
     freqtrade = FreqtradeBot(default_conf)
     patch_get_signal(freqtrade)
 
-    freqtrade.create_trade()
+    assert freqtrade.create_trade()
-
+    assert not freqtrade.create_trade()
-    with pytest.raises(DependencyException, match=r'.*No currency pairs in whitelist.*'):
+    assert log_has("No currency pair in whitelist, but checking to sell open trades.",
-        freqtrade.create_trade()
+                   caplog.record_tuples)
 
 
-def test_create_trade_no_pairs_after_blacklist(default_conf, ticker,
+def test_create_trade_no_pairs_in_whitelist(default_conf, ticker, limit_buy_order, fee,
-                                               limit_buy_order, fee, markets, mocker) -> None:
+                                            markets, mocker, caplog) -> None:
     patch_RPCManager(mocker)
     patch_exchange(mocker)
     mocker.patch.multiple(
@@ -610,15 +640,12 @@ def test_create_trade_no_pairs_after_blacklist(default_conf, ticker,
         get_fee=fee,
         markets=PropertyMock(return_value=markets)
     )
-    default_conf['exchange']['pair_whitelist'] = ["ETH/BTC"]
+    default_conf['exchange']['pair_whitelist'] = []
-    default_conf['exchange']['pair_blacklist'] = ["ETH/BTC"]
     freqtrade = FreqtradeBot(default_conf)
     patch_get_signal(freqtrade)
 
-    freqtrade.create_trade()
+    assert not freqtrade.create_trade()
-
+    assert log_has("Whitelist is empty.", caplog.record_tuples)
-    with pytest.raises(DependencyException, match=r'.*No currency pairs in whitelist.*'):
-        freqtrade.create_trade()
 
 
 def test_create_trade_no_signal(default_conf, fee, mocker) -> None:
@@ -658,7 +685,7 @@ def test_process_trade_creation(default_conf, ticker, limit_buy_order,
     trades = Trade.query.filter(Trade.is_open.is_(True)).all()
     assert not trades
 
-    result = freqtrade._process()
+    result = freqtrade.process()
     assert result is True
 
     trades = Trade.query.filter(Trade.is_open.is_(True)).all()
@@ -689,10 +716,10 @@ def test_process_exchange_failures(default_conf, ticker, markets, mocker) -> Non
     )
     sleep_mock = mocker.patch('time.sleep', side_effect=lambda _: None)
 
-    freqtrade = FreqtradeBot(default_conf)
+    worker = Worker(args=None, config=default_conf)
-    patch_get_signal(freqtrade)
+    patch_get_signal(worker.freqtrade)
 
-    result = freqtrade._process()
+    result = worker._process()
     assert result is False
     assert sleep_mock.has_calls()
 
@@ -706,14 +733,14 @@ def test_process_operational_exception(default_conf, ticker, markets, mocker) ->
         markets=PropertyMock(return_value=markets),
         buy=MagicMock(side_effect=OperationalException)
     )
-    freqtrade = FreqtradeBot(default_conf)
+    worker = Worker(args=None, config=default_conf)
-    patch_get_signal(freqtrade)
+    patch_get_signal(worker.freqtrade)
 
-    assert freqtrade.state == State.RUNNING
+    assert worker.state == State.RUNNING
 
-    result = freqtrade._process()
+    result = worker._process()
     assert result is False
-    assert freqtrade.state == State.STOPPED
+    assert worker.state == State.STOPPED
     assert 'OperationalException' in msg_mock.call_args_list[-1][0][0]['status']
 
 
@@ -734,18 +761,18 @@ def test_process_trade_handling(
 
     trades = Trade.query.filter(Trade.is_open.is_(True)).all()
     assert not trades
-    result = freqtrade._process()
+    result = freqtrade.process()
     assert result is True
     trades = Trade.query.filter(Trade.is_open.is_(True)).all()
     assert len(trades) == 1
 
-    result = freqtrade._process()
+    result = freqtrade.process()
     assert result is False
 
 
 def test_process_trade_no_whitelist_pair(
         default_conf, ticker, limit_buy_order, markets, fee, mocker) -> None:
-    """ Test _process with trade not in pair list """
+    """ Test process with trade not in pair list """
     patch_RPCManager(mocker)
     patch_exchange(mocker)
     mocker.patch.multiple(
@@ -782,7 +809,7 @@ def test_process_trade_no_whitelist_pair(
     ))
 
     assert pair not in freqtrade.active_pair_whitelist
-    result = freqtrade._process()
+    result = freqtrade.process()
     assert pair in freqtrade.active_pair_whitelist
     # Make sure each pair is only in the list once
     assert len(freqtrade.active_pair_whitelist) == len(set(freqtrade.active_pair_whitelist))
@@ -812,7 +839,7 @@ def test_process_informative_pairs_added(default_conf, ticker, markets, mocker)
     freqtrade.strategy.informative_pairs = inf_pairs
     # patch_get_signal(freqtrade)
 
-    freqtrade._process()
+    freqtrade.process()
     assert inf_pairs.call_count == 1
     assert refresh_mock.call_count == 1
     assert ("BTC/ETH", "1m") in refresh_mock.call_args[0][0]
@@ -3083,5 +3110,5 @@ def test_startup_messages(default_conf, mocker):
                                 'config': {'number_assets': 20}
                                 }
     mocker.patch('freqtrade.exchange.Exchange.exchange_has', MagicMock(return_value=True))
-    freqtrade = get_patched_freqtradebot(mocker, default_conf)
+    worker = get_patched_worker(mocker, default_conf)
-    assert freqtrade.state is State.RUNNING
+    assert worker.state is State.RUNNING

freqtrade/tests/test_main.py

@@ -7,8 +7,8 @@ import pytest
 
 from freqtrade import OperationalException
 from freqtrade.arguments import Arguments
-from freqtrade.freqtradebot import FreqtradeBot
+from freqtrade.worker import Worker
-from freqtrade.main import main, reconfigure
+from freqtrade.main import main
 from freqtrade.state import State
 from freqtrade.tests.conftest import log_has, patch_exchange
 
@@ -43,17 +43,14 @@ def test_main_start_hyperopt(mocker) -> None:
 
 def test_main_fatal_exception(mocker, default_conf, caplog) -> None:
     patch_exchange(mocker)
-    mocker.patch.multiple(
+    mocker.patch('freqtrade.freqtradebot.FreqtradeBot.cleanup', MagicMock())
-        'freqtrade.freqtradebot.FreqtradeBot',
+    mocker.patch('freqtrade.worker.Worker._worker', MagicMock(side_effect=Exception))
-        _init_modules=MagicMock(),
-        worker=MagicMock(side_effect=Exception),
-        cleanup=MagicMock(),
-    )
     mocker.patch(
         'freqtrade.configuration.Configuration._load_config_file',
         lambda *args, **kwargs: default_conf
     )
     mocker.patch('freqtrade.freqtradebot.RPCManager', MagicMock())
+    mocker.patch('freqtrade.freqtradebot.persistence.init', MagicMock())
 
     args = ['-c', 'config.json.example']
 
@@ -66,17 +63,14 @@ def test_main_fatal_exception(mocker, default_conf, caplog) -> None:
 
 def test_main_keyboard_interrupt(mocker, default_conf, caplog) -> None:
     patch_exchange(mocker)
-    mocker.patch.multiple(
+    mocker.patch('freqtrade.freqtradebot.FreqtradeBot.cleanup', MagicMock())
-        'freqtrade.freqtradebot.FreqtradeBot',
+    mocker.patch('freqtrade.worker.Worker._worker', MagicMock(side_effect=KeyboardInterrupt))
-        _init_modules=MagicMock(),
-        worker=MagicMock(side_effect=KeyboardInterrupt),
-        cleanup=MagicMock(),
-    )
     mocker.patch(
         'freqtrade.configuration.Configuration._load_config_file',
         lambda *args, **kwargs: default_conf
     )
     mocker.patch('freqtrade.freqtradebot.RPCManager', MagicMock())
+    mocker.patch('freqtrade.freqtradebot.persistence.init', MagicMock())
 
     args = ['-c', 'config.json.example']
 
@@ -89,17 +83,17 @@ def test_main_keyboard_interrupt(mocker, default_conf, caplog) -> None:
 
 def test_main_operational_exception(mocker, default_conf, caplog) -> None:
     patch_exchange(mocker)
-    mocker.patch.multiple(
+    mocker.patch('freqtrade.freqtradebot.FreqtradeBot.cleanup', MagicMock())
-        'freqtrade.freqtradebot.FreqtradeBot',
+    mocker.patch(
-        _init_modules=MagicMock(),
+        'freqtrade.worker.Worker._worker',
-        worker=MagicMock(side_effect=OperationalException('Oh snap!')),
+        MagicMock(side_effect=OperationalException('Oh snap!'))
-        cleanup=MagicMock(),
     )
     mocker.patch(
         'freqtrade.configuration.Configuration._load_config_file',
         lambda *args, **kwargs: default_conf
     )
     mocker.patch('freqtrade.freqtradebot.RPCManager', MagicMock())
+    mocker.patch('freqtrade.freqtradebot.persistence.init', MagicMock())
 
     args = ['-c', 'config.json.example']
 
@@ -112,21 +106,18 @@ def test_main_operational_exception(mocker, default_conf, caplog) -> None:
 
 def test_main_reload_conf(mocker, default_conf, caplog) -> None:
     patch_exchange(mocker)
-    mocker.patch.multiple(
+    mocker.patch('freqtrade.freqtradebot.FreqtradeBot.cleanup', MagicMock())
-        'freqtrade.freqtradebot.FreqtradeBot',
+    mocker.patch('freqtrade.worker.Worker._worker', MagicMock(return_value=State.RELOAD_CONF))
-        _init_modules=MagicMock(),
-        worker=MagicMock(return_value=State.RELOAD_CONF),
-        cleanup=MagicMock(),
-    )
     mocker.patch(
         'freqtrade.configuration.Configuration._load_config_file',
         lambda *args, **kwargs: default_conf
     )
     mocker.patch('freqtrade.freqtradebot.RPCManager', MagicMock())
+    mocker.patch('freqtrade.freqtradebot.persistence.init', MagicMock())
 
     # Raise exception as side effect to avoid endless loop
     reconfigure_mock = mocker.patch(
-        'freqtrade.main.reconfigure', MagicMock(side_effect=Exception)
+        'freqtrade.main.Worker._reconfigure', MagicMock(side_effect=Exception)
     )
 
     with pytest.raises(SystemExit):
@@ -138,19 +129,21 @@ def test_main_reload_conf(mocker, default_conf, caplog) -> None:
 
 def test_reconfigure(mocker, default_conf) -> None:
     patch_exchange(mocker)
-    mocker.patch.multiple(
+    mocker.patch('freqtrade.freqtradebot.FreqtradeBot.cleanup', MagicMock())
-        'freqtrade.freqtradebot.FreqtradeBot',
+    mocker.patch(
-        _init_modules=MagicMock(),
+        'freqtrade.worker.Worker._worker',
-        worker=MagicMock(side_effect=OperationalException('Oh snap!')),
+        MagicMock(side_effect=OperationalException('Oh snap!'))
-        cleanup=MagicMock(),
     )
     mocker.patch(
         'freqtrade.configuration.Configuration._load_config_file',
         lambda *args, **kwargs: default_conf
     )
     mocker.patch('freqtrade.freqtradebot.RPCManager', MagicMock())
+    mocker.patch('freqtrade.freqtradebot.persistence.init', MagicMock())
 
-    freqtrade = FreqtradeBot(default_conf)
+    args = Arguments(['-c', 'config.json.example'], '').get_parsed_arg()
+    worker = Worker(args=args, config=default_conf)
+    freqtrade = worker.freqtrade
 
     # Renew mock to return modified data
     conf = deepcopy(default_conf)
@@ -160,11 +153,10 @@ def test_reconfigure(mocker, default_conf) -> None:
         lambda *args, **kwargs: conf
     )
 
+    worker._config = conf
     # reconfigure should return a new instance
-    freqtrade2 = reconfigure(
+    worker._reconfigure()
-        freqtrade,
+    freqtrade2 = worker.freqtrade
-        Arguments(['-c', 'config.json.example'], '').get_parsed_arg()
-    )
 
     # Verify we have a new instance with the new config
     assert freqtrade is not freqtrade2

freqtrade/worker.py

@@ -0,0 +1,188 @@
+"""
+Main Freqtrade worker class.
+"""
+import logging
+import time
+import traceback
+from argparse import Namespace
+from typing import Any, Callable, Optional
+import sdnotify
+
+from freqtrade import (constants, OperationalException, TemporaryError,
+                       __version__)
+from freqtrade.configuration import Configuration
+from freqtrade.freqtradebot import FreqtradeBot
+from freqtrade.state import State
+from freqtrade.rpc import RPCMessageType
+
+
+logger = logging.getLogger(__name__)
+
+
+class Worker(object):
+    """
+    Freqtradebot worker class
+    """
+
+    def __init__(self, args: Namespace, config=None) -> None:
+        """
+        Init all variables and objects the bot needs to work
+        """
+        logger.info('Starting worker %s', __version__)
+
+        self._args = args
+        self._config = config
+        self._init(False)
+
+        # Tell systemd that we completed initialization phase
+        if self._sd_notify:
+            logger.debug("sd_notify: READY=1")
+            self._sd_notify.notify("READY=1")
+
+    def _init(self, reconfig: bool):
+        """
+        Also called from the _reconfigure() method (with reconfig=True).
+        """
+        if reconfig or self._config is None:
+            # Load configuration
+            self._config = Configuration(self._args, None).get_config()
+
+        # Init the instance of the bot
+        self.freqtrade = FreqtradeBot(self._config)
+
+        self._throttle_secs = self._config.get('internals', {}).get(
+            'process_throttle_secs',
+            constants.PROCESS_THROTTLE_SECS
+        )
+
+        self._sd_notify = sdnotify.SystemdNotifier() if \
+            self._config.get('internals', {}).get('sd_notify', False) else None
+
+    @property
+    def state(self) -> State:
+        return self.freqtrade.state
+
+    @state.setter
+    def state(self, value: State):
+        self.freqtrade.state = value
+
+    def run(self):
+        state = None
+        while True:
+            state = self._worker(old_state=state)
+            if state == State.RELOAD_CONF:
+                self.freqtrade = self._reconfigure()
+
+    def _worker(self, old_state: State, throttle_secs: Optional[float] = None) -> State:
+        """
+        Trading routine that must be run at each loop
+        :param old_state: the previous service state from the previous call
+        :return: current service state
+        """
+        state = self.freqtrade.state
+        if throttle_secs is None:
+            throttle_secs = self._throttle_secs
+
+        # Log state transition
+        if state != old_state:
+            self.freqtrade.rpc.send_msg({
+                'type': RPCMessageType.STATUS_NOTIFICATION,
+                'status': f'{state.name.lower()}'
+            })
+            logger.info('Changing state to: %s', state.name)
+            if state == State.RUNNING:
+                self.freqtrade.rpc.startup_messages(self._config, self.freqtrade.pairlists)
+
+        if state == State.STOPPED:
+            # Ping systemd watchdog before sleeping in the stopped state
+            if self._sd_notify:
+                logger.debug("sd_notify: WATCHDOG=1\\nSTATUS=State: STOPPED.")
+                self._sd_notify.notify("WATCHDOG=1\nSTATUS=State: STOPPED.")
+
+            time.sleep(throttle_secs)
+
+        elif state == State.RUNNING:
+            # Ping systemd watchdog before throttling
+            if self._sd_notify:
+                logger.debug("sd_notify: WATCHDOG=1\\nSTATUS=State: RUNNING.")
+                self._sd_notify.notify("WATCHDOG=1\nSTATUS=State: RUNNING.")
+
+            self._throttle(func=self._process, min_secs=throttle_secs)
+
+        return state
+
+    def _throttle(self, func: Callable[..., Any], min_secs: float, *args, **kwargs) -> Any:
+        """
+        Throttles the given callable that it
+        takes at least `min_secs` to finish execution.
+        :param func: Any callable
+        :param min_secs: minimum execution time in seconds
+        :return: Any
+        """
+        start = time.time()
+        result = func(*args, **kwargs)
+        end = time.time()
+        duration = max(min_secs - (end - start), 0.0)
+        logger.debug('Throttling %s for %.2f seconds', func.__name__, duration)
+        time.sleep(duration)
+        return result
+
+    def _process(self) -> bool:
+        state_changed = False
+        try:
+            state_changed = self.freqtrade.process()
+        except TemporaryError as error:
+            logger.warning(f"Error: {error}, retrying in {constants.RETRY_TIMEOUT} seconds...")
+            time.sleep(constants.RETRY_TIMEOUT)
+        except OperationalException:
+            tb = traceback.format_exc()
+            hint = 'Issue `/start` if you think it is safe to restart.'
+            self.freqtrade.rpc.send_msg({
+                'type': RPCMessageType.STATUS_NOTIFICATION,
+                'status': f'OperationalException:\n```\n{tb}```{hint}'
+            })
+            logger.exception('OperationalException. Stopping trader ...')
+            self.freqtrade.state = State.STOPPED
+            # TODO: The return value of _process() is not used apart tests
+            # and should (could) be eliminated later. See PR #1689.
+#            state_changed = True
+        return state_changed
+
+    def _reconfigure(self):
+        """
+        Cleans up current freqtradebot instance, reloads the configuration and
+        replaces it with the new instance
+        """
+        # Tell systemd that we initiated reconfiguration
+        if self._sd_notify:
+            logger.debug("sd_notify: RELOADING=1")
+            self._sd_notify.notify("RELOADING=1")
+
+        # Clean up current freqtrade modules
+        self.freqtrade.cleanup()
+
+        # Load and validate config and create new instance of the bot
+        self._init(True)
+
+        self.freqtrade.rpc.send_msg({
+            'type': RPCMessageType.STATUS_NOTIFICATION,
+            'status': 'config reloaded'
+        })
+
+        # Tell systemd that we completed reconfiguration
+        if self._sd_notify:
+            logger.debug("sd_notify: READY=1")
+            self._sd_notify.notify("READY=1")
+
+    def exit(self):
+        # Tell systemd that we are exiting now
+        if self._sd_notify:
+            logger.debug("sd_notify: STOPPING=1")
+            self._sd_notify.notify("STOPPING=1")
+
+        if self.freqtrade:
+            self.freqtrade.rpc.send_msg({
+                'type': RPCMessageType.STATUS_NOTIFICATION,
+                'status': 'process died'
+            })
+            self.freqtrade.cleanup()

mkdocs.yml

@@ -3,11 +3,12 @@ nav:
     - About: index.md
     - Installation: installation.md
     - Configuration: configuration.md
-    - Start the bot: bot-usage.md
-    - Stoploss: stoploss.md
     - Custom Strategy: bot-optimization.md
-    - Telegram: telegram-usage.md
+    - Stoploss: stoploss.md
-    - Web Hook: webhook-config.md
+    - Start the bot: bot-usage.md
+    - Control the bot:
+        - Telegram: telegram-usage.md
+        - Web Hook: webhook-config.md
     - Backtesting: backtesting.md
     - Hyperopt: hyperopt.md
     - Edge positioning: edge.md

requirements-dev.txt

@@ -9,4 +9,4 @@ pytest-mock==1.10.3
 pytest-asyncio==0.10.0
 pytest-cov==2.6.1
 coveralls==1.7.0
-mypy==0.670
+mypy==0.700

requirements.txt

@@ -1,5 +1,5 @@
-ccxt==1.18.425
+ccxt==1.18.432
-SQLAlchemy==1.3.1
+SQLAlchemy==1.3.2
 python-telegram-bot==11.1.0
 arrow==0.13.1
 cachetools==3.1.0