Merge branch 'develop' into interface_ordertimeoutcallback

.gitignore

@@ -6,7 +6,6 @@ user_data/*
 !user_data/strategy/sample_strategy.py
 !user_data/notebooks
 user_data/notebooks/*
-!user_data/notebooks/*example.ipynb
 freqtrade-plot.html
 freqtrade-profit-plot.html
 

MANIFEST.in

@@ -2,3 +2,4 @@ include LICENSE
 include README.md
 include config.json.example
 recursive-include freqtrade *.py
+recursive-include freqtrade/templates/ *.j2 *.ipynb

docs/bot-usage.md

@@ -144,10 +144,10 @@ It is recommended to use version control to keep track of changes to your strate
 ### How to use **--strategy**?
 
 This parameter will allow you to load your custom strategy class.
-Per default without `--strategy` or `-s` the bot will load the
-`DefaultStrategy` included with the bot (`freqtrade/strategy/default_strategy.py`).
+To test the bot installation, you can use the `SampleStrategy` installed by the `create-userdir` subcommand (usually `user_data/strategy/sample_strategy.py`).
 
-The bot will search your strategy file within `user_data/strategies` and `freqtrade/strategy`.
+The bot will search your strategy file within `user_data/strategies`.
+To use other directories, please read the next section about `--strategy-path`.
 
 To load a strategy, simply pass the class name (e.g.: `CustomStrategy`) in this parameter.
 

docs/configuration.md

@@ -34,13 +34,13 @@ The prevelance for all Options is as follows:
 
 - CLI arguments override any other option
 - Configuration files are used in sequence (last file wins), and override Strategy configurations.
-- Strategy configurations are only used if they are not set via configuration or via command line arguments. These options are market with [Strategy Override](#parameters-in-the-strategy) in the below table.
+- Strategy configurations are only used if they are not set via configuration or via command line arguments. These options are marked with [Strategy Override](#parameters-in-the-strategy) in the below table.
 
 Mandatory parameters are marked as **Required**, which means that they are required to be set in one of the possible ways.
 
 |  Parameter | Description |
 |------------|-------------|
-| `max_open_trades` | **Required.** Number of trades open your bot will have. If -1 then it is ignored (i.e. potentially unlimited open trades). [More information below](#configuring-amount-per-trade).<br> **Datatype:** Positive integer or -1.
+| `max_open_trades` | **Required.** Number of open trades your bot is allowed to have. Only one open trade per pair is possible, so the length of your pairlist is another limitation which can apply. If -1 then it is ignored (i.e. potentially unlimited open trades, limited by the pairlist). [More information below](#configuring-amount-per-trade).<br> **Datatype:** Positive integer or -1.
 | `stake_currency` | **Required.** Crypto-currency used for trading. [Strategy Override](#parameters-in-the-strategy). <br> **Datatype:** String
 | `stake_amount` | **Required.** Amount of crypto-currency your bot will use for each trade. Set it to `"unlimited"` to allow the bot to use all available balance. [More information below](#configuring-amount-per-trade). [Strategy Override](#parameters-in-the-strategy). <br> **Datatype:** Positive float or `"unlimited"`.
 | `tradable_balance_ratio` | Ratio of the total account balance the bot is allowed to trade. [More information below](#configuring-amount-per-trade). <br>*Defaults to `0.99` 99%).*<br> **Datatype:** Positive float between `0.1` and `1.0`.

docs/exchanges.md

@@ -74,23 +74,13 @@ Should you experience constant errors with Nonce (like `InvalidNonce`), it is be
 $ pip3 install web3
 ```
 
-### Send incomplete candles to the strategy
+### Getting latest price / Incomplete candles
 
 Most exchanges return current incomplete candle via their OHLCV/klines API interface.
 By default, Freqtrade assumes that incomplete candle is fetched from the exchange and removes the last candle assuming it's the incomplete candle.
 
 Whether your exchange returns incomplete candles or not can be checked using [the helper script](developer.md#Incomplete-candles) from the Contributor documentation.
 
-If the exchange does return incomplete candles and you would like to have incomplete candles in your strategy, you can set the following parameter in the configuration file.
+Due to the danger of repainting, Freqtrade does not allow you to use this incomplete candle.
 
-``` json
-{
-
-    "exchange": {
-        "_ft_has_params": {"ohlcv_partial_candle": false}
-    }
-}
-```
-
-!!! Warning "Danger of repainting"
-    Changing this parameter makes the strategy responsible to avoid repainting and handle this accordingly. Doing this is therefore not recommended, and should only be performed by experienced users who are fully aware of the impact this setting has.
+However, if it is based on the need for the latest price for your strategy - then this requirement can be acquired using the [data provider](strategy-customization.md#possible-options-for-dataprovider) from within the strategy.

docs/hyperopt.md

@@ -16,6 +16,24 @@ To learn how to get data for the pairs and exchange you're interested in, head o
 !!! Bug
     Hyperopt can crash when used with only 1 CPU Core as found out in [Issue #1133](https://github.com/freqtrade/freqtrade/issues/1133)
 
+## Install hyperopt dependencies
+
+Since Hyperopt dependencies are not needed to run the bot itself, are heavy, can not be easily built on some platforms (like Raspberry PI), they are not installed by default. Before you run Hyperopt, you need to install the corresponding dependencies, as described in this section below.
+
+!!! Note
+    Since Hyperopt is a resource intensive process, running it on a Raspberry Pi is not recommended nor supported.
+
+### Docker
+
+The docker-image includes hyperopt dependencies, no further action needed.
+
+### Easy installation script (setup.sh) / Manual installation
+
+```bash
+source .env/bin/activate
+pip install -r requirements-hyperopt.txt
+```
+
 ## Prepare Hyperopting
 
 Before we start digging into Hyperopt, we recommend you to take a look at

docs/plotting.md

@@ -23,44 +23,64 @@ The `freqtrade plot-dataframe` subcommand shows an interactive graph with three
 Possible arguments:
 
 ```
-usage: freqtrade plot-dataframe [-h] [-v] [--logfile FILE] [-V] [-c PATH] [-d PATH] [--userdir PATH] [-s NAME]
-                                [--strategy-path PATH] [-p PAIRS [PAIRS ...]] [--indicators1 INDICATORS1 [INDICATORS1 ...]]
-                                [--indicators2 INDICATORS2 [INDICATORS2 ...]] [--plot-limit INT] [--db-url PATH]
-                                [--trade-source {DB,file}] [--export EXPORT] [--export-filename PATH] [--timerange TIMERANGE]
-                                [-i TICKER_INTERVAL]
+usage: freqtrade plot-dataframe [-h] [-v] [--logfile FILE] [-V] [-c PATH]
+                                [-d PATH] [--userdir PATH] [-s NAME]
+                                [--strategy-path PATH] [-p PAIRS [PAIRS ...]]
+                                [--indicators1 INDICATORS1 [INDICATORS1 ...]]
+                                [--indicators2 INDICATORS2 [INDICATORS2 ...]]
+                                [--plot-limit INT] [--db-url PATH]
+                                [--trade-source {DB,file}] [--export EXPORT]
+                                [--export-filename PATH]
+                                [--timerange TIMERANGE] [-i TICKER_INTERVAL]
+                                [--no-trades]
 
 optional arguments:
   -h, --help            show this help message and exit
   -p PAIRS [PAIRS ...], --pairs PAIRS [PAIRS ...]
-                        Show profits for only these pairs. Pairs are space-separated.
+                        Show profits for only these pairs. Pairs are space-
+                        separated.
   --indicators1 INDICATORS1 [INDICATORS1 ...]
-                        Set indicators from your strategy you want in the first row of the graph. Space-separated list. Example:
+                        Set indicators from your strategy you want in the
+                        first row of the graph. Space-separated list. Example:
                         `ema3 ema5`. Default: `['sma', 'ema3', 'ema5']`.
   --indicators2 INDICATORS2 [INDICATORS2 ...]
-                        Set indicators from your strategy you want in the third row of the graph. Space-separated list. Example:
+                        Set indicators from your strategy you want in the
+                        third row of the graph. Space-separated list. Example:
                         `fastd fastk`. Default: `['macd', 'macdsignal']`.
-  --plot-limit INT      Specify tick limit for plotting. Notice: too high values cause huge files. Default: 750.
-  --db-url PATH         Override trades database URL, this is useful in custom deployments (default: `sqlite:///tradesv3.sqlite`
-                        for Live Run mode, `sqlite:///tradesv3.dryrun.sqlite` for Dry Run).
+  --plot-limit INT      Specify tick limit for plotting. Notice: too high
+                        values cause huge files. Default: 750.
+  --db-url PATH         Override trades database URL, this is useful in custom
+                        deployments (default: `sqlite:///tradesv3.sqlite` for
+                        Live Run mode, `sqlite:///tradesv3.dryrun.sqlite` for
+                        Dry Run).
   --trade-source {DB,file}
-                        Specify the source for trades (Can be DB or file (backtest file)) Default: file
-  --export EXPORT       Export backtest results, argument are: trades. Example: `--export=trades`
+                        Specify the source for trades (Can be DB or file
+                        (backtest file)) Default: file
+  --export EXPORT       Export backtest results, argument are: trades.
+                        Example: `--export=trades`
   --export-filename PATH
-                        Save backtest results to the file with this filename. Requires `--export` to be set as well. Example:
-                        `--export-filename=user_data/backtest_results/backtest_today.json`
+                        Save backtest results to the file with this filename.
+                        Requires `--export` to be set as well. Example:
+                        `--export-filename=user_data/backtest_results/backtest
+                        _today.json`
   --timerange TIMERANGE
                         Specify what timerange of data to use.
   -i TICKER_INTERVAL, --ticker-interval TICKER_INTERVAL
-                        Specify ticker interval (`1m`, `5m`, `30m`, `1h`, `1d`).
+                        Specify ticker interval (`1m`, `5m`, `30m`, `1h`,
+                        `1d`).
+  --no-trades           Skip using trades from backtesting file and DB.
 
 Common arguments:
   -v, --verbose         Verbose mode (-vv for more, -vvv to get all messages).
-  --logfile FILE        Log to the file specified. Special values are: 'syslog', 'journald'. See the documentation for more
+  --logfile FILE        Log to the file specified. Special values are:
+                        'syslog', 'journald'. See the documentation for more
                         details.
   -V, --version         show program's version number and exit
   -c PATH, --config PATH
-                        Specify configuration file (default: `config.json`). Multiple --config options may be used. Can be set to
-                        `-` to read config from stdin.
+                        Specify configuration file (default:
+                        `userdir/config.json` or `config.json` whichever
+                        exists). Multiple --config options may be used. Can be
+                        set to `-` to read config from stdin.
   -d PATH, --datadir PATH
                         Path to directory with historical backtesting data.
   --userdir PATH, --user-data-dir PATH
@@ -68,9 +88,9 @@ Common arguments:
 
 Strategy arguments:
   -s NAME, --strategy NAME
-                        Specify strategy class name which will be used by the bot.
+                        Specify strategy class name which will be used by the
+                        bot.
   --strategy-path PATH  Specify additional strategy lookup path.
-
 ```
 
 Example:

docs/requirements-docs.txt

@@ -1,2 +1,2 @@
-mkdocs-material==4.6.3
+mkdocs-material==5.1.0
 mdx_truly_sane_lists==1.2

freqtrade/__init__.py

@@ -24,4 +24,11 @@ if __version__ == 'develop':
         #     stderr=subprocess.DEVNULL).decode("utf-8").rstrip().strip('"')
     except Exception:
         # git not available, ignore
-        pass
+        try:
+            # Try Fallback to freqtrade_commit file (created by CI while building docker image)
+            from pathlib import Path
+            versionfile = Path('./freqtrade_commit')
+            if versionfile.is_file():
+                __version__ = f"docker-{versionfile.read_text()[:8]}"
+        except Exception:
+            pass

freqtrade/commands/arguments.py

@@ -59,7 +59,7 @@ ARGS_DOWNLOAD_DATA = ["pairs", "pairs_file", "days", "download_trades", "exchang
 
 ARGS_PLOT_DATAFRAME = ["pairs", "indicators1", "indicators2", "plot_limit",
                        "db_url", "trade_source", "export", "exportfilename",
-                       "timerange", "ticker_interval"]
+                       "timerange", "ticker_interval", "no_trades"]
 
 ARGS_PLOT_PROFIT = ["pairs", "timerange", "export", "exportfilename", "db_url",
                     "trade_source", "ticker_interval"]

freqtrade/commands/cli_options.py

@@ -413,6 +413,11 @@ AVAILABLE_CLI_OPTIONS = {
         metavar='INT',
         default=750,
     ),
+    "no_trades": Arg(
+        '--no-trades',
+        help='Skip using trades from backtesting file and DB.',
+        action='store_true',
+    ),
     "trade_source": Arg(
         '--trade-source',
         help='Specify the source for trades (Can be DB or file (backtest file)) '

freqtrade/commands/hyperopt_commands.py

@@ -52,8 +52,8 @@ def start_hyperopt_list(args: Dict[str, Any]) -> None:
 
     if not export_csv:
         try:
-            Hyperopt.print_result_table(config, trials, total_epochs,
-                                        not filteroptions['only_best'], print_colorized, 0)
+            print(Hyperopt.get_result_table(config, trials, total_epochs,
+                                            not filteroptions['only_best'], print_colorized, 0))
         except KeyboardInterrupt:
             print('User interrupted..')
 

freqtrade/configuration/configuration.py

@@ -359,6 +359,9 @@ class Configuration:
         self._args_to_config(config, argname='erase',
                              logstring='Erase detected. Deleting existing data.')
 
+        self._args_to_config(config, argname='no_trades',
+                             logstring='Parameter --no-trades detected.')
+
         self._args_to_config(config, argname='timeframes',
                              logstring='timeframes --timeframes: {}')
 

freqtrade/configuration/load_config.py

@@ -1,13 +1,15 @@
 """
 This module contain functions to load the configuration file
 """
-import rapidjson
 import logging
+import re
 import sys
+from pathlib import Path
 from typing import Any, Dict
 
-from freqtrade.exceptions import OperationalException
+import rapidjson
 
+from freqtrade.exceptions import OperationalException
 
 logger = logging.getLogger(__name__)
 
@@ -15,6 +17,26 @@ logger = logging.getLogger(__name__)
 CONFIG_PARSE_MODE = rapidjson.PM_COMMENTS | rapidjson.PM_TRAILING_COMMAS
 
 
+def log_config_error_range(path: str, errmsg: str) -> str:
+    """
+    Parses configuration file and prints range around error
+    """
+    if path != '-':
+        offsetlist = re.findall(r'(?<=Parse\serror\sat\soffset\s)\d+', errmsg)
+        if offsetlist:
+            offset = int(offsetlist[0])
+            text = Path(path).read_text()
+            # Fetch an offset of 80 characters around the error line
+            subtext = text[offset-min(80, offset):offset+80]
+            segments = subtext.split('\n')
+            if len(segments) > 3:
+                # Remove first and last lines, to avoid odd truncations
+                return '\n'.join(segments[1:-1])
+            else:
+                return subtext
+    return ''
+
+
 def load_config_file(path: str) -> Dict[str, Any]:
     """
     Loads a config file from the given path
@@ -29,5 +51,12 @@ def load_config_file(path: str) -> Dict[str, Any]:
         raise OperationalException(
             f'Config file "{path}" not found!'
             ' Please create a config file or check whether it exists.')
+    except rapidjson.JSONDecodeError as e:
+        err_range = log_config_error_range(path, str(e))
+        raise OperationalException(
+            f'{e}\n'
+            f'Please verify the following segment of your configuration:\n{err_range}'
+            if err_range else 'Please verify your configuration file for syntax errors.'
+        )
 
     return config

freqtrade/data/btanalysis.py

@@ -111,7 +111,7 @@ def load_trades_from_db(db_url: str) -> pd.DataFrame:
                             t.calc_profit(), t.calc_profit_ratio(),
                             t.open_rate, t.close_rate, t.amount,
                             (round((t.close_date.timestamp() - t.open_date.timestamp()) / 60, 2)
-                                if t.close_date else None),
+                             if t.close_date else None),
                             t.sell_reason,
                             t.fee_open, t.fee_close,
                             t.open_rate_requested,
@@ -129,7 +129,8 @@ def load_trades_from_db(db_url: str) -> pd.DataFrame:
     return trades
 
 
-def load_trades(source: str, db_url: str, exportfilename: Path) -> pd.DataFrame:
+def load_trades(source: str, db_url: str, exportfilename: Path,
+                no_trades: bool = False) -> pd.DataFrame:
     """
     Based on configuration option "trade_source":
     * loads data from DB (using `db_url`)
@@ -137,21 +138,33 @@ def load_trades(source: str, db_url: str, exportfilename: Path) -> pd.DataFrame:
     :param source: "DB" or "file" - specify source to load from
     :param db_url: sqlalchemy formatted url to a database
     :param exportfilename: Json file generated by backtesting
+    :param no_trades: Skip using trades, only return backtesting data columns
     :return: DataFrame containing trades
     """
+    if no_trades:
+        df = pd.DataFrame(columns=BT_DATA_COLUMNS)
+        return df
+
     if source == "DB":
         return load_trades_from_db(db_url)
     elif source == "file":
         return load_backtest_data(exportfilename)
 
 
-def extract_trades_of_period(dataframe: pd.DataFrame, trades: pd.DataFrame) -> pd.DataFrame:
+def extract_trades_of_period(dataframe: pd.DataFrame, trades: pd.DataFrame,
+                             date_index=False) -> pd.DataFrame:
     """
     Compare trades and backtested pair DataFrames to get trades performed on backtested period
     :return: the DataFrame of a trades of period
     """
-    trades = trades.loc[(trades['open_time'] >= dataframe.iloc[0]['date']) &
-                        (trades['close_time'] <= dataframe.iloc[-1]['date'])]
+    if date_index:
+        trades_start = dataframe.index[0]
+        trades_stop = dataframe.index[-1]
+    else:
+        trades_start = dataframe.iloc[0]['date']
+        trades_stop = dataframe.iloc[-1]['date']
+    trades = trades.loc[(trades['open_time'] >= trades_start) &
+                        (trades['close_time'] <= trades_stop)]
     return trades
 
 
@@ -207,13 +220,15 @@ def calculate_max_drawdown(trades: pd.DataFrame, *, date_col: str = 'close_time'
     """
     if len(trades) == 0:
         raise ValueError("Trade dataframe empty.")
-    profit_results = trades.sort_values(date_col)
+    profit_results = trades.sort_values(date_col).reset_index(drop=True)
     max_drawdown_df = pd.DataFrame()
     max_drawdown_df['cumulative'] = profit_results[value_col].cumsum()
     max_drawdown_df['high_value'] = max_drawdown_df['cumulative'].cummax()
     max_drawdown_df['drawdown'] = max_drawdown_df['cumulative'] - max_drawdown_df['high_value']
 
-    high_date = profit_results.loc[max_drawdown_df['high_value'].idxmax(), date_col]
-    low_date = profit_results.loc[max_drawdown_df['drawdown'].idxmin(), date_col]
-
+    idxmin = max_drawdown_df['drawdown'].idxmin()
+    if idxmin == 0:
+        raise ValueError("No losing trade, therefore no drawdown.")
+    high_date = profit_results.loc[max_drawdown_df.iloc[:idxmin]['high_value'].idxmax(), date_col]
+    low_date = profit_results.loc[idxmin, date_col]
     return abs(min(max_drawdown_df['drawdown'])), high_date, low_date

freqtrade/edge/edge_positioning.py

@@ -8,10 +8,10 @@ import numpy as np
 import utils_find_1st as utf1st
 from pandas import DataFrame
 
-from freqtrade import constants
 from freqtrade.configuration import TimeRange
-from freqtrade.data import history
+from freqtrade.constants import UNLIMITED_STAKE_AMOUNT
 from freqtrade.exceptions import OperationalException
+from freqtrade.data.history import get_timerange, load_data, refresh_data
 from freqtrade.strategy.interface import SellType
 
 logger = logging.getLogger(__name__)
@@ -54,7 +54,7 @@ class Edge:
         if self.config['max_open_trades'] != float('inf'):
             logger.critical('max_open_trades should be -1 in config !')
 
-        if self.config['stake_amount'] != constants.UNLIMITED_STAKE_AMOUNT:
+        if self.config['stake_amount'] != UNLIMITED_STAKE_AMOUNT:
             raise OperationalException('Edge works only with unlimited stake amount')
 
         # Deprecated capital_available_percentage. Will use tradable_balance_ratio in the future.
@@ -96,7 +96,7 @@ class Edge:
         logger.info('Using local backtesting data (using whitelist in given config) ...')
 
         if self._refresh_pairs:
-            history.refresh_data(
+            refresh_data(
                 datadir=self.config['datadir'],
                 pairs=pairs,
                 exchange=self.exchange,
@@ -104,7 +104,7 @@ class Edge:
                 timerange=self._timerange,
             )
 
-        data = history.load_data(
+        data = load_data(
             datadir=self.config['datadir'],
             pairs=pairs,
             timeframe=self.strategy.ticker_interval,
@@ -122,7 +122,7 @@ class Edge:
         preprocessed = self.strategy.ohlcvdata_to_dataframe(data)
 
         # Print timeframe
-        min_date, max_date = history.get_timerange(preprocessed)
+        min_date, max_date = get_timerange(preprocessed)
         logger.info(
             'Measuring data from %s up to %s (%s days) ...',
             min_date.isoformat(),

freqtrade/exchange/exchange.py

@@ -452,6 +452,17 @@ class Exchange:
                 price = ceil(big_price) / pow(10, symbol_prec)
         return price
 
+    def price_get_one_pip(self, pair: str, price: float) -> float:
+        """
+        Get's the "1 pip" value for this pair.
+        Used in PriceFilter to calculate the 1pip movements.
+        """
+        precision = self.markets[pair]['precision']['price']
+        if self.precisionMode == TICK_SIZE:
+            return precision
+        else:
+            return 1 / pow(10, precision)
+
     def dry_run_order(self, pair: str, ordertype: str, side: str, amount: float,
                       rate: float, params: Dict = {}) -> Dict[str, Any]:
         order_id = f'dry_run_{side}_{randint(0, 10**6)}'
@@ -902,6 +913,14 @@ class Exchange:
             self._async_get_trade_history(pair=pair, since=since,
                                           until=until, from_id=from_id))
 
+    def check_order_canceled_empty(self, order: Dict) -> bool:
+        """
+        Verify if an order has been cancelled without being partially filled
+        :param order: Order dict as returned from get_order()
+        :return: True if order has been cancelled without being filled, False otherwise.
+        """
+        return order.get('status') in ('closed', 'canceled') and order.get('filled') == 0.0
+
     @retrier
     def cancel_order(self, order_id: str, pair: str) -> Dict:
         if self._config['dry_run']:
@@ -918,6 +937,37 @@ class Exchange:
         except ccxt.BaseError as e:
             raise OperationalException(e) from e
 
+    def is_cancel_order_result_suitable(self, corder) -> bool:
+        if not isinstance(corder, dict):
+            return False
+
+        required = ('fee', 'status', 'amount')
+        return all(k in corder for k in required)
+
+    def cancel_order_with_result(self, order_id: str, pair: str, amount: float) -> Dict:
+        """
+        Cancel order returning a result.
+        Creates a fake result if cancel order returns a non-usable result
+        and get_order does not work (certain exchanges don't return cancelled orders)
+        :param order_id: Orderid to cancel
+        :param pair: Pair corresponding to order_id
+        :param amount: Amount to use for fake response
+        :return: Result from either cancel_order if usable, or fetch_order
+        """
+        try:
+            corder = self.cancel_order(order_id, pair)
+            if self.is_cancel_order_result_suitable(corder):
+                return corder
+        except InvalidOrderException:
+            logger.warning(f"Could not cancel order {order_id}.")
+        try:
+            order = self.get_order(order_id, pair)
+        except InvalidOrderException:
+            logger.warning(f"Could not fetch cancelled order {order_id}.")
+            order = {'fee': {}, 'status': 'canceled', 'amount': amount, 'info': {}}
+
+        return order
+
     @retrier
     def get_order(self, order_id: str, pair: str) -> Dict:
         if self._config['dry_run']:

freqtrade/freqtradebot.py

@@ -20,6 +20,7 @@ from freqtrade.data.dataprovider import DataProvider
 from freqtrade.edge import Edge
 from freqtrade.exceptions import DependencyException, InvalidOrderException
 from freqtrade.exchange import timeframe_to_minutes, timeframe_to_next_date
+from freqtrade.misc import safe_value_fallback
 from freqtrade.pairlist.pairlistmanager import PairListManager
 from freqtrade.persistence import Trade
 from freqtrade.resolvers import ExchangeResolver, StrategyResolver
@@ -144,6 +145,10 @@ class FreqtradeBot:
         self.dataprovider.refresh(self._create_pair_whitelist(self.active_pair_whitelist),
                                   self.strategy.informative_pairs())
 
+        with self._sell_lock:
+            # Check and handle any timed out open orders
+            self.check_handle_timedout()
+
         # Protect from collisions with forcesell.
         # Without this, freqtrade my try to recreate stoploss_on_exchange orders
         # while selling is in process, since telegram messages arrive in an different thread.
@@ -155,8 +160,6 @@ class FreqtradeBot:
         if self.get_free_open_trades():
             self.enter_positions()
 
-        # Check and handle any timed out open orders
-        self.check_handle_timedout()
         Trade.session.flush()
 
     def _refresh_whitelist(self, trades: List[Trade] = []) -> List[str]:
@@ -395,16 +398,18 @@ class FreqtradeBot:
             logger.info(f"Pair {pair} is currently locked.")
             return False
 
+        # get_free_open_trades is checked before create_trade is called
+        # but it is still used here to prevent opening too many trades within one iteration
+        if not self.get_free_open_trades():
+            logger.debug(f"Can't open a new trade for {pair}: max number of trades is reached.")
+            return False
+
         # running get_signal on historical data fetched
         (buy, sell) = self.strategy.get_signal(
             pair, self.strategy.ticker_interval,
             self.dataprovider.ohlcv(pair, self.strategy.ticker_interval))
 
         if buy and not sell:
-            if not self.get_free_open_trades():
-                logger.debug("Can't open a new trade: max number of trades is reached.")
-                return False
-
             stake_amount = self.get_trade_stake_amount(pair)
             if not stake_amount:
                 logger.debug(f"Stake amount is 0, ignoring possible trade for {pair}.")
@@ -599,7 +604,6 @@ class FreqtradeBot:
         trades_closed = 0
         for trade in trades:
             try:
-                self.update_trade_state(trade)
 
                 if (self.strategy.order_types.get('stoploss_on_exchange') and
                         self.handle_stoploss_on_exchange(trade)):
@@ -859,19 +863,13 @@ class FreqtradeBot:
                     continue
                 order = self.exchange.get_order(trade.open_order_id, trade.pair)
             except (RequestException, DependencyException, InvalidOrderException):
-                logger.info(
-                    'Cannot query order for %s due to %s',
-                    trade,
-                    traceback.format_exc())
+                logger.info('Cannot query order for %s due to %s', trade, traceback.format_exc())
                 continue
 
-            # Check if trade is still actually open
-            if float(order.get('remaining', 0.0)) == 0.0:
-                self.wallets.update()
-                continue
+            trade_state_update = self.update_trade_state(trade, order)
 
             if (order['side'] == 'buy' and (
-                    order['status'] == 'canceled'
+                    trade_state_update
                     or self._check_timed_out('buy', order)
                     or strategy_safe_wrapper(self.strategy.check_buy_timeout,
                                              default_retval=False)(pair=trade.pair,
@@ -884,16 +882,16 @@ class FreqtradeBot:
                 self._notify_buy_cancel(trade, order_type)
 
             elif (order['side'] == 'sell' and (
-                    order['status'] == 'canceled'
+                  trade_state_update
                   or self._check_timed_out('sell', order)
                   or strategy_safe_wrapper(self.strategy.check_sell_timeout,
                                            default_retval=False)(pair=trade.pair,
                                                                  trade=trade,
                                                                  order=order))):
-                self.handle_timedout_limit_sell(trade, order)
+                reason = self.handle_timedout_limit_sell(trade, order)
                 self.wallets.update()
                 order_type = self.strategy.order_types['sell']
-                self._notify_sell_cancel(trade, order_type)
+                self._notify_sell_cancel(trade, order_type, reason)
 
     def handle_timedout_limit_buy(self, trade: Trade, order: Dict) -> bool:
         """
@@ -902,15 +900,17 @@ class FreqtradeBot:
         """
         if order['status'] != 'canceled':
             reason = "cancelled due to timeout"
-            corder = self.exchange.cancel_order(trade.open_order_id, trade.pair)
-            logger.info('Buy order %s for %s.', reason, trade)
+            corder = self.exchange.cancel_order_with_result(trade.open_order_id, trade.pair,
+                                                            trade.amount)
         else:
             # Order was cancelled already, so we can reuse the existing dict
             corder = order
             reason = "cancelled on exchange"
-            logger.info('Buy order %s for %s.', reason, trade)
 
-        if corder.get('remaining', order['remaining']) == order['amount']:
+        logger.info('Buy order %s for %s.', reason, trade)
+
+        if safe_value_fallback(corder, order, 'remaining', 'remaining') == order['amount']:
+            logger.info('Buy order fully cancelled. Removing %s from database.', trade)
             # if trade is not partially completed, just delete the trade
             Trade.session.delete(trade)
             Trade.session.flush()
@@ -921,19 +921,10 @@ class FreqtradeBot:
         # cancel_order may not contain the full order dict, so we need to fallback
         # to the order dict aquired before cancelling.
         # we need to fall back to the values from order if corder does not contain these keys.
-        trade.amount = order['amount'] - corder.get('remaining', order['remaining'])
+        trade.amount = order['amount'] - safe_value_fallback(corder, order,
+                                                             'remaining', 'remaining')
         trade.stake_amount = trade.amount * trade.open_rate
-        # verify if fees were taken from amount to avoid problems during selling
-        try:
-            new_amount = self.get_real_amount(trade, corder if 'fee' in corder else order,
-                                              trade.amount)
-            if not isclose(order['amount'], new_amount, abs_tol=constants.MATH_CLOSE_PREC):
-                trade.amount = new_amount
-                # Fee was applied, so set to 0
-                trade.fee_open = 0
-                trade.recalc_open_trade_price()
-        except DependencyException as e:
-            logger.warning("Could not update trade amount: %s", e)
+        self.update_trade_state(trade, corder, trade.amount)
 
         trade.open_order_id = None
         logger.info('Partial buy order timeout for %s.', trade)
@@ -943,14 +934,14 @@ class FreqtradeBot:
         })
         return False
 
-    def handle_timedout_limit_sell(self, trade: Trade, order: Dict) -> bool:
+    def handle_timedout_limit_sell(self, trade: Trade, order: Dict) -> str:
         """
         Sell timeout - cancel order and update trade
-        :return: True if order was fully cancelled
+        :return: Reason for cancel
         """
         # if trade is not partially completed, just cancel the trade
-        if order['remaining'] == order['amount']:
-            if order["status"] != "canceled":
+        if order['remaining'] == order['amount'] or order.get('filled') == 0.0:
+            if not self.exchange.check_order_canceled_empty(order):
                 reason = "cancelled due to timeout"
                 # if trade is not partially completed, just delete the trade
                 self.exchange.cancel_order(trade.open_order_id, trade.pair)
@@ -960,15 +951,17 @@ class FreqtradeBot:
                 logger.info('Sell order %s for %s.', reason, trade)
 
             trade.close_rate = None
+            trade.close_rate_requested = None
             trade.close_profit = None
+            trade.close_profit_abs = None
             trade.close_date = None
             trade.is_open = True
             trade.open_order_id = None
 
-            return True
+            return reason
 
         # TODO: figure out how to handle partially complete sell orders
-        return False
+        return 'partially filled - keeping order open'
 
     def _safe_sell_amount(self, pair: str, amount: float) -> float:
         """
@@ -1087,7 +1080,7 @@ class FreqtradeBot:
         # Send the message
         self.rpc.send_msg(msg)
 
-    def _notify_sell_cancel(self, trade: Trade, order_type: str) -> None:
+    def _notify_sell_cancel(self, trade: Trade, order_type: str, reason: str) -> None:
         """
         Sends rpc notification when a sell cancel occured.
         """
@@ -1114,6 +1107,7 @@ class FreqtradeBot:
             'close_date': trade.close_date,
             'stake_currency': self.config['stake_currency'],
             'fiat_currency': self.config.get('fiat_display_currency', None),
+            'reason': reason,
         }
 
         if 'fiat_display_currency' in self.config:
@@ -1128,9 +1122,12 @@ class FreqtradeBot:
 # Common update trade state methods
 #
 
-    def update_trade_state(self, trade: Trade, action_order: dict = None) -> None:
+    def update_trade_state(self, trade: Trade, action_order: dict = None,
+                           order_amount: float = None) -> bool:
         """
         Checks trades with open orders and updates the amount if necessary
+        Handles closing both buy and sell orders.
+        :return: True if order has been cancelled without being filled partially, False otherwise
         """
         # Get order details for actual price per unit
         if trade.open_order_id:
@@ -1140,25 +1137,31 @@ class FreqtradeBot:
                 order = action_order or self.exchange.get_order(trade.open_order_id, trade.pair)
             except InvalidOrderException as exception:
                 logger.warning('Unable to fetch order %s: %s', trade.open_order_id, exception)
-                return
+                return False
             # Try update amount (binance-fix)
             try:
-                new_amount = self.get_real_amount(trade, order)
+                new_amount = self.get_real_amount(trade, order, order_amount)
                 if not isclose(order['amount'], new_amount, abs_tol=constants.MATH_CLOSE_PREC):
                     order['amount'] = new_amount
+                    order.pop('filled', None)
                     # Fee was applied, so set to 0
                     trade.fee_open = 0
                     trade.recalc_open_trade_price()
-
             except DependencyException as exception:
                 logger.warning("Could not update trade amount: %s", exception)
 
+            if self.exchange.check_order_canceled_empty(order):
+                # Trade has been cancelled on exchange
+                # Handling of this will happen in check_handle_timeout.
+                return True
             trade.update(order)
 
             # Updating wallets when order is closed
             if not trade.is_open:
                 self.wallets.update()
 
+        return False
+
     def get_real_amount(self, trade: Trade, order: Dict, order_amount: float = None) -> float:
         """
         Get real amount for the trade

freqtrade/loggers.py

@@ -18,13 +18,13 @@ def _set_loggers(verbosity: int = 0) -> None:
     """
 
     logging.getLogger('requests').setLevel(
-            logging.INFO if verbosity <= 1 else logging.DEBUG
+        logging.INFO if verbosity <= 1 else logging.DEBUG
     )
     logging.getLogger("urllib3").setLevel(
-            logging.INFO if verbosity <= 1 else logging.DEBUG
+        logging.INFO if verbosity <= 1 else logging.DEBUG
     )
     logging.getLogger('ccxt.base.exchange').setLevel(
-            logging.INFO if verbosity <= 2 else logging.DEBUG
+        logging.INFO if verbosity <= 2 else logging.DEBUG
     )
     logging.getLogger('telegram').setLevel(logging.INFO)
 

freqtrade/misc.py

@@ -134,6 +134,21 @@ def round_dict(d, n):
     return {k: (round(v, n) if isinstance(v, float) else v) for k, v in d.items()}
 
 
+def safe_value_fallback(dict1: dict, dict2: dict, key1: str, key2: str, default_value=None):
+    """
+    Search a value in dict1, return this if it's not None.
+    Fall back to dict2 - return key2 from dict2 if it's not None.
+    Else falls back to None.
+
+    """
+    if key1 in dict1 and dict1[key1] is not None:
+        return dict1[key1]
+    else:
+        if key2 in dict2 and dict2[key2] is not None:
+            return dict2[key2]
+    return default_value
+
+
 def plural(num: float, singular: str, plural: str = None) -> str:
     return singular if (num == 1 or num == -1) else plural or singular + 's'
 

freqtrade/optimize/backtesting.py

@@ -6,8 +6,7 @@ This module contains the backtesting logic
 import logging
 from copy import deepcopy
 from datetime import datetime, timedelta
-from pathlib import Path
-from typing import Any, Dict, List, NamedTuple, Optional
+from typing import Any, Dict, List, NamedTuple, Optional, Tuple
 
 import arrow
 from pandas import DataFrame
@@ -19,10 +18,8 @@ from freqtrade.data.converter import trim_dataframe
 from freqtrade.data.dataprovider import DataProvider
 from freqtrade.exceptions import OperationalException
 from freqtrade.exchange import timeframe_to_minutes, timeframe_to_seconds
-from freqtrade.misc import file_dump_json
-from freqtrade.optimize.optimize_reports import (
-    generate_text_table, generate_text_table_sell_reason,
-    generate_text_table_strategy)
+from freqtrade.optimize.optimize_reports import (show_backtest_results,
+                                                 store_backtest_result)
 from freqtrade.persistence import Trade
 from freqtrade.resolvers import ExchangeResolver, StrategyResolver
 from freqtrade.state import RunMode
@@ -108,7 +105,7 @@ class Backtesting:
         # And the regular "stoploss" function would not apply to that case
         self.strategy.order_types['stoploss_on_exchange'] = False
 
-    def load_bt_data(self):
+    def load_bt_data(self) -> Tuple[Dict[str, DataFrame], TimeRange]:
         timerange = TimeRange.parse_timerange(None if self.config.get(
             'timerange') is None else str(self.config.get('timerange')))
 
@@ -134,23 +131,6 @@ class Backtesting:
 
         return data, timerange
 
-    def _store_backtest_result(self, recordfilename: Path, results: DataFrame,
-                               strategyname: Optional[str] = None) -> None:
-
-        records = [(t.pair, t.profit_percent, t.open_time.timestamp(),
-                    t.close_time.timestamp(), t.open_index - 1, t.trade_duration,
-                    t.open_rate, t.close_rate, t.open_at_end, t.sell_reason.value)
-                   for index, t in results.iterrows()]
-
-        if records:
-            if strategyname:
-                # Inject strategyname to filename
-                recordfilename = Path.joinpath(
-                    recordfilename.parent,
-                    f'{recordfilename.stem}-{strategyname}').with_suffix(recordfilename.suffix)
-            logger.info(f'Dumping backtest results to {recordfilename}')
-            file_dump_json(recordfilename, records)
-
     def _get_ohlcv_as_lists(self, processed: Dict) -> Dict[str, DataFrame]:
         """
         Helper function to convert a processed dataframes into lists for performance reasons.
@@ -169,8 +149,8 @@ class Backtesting:
 
             # To avoid using data from future, we use buy/sell signals shifted
             # from the previous candle
-            df_analyzed.loc[:, 'buy'] = df_analyzed['buy'].shift(1)
-            df_analyzed.loc[:, 'sell'] = df_analyzed['sell'].shift(1)
+            df_analyzed.loc[:, 'buy'] = df_analyzed.loc[:, 'buy'].shift(1)
+            df_analyzed.loc[:, 'sell'] = df_analyzed.loc[:, 'sell'].shift(1)
 
             df_analyzed.drop(df_analyzed.head(1).index, inplace=True)
 
@@ -418,44 +398,7 @@ class Backtesting:
                 position_stacking=position_stacking,
             )
 
-        for strategy, results in all_results.items():
-
-            if self.config.get('export', False):
-                self._store_backtest_result(self.config['exportfilename'], results,
-                                            strategy if len(self.strategylist) > 1 else None)
-
-            print(f"Result for strategy {strategy}")
-            table = generate_text_table(data, stake_currency=self.config['stake_currency'],
-                                        max_open_trades=self.config['max_open_trades'],
-                                        results=results)
-            if isinstance(table, str):
-                print(' BACKTESTING REPORT '.center(len(table.splitlines()[0]), '='))
-            print(table)
-
-            table = generate_text_table_sell_reason(data,
-                                                    stake_currency=self.config['stake_currency'],
-                                                    max_open_trades=self.config['max_open_trades'],
-                                                    results=results)
-            if isinstance(table, str):
-                print(' SELL REASON STATS '.center(len(table.splitlines()[0]), '='))
-            print(table)
-
-            table = generate_text_table(data,
-                                        stake_currency=self.config['stake_currency'],
-                                        max_open_trades=self.config['max_open_trades'],
-                                        results=results.loc[results.open_at_end], skip_nan=True)
-            if isinstance(table, str):
-                print(' LEFT OPEN TRADES REPORT '.center(len(table.splitlines()[0]), '='))
-            print(table)
-            if isinstance(table, str):
-                print('=' * len(table.splitlines()[0]))
-            print()
-        if len(all_results) > 1:
-            # Print Strategy summary table
-            table = generate_text_table_strategy(self.config['stake_currency'],
-                                                 self.config['max_open_trades'],
-                                                 all_results=all_results)
-            print(' STRATEGY SUMMARY '.center(len(table.splitlines()[0]), '='))
-            print(table)
-            print('=' * len(table.splitlines()[0]))
-            print('\nFor more details, please look at the detail tables above')
+        if self.config.get('export', False):
+            store_backtest_result(self.config['exportfilename'], all_results)
+        # Show backtest results
+        show_backtest_results(self.config, data, all_results)

freqtrade/optimize/hyperopt.py

@@ -7,7 +7,6 @@ This module contains the hyperopt logic
 import locale
 import logging
 import random
-import sys
 import warnings
 from math import ceil
 from collections import OrderedDict
@@ -18,10 +17,10 @@ from typing import Any, Dict, List, Optional
 
 import rapidjson
 from colorama import Fore, Style
-from colorama import init as colorama_init
 from joblib import (Parallel, cpu_count, delayed, dump, load,
                     wrap_non_picklable_objects)
 from pandas import DataFrame, json_normalize, isna
+import progressbar
 import tabulate
 from os import path
 import io
@@ -43,7 +42,8 @@ with warnings.catch_warnings():
     from skopt import Optimizer
     from skopt.space import Dimension
 
-
+progressbar.streams.wrap_stderr()
+progressbar.streams.wrap_stdout()
 logger = logging.getLogger(__name__)
 
 
@@ -266,21 +266,33 @@ class Hyperopt:
         Log results if it is better than any previous evaluation
         """
         is_best = results['is_best']
-        if not self.print_all:
-            # Print '\n' after each 100th epoch to separate dots from the log messages.
-            # Otherwise output is messy on a terminal.
-            print('.', end='' if results['current_epoch'] % 100 != 0 else None)  # type: ignore
-            sys.stdout.flush()
 
         if self.print_all or is_best:
-            if not self.print_all:
-                # Separate the results explanation string from dots
-                print("\n")
-            self.print_result_table(self.config, results, self.total_epochs,
-                                    self.print_all, self.print_colorized,
-                                    self.hyperopt_table_header)
+            print(
+                self.get_result_table(
+                    self.config, results, self.total_epochs,
+                    self.print_all, self.print_colorized,
+                    self.hyperopt_table_header
+                )
+            )
             self.hyperopt_table_header = 2
 
+    def get_results(self, results) -> str:
+        """
+        Log results if it is better than any previous evaluation
+        """
+        output = ''
+        is_best = results['is_best']
+
+        if self.print_all or is_best:
+            output = self.get_result_table(
+                self.config, results, self.total_epochs,
+                self.print_all, self.print_colorized,
+                self.hyperopt_table_header
+            )
+            self.hyperopt_table_header = 2
+        return output
+
     @staticmethod
     def print_results_explanation(results, total_epochs, highlight_best: bool,
                                   print_colorized: bool) -> None:
@@ -304,13 +316,13 @@ class Hyperopt:
                 f"Objective: {results['loss']:.5f}")
 
     @staticmethod
-    def print_result_table(config: dict, results: list, total_epochs: int, highlight_best: bool,
-                           print_colorized: bool, remove_header: int) -> None:
+    def get_result_table(config: dict, results: list, total_epochs: int, highlight_best: bool,
+                         print_colorized: bool, remove_header: int) -> str:
         """
         Log result table
         """
         if not results:
-            return
+            return ''
 
         tabulate.PRESERVE_WHITESPACE = True
 
@@ -381,7 +393,7 @@ class Hyperopt:
                 trials.to_dict(orient='list'), tablefmt='psql',
                 headers='keys', stralign="right"
             )
-        print(table)
+        return table
 
     @staticmethod
     def export_csv_file(config: dict, results: list, total_epochs: int, highlight_best: bool,
@@ -653,48 +665,75 @@ class Hyperopt:
 
         self.dimensions: List[Dimension] = self.hyperopt_space()
         self.opt = self.get_optimizer(self.dimensions, config_jobs)
-
-        if self.print_colorized:
-            colorama_init(autoreset=True)
-
         try:
             with Parallel(n_jobs=config_jobs) as parallel:
                 jobs = parallel._effective_n_jobs()
                 logger.info(f'Effective number of parallel workers used: {jobs}')
-                EVALS = ceil(self.total_epochs / jobs)
-                for i in range(EVALS):
-                    # Correct the number of epochs to be processed for the last
-                    # iteration (should not exceed self.total_epochs in total)
-                    n_rest = (i + 1) * jobs - self.total_epochs
-                    current_jobs = jobs - n_rest if n_rest > 0 else jobs
 
-                    asked = self.opt.ask(n_points=current_jobs)
-                    f_val = self.run_optimizer_parallel(parallel, asked, i)
-                    self.opt.tell(asked, [v['loss'] for v in f_val])
-                    self.fix_optimizer_models_list()
+                # Define progressbar
+                if self.print_colorized:
+                    widgets = [
+                        ' [Epoch ', progressbar.Counter(), ' of ', str(self.total_epochs),
+                        ' (', progressbar.Percentage(), ')] ',
+                        progressbar.Bar(marker=progressbar.AnimatedMarker(
+                            fill='\N{FULL BLOCK}',
+                            fill_wrap=Fore.GREEN + '{}' + Fore.RESET,
+                            marker_wrap=Style.BRIGHT + '{}' + Style.RESET_ALL,
+                        )),
+                        ' [', progressbar.ETA(), ', ', progressbar.Timer(), ']',
+                    ]
+                else:
+                    widgets = [
+                        ' [Epoch ', progressbar.Counter(), ' of ', str(self.total_epochs),
+                        ' (', progressbar.Percentage(), ')] ',
+                        progressbar.Bar(marker=progressbar.AnimatedMarker(
+                            fill='\N{FULL BLOCK}',
+                        )),
+                        ' [', progressbar.ETA(), ', ', progressbar.Timer(), ']',
+                    ]
+                with progressbar.ProgressBar(
+                         maxval=self.total_epochs, redirect_stdout=False, redirect_stderr=False,
+                         widgets=widgets
+                     ) as pbar:
+                    EVALS = ceil(self.total_epochs / jobs)
+                    for i in range(EVALS):
+                        # Correct the number of epochs to be processed for the last
+                        # iteration (should not exceed self.total_epochs in total)
+                        n_rest = (i + 1) * jobs - self.total_epochs
+                        current_jobs = jobs - n_rest if n_rest > 0 else jobs
 
-                    for j, val in enumerate(f_val):
-                        # Use human-friendly indexes here (starting from 1)
-                        current = i * jobs + j + 1
-                        val['current_epoch'] = current
-                        val['is_initial_point'] = current <= INITIAL_POINTS
-                        logger.debug(f"Optimizer epoch evaluated: {val}")
+                        asked = self.opt.ask(n_points=current_jobs)
+                        f_val = self.run_optimizer_parallel(parallel, asked, i)
+                        self.opt.tell(asked, [v['loss'] for v in f_val])
+                        self.fix_optimizer_models_list()
 
-                        is_best = self.is_best_loss(val, self.current_best_loss)
-                        # This value is assigned here and not in the optimization method
-                        # to keep proper order in the list of results. That's because
-                        # evaluations can take different time. Here they are aligned in the
-                        # order they will be shown to the user.
-                        val['is_best'] = is_best
+                        # Calculate progressbar outputs
+                        for j, val in enumerate(f_val):
+                            # Use human-friendly indexes here (starting from 1)
+                            current = i * jobs + j + 1
+                            val['current_epoch'] = current
+                            val['is_initial_point'] = current <= INITIAL_POINTS
 
-                        self.print_results(val)
+                            logger.debug(f"Optimizer epoch evaluated: {val}")
+
+                            is_best = self.is_best_loss(val, self.current_best_loss)
+                            # This value is assigned here and not in the optimization method
+                            # to keep proper order in the list of results. That's because
+                            # evaluations can take different time. Here they are aligned in the
+                            # order they will be shown to the user.
+                            val['is_best'] = is_best
+                            self.print_results(val)
+
+                            if is_best:
+                                self.current_best_loss = val['loss']
+                            self.trials.append(val)
+
+                            # Save results after each best epoch and every 100 epochs
+                            if is_best or current % 100 == 0:
+                                self.save_trials()
+
+                            pbar.update(current)
 
-                        if is_best:
-                            self.current_best_loss = val['loss']
-                        self.trials.append(val)
-                        # Save results after each best epoch and every 100 epochs
-                        if is_best or current % 100 == 0:
-                            self.save_trials()
         except KeyboardInterrupt:
             print('User interrupted..')
 

freqtrade/optimize/optimize_reports.py

@@ -1,9 +1,38 @@
+import logging
 from datetime import timedelta
+from pathlib import Path
 from typing import Dict
 
 from pandas import DataFrame
 from tabulate import tabulate
 
+from freqtrade.misc import file_dump_json
+
+logger = logging.getLogger(__name__)
+
+
+def store_backtest_result(recordfilename: Path, all_results: Dict[str, DataFrame]) -> None:
+    """
+    Stores backtest results to file (one file per strategy)
+    :param recordfilename: Destination filename
+    :param all_results: Dict of Dataframes, one results dataframe per strategy
+    """
+    for strategy, results in all_results.items():
+        records = [(t.pair, t.profit_percent, t.open_time.timestamp(),
+                    t.close_time.timestamp(), t.open_index - 1, t.trade_duration,
+                    t.open_rate, t.close_rate, t.open_at_end, t.sell_reason.value)
+                   for index, t in results.iterrows()]
+
+        if records:
+            filename = recordfilename
+            if len(all_results) > 1:
+                # Inject strategy to filename
+                filename = Path.joinpath(
+                    recordfilename.parent,
+                    f'{recordfilename.stem}-{strategy}').with_suffix(recordfilename.suffix)
+            logger.info(f'Dumping backtest results to {filename}')
+            file_dump_json(filename, records)
+
 
 def generate_text_table(data: Dict[str, Dict], stake_currency: str, max_open_trades: int,
                         results: DataFrame, skip_nan: bool = False) -> str:
@@ -69,12 +98,12 @@ def generate_text_table(data: Dict[str, Dict], stake_currency: str, max_open_tra
                     floatfmt=floatfmt, tablefmt="orgtbl", stralign="right")  # type: ignore
 
 
-def generate_text_table_sell_reason(
-    data: Dict[str, Dict], stake_currency: str, max_open_trades: int, results: DataFrame
-) -> str:
+def generate_text_table_sell_reason(stake_currency: str, max_open_trades: int,
+                                    results: DataFrame) -> str:
     """
     Generate small table outlining Backtest results
-    :param data: Dict of <pair: dataframe> containing data that was used during backtesting.
+    :param stake_currency: Stakecurrency used
+    :param max_open_trades: Max_open_trades parameter
     :param results: Dataframe containing the backtest results
     :return: pretty printed table with tabulate as string
     """
@@ -173,3 +202,43 @@ def generate_edge_table(results: dict) -> str:
     # Ignore type as floatfmt does allow tuples but mypy does not know that
     return tabulate(tabular_data, headers=headers,
                     floatfmt=floatfmt, tablefmt="orgtbl", stralign="right")  # type: ignore
+
+
+def show_backtest_results(config: Dict, btdata: Dict[str, DataFrame],
+                          all_results: Dict[str, DataFrame]):
+    for strategy, results in all_results.items():
+
+        print(f"Result for strategy {strategy}")
+        table = generate_text_table(btdata, stake_currency=config['stake_currency'],
+                                    max_open_trades=config['max_open_trades'],
+                                    results=results)
+        if isinstance(table, str):
+            print(' BACKTESTING REPORT '.center(len(table.splitlines()[0]), '='))
+        print(table)
+
+        table = generate_text_table_sell_reason(stake_currency=config['stake_currency'],
+                                                max_open_trades=config['max_open_trades'],
+                                                results=results)
+        if isinstance(table, str):
+            print(' SELL REASON STATS '.center(len(table.splitlines()[0]), '='))
+        print(table)
+
+        table = generate_text_table(btdata,
+                                    stake_currency=config['stake_currency'],
+                                    max_open_trades=config['max_open_trades'],
+                                    results=results.loc[results.open_at_end], skip_nan=True)
+        if isinstance(table, str):
+            print(' LEFT OPEN TRADES REPORT '.center(len(table.splitlines()[0]), '='))
+        print(table)
+        if isinstance(table, str):
+            print('=' * len(table.splitlines()[0]))
+        print()
+    if len(all_results) > 1:
+        # Print Strategy summary table
+        table = generate_text_table_strategy(config['stake_currency'],
+                                             config['max_open_trades'],
+                                             all_results=all_results)
+        print(' STRATEGY SUMMARY '.center(len(table.splitlines()[0]), '='))
+        print(table)
+        print('=' * len(table.splitlines()[0]))
+        print('\nFor more details, please look at the detail tables above')

freqtrade/pairlist/IPairList.py

@@ -9,6 +9,8 @@ from abc import ABC, abstractmethod, abstractproperty
 from copy import deepcopy
 from typing import Any, Dict, List
 
+from cachetools import TTLCache, cached
+
 from freqtrade.exchange import market_is_active
 
 logger = logging.getLogger(__name__)
@@ -31,6 +33,9 @@ class IPairList(ABC):
         self._config = config
         self._pairlistconfig = pairlistconfig
         self._pairlist_pos = pairlist_pos
+        self.refresh_period = self._pairlistconfig.get('refresh_period', 1800)
+        self._last_refresh = 0
+        self._log_cache = TTLCache(maxsize=1024, ttl=self.refresh_period)
 
     @property
     def name(self) -> str:
@@ -40,6 +45,24 @@ class IPairList(ABC):
         """
         return self.__class__.__name__
 
+    def log_on_refresh(self, logmethod, message: str) -> None:
+        """
+        Logs message - not more often than "refresh_period" to avoid log spamming
+        Logs the log-message as debug as well to simplify debugging.
+        :param logmethod: Function that'll be called. Most likely `logger.info`.
+        :param message: String containing the message to be sent to the function.
+        :return: None.
+        """
+
+        @cached(cache=self._log_cache)
+        def _log_on_refresh(message: str):
+            logmethod(message)
+
+        # Log as debug first
+        logger.debug(message)
+        # Call hidden function.
+        _log_on_refresh(message)
+
     @abstractproperty
     def needstickers(self) -> bool:
         """

freqtrade/pairlist/PrecisionFilter.py

@@ -39,8 +39,9 @@ class PrecisionFilter(IPairList):
         stop_gap_price = self._exchange.price_to_precision(ticker["symbol"], stop_price * 0.99)
         logger.debug(f"{ticker['symbol']} - {sp} : {stop_gap_price}")
         if sp <= stop_gap_price:
-            logger.info(f"Removed {ticker['symbol']} from whitelist, "
-                        f"because stop price {sp} would be <= stop limit {stop_gap_price}")
+            self.log_on_refresh(logger.info,
+                                f"Removed {ticker['symbol']} from whitelist, "
+                                f"because stop price {sp} would be <= stop limit {stop_gap_price}")
             return False
         return True
 

freqtrade/pairlist/PriceFilter.py

@@ -35,16 +35,14 @@ class PriceFilter(IPairList):
         """
         Check if if one price-step (pip) is > than a certain barrier.
         :param ticker: ticker dict as returned from ccxt.load_markets()
-        :param precision: Precision
         :return: True if the pair can stay, false if it should be removed
         """
-        precision = self._exchange.markets[ticker['symbol']]['precision']['price']
-
-        compare = ticker['last'] + 1 / pow(10, precision)
+        compare = ticker['last'] + self._exchange.price_get_one_pip(ticker['symbol'],
+                                                                    ticker['last'])
         changeperc = (compare - ticker['last']) / ticker['last']
         if changeperc > self._low_price_ratio:
-            logger.info(f"Removed {ticker['symbol']} from whitelist, "
-                        f"because 1 unit is {changeperc * 100:.3f}%")
+            self.log_on_refresh(logger.info, f"Removed {ticker['symbol']} from whitelist, "
+                                             f"because 1 unit is {changeperc * 100:.3f}%")
             return False
         return True
 

freqtrade/pairlist/SpreadFilter.py

@@ -49,9 +49,9 @@ class SpreadFilter(IPairList):
             if 'bid' in ticker and 'ask' in ticker:
                 spread = 1 - ticker['bid'] / ticker['ask']
                 if not ticker or spread > self._max_spread_ratio:
-                    logger.info(f"Removed {ticker['symbol']} from whitelist, "
-                                f"because spread {spread * 100:.3f}% >"
-                                f"{self._max_spread_ratio * 100}%")
+                    self.log_on_refresh(logger.info, f"Removed {ticker['symbol']} from whitelist, "
+                                                     f"because spread {spread * 100:.3f}% >"
+                                                     f"{self._max_spread_ratio * 100}%")
                     pairlist.remove(p)
             else:
                 pairlist.remove(p)

freqtrade/pairlist/VolumePairList.py

@@ -39,7 +39,6 @@ class VolumePairList(IPairList):
         if not self._validate_keys(self._sort_key):
             raise OperationalException(
                 f'key {self._sort_key} not in {SORT_VALUES}')
-        self._last_refresh = 0
 
     @property
     def needstickers(self) -> bool:
@@ -68,16 +67,18 @@ class VolumePairList(IPairList):
         :return: new whitelist
         """
         # Generate dynamic whitelist
-        if self._last_refresh + self.refresh_period < datetime.now().timestamp():
+        # Must always run if this pairlist is not the first in the list.
+        if (self._pairlist_pos != 0 or
+                (self._last_refresh + self.refresh_period < datetime.now().timestamp())):
+
             self._last_refresh = int(datetime.now().timestamp())
-            return self._gen_pair_whitelist(pairlist,
-                                            tickers,
-                                            self._config['stake_currency'],
-                                            self._sort_key,
-                                            self._min_value
-                                            )
+            pairs = self._gen_pair_whitelist(pairlist, tickers,
+                                             self._config['stake_currency'],
+                                             self._sort_key, self._min_value)
         else:
-            return pairlist
+            pairs = pairlist
+        self.log_on_refresh(logger.info, f"Searching {self._number_pairs} pairs: {pairs}")
+        return pairs
 
     def _gen_pair_whitelist(self, pairlist: List[str], tickers: Dict,
                             base_currency: str, key: str, min_val: int) -> List[str]:
@@ -88,7 +89,6 @@ class VolumePairList(IPairList):
         :param tickers: Tickers (from exchange.get_tickers()).
         :return: List of pairs
         """
-
         if self._pairlist_pos == 0:
             # If VolumePairList is the first in the list, use fresh pairlist
             # Check if pair quote currency equals to the stake currency.
@@ -109,6 +109,5 @@ class VolumePairList(IPairList):
         pairs = self._verify_blacklist(pairs, aswarning=False)
         # Limit to X number of pairs
         pairs = pairs[:self._number_pairs]
-        logger.info(f"Searching {self._number_pairs} pairs: {pairs}")
 
         return pairs

freqtrade/persistence.py

@@ -86,7 +86,7 @@ def check_migrate(engine) -> None:
         logger.debug(f'trying {table_back_name}')
 
     # Check for latest column
-    if not has_column(cols, 'open_trade_price'):
+    if not has_column(cols, 'close_profit_abs'):
         logger.info(f'Running database migration - backup available as {table_back_name}')
 
         fee_open = get_column_def(cols, 'fee_open', 'fee')
@@ -106,6 +106,9 @@ def check_migrate(engine) -> None:
         ticker_interval = get_column_def(cols, 'ticker_interval', 'null')
         open_trade_price = get_column_def(cols, 'open_trade_price',
                                           f'amount * open_rate * (1 + {fee_open})')
+        close_profit_abs = get_column_def(
+            cols, 'close_profit_abs',
+            f"(amount * close_rate * (1 - {fee_close})) - {open_trade_price}")
 
         # Schema migration necessary
         engine.execute(f"alter table trades rename to {table_back_name}")
@@ -123,7 +126,7 @@ def check_migrate(engine) -> None:
                 stop_loss, stop_loss_pct, initial_stop_loss, initial_stop_loss_pct,
                 stoploss_order_id, stoploss_last_update,
                 max_rate, min_rate, sell_reason, strategy,
-                ticker_interval, open_trade_price
+                ticker_interval, open_trade_price, close_profit_abs
                 )
             select id, lower(exchange),
                 case
@@ -143,7 +146,7 @@ def check_migrate(engine) -> None:
                 {stoploss_order_id} stoploss_order_id, {stoploss_last_update} stoploss_last_update,
                 {max_rate} max_rate, {min_rate} min_rate, {sell_reason} sell_reason,
                 {strategy} strategy, {ticker_interval} ticker_interval,
-                {open_trade_price} open_trade_price
+                {open_trade_price} open_trade_price, {close_profit_abs} close_profit_abs
                 from {table_back_name}
              """)
 
@@ -185,11 +188,12 @@ class Trade(_DECL_BASE):
     fee_close = Column(Float, nullable=False, default=0.0)
     open_rate = Column(Float)
     open_rate_requested = Column(Float)
-    # open_trade_price - calcuated via _calc_open_trade_price
+    # open_trade_price - calculated via _calc_open_trade_price
     open_trade_price = Column(Float)
     close_rate = Column(Float)
     close_rate_requested = Column(Float)
     close_profit = Column(Float)
+    close_profit_abs = Column(Float)
     stake_amount = Column(Float, nullable=False)
     amount = Column(Float)
     open_date = Column(DateTime, nullable=False, default=datetime.utcnow)
@@ -229,6 +233,9 @@ class Trade(_DECL_BASE):
         return {
             'trade_id': self.id,
             'pair': self.pair,
+            'is_open': self.is_open,
+            'fee_open': self.fee_open,
+            'fee_close': self.fee_close,
             'open_date_hum': arrow.get(self.open_date).humanize(),
             'open_date': self.open_date.strftime("%Y-%m-%d %H:%M:%S"),
             'close_date_hum': (arrow.get(self.close_date).humanize()
@@ -236,14 +243,24 @@ class Trade(_DECL_BASE):
             'close_date': (self.close_date.strftime("%Y-%m-%d %H:%M:%S")
                            if self.close_date else None),
             'open_rate': self.open_rate,
+            'open_rate_requested': self.open_rate_requested,
+            'open_trade_price': self.open_trade_price,
             'close_rate': self.close_rate,
+            'close_rate_requested': self.close_rate_requested,
             'amount': round(self.amount, 8),
             'stake_amount': round(self.stake_amount, 8),
+            'close_profit': self.close_profit,
+            'sell_reason': self.sell_reason,
             'stop_loss': self.stop_loss,
             'stop_loss_pct': (self.stop_loss_pct * 100) if self.stop_loss_pct else None,
             'initial_stop_loss': self.initial_stop_loss,
             'initial_stop_loss_pct': (self.initial_stop_loss_pct * 100
                                       if self.initial_stop_loss_pct else None),
+            'min_rate': self.min_rate,
+            'max_rate': self.max_rate,
+            'strategy': self.strategy,
+            'ticker_interval': self.ticker_interval,
+            'open_order_id': self.open_order_id,
         }
 
     def adjust_min_max_rates(self, current_price: float) -> None:
@@ -311,7 +328,7 @@ class Trade(_DECL_BASE):
         if order_type in ('market', 'limit') and order['side'] == 'buy':
             # Update open rate and actual amount
             self.open_rate = Decimal(order['price'])
-            self.amount = Decimal(order['amount'])
+            self.amount = Decimal(order.get('filled', order['amount']))
             self.recalc_open_trade_price()
             logger.info('%s_BUY has been fulfilled for %s.', order_type.upper(), self)
             self.open_order_id = None
@@ -334,6 +351,7 @@ class Trade(_DECL_BASE):
         """
         self.close_rate = Decimal(rate)
         self.close_profit = self.calc_profit_ratio()
+        self.close_profit_abs = self.calc_profit()
         self.close_date = datetime.utcnow()
         self.is_open = False
         self.open_order_id = None

freqtrade/plot/plotting.py

@@ -10,6 +10,7 @@ from freqtrade.data.btanalysis import (calculate_max_drawdown,
                                        create_cum_profit,
                                        extract_trades_of_period, load_trades)
 from freqtrade.data.converter import trim_dataframe
+from freqtrade.exchange import timeframe_to_prev_date
 from freqtrade.data.history import load_data
 from freqtrade.misc import pair_to_filename
 from freqtrade.resolvers import StrategyResolver
@@ -48,11 +49,21 @@ def init_plotscript(config):
         data_format=config.get('dataformat_ohlcv', 'json'),
     )
 
-    trades = load_trades(config['trade_source'],
-                         db_url=config.get('db_url'),
-                         exportfilename=config.get('exportfilename'),
-                         )
+    no_trades = False
+    if config.get('no_trades', False):
+        no_trades = True
+    elif not config['exportfilename'].is_file() and config['trade_source'] == 'file':
+        logger.warning("Backtest file is missing skipping trades.")
+        no_trades = True
+
+    trades = load_trades(
+        config['trade_source'],
+        db_url=config.get('db_url'),
+        exportfilename=config.get('exportfilename'),
+        no_trades=no_trades
+    )
     trades = trim_dataframe(trades, timerange, 'open_time')
+
     return {"ohlcv": data,
             "trades": trades,
             "pairs": pairs,
@@ -112,7 +123,8 @@ def add_profit(fig, row, data: pd.DataFrame, column: str, name: str) -> make_sub
     return fig
 
 
-def add_max_drawdown(fig, row, trades: pd.DataFrame, df_comb: pd.DataFrame) -> make_subplots:
+def add_max_drawdown(fig, row, trades: pd.DataFrame, df_comb: pd.DataFrame,
+                     timeframe: str) -> make_subplots:
     """
     Add scatter points indicating max drawdown
     """
@@ -122,12 +134,12 @@ def add_max_drawdown(fig, row, trades: pd.DataFrame, df_comb: pd.DataFrame) -> m
         drawdown = go.Scatter(
             x=[highdate, lowdate],
             y=[
-                df_comb.loc[highdate, 'cum_profit'],
-                df_comb.loc[lowdate, 'cum_profit'],
+                df_comb.loc[timeframe_to_prev_date(timeframe, highdate), 'cum_profit'],
+                df_comb.loc[timeframe_to_prev_date(timeframe, lowdate), 'cum_profit'],
             ],
             mode='markers',
-            name=f"Max drawdown {max_drawdown:.2f}%",
-            text=f"Max drawdown {max_drawdown:.2f}%",
+            name=f"Max drawdown {max_drawdown * 100:.2f}%",
+            text=f"Max drawdown {max_drawdown * 100:.2f}%",
             marker=dict(
                 symbol='square-open',
                 size=9,
@@ -373,6 +385,9 @@ def generate_profit_graph(pairs: str, data: Dict[str, pd.DataFrame],
     # Combine close-values for all pairs, rename columns to "pair"
     df_comb = combine_dataframes_with_mean(data, "close")
 
+    # Trim trades to available OHLCV data
+    trades = extract_trades_of_period(df_comb, trades, date_index=True)
+
     # Add combined cumulative profit
     df_comb = create_cum_profit(df_comb, trades, 'cum_profit', timeframe)
 
@@ -395,7 +410,7 @@ def generate_profit_graph(pairs: str, data: Dict[str, pd.DataFrame],
 
     fig.add_trace(avgclose, 1, 1)
     fig = add_profit(fig, 2, df_comb, 'cum_profit', 'Profit')
-    fig = add_max_drawdown(fig, 2, trades, df_comb)
+    fig = add_max_drawdown(fig, 2, trades, df_comb, timeframe)
 
     for pair in pairs:
         profit_col = f'cum_profit_{pair}'

freqtrade/rpc/api_server.py

@@ -173,7 +173,8 @@ class ApiServer(RPC):
                               view_func=self._show_config, methods=['GET'])
         self.app.add_url_rule(f'{BASE_URI}/ping', 'ping',
                               view_func=self._ping, methods=['GET'])
-
+        self.app.add_url_rule(f'{BASE_URI}/trades', 'trades',
+                              view_func=self._trades, methods=['GET'])
         # Combined actions and infos
         self.app.add_url_rule(f'{BASE_URI}/blacklist', 'blacklist', view_func=self._blacklist,
                               methods=['GET', 'POST'])
@@ -358,6 +359,18 @@ class ApiServer(RPC):
                                     self._config.get('fiat_display_currency', ''))
         return self.rest_dump(results)
 
+    @require_login
+    @rpc_catch_errors
+    def _trades(self):
+        """
+        Handler for /trades.
+
+        Returns the X last trades in json format
+        """
+        limit = int(request.args.get('limit', 0))
+        results = self._rpc_trade_history(limit)
+        return self.rest_dump(results)
+
     @require_login
     @rpc_catch_errors
     def _whitelist(self):

freqtrade/rpc/rpc.py

@@ -197,7 +197,7 @@ class RPC:
                 Trade.close_date >= profitday,
                 Trade.close_date < (profitday + timedelta(days=1))
             ]).order_by(Trade.close_date).all()
-            curdayprofit = sum(trade.calc_profit() for trade in trades)
+            curdayprofit = sum(trade.close_profit_abs for trade in trades)
             profit_days[profitday] = {
                 'amount': f'{curdayprofit:.8f}',
                 'trades': len(trades)
@@ -226,6 +226,20 @@ class RPC:
             for key, value in profit_days.items()
         ]
 
+    def _rpc_trade_history(self, limit: int) -> Dict:
+        """ Returns the X last trades """
+        if limit > 0:
+            trades = Trade.get_trades().order_by(Trade.id.desc()).limit(limit)
+        else:
+            trades = Trade.get_trades().order_by(Trade.id.desc()).all()
+
+        output = [trade.to_json() for trade in trades]
+
+        return {
+            "trades": output,
+            "trades_count": len(output)
+        }
+
     def _rpc_trade_statistics(
             self, stake_currency: str, fiat_display_currency: str) -> Dict[str, Any]:
         """ Returns cumulative profit statistics """
@@ -246,8 +260,8 @@ class RPC:
                 durations.append((trade.close_date - trade.open_date).total_seconds())
 
             if not trade.is_open:
-                profit_ratio = trade.calc_profit_ratio()
-                profit_closed_coin.append(trade.calc_profit())
+                profit_ratio = trade.close_profit
+                profit_closed_coin.append(trade.close_profit_abs)
                 profit_closed_ratio.append(profit_ratio)
             else:
                 # Get current rate

freqtrade/rpc/telegram.py

@@ -172,7 +172,8 @@ class Telegram(RPC):
                             ' / {profit_fiat:.3f} {fiat_currency})`').format(**msg)
 
         elif msg['type'] == RPCMessageType.SELL_CANCEL_NOTIFICATION:
-            message = "*{exchange}:* Cancelling Open Sell Order for {pair}".format(**msg)
+            message = ("*{exchange}:* Cancelling Open Sell Order "
+                       "for {pair}. Reason: {reason}").format(**msg)
 
         elif msg['type'] == RPCMessageType.STATUS_NOTIFICATION:
             message = '*Status:* `{status}`'.format(**msg)

freqtrade/strategy/interface.py

@@ -278,8 +278,25 @@ class IStrategy(ABC):
 
         return dataframe
 
-    def get_signal(self, pair: str, interval: str,
-                   dataframe: DataFrame) -> Tuple[bool, bool]:
+    @staticmethod
+    def preserve_df(dataframe: DataFrame) -> Tuple[int, float, datetime]:
+        """ keep some data for dataframes """
+        return len(dataframe), dataframe["close"].iloc[-1], dataframe["date"].iloc[-1]
+
+    @staticmethod
+    def assert_df(dataframe: DataFrame, df_len: int, df_close: float, df_date: datetime):
+        """ make sure data is unmodified """
+        message = ""
+        if df_len != len(dataframe):
+            message = "length"
+        elif df_close != dataframe["close"].iloc[-1]:
+            message = "last close price"
+        elif df_date != dataframe["date"].iloc[-1]:
+            message = "last date"
+        if message:
+            raise StrategyError(f"Dataframe returned from strategy has mismatching {message}.")
+
+    def get_signal(self, pair: str, interval: str, dataframe: DataFrame) -> Tuple[bool, bool]:
         """
         Calculates current signal based several technical analysis indicators
         :param pair: pair in format ANT/BTC
@@ -291,10 +308,13 @@ class IStrategy(ABC):
             logger.warning('Empty candle (OHLCV) data for pair %s', pair)
             return False, False
 
+        latest_date = dataframe['date'].max()
         try:
+            df_len, df_close, df_date = self.preserve_df(dataframe)
             dataframe = strategy_safe_wrapper(
                 self._analyze_ticker_internal, message=""
                 )(dataframe, {'pair': pair})
+            self.assert_df(dataframe, df_len, df_close, df_date)
         except StrategyError as error:
             logger.warning(f"Unable to analyze candle (OHLCV) data for pair {pair}: {error}")
 
@@ -304,7 +324,7 @@ class IStrategy(ABC):
             logger.warning('Empty dataframe for pair %s', pair)
             return False, False
 
-        latest = dataframe.iloc[-1]
+        latest = dataframe.loc[dataframe['date'] == latest_date].iloc[-1]
 
         # Check if dataframe is out of date
         signal_date = arrow.get(latest['date'])
@@ -473,8 +493,11 @@ class IStrategy(ABC):
         """
         Creates a dataframe and populates indicators for given candle (OHLCV) data
         Used by optimize operations only, not during dry / live runs.
+        Using .copy() to get a fresh copy of the dataframe for every strategy run.
+        Has positive effects on memory usage for whatever reason - also when
+        using only one strategy.
         """
-        return {pair: self.advise_indicators(pair_data, {'pair': pair})
+        return {pair: self.advise_indicators(pair_data.copy(), {'pair': pair})
                 for pair, pair_data in data.items()}
 
     def advise_indicators(self, dataframe: DataFrame, metadata: dict) -> DataFrame:

requirements-common.txt

@@ -1,18 +1,18 @@
 # requirements without requirements installable via conda
 # mainly used for Raspberry pi installs
-ccxt==1.23.81
-SQLAlchemy==1.3.13
-python-telegram-bot==12.4.2
+ccxt==1.26.32
+SQLAlchemy==1.3.16
+python-telegram-bot==12.6.1
 arrow==0.15.5
-cachetools==4.0.0
+cachetools==4.1.0
 requests==2.23.0
 urllib3==1.25.8
 wrapt==1.12.1
 jsonschema==3.2.0
 TA-Lib==0.4.17
-tabulate==0.8.6
+tabulate==0.8.7
 pycoingecko==1.2.0
-jinja2==2.11.1
+jinja2==2.11.2
 
 # find first, C search in arrays
 py_find_1st==1.1.4
@@ -24,10 +24,10 @@ python-rapidjson==0.9.1
 sdnotify==0.3.2
 
 # Api server
-flask==1.1.1
+flask==1.1.2
 
 # Support for colorized terminal output
 colorama==0.4.3
 # Building config files interactively
 questionary==1.5.1
-prompt-toolkit==3.0.4
+prompt-toolkit==3.0.5

requirements-dev.txt

@@ -3,15 +3,15 @@
 -r requirements-plot.txt
 -r requirements-hyperopt.txt
 
-coveralls==1.11.1
+coveralls==2.0.0
 flake8==3.7.9
 flake8-type-annotations==0.1.0
-flake8-tidy-imports==4.0.0
-mypy==0.761
-pytest==5.3.5
+flake8-tidy-imports==4.1.0
+mypy==0.770
+pytest==5.4.1
 pytest-asyncio==0.10.0
 pytest-cov==2.8.1
-pytest-mock==2.0.0
+pytest-mock==3.0.0
 pytest-random-order==1.0.4
 
 # Convert jupyter notebooks to markdown documents

requirements-hyperopt.txt

@@ -7,3 +7,4 @@ scikit-learn==0.22.2.post1
 scikit-optimize==0.7.4
 filelock==3.0.12
 joblib==0.14.1
+progressbar2==3.50.1

requirements-plot.txt

@@ -1,5 +1,5 @@
 # Include all requirements to run the bot.
 -r requirements.txt
 
-plotly==4.5.3
+plotly==4.6.0
 

requirements.txt

@@ -1,5 +1,5 @@
 # Load common requirements
 -r requirements-common.txt
 
-numpy==1.18.1
-pandas==1.0.1
+numpy==1.18.2
+pandas==1.0.3

scripts/rest_client.py

@@ -156,6 +156,14 @@ class FtRestClient():
         """
         return self._get("show_config")
 
+    def trades(self, limit=None):
+        """Return trades history.
+
+        :param limit: Limits trades to the X last trades. No limit to get all the trades.
+        :return: json object
+        """
+        return self._get("trades", params={"limit": limit} if limit else 0)
+
     def whitelist(self):
         """Show the current whitelist.
 

setup.py

@@ -24,6 +24,7 @@ hyperopt = [
     'scikit-optimize',
     'filelock',
     'joblib',
+    'progressbar2',
     ]
 
 develop = [

tests/conftest.py

@@ -166,6 +166,52 @@ def patch_get_signal(freqtrade: FreqtradeBot, value=(True, False)) -> None:
     freqtrade.exchange.refresh_latest_ohlcv = lambda p: None
 
 
+def create_mock_trades(fee):
+    """
+    Create some fake trades ...
+    """
+    # Simulate dry_run entries
+    trade = Trade(
+        pair='ETH/BTC',
+        stake_amount=0.001,
+        amount=123.0,
+        fee_open=fee.return_value,
+        fee_close=fee.return_value,
+        open_rate=0.123,
+        exchange='bittrex',
+        open_order_id='dry_run_buy_12345'
+    )
+    Trade.session.add(trade)
+
+    trade = Trade(
+        pair='ETC/BTC',
+        stake_amount=0.001,
+        amount=123.0,
+        fee_open=fee.return_value,
+        fee_close=fee.return_value,
+        open_rate=0.123,
+        close_rate=0.128,
+        close_profit=0.005,
+        exchange='bittrex',
+        is_open=False,
+        open_order_id='dry_run_sell_12345'
+    )
+    Trade.session.add(trade)
+
+    # Simulate prod entry
+    trade = Trade(
+        pair='ETC/BTC',
+        stake_amount=0.001,
+        amount=123.0,
+        fee_open=fee.return_value,
+        fee_close=fee.return_value,
+        open_rate=0.123,
+        exchange='bittrex',
+        open_order_id='prod_buy_12345'
+    )
+    Trade.session.add(trade)
+
+
 @pytest.fixture(autouse=True)
 def patch_coingekko(mocker) -> None:
     """
@@ -712,6 +758,7 @@ def limit_buy_order():
         'datetime': arrow.utcnow().isoformat(),
         'price': 0.00001099,
         'amount': 90.99181073,
+        'filled': 90.99181073,
         'remaining': 0.0,
         'status': 'closed'
     }
@@ -727,6 +774,7 @@ def market_buy_order():
         'datetime': arrow.utcnow().isoformat(),
         'price': 0.00004099,
         'amount': 91.99181073,
+        'filled': 91.99181073,
         'remaining': 0.0,
         'status': 'closed'
     }
@@ -742,6 +790,7 @@ def market_sell_order():
         'datetime': arrow.utcnow().isoformat(),
         'price': 0.00004173,
         'amount': 91.99181073,
+        'filled': 91.99181073,
         'remaining': 0.0,
         'status': 'closed'
     }
@@ -757,6 +806,7 @@ def limit_buy_order_old():
         'datetime': str(arrow.utcnow().shift(minutes=-601).datetime),
         'price': 0.00001099,
         'amount': 90.99181073,
+        'filled': 0.0,
         'remaining': 90.99181073,
         'status': 'open'
     }
@@ -772,6 +822,7 @@ def limit_sell_order_old():
         'datetime': arrow.utcnow().shift(minutes=-601).isoformat(),
         'price': 0.00001099,
         'amount': 90.99181073,
+        'filled': 0.0,
         'remaining': 90.99181073,
         'status': 'open'
     }
@@ -787,6 +838,7 @@ def limit_buy_order_old_partial():
         'datetime': arrow.utcnow().shift(minutes=-601).isoformat(),
         'price': 0.00001099,
         'amount': 90.99181073,
+        'filled': 23.0,
         'remaining': 67.99181073,
         'status': 'open'
     }
@@ -810,6 +862,7 @@ def limit_sell_order():
         'datetime': arrow.utcnow().isoformat(),
         'price': 0.00001173,
         'amount': 90.99181073,
+        'filled': 90.99181073,
         'remaining': 0.0,
         'status': 'closed'
     }

tests/data/test_btanalysis.py

@@ -15,7 +15,7 @@ from freqtrade.data.btanalysis import (BT_DATA_COLUMNS,
                                        load_backtest_data, load_trades,
                                        load_trades_from_db)
 from freqtrade.data.history import load_data, load_pair_history
-from tests.test_persistence import create_mock_trades
+from tests.conftest import create_mock_trades
 
 
 def test_load_backtest_data(testdatadir):
@@ -105,6 +105,7 @@ def test_load_trades(default_conf, mocker):
     load_trades("DB",
                 db_url=default_conf.get('db_url'),
                 exportfilename=default_conf.get('exportfilename'),
+                no_trades=False
                 )
 
     assert db_mock.call_count == 1
@@ -115,11 +116,24 @@ def test_load_trades(default_conf, mocker):
     default_conf['exportfilename'] = Path("testfile.json")
     load_trades("file",
                 db_url=default_conf.get('db_url'),
-                exportfilename=default_conf.get('exportfilename'),)
+                exportfilename=default_conf.get('exportfilename'),
+                )
 
     assert db_mock.call_count == 0
     assert bt_mock.call_count == 1
 
+    db_mock.reset_mock()
+    bt_mock.reset_mock()
+    default_conf['exportfilename'] = "testfile.json"
+    load_trades("file",
+                db_url=default_conf.get('db_url'),
+                exportfilename=default_conf.get('exportfilename'),
+                no_trades=True
+                )
+
+    assert db_mock.call_count == 0
+    assert bt_mock.call_count == 0
+
 
 def test_combine_dataframes_with_mean(testdatadir):
     pairs = ["ETH/BTC", "ADA/BTC"]
@@ -177,3 +191,28 @@ def test_calculate_max_drawdown(testdatadir):
     assert low == Timestamp('2018-01-30 04:45:00', tz='UTC')
     with pytest.raises(ValueError, match='Trade dataframe empty.'):
         drawdown, h, low = calculate_max_drawdown(DataFrame())
+
+
+def test_calculate_max_drawdown2():
+    values = [0.011580, 0.010048, 0.011340, 0.012161, 0.010416, 0.010009, 0.020024,
+              -0.024662, -0.022350, 0.020496, -0.029859, -0.030511, 0.010041, 0.010872,
+              -0.025782, 0.010400, 0.012374, 0.012467, 0.114741, 0.010303, 0.010088,
+              -0.033961, 0.010680, 0.010886, -0.029274, 0.011178, 0.010693, 0.010711]
+
+    dates = [Arrow(2020, 1, 1).shift(days=i) for i in range(len(values))]
+    df = DataFrame(zip(values, dates), columns=['profit', 'open_time'])
+    # sort by profit and reset index
+    df = df.sort_values('profit').reset_index(drop=True)
+    df1 = df.copy()
+    drawdown, h, low = calculate_max_drawdown(df, date_col='open_time', value_col='profit')
+    # Ensure df has not been altered.
+    assert df.equals(df1)
+
+    assert isinstance(drawdown, float)
+    # High must be before low
+    assert h < low
+    assert drawdown == 0.091755
+
+    df = DataFrame(zip(values[:5], dates[:5]), columns=['profit', 'open_time'])
+    with pytest.raises(ValueError, match='No losing trade, therefore no drawdown.'):
+        calculate_max_drawdown(df, date_col='open_time', value_col='profit')

tests/edge/test_edge.py

@@ -292,8 +292,8 @@ def mocked_load_data(datadir, pairs=[], timeframe='0m',
 def test_edge_process_downloaded_data(mocker, edge_conf):
     freqtrade = get_patched_freqtradebot(mocker, edge_conf)
     mocker.patch('freqtrade.exchange.Exchange.get_fee', MagicMock(return_value=0.001))
-    mocker.patch('freqtrade.data.history.refresh_data', MagicMock())
-    mocker.patch('freqtrade.data.history.load_data', mocked_load_data)
+    mocker.patch('freqtrade.edge.edge_positioning.refresh_data', MagicMock())
+    mocker.patch('freqtrade.edge.edge_positioning.load_data', mocked_load_data)
     edge = Edge(edge_conf, freqtrade.exchange, freqtrade.strategy)
 
     assert edge.calculate()
@@ -304,8 +304,8 @@ def test_edge_process_downloaded_data(mocker, edge_conf):
 def test_edge_process_no_data(mocker, edge_conf, caplog):
     freqtrade = get_patched_freqtradebot(mocker, edge_conf)
     mocker.patch('freqtrade.exchange.Exchange.get_fee', MagicMock(return_value=0.001))
-    mocker.patch('freqtrade.data.history.refresh_data', MagicMock())
-    mocker.patch('freqtrade.data.history.load_data', MagicMock(return_value={}))
+    mocker.patch('freqtrade.edge.edge_positioning.refresh_data', MagicMock())
+    mocker.patch('freqtrade.edge.edge_positioning.load_data', MagicMock(return_value={}))
     edge = Edge(edge_conf, freqtrade.exchange, freqtrade.strategy)
 
     assert not edge.calculate()
@@ -317,8 +317,8 @@ def test_edge_process_no_data(mocker, edge_conf, caplog):
 def test_edge_process_no_trades(mocker, edge_conf, caplog):
     freqtrade = get_patched_freqtradebot(mocker, edge_conf)
     mocker.patch('freqtrade.exchange.Exchange.get_fee', MagicMock(return_value=0.001))
-    mocker.patch('freqtrade.data.history.refresh_data', MagicMock())
-    mocker.patch('freqtrade.data.history.load_data', mocked_load_data)
+    mocker.patch('freqtrade.edge.edge_positioning.refresh_data', MagicMock())
+    mocker.patch('freqtrade.edge.edge_positioning.load_data', mocked_load_data)
     # Return empty
     mocker.patch('freqtrade.edge.Edge._find_trades_for_stoploss_range', MagicMock(return_value=[]))
     edge = Edge(edge_conf, freqtrade.exchange, freqtrade.strategy)

tests/exchange/test_exchange.py

@@ -253,6 +253,32 @@ def test_price_to_precision(default_conf, mocker, price, precision_mode, precisi
     assert pytest.approx(exchange.price_to_precision(pair, price)) == expected
 
 
+@pytest.mark.parametrize("price,precision_mode,precision,expected", [
+    (2.34559, 2, 4, 0.0001),
+    (2.34559, 2, 5, 0.00001),
+    (2.34559, 2, 3, 0.001),
+    (2.9999, 2, 3, 0.001),
+    (200.0511, 2, 3, 0.001),
+    # Tests for Tick_size
+    (2.34559, 4, 0.0001, 0.0001),
+    (2.34559, 4, 0.00001, 0.00001),
+    (2.34559, 4, 0.0025, 0.0025),
+    (2.9909, 4, 0.0025, 0.0025),
+    (234.43, 4, 0.5, 0.5),
+    (234.43, 4, 0.0025, 0.0025),
+    (234.43, 4, 0.00013, 0.00013),
+
+])
+def test_price_get_one_pip(default_conf, mocker, price, precision_mode, precision, expected):
+    markets = PropertyMock(return_value={'ETH/BTC': {'precision': {'price': precision}}})
+    exchange = get_patched_exchange(mocker, default_conf, id="binance")
+    mocker.patch('freqtrade.exchange.Exchange.markets', markets)
+    mocker.patch('freqtrade.exchange.Exchange.precisionMode',
+                 PropertyMock(return_value=precision_mode))
+    pair = 'ETH/BTC'
+    assert pytest.approx(exchange.price_get_one_pip(pair, price)) == expected
+
+
 def test_set_sandbox(default_conf, mocker):
     """
     Test working scenario
@@ -1705,6 +1731,68 @@ def test_cancel_order_dry_run(default_conf, mocker, exchange_name):
     assert exchange.cancel_order(order_id='123', pair='TKN/BTC') == {}
 
 
+@pytest.mark.parametrize("exchange_name", EXCHANGES)
+@pytest.mark.parametrize("order,result", [
+    ({'status': 'closed', 'filled': 10}, False),
+    ({'status': 'closed', 'filled': 0.0}, True),
+    ({'status': 'canceled', 'filled': 0.0}, True),
+    ({'status': 'canceled', 'filled': 10.0}, False),
+    ({'status': 'unknown', 'filled': 10.0}, False),
+    ({'result': 'testest123'}, False),
+    ])
+def test_check_order_canceled_empty(mocker, default_conf, exchange_name, order, result):
+    exchange = get_patched_exchange(mocker, default_conf, id=exchange_name)
+    assert exchange.check_order_canceled_empty(order) == result
+
+
+@pytest.mark.parametrize("exchange_name", EXCHANGES)
+@pytest.mark.parametrize("order,result", [
+    ({'status': 'closed', 'amount': 10, 'fee': {}}, True),
+    ({'status': 'closed', 'amount': 0.0, 'fee': {}}, True),
+    ({'status': 'canceled', 'amount': 0.0, 'fee': {}}, True),
+    ({'status': 'canceled', 'amount': 10.0}, False),
+    ({'amount': 10.0, 'fee': {}}, False),
+    ({'result': 'testest123'}, False),
+    ('hello_world', False),
+])
+def test_is_cancel_order_result_suitable(mocker, default_conf, exchange_name, order, result):
+    exchange = get_patched_exchange(mocker, default_conf, id=exchange_name)
+    assert exchange.is_cancel_order_result_suitable(order) == result
+
+
+@pytest.mark.parametrize("exchange_name", EXCHANGES)
+@pytest.mark.parametrize("corder,call_corder,call_forder", [
+    ({'status': 'closed', 'amount': 10, 'fee': {}}, 1, 0),
+    ({'amount': 10, 'fee': {}}, 1, 1),
+])
+def test_cancel_order_with_result(default_conf, mocker, exchange_name, corder,
+                                  call_corder, call_forder):
+    default_conf['dry_run'] = False
+    api_mock = MagicMock()
+    api_mock.cancel_order = MagicMock(return_value=corder)
+    api_mock.fetch_order = MagicMock(return_value={})
+    exchange = get_patched_exchange(mocker, default_conf, api_mock, id=exchange_name)
+    res = exchange.cancel_order_with_result('1234', 'ETH/BTC', 1234)
+    assert isinstance(res, dict)
+    assert api_mock.cancel_order.call_count == call_corder
+    assert api_mock.fetch_order.call_count == call_forder
+
+
+@pytest.mark.parametrize("exchange_name", EXCHANGES)
+def test_cancel_order_with_result_error(default_conf, mocker, exchange_name, caplog):
+    default_conf['dry_run'] = False
+    api_mock = MagicMock()
+    api_mock.cancel_order = MagicMock(side_effect=ccxt.InvalidOrder("Did not find order"))
+    api_mock.fetch_order = MagicMock(side_effect=ccxt.InvalidOrder("Did not find order"))
+    exchange = get_patched_exchange(mocker, default_conf, api_mock, id=exchange_name)
+
+    res = exchange.cancel_order_with_result('1234', 'ETH/BTC', 1541)
+    assert isinstance(res, dict)
+    assert log_has("Could not cancel order 1234.", caplog)
+    assert log_has("Could not fetch cancelled order 1234.", caplog)
+    assert res['amount'] == 1541
+
+
 # Ensure that if not dry_run, we should call API
 @pytest.mark.parametrize("exchange_name", EXCHANGES)
 def test_cancel_order(default_conf, mocker, exchange_name):

tests/optimize/test_backtesting.py

@@ -331,8 +331,8 @@ def test_backtesting_start(default_conf, mocker, testdatadir, caplog) -> None:
 
     mocker.patch('freqtrade.data.history.get_timerange', get_timerange)
     patch_exchange(mocker)
-    mocker.patch('freqtrade.optimize.backtesting.Backtesting.backtest', MagicMock())
-    mocker.patch('freqtrade.optimize.backtesting.generate_text_table', MagicMock(return_value=1))
+    mocker.patch('freqtrade.optimize.backtesting.Backtesting.backtest')
+    mocker.patch('freqtrade.optimize.backtesting.show_backtest_results')
 
     default_conf['exchange']['pair_whitelist'] = ['UNITTEST/BTC']
     default_conf['ticker_interval'] = '1m'
@@ -361,8 +361,8 @@ def test_backtesting_start_no_data(default_conf, mocker, caplog, testdatadir) ->
                  MagicMock(return_value=pd.DataFrame()))
     mocker.patch('freqtrade.data.history.get_timerange', get_timerange)
     patch_exchange(mocker)
-    mocker.patch('freqtrade.optimize.backtesting.Backtesting.backtest', MagicMock())
-    mocker.patch('freqtrade.optimize.backtesting.generate_text_table', MagicMock(return_value=1))
+    mocker.patch('freqtrade.optimize.backtesting.Backtesting.backtest')
+    mocker.patch('freqtrade.optimize.backtesting.show_backtest_results')
 
     default_conf['exchange']['pair_whitelist'] = ['UNITTEST/BTC']
     default_conf['ticker_interval'] = "1m"
@@ -507,7 +507,6 @@ def test_backtest_only_sell(mocker, default_conf, testdatadir):
 
 def test_backtest_alternate_buy_sell(default_conf, fee, mocker, testdatadir):
     mocker.patch('freqtrade.exchange.Exchange.get_fee', fee)
-    mocker.patch('freqtrade.optimize.backtesting.file_dump_json', MagicMock())
     backtest_conf = _make_backtest_conf(mocker, conf=default_conf,
                                         pair='UNITTEST/BTC', datadir=testdatadir)
     default_conf['ticker_interval'] = '1m'
@@ -515,7 +514,6 @@ def test_backtest_alternate_buy_sell(default_conf, fee, mocker, testdatadir):
     backtesting.strategy.advise_buy = _trend_alternate  # Override
     backtesting.strategy.advise_sell = _trend_alternate  # Override
     results = backtesting.backtest(**backtest_conf)
-    backtesting._store_backtest_result("test_.json", results)
     # 200 candles in backtest data
     # won't buy on first (shifted by 1)
     # 100 buys signals
@@ -586,84 +584,12 @@ def test_backtest_multi_pair(default_conf, fee, mocker, tres, pair, testdatadir)
     assert len(evaluate_result_multi(results, '5m', 1)) == 0
 
 
-def test_backtest_record(default_conf, fee, mocker):
-    names = []
-    records = []
-    patch_exchange(mocker)
-    mocker.patch('freqtrade.exchange.Exchange.get_fee', fee)
-    mocker.patch(
-        'freqtrade.optimize.backtesting.file_dump_json',
-        new=lambda n, r: (names.append(n), records.append(r))
-    )
-
-    backtesting = Backtesting(default_conf)
-    results = pd.DataFrame({"pair": ["UNITTEST/BTC", "UNITTEST/BTC",
-                                     "UNITTEST/BTC", "UNITTEST/BTC"],
-                            "profit_percent": [0.003312, 0.010801, 0.013803, 0.002780],
-                            "profit_abs": [0.000003, 0.000011, 0.000014, 0.000003],
-                            "open_time": [Arrow(2017, 11, 14, 19, 32, 00).datetime,
-                                          Arrow(2017, 11, 14, 21, 36, 00).datetime,
-                                          Arrow(2017, 11, 14, 22, 12, 00).datetime,
-                                          Arrow(2017, 11, 14, 22, 44, 00).datetime],
-                            "close_time": [Arrow(2017, 11, 14, 21, 35, 00).datetime,
-                                           Arrow(2017, 11, 14, 22, 10, 00).datetime,
-                                           Arrow(2017, 11, 14, 22, 43, 00).datetime,
-                                           Arrow(2017, 11, 14, 22, 58, 00).datetime],
-                            "open_rate": [0.002543, 0.003003, 0.003089, 0.003214],
-                            "close_rate": [0.002546, 0.003014, 0.003103, 0.003217],
-                            "open_index": [1, 119, 153, 185],
-                            "close_index": [118, 151, 184, 199],
-                            "trade_duration": [123, 34, 31, 14],
-                            "open_at_end": [False, False, False, True],
-                            "sell_reason": [SellType.ROI, SellType.STOP_LOSS,
-                                            SellType.ROI, SellType.FORCE_SELL]
-                            })
-    backtesting._store_backtest_result("backtest-result.json", results)
-    assert len(results) == 4
-    # Assert file_dump_json was only called once
-    assert names == ['backtest-result.json']
-    records = records[0]
-    # Ensure records are of correct type
-    assert len(records) == 4
-
-    # reset test to test with strategy name
-    names = []
-    records = []
-    backtesting._store_backtest_result(Path("backtest-result.json"), results, "DefStrat")
-    assert len(results) == 4
-    # Assert file_dump_json was only called once
-    assert names == [Path('backtest-result-DefStrat.json')]
-    records = records[0]
-    # Ensure records are of correct type
-    assert len(records) == 4
-
-    # ('UNITTEST/BTC', 0.00331158, '1510684320', '1510691700', 0, 117)
-    # Below follows just a typecheck of the schema/type of trade-records
-    oix = None
-    for (pair, profit, date_buy, date_sell, buy_index, dur,
-         openr, closer, open_at_end, sell_reason) in records:
-        assert pair == 'UNITTEST/BTC'
-        assert isinstance(profit, float)
-        # FIX: buy/sell should be converted to ints
-        assert isinstance(date_buy, float)
-        assert isinstance(date_sell, float)
-        assert isinstance(openr, float)
-        assert isinstance(closer, float)
-        assert isinstance(open_at_end, bool)
-        assert isinstance(sell_reason, str)
-        isinstance(buy_index, pd._libs.tslib.Timestamp)
-        if oix:
-            assert buy_index > oix
-        oix = buy_index
-        assert dur > 0
-
-
 def test_backtest_start_timerange(default_conf, mocker, caplog, testdatadir):
     default_conf['exchange']['pair_whitelist'] = ['UNITTEST/BTC']
 
     patch_exchange(mocker)
     mocker.patch('freqtrade.optimize.backtesting.Backtesting.backtest', MagicMock())
-    mocker.patch('freqtrade.optimize.backtesting.generate_text_table', MagicMock())
+    mocker.patch('freqtrade.optimize.backtesting.show_backtest_results', MagicMock())
 
     patched_configuration_load_config_file(mocker, default_conf)
 
@@ -705,9 +631,10 @@ def test_backtest_start_multi_strat(default_conf, mocker, caplog, testdatadir):
     backtestmock = MagicMock()
     mocker.patch('freqtrade.optimize.backtesting.Backtesting.backtest', backtestmock)
     gen_table_mock = MagicMock()
-    mocker.patch('freqtrade.optimize.backtesting.generate_text_table', gen_table_mock)
+    mocker.patch('freqtrade.optimize.optimize_reports.generate_text_table', gen_table_mock)
     gen_strattable_mock = MagicMock()
-    mocker.patch('freqtrade.optimize.backtesting.generate_text_table_strategy', gen_strattable_mock)
+    mocker.patch('freqtrade.optimize.optimize_reports.generate_text_table_strategy',
+                 gen_strattable_mock)
     patched_configuration_load_config_file(mocker, default_conf)
 
     args = [

tests/optimize/test_optimize_reports.py

@@ -1,10 +1,14 @@
+from pathlib import Path
+
 import pandas as pd
+from arrow import Arrow
 
 from freqtrade.edge import PairInfo
 from freqtrade.optimize.optimize_reports import (
     generate_edge_table, generate_text_table, generate_text_table_sell_reason,
-    generate_text_table_strategy)
+    generate_text_table_strategy, store_backtest_result)
 from freqtrade.strategy.interface import SellType
+from tests.conftest import patch_exchange
 
 
 def test_generate_text_table(default_conf, mocker):
@@ -61,10 +65,8 @@ def test_generate_text_table_sell_reason(default_conf, mocker):
         '|     stop_loss |       1 |      0 |       0 |        1 |'
         '            -10 |            -10 |             -0.2 |             -5 |'
     )
-    assert generate_text_table_sell_reason(
-        data={'ETH/BTC': {}},
-        stake_currency='BTC', max_open_trades=2,
-        results=results) == result_str
+    assert generate_text_table_sell_reason(stake_currency='BTC', max_open_trades=2,
+                                           results=results) == result_str
 
 
 def test_generate_text_table_strategy(default_conf, mocker):
@@ -115,3 +117,77 @@ def test_generate_edge_table(edge_conf, mocker):
     assert generate_edge_table(results).count('| ETH/BTC |') == 1
     assert generate_edge_table(results).count(
         '|   Risk Reward Ratio |   Required Risk Reward |   Expectancy |') == 1
+
+
+def test_backtest_record(default_conf, fee, mocker):
+    names = []
+    records = []
+    patch_exchange(mocker)
+    mocker.patch('freqtrade.exchange.Exchange.get_fee', fee)
+    mocker.patch(
+        'freqtrade.optimize.optimize_reports.file_dump_json',
+        new=lambda n, r: (names.append(n), records.append(r))
+    )
+
+    results = {'DefStrat': pd.DataFrame({"pair": ["UNITTEST/BTC", "UNITTEST/BTC",
+                                                  "UNITTEST/BTC", "UNITTEST/BTC"],
+                                         "profit_percent": [0.003312, 0.010801, 0.013803, 0.002780],
+                                         "profit_abs": [0.000003, 0.000011, 0.000014, 0.000003],
+                                         "open_time": [Arrow(2017, 11, 14, 19, 32, 00).datetime,
+                                                       Arrow(2017, 11, 14, 21, 36, 00).datetime,
+                                                       Arrow(2017, 11, 14, 22, 12, 00).datetime,
+                                                       Arrow(2017, 11, 14, 22, 44, 00).datetime],
+                                         "close_time": [Arrow(2017, 11, 14, 21, 35, 00).datetime,
+                                                        Arrow(2017, 11, 14, 22, 10, 00).datetime,
+                                                        Arrow(2017, 11, 14, 22, 43, 00).datetime,
+                                                        Arrow(2017, 11, 14, 22, 58, 00).datetime],
+                                         "open_rate": [0.002543, 0.003003, 0.003089, 0.003214],
+                                         "close_rate": [0.002546, 0.003014, 0.003103, 0.003217],
+                                         "open_index": [1, 119, 153, 185],
+                                         "close_index": [118, 151, 184, 199],
+                                         "trade_duration": [123, 34, 31, 14],
+                                         "open_at_end": [False, False, False, True],
+                                         "sell_reason": [SellType.ROI, SellType.STOP_LOSS,
+                                                         SellType.ROI, SellType.FORCE_SELL]
+                                         })}
+    store_backtest_result(Path("backtest-result.json"), results)
+    # Assert file_dump_json was only called once
+    assert names == [Path('backtest-result.json')]
+    records = records[0]
+    # Ensure records are of correct type
+    assert len(records) == 4
+
+    # reset test to test with strategy name
+    names = []
+    records = []
+    results['Strat'] = results['DefStrat']
+    results['Strat2'] = results['DefStrat']
+    store_backtest_result(Path("backtest-result.json"), results)
+    assert names == [
+        Path('backtest-result-DefStrat.json'),
+        Path('backtest-result-Strat.json'),
+        Path('backtest-result-Strat2.json'),
+    ]
+    records = records[0]
+    # Ensure records are of correct type
+    assert len(records) == 4
+
+    # ('UNITTEST/BTC', 0.00331158, '1510684320', '1510691700', 0, 117)
+    # Below follows just a typecheck of the schema/type of trade-records
+    oix = None
+    for (pair, profit, date_buy, date_sell, buy_index, dur,
+         openr, closer, open_at_end, sell_reason) in records:
+        assert pair == 'UNITTEST/BTC'
+        assert isinstance(profit, float)
+        # FIX: buy/sell should be converted to ints
+        assert isinstance(date_buy, float)
+        assert isinstance(date_sell, float)
+        assert isinstance(openr, float)
+        assert isinstance(closer, float)
+        assert isinstance(open_at_end, bool)
+        assert isinstance(sell_reason, str)
+        isinstance(buy_index, pd._libs.tslib.Timestamp)
+        if oix:
+            assert buy_index > oix
+        oix = buy_index
+        assert dur > 0

tests/pairlist/test_pairlist.py

@@ -46,6 +46,28 @@ def static_pl_conf(whitelist_conf):
     return whitelist_conf
 
 
+def test_log_on_refresh(mocker, static_pl_conf, markets, tickers):
+    mocker.patch.multiple('freqtrade.exchange.Exchange',
+                          markets=PropertyMock(return_value=markets),
+                          exchange_has=MagicMock(return_value=True),
+                          get_tickers=tickers
+                          )
+    freqtrade = get_patched_freqtradebot(mocker, static_pl_conf)
+    logmock = MagicMock()
+    # Assign starting whitelist
+    pl = freqtrade.pairlists._pairlists[0]
+    pl.log_on_refresh(logmock, 'Hello world')
+    assert logmock.call_count == 1
+    pl.log_on_refresh(logmock, 'Hello world')
+    assert logmock.call_count == 1
+    assert pl._log_cache.currsize == 1
+    assert ('Hello world',) in pl._log_cache._Cache__data
+
+    pl.log_on_refresh(logmock, 'Hello world2')
+    assert logmock.call_count == 2
+    assert pl._log_cache.currsize == 2
+
+
 def test_load_pairlist_noexist(mocker, markets, default_conf):
     bot = get_patched_freqtradebot(mocker, default_conf)
     mocker.patch('freqtrade.exchange.Exchange.markets', PropertyMock(return_value=markets))

tests/rpc/test_rpc.py

@@ -13,7 +13,7 @@ from freqtrade.persistence import Trade
 from freqtrade.rpc import RPC, RPCException
 from freqtrade.rpc.fiat_convert import CryptoToFiatConverter
 from freqtrade.state import State
-from tests.conftest import get_patched_freqtradebot, patch_get_signal
+from tests.conftest import get_patched_freqtradebot, patch_get_signal, create_mock_trades
 
 
 # Functions for recurrent object patching
@@ -49,6 +49,18 @@ def test_rpc_trade_status(default_conf, ticker, fee, mocker) -> None:
         'base_currency': 'BTC',
         'open_date': ANY,
         'open_date_hum': ANY,
+        'is_open': ANY,
+        'fee_open': ANY,
+        'fee_close': ANY,
+        'open_rate_requested': ANY,
+        'open_trade_price': ANY,
+        'close_rate_requested': ANY,
+        'sell_reason': ANY,
+        'min_rate': ANY,
+        'max_rate': ANY,
+        'strategy': ANY,
+        'ticker_interval': ANY,
+        'open_order_id': ANY,
         'close_date': None,
         'close_date_hum': None,
         'open_rate': 1.098e-05,
@@ -76,6 +88,18 @@ def test_rpc_trade_status(default_conf, ticker, fee, mocker) -> None:
         'base_currency': 'BTC',
         'open_date': ANY,
         'open_date_hum': ANY,
+        'is_open': ANY,
+        'fee_open': ANY,
+        'fee_close': ANY,
+        'open_rate_requested': ANY,
+        'open_trade_price': ANY,
+        'close_rate_requested': ANY,
+        'sell_reason': ANY,
+        'min_rate': ANY,
+        'max_rate': ANY,
+        'strategy': ANY,
+        'ticker_interval': ANY,
+        'open_order_id': ANY,
         'close_date': None,
         'close_date_hum': None,
         'open_rate': 1.098e-05,
@@ -187,6 +211,32 @@ def test_rpc_daily_profit(default_conf, update, ticker, fee,
         rpc._rpc_daily_profit(0, stake_currency, fiat_display_currency)
 
 
+def test_rpc_trade_history(mocker, default_conf, markets, fee):
+    mocker.patch('freqtrade.rpc.telegram.Telegram', MagicMock())
+    mocker.patch.multiple(
+        'freqtrade.exchange.Exchange',
+        markets=PropertyMock(return_value=markets)
+    )
+
+    freqtradebot = get_patched_freqtradebot(mocker, default_conf)
+    create_mock_trades(fee)
+    rpc = RPC(freqtradebot)
+    rpc._fiat_converter = CryptoToFiatConverter()
+    trades = rpc._rpc_trade_history(2)
+    assert len(trades['trades']) == 2
+    assert trades['trades_count'] == 2
+    assert isinstance(trades['trades'][0], dict)
+    assert isinstance(trades['trades'][1], dict)
+
+    trades = rpc._rpc_trade_history(0)
+    assert len(trades['trades']) == 3
+    assert trades['trades_count'] == 3
+    # The first trade is for ETH ... sorting is descending
+    assert trades['trades'][-1]['pair'] == 'ETH/BTC'
+    assert trades['trades'][0]['pair'] == 'ETC/BTC'
+    assert trades['trades'][1]['pair'] == 'ETC/BTC'
+
+
 def test_rpc_trade_statistics(default_conf, ticker, ticker_sell_up, fee,
                               limit_buy_order, limit_sell_order, mocker) -> None:
     mocker.patch.multiple(

tests/rpc/test_rpc_apiserver.py

@@ -13,7 +13,7 @@ from freqtrade.__init__ import __version__
 from freqtrade.persistence import Trade
 from freqtrade.rpc.api_server import BASE_URI, ApiServer
 from freqtrade.state import State
-from tests.conftest import get_patched_freqtradebot, log_has, patch_get_signal
+from tests.conftest import get_patched_freqtradebot, log_has, patch_get_signal, create_mock_trades
 
 _TEST_USER = "FreqTrader"
 _TEST_PASS = "SuperSecurePassword1!"
@@ -302,6 +302,30 @@ def test_api_daily(botclient, mocker, ticker, fee, markets):
     assert rc.json[0][0] == str(datetime.utcnow().date())
 
 
+def test_api_trades(botclient, mocker, ticker, fee, markets):
+    ftbot, client = botclient
+    patch_get_signal(ftbot, (True, False))
+    mocker.patch.multiple(
+        'freqtrade.exchange.Exchange',
+        markets=PropertyMock(return_value=markets)
+    )
+    rc = client_get(client, f"{BASE_URI}/trades")
+    assert_response(rc)
+    assert len(rc.json) == 2
+    assert rc.json['trades_count'] == 0
+
+    create_mock_trades(fee)
+
+    rc = client_get(client, f"{BASE_URI}/trades")
+    assert_response(rc)
+    assert len(rc.json['trades']) == 3
+    assert rc.json['trades_count'] == 3
+    rc = client_get(client, f"{BASE_URI}/trades?limit=2")
+    assert_response(rc)
+    assert len(rc.json['trades']) == 2
+    assert rc.json['trades_count'] == 2
+
+
 def test_api_edge_disabled(botclient, mocker, ticker, fee, markets):
     ftbot, client = botclient
     patch_get_signal(ftbot, (True, False))
@@ -444,7 +468,21 @@ def test_api_status(botclient, mocker, ticker, fee, markets):
                         'stake_amount': 0.001,
                         'stop_loss': 0.0,
                         'stop_loss_pct': None,
-                        'trade_id': 1}]
+                        'trade_id': 1,
+                        'close_rate_requested': None,
+                        'current_rate': 1.099e-05,
+                        'fee_close': 0.0025,
+                        'fee_open': 0.0025,
+                        'open_date': ANY,
+                        'is_open': True,
+                        'max_rate': 0.0,
+                        'min_rate': None,
+                        'open_order_id': ANY,
+                        'open_rate_requested': 1.098e-05,
+                        'open_trade_price': 0.0010025,
+                        'sell_reason': None,
+                        'strategy': 'DefaultStrategy',
+                        'ticker_interval': 5}]
 
 
 def test_api_version(botclient):
@@ -533,7 +571,21 @@ def test_api_forcebuy(botclient, mocker, fee):
                        'stake_amount': 1,
                        'stop_loss': None,
                        'stop_loss_pct': None,
-                       'trade_id': None}
+                       'trade_id': None,
+                       'close_profit': None,
+                       'close_rate_requested': None,
+                       'fee_close': 0.0025,
+                       'fee_open': 0.0025,
+                       'is_open': False,
+                       'max_rate': None,
+                       'min_rate': None,
+                       'open_order_id': '123456',
+                       'open_rate_requested': None,
+                       'open_trade_price': 0.2460546025,
+                       'sell_reason': None,
+                       'strategy': None,
+                       'ticker_interval': None
+                       }
 
 
 def test_api_forcesell(botclient, mocker, ticker, fee, markets):

tests/rpc/test_rpc_telegram.py

@@ -1316,18 +1316,20 @@ def test_send_msg_sell_cancel_notification(default_conf, mocker) -> None:
         'type': RPCMessageType.SELL_CANCEL_NOTIFICATION,
         'exchange': 'Binance',
         'pair': 'KEY/ETH',
+        'reason': 'Cancelled on exchange'
     })
     assert msg_mock.call_args[0][0] \
-        == ('*Binance:* Cancelling Open Sell Order for KEY/ETH')
+        == ('*Binance:* Cancelling Open Sell Order for KEY/ETH. Reason: Cancelled on exchange')
 
     msg_mock.reset_mock()
     telegram.send_msg({
         'type': RPCMessageType.SELL_CANCEL_NOTIFICATION,
         'exchange': 'Binance',
         'pair': 'KEY/ETH',
+        'reason': 'timeout'
     })
     assert msg_mock.call_args[0][0] \
-        == ('*Binance:* Cancelling Open Sell Order for KEY/ETH')
+        == ('*Binance:* Cancelling Open Sell Order for KEY/ETH. Reason: timeout')
     # Reset singleton function to avoid random breaks
     telegram._fiat_converter.convert_amount = old_convamount
 

tests/strategy/test_interface.py

@@ -21,33 +21,36 @@ from .strats.default_strategy import DefaultStrategy
 _STRATEGY = DefaultStrategy(config={})
 
 
-def test_returns_latest_buy_signal(mocker, default_conf, ohlcv_history):
-    mocker.patch.object(
-        _STRATEGY, '_analyze_ticker_internal',
-        return_value=DataFrame([{'buy': 1, 'sell': 0, 'date': arrow.utcnow()}])
-    )
-    assert _STRATEGY.get_signal('ETH/BTC', '5m', ohlcv_history) == (True, False)
+def test_returns_latest_signal(mocker, default_conf, ohlcv_history):
+    ohlcv_history.loc[1, 'date'] = arrow.utcnow()
+    # Take a copy to correctly modify the call
+    mocked_history = ohlcv_history.copy()
+    mocked_history['sell'] = 0
+    mocked_history['buy'] = 0
+    mocked_history.loc[1, 'sell'] = 1
 
     mocker.patch.object(
         _STRATEGY, '_analyze_ticker_internal',
-        return_value=DataFrame([{'buy': 0, 'sell': 1, 'date': arrow.utcnow()}])
-    )
-    assert _STRATEGY.get_signal('ETH/BTC', '5m', ohlcv_history) == (False, True)
-
-
-def test_returns_latest_sell_signal(mocker, default_conf, ohlcv_history):
-    mocker.patch.object(
-        _STRATEGY, '_analyze_ticker_internal',
-        return_value=DataFrame([{'sell': 1, 'buy': 0, 'date': arrow.utcnow()}])
+        return_value=mocked_history
     )
 
     assert _STRATEGY.get_signal('ETH/BTC', '5m', ohlcv_history) == (False, True)
+    mocked_history.loc[1, 'sell'] = 0
+    mocked_history.loc[1, 'buy'] = 1
 
     mocker.patch.object(
         _STRATEGY, '_analyze_ticker_internal',
-        return_value=DataFrame([{'sell': 0, 'buy': 1, 'date': arrow.utcnow()}])
+        return_value=mocked_history
     )
     assert _STRATEGY.get_signal('ETH/BTC', '5m', ohlcv_history) == (True, False)
+    mocked_history.loc[1, 'sell'] = 0
+    mocked_history.loc[1, 'buy'] = 0
+
+    mocker.patch.object(
+        _STRATEGY, '_analyze_ticker_internal',
+        return_value=mocked_history
+    )
+    assert _STRATEGY.get_signal('ETH/BTC', '5m', ohlcv_history) == (False, False)
 
 
 def test_get_signal_empty(default_conf, mocker, caplog):
@@ -78,26 +81,74 @@ def test_get_signal_empty_dataframe(default_conf, mocker, caplog, ohlcv_history)
         _STRATEGY, '_analyze_ticker_internal',
         return_value=DataFrame([])
     )
+    mocker.patch.object(_STRATEGY, 'assert_df')
+
     assert (False, False) == _STRATEGY.get_signal('xyz', default_conf['ticker_interval'],
                                                   ohlcv_history)
     assert log_has('Empty dataframe for pair xyz', caplog)
 
 
 def test_get_signal_old_dataframe(default_conf, mocker, caplog, ohlcv_history):
-    caplog.set_level(logging.INFO)
     # default_conf defines a 5m interval. we check interval * 2 + 5m
     # this is necessary as the last candle is removed (partial candles) by default
-    oldtime = arrow.utcnow().shift(minutes=-16)
-    ticks = DataFrame([{'buy': 1, 'date': oldtime}])
+    ohlcv_history.loc[1, 'date'] = arrow.utcnow().shift(minutes=-16)
+    # Take a copy to correctly modify the call
+    mocked_history = ohlcv_history.copy()
+    mocked_history['sell'] = 0
+    mocked_history['buy'] = 0
+    mocked_history.loc[1, 'buy'] = 1
+
+    caplog.set_level(logging.INFO)
     mocker.patch.object(
         _STRATEGY, '_analyze_ticker_internal',
-        return_value=DataFrame(ticks)
+        return_value=mocked_history
     )
+    mocker.patch.object(_STRATEGY, 'assert_df')
     assert (False, False) == _STRATEGY.get_signal('xyz', default_conf['ticker_interval'],
                                                   ohlcv_history)
     assert log_has('Outdated history for pair xyz. Last tick is 16 minutes old', caplog)
 
 
+def test_assert_df_raise(default_conf, mocker, caplog, ohlcv_history):
+    # default_conf defines a 5m interval. we check interval * 2 + 5m
+    # this is necessary as the last candle is removed (partial candles) by default
+    ohlcv_history.loc[1, 'date'] = arrow.utcnow().shift(minutes=-16)
+    # Take a copy to correctly modify the call
+    mocked_history = ohlcv_history.copy()
+    mocked_history['sell'] = 0
+    mocked_history['buy'] = 0
+    mocked_history.loc[1, 'buy'] = 1
+
+    caplog.set_level(logging.INFO)
+    mocker.patch.object(
+        _STRATEGY, 'assert_df',
+        side_effect=StrategyError('Dataframe returned...')
+    )
+    assert (False, False) == _STRATEGY.get_signal('xyz', default_conf['ticker_interval'],
+                                                  ohlcv_history)
+    assert log_has('Unable to analyze candle (OHLCV) data for pair xyz: Dataframe returned...',
+                   caplog)
+
+
+def test_assert_df(default_conf, mocker, ohlcv_history):
+    # Ensure it's running when passed correctly
+    _STRATEGY.assert_df(ohlcv_history, len(ohlcv_history),
+                        ohlcv_history.loc[1, 'close'], ohlcv_history.loc[1, 'date'])
+
+    with pytest.raises(StrategyError, match=r"Dataframe returned from strategy.*length\."):
+        _STRATEGY.assert_df(ohlcv_history, len(ohlcv_history) + 1,
+                            ohlcv_history.loc[1, 'close'], ohlcv_history.loc[1, 'date'])
+
+    with pytest.raises(StrategyError,
+                       match=r"Dataframe returned from strategy.*last close price\."):
+        _STRATEGY.assert_df(ohlcv_history, len(ohlcv_history),
+                            ohlcv_history.loc[1, 'close'] + 0.01, ohlcv_history.loc[1, 'date'])
+    with pytest.raises(StrategyError,
+                       match=r"Dataframe returned from strategy.*last date\."):
+        _STRATEGY.assert_df(ohlcv_history, len(ohlcv_history),
+                            ohlcv_history.loc[1, 'close'], ohlcv_history.loc[0, 'date'])
+
+
 def test_get_signal_handles_exceptions(mocker, default_conf):
     exchange = get_patched_exchange(mocker, default_conf)
     mocker.patch.object(
@@ -118,6 +169,19 @@ def test_ohlcvdata_to_dataframe(default_conf, testdatadir) -> None:
     assert len(processed['UNITTEST/BTC']) == 102  # partial candle was removed
 
 
+def test_ohlcvdata_to_dataframe_copy(mocker, default_conf, testdatadir) -> None:
+    default_conf.update({'strategy': 'DefaultStrategy'})
+    strategy = StrategyResolver.load_strategy(default_conf)
+    aimock = mocker.patch('freqtrade.strategy.interface.IStrategy.advise_indicators')
+    timerange = TimeRange.parse_timerange('1510694220-1510700340')
+    data = load_data(testdatadir, '1m', ['UNITTEST/BTC'], timerange=timerange,
+                     fill_up_missing=True)
+    strategy.ohlcvdata_to_dataframe(data)
+    assert aimock.call_count == 1
+    # Ensure that a copy of the dataframe is passed to advice_indicators
+    assert aimock.call_args_list[0][0][0] is not data
+
+
 def test_min_roi_reached(default_conf, fee) -> None:
 
     # Use list to confirm sequence does not matter

tests/test_configuration.py

@@ -18,7 +18,7 @@ from freqtrade.configuration.config_validation import validate_config_schema
 from freqtrade.configuration.deprecated_settings import (
     check_conflicting_settings, process_deprecated_setting,
     process_temporary_deprecated_settings)
-from freqtrade.configuration.load_config import load_config_file
+from freqtrade.configuration.load_config import load_config_file, log_config_error_range
 from freqtrade.constants import DEFAULT_DB_DRYRUN_URL, DEFAULT_DB_PROD_URL
 from freqtrade.exceptions import OperationalException
 from freqtrade.loggers import _set_loggers, setup_logging
@@ -66,6 +66,30 @@ def test_load_config_file(default_conf, mocker, caplog) -> None:
     assert validated_conf.items() >= default_conf.items()
 
 
+def test_load_config_file_error(default_conf, mocker, caplog) -> None:
+    del default_conf['user_data_dir']
+    filedata = json.dumps(default_conf).replace(
+        '"stake_amount": 0.001,', '"stake_amount": .001,')
+    mocker.patch('freqtrade.configuration.load_config.open', mocker.mock_open(read_data=filedata))
+    mocker.patch.object(Path, "read_text", MagicMock(return_value=filedata))
+
+    with pytest.raises(OperationalException, match=f".*Please verify the following segment.*"):
+        load_config_file('somefile')
+
+
+def test_load_config_file_error_range(default_conf, mocker, caplog) -> None:
+    del default_conf['user_data_dir']
+    filedata = json.dumps(default_conf).replace(
+        '"stake_amount": 0.001,', '"stake_amount": .001,')
+    mocker.patch.object(Path, "read_text", MagicMock(return_value=filedata))
+
+    x = log_config_error_range('somefile', 'Parse error at offset 64: Invalid value.')
+    assert isinstance(x, str)
+    assert (x == '{"max_open_trades": 1, "stake_currency": "BTC", '
+            '"stake_amount": .001, "fiat_display_currency": "USD", '
+            '"ticker_interval": "5m", "dry_run": true, ')
+
+
 def test__args_to_config(caplog):
 
     arg_list = ['trade', '--strategy-path', 'TestTest']
@@ -73,6 +97,7 @@ def test__args_to_config(caplog):
     configuration = Configuration(args)
     config = {}
     with warnings.catch_warnings(record=True) as w:
+        warnings.simplefilter("always")
         # No warnings ...
         configuration._args_to_config(config, argname="strategy_path", logstring="DeadBeef")
         assert len(w) == 0
@@ -82,6 +107,7 @@ def test__args_to_config(caplog):
     configuration = Configuration(args)
     config = {}
     with warnings.catch_warnings(record=True) as w:
+        warnings.simplefilter("always")
         # Deprecation warnings!
         configuration._args_to_config(config, argname="strategy_path", logstring="DeadBeef",
                                       deprecated_msg="Going away soon!")

tests/test_freqtradebot.py

@@ -1592,13 +1592,13 @@ def test_exit_positions_exception(mocker, default_conf, limit_buy_order, caplog)
     mocker.patch('freqtrade.exchange.Exchange.get_order', return_value=limit_buy_order)
 
     trade = MagicMock()
-    trade.open_order_id = '123'
+    trade.open_order_id = None
     trade.open_fee = 0.001
     trades = [trade]
 
     # Test raise of DependencyException exception
     mocker.patch(
-        'freqtrade.freqtradebot.FreqtradeBot.update_trade_state',
+        'freqtrade.freqtradebot.FreqtradeBot.handle_trade',
         side_effect=DependencyException()
     )
     n = freqtrade.exit_positions(trades)
@@ -1995,7 +1995,7 @@ def test_check_handle_timedout_buy(default_conf, ticker, limit_buy_order_old, op
         'freqtrade.exchange.Exchange',
         fetch_ticker=ticker,
         get_order=MagicMock(return_value=limit_buy_order_old),
-        cancel_order=cancel_order_mock,
+        cancel_order_with_result=cancel_order_mock,
         get_fee=fee
     )
     freqtrade = FreqtradeBot(default_conf)
@@ -2020,7 +2020,7 @@ def test_check_handle_cancelled_buy(default_conf, ticker, limit_buy_order_old, o
     rpc_mock = patch_RPCManager(mocker)
     cancel_order_mock = MagicMock()
     patch_exchange(mocker)
-    limit_buy_order_old.update({"status": "canceled"})
+    limit_buy_order_old.update({"status": "canceled", 'filled': 0.0})
     mocker.patch.multiple(
         'freqtrade.exchange.Exchange',
         fetch_ticker=ticker,
@@ -2147,13 +2147,13 @@ def test_check_handle_cancelled_sell(default_conf, ticker, limit_sell_order_old,
     """ Handle sell order cancelled on exchange"""
     rpc_mock = patch_RPCManager(mocker)
     cancel_order_mock = MagicMock()
-    limit_sell_order_old.update({"status": "canceled"})
+    limit_sell_order_old.update({"status": "canceled", 'filled': 0.0})
     patch_exchange(mocker)
     mocker.patch.multiple(
         'freqtrade.exchange.Exchange',
         fetch_ticker=ticker,
         get_order=MagicMock(return_value=limit_sell_order_old),
-        cancel_order=cancel_order_mock
+        cancel_order_with_result=cancel_order_mock
     )
     freqtrade = FreqtradeBot(default_conf)
 
@@ -2180,7 +2180,7 @@ def test_check_handle_timedout_partial(default_conf, ticker, limit_buy_order_old
         'freqtrade.exchange.Exchange',
         fetch_ticker=ticker,
         get_order=MagicMock(return_value=limit_buy_order_old_partial),
-        cancel_order=cancel_order_mock
+        cancel_order_with_result=cancel_order_mock
     )
     freqtrade = FreqtradeBot(default_conf)
 
@@ -2207,7 +2207,7 @@ def test_check_handle_timedout_partial_fee(default_conf, ticker, open_trade, cap
         'freqtrade.exchange.Exchange',
         fetch_ticker=ticker,
         get_order=MagicMock(return_value=limit_buy_order_old_partial),
-        cancel_order=cancel_order_mock,
+        cancel_order_with_result=cancel_order_mock,
         get_trades_for_order=MagicMock(return_value=trades_for_order),
     )
     freqtrade = FreqtradeBot(default_conf)
@@ -2227,7 +2227,7 @@ def test_check_handle_timedout_partial_fee(default_conf, ticker, open_trade, cap
     assert rpc_mock.call_count == 2
     trades = Trade.query.filter(Trade.open_order_id.is_(open_trade.open_order_id)).all()
     assert len(trades) == 1
-    # Verify that tradehas been updated
+    # Verify that trade has been updated
     assert trades[0].amount == (limit_buy_order_old_partial['amount'] -
                                 limit_buy_order_old_partial['remaining']) - 0.0001
     assert trades[0].open_order_id is None
@@ -2244,7 +2244,7 @@ def test_check_handle_timedout_partial_except(default_conf, ticker, open_trade,
         'freqtrade.exchange.Exchange',
         fetch_ticker=ticker,
         get_order=MagicMock(return_value=limit_buy_order_old_partial),
-        cancel_order=cancel_order_mock,
+        cancel_order_with_result=cancel_order_mock,
         get_trades_for_order=MagicMock(return_value=trades_for_order),
     )
     mocker.patch('freqtrade.freqtradebot.FreqtradeBot.get_real_amount',
@@ -2266,7 +2266,7 @@ def test_check_handle_timedout_partial_except(default_conf, ticker, open_trade,
     assert rpc_mock.call_count == 2
     trades = Trade.query.filter(Trade.open_order_id.is_(open_trade.open_order_id)).all()
     assert len(trades) == 1
-    # Verify that tradehas been updated
+    # Verify that trade has been updated
 
     assert trades[0].amount == (limit_buy_order_old_partial['amount'] -
                                 limit_buy_order_old_partial['remaining'])
@@ -2302,14 +2302,11 @@ def test_check_handle_timedout_exception(default_conf, ticker, open_trade, mocke
                       caplog)
 
 
-def test_handle_timedout_limit_buy(mocker, default_conf, limit_buy_order) -> None:
+def test_handle_timedout_limit_buy(mocker, caplog, default_conf, limit_buy_order) -> None:
     patch_RPCManager(mocker)
     patch_exchange(mocker)
     cancel_order_mock = MagicMock(return_value=limit_buy_order)
-    mocker.patch.multiple(
-        'freqtrade.exchange.Exchange',
-        cancel_order=cancel_order_mock
-    )
+    mocker.patch('freqtrade.exchange.Exchange.cancel_order_with_result', cancel_order_mock)
 
     freqtrade = FreqtradeBot(default_conf)
 
@@ -2325,11 +2322,21 @@ def test_handle_timedout_limit_buy(mocker, default_conf, limit_buy_order) -> Non
     assert not freqtrade.handle_timedout_limit_buy(trade, limit_buy_order)
     assert cancel_order_mock.call_count == 1
 
+    mocker.patch('freqtrade.exchange.Exchange.cancel_order', side_effect=InvalidOrderException)
+    assert not freqtrade.handle_timedout_limit_buy(trade, limit_buy_order)
 
-def test_handle_timedout_limit_buy_corder_empty(mocker, default_conf, limit_buy_order) -> None:
+
+@pytest.mark.parametrize('cancelorder', [
+    {},
+    {'remaining': None},
+    'String Return value',
+    123
+])
+def test_handle_timedout_limit_buy_corder_empty(mocker, default_conf, limit_buy_order,
+                                                cancelorder) -> None:
     patch_RPCManager(mocker)
     patch_exchange(mocker)
-    cancel_order_mock = MagicMock(return_value={})
+    cancel_order_mock = MagicMock(return_value=cancelorder)
     mocker.patch.multiple(
         'freqtrade.exchange.Exchange',
         cancel_order=cancel_order_mock
@@ -2368,7 +2375,8 @@ def test_handle_timedout_limit_sell(mocker, default_conf) -> None:
     assert freqtrade.handle_timedout_limit_sell(trade, order)
     assert cancel_order_mock.call_count == 1
     order['amount'] = 2
-    assert not freqtrade.handle_timedout_limit_sell(trade, order)
+    assert (freqtrade.handle_timedout_limit_sell(trade, order)
+            == 'partially filled - keeping order open')
     # Assert cancel_order was not called (callcount remains unchanged)
     assert cancel_order_mock.call_count == 1
 
@@ -2591,6 +2599,7 @@ def test_execute_sell_with_stoploss_on_exchange(default_conf, ticker, fee, ticke
     assert trade
     trades = [trade]
 
+    freqtrade.check_handle_timedout()
     freqtrade.exit_positions(trades)
 
     # Increase the price and sell it
@@ -2636,8 +2645,11 @@ def test_may_execute_sell_after_stoploss_on_exchange_hit(default_conf, ticker, f
 
     # Create some test data
     freqtrade.enter_positions()
+    freqtrade.check_handle_timedout()
     trade = Trade.query.first()
     trades = [trade]
+    assert trade.stoploss_order_id is None
+
     freqtrade.exit_positions(trades)
     assert trade
     assert trade.stoploss_order_id == '123'

tests/test_misc.py

@@ -10,7 +10,8 @@ from freqtrade.data.converter import ohlcv_to_dataframe
 from freqtrade.misc import (datesarray_to_datetimearray, file_dump_json,
                             file_load_json, format_ms_time, pair_to_filename,
                             plural, render_template,
-                            render_template_with_fallback, shorten_date)
+                            render_template_with_fallback, safe_value_fallback,
+                            shorten_date)
 
 
 def test_shorten_date() -> None:
@@ -94,6 +95,27 @@ def test_format_ms_time() -> None:
     assert format_ms_time(date_in_epoch_ms) == res.astimezone(None).strftime('%Y-%m-%dT%H:%M:%S')
 
 
+def test_safe_value_fallback():
+    dict1 = {'keya': None, 'keyb': 2, 'keyc': 5, 'keyd': None}
+    dict2 = {'keya': 20, 'keyb': None, 'keyc': 6, 'keyd': None}
+    assert safe_value_fallback(dict1, dict2, 'keya', 'keya') == 20
+    assert safe_value_fallback(dict2, dict1, 'keya', 'keya') == 20
+
+    assert safe_value_fallback(dict1, dict2, 'keyb', 'keyb') == 2
+    assert safe_value_fallback(dict2, dict1, 'keyb', 'keyb') == 2
+
+    assert safe_value_fallback(dict1, dict2, 'keyc', 'keyc') == 5
+    assert safe_value_fallback(dict2, dict1, 'keyc', 'keyc') == 6
+
+    assert safe_value_fallback(dict1, dict2, 'keyd', 'keyd') is None
+    assert safe_value_fallback(dict2, dict1, 'keyd', 'keyd') is None
+    assert safe_value_fallback(dict2, dict1, 'keyd', 'keyd', 1234) == 1234
+
+    assert safe_value_fallback(dict1, dict2, 'keyNo', 'keyNo') is None
+    assert safe_value_fallback(dict2, dict1, 'keyNo', 'keyNo') is None
+    assert safe_value_fallback(dict2, dict1, 'keyNo', 'keyNo', 1234) == 1234
+
+
 def test_plural() -> None:
     assert plural(0, "page") == "pages"
     assert plural(0.0, "page") == "pages"

tests/test_persistence.py

@@ -9,53 +9,7 @@ from sqlalchemy import create_engine
 from freqtrade import constants
 from freqtrade.exceptions import OperationalException
 from freqtrade.persistence import Trade, clean_dry_run_db, init
-from tests.conftest import log_has
-
-
-def create_mock_trades(fee):
-    """
-    Create some fake trades ...
-    """
-    # Simulate dry_run entries
-    trade = Trade(
-        pair='ETH/BTC',
-        stake_amount=0.001,
-        amount=123.0,
-        fee_open=fee.return_value,
-        fee_close=fee.return_value,
-        open_rate=0.123,
-        exchange='bittrex',
-        open_order_id='dry_run_buy_12345'
-    )
-    Trade.session.add(trade)
-
-    trade = Trade(
-        pair='ETC/BTC',
-        stake_amount=0.001,
-        amount=123.0,
-        fee_open=fee.return_value,
-        fee_close=fee.return_value,
-        open_rate=0.123,
-        close_rate=0.128,
-        close_profit=0.005,
-        exchange='bittrex',
-        is_open=False,
-        open_order_id='dry_run_sell_12345'
-    )
-    Trade.session.add(trade)
-
-    # Simulate prod entry
-    trade = Trade(
-        pair='ETC/BTC',
-        stake_amount=0.001,
-        amount=123.0,
-        fee_open=fee.return_value,
-        fee_close=fee.return_value,
-        open_rate=0.123,
-        exchange='bittrex',
-        open_order_id='prod_buy_12345'
-    )
-    Trade.session.add(trade)
+from tests.conftest import log_has, create_mock_trades
 
 
 def test_init_create_session(default_conf):
@@ -476,12 +430,22 @@ def test_migrate_old(mocker, default_conf, fee):
                                      stake=default_conf.get("stake_amount"),
                                      amount=amount
                                      )
+    insert_table_old2 = """INSERT INTO trades (exchange, pair, is_open, fee,
+                          open_rate, close_rate, stake_amount, amount, open_date)
+                          VALUES ('BITTREX', 'BTC_ETC', 0, {fee},
+                          0.00258580, 0.00268580, {stake}, {amount},
+                          '2017-11-28 12:44:24.000000')
+                          """.format(fee=fee.return_value,
+                                     stake=default_conf.get("stake_amount"),
+                                     amount=amount
+                                     )
     engine = create_engine('sqlite://')
     mocker.patch('freqtrade.persistence.create_engine', lambda *args, **kwargs: engine)
 
     # Create table using the old format
     engine.execute(create_table_old)
     engine.execute(insert_table_old)
+    engine.execute(insert_table_old2)
     # Run init to test migration
     init(default_conf['db_url'], default_conf['dry_run'])
 
@@ -500,6 +464,15 @@ def test_migrate_old(mocker, default_conf, fee):
     assert trade.stop_loss == 0.0
     assert trade.initial_stop_loss == 0.0
     assert trade.open_trade_price == trade._calc_open_trade_price()
+    assert trade.close_profit_abs is None
+
+    trade = Trade.query.filter(Trade.id == 2).first()
+    assert trade.close_rate is not None
+    assert trade.is_open == 0
+    assert trade.open_rate_requested is None
+    assert trade.close_rate_requested is None
+    assert trade.close_rate is not None
+    assert pytest.approx(trade.close_profit_abs) == trade.calc_profit()
 
 
 def test_migrate_new(mocker, default_conf, fee, caplog):
@@ -583,6 +556,7 @@ def test_migrate_new(mocker, default_conf, fee, caplog):
     assert log_has("trying trades_bak2", caplog)
     assert log_has("Running database migration - backup available as trades_bak2", caplog)
     assert trade.open_trade_price == trade._calc_open_trade_price()
+    assert trade.close_profit_abs is None
 
 
 def test_migrate_mid_state(mocker, default_conf, fee, caplog):
@@ -757,18 +731,31 @@ def test_to_json(default_conf, fee):
 
     assert result == {'trade_id': None,
                       'pair': 'ETH/BTC',
+                      'is_open': None,
                       'open_date_hum': '2 hours ago',
                       'open_date': trade.open_date.strftime("%Y-%m-%d %H:%M:%S"),
+                      'open_order_id': 'dry_run_buy_12345',
                       'close_date_hum': None,
                       'close_date': None,
                       'open_rate': 0.123,
+                      'open_rate_requested': None,
+                      'open_trade_price': 15.1668225,
+                      'fee_close': 0.0025,
+                      'fee_open': 0.0025,
                       'close_rate': None,
+                      'close_rate_requested': None,
                       'amount': 123.0,
                       'stake_amount': 0.001,
+                      'close_profit': None,
+                      'sell_reason': None,
                       'stop_loss': None,
                       'stop_loss_pct': None,
                       'initial_stop_loss': None,
-                      'initial_stop_loss_pct': None}
+                      'initial_stop_loss_pct': None,
+                      'min_rate': None,
+                      'max_rate': None,
+                      'strategy': None,
+                      'ticker_interval': None}
 
     # Simulate dry_run entries
     trade = Trade(
@@ -799,7 +786,20 @@ def test_to_json(default_conf, fee):
                       'stop_loss': None,
                       'stop_loss_pct': None,
                       'initial_stop_loss': None,
-                      'initial_stop_loss_pct': None}
+                      'initial_stop_loss_pct': None,
+                      'close_profit': None,
+                      'close_rate_requested': None,
+                      'fee_close': 0.0025,
+                      'fee_open': 0.0025,
+                      'is_open': None,
+                      'max_rate': None,
+                      'min_rate': None,
+                      'open_order_id': None,
+                      'open_rate_requested': None,
+                      'open_trade_price': 12.33075,
+                      'sell_reason': None,
+                      'strategy': None,
+                      'ticker_interval': None}
 
 
 def test_stoploss_reinitialization(default_conf, fee):

tests/test_plotting.py

@@ -266,7 +266,7 @@ def test_generate_profit_graph(testdatadir):
     filename = testdatadir / "backtest-result_test.json"
     trades = load_backtest_data(filename)
     timerange = TimeRange.parse_timerange("20180110-20180112")
-    pairs = ["TRX/BTC", "ADA/BTC"]
+    pairs = ["TRX/BTC", "XLM/BTC"]
     trades = trades[trades['close_time'] < pd.Timestamp('2018-01-12', tz='UTC')]
 
     data = history.load_data(datadir=testdatadir,
@@ -292,7 +292,7 @@ def test_generate_profit_graph(testdatadir):
 
     profit = find_trace_in_fig_data(figure.data, "Profit")
     assert isinstance(profit, go.Scatter)
-    profit = find_trace_in_fig_data(figure.data, "Max drawdown 0.00%")
+    profit = find_trace_in_fig_data(figure.data, "Max drawdown 10.45%")
     assert isinstance(profit, go.Scatter)
 
     for pair in pairs: