Merge remote-tracking branch 'origin/develop' into list-pairs2

docs/backtesting.md

@@ -201,6 +201,8 @@ Since backtesting lacks some detailed information about what happens within a ca
 Taking these assumptions, backtesting tries to mirror real trading as closely as possible. However, backtesting will **never** replace running a strategy in dry-run mode.
 Also, keep in mind that past results don't guarantee future success.
 
+In addition to the above assumptions, strategy authors should carefully read the [Common Mistakes](strategy-customization.md#common-mistakes-when-developing-strategies) section, to avoid using data in backtesting which is not available in real market conditions.
+
 ### Further backtest-result analysis
 
 To further analyze your backtest results, you can [export the trades](#exporting-trades-to-file).

docs/strategy-customization.md

@@ -60,8 +60,7 @@ file as reference.**
 !!! Warning Using future data
     Since backtesting passes the full time interval to the `populate_*()` methods, the strategy author
     needs to take care to avoid having the strategy utilize data from the future.
-    Samples for usage of future data are `dataframe.shift(-1)`, `dataframe.resample("1h")` (this uses the left border of the interval, so moves data from an hour to the start of the hour).
+    Some common patterns for this are listed in the [Common Mistakes](#common-mistakes-when-developing-strategies) section of this document.
-    They all use data which is not available during regular operations, so these strategies will perform well during backtesting, but will fail / perform badly in dry-runs.
 
 ### Customize Indicators
 
@@ -399,10 +398,10 @@ def populate_buy_trend(self, dataframe: DataFrame, metadata: dict) -> DataFrame:
 
 Printing more than a few rows is also possible (simply use  `print(dataframe)` instead of `print(dataframe.tail())`), however not recommended, as that will be very verbose (~500 lines per pair every 5 seconds).
 
-### Where is the default strategy?
+### Where can i find a strategy template?
 
-The default buy strategy is located in the file
+The strategy template is located in the file
-[freqtrade/default_strategy.py](https://github.com/freqtrade/freqtrade/blob/develop/freqtrade/strategy/default_strategy.py).
+[user_data/strategies/sample_strategy.py](https://github.com/freqtrade/freqtrade/blob/develop/user_data/strategies/sample_strategy.py).
 
 ### Specify custom strategy location
 
@@ -412,6 +411,18 @@ If you want to use a strategy from a different directory you can pass `--strateg
 freqtrade --strategy AwesomeStrategy --strategy-path /some/directory
 ```
 
+### Common mistakes when developing strategies
+
+Backtesting analyzes the whole time-range at once for performance reasons. Because of this, strategy authors need to make sure that strategies do not look-ahead into the future.
+This is a common pain-point, which can cause huge differences between backtesting and dry/live run methods, since they all use data which is not available during dry/live runs, so these strategies will perform well during backtesting, but will fail / perform badly in real conditions.
+
+The following lists some common patterns which should be avoided to prevent frustration:
+
+- don't use `shift(-1)`. This uses data from the future, which is not available.
+- don't use `.iloc[-1]` or any other absolute position in the dataframe, this will be different between dry-run and backtesting.
+- don't use `dataframe['volume'].mean()`. This uses the full DataFrame for backtesting, including data from the future. Use `dataframe['volume'].rolling(<window>).mean()` instead
+- don't use `.resample('1h')`. This uses the left border of the interval, so moves data from an hour to the start of the hour. Use `.resample('1h', label='right')` instead.
+
 ### Further strategy ideas
 
 To get additional Ideas for strategies, head over to our [strategy repository](https://github.com/freqtrade/freqtrade-strategies). Feel free to use them as they are - but results will depend on the current market situation, pairs used etc. - therefore please backtest the strategy for your exchange/desired pairs first, evaluate carefully, use at your own risk.

freqtrade/exchange/exchange.py

@@ -166,7 +166,7 @@ class Exchange:
     }
     _ft_has: Dict = {}
 
-    def __init__(self, config: dict) -> None:
+    def __init__(self, config: dict, validate: bool = True) -> None:
         """
         Initializes this module with the given config,
         it does basic validation whether the specified exchange and pairs are valid.
@@ -223,13 +223,13 @@ class Exchange:
         # Converts the interval provided in minutes in config to seconds
         self.markets_refresh_interval: int = exchange_config.get(
             "markets_refresh_interval", 60) * 60
-        # Initial markets load
+        if validate:
-        self._load_markets()
+            # Initial markets load
-
+            self._load_markets()
-        # Check if all pairs are available
+            # Check if all pairs are available
-        self.validate_pairs(config['exchange']['pair_whitelist'])
+            self.validate_pairs(config['exchange']['pair_whitelist'])
-        self.validate_ordertypes(config.get('order_types', {}))
+            self.validate_ordertypes(config.get('order_types', {}))
-        self.validate_order_time_in_force(config.get('order_time_in_force', {}))
+            self.validate_order_time_in_force(config.get('order_time_in_force', {}))
 
     def __del__(self):
         """

freqtrade/resolvers/exchange_resolver.py

@@ -17,7 +17,7 @@ class ExchangeResolver(IResolver):
 
     __slots__ = ['exchange']
 
-    def __init__(self, exchange_name: str, config: dict) -> None:
+    def __init__(self, exchange_name: str, config: dict, validate: bool = True) -> None:
         """
         Load the custom class from config parameter
         :param config: configuration dictionary
@@ -26,7 +26,8 @@ class ExchangeResolver(IResolver):
         exchange_name = MAP_EXCHANGE_CHILDCLASS.get(exchange_name, exchange_name)
         exchange_name = exchange_name.title()
         try:
-            self.exchange = self._load_exchange(exchange_name, kwargs={'config': config})
+            self.exchange = self._load_exchange(exchange_name, kwargs={'config': config,
+                                                                       'validate': validate})
         except ImportError:
             logger.info(
                 f"No {exchange_name} specific subclass found. Using the generic class instead.")
@@ -45,7 +46,7 @@ class ExchangeResolver(IResolver):
         try:
             ex_class = getattr(exchanges, exchange_name)
 
-            exchange = ex_class(kwargs['config'])
+            exchange = ex_class(**kwargs)
             if exchange:
                 logger.info(f"Using resolved exchange '{exchange_name}'...")
                 return exchange

freqtrade/rpc/api_server.py

@@ -2,7 +2,7 @@ import logging
 import threading
 from datetime import date, datetime
 from ipaddress import IPv4Address
-from typing import Dict
+from typing import Dict, Callable, Any
 
 from arrow import Arrow
 from flask import Flask, jsonify, request
@@ -34,41 +34,45 @@ class ArrowJSONEncoder(JSONEncoder):
         return JSONEncoder.default(self, obj)
 
 
+# Type should really be Callable[[ApiServer, Any], Any], but that will create a circular dependency
+def require_login(func: Callable[[Any, Any], Any]):
+
+    def func_wrapper(obj, *args, **kwargs):
+
+        auth = request.authorization
+        if auth and obj.check_auth(auth.username, auth.password):
+            return func(obj, *args, **kwargs)
+        else:
+            return jsonify({"error": "Unauthorized"}), 401
+
+    return func_wrapper
+
+
+# Type should really be Callable[[ApiServer], Any], but that will create a circular dependency
+def rpc_catch_errors(func: Callable[[Any], Any]):
+
+    def func_wrapper(obj, *args, **kwargs):
+
+        try:
+            return func(obj, *args, **kwargs)
+        except RPCException as e:
+            logger.exception("API Error calling %s: %s", func.__name__, e)
+            return obj.rest_error(f"Error querying {func.__name__}: {e}")
+
+    return func_wrapper
+
+
 class ApiServer(RPC):
     """
     This class runs api server and provides rpc.rpc functionality to it
 
-    This class starts a none blocking thread the api server runs within
+    This class starts a non-blocking thread the api server runs within
     """
 
-    def rpc_catch_errors(func):
-
-        def func_wrapper(self, *args, **kwargs):
-
-            try:
-                return func(self, *args, **kwargs)
-            except RPCException as e:
-                logger.exception("API Error calling %s: %s", func.__name__, e)
-                return self.rest_error(f"Error querying {func.__name__}: {e}")
-
-        return func_wrapper
-
     def check_auth(self, username, password):
         return (username == self._config['api_server'].get('username') and
                 password == self._config['api_server'].get('password'))
 
-    def require_login(func):
-
-        def func_wrapper(self, *args, **kwargs):
-
-            auth = request.authorization
-            if auth and self.check_auth(auth.username, auth.password):
-                return func(self, *args, **kwargs)
-            else:
-                return jsonify({"error": "Unauthorized"}), 401
-
-        return func_wrapper
-
     def __init__(self, freqtrade) -> None:
         """
         Init the api server, and init the super class RPC

freqtrade/utils.py

@@ -128,7 +128,7 @@ def start_list_timeframes(args: Dict[str, Any]) -> None:
     config['ticker_interval'] = None
 
     # Init exchange
-    exchange = ExchangeResolver(config['exchange']['name'], config).exchange
+    exchange = ExchangeResolver(config['exchange']['name'], config, validate=False).exchange
 
     if args['print_one_column']:
         print('\n'.join(exchange.timeframes))

requirements-common.txt

@@ -1,8 +1,8 @@
 # requirements without requirements installable via conda
 # mainly used for Raspberry pi installs
-ccxt==1.18.1260
+ccxt==1.18.1306
 SQLAlchemy==1.3.10
-python-telegram-bot==12.1.1
+python-telegram-bot==12.2.0
 arrow==0.15.2
 cachetools==3.1.1
 requests==2.22.0

requirements-dev.txt

@@ -6,8 +6,8 @@
 coveralls==1.8.2
 flake8==3.7.8
 flake8-type-annotations==0.1.0
-flake8-tidy-imports==2.0.0
+flake8-tidy-imports==3.0.0
-mypy==0.730
+mypy==0.740
 pytest==5.2.1
 pytest-asyncio==0.10.0
 pytest-cov==2.8.1

requirements-plot.txt

@@ -1,5 +1,5 @@
 # Include all requirements to run the bot.
 -r requirements.txt
 
-plotly==4.1.1
+plotly==4.2.1
 

requirements.txt

@@ -1,5 +1,5 @@
 # Load common requirements
 -r requirements-common.txt
 
-numpy==1.17.2
+numpy==1.17.3
-pandas==0.25.1
+pandas==0.25.2