Merge branch 'develop' into pr/jonny07/5674

README.md

@@ -53,7 +53,7 @@ Please find the complete documentation on our [website](https://www.freqtrade.io
 - [x] **Dry-run**: Run the bot without paying money.
 - [x] **Backtesting**: Run a simulation of your buy/sell strategy.
 - [x] **Strategy Optimization by machine learning**: Use machine learning to optimize your buy/sell strategy parameters with real exchange data.
-- [x] **Edge position sizing** Calculate your win rate, risk reward ratio, the best stoploss and adjust your position size before taking a position for each specific market. [Learn more](https://www.freqtrade.io/en/latest/edge/).
+- [x] **Edge position sizing** Calculate your win rate, risk reward ratio, the best stoploss and adjust your position size before taking a position for each specific market. [Learn more](https://www.freqtrade.io/en/stable/edge/).
 - [x] **Whitelist crypto-currencies**: Select which crypto-currency you want to trade or use dynamic whitelists.
 - [x] **Blacklist crypto-currencies**: Select which crypto-currency you want to avoid.
 - [x] **Manageable via Telegram**: Manage the bot with Telegram.
@@ -66,12 +66,12 @@ Please find the complete documentation on our [website](https://www.freqtrade.io
 Freqtrade provides a Linux/macOS script to install all dependencies and help you to configure the bot.
 
 ```bash
-git clone -b develop https://github.com/freqtrade/freqtrade.git 
+git clone -b develop https://github.com/freqtrade/freqtrade.git
 cd freqtrade
 ./setup.sh --install
 ```
 
-For any other type of installation please refer to [Installation doc](https://www.freqtrade.io/en/latest/installation/).
+For any other type of installation please refer to [Installation doc](https://www.freqtrade.io/en/stable/installation/).
 
 ## Basic Usage
 

docker-compose.yml

@@ -15,10 +15,10 @@ services:
     volumes:
       - "./user_data:/freqtrade/user_data"
     # Expose api on port 8080 (localhost only)
-    # Please read the https://www.freqtrade.io/en/latest/rest-api/ documentation
+    # Please read the https://www.freqtrade.io/en/stable/rest-api/ documentation
     # before enabling this.
-    # ports:
+    ports:
-    #   - "127.0.0.1:8080:8080"
+      - "127.0.0.1:8080:8080"
     # Default command used when running `docker compose up`
     command: >
       trade

docs/docker_quickstart.md

@@ -182,27 +182,9 @@ You'll then also need to modify the `docker-compose.yml` file and uncomment the
       dockerfile: "./Dockerfile.<yourextension>"
 ```
 
-You can then run `docker-compose build` to build the docker image, and run it using the commands described above.
+You can then run `docker-compose build --pull` to build the docker image, and run it using the commands described above.
 
-### Troubleshooting
+### Plotting with docker-compose
-
-#### Docker on Windows
-
-* Error: `"Timestamp for this request is outside of the recvWindow."`
-  * The market api requests require a synchronized clock but the time in the docker container shifts a bit over time into the past.
-    To fix this issue temporarily you need to run `wsl --shutdown` and restart docker again (a popup on windows 10 will ask you to do so).
-    A permanent solution is either to host the docker container on a linux host or restart the wsl from time to time with the scheduler.
-    ```
-    taskkill /IM "Docker Desktop.exe" /F
-    wsl --shutdown
-    start "" "C:\Program Files\Docker\Docker\Docker Desktop.exe"
-    ```
-
-!!! Warning
-    Due to the above, we do not recommend the usage of docker on windows for production setups, but only for experimentation, datadownload and backtesting.
-    Best use a linux-VPS for running freqtrade reliably.
-
-## Plotting with docker-compose
 
 Commands `freqtrade plot-profit` and `freqtrade plot-dataframe` ([Documentation](plotting.md)) are available by changing the image to `*_plot` in your docker-compose.yml file.
 You can then use these commands as follows:
@@ -213,7 +195,7 @@ docker-compose run --rm freqtrade plot-dataframe --strategy AwesomeStrategy -p B
 
 The output will be stored in the `user_data/plot` directory, and can be opened with any modern browser.
 
-## Data analysis using docker compose
+### Data analysis using docker compose
 
 Freqtrade provides a docker-compose file which starts up a jupyter lab server.
 You can run this server using the following command:
@@ -230,3 +212,22 @@ Since part of this image is built on your machine, it is recommended to rebuild
 ``` bash
 docker-compose -f docker/docker-compose-jupyter.yml build --no-cache
 ```
+
+## Troubleshooting
+
+### Docker on Windows
+
+* Error: `"Timestamp for this request is outside of the recvWindow."`
+  * The market api requests require a synchronized clock but the time in the docker container shifts a bit over time into the past.
+    To fix this issue temporarily you need to run `wsl --shutdown` and restart docker again (a popup on windows 10 will ask you to do so).
+    A permanent solution is either to host the docker container on a linux host or restart the wsl from time to time with the scheduler.
+
+    ``` bash
+    taskkill /IM "Docker Desktop.exe" /F
+    wsl --shutdown
+    start "" "C:\Program Files\Docker\Docker\Docker Desktop.exe"
+    ```
+
+!!! Warning
+    Due to the above, we do not recommend the usage of docker on windows for production setups, but only for experimentation, datadownload and backtesting.
+    Best use a linux-VPS for running freqtrade reliably.

docs/faq.md

@@ -54,9 +54,11 @@ you can't say much from few trades.
 
 Yes. You can edit your config and use the `/reload_config` command to reload the configuration. The bot will stop, reload the configuration and strategy and will restart with the new configuration and strategy.
 
-### I want to improve the bot with a new strategy
+### I want to use incomplete candles
 
-That's great. We have a nice backtesting and hyperoptimization setup. See the tutorial [here|Testing-new-strategies-with-Hyperopt](bot-usage.md#hyperopt-commands).
+Freqtrade will not provide incomplete candles to strategies. Using incomplete candles will lead to repainting and consequently to strategies with "ghost" buys, which are impossible to both backtest, and verify after they happened.
+
+You can use "current" market data by using the [dataprovider](strategy-customization.md#orderbookpair-maximum)'s orderbook or ticker methods - which however cannot be used during backtesting.
 
 ### Is there a setting to only SELL the coins being held and not perform anymore BUYS?
 

docs/hyperopt.md

@@ -60,7 +60,7 @@ optional arguments:
                         Specify what timerange of data to use.
   --data-format-ohlcv {json,jsongz,hdf5}
                         Storage format for downloaded candle (OHLCV) data.
-                        (default: `None`).
+                        (default: `json`).
   --max-open-trades INT
                         Override the value of the `max_open_trades`
                         configuration setting.
@@ -114,7 +114,8 @@ optional arguments:
                         Hyperopt-loss-functions are:
                         ShortTradeDurHyperOptLoss, OnlyProfitHyperOptLoss,
                         SharpeHyperOptLoss, SharpeHyperOptLossDaily,
-                        SortinoHyperOptLoss, SortinoHyperOptLossDaily
+                        SortinoHyperOptLoss, SortinoHyperOptLossDaily,
+                        MaxDrawDownHyperOptLoss
   --disable-param-export
                         Disable automatic hyperopt parameter export.
 
@@ -512,12 +513,13 @@ This class should be in its own file within the `user_data/hyperopts/` directory
 
 Currently, the following loss functions are builtin:
 
-* `ShortTradeDurHyperOptLoss` (default legacy Freqtrade hyperoptimization loss function) - Mostly for short trade duration and avoiding losses.
+* `ShortTradeDurHyperOptLoss` - (default legacy Freqtrade hyperoptimization loss function) - Mostly for short trade duration and avoiding losses.
-* `OnlyProfitHyperOptLoss` (which takes only amount of profit into consideration)
+* `OnlyProfitHyperOptLoss` - takes only amount of profit into consideration.
-* `SharpeHyperOptLoss` (optimizes Sharpe Ratio calculated on trade returns relative to standard deviation)
+* `SharpeHyperOptLoss` - optimizes Sharpe Ratio calculated on trade returns relative to standard deviation.
-* `SharpeHyperOptLossDaily` (optimizes Sharpe Ratio calculated on **daily** trade returns relative to standard deviation)
+* `SharpeHyperOptLossDaily` - optimizes Sharpe Ratio calculated on **daily** trade returns relative to standard deviation.
-* `SortinoHyperOptLoss` (optimizes Sortino Ratio calculated on trade returns relative to **downside** standard deviation)
+* `SortinoHyperOptLoss` - optimizes Sortino Ratio calculated on trade returns relative to **downside** standard deviation.
-* `SortinoHyperOptLossDaily` (optimizes Sortino Ratio calculated on **daily** trade returns relative to **downside** standard deviation)
+* `SortinoHyperOptLossDaily` - optimizes Sortino Ratio calculated on **daily** trade returns relative to **downside** standard deviation.
+* `MaxDrawDownHyperOptLoss` - Optimizes Maximum drawdown.
 
 Creation of a custom loss function is covered in the [Advanced Hyperopt](advanced-hyperopt.md) part of the documentation.
 

docs/installation.md

@@ -113,6 +113,13 @@ git checkout develop
 
 You may later switch between branches at any time with the `git checkout stable`/`git checkout develop` commands.
 
+??? Note "Install from pypi"
+    An alternative way to install Freqtrade is from [pypi](https://pypi.org/project/freqtrade/). The downside is that this method requires ta-lib to be correctly installed beforehand, and is therefore currently not the recommended way to install Freqtrade.
+
+    ``` bash
+    pip install freqtrade
+    ```
+
 ------
 
 ## Script Installation

docs/rest-api.md

@@ -78,7 +78,7 @@ If you run your bot using docker, you'll need to have the bot listen to incoming
     },
 ```
 
-Uncomment the following from your docker-compose file:
+Make sure that the following 2 lines are available in your docker-compose file:
 
 ```yml
     ports:

freqtrade/commands/build_config_commands.py

@@ -163,7 +163,8 @@ def ask_user_config() -> Dict[str, Any]:
         {
             "type": "text",
             "name": "api_server_listen_addr",
-            "message": "Insert Api server Listen Address (best left untouched default!)",
+            "message": ("Insert Api server Listen Address (0.0.0.0 for docker, "
+                        "otherwise best left untouched)"),
             "default": "127.0.0.1",
             "when": lambda x: x['api_server']
         },

freqtrade/constants.py

@@ -24,7 +24,8 @@ ORDERTYPE_POSSIBILITIES = ['limit', 'market']
 ORDERTIF_POSSIBILITIES = ['gtc', 'fok', 'ioc']
 HYPEROPT_LOSS_BUILTIN = ['ShortTradeDurHyperOptLoss', 'OnlyProfitHyperOptLoss',
                          'SharpeHyperOptLoss', 'SharpeHyperOptLossDaily',
-                         'SortinoHyperOptLoss', 'SortinoHyperOptLossDaily']
+                         'SortinoHyperOptLoss', 'SortinoHyperOptLossDaily',
+                         'MaxDrawDownHyperOptLoss']
 AVAILABLE_PAIRLISTS = ['StaticPairList', 'VolumePairList',
                        'AgeFilter', 'OffsetFilter', 'PerformanceFilter',
                        'PrecisionFilter', 'PriceFilter', 'RangeStabilityFilter',

freqtrade/exchange/exchange.py

@@ -480,7 +480,7 @@ class Exchange:
         if startup_candles + 5 > candle_limit:
             raise OperationalException(
                 f"This strategy requires {startup_candles} candles to start. "
-                f"{self.name} only provides {candle_limit} for {timeframe}.")
+                f"{self.name} only provides {candle_limit - 5} for {timeframe}.")
 
     def exchange_has(self, endpoint: str) -> bool:
         """
@@ -1058,7 +1058,7 @@ class Exchange:
             ticker_rate = ticker[conf_strategy['price_side']]
             if ticker['last'] and ticker_rate:
                 if side == 'buy' and ticker_rate > ticker['last']:
-                    balance = conf_strategy['ask_last_balance']
+                    balance = conf_strategy.get('ask_last_balance', 0.0)
                     ticker_rate = ticker_rate + balance * (ticker['last'] - ticker_rate)
                 elif side == 'sell' and ticker_rate < ticker['last']:
                     balance = conf_strategy.get('bid_last_balance', 0.0)

freqtrade/optimize/hyperopt_loss_max_drawdown.py

@@ -0,0 +1,41 @@
+"""
+MaxDrawDownHyperOptLoss
+
+This module defines the alternative HyperOptLoss class which can be used for
+Hyperoptimization.
+"""
+from datetime import datetime
+
+from pandas import DataFrame
+
+from freqtrade.data.btanalysis import calculate_max_drawdown
+from freqtrade.optimize.hyperopt import IHyperOptLoss
+
+
+class MaxDrawDownHyperOptLoss(IHyperOptLoss):
+
+    """
+    Defines the loss function for hyperopt.
+
+    This implementation optimizes for max draw down and profit
+    Less max drawdown more profit -> Lower return value
+    """
+
+    @staticmethod
+    def hyperopt_loss_function(results: DataFrame, trade_count: int,
+                               min_date: datetime, max_date: datetime,
+                               *args, **kwargs) -> float:
+
+        """
+        Objective function.
+
+        Uses profit ratio weighted max_drawdown when drawdown is available.
+        Otherwise directly optimizes profit ratio.
+        """
+        total_profit = results['profit_abs'].sum()
+        try:
+            max_drawdown = calculate_max_drawdown(results, value_col='profit_abs')
+        except ValueError:
+            # No losing trade, therefore no drawdown.
+            return -total_profit
+        return -total_profit / max_drawdown[0]

freqtrade/rpc/api_server/api_schemas.py

@@ -347,3 +347,8 @@ class BacktestResponse(BaseModel):
     trade_count: Optional[float]
     # TODO: Properly type backtestresult...
     backtest_result: Optional[Dict[str, Any]]
+
+
+class SysInfo(BaseModel):
+    cpu_pct: List[float]
+    ram_pct: float

freqtrade/rpc/api_server/api_v1.py

@@ -18,7 +18,8 @@ from freqtrade.rpc.api_server.api_schemas import (AvailablePairs, Balances, Blac
                                                   OpenTradeSchema, PairHistory, PerformanceEntry,
                                                   Ping, PlotConfig, Profit, ResultMsg, ShowConfig,
                                                   Stats, StatusMsg, StrategyListResponse,
-                                                  StrategyResponse, Version, WhitelistResponse)
+                                                  StrategyResponse, SysInfo, Version,
+                                                  WhitelistResponse)
 from freqtrade.rpc.api_server.deps import get_config, get_rpc, get_rpc_optional
 from freqtrade.rpc.rpc import RPCException
 
@@ -259,3 +260,8 @@ def list_available_pairs(timeframe: Optional[str] = None, stake_currency: Option
         'pair_interval': pair_interval,
     }
     return result
+
+
+@router.get('/sysinfo', response_model=SysInfo, tags=['info'])
+def sysinfo():
+    return RPC._rpc_sysinfo()

freqtrade/rpc/rpc.py

@@ -8,6 +8,7 @@ from math import isnan
 from typing import Any, Dict, List, Optional, Tuple, Union
 
 import arrow
+import psutil
 from numpy import NAN, inf, int64, mean
 from pandas import DataFrame
 
@@ -870,3 +871,10 @@ class RPC:
                 'subplots' not in self._freqtrade.strategy.plot_config):
             self._freqtrade.strategy.plot_config['subplots'] = {}
         return self._freqtrade.strategy.plot_config
+
+    @staticmethod
+    def _rpc_sysinfo() -> Dict[str, Any]:
+        return {
+            "cpu_pct": psutil.cpu_percent(interval=1, percpu=True),
+            "ram_pct": psutil.virtual_memory().percent
+        }

requirements.txt

@@ -36,6 +36,7 @@ fastapi==0.68.1
 uvicorn==0.15.0
 pyjwt==2.1.0
 aiofiles==0.7.0
+psutil==5.8.0
 
 # Support for colorized terminal output
 colorama==0.4.4

scripts/rest_client.py

@@ -334,6 +334,13 @@ class FtRestClient():
             "timerange": timerange if timerange else '',
         })
 
+    def sysinfo(self):
+        """Provides system information (CPU, RAM usage)
+
+        :return: json object
+        """
+        return self._get("sysinfo")
+
 
 def add_arguments():
     parser = argparse.ArgumentParser()

tests/exchange/test_exchange.py

@@ -1832,6 +1832,7 @@ def test_fetch_l2_order_book_exception(default_conf, mocker, exchange_name):
     ('ask', 20, 19, 10, 0.3, 17),  # Between ask and last
     ('ask', 5, 6, 10, 1.0, 5),  # last bigger than ask
     ('ask', 5, 6, 10, 0.5, 5),  # last bigger than ask
+    ('ask', 20, 19, 10, None, 20),  # ask_last_balance missing
     ('ask', 10, 20, None, 0.5, 10),  # last not available - uses ask
     ('ask', 4, 5, None, 0.5, 4),  # last not available - uses ask
     ('ask', 4, 5, None, 1, 4),  # last not available - uses ask
@@ -1842,6 +1843,7 @@ def test_fetch_l2_order_book_exception(default_conf, mocker, exchange_name):
     ('bid', 21, 20, 10, 0.7, 13),  # Between bid and last
     ('bid', 21, 20, 10, 0.3, 17),  # Between bid and last
     ('bid', 6, 5, 10, 1.0, 5),  # last bigger than bid
+    ('bid', 21, 20, 10, None, 20),  # ask_last_balance missing
     ('bid', 6, 5, 10, 0.5, 5),  # last bigger than bid
     ('bid', 21, 20, None, 0.5, 20),  # last not available - uses bid
     ('bid', 6, 5, None, 0.5, 5),  # last not available - uses bid
@@ -1851,7 +1853,10 @@ def test_fetch_l2_order_book_exception(default_conf, mocker, exchange_name):
 def test_get_buy_rate(mocker, default_conf, caplog, side, ask, bid,
                       last, last_ab, expected) -> None:
     caplog.set_level(logging.DEBUG)
-    default_conf['bid_strategy']['ask_last_balance'] = last_ab
+    if last_ab is None:
+        del default_conf['bid_strategy']['ask_last_balance']
+    else:
+        default_conf['bid_strategy']['ask_last_balance'] = last_ab
     default_conf['bid_strategy']['price_side'] = side
     exchange = get_patched_exchange(mocker, default_conf)
     mocker.patch('freqtrade.exchange.Exchange.fetch_ticker',
@@ -1876,6 +1881,7 @@ def test_get_buy_rate(mocker, default_conf, caplog, side, ask, bid,
     ('bid', 12.0, 11.2, 10.5, 1.0, 11.2),  # Last smaller than bid - uses bid
     ('bid', 12.0, 11.2, 10.5, 0.5, 11.2),  # Last smaller than bid - uses bid
     ('bid', 0.003, 0.002, 0.005, 0.0, 0.002),
+    ('bid', 0.003, 0.002, 0.005, None, 0.002),
     ('ask', 12.0, 11.0, 12.5, 0.0, 12.0),  # full ask side
     ('ask', 12.0, 11.0, 12.5, 1.0, 12.5),  # full last side
     ('ask', 12.0, 11.0, 12.5, 0.5, 12.25),  # between bid and lat
@@ -1886,13 +1892,15 @@ def test_get_buy_rate(mocker, default_conf, caplog, side, ask, bid,
     ('ask', 10.11, 11.2, 11.0, 0.0, 10.11),
     ('ask', 0.001, 0.002, 11.0, 0.0, 0.001),
     ('ask', 0.006, 1.0, 11.0, 0.0, 0.006),
+    ('ask', 0.006, 1.0, 11.0, None, 0.006),
 ])
 def test_get_sell_rate(default_conf, mocker, caplog, side, bid, ask,
                        last, last_ab, expected) -> None:
     caplog.set_level(logging.DEBUG)
 
     default_conf['ask_strategy']['price_side'] = side
-    default_conf['ask_strategy']['bid_last_balance'] = last_ab
+    if last_ab is not None:
+        default_conf['ask_strategy']['bid_last_balance'] = last_ab
     mocker.patch('freqtrade.exchange.Exchange.fetch_ticker',
                  return_value={'ask': ask, 'bid': bid, 'last': last})
     pair = "ETH/BTC"

tests/optimize/test_hyperoptloss.py

@@ -84,13 +84,14 @@ def test_loss_calculation_has_limited_profit(hyperopt_conf, hyperopt_results) ->
     "SortinoHyperOptLossDaily",
     "SharpeHyperOptLoss",
     "SharpeHyperOptLossDaily",
+    "MaxDrawDownHyperOptLoss",
 ])
 def test_loss_functions_better_profits(default_conf, hyperopt_results, lossfunction) -> None:
     results_over = hyperopt_results.copy()
-    results_over['profit_abs'] = hyperopt_results['profit_abs'] * 2
+    results_over['profit_abs'] = hyperopt_results['profit_abs'] * 2 + 0.2
     results_over['profit_ratio'] = hyperopt_results['profit_ratio'] * 2
     results_under = hyperopt_results.copy()
-    results_under['profit_abs'] = hyperopt_results['profit_abs'] / 2
+    results_under['profit_abs'] = hyperopt_results['profit_abs'] / 2 - 0.2
     results_under['profit_ratio'] = hyperopt_results['profit_ratio'] / 2
 
     default_conf.update({'hyperopt_loss': lossfunction})

tests/rpc/test_rpc_apiserver.py

@@ -1271,6 +1271,16 @@ def test_list_available_pairs(botclient):
     assert len(rc.json()['pair_interval']) == 1
 
 
+def test_sysinfo(botclient):
+    ftbot, client = botclient
+
+    rc = client_get(client, f"{BASE_URI}/sysinfo")
+    assert_response(rc)
+    result = rc.json()
+    assert 'cpu_pct' in result
+    assert 'ram_pct' in result
+
+
 def test_api_backtesting(botclient, mocker, fee, caplog):
     ftbot, client = botclient
     mocker.patch('freqtrade.exchange.Exchange.get_fee', fee)