Merge branch 'develop' into feat/short

docs/data-download.md

@@ -11,7 +11,7 @@ Otherwise `--exchange` becomes mandatory.
 You can use a relative timerange (`--days 20`) or an absolute starting point (`--timerange 20200101-`). For incremental downloads, the relative approach should be used.
 
 !!! Tip "Tip: Updating existing data"
-    If you already have backtesting data available in your data-directory and would like to refresh this data up to today, do not use `--days` or `--timerange` parameters. Freqtrade will keep the available data and only download the missing data.
+    If you already have backtesting data available in your data-directory and would like to refresh this data up to today, freqtrade will automatically calculate the data missing for the existing pairs and the download will occur from the latest available point until "now", neither --days or --timerange parameters are required. Freqtrade will keep the available data and only download the missing data.
     If you are updating existing data after inserting new pairs that you have no data for, use `--new-pairs-days xx` parameter. Specified number of days will be downloaded for new pairs while old pairs will be updated with missing data only.
     If you use `--days xx` parameter alone - data for specified number of days will be downloaded for _all_ pairs. Be careful, if specified number of days is smaller than gap between now and last downloaded candle - freqtrade will delete all existing data to avoid gaps in candle data.
 

docs/faq.md

@@ -54,6 +54,21 @@ you can't say much from few trades.
 
 Yes. You can edit your config and use the `/reload_config` command to reload the configuration. The bot will stop, reload the configuration and strategy and will restart with the new configuration and strategy.
 
+### Why does my bot not sell everything it bought?
+
+This is called "coin dust" and can happen on all exchanges.
+It happens because many exchanges subtract fees from the "receiving currency" - so you buy 100 COIN - but you only get 99.9 COIN.
+As COIN is trading in full lot sizes (1COIN steps), you cannot sell 0.9 COIN (or 99.9 COIN) - but you need to round down to 99 COIN.
+
+This is not a bot-problem, but will also happen while manual trading.
+
+While freqtrade can handle this (it'll sell 99 COIN), fees are often below the minimum tradable lot-size (you can only trade full COIN, not 0.9 COIN).
+Leaving the dust (0.9 COIN) on the exchange makes usually sense, as the next time freqtrade buys COIN, it'll eat into the remaining small balance, this time selling everything it bought, and therefore slowly declining the dust balance (although it most likely will never reach exactly 0).
+
+Where possible (e.g. on binance), the use of the exchange's dedicated fee currency will fix this.  
+On binance, it's sufficient to have BNB in your account, and have "Pay fees in BNB" enabled in your profile. Your BNB balance will slowly decline (as it's used to pay fees) - but you'll no longer encounter dust (Freqtrade will include the fees in the profit calculations).
+Other exchanges don't offer such possibilities, where it's simply something you'll have to accept or move to a different exchange.
+
 ### I want to use incomplete candles
 
 Freqtrade will not provide incomplete candles to strategies. Using incomplete candles will lead to repainting and consequently to strategies with "ghost" buys, which are impossible to both backtest, and verify after they happened.

docs/hyperopt.md

@@ -116,7 +116,7 @@ optional arguments:
                         ShortTradeDurHyperOptLoss, OnlyProfitHyperOptLoss,
                         SharpeHyperOptLoss, SharpeHyperOptLossDaily,
                         SortinoHyperOptLoss, SortinoHyperOptLossDaily,
-                        MaxDrawDownHyperOptLoss
+                        CalmarHyperOptLoss, MaxDrawDownHyperOptLoss
   --disable-param-export
                         Disable automatic hyperopt parameter export.
   --ignore-missing-spaces, --ignore-unparameterized-spaces
@@ -524,6 +524,7 @@ Currently, the following loss functions are builtin:
 * `SortinoHyperOptLoss` - optimizes Sortino Ratio calculated on trade returns relative to **downside** standard deviation.
 * `SortinoHyperOptLossDaily` - optimizes Sortino Ratio calculated on **daily** trade returns relative to **downside** standard deviation.
 * `MaxDrawDownHyperOptLoss` - Optimizes Maximum drawdown.
+* `CalmarHyperOptLoss` - Optimizes Calmar Ratio calculated on trade returns relative to max drawdown.
 
 Creation of a custom loss function is covered in the [Advanced Hyperopt](advanced-hyperopt.md) part of the documentation.
 

docs/strategy-customization.md

@@ -312,7 +312,7 @@ Currently this is `pair`, which can be accessed using `metadata['pair']` - and w
 The Metadata-dict should not be modified and does not persist information across multiple calls.
 Instead, have a look at the section [Storing information](strategy-advanced.md#Storing-information)
 
-## Additional data (informative_pairs)
+## Informative Pairs
 
 ### Get data for non-tradeable pairs
 
@@ -341,6 +341,133 @@ A full sample can be found [in the DataProvider section](#complete-data-provider
 
 ***
 
+### Informative pairs decorator (`@informative()`)
+
+In most common case it is possible to easily define informative pairs by using a decorator. All decorated `populate_indicators_*` methods run in isolation,
+not having access to data from other informative pairs, in the end all informative dataframes are merged and passed to main `populate_indicators()` method.
+When hyperopting, use of hyperoptable parameter `.value` attribute is not supported. Please use `.range` attribute. See [optimizing an indicator parameter](hyperopt.md#optimizing-an-indicator-parameter)
+for more information.
+
+??? info "Full documentation"
+    ``` python
+    def informative(timeframe: str, asset: str = '',
+                    fmt: Optional[Union[str, Callable[[KwArg(str)], str]]] = None,
+                    ffill: bool = True) -> Callable[[PopulateIndicators], PopulateIndicators]:
+        """
+        A decorator for populate_indicators_Nn(self, dataframe, metadata), allowing these functions to
+        define informative indicators.
+
+        Example usage:
+
+            @informative('1h')
+            def populate_indicators_1h(self, dataframe: DataFrame, metadata: dict) -> DataFrame:
+                dataframe['rsi'] = ta.RSI(dataframe, timeperiod=14)
+                return dataframe
+
+        :param timeframe: Informative timeframe. Must always be equal or higher than strategy timeframe.
+        :param asset: Informative asset, for example BTC, BTC/USDT, ETH/BTC. Do not specify to use
+        current pair.
+        :param fmt: Column format (str) or column formatter (callable(name, asset, timeframe)). When not
+        specified, defaults to:
+        * {base}_{quote}_{column}_{timeframe} if asset is specified. 
+        * {column}_{timeframe} if asset is not specified.
+        Format string supports these format variables:
+        * {asset} - full name of the asset, for example 'BTC/USDT'.
+        * {base} - base currency in lower case, for example 'eth'.
+        * {BASE} - same as {base}, except in upper case.
+        * {quote} - quote currency in lower case, for example 'usdt'.
+        * {QUOTE} - same as {quote}, except in upper case.
+        * {column} - name of dataframe column.
+        * {timeframe} - timeframe of informative dataframe.
+        :param ffill: ffill dataframe after merging informative pair.
+        """
+    ```
+
+??? Example "Fast and easy way to define informative pairs"
+
+    Most of the time we do not need power and flexibility offered by `merge_informative_pair()`, therefore we can use a decorator to quickly define informative pairs.
+
+    ``` python
+
+    from datetime import datetime
+    from freqtrade.persistence import Trade
+    from freqtrade.strategy import IStrategy, informative
+
+    class AwesomeStrategy(IStrategy):
+        
+        # This method is not required. 
+        # def informative_pairs(self): ...
+
+        # Define informative upper timeframe for each pair. Decorators can be stacked on same 
+        # method. Available in populate_indicators as 'rsi_30m' and 'rsi_1h'.
+        @informative('30m')
+        @informative('1h')
+        def populate_indicators_1h(self, dataframe: DataFrame, metadata: dict) -> DataFrame:
+            dataframe['rsi'] = ta.RSI(dataframe, timeperiod=14)
+            return dataframe
+
+        # Define BTC/STAKE informative pair. Available in populate_indicators and other methods as
+        # 'btc_rsi_1h'. Current stake currency should be specified as {stake} format variable 
+        # instead of hardcoding actual stake currency. Available in populate_indicators and other 
+        # methods as 'btc_usdt_rsi_1h' (when stake currency is USDT).
+        @informative('1h', 'BTC/{stake}')
+        def populate_indicators_btc_1h(self, dataframe: DataFrame, metadata: dict) -> DataFrame:
+            dataframe['rsi'] = ta.RSI(dataframe, timeperiod=14)
+            return dataframe
+
+        # Define BTC/ETH informative pair. You must specify quote currency if it is different from
+        # stake currency. Available in populate_indicators and other methods as 'eth_btc_rsi_1h'.
+        @informative('1h', 'ETH/BTC')
+        def populate_indicators_eth_btc_1h(self, dataframe: DataFrame, metadata: dict) -> DataFrame:
+            dataframe['rsi'] = ta.RSI(dataframe, timeperiod=14)
+            return dataframe
+    
+        # Define BTC/STAKE informative pair. A custom formatter may be specified for formatting
+        # column names. A callable `fmt(**kwargs) -> str` may be specified, to implement custom
+        # formatting. Available in populate_indicators and other methods as 'rsi_upper'.
+        @informative('1h', 'BTC/{stake}', '{column}')
+        def populate_indicators_btc_1h_2(self, dataframe: DataFrame, metadata: dict) -> DataFrame:
+            dataframe['rsi_upper'] = ta.RSI(dataframe, timeperiod=14)
+            return dataframe
+    
+        def populate_indicators(self, dataframe: DataFrame, metadata: dict) -> DataFrame:
+            # Strategy timeframe indicators for current pair.
+            dataframe['rsi'] = ta.RSI(dataframe, timeperiod=14)
+            # Informative pairs are available in this method.
+            dataframe['rsi_less'] = dataframe['rsi'] < dataframe['rsi_1h']
+            return dataframe
+
+    ```
+
+!!! Note
+    Do not use `@informative` decorator if you need to use data of one informative pair when generating another informative pair. Instead, define informative pairs
+    manually as described [in the DataProvider section](#complete-data-provider-sample).
+
+!!! Note
+    Use string formatting when accessing informative dataframes of other pairs. This will allow easily changing stake currency in config without having to adjust strategy code.
+
+    ``` python
+    def populate_buy_trend(self, dataframe: DataFrame, metadata: dict) -> DataFrame:
+        stake = self.config['stake_currency']
+        dataframe.loc[
+            (
+                (dataframe[f'btc_{stake}_rsi_1h'] < 35)
+                &
+                (dataframe['volume'] > 0)
+            ),
+            ['buy', 'buy_tag']] = (1, 'buy_signal_rsi')
+    
+        return dataframe
+    ```
+
+    Alternatively column renaming may be used to remove stake currency from column names: `@informative('1h', 'BTC/{stake}', fmt='{base}_{column}_{timeframe}')`.
+
+!!! Warning "Duplicate method names"
+    Methods tagged with `@informative()` decorator must always have unique names! Re-using same name (for example when copy-pasting already defined informative method)
+    will overwrite previously defined method and not produce any errors due to limitations of Python programming language. In such cases you will find that indicators
+    created in earlier-defined methods are not available in the dataframe. Carefully review method names and make sure they are unique!
+
+
 ## Additional data (DataProvider)
 
 The strategy provides access to the `DataProvider`. This allows you to get additional data to use in your strategy.
@@ -686,131 +813,6 @@ In some situations it may be confusing to deal with stops relative to current ra
 
     ```
 
-### *@informative()*
-
-``` python
-def informative(timeframe: str, asset: str = '',
-                fmt: Optional[Union[str, Callable[[KwArg(str)], str]]] = None,
-                ffill: bool = True) -> Callable[[PopulateIndicators], PopulateIndicators]:
-    """
-    A decorator for populate_indicators_Nn(self, dataframe, metadata), allowing these functions to
-    define informative indicators.
-
-    Example usage:
-
-        @informative('1h')
-        def populate_indicators_1h(self, dataframe: DataFrame, metadata: dict) -> DataFrame:
-            dataframe['rsi'] = ta.RSI(dataframe, timeperiod=14)
-            return dataframe
-
-    :param timeframe: Informative timeframe. Must always be equal or higher than strategy timeframe.
-    :param asset: Informative asset, for example BTC, BTC/USDT, ETH/BTC. Do not specify to use
-    current pair.
-    :param fmt: Column format (str) or column formatter (callable(name, asset, timeframe)). When not
-    specified, defaults to:
-    * {base}_{quote}_{column}_{timeframe} if asset is specified. 
-    * {column}_{timeframe} if asset is not specified.
-    Format string supports these format variables:
-    * {asset} - full name of the asset, for example 'BTC/USDT'.
-    * {base} - base currency in lower case, for example 'eth'.
-    * {BASE} - same as {base}, except in upper case.
-    * {quote} - quote currency in lower case, for example 'usdt'.
-    * {QUOTE} - same as {quote}, except in upper case.
-    * {column} - name of dataframe column.
-    * {timeframe} - timeframe of informative dataframe.
-    :param ffill: ffill dataframe after merging informative pair.
-    """
-```
-
-In most common case it is possible to easily define informative pairs by using a decorator. All decorated `populate_indicators_*` methods run in isolation,
-not having access to data from other informative pairs, in the end all informative dataframes are merged and passed to main `populate_indicators()` method.
-When hyperopting, use of hyperoptable parameter `.value` attribute is not supported. Please use `.range` attribute. See [optimizing an indicator parameter](hyperopt.md#optimizing-an-indicator-parameter)
-for more information.
-
-??? Example "Fast and easy way to define informative pairs"
-
-    Most of the time we do not need power and flexibility offered by `merge_informative_pair()`, therefore we can use a decorator to quickly define informative pairs.
-
-    ``` python
-
-    from datetime import datetime
-    from freqtrade.persistence import Trade
-    from freqtrade.strategy import IStrategy, informative
-
-    class AwesomeStrategy(IStrategy):
-        
-        # This method is not required. 
-        # def informative_pairs(self): ...
-
-        # Define informative upper timeframe for each pair. Decorators can be stacked on same 
-        # method. Available in populate_indicators as 'rsi_30m' and 'rsi_1h'.
-        @informative('30m')
-        @informative('1h')
-        def populate_indicators_1h(self, dataframe: DataFrame, metadata: dict) -> DataFrame:
-            dataframe['rsi'] = ta.RSI(dataframe, timeperiod=14)
-            return dataframe
-
-        # Define BTC/STAKE informative pair. Available in populate_indicators and other methods as
-        # 'btc_rsi_1h'. Current stake currency should be specified as {stake} format variable 
-        # instead of hardcoding actual stake currency. Available in populate_indicators and other 
-        # methods as 'btc_usdt_rsi_1h' (when stake currency is USDT).
-        @informative('1h', 'BTC/{stake}')
-        def populate_indicators_btc_1h(self, dataframe: DataFrame, metadata: dict) -> DataFrame:
-            dataframe['rsi'] = ta.RSI(dataframe, timeperiod=14)
-            return dataframe
-
-        # Define BTC/ETH informative pair. You must specify quote currency if it is different from
-        # stake currency. Available in populate_indicators and other methods as 'eth_btc_rsi_1h'.
-        @informative('1h', 'ETH/BTC')
-        def populate_indicators_eth_btc_1h(self, dataframe: DataFrame, metadata: dict) -> DataFrame:
-            dataframe['rsi'] = ta.RSI(dataframe, timeperiod=14)
-            return dataframe
-    
-        # Define BTC/STAKE informative pair. A custom formatter may be specified for formatting
-        # column names. A callable `fmt(**kwargs) -> str` may be specified, to implement custom
-        # formatting. Available in populate_indicators and other methods as 'rsi_upper'.
-        @informative('1h', 'BTC/{stake}', '{column}')
-        def populate_indicators_btc_1h_2(self, dataframe: DataFrame, metadata: dict) -> DataFrame:
-            dataframe['rsi_upper'] = ta.RSI(dataframe, timeperiod=14)
-            return dataframe
-    
-        def populate_indicators(self, dataframe: DataFrame, metadata: dict) -> DataFrame:
-            # Strategy timeframe indicators for current pair.
-            dataframe['rsi'] = ta.RSI(dataframe, timeperiod=14)
-            # Informative pairs are available in this method.
-            dataframe['rsi_less'] = dataframe['rsi'] < dataframe['rsi_1h']
-            return dataframe
-
-    ```
-
-!!! Note
-    Do not use `@informative` decorator if you need to use data of one informative pair when generating another informative pair. Instead, define informative pairs
-    manually as described [in the DataProvider section](#complete-data-provider-sample).
-
-!!! Note
-    Use string formatting when accessing informative dataframes of other pairs. This will allow easily changing stake currency in config without having to adjust strategy code.
-
-    ``` python
-    def populate_buy_trend(self, dataframe: DataFrame, metadata: dict) -> DataFrame:
-        stake = self.config['stake_currency']
-        dataframe.loc[
-            (
-                (dataframe[f'btc_{stake}_rsi_1h'] < 35)
-                &
-                (dataframe['volume'] > 0)
-            ),
-            ['buy', 'buy_tag']] = (1, 'buy_signal_rsi')
-    
-        return dataframe
-    ```
-
-    Alternatively column renaming may be used to remove stake currency from column names: `@informative('1h', 'BTC/{stake}', fmt='{base}_{column}_{timeframe}')`.
-
-!!! Warning "Duplicate method names"
-    Methods tagged with `@informative()` decorator must always have unique names! Re-using same name (for example when copy-pasting already defined informative method)
-    will overwrite previously defined method and not produce any errors due to limitations of Python programming language. In such cases you will find that indicators
-    created in earlier-defined methods are not available in the dataframe. Carefully review method names and make sure they are unique!
-
 ## Additional data (Wallets)
 
 The strategy provides access to the `Wallets` object. This contains the current balances on the exchange.
@@ -894,7 +896,8 @@ Sometimes it may be desired to lock a pair after certain events happen (e.g. mul
 Freqtrade has an easy method to do this from within the strategy, by calling `self.lock_pair(pair, until, [reason])`.
 `until` must be a datetime object in the future, after which trading will be re-enabled for that pair, while `reason` is an optional string detailing why the pair was locked.
 
-Locks can also be lifted manually, by calling `self.unlock_pair(pair)`.
+Locks can also be lifted manually, by calling `self.unlock_pair(pair)` or `self.unlock_reason(<reason>)` - providing reason the pair was locked with.
+`self.unlock_reason(<reason>)` will unlock all pairs currently locked with the provided reason.
 
 To verify if a pair is currently locked, use `self.is_pair_locked(pair)`.
 

docs/telegram-usage.md

@@ -171,7 +171,7 @@ official commands. You can ask at any moment for help with `/help`.
 | `/profit [<n>]` | Display a summary of your profit/loss from close trades and some stats about your performance, over the last n days (all trades by default)
 | `/forcesell <trade_id>` | Instantly sells the given trade  (Ignoring `minimum_roi`).
 | `/forcesell all` | Instantly sells all open trades (Ignoring `minimum_roi`).
-| `/forcebuy <pair> [rate]` | Instantly buys the given pair. Rate is optional. (`forcebuy_enable` must be set to True)
+| `/forcebuy <pair> [rate]` | Instantly buys the given pair. Rate is optional and only applies to limit orders. (`forcebuy_enable` must be set to True)
 | `/performance` | Show performance of each finished trade grouped by pair
 | `/balance` | Show account balance per currency
 | `/daily <n>` | Shows profit or loss per day, over the last n days (n defaults to 7)

freqtrade/constants.py

@@ -25,6 +25,7 @@ ORDERTIF_POSSIBILITIES = ['gtc', 'fok', 'ioc']
 HYPEROPT_LOSS_BUILTIN = ['ShortTradeDurHyperOptLoss', 'OnlyProfitHyperOptLoss',
                          'SharpeHyperOptLoss', 'SharpeHyperOptLossDaily',
                          'SortinoHyperOptLoss', 'SortinoHyperOptLossDaily',
+                         'CalmarHyperOptLoss',
                          'MaxDrawDownHyperOptLoss']
 AVAILABLE_PAIRLISTS = ['StaticPairList', 'VolumePairList',
                        'AgeFilter', 'OffsetFilter', 'PerformanceFilter',
@@ -55,7 +56,6 @@ ENV_VAR_PREFIX = 'FREQTRADE__'
 
 NON_OPEN_EXCHANGE_STATES = ('cancelled', 'canceled', 'closed', 'expired')
 
-
 # Define decimals per coin for outputs
 # Only used for outputs.
 DECIMAL_PER_COIN_FALLBACK = 3  # Should be low to avoid listing all possible FIAT's
@@ -69,7 +69,6 @@ DUST_PER_COIN = {
     'ETH': 0.01
 }
 
-
 # Source files with destination directories within user-directory
 USER_DATA_FILES = {
     'sample_strategy.py': USERPATH_STRATEGIES,

freqtrade/optimize/hyperopt_loss_calmar.py

@@ -0,0 +1,64 @@
+"""
+CalmarHyperOptLoss
+
+This module defines the alternative HyperOptLoss class which can be used for
+Hyperoptimization.
+"""
+from datetime import datetime
+from math import sqrt as msqrt
+from typing import Any, Dict
+
+from pandas import DataFrame
+
+from freqtrade.data.btanalysis import calculate_max_drawdown
+from freqtrade.optimize.hyperopt import IHyperOptLoss
+
+
+class CalmarHyperOptLoss(IHyperOptLoss):
+    """
+    Defines the loss function for hyperopt.
+
+    This implementation uses the Calmar Ratio calculation.
+    """
+
+    @staticmethod
+    def hyperopt_loss_function(
+        results: DataFrame,
+        trade_count: int,
+        min_date: datetime,
+        max_date: datetime,
+        config: Dict,
+        processed: Dict[str, DataFrame],
+        backtest_stats: Dict[str, Any],
+        *args,
+        **kwargs
+    ) -> float:
+        """
+        Objective function, returns smaller number for more optimal results.
+
+        Uses Calmar Ratio calculation.
+        """
+        total_profit = backtest_stats["profit_total"]
+        days_period = (max_date - min_date).days
+
+        # adding slippage of 0.1% per trade
+        total_profit = total_profit - 0.0005
+        expected_returns_mean = total_profit.sum() / days_period * 100
+
+        # calculate max drawdown
+        try:
+            _, _, _, high_val, low_val = calculate_max_drawdown(
+                results, value_col="profit_abs"
+            )
+            max_drawdown = (high_val - low_val) / high_val
+        except ValueError:
+            max_drawdown = 0
+
+        if max_drawdown != 0:
+            calmar_ratio = expected_returns_mean / max_drawdown * msqrt(365)
+        else:
+            # Define high (negative) calmar ratio to be clear that this is NOT optimal.
+            calmar_ratio = -20.0
+
+        # print(expected_returns_mean, max_drawdown, calmar_ratio)
+        return -calmar_ratio

freqtrade/optimize/hyperopt_tools.py

@@ -1,4 +1,3 @@
-
 import io
 import logging
 from copy import deepcopy
@@ -64,7 +63,8 @@ class HyperoptTools():
             'export_time': datetime.now(timezone.utc),
         }
         logger.info(f"Dumping parameters to {filename}")
-        rapidjson.dump(final_params, filename.open('w'), indent=2,
+        with filename.open('w') as f:
+            rapidjson.dump(final_params, f, indent=2,
                            default=hyperopt_serializer,
                            number_mode=rapidjson.NM_NATIVE | rapidjson.NM_NAN
                            )

freqtrade/optimize/space/decimalspace.py

@@ -7,11 +7,15 @@ class SKDecimal(Integer):
     def __init__(self, low, high, decimals=3, prior="uniform", base=10, transform=None,
                  name=None, dtype=np.int64):
         self.decimals = decimals
-        _low = int(low * pow(10, self.decimals))
+
-        _high = int(high * pow(10, self.decimals))
+        self.pow_dot_one = pow(0.1, self.decimals)
+        self.pow_ten = pow(10, self.decimals)
+
+        _low = int(low * self.pow_ten)
+        _high = int(high * self.pow_ten)
         # trunc to precision to avoid points out of space
-        self.low_orig = round(_low * pow(0.1, self.decimals), self.decimals)
+        self.low_orig = round(_low * self.pow_dot_one, self.decimals)
-        self.high_orig = round(_high * pow(0.1, self.decimals), self.decimals)
+        self.high_orig = round(_high * self.pow_dot_one, self.decimals)
 
         super().__init__(_low, _high, prior, base, transform, name, dtype)
 
@@ -25,9 +29,9 @@ class SKDecimal(Integer):
         return self.low_orig <= point <= self.high_orig
 
     def transform(self, Xt):
-        aa = [int(x * pow(10, self.decimals)) for x in Xt]
+        return super().transform([int(v * self.pow_ten) for v in Xt])
-        return super().transform(aa)
 
     def inverse_transform(self, Xt):
         res = super().inverse_transform(Xt)
-        return [round(x * pow(0.1, self.decimals), self.decimals) for x in res]
+        # equivalent to [round(x * pow(0.1, self.decimals), self.decimals) for x in res]
+        return [int(v) / self.pow_ten for v in res]

freqtrade/persistence/models.py

@@ -1123,7 +1123,7 @@ class PairLock(_DECL_BASE):
         lock_time = self.lock_time.strftime(DATETIME_PRINT_FORMAT)
         lock_end_time = self.lock_end_time.strftime(DATETIME_PRINT_FORMAT)
         return (f'PairLock(id={self.id}, pair={self.pair}, lock_time={lock_time}, '
-                f'lock_end_time={lock_end_time})')
+                f'lock_end_time={lock_end_time}, reason={self.reason}, active={self.active})')
 
     @staticmethod
     def query_pair_locks(pair: Optional[str], now: datetime) -> Query:
@@ -1132,7 +1132,6 @@ class PairLock(_DECL_BASE):
         :param pair: Pair to check for. Returns all current locks if pair is empty
         :param now: Datetime object (generated via datetime.now(timezone.utc)).
         """
-
         filters = [PairLock.lock_end_time > now,
                    # Only active locks
                    PairLock.active.is_(True), ]

freqtrade/persistence/pairlock_middleware.py

@@ -103,6 +103,36 @@ class PairLocks():
         if PairLocks.use_db:
             PairLock.query.session.commit()
 
+    @staticmethod
+    def unlock_reason(reason: str, now: Optional[datetime] = None) -> None:
+        """
+        Release all locks for this reason.
+        :param reason: Which reason to unlock
+        :param now: Datetime object (generated via datetime.now(timezone.utc)).
+            defaults to datetime.now(timezone.utc)
+        """
+        if not now:
+            now = datetime.now(timezone.utc)
+
+        if PairLocks.use_db:
+            # used in live modes
+            logger.info(f"Releasing all locks with reason '{reason}':")
+            filters = [PairLock.lock_end_time > now,
+                       PairLock.active.is_(True),
+                       PairLock.reason == reason
+                       ]
+            locks = PairLock.query.filter(*filters)
+            for lock in locks:
+                logger.info(f"Releasing lock for {lock.pair} with reason '{reason}'.")
+                lock.active = False
+            PairLock.query.session.commit()
+        else:
+            # used in backtesting mode; don't show log messages for speed
+            locks = PairLocks.get_pair_locks(None)
+            for lock in locks:
+                if lock.reason == reason:
+                    lock.active = False
+
     @staticmethod
     def is_global_lock(now: Optional[datetime] = None) -> bool:
         """
@@ -128,7 +158,9 @@ class PairLocks():
 
     @staticmethod
     def get_all_locks() -> List[PairLock]:
-
+        """
+        Return all locks, also locks with expired end date
+        """
         if PairLocks.use_db:
             return PairLock.query.all()
         else:

freqtrade/resolvers/iresolver.py

@@ -91,7 +91,7 @@ class IResolver:
         logger.debug(f"Searching for {cls.object_type.__name__} {object_name} in '{directory}'")
         for entry in directory.iterdir():
             # Only consider python files
-            if not str(entry).endswith('.py'):
+            if entry.suffix != '.py':
                 logger.debug('Ignoring %s', entry)
                 continue
             if entry.is_symlink() and not entry.is_file():
@@ -169,7 +169,7 @@ class IResolver:
         objects = []
         for entry in directory.iterdir():
             # Only consider python files
-            if not str(entry).endswith('.py'):
+            if entry.suffix != '.py':
                 logger.debug('Ignoring %s', entry)
                 continue
             module_path = entry.resolve()

freqtrade/resolvers/strategy_resolver.py

@@ -56,17 +56,21 @@ class StrategyResolver(IResolver):
         if strategy._ft_params_from_file:
             # Set parameters from Hyperopt results file
             params = strategy._ft_params_from_file
-            strategy.minimal_roi = params.get('roi', strategy.minimal_roi)
+            strategy.minimal_roi = params.get('roi', getattr(strategy, 'minimal_roi', {}))
 
-            strategy.stoploss = params.get('stoploss', {}).get('stoploss', strategy.stoploss)
+            strategy.stoploss = params.get('stoploss', {}).get(
+                'stoploss', getattr(strategy, 'stoploss', -0.1))
             trailing = params.get('trailing', {})
-            strategy.trailing_stop = trailing.get('trailing_stop', strategy.trailing_stop)
+            strategy.trailing_stop = trailing.get(
-            strategy.trailing_stop_positive = trailing.get('trailing_stop_positive',
+                'trailing_stop', getattr(strategy, 'trailing_stop', False))
-                                                           strategy.trailing_stop_positive)
+            strategy.trailing_stop_positive = trailing.get(
+                'trailing_stop_positive', getattr(strategy, 'trailing_stop_positive', None))
             strategy.trailing_stop_positive_offset = trailing.get(
-                'trailing_stop_positive_offset', strategy.trailing_stop_positive_offset)
+                'trailing_stop_positive_offset',
+                getattr(strategy, 'trailing_stop_positive_offset', 0))
             strategy.trailing_only_offset_is_reached = trailing.get(
-                'trailing_only_offset_is_reached', strategy.trailing_only_offset_is_reached)
+                'trailing_only_offset_is_reached',
+                getattr(strategy, 'trailing_only_offset_is_reached', 0.0))
 
         # Set attributes
         # Check if we need to override configuration

freqtrade/rpc/telegram.py

@@ -1033,7 +1033,8 @@ class Telegram(RPCHandler):
         :return: None
         """
         forcebuy_text = ("*/forcebuy <pair> [<rate>]:* `Instantly buys the given pair. "
-                         "Optionally takes a rate at which to buy.` \n")
+                         "Optionally takes a rate at which to buy "
+                         "(only applies to limit orders).` \n")
         message = ("*/start:* `Starts the trader`\n"
                    "*/stop:* `Stops the trader`\n"
                    "*/status <trade_id>|[table]:* `Lists all open trades`\n"

freqtrade/strategy/hyper.py

@@ -381,7 +381,8 @@ class HyperStrategyMixin(object):
         if filename.is_file():
             logger.info(f"Loading parameters from file {filename}")
             try:
-                params = json_load(filename.open('r'))
+                with filename.open('r') as f:
+                    params = json_load(f)
                 if params.get('strategy_name') != self.__class__.__name__:
                     raise OperationalException('Invalid parameter file provided.')
                 return params

freqtrade/strategy/interface.py

@@ -65,9 +65,9 @@ class IStrategy(ABC, HyperStrategyMixin):
     _populate_fun_len: int = 0
     _buy_fun_len: int = 0
     _sell_fun_len: int = 0
-    _ft_params_from_file: Dict = {}
+    _ft_params_from_file: Dict
     # associated minimal roi
-    minimal_roi: Dict
+    minimal_roi: Dict = {}
 
     # associated stoploss
     stoploss: float
@@ -462,6 +462,15 @@ class IStrategy(ABC, HyperStrategyMixin):
         """
         PairLocks.unlock_pair(pair, datetime.now(timezone.utc))
 
+    def unlock_reason(self, reason: str) -> None:
+        """
+        Unlocks all pairs previously locked using lock_pair with specified reason.
+        Not used by freqtrade itself, but intended to be used if users lock pairs
+        manually from within the strategy, to allow an easy way to unlock pairs.
+        :param reason: Unlock pairs to allow trading again
+        """
+        PairLocks.unlock_reason(reason, datetime.now(timezone.utc))
+
     def is_pair_locked(self, pair: str, candle_date: datetime = None) -> bool:
         """
         Checks if a pair is currently locked

requirements.txt

@@ -1,18 +1,18 @@
-numpy==1.21.2
+numpy==1.21.3
 pandas==1.3.4
 pandas-ta==0.3.14b
 
-ccxt==1.58.47
+ccxt==1.59.2
 # Pin cryptography for now due to rust build errors with piwheels
 cryptography==35.0.0
 aiohttp==3.7.4.post0
-SQLAlchemy==1.4.25
+SQLAlchemy==1.4.26
 python-telegram-bot==13.7
-arrow==1.2.0
+arrow==1.2.1
 cachetools==4.2.2
 requests==2.26.0
 urllib3==1.26.7
-jsonschema==4.1.0
+jsonschema==4.1.2
 TA-Lib==0.4.21
 technical==1.3.0
 tabulate==0.8.9
@@ -41,7 +41,7 @@ psutil==5.8.0
 colorama==0.4.4
 # Building config files interactively
 questionary==1.10.0
-prompt-toolkit==3.0.20
+prompt-toolkit==3.0.21
 
 #Futures
 schedule==1.1.0

tests/optimize/test_hyperopt_tools.py

@@ -209,7 +209,8 @@ def test_export_params(tmpdir):
 
     assert filename.is_file()
 
-    content = rapidjson.load(filename.open('r'))
+    with filename.open('r') as f:
+        content = rapidjson.load(f)
     assert content['strategy_name'] == CURRENT_TEST_STRATEGY
     assert 'params' in content
     assert "buy" in content["params"]

tests/optimize/test_hyperoptloss.py

@@ -85,6 +85,8 @@ def test_loss_calculation_has_limited_profit(hyperopt_conf, hyperopt_results) ->
     "SharpeHyperOptLoss",
     "SharpeHyperOptLossDaily",
     "MaxDrawDownHyperOptLoss",
+    "CalmarHyperOptLoss",
+
 ])
 def test_loss_functions_better_profits(default_conf, hyperopt_results, lossfunction) -> None:
     results_over = hyperopt_results.copy()
@@ -96,11 +98,32 @@ def test_loss_functions_better_profits(default_conf, hyperopt_results, lossfunct
 
     default_conf.update({'hyperopt_loss': lossfunction})
     hl = HyperOptLossResolver.load_hyperoptloss(default_conf)
-    correct = hl.hyperopt_loss_function(hyperopt_results, len(hyperopt_results),
+    correct = hl.hyperopt_loss_function(
-                                        datetime(2019, 1, 1), datetime(2019, 5, 1))
+        hyperopt_results,
-    over = hl.hyperopt_loss_function(results_over, len(results_over),
+        trade_count=len(hyperopt_results),
-                                     datetime(2019, 1, 1), datetime(2019, 5, 1))
+        min_date=datetime(2019, 1, 1),
-    under = hl.hyperopt_loss_function(results_under, len(results_under),
+        max_date=datetime(2019, 5, 1),
-                                      datetime(2019, 1, 1), datetime(2019, 5, 1))
+        config=default_conf,
+        processed=None,
+        backtest_stats={'profit_total': hyperopt_results['profit_abs'].sum()}
+        )
+    over = hl.hyperopt_loss_function(
+        results_over,
+        trade_count=len(results_over),
+        min_date=datetime(2019, 1, 1),
+        max_date=datetime(2019, 5, 1),
+        config=default_conf,
+        processed=None,
+        backtest_stats={'profit_total': results_over['profit_abs'].sum()}
+    )
+    under = hl.hyperopt_loss_function(
+        results_under,
+        trade_count=len(results_under),
+        min_date=datetime(2019, 1, 1),
+        max_date=datetime(2019, 5, 1),
+        config=default_conf,
+        processed=None,
+        backtest_stats={'profit_total': results_under['profit_abs'].sum()}
+    )
     assert over < correct
     assert under > correct

tests/plugins/test_pairlocks.py

@@ -116,3 +116,28 @@ def test_PairLocks_getlongestlock(use_db):
 
     PairLocks.reset_locks()
     PairLocks.use_db = True
+
+
+@pytest.mark.parametrize('use_db', (False, True))
+@pytest.mark.usefixtures("init_persistence")
+def test_PairLocks_reason(use_db):
+    PairLocks.timeframe = '5m'
+    PairLocks.use_db = use_db
+    # No lock should be present
+    if use_db:
+        assert len(PairLock.query.all()) == 0
+
+    assert PairLocks.use_db == use_db
+
+    PairLocks.lock_pair('XRP/USDT', arrow.utcnow().shift(minutes=4).datetime, 'TestLock1')
+    PairLocks.lock_pair('ETH/USDT', arrow.utcnow().shift(minutes=4).datetime, 'TestLock2')
+
+    assert PairLocks.is_pair_locked('XRP/USDT')
+    assert PairLocks.is_pair_locked('ETH/USDT')
+
+    PairLocks.unlock_reason('TestLock1')
+    assert not PairLocks.is_pair_locked('XRP/USDT')
+    assert PairLocks.is_pair_locked('ETH/USDT')
+
+    PairLocks.reset_locks()
+    PairLocks.use_db = True

tests/strategy/test_interface.py

@@ -633,6 +633,13 @@ def test_is_pair_locked(default_conf):
     strategy.unlock_pair(pair)
     assert not strategy.is_pair_locked(pair)
 
+    # Lock with reason
+    reason = "TestLockR"
+    strategy.lock_pair(pair, arrow.now(timezone.utc).shift(minutes=4).datetime, reason)
+    assert strategy.is_pair_locked(pair)
+    strategy.unlock_reason(reason)
+    assert not strategy.is_pair_locked(pair)
+
     pair = 'BTC/USDT'
     # Lock until 14:30
     lock_time = datetime(2020, 5, 1, 14, 30, 0, tzinfo=timezone.utc)

tests/strategy/test_strategy_loading.py

@@ -62,8 +62,8 @@ def test_load_strategy(default_conf, result):
 
 
 def test_load_strategy_base64(result, caplog, default_conf):
-    with (Path(__file__).parents[2] / 'freqtrade/templates/sample_strategy.py').open("rb") as file:
+    filepath = Path(__file__).parents[2] / 'freqtrade/templates/sample_strategy.py'
-        encoded_string = urlsafe_b64encode(file.read()).decode("utf-8")
+    encoded_string = urlsafe_b64encode(filepath.read_bytes()).decode("utf-8")
     default_conf.update({'strategy': 'SampleStrategy:{}'.format(encoded_string)})
 
     strategy = StrategyResolver.load_strategy(default_conf)