remove add pair to column from docs, fix keyerror bug and adjust hybrid strategy example

docs/freqai-feature-engineering.md

@@ -2,7 +2,7 @@
 
 ## Defining the features
 
-Low level feature engineering is performed in the user strategy within a set of functions called `feature_engineering_*`. These function set the `base features` such as, `RSI`, `MFI`, `EMA`, `SMA`, time of day, volume, etc. The `base features` can be custom indicators or they can be imported from any technical-analysis library that you can find. One important syntax rule is that all `base features` string names defined within `feature_engineering_*` functions must be prepended with `%-{pair}`. FreqAI is equipped with a set of functions to simplify rapid large-scale feature engineering:
+Low level feature engineering is performed in the user strategy within a set of functions called `feature_engineering_*`. These function set the `base features` such as, `RSI`, `MFI`, `EMA`, `SMA`, time of day, volume, etc. The `base features` can be custom indicators or they can be imported from any technical-analysis library that you can find. FreqAI is equipped with a set of functions to simplify rapid large-scale feature engineering:
 
 |  Function | Description |
 |---------------|-------------|
@@ -11,10 +11,6 @@ Low level feature engineering is performed in the user strategy within a set of
 | `feature_engineering_standard()` | This optional function will be called once with the dataframe of the base timeframe. This is the final function to be called, which means that the dataframe entering this function will contain all the features and columns created by all other `feature_engineering_expand` functions. This function is a good place to do custom exotic feature extractions (e.g. tsfresh). This function is also a good place for any feature that should not be auto-expanded upon (e.g. day of the week).
 | `set_freqai_targets()` | Required function to set the targets for the model. All targets must be prepended with `&` to be recognized by the FreqAI internals.
 
-
-!!! Note
-    Adding the full pair string, e.g. XYZ/USD, in the feature name enables improved performance for dataframe caching on the backend. If you decide *not* to add the full pair string in the feature string, FreqAI will operate in a reduced performance mode.
-
 Meanwhile, high level feature engineering is handled within `"feature_parameters":{}` in the FreqAI config. Within this file, it is possible to decide large scale feature expansions on top of the `base_features` such as "including correlated pairs" or "including informative timeframes" or even "including recent candles."
 
 It is advisable to start from the template `feature_engineering_*` functions in the source provided example strategy (found in `templates/FreqaiExampleStrategy.py`) to ensure that the feature definitions are following the correct conventions. Here is an example of how to set the indicators and labels in the strategy:

docs/freqai-running.md

@@ -135,7 +135,7 @@ freqtrade hyperopt --hyperopt-loss SharpeHyperOptLoss --strategy FreqaiExampleSt
 `hyperopt` requires you to have the data pre-downloaded in the same fashion as if you were doing [backtesting](#backtesting). In addition, you must consider some restrictions when trying to hyperopt FreqAI strategies:
 
 - The `--analyze-per-epoch` hyperopt parameter is not compatible with FreqAI.
-- It's not possible to hyperopt indicators in the `populate_any_indicators()` function. This means that you cannot optimize model parameters using hyperopt. Apart from this exception, it is possible to optimize all other [spaces](hyperopt.md#running-hyperopt-with-smaller-search-space).
+- It's not possible to hyperopt indicators in the `feature_engineering_*()` and `set_freqai_targets()` functions. This means that you cannot optimize model parameters using hyperopt. Apart from this exception, it is possible to optimize all other [spaces](hyperopt.md#running-hyperopt-with-smaller-search-space).
 - The backtesting instructions also apply to hyperopt.
 
 The best method for combining hyperopt and FreqAI is to focus on hyperopting entry/exit thresholds/criteria. You need to focus on hyperopting parameters that are not used in your features. For example, you should not try to hyperopt rolling window lengths in the feature creation, or any part of the FreqAI config which changes predictions. In order to efficiently hyperopt the FreqAI strategy, FreqAI stores predictions as dataframes and reuses them. Hence the requirement to hyperopt entry/exit thresholds/criteria only.

freqtrade/freqai/data_kitchen.py

@@ -1311,10 +1311,10 @@ class FreqaiDataKitchen:
             for tf in tfs:
                 if tf not in base_dataframes:
                     base_dataframes[tf] = pd.DataFrame()
-                if not corr_dataframes.keys():
+                for p in pairs:
-                    for p in pairs:
+                    if p not in corr_dataframes:
-                        if p not in corr_dataframes:
+                        corr_dataframes[p] = {}
-                            corr_dataframes[p] = {}
+                    if tf not in corr_dataframes[p]:
                         corr_dataframes[p][tf] = pd.DataFrame()
 
             if not prediction_dataframe.empty:

freqtrade/freqai/freqai_interface.py

@@ -272,11 +272,7 @@ class IFreqaiModel(ABC):
         self.pair_it += 1
         train_it = 0
         pair = metadata["pair"]
-
         populate_indicators = True
-        timerange = TimeRange.parse_timerange(self.dk.full_timerange)
-        self.dd.load_all_pair_histories(timerange, self.dk)
-        corr_df, base_df = self.dd.get_base_and_corr_dataframes(timerange, pair, dk)
 
         # Loop enforcing the sliding window training/backtesting paradigm
         # tr_train is the training time range e.g. 1 historical month
@@ -312,6 +308,9 @@ class IFreqaiModel(ABC):
                 dk.append_predictions(append_df)
             else:
                 if populate_indicators:
+                    timerange = TimeRange.parse_timerange(self.dk.full_timerange)
+                    self.dd.load_all_pair_histories(timerange, self.dk)
+                    corr_df, base_df = self.dd.get_base_and_corr_dataframes(timerange, pair, dk)
                     dataframe = self.dk.use_strategy_to_populate_indicators(
                         strategy, prediction_dataframe=dataframe, pair=metadata["pair"],
                         corr_dataframes=corr_df, base_dataframes=base_df

freqtrade/templates/FreqaiExampleHybridStrategy.py

@@ -95,65 +95,132 @@ class FreqaiExampleHybridStrategy(IStrategy):
     short_rsi = IntParameter(low=51, high=100, default=70, space='sell', optimize=True, load=True)
     exit_short_rsi = IntParameter(low=1, high=50, default=30, space='buy', optimize=True, load=True)
 
-    # FreqAI required function, user can add or remove indicators, but general structure
+    def feature_engineering_expand_all(self, dataframe, period, **kwargs):
-    # must stay the same.
-    def populate_any_indicators(
-        self, pair, df, tf, informative=None, set_generalized_indicators=False
-    ):
         """
-        User feeds these indicators to FreqAI to train a classifier to decide
+        *Only functional with FreqAI enabled strategies*
-        if the market will go up or down.
+        This function will automatically expand the defined features on the config defined
+        `indicator_periods_candles`, `include_timeframes`, `include_shifted_candles`, and
+        `include_corr_pairs`. In other words, a single feature defined in this function
+        will automatically expand to a total of
+        `indicator_periods_candles` * `include_timeframes` * `include_shifted_candles` *
+        `include_corr_pairs` numbers of features added to the model.
 
-        :param pair: pair to be used as informative
+        All features must be prepended with `%` to be recognized by FreqAI internals.
-        :param df: strategy dataframe which will receive merges from informatives
+
-        :param tf: timeframe of the dataframe which will modify the feature names
+        More details on how these config defined parameters accelerate feature engineering
-        :param informative: the dataframe associated with the informative pair
+        in the documentation at:
+
+        https://www.freqtrade.io/en/latest/freqai-parameter-table/#feature-parameters
+
+        https://www.freqtrade.io/en/latest/freqai-feature-engineering/#defining-the-features
+
+        :param df: strategy dataframe which will receive the features
+        :param period: period of the indicator - usage example:
+        dataframe["%-ema-period"] = ta.EMA(dataframe, timeperiod=period)
         """
 
-        if informative is None:
+        dataframe["%-rsi-period"] = ta.RSI(dataframe, timeperiod=period)
-            informative = self.dp.get_pair_dataframe(pair, tf)
+        dataframe["%-mfi-period"] = ta.MFI(dataframe, timeperiod=period)
+        dataframe["%-adx-period"] = ta.ADX(dataframe, timeperiod=period)
+        dataframe["%-sma-period"] = ta.SMA(dataframe, timeperiod=period)
+        dataframe["%-ema-period"] = ta.EMA(dataframe, timeperiod=period)
 
-        # first loop is automatically duplicating indicators for time periods
+        bollinger = qtpylib.bollinger_bands(
-        for t in self.freqai_info["feature_parameters"]["indicator_periods_candles"]:
+            qtpylib.typical_price(dataframe), window=period, stds=2.2
+        )
+        dataframe["bb_lowerband-period"] = bollinger["lower"]
+        dataframe["bb_middleband-period"] = bollinger["mid"]
+        dataframe["bb_upperband-period"] = bollinger["upper"]
 
-            t = int(t)
+        dataframe["%-bb_width-period"] = (
-            informative[f"%-{pair}rsi-period_{t}"] = ta.RSI(informative, timeperiod=t)
+            dataframe["bb_upperband-period"]
-            informative[f"%-{pair}mfi-period_{t}"] = ta.MFI(informative, timeperiod=t)
+            - dataframe["bb_lowerband-period"]
-            informative[f"%-{pair}adx-period_{t}"] = ta.ADX(informative, timeperiod=t)
+        ) / dataframe["bb_middleband-period"]
-            informative[f"%-{pair}sma-period_{t}"] = ta.SMA(informative, timeperiod=t)
+        dataframe["%-close-bb_lower-period"] = (
-            informative[f"%-{pair}ema-period_{t}"] = ta.EMA(informative, timeperiod=t)
+            dataframe["close"] / dataframe["bb_lowerband-period"]
-            informative[f"%-{pair}roc-period_{t}"] = ta.ROC(informative, timeperiod=t)
+        )
-            informative[f"%-{pair}relative_volume-period_{t}"] = (
-                informative["volume"] / informative["volume"].rolling(t).mean()
-            )
 
-        # FreqAI needs the following lines in order to detect features and automatically
+        dataframe["%-roc-period"] = ta.ROC(dataframe, timeperiod=period)
-        # expand upon them.
-        indicators = [col for col in informative if col.startswith("%")]
-        # This loop duplicates and shifts all indicators to add a sense of recency to data
-        for n in range(self.freqai_info["feature_parameters"]["include_shifted_candles"] + 1):
-            if n == 0:
-                continue
-            informative_shift = informative[indicators].shift(n)
-            informative_shift = informative_shift.add_suffix("_shift-" + str(n))
-            informative = pd.concat((informative, informative_shift), axis=1)
 
-        df = merge_informative_pair(df, informative, self.config["timeframe"], tf, ffill=True)
+        dataframe["%-relative_volume-period"] = (
-        skip_columns = [
+            dataframe["volume"] / dataframe["volume"].rolling(period).mean()
-            (s + "_" + tf) for s in ["date", "open", "high", "low", "close", "volume"]
+        )
-        ]
-        df = df.drop(columns=skip_columns)
 
-        # User can set the "target" here (in present case it is the
+        return dataframe
-        # "up" or "down")
-        if set_generalized_indicators:
-            # User "looks into the future" here to figure out if the future
-            # will be "up" or "down". This same column name is available to
-            # the user
-            df['&s-up_or_down'] = np.where(df["close"].shift(-50) >
-                                           df["close"], 'up', 'down')
 
-        return df
+    def feature_engineering_expand_basic(self, dataframe, **kwargs):
+        """
+        *Only functional with FreqAI enabled strategies*
+        This function will automatically expand the defined features on the config defined
+        `include_timeframes`, `include_shifted_candles`, and `include_corr_pairs`.
+        In other words, a single feature defined in this function
+        will automatically expand to a total of
+        `include_timeframes` * `include_shifted_candles` * `include_corr_pairs`
+        numbers of features added to the model.
+
+        Features defined here will *not* be automatically duplicated on user defined
+        `indicator_periods_candles`
+
+        All features must be prepended with `%` to be recognized by FreqAI internals.
+
+        More details on how these config defined parameters accelerate feature engineering
+        in the documentation at:
+
+        https://www.freqtrade.io/en/latest/freqai-parameter-table/#feature-parameters
+
+        https://www.freqtrade.io/en/latest/freqai-feature-engineering/#defining-the-features
+
+        :param df: strategy dataframe which will receive the features
+        dataframe["%-pct-change"] = dataframe["close"].pct_change()
+        dataframe["%-ema-200"] = ta.EMA(dataframe, timeperiod=200)
+        """
+        dataframe["%-pct-change"] = dataframe["close"].pct_change()
+        dataframe["%-raw_volume"] = dataframe["volume"]
+        dataframe["%-raw_price"] = dataframe["close"]
+        return dataframe
+
+    def feature_engineering_standard(self, dataframe, **kwargs):
+        """
+        *Only functional with FreqAI enabled strategies*
+        This optional function will be called once with the dataframe of the base timeframe.
+        This is the final function to be called, which means that the dataframe entering this
+        function will contain all the features and columns created by all other
+        freqai_feature_engineering_* functions.
+
+        This function is a good place to do custom exotic feature extractions (e.g. tsfresh).
+        This function is a good place for any feature that should not be auto-expanded upon
+        (e.g. day of the week).
+
+        All features must be prepended with `%` to be recognized by FreqAI internals.
+
+        More details about feature engineering available:
+
+        https://www.freqtrade.io/en/latest/freqai-feature-engineering
+
+        :param df: strategy dataframe which will receive the features
+        usage example: dataframe["%-day_of_week"] = (dataframe["date"].dt.dayofweek + 1) / 7
+        """
+        dataframe["%-day_of_week"] = dataframe["date"].dt.dayofweek
+        dataframe["%-hour_of_day"] = dataframe["date"].dt.hour
+        return dataframe
+
+    def set_freqai_targets(self, dataframe, **kwargs):
+        """
+        *Only functional with FreqAI enabled strategies*
+        Required function to set the targets for the model.
+        All targets must be prepended with `&` to be recognized by the FreqAI internals.
+
+        More details about feature engineering available:
+
+        https://www.freqtrade.io/en/latest/freqai-feature-engineering
+
+        :param df: strategy dataframe which will receive the targets
+        usage example: dataframe["&-target"] = dataframe["close"].shift(-1) / dataframe["close"]
+        """
+        dataframe['&s-up_or_down'] = np.where(dataframe["close"].shift(-50) >
+                                        dataframe["close"], 'up', 'down')
+
+        return dataframe
 
     # flake8: noqa: C901
     def populate_indicators(self, dataframe: DataFrame, metadata: dict) -> DataFrame: