Merge pull request #3497 from freqtrade/keep_dataframe_noapi

Analyze dataframe and keep it until the next analysis
This commit is contained in:
hroff-1902 2020-07-05 13:46:02 +03:00 committed by GitHub
commit 839b3340e6
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
16 changed files with 603 additions and 123 deletions

58
docs/bot-basics.md Normal file
View File

@ -0,0 +1,58 @@
# Freqtrade basics
This page provides you some basic concepts on how Freqtrade works and operates.
## Freqtrade terminology
* Trade: Open position.
* Open Order: Order which is currently placed on the exchange, and is not yet complete.
* Pair: Tradable pair, usually in the format of Quote/Base (e.g. XRP/USDT).
* Timeframe: Candle length to use (e.g. `"5m"`, `"1h"`, ...).
* Indicators: Technical indicators (SMA, EMA, RSI, ...).
* Limit order: Limit orders which execute at the defined limit price or better.
* Market order: Guaranteed to fill, may move price depending on the order size.
## Fee handling
All profit calculations of Freqtrade include fees. For Backtesting / Hyperopt / Dry-run modes, the exchange default fee is used (lowest tier on the exchange). For live operations, fees are used as applied by the exchange (this includes BNB rebates etc.).
## Bot execution logic
Starting freqtrade in dry-run or live mode (using `freqtrade trade`) will start the bot and start the bot iteration loop.
By default, loop runs every few seconds (`internals.process_throttle_secs`) and does roughly the following in the following sequence:
* Fetch open trades from persistence.
* Calculate current list of tradable pairs.
* Download ohlcv data for the pairlist including all [informative pairs](strategy-customization.md#get-data-for-non-tradeable-pairs)
This step is only executed once per Candle to avoid unnecessary network traffic.
* Call `bot_loop_start()` strategy callback.
* Analyze strategy per pair.
* Call `populate_indicators()`
* Call `populate_buy_trend()`
* Call `populate_sell_trend()`
* Check timeouts for open orders.
* Calls `check_buy_timeout()` strategy callback for open buy orders.
* Calls `check_sell_timeout()` strategy callback for open sell orders.
* Verifies existing positions and eventually places sell orders.
* Considers stoploss, ROI and sell-signal.
* Determine sell-price based on `ask_strategy` configuration setting.
* Before a sell order is placed, `confirm_trade_exit()` strategy callback is called.
* Check if trade-slots are still available (if `max_open_trades` is reached).
* Verifies buy signal trying to enter new positions.
* Determine buy-price based on `bid_strategy` configuration setting.
* Before a buy order is placed, `confirm_trade_entry()` strategy callback is called.
This loop will be repeated again and again until the bot is stopped.
## Backtesting / Hyperopt execution logic
[backtesting](backtesting.md) or [hyperopt](hyperopt.md) do only part of the above logic, since most of the trading operations are fully simulated.
* Load historic data for configured pairlist.
* Calculate indicators (calls `populate_indicators()`).
* Calls `populate_buy_trend()` and `populate_sell_trend()`
* Loops per candle simulating entry and exit points.
* Generate backtest report output
!!! Note
Both Backtesting and Hyperopt include exchange default Fees in the calculation. Custom fees can be passed to backtesting / hyperopt by specifying the `--fee` argument.

View File

@ -498,8 +498,3 @@ After you run Hyperopt for the desired amount of epochs, you can later list all
Once the optimized strategy has been implemented into your strategy, you should backtest this strategy to make sure everything is working as expected.
To achieve same results (number of trades, their durations, profit, etc.) than during Hyperopt, please use same set of arguments `--dmmp`/`--disable-max-market-positions` and `--eps`/`--enable-position-stacking` for Backtesting.
## Next Step
Now you have a perfect bot and want to control it from Telegram. Your
next step is to learn the [Telegram usage](telegram-usage.md).

View File

@ -1,7 +1,12 @@
# Advanced Strategies
This page explains some advanced concepts available for strategies.
If you're just getting started, please be familiar with the methods described in the [Strategy Customization](strategy-customization.md) documentation first.
If you're just getting started, please be familiar with the methods described in the [Strategy Customization](strategy-customization.md) documentation and with the [Freqtrade basics](bot-basics.md) first.
[Freqtrade basics](bot-basics.md) describes in which sequence each method described below is called, which can be helpful to understand which method to use for your custom needs.
!!! Note
All callback methods described below should only be implemented in a strategy if they are actually used.
## Custom order timeout rules
@ -89,3 +94,108 @@ class Awesomestrategy(IStrategy):
return True
return False
```
## Bot loop start callback
A simple callback which is called once at the start of every bot throttling iteration.
This can be used to perform calculations which are pair independent (apply to all pairs), loading of external data, etc.
``` python
import requests
class Awesomestrategy(IStrategy):
# ... populate_* methods
def bot_loop_start(self, **kwargs) -> None:
"""
Called at the start of the bot iteration (one loop).
Might be used to perform pair-independent tasks
(e.g. gather some remote resource for comparison)
:param **kwargs: Ensure to keep this here so updates to this won't break your strategy.
"""
if self.config['runmode'].value in ('live', 'dry_run'):
# Assign this to the class by using self.*
# can then be used by populate_* methods
self.remote_data = requests.get('https://some_remote_source.example.com')
```
## Bot order confirmation
### Trade entry (buy order) confirmation
`confirm_trade_entry()` can be used to abort a trade entry at the latest second (maybe because the price is not what we expect).
``` python
class Awesomestrategy(IStrategy):
# ... populate_* methods
def confirm_trade_entry(self, pair: str, order_type: str, amount: float, rate: float,
time_in_force: str, **kwargs) -> bool:
"""
Called right before placing a buy order.
Timing for this function is critical, so avoid doing heavy computations or
network requests in this method.
For full documentation please go to https://www.freqtrade.io/en/latest/strategy-advanced/
When not implemented by a strategy, returns True (always confirming).
:param pair: Pair that's about to be bought.
:param order_type: Order type (as configured in order_types). usually limit or market.
:param amount: Amount in target (quote) currency that's going to be traded.
:param rate: Rate that's going to be used when using limit orders
:param time_in_force: Time in force. Defaults to GTC (Good-til-cancelled).
:param **kwargs: Ensure to keep this here so updates to this won't break your strategy.
:return bool: When True is returned, then the buy-order is placed on the exchange.
False aborts the process
"""
return True
```
### Trade exit (sell order) confirmation
`confirm_trade_exit()` can be used to abort a trade exit (sell) at the latest second (maybe because the price is not what we expect).
``` python
from freqtrade.persistence import Trade
class Awesomestrategy(IStrategy):
# ... populate_* methods
def confirm_trade_exit(self, pair: str, trade: Trade, order_type: str, amount: float,
rate: float, time_in_force: str, sell_reason: str, **kwargs) -> bool:
"""
Called right before placing a regular sell order.
Timing for this function is critical, so avoid doing heavy computations or
network requests in this method.
For full documentation please go to https://www.freqtrade.io/en/latest/strategy-advanced/
When not implemented by a strategy, returns True (always confirming).
:param pair: Pair that's about to be sold.
:param order_type: Order type (as configured in order_types). usually limit or market.
:param amount: Amount in quote currency.
:param rate: Rate that's going to be used when using limit orders
:param time_in_force: Time in force. Defaults to GTC (Good-til-cancelled).
:param sell_reason: Sell reason.
Can be any of ['roi', 'stop_loss', 'stoploss_on_exchange', 'trailing_stop_loss',
'sell_signal', 'force_sell', 'emergency_sell']
:param **kwargs: Ensure to keep this here so updates to this won't break your strategy.
:return bool: When True is returned, then the sell-order is placed on the exchange.
False aborts the process
"""
if sell_reason == 'force_sell' and trade.calc_profit_ratio(rate) < 0:
# Reject force-sells with negative profit
# This is just a sample, please adjust to your needs
# (this does not necessarily make sense, assuming you know when you're force-selling)
return False
return True
```

View File

@ -1,6 +1,8 @@
# Strategy Customization
This page explains where to customize your strategies, and add new indicators.
This page explains how to customize your strategies, add new indicators and set up trading rules.
Please familiarize yourself with [Freqtrade basics](bot-basics.md) first, which provides overall info on how the bot operates.
## Install a custom strategy file
@ -366,6 +368,7 @@ Please always check the mode of operation to select the correct method to get da
- [`available_pairs`](#available_pairs) - Property with tuples listing cached pairs with their intervals (pair, interval).
- [`current_whitelist()`](#current_whitelist) - Returns a current list of whitelisted pairs. Useful for accessing dynamic whitelists (ie. VolumePairlist)
- [`get_pair_dataframe(pair, timeframe)`](#get_pair_dataframepair-timeframe) - This is a universal method, which returns either historical data (for backtesting) or cached live data (for the Dry-Run and Live-Run modes).
- [`get_analyzed_dataframe(pair, timeframe)`](#get_analyzed_dataframepair-timeframe) - Returns the analyzed dataframe (after calling `populate_indicators()`, `populate_buy()`, `populate_sell()`) and the time of the latest analysis.
- `historic_ohlcv(pair, timeframe)` - Returns historical data stored on disk.
- `market(pair)` - Returns market data for the pair: fees, limits, precisions, activity flag, etc. See [ccxt documentation](https://github.com/ccxt/ccxt/wiki/Manual#markets) for more details on the Market data structure.
- `ohlcv(pair, timeframe)` - Currently cached candle (OHLCV) data for the pair, returns DataFrame or empty DataFrame.
@ -384,6 +387,7 @@ if self.dp:
```
#### *current_whitelist()*
Imagine you've developed a strategy that trades the `5m` timeframe using signals generated from a `1d` timeframe on the top 10 volume pairs by volume.
The strategy might look something like this:
@ -431,13 +435,32 @@ if self.dp:
```
!!! Warning "Warning about backtesting"
Be carefull when using dataprovider in backtesting. `historic_ohlcv()` (and `get_pair_dataframe()`
Be careful when using dataprovider in backtesting. `historic_ohlcv()` (and `get_pair_dataframe()`
for the backtesting runmode) provides the full time-range in one go,
so please be aware of it and make sure to not "look into the future" to avoid surprises when running in dry/live mode).
!!! Warning "Warning in hyperopt"
This option cannot currently be used during hyperopt.
#### *get_analyzed_dataframe(pair, timeframe)*
This method is used by freqtrade internally to determine the last signal.
It can also be used in specific callbacks to get the signal that caused the action (see [Advanced Strategy Documentation](strategy-advanced.md) for more details on available callbacks).
``` python
# fetch current dataframe
if self.dp:
dataframe, last_updated = self.dp.get_analyzed_dataframe(pair=metadata['pair'],
timeframe=self.ticker_interval)
```
!!! Note "No data available"
Returns an empty dataframe if the requested pair was not cached.
This should not happen when using whitelisted pairs.
!!! Warning "Warning in hyperopt"
This option cannot currently be used during hyperopt.
#### *orderbook(pair, maximum)*
``` python

View File

@ -339,4 +339,5 @@ CANCEL_REASON = {
}
# List of pairs with their timeframes
ListPairsWithTimeframes = List[Tuple[str, str]]
PairWithTimeframe = Tuple[str, str]
ListPairsWithTimeframes = List[PairWithTimeframe]

View File

@ -5,16 +5,17 @@ including ticker and orderbook data, live and historical candle (OHLCV) data
Common Interface for bot and strategy to access data.
"""
import logging
from typing import Any, Dict, List, Optional
from datetime import datetime, timezone
from typing import Any, Dict, List, Optional, Tuple
from arrow import Arrow
from pandas import DataFrame
from freqtrade.constants import ListPairsWithTimeframes, PairWithTimeframe
from freqtrade.data.history import load_pair_history
from freqtrade.exceptions import ExchangeError, OperationalException
from freqtrade.exchange import Exchange
from freqtrade.state import RunMode
from freqtrade.constants import ListPairsWithTimeframes
logger = logging.getLogger(__name__)
@ -25,6 +26,18 @@ class DataProvider:
self._config = config
self._exchange = exchange
self._pairlists = pairlists
self.__cached_pairs: Dict[PairWithTimeframe, Tuple[DataFrame, datetime]] = {}
def _set_cached_df(self, pair: str, timeframe: str, dataframe: DataFrame) -> None:
"""
Store cached Dataframe.
Using private method as this should never be used by a user
(but the class is exposed via `self.dp` to the strategy)
:param pair: pair to get the data for
:param timeframe: Timeframe to get data for
:param dataframe: analyzed dataframe
"""
self.__cached_pairs[(pair, timeframe)] = (dataframe, Arrow.utcnow().datetime)
def refresh(self,
pairlist: ListPairsWithTimeframes,
@ -89,6 +102,20 @@ class DataProvider:
logger.warning(f"No data found for ({pair}, {timeframe}).")
return data
def get_analyzed_dataframe(self, pair: str, timeframe: str) -> Tuple[DataFrame, datetime]:
"""
:param pair: pair to get the data for
:param timeframe: timeframe to get data for
:return: Tuple of (Analyzed Dataframe, lastrefreshed) for the requested pair / timeframe
combination.
Returns empty dataframe and Epoch 0 (1970-01-01) if no dataframe was cached.
"""
if (pair, timeframe) in self.__cached_pairs:
return self.__cached_pairs[(pair, timeframe)]
else:
return (DataFrame(), datetime.fromtimestamp(0, tz=timezone.utc))
def market(self, pair: str) -> Optional[Dict[str, Any]]:
"""
Return market data for the pair

View File

@ -153,6 +153,10 @@ class FreqtradeBot:
self.dataprovider.refresh(self.pairlists.create_pair_list(self.active_pair_whitelist),
self.strategy.informative_pairs())
strategy_safe_wrapper(self.strategy.bot_loop_start, supress_error=True)()
self.strategy.analyze(self.active_pair_whitelist)
with self._sell_lock:
# Check and handle any timed out open orders
self.check_handle_timedout()
@ -440,9 +444,8 @@ class FreqtradeBot:
return False
# running get_signal on historical data fetched
(buy, sell) = self.strategy.get_signal(
pair, self.strategy.timeframe,
self.dataprovider.ohlcv(pair, self.strategy.timeframe))
analyzed_df, _ = self.dataprovider.get_analyzed_dataframe(pair, self.strategy.timeframe)
(buy, sell) = self.strategy.get_signal(pair, self.strategy.timeframe, analyzed_df)
if buy and not sell:
stake_amount = self.get_trade_stake_amount(pair)
@ -515,6 +518,12 @@ class FreqtradeBot:
amount = stake_amount / buy_limit_requested
order_type = self.strategy.order_types['buy']
if not strategy_safe_wrapper(self.strategy.confirm_trade_entry, default_retval=True)(
pair=pair, order_type=order_type, amount=amount, rate=buy_limit_requested,
time_in_force=time_in_force):
logger.info(f"User requested abortion of buying {pair}")
return False
order = self.exchange.buy(pair=pair, ordertype=order_type,
amount=amount, rate=buy_limit_requested,
time_in_force=time_in_force)
@ -717,9 +726,10 @@ class FreqtradeBot:
if (config_ask_strategy.get('use_sell_signal', True) or
config_ask_strategy.get('ignore_roi_if_buy_signal', False)):
(buy, sell) = self.strategy.get_signal(
trade.pair, self.strategy.timeframe,
self.dataprovider.ohlcv(trade.pair, self.strategy.timeframe))
analyzed_df, _ = self.dataprovider.get_analyzed_dataframe(trade.pair,
self.strategy.timeframe)
(buy, sell) = self.strategy.get_signal(trade.pair, self.strategy.timeframe, analyzed_df)
if config_ask_strategy.get('use_order_book', False):
order_book_min = config_ask_strategy.get('order_book_min', 1)
@ -1097,12 +1107,20 @@ class FreqtradeBot:
order_type = self.strategy.order_types.get("emergencysell", "market")
amount = self._safe_sell_amount(trade.pair, trade.amount)
time_in_force = self.strategy.order_time_in_force['sell']
if not strategy_safe_wrapper(self.strategy.confirm_trade_exit, default_retval=True)(
pair=trade.pair, trade=trade, order_type=order_type, amount=amount, rate=limit,
time_in_force=time_in_force,
sell_reason=sell_reason.value):
logger.info(f"User requested abortion of selling {trade.pair}")
return False
# Execute sell and update trade record
order = self.exchange.sell(pair=str(trade.pair),
ordertype=order_type,
amount=amount, rate=limit,
time_in_force=self.strategy.order_time_in_force['sell']
time_in_force=time_in_force
)
trade.open_order_id = order['id']

View File

@ -7,20 +7,19 @@ import warnings
from abc import ABC, abstractmethod
from datetime import datetime, timezone
from enum import Enum
from typing import Dict, NamedTuple, Optional, Tuple
from typing import Dict, List, NamedTuple, Optional, Tuple
import arrow
from pandas import DataFrame
from freqtrade.constants import ListPairsWithTimeframes
from freqtrade.data.dataprovider import DataProvider
from freqtrade.exceptions import StrategyError
from freqtrade.exceptions import StrategyError, OperationalException
from freqtrade.exchange import timeframe_to_minutes
from freqtrade.persistence import Trade
from freqtrade.strategy.strategy_wrapper import strategy_safe_wrapper
from freqtrade.constants import ListPairsWithTimeframes
from freqtrade.wallets import Wallets
logger = logging.getLogger(__name__)
@ -191,6 +190,63 @@ class IStrategy(ABC):
"""
return False
def bot_loop_start(self, **kwargs) -> None:
"""
Called at the start of the bot iteration (one loop).
Might be used to perform pair-independent tasks
(e.g. gather some remote resource for comparison)
:param **kwargs: Ensure to keep this here so updates to this won't break your strategy.
"""
pass
def confirm_trade_entry(self, pair: str, order_type: str, amount: float, rate: float,
time_in_force: str, **kwargs) -> bool:
"""
Called right before placing a buy order.
Timing for this function is critical, so avoid doing heavy computations or
network requests in this method.
For full documentation please go to https://www.freqtrade.io/en/latest/strategy-advanced/
When not implemented by a strategy, returns True (always confirming).
:param pair: Pair that's about to be bought.
:param order_type: Order type (as configured in order_types). usually limit or market.
:param amount: Amount in target (quote) currency that's going to be traded.
:param rate: Rate that's going to be used when using limit orders
:param time_in_force: Time in force. Defaults to GTC (Good-til-cancelled).
:param **kwargs: Ensure to keep this here so updates to this won't break your strategy.
:return bool: When True is returned, then the buy-order is placed on the exchange.
False aborts the process
"""
return True
def confirm_trade_exit(self, pair: str, trade: Trade, order_type: str, amount: float,
rate: float, time_in_force: str, sell_reason: str, **kwargs) -> bool:
"""
Called right before placing a regular sell order.
Timing for this function is critical, so avoid doing heavy computations or
network requests in this method.
For full documentation please go to https://www.freqtrade.io/en/latest/strategy-advanced/
When not implemented by a strategy, returns True (always confirming).
:param pair: Pair that's about to be sold.
:param trade: trade object.
:param order_type: Order type (as configured in order_types). usually limit or market.
:param amount: Amount in quote currency.
:param rate: Rate that's going to be used when using limit orders
:param time_in_force: Time in force. Defaults to GTC (Good-til-cancelled).
:param sell_reason: Sell reason.
Can be any of ['roi', 'stop_loss', 'stoploss_on_exchange', 'trailing_stop_loss',
'sell_signal', 'force_sell', 'emergency_sell']
:param **kwargs: Ensure to keep this here so updates to this won't break your strategy.
:return bool: When True is returned, then the sell-order is placed on the exchange.
False aborts the process
"""
return True
def informative_pairs(self) -> ListPairsWithTimeframes:
"""
Define additional, informative pair/interval combinations to be cached from the exchange.
@ -204,6 +260,10 @@ class IStrategy(ABC):
"""
return []
###
# END - Intended to be overridden by strategy
###
def get_strategy_name(self) -> str:
"""
Returns strategy class name
@ -273,6 +333,8 @@ class IStrategy(ABC):
# Defs that only make change on new candle data.
dataframe = self.analyze_ticker(dataframe, metadata)
self._last_candle_seen_per_pair[pair] = dataframe.iloc[-1]['date']
if self.dp:
self.dp._set_cached_df(pair, self.timeframe, dataframe)
else:
logger.debug("Skipping TA Analysis for already analyzed candle")
dataframe['buy'] = 0
@ -284,13 +346,53 @@ class IStrategy(ABC):
return dataframe
def analyze_pair(self, pair: str) -> None:
"""
Fetch data for this pair from dataprovider and analyze.
Stores the dataframe into the dataprovider.
The analyzed dataframe is then accessible via `dp.get_analyzed_dataframe()`.
:param pair: Pair to analyze.
"""
if not self.dp:
raise OperationalException("DataProvider not found.")
dataframe = self.dp.ohlcv(pair, self.timeframe)
if not isinstance(dataframe, DataFrame) or dataframe.empty:
logger.warning('Empty candle (OHLCV) data for pair %s', pair)
return
try:
df_len, df_close, df_date = self.preserve_df(dataframe)
dataframe = strategy_safe_wrapper(
self._analyze_ticker_internal, message=""
)(dataframe, {'pair': pair})
self.assert_df(dataframe, df_len, df_close, df_date)
except StrategyError as error:
logger.warning(f"Unable to analyze candle (OHLCV) data for pair {pair}: {error}")
return
if dataframe.empty:
logger.warning('Empty dataframe for pair %s', pair)
return
def analyze(self, pairs: List[str]) -> None:
"""
Analyze all pairs using analyze_pair().
:param pairs: List of pairs to analyze
"""
for pair in pairs:
self.analyze_pair(pair)
@staticmethod
def preserve_df(dataframe: DataFrame) -> Tuple[int, float, datetime]:
""" keep some data for dataframes """
return len(dataframe), dataframe["close"].iloc[-1], dataframe["date"].iloc[-1]
def assert_df(self, dataframe: DataFrame, df_len: int, df_close: float, df_date: datetime):
""" make sure data is unmodified """
"""
Ensure dataframe (length, last candle) was not modified, and has all elements we need.
"""
message = ""
if df_len != len(dataframe):
message = "length"
@ -304,31 +406,17 @@ class IStrategy(ABC):
else:
raise StrategyError(f"Dataframe returned from strategy has mismatching {message}.")
def get_signal(self, pair: str, interval: str, dataframe: DataFrame) -> Tuple[bool, bool]:
def get_signal(self, pair: str, timeframe: str, dataframe: DataFrame) -> Tuple[bool, bool]:
"""
Calculates current signal based several technical analysis indicators
Calculates current signal based based on the buy / sell columns of the dataframe.
Used by Bot to get the signal to buy or sell
:param pair: pair in format ANT/BTC
:param interval: Interval to use (in min)
:param dataframe: Dataframe to analyze
:param timeframe: timeframe to use
:param dataframe: Analyzed dataframe to get signal from.
:return: (Buy, Sell) A bool-tuple indicating buy/sell signal
"""
if not isinstance(dataframe, DataFrame) or dataframe.empty:
logger.warning('Empty candle (OHLCV) data for pair %s', pair)
return False, False
try:
df_len, df_close, df_date = self.preserve_df(dataframe)
dataframe = strategy_safe_wrapper(
self._analyze_ticker_internal, message=""
)(dataframe, {'pair': pair})
self.assert_df(dataframe, df_len, df_close, df_date)
except StrategyError as error:
logger.warning(f"Unable to analyze candle (OHLCV) data for pair {pair}: {error}")
return False, False
if dataframe.empty:
logger.warning('Empty dataframe for pair %s', pair)
logger.warning(f'Empty candle (OHLCV) data for pair {pair}')
return False, False
latest_date = dataframe['date'].max()
@ -337,24 +425,18 @@ class IStrategy(ABC):
latest_date = arrow.get(latest_date)
# Check if dataframe is out of date
interval_minutes = timeframe_to_minutes(interval)
timeframe_minutes = timeframe_to_minutes(timeframe)
offset = self.config.get('exchange', {}).get('outdated_offset', 5)
if latest_date < (arrow.utcnow().shift(minutes=-(interval_minutes * 2 + offset))):
if latest_date < (arrow.utcnow().shift(minutes=-(timeframe_minutes * 2 + offset))):
logger.warning(
'Outdated history for pair %s. Last tick is %s minutes old',
pair,
(arrow.utcnow() - latest_date).seconds // 60
pair, (arrow.utcnow() - latest_date).seconds // 60
)
return False, False
(buy, sell) = latest[SignalType.BUY.value] == 1, latest[SignalType.SELL.value] == 1
logger.debug(
'trigger: %s (pair=%s) buy=%s sell=%s',
latest['date'],
pair,
str(buy),
str(sell)
)
logger.debug('trigger: %s (pair=%s) buy=%s sell=%s',
latest['date'], pair, str(buy), str(sell))
return buy, sell
def should_sell(self, trade: Trade, rate: float, date: datetime, buy: bool,
@ -500,7 +582,8 @@ class IStrategy(ABC):
def ohlcvdata_to_dataframe(self, data: Dict[str, DataFrame]) -> Dict[str, DataFrame]:
"""
Creates a dataframe and populates indicators for given candle (OHLCV) data
Populates indicators for given candle (OHLCV) data (for multiple pairs)
Does not run advice_buy or advise_sell!
Used by optimize operations only, not during dry / live runs.
Using .copy() to get a fresh copy of the dataframe for every strategy run.
Has positive effects on memory usage for whatever reason - also when

View File

@ -5,7 +5,7 @@ from freqtrade.exceptions import StrategyError
logger = logging.getLogger(__name__)
def strategy_safe_wrapper(f, message: str = "", default_retval=None):
def strategy_safe_wrapper(f, message: str = "", default_retval=None, supress_error=False):
"""
Wrapper around user-provided methods and functions.
Caches all exceptions and returns either the default_retval (if it's not None) or raises
@ -20,7 +20,7 @@ def strategy_safe_wrapper(f, message: str = "", default_retval=None):
f"Strategy caused the following exception: {error}"
f"{f}"
)
if default_retval is None:
if default_retval is None and not supress_error:
raise StrategyError(str(error)) from error
return default_retval
except Exception as error:
@ -28,7 +28,7 @@ def strategy_safe_wrapper(f, message: str = "", default_retval=None):
f"{message}"
f"Unexpected error {error} calling {f}"
)
if default_retval is None:
if default_retval is None and not supress_error:
raise StrategyError(str(error)) from error
return default_retval

View File

@ -1,4 +1,65 @@
def bot_loop_start(self, **kwargs) -> None:
"""
Called at the start of the bot iteration (one loop).
Might be used to perform pair-independent tasks
(e.g. gather some remote ressource for comparison)
For full documentation please go to https://www.freqtrade.io/en/latest/strategy-advanced/
When not implemented by a strategy, this simply does nothing.
:param **kwargs: Ensure to keep this here so updates to this won't break your strategy.
"""
pass
def confirm_trade_entry(self, pair: str, order_type: str, amount: float, rate: float,
time_in_force: str, **kwargs) -> bool:
"""
Called right before placing a buy order.
Timing for this function is critical, so avoid doing heavy computations or
network requests in this method.
For full documentation please go to https://www.freqtrade.io/en/latest/strategy-advanced/
When not implemented by a strategy, returns True (always confirming).
:param pair: Pair that's about to be bought.
:param order_type: Order type (as configured in order_types). usually limit or market.
:param amount: Amount in target (quote) currency that's going to be traded.
:param rate: Rate that's going to be used when using limit orders
:param time_in_force: Time in force. Defaults to GTC (Good-til-cancelled).
:param **kwargs: Ensure to keep this here so updates to this won't break your strategy.
:return bool: When True is returned, then the buy-order is placed on the exchange.
False aborts the process
"""
return True
def confirm_trade_exit(self, pair: str, trade: Trade, order_type: str, amount: float,
rate: float, time_in_force: str, sell_reason: str, **kwargs) -> bool:
"""
Called right before placing a regular sell order.
Timing for this function is critical, so avoid doing heavy computations or
network requests in this method.
For full documentation please go to https://www.freqtrade.io/en/latest/strategy-advanced/
When not implemented by a strategy, returns True (always confirming).
:param pair: Pair that's about to be sold.
:param trade: trade object.
:param order_type: Order type (as configured in order_types). usually limit or market.
:param amount: Amount in quote currency.
:param rate: Rate that's going to be used when using limit orders
:param time_in_force: Time in force. Defaults to GTC (Good-til-cancelled).
:param sell_reason: Sell reason.
Can be any of ['roi', 'stop_loss', 'stoploss_on_exchange', 'trailing_stop_loss',
'sell_signal', 'force_sell', 'emergency_sell']
:param **kwargs: Ensure to keep this here so updates to this won't break your strategy.
:return bool: When True is returned, then the sell-order is placed on the exchange.
False aborts the process
"""
return True
def check_buy_timeout(self, pair: str, trade: 'Trade', order: dict, **kwargs) -> bool:
"""
Check buy timeout function callback.

View File

@ -3,6 +3,7 @@ nav:
- Home: index.md
- Installation Docker: docker.md
- Installation: installation.md
- Freqtrade Basics: bot-basics.md
- Configuration: configuration.md
- Strategy Customization: strategy-customization.md
- Stoploss: stoploss.md

View File

@ -163,7 +163,7 @@ def patch_get_signal(freqtrade: FreqtradeBot, value=(True, False)) -> None:
:param value: which value IStrategy.get_signal() must return
:return: None
"""
freqtrade.strategy.get_signal = lambda e, s, t: value
freqtrade.strategy.get_signal = lambda e, s, x: value
freqtrade.exchange.refresh_latest_ohlcv = lambda p: None
@ -787,6 +787,7 @@ def limit_buy_order():
'price': 0.00001099,
'amount': 90.99181073,
'filled': 90.99181073,
'cost': 0.0009999,
'remaining': 0.0,
'status': 'closed'
}

View File

@ -1,3 +1,4 @@
from datetime import datetime, timezone
from unittest.mock import MagicMock
import pytest
@ -194,3 +195,29 @@ def test_current_whitelist(mocker, default_conf, tickers):
with pytest.raises(OperationalException):
dp = DataProvider(default_conf, exchange)
dp.current_whitelist()
def test_get_analyzed_dataframe(mocker, default_conf, ohlcv_history):
default_conf["runmode"] = RunMode.DRY_RUN
timeframe = default_conf["timeframe"]
exchange = get_patched_exchange(mocker, default_conf)
dp = DataProvider(default_conf, exchange)
dp._set_cached_df("XRP/BTC", timeframe, ohlcv_history)
dp._set_cached_df("UNITTEST/BTC", timeframe, ohlcv_history)
assert dp.runmode == RunMode.DRY_RUN
dataframe, time = dp.get_analyzed_dataframe("UNITTEST/BTC", timeframe)
assert ohlcv_history.equals(dataframe)
assert isinstance(time, datetime)
dataframe, time = dp.get_analyzed_dataframe("XRP/BTC", timeframe)
assert ohlcv_history.equals(dataframe)
assert isinstance(time, datetime)
dataframe, time = dp.get_analyzed_dataframe("NOTHING/BTC", timeframe)
assert dataframe.empty
assert isinstance(time, datetime)
assert time == datetime(1970, 1, 1, tzinfo=timezone.utc)

View File

@ -13,12 +13,14 @@ from freqtrade.exceptions import StrategyError
from freqtrade.persistence import Trade
from freqtrade.resolvers import StrategyResolver
from freqtrade.strategy.strategy_wrapper import strategy_safe_wrapper
from tests.conftest import get_patched_exchange, log_has, log_has_re
from freqtrade.data.dataprovider import DataProvider
from tests.conftest import log_has, log_has_re
from .strats.default_strategy import DefaultStrategy
# Avoid to reinit the same object again and again
_STRATEGY = DefaultStrategy(config={})
_STRATEGY.dp = DataProvider({}, None, None)
def test_returns_latest_signal(mocker, default_conf, ohlcv_history):
@ -29,63 +31,60 @@ def test_returns_latest_signal(mocker, default_conf, ohlcv_history):
mocked_history['buy'] = 0
mocked_history.loc[1, 'sell'] = 1
mocker.patch.object(
_STRATEGY, '_analyze_ticker_internal',
return_value=mocked_history
)
assert _STRATEGY.get_signal('ETH/BTC', '5m', ohlcv_history) == (False, True)
assert _STRATEGY.get_signal('ETH/BTC', '5m', mocked_history) == (False, True)
mocked_history.loc[1, 'sell'] = 0
mocked_history.loc[1, 'buy'] = 1
mocker.patch.object(
_STRATEGY, '_analyze_ticker_internal',
return_value=mocked_history
)
assert _STRATEGY.get_signal('ETH/BTC', '5m', ohlcv_history) == (True, False)
assert _STRATEGY.get_signal('ETH/BTC', '5m', mocked_history) == (True, False)
mocked_history.loc[1, 'sell'] = 0
mocked_history.loc[1, 'buy'] = 0
mocker.patch.object(
_STRATEGY, '_analyze_ticker_internal',
return_value=mocked_history
)
assert _STRATEGY.get_signal('ETH/BTC', '5m', ohlcv_history) == (False, False)
assert _STRATEGY.get_signal('ETH/BTC', '5m', mocked_history) == (False, False)
def test_get_signal_empty(default_conf, mocker, caplog):
assert (False, False) == _STRATEGY.get_signal('foo', default_conf['timeframe'],
DataFrame())
assert log_has('Empty candle (OHLCV) data for pair foo', caplog)
caplog.clear()
assert (False, False) == _STRATEGY.get_signal('bar', default_conf['timeframe'],
[])
assert log_has('Empty candle (OHLCV) data for pair bar', caplog)
def test_get_signal_exception_valueerror(default_conf, mocker, caplog, ohlcv_history):
caplog.set_level(logging.INFO)
mocker.patch.object(
_STRATEGY, '_analyze_ticker_internal',
side_effect=ValueError('xyz')
)
assert (False, False) == _STRATEGY.get_signal('foo', default_conf['timeframe'],
ohlcv_history)
assert log_has_re(r'Strategy caused the following exception: xyz.*', caplog)
def test_get_signal_empty_dataframe(default_conf, mocker, caplog, ohlcv_history):
caplog.set_level(logging.INFO)
def test_analyze_pair_empty(default_conf, mocker, caplog, ohlcv_history):
mocker.patch.object(_STRATEGY.dp, 'ohlcv', return_value=ohlcv_history)
mocker.patch.object(
_STRATEGY, '_analyze_ticker_internal',
return_value=DataFrame([])
)
mocker.patch.object(_STRATEGY, 'assert_df')
assert (False, False) == _STRATEGY.get_signal('xyz', default_conf['timeframe'],
ohlcv_history)
assert log_has('Empty dataframe for pair xyz', caplog)
_STRATEGY.analyze_pair('ETH/BTC')
assert log_has('Empty dataframe for pair ETH/BTC', caplog)
def test_get_signal_empty(default_conf, mocker, caplog):
assert (False, False) == _STRATEGY.get_signal('foo', default_conf['timeframe'], DataFrame())
assert log_has('Empty candle (OHLCV) data for pair foo', caplog)
caplog.clear()
assert (False, False) == _STRATEGY.get_signal('bar', default_conf['timeframe'], None)
assert log_has('Empty candle (OHLCV) data for pair bar', caplog)
caplog.clear()
assert (False, False) == _STRATEGY.get_signal('baz', default_conf['timeframe'], DataFrame([]))
assert log_has('Empty candle (OHLCV) data for pair baz', caplog)
def test_get_signal_exception_valueerror(default_conf, mocker, caplog, ohlcv_history):
caplog.set_level(logging.INFO)
mocker.patch.object(_STRATEGY.dp, 'ohlcv', return_value=ohlcv_history)
mocker.patch.object(
_STRATEGY, '_analyze_ticker_internal',
side_effect=ValueError('xyz')
)
_STRATEGY.analyze_pair('foo')
assert log_has_re(r'Strategy caused the following exception: xyz.*', caplog)
caplog.clear()
mocker.patch.object(
_STRATEGY, 'analyze_ticker',
side_effect=Exception('invalid ticker history ')
)
_STRATEGY.analyze_pair('foo')
assert log_has_re(r'Strategy caused the following exception: xyz.*', caplog)
def test_get_signal_old_dataframe(default_conf, mocker, caplog, ohlcv_history):
@ -99,13 +98,9 @@ def test_get_signal_old_dataframe(default_conf, mocker, caplog, ohlcv_history):
mocked_history.loc[1, 'buy'] = 1
caplog.set_level(logging.INFO)
mocker.patch.object(
_STRATEGY, '_analyze_ticker_internal',
return_value=mocked_history
)
mocker.patch.object(_STRATEGY, 'assert_df')
assert (False, False) == _STRATEGY.get_signal('xyz', default_conf['timeframe'],
ohlcv_history)
assert (False, False) == _STRATEGY.get_signal('xyz', default_conf['timeframe'], mocked_history)
assert log_has('Outdated history for pair xyz. Last tick is 16 minutes old', caplog)
@ -120,12 +115,13 @@ def test_assert_df_raise(default_conf, mocker, caplog, ohlcv_history):
mocked_history.loc[1, 'buy'] = 1
caplog.set_level(logging.INFO)
mocker.patch.object(_STRATEGY.dp, 'ohlcv', return_value=ohlcv_history)
mocker.patch.object(_STRATEGY.dp, 'get_analyzed_dataframe', return_value=(mocked_history, 0))
mocker.patch.object(
_STRATEGY, 'assert_df',
side_effect=StrategyError('Dataframe returned...')
)
assert (False, False) == _STRATEGY.get_signal('xyz', default_conf['timeframe'],
ohlcv_history)
_STRATEGY.analyze_pair('xyz')
assert log_has('Unable to analyze candle (OHLCV) data for pair xyz: Dataframe returned...',
caplog)
@ -157,15 +153,6 @@ def test_assert_df(default_conf, mocker, ohlcv_history, caplog):
_STRATEGY.disable_dataframe_checks = False
def test_get_signal_handles_exceptions(mocker, default_conf):
exchange = get_patched_exchange(mocker, default_conf)
mocker.patch.object(
_STRATEGY, 'analyze_ticker',
side_effect=Exception('invalid ticker history ')
)
assert _STRATEGY.get_signal(exchange, 'ETH/BTC', '5m') == (False, False)
def test_ohlcvdata_to_dataframe(default_conf, testdatadir) -> None:
default_conf.update({'strategy': 'DefaultStrategy'})
strategy = StrategyResolver.load_strategy(default_conf)
@ -342,6 +329,7 @@ def test__analyze_ticker_internal_skip_analyze(ohlcv_history, mocker, caplog) ->
)
strategy = DefaultStrategy({})
strategy.dp = DataProvider({}, None, None)
strategy.process_only_new_candles = True
ret = strategy._analyze_ticker_internal(ohlcv_history, {'pair': 'ETH/BTC'})
@ -400,6 +388,14 @@ def test_is_pair_locked(default_conf):
assert not strategy.is_pair_locked(pair)
def test_is_informative_pairs_callback(default_conf):
default_conf.update({'strategy': 'TestStrategyLegacy'})
strategy = StrategyResolver.load_strategy(default_conf)
# Should return empty
# Uses fallback to base implementation
assert [] == strategy.informative_pairs()
@pytest.mark.parametrize('error', [
ValueError, KeyError, Exception,
])
@ -419,6 +415,11 @@ def test_strategy_safe_wrapper_error(caplog, error):
assert isinstance(ret, bool)
assert ret
caplog.clear()
# Test supressing error
ret = strategy_safe_wrapper(failing_method, message='DeadBeef', supress_error=True)()
assert log_has_re(r'DeadBeef.*', caplog)
@pytest.mark.parametrize('value', [
1, 22, 55, True, False, {'a': 1, 'b': '112'},

View File

@ -911,6 +911,7 @@ def test_process_informative_pairs_added(default_conf, ticker, mocker) -> None:
refresh_latest_ohlcv=refresh_mock,
)
inf_pairs = MagicMock(return_value=[("BTC/ETH", '1m'), ("ETH/USDT", "1h")])
mocker.patch('freqtrade.strategy.interface.IStrategy.get_signal', return_value=(False, False))
mocker.patch('time.sleep', return_value=None)
freqtrade = FreqtradeBot(default_conf)
@ -973,6 +974,7 @@ def test_execute_buy(mocker, default_conf, fee, limit_buy_order) -> None:
patch_RPCManager(mocker)
patch_exchange(mocker)
freqtrade = FreqtradeBot(default_conf)
freqtrade.strategy.confirm_trade_entry = MagicMock(return_value=False)
stake_amount = 2
bid = 0.11
buy_rate_mock = MagicMock(return_value=bid)
@ -994,6 +996,13 @@ def test_execute_buy(mocker, default_conf, fee, limit_buy_order) -> None:
)
pair = 'ETH/BTC'
assert not freqtrade.execute_buy(pair, stake_amount)
assert buy_rate_mock.call_count == 1
assert buy_mm.call_count == 0
assert freqtrade.strategy.confirm_trade_entry.call_count == 1
buy_rate_mock.reset_mock()
freqtrade.strategy.confirm_trade_entry = MagicMock(return_value=True)
assert freqtrade.execute_buy(pair, stake_amount)
assert buy_rate_mock.call_count == 1
assert buy_mm.call_count == 1
@ -1001,6 +1010,7 @@ def test_execute_buy(mocker, default_conf, fee, limit_buy_order) -> None:
assert call_args['pair'] == pair
assert call_args['rate'] == bid
assert call_args['amount'] == stake_amount / bid
buy_rate_mock.reset_mock()
# Should create an open trade with an open order id
# As the order is not fulfilled yet
@ -1013,7 +1023,7 @@ def test_execute_buy(mocker, default_conf, fee, limit_buy_order) -> None:
fix_price = 0.06
assert freqtrade.execute_buy(pair, stake_amount, fix_price)
# Make sure get_buy_rate wasn't called again
assert buy_rate_mock.call_count == 1
assert buy_rate_mock.call_count == 0
assert buy_mm.call_count == 2
call_args = buy_mm.call_args_list[1][1]
@ -1059,6 +1069,39 @@ def test_execute_buy(mocker, default_conf, fee, limit_buy_order) -> None:
assert not freqtrade.execute_buy(pair, stake_amount)
def test_execute_buy_confirm_error(mocker, default_conf, fee, limit_buy_order) -> None:
freqtrade = get_patched_freqtradebot(mocker, default_conf)
mocker.patch.multiple(
'freqtrade.freqtradebot.FreqtradeBot',
get_buy_rate=MagicMock(return_value=0.11),
_get_min_pair_stake_amount=MagicMock(return_value=1)
)
mocker.patch.multiple(
'freqtrade.exchange.Exchange',
fetch_ticker=MagicMock(return_value={
'bid': 0.00001172,
'ask': 0.00001173,
'last': 0.00001172
}),
buy=MagicMock(return_value=limit_buy_order),
get_fee=fee,
)
stake_amount = 2
pair = 'ETH/BTC'
freqtrade.strategy.confirm_trade_entry = MagicMock(side_effect=ValueError)
assert freqtrade.execute_buy(pair, stake_amount)
freqtrade.strategy.confirm_trade_entry = MagicMock(side_effect=Exception)
assert freqtrade.execute_buy(pair, stake_amount)
freqtrade.strategy.confirm_trade_entry = MagicMock(return_value=True)
assert freqtrade.execute_buy(pair, stake_amount)
freqtrade.strategy.confirm_trade_entry = MagicMock(return_value=False)
assert not freqtrade.execute_buy(pair, stake_amount)
def test_add_stoploss_on_exchange(mocker, default_conf, limit_buy_order) -> None:
patch_RPCManager(mocker)
patch_exchange(mocker)
@ -1962,6 +2005,18 @@ def test_close_trade(default_conf, ticker, limit_buy_order, limit_sell_order,
freqtrade.handle_trade(trade)
def test_bot_loop_start_called_once(mocker, default_conf, caplog):
ftbot = get_patched_freqtradebot(mocker, default_conf)
patch_get_signal(ftbot)
ftbot.strategy.bot_loop_start = MagicMock(side_effect=ValueError)
ftbot.strategy.analyze = MagicMock()
ftbot.process()
assert log_has_re(r'Strategy caused the following exception.*', caplog)
assert ftbot.strategy.bot_loop_start.call_count == 1
assert ftbot.strategy.analyze.call_count == 1
def test_check_handle_timedout_buy_usercustom(default_conf, ticker, limit_buy_order_old, open_trade,
fee, mocker) -> None:
default_conf["unfilledtimeout"] = {"buy": 1400, "sell": 30}
@ -2488,22 +2543,33 @@ def test_execute_sell_up(default_conf, ticker, fee, ticker_sell_up, mocker) -> N
patch_whitelist(mocker, default_conf)
freqtrade = FreqtradeBot(default_conf)
patch_get_signal(freqtrade)
freqtrade.strategy.confirm_trade_exit = MagicMock(return_value=False)
# Create some test data
freqtrade.enter_positions()
rpc_mock.reset_mock()
trade = Trade.query.first()
assert trade
assert freqtrade.strategy.confirm_trade_exit.call_count == 0
# Increase the price and sell it
mocker.patch.multiple(
'freqtrade.exchange.Exchange',
fetch_ticker=ticker_sell_up
)
# Prevented sell ...
freqtrade.execute_sell(trade=trade, limit=ticker_sell_up()['bid'], sell_reason=SellType.ROI)
assert rpc_mock.call_count == 0
assert freqtrade.strategy.confirm_trade_exit.call_count == 1
# Repatch with true
freqtrade.strategy.confirm_trade_exit = MagicMock(return_value=True)
freqtrade.execute_sell(trade=trade, limit=ticker_sell_up()['bid'], sell_reason=SellType.ROI)
assert freqtrade.strategy.confirm_trade_exit.call_count == 1
assert rpc_mock.call_count == 2
assert rpc_mock.call_count == 1
last_msg = rpc_mock.call_args_list[-1][0][0]
assert {
'type': RPCMessageType.SELL_NOTIFICATION,

View File

@ -79,10 +79,15 @@ def test_may_execute_sell_stoploss_on_exchange_multi(default_conf, ticker, fee,
freqtrade.strategy.order_types['stoploss_on_exchange'] = True
# Switch ordertype to market to close trade immediately
freqtrade.strategy.order_types['sell'] = 'market'
freqtrade.strategy.confirm_trade_entry = MagicMock(return_value=True)
freqtrade.strategy.confirm_trade_exit = MagicMock(return_value=True)
patch_get_signal(freqtrade)
# Create some test data
freqtrade.enter_positions()
assert freqtrade.strategy.confirm_trade_entry.call_count == 3
freqtrade.strategy.confirm_trade_entry.reset_mock()
assert freqtrade.strategy.confirm_trade_exit.call_count == 0
wallets_mock.reset_mock()
Trade.session = MagicMock()
@ -95,6 +100,9 @@ def test_may_execute_sell_stoploss_on_exchange_multi(default_conf, ticker, fee,
n = freqtrade.exit_positions(trades)
assert n == 2
assert should_sell_mock.call_count == 2
assert freqtrade.strategy.confirm_trade_entry.call_count == 0
assert freqtrade.strategy.confirm_trade_exit.call_count == 1
freqtrade.strategy.confirm_trade_exit.reset_mock()
# Only order for 3rd trade needs to be cancelled
assert cancel_order_mock.call_count == 1