diff --git a/docker-compose.yml b/docker-compose.yml index 3a4c4c2db..49d83aa5e 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -3,6 +3,7 @@ version: '3' services: freqtrade: image: freqtradeorg/freqtrade:master + # image: freqtradeorg/freqtrade:develop # Build step - only needed when additional dependencies are needed # build: # context: . @@ -14,7 +15,7 @@ services: # Default command used when running `docker compose up` command: > trade - --logfile /freqtrade/user_data/freqtrade.log + --logfile /freqtrade/user_data/logs/freqtrade.log --db-url sqlite:////freqtrade/user_data/tradesv3.sqlite --config /freqtrade/user_data/config.json --strategy SampleStrategy diff --git a/docs/advanced-setup.md b/docs/advanced-setup.md index 2d3fe36f5..95480a2c6 100644 --- a/docs/advanced-setup.md +++ b/docs/advanced-setup.md @@ -37,30 +37,30 @@ as the watchdog. ## Advanced Logging -On many Linux systems the bot can be configured to send its log messages to `syslog` or `journald` system services. Logging to a remote `syslog` server is also available on Windows. The special values for the `--logfilename` command line option can be used for this. +On many Linux systems the bot can be configured to send its log messages to `syslog` or `journald` system services. Logging to a remote `syslog` server is also available on Windows. The special values for the `--logfile` command line option can be used for this. ### Logging to syslog -To send Freqtrade log messages to a local or remote `syslog` service use the `--logfilename` command line option with the value in the following format: +To send Freqtrade log messages to a local or remote `syslog` service use the `--logfile` command line option with the value in the following format: -* `--logfilename syslog:` -- send log messages to `syslog` service using the `` as the syslog address. +* `--logfile syslog:` -- send log messages to `syslog` service using the `` as the syslog address. The syslog address can be either a Unix domain socket (socket filename) or a UDP socket specification, consisting of IP address and UDP port, separated by the `:` character. So, the following are the examples of possible usages: -* `--logfilename syslog:/dev/log` -- log to syslog (rsyslog) using the `/dev/log` socket, suitable for most systems. -* `--logfilename syslog` -- same as above, the shortcut for `/dev/log`. -* `--logfilename syslog:/var/run/syslog` -- log to syslog (rsyslog) using the `/var/run/syslog` socket. Use this on MacOS. -* `--logfilename syslog:localhost:514` -- log to local syslog using UDP socket, if it listens on port 514. -* `--logfilename syslog::514` -- log to remote syslog at IP address and port 514. This may be used on Windows for remote logging to an external syslog server. +* `--logfile syslog:/dev/log` -- log to syslog (rsyslog) using the `/dev/log` socket, suitable for most systems. +* `--logfile syslog` -- same as above, the shortcut for `/dev/log`. +* `--logfile syslog:/var/run/syslog` -- log to syslog (rsyslog) using the `/var/run/syslog` socket. Use this on MacOS. +* `--logfile syslog:localhost:514` -- log to local syslog using UDP socket, if it listens on port 514. +* `--logfile syslog::514` -- log to remote syslog at IP address and port 514. This may be used on Windows for remote logging to an external syslog server. Log messages are send to `syslog` with the `user` facility. So you can see them with the following commands: * `tail -f /var/log/user`, or * install a comprehensive graphical viewer (for instance, 'Log File Viewer' for Ubuntu). -On many systems `syslog` (`rsyslog`) fetches data from `journald` (and vice versa), so both `--logfilename syslog` or `--logfilename journald` can be used and the messages be viewed with both `journalctl` and a syslog viewer utility. You can combine this in any way which suites you better. +On many systems `syslog` (`rsyslog`) fetches data from `journald` (and vice versa), so both `--logfile syslog` or `--logfile journald` can be used and the messages be viewed with both `journalctl` and a syslog viewer utility. You can combine this in any way which suites you better. For `rsyslog` the messages from the bot can be redirected into a separate dedicated log file. To achieve this, add ``` @@ -78,9 +78,9 @@ $RepeatedMsgReduction on This needs the `systemd` python package installed as the dependency, which is not available on Windows. Hence, the whole journald logging functionality is not available for a bot running on Windows. -To send Freqtrade log messages to `journald` system service use the `--logfilename` command line option with the value in the following format: +To send Freqtrade log messages to `journald` system service use the `--logfile` command line option with the value in the following format: -* `--logfilename journald` -- send log messages to `journald`. +* `--logfile journald` -- send log messages to `journald`. Log messages are send to `journald` with the `user` facility. So you can see them with the following commands: @@ -89,4 +89,4 @@ Log messages are send to `journald` with the `user` facility. So you can see the There are many other options in the `journalctl` utility to filter the messages, see manual pages for this utility. -On many systems `syslog` (`rsyslog`) fetches data from `journald` (and vice versa), so both `--logfilename syslog` or `--logfilename journald` can be used and the messages be viewed with both `journalctl` and a syslog viewer utility. You can combine this in any way which suites you better. +On many systems `syslog` (`rsyslog`) fetches data from `journald` (and vice versa), so both `--logfile syslog` or `--logfile journald` can be used and the messages be viewed with both `journalctl` and a syslog viewer utility. You can combine this in any way which suites you better. diff --git a/docs/bot-usage.md b/docs/bot-usage.md index 78e137676..b1649374a 100644 --- a/docs/bot-usage.md +++ b/docs/bot-usage.md @@ -144,10 +144,10 @@ It is recommended to use version control to keep track of changes to your strate ### How to use **--strategy**? This parameter will allow you to load your custom strategy class. -Per default without `--strategy` or `-s` the bot will load the -`DefaultStrategy` included with the bot (`freqtrade/strategy/default_strategy.py`). +To test the bot installation, you can use the `SampleStrategy` installed by the `create-userdir` subcommand (usually `user_data/strategy/sample_strategy.py`). -The bot will search your strategy file within `user_data/strategies` and `freqtrade/strategy`. +The bot will search your strategy file within `user_data/strategies`. +To use other directories, please read the next section about `--strategy-path`. To load a strategy, simply pass the class name (e.g.: `CustomStrategy`) in this parameter. diff --git a/docs/configuration.md b/docs/configuration.md index d8a9653c3..338299781 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -34,7 +34,7 @@ The prevelance for all Options is as follows: - CLI arguments override any other option - Configuration files are used in sequence (last file wins), and override Strategy configurations. -- Strategy configurations are only used if they are not set via configuration or via command line arguments. These options are market with [Strategy Override](#parameters-in-the-strategy) in the below table. +- Strategy configurations are only used if they are not set via configuration or via command line arguments. These options are marked with [Strategy Override](#parameters-in-the-strategy) in the below table. Mandatory parameters are marked as **Required**, which means that they are required to be set in one of the possible ways. diff --git a/docs/edge.md b/docs/edge.md index 721f570c7..029844c0b 100644 --- a/docs/edge.md +++ b/docs/edge.md @@ -79,7 +79,7 @@ So lets say your Win rate is 28% and your Risk Reward Ratio is 5: Expectancy = (5 X 0.28) – 0.72 = 0.68 ``` -Superficially, this means that on average you expect this strategy’s trades to return .68 times the size of your loses. This is important for two reasons: First, it may seem obvious, but you know right away that you have a positive return. Second, you now have a number you can compare to other candidate systems to make decisions about which ones you employ. +Superficially, this means that on average you expect this strategy’s trades to return 1.68 times the size of your loses. Said another way, you can expect to win $1.68 for every $1 you lose. This is important for two reasons: First, it may seem obvious, but you know right away that you have a positive return. Second, you now have a number you can compare to other candidate systems to make decisions about which ones you employ. It is important to remember that any system with an expectancy greater than 0 is profitable using past data. The key is finding one that will be profitable in the future. diff --git a/docs/faq.md b/docs/faq.md index 94818964b..8e8a1bf35 100644 --- a/docs/faq.md +++ b/docs/faq.md @@ -100,7 +100,7 @@ $ tail -f /path/to/mylogfile.log | grep 'something' ``` from a separate terminal window. -On Windows, the `--logfilename` option is also supported by Freqtrade and you can use the `findstr` command to search the log for the string of interest: +On Windows, the `--logfile` option is also supported by Freqtrade and you can use the `findstr` command to search the log for the string of interest: ``` > type \path\to\mylogfile.log | findstr "something" ``` diff --git a/docs/hyperopt.md b/docs/hyperopt.md index c5055a3a8..11161e58b 100644 --- a/docs/hyperopt.md +++ b/docs/hyperopt.md @@ -6,9 +6,7 @@ algorithms included in the `scikit-optimize` package to accomplish this. The search will burn all your CPU cores, make your laptop sound like a fighter jet and still take a long time. -In general, the search for best parameters starts with a few random combinations and then uses Bayesian search with a -ML regressor algorithm (currently ExtraTreesRegressor) to quickly find a combination of parameters in the search hyperspace -that minimizes the value of the [loss function](#loss-functions). +In general, the search for best parameters starts with a few random combinations (see [below](#reproducible-results) for more details) and then uses Bayesian search with a ML regressor algorithm (currently ExtraTreesRegressor) to quickly find a combination of parameters in the search hyperspace that minimizes the value of the [loss function](#loss-functions). Hyperopt requires historic data to be available, just as backtesting does. To learn how to get data for the pairs and exchange you're interested in, head over to the [Data Downloading](data-download.md) section of the documentation. @@ -16,6 +14,24 @@ To learn how to get data for the pairs and exchange you're interested in, head o !!! Bug Hyperopt can crash when used with only 1 CPU Core as found out in [Issue #1133](https://github.com/freqtrade/freqtrade/issues/1133) +## Install hyperopt dependencies + +Since Hyperopt dependencies are not needed to run the bot itself, are heavy, can not be easily built on some platforms (like Raspberry PI), they are not installed by default. Before you run Hyperopt, you need to install the corresponding dependencies, as described in this section below. + +!!! Note + Since Hyperopt is a resource intensive process, running it on a Raspberry Pi is not recommended nor supported. + +### Docker + +The docker-image includes hyperopt dependencies, no further action needed. + +### Easy installation script (setup.sh) / Manual installation + +```bash +source .env/bin/activate +pip install -r requirements-hyperopt.txt +``` + ## Prepare Hyperopting Before we start digging into Hyperopt, we recommend you to take a look at @@ -47,6 +63,9 @@ Optional - can also be loaded from a strategy: !!! Note Assuming the optional methods are not in your hyperopt file, please use `--strategy AweSomeStrategy` which contains these methods so hyperopt can use these methods instead. +!!! Note + You always have to provide a strategy to Hyperopt, even if your custom Hyperopt class contains all methods. + Rarely you may also need to override: * `roi_space` - for custom ROI optimization (if you need the ranges for the ROI parameters in the optimization hyperspace that differ from default) @@ -293,7 +312,7 @@ You can also enable position stacking in the configuration file by explicitly se ### Reproducible results -The search for optimal parameters starts with a few (currently 30) random combinations in the hyperspace of parameters, random Hyperopt epochs. These random epochs are marked with a leading asterisk sign at the Hyperopt output. +The search for optimal parameters starts with a few (currently 30) random combinations in the hyperspace of parameters, random Hyperopt epochs. These random epochs are marked with an asterisk character (`*`) in the first column in the Hyperopt output. The initial state for generation of these random values (random state) is controlled by the value of the `--random-state` command line option. You can set it to some arbitrary value of your choice to obtain reproducible results. diff --git a/docs/requirements-docs.txt b/docs/requirements-docs.txt index 48ade026e..1c0e280ae 100644 --- a/docs/requirements-docs.txt +++ b/docs/requirements-docs.txt @@ -1,2 +1,2 @@ -mkdocs-material==4.6.3 +mkdocs-material==5.1.3 mdx_truly_sane_lists==1.2 diff --git a/docs/sql_cheatsheet.md b/docs/sql_cheatsheet.md index f41520bd9..b7b38c3dc 100644 --- a/docs/sql_cheatsheet.md +++ b/docs/sql_cheatsheet.md @@ -67,22 +67,32 @@ SELECT * FROM trades; !!! Warning Manually selling a pair on the exchange will not be detected by the bot and it will try to sell anyway. Whenever possible, forcesell should be used to accomplish the same thing. - It is strongly advised to backup your database file before making any manual changes. + It is strongly advised to backup your database file before making any manual changes. !!! Note This should not be necessary after /forcesell, as forcesell orders are closed automatically by the bot on the next iteration. ```sql UPDATE trades -SET is_open=0, close_date=, close_rate=, close_profit=close_rate/open_rate-1, sell_reason= +SET is_open=0, + close_date=, + close_rate=, + close_profit=close_rate/open_rate-1, + close_profit_abs = (amount * * (1 - fee_close) - (amount * open_rate * 1 - fee_open), + sell_reason= WHERE id=; ``` -##### Example +### Example ```sql UPDATE trades -SET is_open=0, close_date='2017-12-20 03:08:45.103418', close_rate=0.19638016, close_profit=0.0496, sell_reason='force_sell' +SET is_open=0, + close_date='2017-12-20 03:08:45.103418', + close_rate=0.19638016, + close_profit=0.0496, + close_profit_abs = (amount * 0.19638016 * (1 - fee_close) - (amount * open_rate * 1 - fee_open) + sell_reason='force_sell' WHERE id=31; ``` @@ -99,10 +109,3 @@ VALUES ('bittrex', 'ETH/BTC', 1, 0.0025, 0.0025, , , bool: + if trade.open_rate > 100 and trade.open_date < datetime.utcnow() - timedelta(minutes=5): + return True + elif trade.open_rate > 10 and trade.open_date < datetime.utcnow() - timedelta(minutes=3): + return True + elif trade.open_rate < 1 and trade.open_date < datetime.utcnow() - timedelta(hours=24): + return True + return False + + + def check_sell_timeout(self, pair: str, trade: 'Trade', order: dict, **kwargs) -> bool: + if trade.open_rate > 100 and trade.open_date < datetime.utcnow() - timedelta(minutes=5): + return True + elif trade.open_rate > 10 and trade.open_date < datetime.utcnow() - timedelta(minutes=3): + return True + elif trade.open_rate < 1 and trade.open_date < datetime.utcnow() - timedelta(hours=24): + return True + return False +``` + +!!! Note + For the above example, `unfilledtimeout` must be set to something bigger than 24h, otherwise that type of timeout will apply first. + +### Custom order timeout example (using additional data) + +``` python +from datetime import datetime, timestamp +from freqtrade.persistence import Trade + +class Awesomestrategy(IStrategy): + + # ... populate_* methods + + # Set unfilledtimeout to 25 hours, since our maximum timeout from below is 24 hours. + unfilledtimeout = { + 'buy': 60 * 25, + 'sell': 60 * 25 + } + + def check_buy_timeout(self, pair: str, trade: Trade, order: dict, **kwargs) -> bool: + ob = self.dp.orderbook(pair, 1) + current_price = ob['bids'][0][0] + # Cancel buy order if price is more than 2% above the order. + if current_price > order['price'] * 1.02: + return True + return False + + + def check_sell_timeout(self, pair: str, trade: Trade, order: dict, **kwargs) -> bool: + ob = self.dp.orderbook(pair, 1) + current_price = ob['asks'][0][0] + # Cancel sell order if price is more than 2% below the order. + if current_price < order['price'] * 0.98: + return True + return False +``` diff --git a/docs/strategy-customization.md b/docs/strategy-customization.md index 7793ea148..c4fc55811 100644 --- a/docs/strategy-customization.md +++ b/docs/strategy-customization.md @@ -1,7 +1,6 @@ # Strategy Customization -This page explains where to customize your strategies, and add new -indicators. +This page explains where to customize your strategies, and add new indicators. ## Install a custom strategy file diff --git a/docs/utils.md b/docs/utils.md index 269b9affd..57210ac7e 100644 --- a/docs/utils.md +++ b/docs/utils.md @@ -77,7 +77,7 @@ Results will be located in `user_data/strategies/.py`. ``` output usage: freqtrade new-strategy [-h] [--userdir PATH] [-s NAME] - [--template {full,minimal}] + [--template {full,minimal,advanced}] optional arguments: -h, --help show this help message and exit @@ -86,10 +86,10 @@ optional arguments: -s NAME, --strategy NAME Specify strategy class name which will be used by the bot. - --template {full,minimal} - Use a template which is either `minimal` or `full` - (containing multiple sample indicators). Default: - `full`. + --template {full,minimal,advanced} + Use a template which is either `minimal`, `full` + (containing multiple sample indicators) or `advanced`. + Default: `full`. ``` @@ -105,6 +105,12 @@ With custom user directory freqtrade new-strategy --userdir ~/.freqtrade/ --strategy AwesomeStrategy ``` +Using the advanced template (populates all optional functions and methods) + +```bash +freqtrade new-strategy --strategy AwesomeStrategy --template advanced +``` + ## Create new hyperopt Creates a new hyperopt from a template similar to SampleHyperopt. @@ -114,7 +120,7 @@ Results will be located in `user_data/hyperopts/.py`. ``` output usage: freqtrade new-hyperopt [-h] [--userdir PATH] [--hyperopt NAME] - [--template {full,minimal}] + [--template {full,minimal,advanced}] optional arguments: -h, --help show this help message and exit @@ -122,10 +128,10 @@ optional arguments: Path to userdata directory. --hyperopt NAME Specify hyperopt class name which will be used by the bot. - --template {full,minimal} - Use a template which is either `minimal` or `full` - (containing multiple sample indicators). Default: - `full`. + --template {full,minimal,advanced} + Use a template which is either `minimal`, `full` + (containing multiple sample indicators) or `advanced`. + Default: `full`. ``` ### Sample usage of new-hyperopt diff --git a/freqtrade/__init__.py b/freqtrade/__init__.py index 080fde242..647682d70 100644 --- a/freqtrade/__init__.py +++ b/freqtrade/__init__.py @@ -1,5 +1,5 @@ """ Freqtrade bot """ -__version__ = '2020.3' +__version__ = '2020.4' if __version__ == 'develop': @@ -24,4 +24,11 @@ if __version__ == 'develop': # stderr=subprocess.DEVNULL).decode("utf-8").rstrip().strip('"') except Exception: # git not available, ignore - pass + try: + # Try Fallback to freqtrade_commit file (created by CI while building docker image) + from pathlib import Path + versionfile = Path('./freqtrade_commit') + if versionfile.is_file(): + __version__ = f"docker-{versionfile.read_text()[:8]}" + except Exception: + pass diff --git a/freqtrade/commands/cli_options.py b/freqtrade/commands/cli_options.py index 5cf1b7fce..498ea9359 100644 --- a/freqtrade/commands/cli_options.py +++ b/freqtrade/commands/cli_options.py @@ -387,9 +387,9 @@ AVAILABLE_CLI_OPTIONS = { # Templating options "template": Arg( '--template', - help='Use a template which is either `minimal` or ' - '`full` (containing multiple sample indicators). Default: `%(default)s`.', - choices=['full', 'minimal'], + help='Use a template which is either `minimal`, ' + '`full` (containing multiple sample indicators) or `advanced`. Default: `%(default)s`.', + choices=['full', 'minimal', 'advanced'], default='full', ), # Plot dataframe diff --git a/freqtrade/commands/deploy_commands.py b/freqtrade/commands/deploy_commands.py index f5a68f748..a29ba346f 100644 --- a/freqtrade/commands/deploy_commands.py +++ b/freqtrade/commands/deploy_commands.py @@ -8,7 +8,7 @@ from freqtrade.configuration.directory_operations import (copy_sample_files, create_userdata_dir) from freqtrade.constants import USERPATH_HYPEROPTS, USERPATH_STRATEGIES from freqtrade.exceptions import OperationalException -from freqtrade.misc import render_template +from freqtrade.misc import render_template, render_template_with_fallback from freqtrade.state import RunMode logger = logging.getLogger(__name__) @@ -32,10 +32,27 @@ def deploy_new_strategy(strategy_name: str, strategy_path: Path, subtemplate: st """ Deploy new strategy from template to strategy_path """ - indicators = render_template(templatefile=f"subtemplates/indicators_{subtemplate}.j2",) - buy_trend = render_template(templatefile=f"subtemplates/buy_trend_{subtemplate}.j2",) - sell_trend = render_template(templatefile=f"subtemplates/sell_trend_{subtemplate}.j2",) - plot_config = render_template(templatefile=f"subtemplates/plot_config_{subtemplate}.j2",) + fallback = 'full' + indicators = render_template_with_fallback( + templatefile=f"subtemplates/indicators_{subtemplate}.j2", + templatefallbackfile=f"subtemplates/indicators_{fallback}.j2", + ) + buy_trend = render_template_with_fallback( + templatefile=f"subtemplates/buy_trend_{subtemplate}.j2", + templatefallbackfile=f"subtemplates/buy_trend_{fallback}.j2", + ) + sell_trend = render_template_with_fallback( + templatefile=f"subtemplates/sell_trend_{subtemplate}.j2", + templatefallbackfile=f"subtemplates/sell_trend_{fallback}.j2", + ) + plot_config = render_template_with_fallback( + templatefile=f"subtemplates/plot_config_{subtemplate}.j2", + templatefallbackfile=f"subtemplates/plot_config_{fallback}.j2", + ) + additional_methods = render_template_with_fallback( + templatefile=f"subtemplates/strategy_methods_{subtemplate}.j2", + templatefallbackfile=f"subtemplates/strategy_methods_empty.j2", + ) strategy_text = render_template(templatefile='base_strategy.py.j2', arguments={"strategy": strategy_name, @@ -43,6 +60,7 @@ def deploy_new_strategy(strategy_name: str, strategy_path: Path, subtemplate: st "buy_trend": buy_trend, "sell_trend": sell_trend, "plot_config": plot_config, + "additional_methods": additional_methods, }) logger.info(f"Writing strategy to `{strategy_path}`.") @@ -73,14 +91,23 @@ def deploy_new_hyperopt(hyperopt_name: str, hyperopt_path: Path, subtemplate: st """ Deploys a new hyperopt template to hyperopt_path """ - buy_guards = render_template( - templatefile=f"subtemplates/hyperopt_buy_guards_{subtemplate}.j2",) - sell_guards = render_template( - templatefile=f"subtemplates/hyperopt_sell_guards_{subtemplate}.j2",) - buy_space = render_template( - templatefile=f"subtemplates/hyperopt_buy_space_{subtemplate}.j2",) - sell_space = render_template( - templatefile=f"subtemplates/hyperopt_sell_space_{subtemplate}.j2",) + fallback = 'full' + buy_guards = render_template_with_fallback( + templatefile=f"subtemplates/hyperopt_buy_guards_{subtemplate}.j2", + templatefallbackfile=f"subtemplates/hyperopt_buy_guards_{fallback}.j2", + ) + sell_guards = render_template_with_fallback( + templatefile=f"subtemplates/hyperopt_sell_guards_{subtemplate}.j2", + templatefallbackfile=f"subtemplates/hyperopt_sell_guards_{fallback}.j2", + ) + buy_space = render_template_with_fallback( + templatefile=f"subtemplates/hyperopt_buy_space_{subtemplate}.j2", + templatefallbackfile=f"subtemplates/hyperopt_buy_space_{fallback}.j2", + ) + sell_space = render_template_with_fallback( + templatefile=f"subtemplates/hyperopt_sell_space_{subtemplate}.j2", + templatefallbackfile=f"subtemplates/hyperopt_sell_space_{fallback}.j2", + ) strategy_text = render_template(templatefile='base_hyperopt.py.j2', arguments={"hyperopt": hyperopt_name, diff --git a/freqtrade/commands/hyperopt_commands.py b/freqtrade/commands/hyperopt_commands.py index 5b2388252..3f61ea66c 100755 --- a/freqtrade/commands/hyperopt_commands.py +++ b/freqtrade/commands/hyperopt_commands.py @@ -52,8 +52,8 @@ def start_hyperopt_list(args: Dict[str, Any]) -> None: if not export_csv: try: - Hyperopt.print_result_table(config, trials, total_epochs, - not filteroptions['only_best'], print_colorized, 0) + print(Hyperopt.get_result_table(config, trials, total_epochs, + not filteroptions['only_best'], print_colorized, 0)) except KeyboardInterrupt: print('User interrupted..') diff --git a/freqtrade/configuration/directory_operations.py b/freqtrade/configuration/directory_operations.py index 5f8eb76b0..6b8c8cb5a 100644 --- a/freqtrade/configuration/directory_operations.py +++ b/freqtrade/configuration/directory_operations.py @@ -33,8 +33,8 @@ def create_userdata_dir(directory: str, create_dir: bool = False) -> Path: :param create_dir: Create directory if it does not exist. :return: Path object containing the directory """ - sub_dirs = ["backtest_results", "data", "hyperopts", "hyperopt_results", "notebooks", - "plot", "strategies", ] + sub_dirs = ["backtest_results", "data", "hyperopts", "hyperopt_results", "logs", + "notebooks", "plot", "strategies", ] folder = Path(directory) if not folder.is_dir(): if create_dir: diff --git a/freqtrade/data/btanalysis.py b/freqtrade/data/btanalysis.py index 23a9f720c..b0c642c1d 100644 --- a/freqtrade/data/btanalysis.py +++ b/freqtrade/data/btanalysis.py @@ -151,13 +151,20 @@ def load_trades(source: str, db_url: str, exportfilename: Path, return load_backtest_data(exportfilename) -def extract_trades_of_period(dataframe: pd.DataFrame, trades: pd.DataFrame) -> pd.DataFrame: +def extract_trades_of_period(dataframe: pd.DataFrame, trades: pd.DataFrame, + date_index=False) -> pd.DataFrame: """ Compare trades and backtested pair DataFrames to get trades performed on backtested period :return: the DataFrame of a trades of period """ - trades = trades.loc[(trades['open_time'] >= dataframe.iloc[0]['date']) & - (trades['close_time'] <= dataframe.iloc[-1]['date'])] + if date_index: + trades_start = dataframe.index[0] + trades_stop = dataframe.index[-1] + else: + trades_start = dataframe.iloc[0]['date'] + trades_stop = dataframe.iloc[-1]['date'] + trades = trades.loc[(trades['open_time'] >= trades_start) & + (trades['close_time'] <= trades_stop)] return trades @@ -213,13 +220,15 @@ def calculate_max_drawdown(trades: pd.DataFrame, *, date_col: str = 'close_time' """ if len(trades) == 0: raise ValueError("Trade dataframe empty.") - profit_results = trades.sort_values(date_col) + profit_results = trades.sort_values(date_col).reset_index(drop=True) max_drawdown_df = pd.DataFrame() max_drawdown_df['cumulative'] = profit_results[value_col].cumsum() max_drawdown_df['high_value'] = max_drawdown_df['cumulative'].cummax() max_drawdown_df['drawdown'] = max_drawdown_df['cumulative'] - max_drawdown_df['high_value'] - high_date = profit_results.loc[max_drawdown_df['high_value'].idxmax(), date_col] - low_date = profit_results.loc[max_drawdown_df['drawdown'].idxmin(), date_col] - + idxmin = max_drawdown_df['drawdown'].idxmin() + if idxmin == 0: + raise ValueError("No losing trade, therefore no drawdown.") + high_date = profit_results.loc[max_drawdown_df.iloc[:idxmin]['high_value'].idxmax(), date_col] + low_date = profit_results.loc[idxmin, date_col] return abs(min(max_drawdown_df['drawdown'])), high_date, low_date diff --git a/freqtrade/exceptions.py b/freqtrade/exceptions.py index 2f05ddb57..553a691ef 100644 --- a/freqtrade/exceptions.py +++ b/freqtrade/exceptions.py @@ -35,3 +35,10 @@ class TemporaryError(FreqtradeException): This could happen when an exchange is congested, unavailable, or the user has networking problems. Usually resolves itself after a time. """ + + +class StrategyError(FreqtradeException): + """ + Errors with custom user-code deteced. + Usually caused by errors in the strategy. + """ diff --git a/freqtrade/exchange/exchange.py b/freqtrade/exchange/exchange.py index f4c94a1ca..1a0565959 100644 --- a/freqtrade/exchange/exchange.py +++ b/freqtrade/exchange/exchange.py @@ -452,6 +452,17 @@ class Exchange: price = ceil(big_price) / pow(10, symbol_prec) return price + def price_get_one_pip(self, pair: str, price: float) -> float: + """ + Get's the "1 pip" value for this pair. + Used in PriceFilter to calculate the 1pip movements. + """ + precision = self.markets[pair]['precision']['price'] + if self.precisionMode == TICK_SIZE: + return precision + else: + return 1 / pow(10, precision) + def dry_run_order(self, pair: str, ordertype: str, side: str, amount: float, rate: float, params: Dict = {}) -> Dict[str, Any]: order_id = f'dry_run_{side}_{randint(0, 10**6)}' @@ -902,10 +913,18 @@ class Exchange: self._async_get_trade_history(pair=pair, since=since, until=until, from_id=from_id)) + def check_order_canceled_empty(self, order: Dict) -> bool: + """ + Verify if an order has been cancelled without being partially filled + :param order: Order dict as returned from get_order() + :return: True if order has been cancelled without being filled, False otherwise. + """ + return order.get('status') in ('closed', 'canceled') and order.get('filled') == 0.0 + @retrier - def cancel_order(self, order_id: str, pair: str) -> None: + def cancel_order(self, order_id: str, pair: str) -> Dict: if self._config['dry_run']: - return + return {} try: return self._api.cancel_order(order_id, pair) @@ -918,6 +937,37 @@ class Exchange: except ccxt.BaseError as e: raise OperationalException(e) from e + def is_cancel_order_result_suitable(self, corder) -> bool: + if not isinstance(corder, dict): + return False + + required = ('fee', 'status', 'amount') + return all(k in corder for k in required) + + def cancel_order_with_result(self, order_id: str, pair: str, amount: float) -> Dict: + """ + Cancel order returning a result. + Creates a fake result if cancel order returns a non-usable result + and get_order does not work (certain exchanges don't return cancelled orders) + :param order_id: Orderid to cancel + :param pair: Pair corresponding to order_id + :param amount: Amount to use for fake response + :return: Result from either cancel_order if usable, or fetch_order + """ + try: + corder = self.cancel_order(order_id, pair) + if self.is_cancel_order_result_suitable(corder): + return corder + except InvalidOrderException: + logger.warning(f"Could not cancel order {order_id}.") + try: + order = self.get_order(order_id, pair) + except InvalidOrderException: + logger.warning(f"Could not fetch cancelled order {order_id}.") + order = {'fee': {}, 'status': 'canceled', 'amount': amount, 'info': {}} + + return order + @retrier def get_order(self, order_id: str, pair: str) -> Dict: if self._config['dry_run']: diff --git a/freqtrade/freqtradebot.py b/freqtrade/freqtradebot.py index 570f8bea8..7ae87e807 100644 --- a/freqtrade/freqtradebot.py +++ b/freqtrade/freqtradebot.py @@ -20,12 +20,14 @@ from freqtrade.data.dataprovider import DataProvider from freqtrade.edge import Edge from freqtrade.exceptions import DependencyException, InvalidOrderException from freqtrade.exchange import timeframe_to_minutes, timeframe_to_next_date +from freqtrade.misc import safe_value_fallback from freqtrade.pairlist.pairlistmanager import PairListManager from freqtrade.persistence import Trade from freqtrade.resolvers import ExchangeResolver, StrategyResolver from freqtrade.rpc import RPCManager, RPCMessageType from freqtrade.state import State from freqtrade.strategy.interface import IStrategy, SellType +from freqtrade.strategy.strategy_wrapper import strategy_safe_wrapper from freqtrade.wallets import Wallets logger = logging.getLogger(__name__) @@ -143,6 +145,10 @@ class FreqtradeBot: self.dataprovider.refresh(self._create_pair_whitelist(self.active_pair_whitelist), self.strategy.informative_pairs()) + with self._sell_lock: + # Check and handle any timed out open orders + self.check_handle_timedout() + # Protect from collisions with forcesell. # Without this, freqtrade my try to recreate stoploss_on_exchange orders # while selling is in process, since telegram messages arrive in an different thread. @@ -154,8 +160,6 @@ class FreqtradeBot: if self.get_free_open_trades(): self.enter_positions() - # Check and handle any timed out open orders - self.check_handle_timedout() Trade.session.flush() def _refresh_whitelist(self, trades: List[Trade] = []) -> List[str]: @@ -600,7 +604,6 @@ class FreqtradeBot: trades_closed = 0 for trade in trades: try: - self.update_trade_state(trade) if (self.strategy.order_types.get('stoploss_on_exchange') and self.handle_stoploss_on_exchange(trade)): @@ -860,30 +863,35 @@ class FreqtradeBot: continue order = self.exchange.get_order(trade.open_order_id, trade.pair) except (RequestException, DependencyException, InvalidOrderException): - logger.info( - 'Cannot query order for %s due to %s', - trade, - traceback.format_exc()) + logger.info('Cannot query order for %s due to %s', trade, traceback.format_exc()) continue - # Check if trade is still actually open - if float(order.get('remaining', 0.0)) == 0.0: - self.wallets.update() - continue + trade_state_update = self.update_trade_state(trade, order) + + if (order['side'] == 'buy' and ( + trade_state_update + or self._check_timed_out('buy', order) + or strategy_safe_wrapper(self.strategy.check_buy_timeout, + default_retval=False)(pair=trade.pair, + trade=trade, + order=order))): - if ((order['side'] == 'buy' and order['status'] == 'canceled') - or (self._check_timed_out('buy', order))): self.handle_timedout_limit_buy(trade, order) self.wallets.update() order_type = self.strategy.order_types['buy'] self._notify_buy_cancel(trade, order_type) - elif ((order['side'] == 'sell' and order['status'] == 'canceled') - or (self._check_timed_out('sell', order))): - self.handle_timedout_limit_sell(trade, order) + elif (order['side'] == 'sell' and ( + trade_state_update + or self._check_timed_out('sell', order) + or strategy_safe_wrapper(self.strategy.check_sell_timeout, + default_retval=False)(pair=trade.pair, + trade=trade, + order=order))): + reason = self.handle_timedout_limit_sell(trade, order) self.wallets.update() order_type = self.strategy.order_types['sell'] - self._notify_sell_cancel(trade, order_type) + self._notify_sell_cancel(trade, order_type, reason) def handle_timedout_limit_buy(self, trade: Trade, order: Dict) -> bool: """ @@ -892,18 +900,17 @@ class FreqtradeBot: """ if order['status'] != 'canceled': reason = "cancelled due to timeout" - corder = self.exchange.cancel_order(trade.open_order_id, trade.pair) - # Some exchanges don't return a dict here. - if not isinstance(corder, dict): - corder = {} - logger.info('Buy order %s for %s.', reason, trade) + corder = self.exchange.cancel_order_with_result(trade.open_order_id, trade.pair, + trade.amount) else: # Order was cancelled already, so we can reuse the existing dict corder = order reason = "cancelled on exchange" - logger.info('Buy order %s for %s.', reason, trade) - if corder.get('remaining', order['remaining']) == order['amount']: + logger.info('Buy order %s for %s.', reason, trade) + + if safe_value_fallback(corder, order, 'remaining', 'remaining') == order['amount']: + logger.info('Buy order fully cancelled. Removing %s from database.', trade) # if trade is not partially completed, just delete the trade Trade.session.delete(trade) Trade.session.flush() @@ -914,19 +921,10 @@ class FreqtradeBot: # cancel_order may not contain the full order dict, so we need to fallback # to the order dict aquired before cancelling. # we need to fall back to the values from order if corder does not contain these keys. - trade.amount = order['amount'] - corder.get('remaining', order['remaining']) + trade.amount = order['amount'] - safe_value_fallback(corder, order, + 'remaining', 'remaining') trade.stake_amount = trade.amount * trade.open_rate - # verify if fees were taken from amount to avoid problems during selling - try: - new_amount = self.get_real_amount(trade, corder if 'fee' in corder else order, - trade.amount) - if not isclose(order['amount'], new_amount, abs_tol=constants.MATH_CLOSE_PREC): - trade.amount = new_amount - # Fee was applied, so set to 0 - trade.fee_open = 0 - trade.recalc_open_trade_price() - except DependencyException as e: - logger.warning("Could not update trade amount: %s", e) + self.update_trade_state(trade, corder, trade.amount) trade.open_order_id = None logger.info('Partial buy order timeout for %s.', trade) @@ -936,14 +934,14 @@ class FreqtradeBot: }) return False - def handle_timedout_limit_sell(self, trade: Trade, order: Dict) -> bool: + def handle_timedout_limit_sell(self, trade: Trade, order: Dict) -> str: """ Sell timeout - cancel order and update trade - :return: True if order was fully cancelled + :return: Reason for cancel """ # if trade is not partially completed, just cancel the trade - if order['remaining'] == order['amount']: - if order["status"] != "canceled": + if order['remaining'] == order['amount'] or order.get('filled') == 0.0: + if not self.exchange.check_order_canceled_empty(order): reason = "cancelled due to timeout" # if trade is not partially completed, just delete the trade self.exchange.cancel_order(trade.open_order_id, trade.pair) @@ -953,16 +951,17 @@ class FreqtradeBot: logger.info('Sell order %s for %s.', reason, trade) trade.close_rate = None + trade.close_rate_requested = None trade.close_profit = None trade.close_profit_abs = None trade.close_date = None trade.is_open = True trade.open_order_id = None - return True + return reason # TODO: figure out how to handle partially complete sell orders - return False + return 'partially filled - keeping order open' def _safe_sell_amount(self, pair: str, amount: float) -> float: """ @@ -1081,7 +1080,7 @@ class FreqtradeBot: # Send the message self.rpc.send_msg(msg) - def _notify_sell_cancel(self, trade: Trade, order_type: str) -> None: + def _notify_sell_cancel(self, trade: Trade, order_type: str, reason: str) -> None: """ Sends rpc notification when a sell cancel occured. """ @@ -1108,6 +1107,7 @@ class FreqtradeBot: 'close_date': trade.close_date, 'stake_currency': self.config['stake_currency'], 'fiat_currency': self.config.get('fiat_display_currency', None), + 'reason': reason, } if 'fiat_display_currency' in self.config: @@ -1122,9 +1122,12 @@ class FreqtradeBot: # Common update trade state methods # - def update_trade_state(self, trade: Trade, action_order: dict = None) -> None: + def update_trade_state(self, trade: Trade, action_order: dict = None, + order_amount: float = None) -> bool: """ Checks trades with open orders and updates the amount if necessary + Handles closing both buy and sell orders. + :return: True if order has been cancelled without being filled partially, False otherwise """ # Get order details for actual price per unit if trade.open_order_id: @@ -1134,25 +1137,31 @@ class FreqtradeBot: order = action_order or self.exchange.get_order(trade.open_order_id, trade.pair) except InvalidOrderException as exception: logger.warning('Unable to fetch order %s: %s', trade.open_order_id, exception) - return + return False # Try update amount (binance-fix) try: - new_amount = self.get_real_amount(trade, order) + new_amount = self.get_real_amount(trade, order, order_amount) if not isclose(order['amount'], new_amount, abs_tol=constants.MATH_CLOSE_PREC): order['amount'] = new_amount + order.pop('filled', None) # Fee was applied, so set to 0 trade.fee_open = 0 trade.recalc_open_trade_price() - except DependencyException as exception: logger.warning("Could not update trade amount: %s", exception) + if self.exchange.check_order_canceled_empty(order): + # Trade has been cancelled on exchange + # Handling of this will happen in check_handle_timeout. + return True trade.update(order) # Updating wallets when order is closed if not trade.is_open: self.wallets.update() + return False + def get_real_amount(self, trade: Trade, order: Dict, order_amount: float = None) -> float: """ Get real amount for the trade diff --git a/freqtrade/loggers.py b/freqtrade/loggers.py index c69388430..153ce8c80 100644 --- a/freqtrade/loggers.py +++ b/freqtrade/loggers.py @@ -18,13 +18,13 @@ def _set_loggers(verbosity: int = 0) -> None: """ logging.getLogger('requests').setLevel( - logging.INFO if verbosity <= 1 else logging.DEBUG + logging.INFO if verbosity <= 1 else logging.DEBUG ) logging.getLogger("urllib3").setLevel( - logging.INFO if verbosity <= 1 else logging.DEBUG + logging.INFO if verbosity <= 1 else logging.DEBUG ) logging.getLogger('ccxt.base.exchange').setLevel( - logging.INFO if verbosity <= 2 else logging.DEBUG + logging.INFO if verbosity <= 2 else logging.DEBUG ) logging.getLogger('telegram').setLevel(logging.INFO) diff --git a/freqtrade/misc.py b/freqtrade/misc.py index 1f52b75ec..ac6084eb7 100644 --- a/freqtrade/misc.py +++ b/freqtrade/misc.py @@ -134,6 +134,21 @@ def round_dict(d, n): return {k: (round(v, n) if isinstance(v, float) else v) for k, v in d.items()} +def safe_value_fallback(dict1: dict, dict2: dict, key1: str, key2: str, default_value=None): + """ + Search a value in dict1, return this if it's not None. + Fall back to dict2 - return key2 from dict2 if it's not None. + Else falls back to None. + + """ + if key1 in dict1 and dict1[key1] is not None: + return dict1[key1] + else: + if key2 in dict2 and dict2[key2] is not None: + return dict2[key2] + return default_value + + def plural(num: float, singular: str, plural: str = None) -> str: return singular if (num == 1 or num == -1) else plural or singular + 's' @@ -148,3 +163,15 @@ def render_template(templatefile: str, arguments: dict = {}) -> str: ) template = env.get_template(templatefile) return template.render(**arguments) + + +def render_template_with_fallback(templatefile: str, templatefallbackfile: str, + arguments: dict = {}) -> str: + """ + Use templatefile if possible, otherwise fall back to templatefallbackfile + """ + from jinja2.exceptions import TemplateNotFound + try: + return render_template(templatefile, arguments) + except TemplateNotFound: + return render_template(templatefallbackfile, arguments) diff --git a/freqtrade/optimize/backtesting.py b/freqtrade/optimize/backtesting.py index 1725a7d13..f29f599a6 100644 --- a/freqtrade/optimize/backtesting.py +++ b/freqtrade/optimize/backtesting.py @@ -149,8 +149,8 @@ class Backtesting: # To avoid using data from future, we use buy/sell signals shifted # from the previous candle - df_analyzed.loc[:, 'buy'] = df_analyzed['buy'].shift(1) - df_analyzed.loc[:, 'sell'] = df_analyzed['sell'].shift(1) + df_analyzed.loc[:, 'buy'] = df_analyzed.loc[:, 'buy'].shift(1) + df_analyzed.loc[:, 'sell'] = df_analyzed.loc[:, 'sell'].shift(1) df_analyzed.drop(df_analyzed.head(1).index, inplace=True) diff --git a/freqtrade/optimize/hyperopt.py b/freqtrade/optimize/hyperopt.py index c6ac3acbc..79b6b8cb0 100644 --- a/freqtrade/optimize/hyperopt.py +++ b/freqtrade/optimize/hyperopt.py @@ -7,7 +7,6 @@ This module contains the hyperopt logic import locale import logging import random -import sys import warnings from math import ceil from collections import OrderedDict @@ -18,10 +17,10 @@ from typing import Any, Dict, List, Optional import rapidjson from colorama import Fore, Style -from colorama import init as colorama_init from joblib import (Parallel, cpu_count, delayed, dump, load, wrap_non_picklable_objects) from pandas import DataFrame, json_normalize, isna +import progressbar import tabulate from os import path import io @@ -43,7 +42,8 @@ with warnings.catch_warnings(): from skopt import Optimizer from skopt.space import Dimension - +progressbar.streams.wrap_stderr() +progressbar.streams.wrap_stdout() logger = logging.getLogger(__name__) @@ -266,36 +266,17 @@ class Hyperopt: Log results if it is better than any previous evaluation """ is_best = results['is_best'] - if not self.print_all: - # Print '\n' after each 100th epoch to separate dots from the log messages. - # Otherwise output is messy on a terminal. - print('.', end='' if results['current_epoch'] % 100 != 0 else None) # type: ignore - sys.stdout.flush() if self.print_all or is_best: - if not self.print_all: - # Separate the results explanation string from dots - print("\n") - self.print_result_table(self.config, results, self.total_epochs, - self.print_all, self.print_colorized, - self.hyperopt_table_header) + print( + self.get_result_table( + self.config, results, self.total_epochs, + self.print_all, self.print_colorized, + self.hyperopt_table_header + ) + ) self.hyperopt_table_header = 2 - @staticmethod - def print_results_explanation(results, total_epochs, highlight_best: bool, - print_colorized: bool) -> None: - """ - Log results explanation string - """ - explanation_str = Hyperopt._format_explanation_string(results, total_epochs) - # Colorize output - if print_colorized: - if results['total_profit'] > 0: - explanation_str = Fore.GREEN + explanation_str - if highlight_best and results['is_best']: - explanation_str = Style.BRIGHT + explanation_str - print(explanation_str) - @staticmethod def _format_explanation_string(results, total_epochs) -> str: return (("*" if results['is_initial_point'] else " ") + @@ -304,13 +285,13 @@ class Hyperopt: f"Objective: {results['loss']:.5f}") @staticmethod - def print_result_table(config: dict, results: list, total_epochs: int, highlight_best: bool, - print_colorized: bool, remove_header: int) -> None: + def get_result_table(config: dict, results: list, total_epochs: int, highlight_best: bool, + print_colorized: bool, remove_header: int) -> str: """ Log result table """ if not results: - return + return '' tabulate.PRESERVE_WHITESPACE = True @@ -323,8 +304,9 @@ class Hyperopt: trials.columns = ['Best', 'Epoch', 'Trades', 'Avg profit', 'Total profit', 'Profit', 'Avg duration', 'Objective', 'is_initial_point', 'is_best'] trials['is_profit'] = False - trials.loc[trials['is_initial_point'], 'Best'] = '*' + trials.loc[trials['is_initial_point'], 'Best'] = '* ' trials.loc[trials['is_best'], 'Best'] = 'Best' + trials.loc[trials['is_initial_point'] & trials['is_best'], 'Best'] = '* Best' trials.loc[trials['Total profit'] > 0, 'is_profit'] = True trials['Trades'] = trials['Trades'].astype(str) @@ -381,7 +363,7 @@ class Hyperopt: trials.to_dict(orient='list'), tablefmt='psql', headers='keys', stralign="right" ) - print(table) + return table @staticmethod def export_csv_file(config: dict, results: list, total_epochs: int, highlight_best: bool, @@ -415,6 +397,7 @@ class Hyperopt: trials['is_profit'] = False trials.loc[trials['is_initial_point'], 'Best'] = '*' trials.loc[trials['is_best'], 'Best'] = 'Best' + trials.loc[trials['is_initial_point'] & trials['is_best'], 'Best'] = '* Best' trials.loc[trials['Total profit'] > 0, 'is_profit'] = True trials['Epoch'] = trials['Epoch'].astype(str) trials['Trades'] = trials['Trades'].astype(str) @@ -653,48 +636,75 @@ class Hyperopt: self.dimensions: List[Dimension] = self.hyperopt_space() self.opt = self.get_optimizer(self.dimensions, config_jobs) - - if self.print_colorized: - colorama_init(autoreset=True) - try: with Parallel(n_jobs=config_jobs) as parallel: jobs = parallel._effective_n_jobs() logger.info(f'Effective number of parallel workers used: {jobs}') - EVALS = ceil(self.total_epochs / jobs) - for i in range(EVALS): - # Correct the number of epochs to be processed for the last - # iteration (should not exceed self.total_epochs in total) - n_rest = (i + 1) * jobs - self.total_epochs - current_jobs = jobs - n_rest if n_rest > 0 else jobs - asked = self.opt.ask(n_points=current_jobs) - f_val = self.run_optimizer_parallel(parallel, asked, i) - self.opt.tell(asked, [v['loss'] for v in f_val]) - self.fix_optimizer_models_list() + # Define progressbar + if self.print_colorized: + widgets = [ + ' [Epoch ', progressbar.Counter(), ' of ', str(self.total_epochs), + ' (', progressbar.Percentage(), ')] ', + progressbar.Bar(marker=progressbar.AnimatedMarker( + fill='\N{FULL BLOCK}', + fill_wrap=Fore.GREEN + '{}' + Fore.RESET, + marker_wrap=Style.BRIGHT + '{}' + Style.RESET_ALL, + )), + ' [', progressbar.ETA(), ', ', progressbar.Timer(), ']', + ] + else: + widgets = [ + ' [Epoch ', progressbar.Counter(), ' of ', str(self.total_epochs), + ' (', progressbar.Percentage(), ')] ', + progressbar.Bar(marker=progressbar.AnimatedMarker( + fill='\N{FULL BLOCK}', + )), + ' [', progressbar.ETA(), ', ', progressbar.Timer(), ']', + ] + with progressbar.ProgressBar( + maxval=self.total_epochs, redirect_stdout=False, redirect_stderr=False, + widgets=widgets + ) as pbar: + EVALS = ceil(self.total_epochs / jobs) + for i in range(EVALS): + # Correct the number of epochs to be processed for the last + # iteration (should not exceed self.total_epochs in total) + n_rest = (i + 1) * jobs - self.total_epochs + current_jobs = jobs - n_rest if n_rest > 0 else jobs - for j, val in enumerate(f_val): - # Use human-friendly indexes here (starting from 1) - current = i * jobs + j + 1 - val['current_epoch'] = current - val['is_initial_point'] = current <= INITIAL_POINTS - logger.debug(f"Optimizer epoch evaluated: {val}") + asked = self.opt.ask(n_points=current_jobs) + f_val = self.run_optimizer_parallel(parallel, asked, i) + self.opt.tell(asked, [v['loss'] for v in f_val]) + self.fix_optimizer_models_list() - is_best = self.is_best_loss(val, self.current_best_loss) - # This value is assigned here and not in the optimization method - # to keep proper order in the list of results. That's because - # evaluations can take different time. Here they are aligned in the - # order they will be shown to the user. - val['is_best'] = is_best + # Calculate progressbar outputs + for j, val in enumerate(f_val): + # Use human-friendly indexes here (starting from 1) + current = i * jobs + j + 1 + val['current_epoch'] = current + val['is_initial_point'] = current <= INITIAL_POINTS - self.print_results(val) + logger.debug(f"Optimizer epoch evaluated: {val}") + + is_best = self.is_best_loss(val, self.current_best_loss) + # This value is assigned here and not in the optimization method + # to keep proper order in the list of results. That's because + # evaluations can take different time. Here they are aligned in the + # order they will be shown to the user. + val['is_best'] = is_best + self.print_results(val) + + if is_best: + self.current_best_loss = val['loss'] + self.trials.append(val) + + # Save results after each best epoch and every 100 epochs + if is_best or current % 100 == 0: + self.save_trials() + + pbar.update(current) - if is_best: - self.current_best_loss = val['loss'] - self.trials.append(val) - # Save results after each best epoch and every 100 epochs - if is_best or current % 100 == 0: - self.save_trials() except KeyboardInterrupt: print('User interrupted..') diff --git a/freqtrade/optimize/optimize_reports.py b/freqtrade/optimize/optimize_reports.py index 251da9159..646afb5df 100644 --- a/freqtrade/optimize/optimize_reports.py +++ b/freqtrade/optimize/optimize_reports.py @@ -24,13 +24,14 @@ def store_backtest_result(recordfilename: Path, all_results: Dict[str, DataFrame for index, t in results.iterrows()] if records: + filename = recordfilename if len(all_results) > 1: # Inject strategy to filename - recordfilename = Path.joinpath( + filename = Path.joinpath( recordfilename.parent, f'{recordfilename.stem}-{strategy}').with_suffix(recordfilename.suffix) - logger.info(f'Dumping backtest results to {recordfilename}') - file_dump_json(recordfilename, records) + logger.info(f'Dumping backtest results to {filename}') + file_dump_json(filename, records) def generate_text_table(data: Dict[str, Dict], stake_currency: str, max_open_trades: int, diff --git a/freqtrade/pairlist/IPairList.py b/freqtrade/pairlist/IPairList.py index 35844a99e..e089e546c 100644 --- a/freqtrade/pairlist/IPairList.py +++ b/freqtrade/pairlist/IPairList.py @@ -9,6 +9,8 @@ from abc import ABC, abstractmethod, abstractproperty from copy import deepcopy from typing import Any, Dict, List +from cachetools import TTLCache, cached + from freqtrade.exchange import market_is_active logger = logging.getLogger(__name__) @@ -31,6 +33,9 @@ class IPairList(ABC): self._config = config self._pairlistconfig = pairlistconfig self._pairlist_pos = pairlist_pos + self.refresh_period = self._pairlistconfig.get('refresh_period', 1800) + self._last_refresh = 0 + self._log_cache = TTLCache(maxsize=1024, ttl=self.refresh_period) @property def name(self) -> str: @@ -40,6 +45,24 @@ class IPairList(ABC): """ return self.__class__.__name__ + def log_on_refresh(self, logmethod, message: str) -> None: + """ + Logs message - not more often than "refresh_period" to avoid log spamming + Logs the log-message as debug as well to simplify debugging. + :param logmethod: Function that'll be called. Most likely `logger.info`. + :param message: String containing the message to be sent to the function. + :return: None. + """ + + @cached(cache=self._log_cache) + def _log_on_refresh(message: str): + logmethod(message) + + # Log as debug first + logger.debug(message) + # Call hidden function. + _log_on_refresh(message) + @abstractproperty def needstickers(self) -> bool: """ diff --git a/freqtrade/pairlist/PrecisionFilter.py b/freqtrade/pairlist/PrecisionFilter.py index f16458ca5..2a2ba46b7 100644 --- a/freqtrade/pairlist/PrecisionFilter.py +++ b/freqtrade/pairlist/PrecisionFilter.py @@ -39,8 +39,9 @@ class PrecisionFilter(IPairList): stop_gap_price = self._exchange.price_to_precision(ticker["symbol"], stop_price * 0.99) logger.debug(f"{ticker['symbol']} - {sp} : {stop_gap_price}") if sp <= stop_gap_price: - logger.info(f"Removed {ticker['symbol']} from whitelist, " - f"because stop price {sp} would be <= stop limit {stop_gap_price}") + self.log_on_refresh(logger.info, + f"Removed {ticker['symbol']} from whitelist, " + f"because stop price {sp} would be <= stop limit {stop_gap_price}") return False return True diff --git a/freqtrade/pairlist/PriceFilter.py b/freqtrade/pairlist/PriceFilter.py index dc02ae251..2f7e98e24 100644 --- a/freqtrade/pairlist/PriceFilter.py +++ b/freqtrade/pairlist/PriceFilter.py @@ -35,21 +35,24 @@ class PriceFilter(IPairList): """ Check if if one price-step (pip) is > than a certain barrier. :param ticker: ticker dict as returned from ccxt.load_markets() - :param precision: Precision :return: True if the pair can stay, false if it should be removed """ - precision = self._exchange.markets[ticker['symbol']]['precision']['price'] + if ticker['last'] is None: - compare = ticker['last'] + 1 / pow(10, precision) + self.log_on_refresh(logger.info, + f"Removed {ticker['symbol']} from whitelist, because " + "ticker['last'] is empty (Usually no trade in the last 24h).") + return False + compare = ticker['last'] + self._exchange.price_get_one_pip(ticker['symbol'], + ticker['last']) changeperc = (compare - ticker['last']) / ticker['last'] if changeperc > self._low_price_ratio: - logger.info(f"Removed {ticker['symbol']} from whitelist, " - f"because 1 unit is {changeperc * 100:.3f}%") + self.log_on_refresh(logger.info, f"Removed {ticker['symbol']} from whitelist, " + f"because 1 unit is {changeperc * 100:.3f}%") return False return True def filter_pairlist(self, pairlist: List[str], tickers: Dict) -> List[str]: - """ Filters and sorts pairlist and returns the whitelist again. Called on each bot iteration - please use internal caching if necessary diff --git a/freqtrade/pairlist/SpreadFilter.py b/freqtrade/pairlist/SpreadFilter.py index 9361837cc..49731ef11 100644 --- a/freqtrade/pairlist/SpreadFilter.py +++ b/freqtrade/pairlist/SpreadFilter.py @@ -49,9 +49,9 @@ class SpreadFilter(IPairList): if 'bid' in ticker and 'ask' in ticker: spread = 1 - ticker['bid'] / ticker['ask'] if not ticker or spread > self._max_spread_ratio: - logger.info(f"Removed {ticker['symbol']} from whitelist, " - f"because spread {spread * 100:.3f}% >" - f"{self._max_spread_ratio * 100}%") + self.log_on_refresh(logger.info, f"Removed {ticker['symbol']} from whitelist, " + f"because spread {spread * 100:.3f}% >" + f"{self._max_spread_ratio * 100}%") pairlist.remove(p) else: pairlist.remove(p) diff --git a/freqtrade/pairlist/VolumePairList.py b/freqtrade/pairlist/VolumePairList.py index 9ce2adc9e..65f43245c 100644 --- a/freqtrade/pairlist/VolumePairList.py +++ b/freqtrade/pairlist/VolumePairList.py @@ -39,7 +39,6 @@ class VolumePairList(IPairList): if not self._validate_keys(self._sort_key): raise OperationalException( f'key {self._sort_key} not in {SORT_VALUES}') - self._last_refresh = 0 @property def needstickers(self) -> bool: @@ -68,16 +67,18 @@ class VolumePairList(IPairList): :return: new whitelist """ # Generate dynamic whitelist - if self._last_refresh + self.refresh_period < datetime.now().timestamp(): + # Must always run if this pairlist is not the first in the list. + if (self._pairlist_pos != 0 or + (self._last_refresh + self.refresh_period < datetime.now().timestamp())): + self._last_refresh = int(datetime.now().timestamp()) - return self._gen_pair_whitelist(pairlist, - tickers, - self._config['stake_currency'], - self._sort_key, - self._min_value - ) + pairs = self._gen_pair_whitelist(pairlist, tickers, + self._config['stake_currency'], + self._sort_key, self._min_value) else: - return pairlist + pairs = pairlist + self.log_on_refresh(logger.info, f"Searching {self._number_pairs} pairs: {pairs}") + return pairs def _gen_pair_whitelist(self, pairlist: List[str], tickers: Dict, base_currency: str, key: str, min_val: int) -> List[str]: @@ -88,7 +89,6 @@ class VolumePairList(IPairList): :param tickers: Tickers (from exchange.get_tickers()). :return: List of pairs """ - if self._pairlist_pos == 0: # If VolumePairList is the first in the list, use fresh pairlist # Check if pair quote currency equals to the stake currency. @@ -109,6 +109,5 @@ class VolumePairList(IPairList): pairs = self._verify_blacklist(pairs, aswarning=False) # Limit to X number of pairs pairs = pairs[:self._number_pairs] - logger.info(f"Searching {self._number_pairs} pairs: {pairs}") return pairs diff --git a/freqtrade/persistence.py b/freqtrade/persistence.py index 0d668596c..fb314f439 100644 --- a/freqtrade/persistence.py +++ b/freqtrade/persistence.py @@ -188,7 +188,7 @@ class Trade(_DECL_BASE): fee_close = Column(Float, nullable=False, default=0.0) open_rate = Column(Float) open_rate_requested = Column(Float) - # open_trade_price - calcuated via _calc_open_trade_price + # open_trade_price - calculated via _calc_open_trade_price open_trade_price = Column(Float) close_rate = Column(Float) close_rate_requested = Column(Float) @@ -233,6 +233,9 @@ class Trade(_DECL_BASE): return { 'trade_id': self.id, 'pair': self.pair, + 'is_open': self.is_open, + 'fee_open': self.fee_open, + 'fee_close': self.fee_close, 'open_date_hum': arrow.get(self.open_date).humanize(), 'open_date': self.open_date.strftime("%Y-%m-%d %H:%M:%S"), 'close_date_hum': (arrow.get(self.close_date).humanize() @@ -240,14 +243,24 @@ class Trade(_DECL_BASE): 'close_date': (self.close_date.strftime("%Y-%m-%d %H:%M:%S") if self.close_date else None), 'open_rate': self.open_rate, + 'open_rate_requested': self.open_rate_requested, + 'open_trade_price': self.open_trade_price, 'close_rate': self.close_rate, + 'close_rate_requested': self.close_rate_requested, 'amount': round(self.amount, 8), 'stake_amount': round(self.stake_amount, 8), + 'close_profit': self.close_profit, + 'sell_reason': self.sell_reason, 'stop_loss': self.stop_loss, 'stop_loss_pct': (self.stop_loss_pct * 100) if self.stop_loss_pct else None, 'initial_stop_loss': self.initial_stop_loss, 'initial_stop_loss_pct': (self.initial_stop_loss_pct * 100 if self.initial_stop_loss_pct else None), + 'min_rate': self.min_rate, + 'max_rate': self.max_rate, + 'strategy': self.strategy, + 'ticker_interval': self.ticker_interval, + 'open_order_id': self.open_order_id, } def adjust_min_max_rates(self, current_price: float) -> None: @@ -315,7 +328,7 @@ class Trade(_DECL_BASE): if order_type in ('market', 'limit') and order['side'] == 'buy': # Update open rate and actual amount self.open_rate = Decimal(order['price']) - self.amount = Decimal(order['amount']) + self.amount = Decimal(order.get('filled', order['amount'])) self.recalc_open_trade_price() logger.info('%s_BUY has been fulfilled for %s.', order_type.upper(), self) self.open_order_id = None diff --git a/freqtrade/plot/plotting.py b/freqtrade/plot/plotting.py index fc8f25612..60f838db2 100644 --- a/freqtrade/plot/plotting.py +++ b/freqtrade/plot/plotting.py @@ -10,6 +10,7 @@ from freqtrade.data.btanalysis import (calculate_max_drawdown, create_cum_profit, extract_trades_of_period, load_trades) from freqtrade.data.converter import trim_dataframe +from freqtrade.exchange import timeframe_to_prev_date from freqtrade.data.history import load_data from freqtrade.misc import pair_to_filename from freqtrade.resolvers import StrategyResolver @@ -122,7 +123,8 @@ def add_profit(fig, row, data: pd.DataFrame, column: str, name: str) -> make_sub return fig -def add_max_drawdown(fig, row, trades: pd.DataFrame, df_comb: pd.DataFrame) -> make_subplots: +def add_max_drawdown(fig, row, trades: pd.DataFrame, df_comb: pd.DataFrame, + timeframe: str) -> make_subplots: """ Add scatter points indicating max drawdown """ @@ -132,12 +134,12 @@ def add_max_drawdown(fig, row, trades: pd.DataFrame, df_comb: pd.DataFrame) -> m drawdown = go.Scatter( x=[highdate, lowdate], y=[ - df_comb.loc[highdate, 'cum_profit'], - df_comb.loc[lowdate, 'cum_profit'], + df_comb.loc[timeframe_to_prev_date(timeframe, highdate), 'cum_profit'], + df_comb.loc[timeframe_to_prev_date(timeframe, lowdate), 'cum_profit'], ], mode='markers', - name=f"Max drawdown {max_drawdown:.2f}%", - text=f"Max drawdown {max_drawdown:.2f}%", + name=f"Max drawdown {max_drawdown * 100:.2f}%", + text=f"Max drawdown {max_drawdown * 100:.2f}%", marker=dict( symbol='square-open', size=9, @@ -383,6 +385,9 @@ def generate_profit_graph(pairs: str, data: Dict[str, pd.DataFrame], # Combine close-values for all pairs, rename columns to "pair" df_comb = combine_dataframes_with_mean(data, "close") + # Trim trades to available OHLCV data + trades = extract_trades_of_period(df_comb, trades, date_index=True) + # Add combined cumulative profit df_comb = create_cum_profit(df_comb, trades, 'cum_profit', timeframe) @@ -405,7 +410,7 @@ def generate_profit_graph(pairs: str, data: Dict[str, pd.DataFrame], fig.add_trace(avgclose, 1, 1) fig = add_profit(fig, 2, df_comb, 'cum_profit', 'Profit') - fig = add_max_drawdown(fig, 2, trades, df_comb) + fig = add_max_drawdown(fig, 2, trades, df_comb, timeframe) for pair in pairs: profit_col = f'cum_profit_{pair}' diff --git a/freqtrade/rpc/api_server.py b/freqtrade/rpc/api_server.py index 8f4cc4787..0335bb151 100644 --- a/freqtrade/rpc/api_server.py +++ b/freqtrade/rpc/api_server.py @@ -173,7 +173,8 @@ class ApiServer(RPC): view_func=self._show_config, methods=['GET']) self.app.add_url_rule(f'{BASE_URI}/ping', 'ping', view_func=self._ping, methods=['GET']) - + self.app.add_url_rule(f'{BASE_URI}/trades', 'trades', + view_func=self._trades, methods=['GET']) # Combined actions and infos self.app.add_url_rule(f'{BASE_URI}/blacklist', 'blacklist', view_func=self._blacklist, methods=['GET', 'POST']) @@ -358,6 +359,18 @@ class ApiServer(RPC): self._config.get('fiat_display_currency', '')) return self.rest_dump(results) + @require_login + @rpc_catch_errors + def _trades(self): + """ + Handler for /trades. + + Returns the X last trades in json format + """ + limit = int(request.args.get('limit', 0)) + results = self._rpc_trade_history(limit) + return self.rest_dump(results) + @require_login @rpc_catch_errors def _whitelist(self): diff --git a/freqtrade/rpc/rpc.py b/freqtrade/rpc/rpc.py index a0f50b070..8645e466e 100644 --- a/freqtrade/rpc/rpc.py +++ b/freqtrade/rpc/rpc.py @@ -226,6 +226,20 @@ class RPC: for key, value in profit_days.items() ] + def _rpc_trade_history(self, limit: int) -> Dict: + """ Returns the X last trades """ + if limit > 0: + trades = Trade.get_trades().order_by(Trade.id.desc()).limit(limit) + else: + trades = Trade.get_trades().order_by(Trade.id.desc()).all() + + output = [trade.to_json() for trade in trades] + + return { + "trades": output, + "trades_count": len(output) + } + def _rpc_trade_statistics( self, stake_currency: str, fiat_display_currency: str) -> Dict[str, Any]: """ Returns cumulative profit statistics """ diff --git a/freqtrade/rpc/telegram.py b/freqtrade/rpc/telegram.py index ad01700ab..a21f7556c 100644 --- a/freqtrade/rpc/telegram.py +++ b/freqtrade/rpc/telegram.py @@ -172,7 +172,8 @@ class Telegram(RPC): ' / {profit_fiat:.3f} {fiat_currency})`').format(**msg) elif msg['type'] == RPCMessageType.SELL_CANCEL_NOTIFICATION: - message = "*{exchange}:* Cancelling Open Sell Order for {pair}".format(**msg) + message = ("*{exchange}:* Cancelling Open Sell Order " + "for {pair}. Reason: {reason}").format(**msg) elif msg['type'] == RPCMessageType.STATUS_NOTIFICATION: message = '*Status:* `{status}`'.format(**msg) diff --git a/freqtrade/strategy/interface.py b/freqtrade/strategy/interface.py index 696d2b2d2..6268b8a43 100644 --- a/freqtrade/strategy/interface.py +++ b/freqtrade/strategy/interface.py @@ -3,21 +3,22 @@ IStrategy interface This module defines the interface to apply for strategies """ import logging +import warnings from abc import ABC, abstractmethod from datetime import datetime, timezone from enum import Enum from typing import Dict, List, NamedTuple, Optional, Tuple -import warnings import arrow from pandas import DataFrame from freqtrade.data.dataprovider import DataProvider +from freqtrade.exceptions import StrategyError from freqtrade.exchange import timeframe_to_minutes from freqtrade.persistence import Trade +from freqtrade.strategy.strategy_wrapper import strategy_safe_wrapper from freqtrade.wallets import Wallets - logger = logging.getLogger(__name__) @@ -148,6 +149,42 @@ class IStrategy(ABC): :return: DataFrame with sell column """ + def check_buy_timeout(self, pair: str, trade: Trade, order: dict, **kwargs) -> bool: + """ + Check buy timeout function callback. + This method can be used to override the buy-timeout. + It is called whenever a limit buy order has been created, + and is not yet fully filled. + Configuration options in `unfilledtimeout` will be verified before this, + so ensure to set these timeouts high enough. + + When not implemented by a strategy, this simply returns False. + :param pair: Pair the trade is for + :param trade: trade object. + :param order: Order dictionary as returned from CCXT. + :param **kwargs: Ensure to keep this here so updates to this won't break your strategy. + :return bool: When True is returned, then the buy-order is cancelled. + """ + return False + + def check_sell_timeout(self, pair: str, trade: Trade, order: dict, **kwargs) -> bool: + """ + Check sell timeout function callback. + This method can be used to override the sell-timeout. + It is called whenever a limit sell order has been created, + and is not yet fully filled. + Configuration options in `unfilledtimeout` will be verified before this, + so ensure to set these timeouts high enough. + + When not implemented by a strategy, this simply returns False. + :param pair: Pair the trade is for + :param trade: trade object. + :param order: Order dictionary as returned from CCXT. + :param **kwargs: Ensure to keep this here so updates to this won't break your strategy. + :return bool: When True is returned, then the sell-order is cancelled. + """ + return False + def informative_pairs(self) -> List[Tuple[str, str]]: """ Define additional, informative pair/interval combinations to be cached from the exchange. @@ -241,8 +278,25 @@ class IStrategy(ABC): return dataframe - def get_signal(self, pair: str, interval: str, - dataframe: DataFrame) -> Tuple[bool, bool]: + @staticmethod + def preserve_df(dataframe: DataFrame) -> Tuple[int, float, datetime]: + """ keep some data for dataframes """ + return len(dataframe), dataframe["close"].iloc[-1], dataframe["date"].iloc[-1] + + @staticmethod + def assert_df(dataframe: DataFrame, df_len: int, df_close: float, df_date: datetime): + """ make sure data is unmodified """ + message = "" + if df_len != len(dataframe): + message = "length" + elif df_close != dataframe["close"].iloc[-1]: + message = "last close price" + elif df_date != dataframe["date"].iloc[-1]: + message = "last date" + if message: + raise StrategyError(f"Dataframe returned from strategy has mismatching {message}.") + + def get_signal(self, pair: str, interval: str, dataframe: DataFrame) -> Tuple[bool, bool]: """ Calculates current signal based several technical analysis indicators :param pair: pair in format ANT/BTC @@ -254,28 +308,23 @@ class IStrategy(ABC): logger.warning('Empty candle (OHLCV) data for pair %s', pair) return False, False + latest_date = dataframe['date'].max() try: - dataframe = self._analyze_ticker_internal(dataframe, {'pair': pair}) - except ValueError as error: - logger.warning( - 'Unable to analyze candle (OHLCV) data for pair %s: %s', - pair, - str(error) - ) - return False, False - except Exception as error: - logger.exception( - 'Unexpected error when analyzing candle (OHLCV) data for pair %s: %s', - pair, - str(error) - ) + df_len, df_close, df_date = self.preserve_df(dataframe) + dataframe = strategy_safe_wrapper( + self._analyze_ticker_internal, message="" + )(dataframe, {'pair': pair}) + self.assert_df(dataframe, df_len, df_close, df_date) + except StrategyError as error: + logger.warning(f"Unable to analyze candle (OHLCV) data for pair {pair}: {error}") + return False, False if dataframe.empty: logger.warning('Empty dataframe for pair %s', pair) return False, False - latest = dataframe.iloc[-1] + latest = dataframe.loc[dataframe['date'] == latest_date].iloc[-1] # Check if dataframe is out of date signal_date = arrow.get(latest['date']) @@ -444,8 +493,11 @@ class IStrategy(ABC): """ Creates a dataframe and populates indicators for given candle (OHLCV) data Used by optimize operations only, not during dry / live runs. + Using .copy() to get a fresh copy of the dataframe for every strategy run. + Has positive effects on memory usage for whatever reason - also when + using only one strategy. """ - return {pair: self.advise_indicators(pair_data, {'pair': pair}) + return {pair: self.advise_indicators(pair_data.copy(), {'pair': pair}) for pair, pair_data in data.items()} def advise_indicators(self, dataframe: DataFrame, metadata: dict) -> DataFrame: diff --git a/freqtrade/strategy/strategy_wrapper.py b/freqtrade/strategy/strategy_wrapper.py new file mode 100644 index 000000000..7b9da9140 --- /dev/null +++ b/freqtrade/strategy/strategy_wrapper.py @@ -0,0 +1,35 @@ +import logging + +from freqtrade.exceptions import StrategyError + +logger = logging.getLogger(__name__) + + +def strategy_safe_wrapper(f, message: str = "", default_retval=None): + """ + Wrapper around user-provided methods and functions. + Caches all exceptions and returns either the default_retval (if it's not None) or raises + a StrategyError exception, which then needs to be handled by the calling method. + """ + def wrapper(*args, **kwargs): + try: + return f(*args, **kwargs) + except ValueError as error: + logger.warning( + f"{message}" + f"Strategy caused the following exception: {error}" + f"{f}" + ) + if default_retval is None: + raise StrategyError(str(error)) from error + return default_retval + except Exception as error: + logger.exception( + f"{message}" + f"Unexpected error {error} calling {f}" + ) + if default_retval is None: + raise StrategyError(str(error)) from error + return default_retval + + return wrapper diff --git a/freqtrade/templates/base_strategy.py.j2 b/freqtrade/templates/base_strategy.py.j2 index 97a189ff4..c37164568 100644 --- a/freqtrade/templates/base_strategy.py.j2 +++ b/freqtrade/templates/base_strategy.py.j2 @@ -137,3 +137,4 @@ class {{ strategy }}(IStrategy): ), 'sell'] = 1 return dataframe + {{ additional_methods | indent(4) }} diff --git a/freqtrade/templates/subtemplates/strategy_methods_advanced.j2 b/freqtrade/templates/subtemplates/strategy_methods_advanced.j2 new file mode 100644 index 000000000..0ca35e117 --- /dev/null +++ b/freqtrade/templates/subtemplates/strategy_methods_advanced.j2 @@ -0,0 +1,40 @@ + +def check_buy_timeout(self, pair: str, trade: 'Trade', order: dict, **kwargs) -> bool: + """ + Check buy timeout function callback. + This method can be used to override the buy-timeout. + It is called whenever a limit buy order has been created, + and is not yet fully filled. + Configuration options in `unfilledtimeout` will be verified before this, + so ensure to set these timeouts high enough. + + For full documentation please go to https://www.freqtrade.io/en/latest/strategy-advanced/ + + When not implemented by a strategy, this simply returns False. + :param pair: Pair the trade is for + :param trade: trade object. + :param order: Order dictionary as returned from CCXT. + :param **kwargs: Ensure to keep this here so updates to this won't break your strategy. + :return bool: When True is returned, then the buy-order is cancelled. + """ + return False + +def check_sell_timeout(self, pair: str, trade: 'Trade', order: dict, **kwargs) -> bool: + """ + Check sell timeout function callback. + This method can be used to override the sell-timeout. + It is called whenever a limit sell order has been created, + and is not yet fully filled. + Configuration options in `unfilledtimeout` will be verified before this, + so ensure to set these timeouts high enough. + + For full documentation please go to https://www.freqtrade.io/en/latest/strategy-advanced/ + + When not implemented by a strategy, this simply returns False. + :param pair: Pair the trade is for + :param trade: trade object. + :param order: Order dictionary as returned from CCXT. + :param **kwargs: Ensure to keep this here so updates to this won't break your strategy. + :return bool: When True is returned, then the sell-order is cancelled. + """ + return False diff --git a/freqtrade/templates/subtemplates/strategy_methods_empty.j2 b/freqtrade/templates/subtemplates/strategy_methods_empty.j2 new file mode 100644 index 000000000..e69de29bb diff --git a/mkdocs.yml b/mkdocs.yml index 4e7e6ff75..ae24e150c 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -24,6 +24,7 @@ nav: - Plotting: plotting.md - SQL Cheatsheet: sql_cheatsheet.md - Advanced Post-installation Tasks: advanced-setup.md + - Advanced Strategy: strategy-advanced.md - Advanced Hyperopt: advanced-hyperopt.md - Sandbox Testing: sandbox-testing.md - Deprecated Features: deprecated.md diff --git a/requirements-common.txt b/requirements-common.txt index e490c8927..a53fc3999 100644 --- a/requirements-common.txt +++ b/requirements-common.txt @@ -1,18 +1,18 @@ # requirements without requirements installable via conda # mainly used for Raspberry pi installs -ccxt==1.24.83 -SQLAlchemy==1.3.15 -python-telegram-bot==12.4.2 +ccxt==1.27.1 +SQLAlchemy==1.3.16 +python-telegram-bot==12.6.1 arrow==0.15.5 -cachetools==4.0.0 +cachetools==4.1.0 requests==2.23.0 -urllib3==1.25.8 +urllib3==1.25.9 wrapt==1.12.1 jsonschema==3.2.0 TA-Lib==0.4.17 tabulate==0.8.7 pycoingecko==1.2.0 -jinja2==2.11.1 +jinja2==2.11.2 # find first, C search in arrays py_find_1st==1.1.4 @@ -24,10 +24,10 @@ python-rapidjson==0.9.1 sdnotify==0.3.2 # Api server -flask==1.1.1 +flask==1.1.2 # Support for colorized terminal output colorama==0.4.3 # Building config files interactively -questionary==1.5.1 -prompt-toolkit==3.0.4 +questionary==1.5.2 +prompt-toolkit==3.0.5 diff --git a/requirements-dev.txt b/requirements-dev.txt index a4d83eb4f..508716bde 100644 --- a/requirements-dev.txt +++ b/requirements-dev.txt @@ -3,15 +3,15 @@ -r requirements-plot.txt -r requirements-hyperopt.txt -coveralls==1.11.1 +coveralls==2.0.0 flake8==3.7.9 flake8-type-annotations==0.1.0 -flake8-tidy-imports==4.0.0 +flake8-tidy-imports==4.1.0 mypy==0.770 pytest==5.4.1 -pytest-asyncio==0.10.0 +pytest-asyncio==0.11.0 pytest-cov==2.8.1 -pytest-mock==2.0.0 +pytest-mock==3.1.0 pytest-random-order==1.0.4 # Convert jupyter notebooks to markdown documents diff --git a/requirements-hyperopt.txt b/requirements-hyperopt.txt index c7e586a33..b0e18867d 100644 --- a/requirements-hyperopt.txt +++ b/requirements-hyperopt.txt @@ -7,3 +7,4 @@ scikit-learn==0.22.2.post1 scikit-optimize==0.7.4 filelock==3.0.12 joblib==0.14.1 +progressbar2==3.51.0 diff --git a/requirements-plot.txt b/requirements-plot.txt index 7a5b21e2d..3db48a201 100644 --- a/requirements-plot.txt +++ b/requirements-plot.txt @@ -1,5 +1,5 @@ # Include all requirements to run the bot. -r requirements.txt -plotly==4.5.4 +plotly==4.6.0 diff --git a/requirements.txt b/requirements.txt index b1a4b4403..967f8df10 100644 --- a/requirements.txt +++ b/requirements.txt @@ -1,5 +1,5 @@ # Load common requirements -r requirements-common.txt -numpy==1.18.2 +numpy==1.18.3 pandas==1.0.3 diff --git a/scripts/rest_client.py b/scripts/rest_client.py index ccb33604f..b26c32479 100755 --- a/scripts/rest_client.py +++ b/scripts/rest_client.py @@ -156,6 +156,14 @@ class FtRestClient(): """ return self._get("show_config") + def trades(self, limit=None): + """Return trades history. + + :param limit: Limits trades to the X last trades. No limit to get all the trades. + :return: json object + """ + return self._get("trades", params={"limit": limit} if limit else 0) + def whitelist(self): """Show the current whitelist. diff --git a/setup.py b/setup.py index 7890f862e..94c48a6a7 100644 --- a/setup.py +++ b/setup.py @@ -24,6 +24,7 @@ hyperopt = [ 'scikit-optimize', 'filelock', 'joblib', + 'progressbar2', ] develop = [ diff --git a/setup.sh b/setup.sh index e120190ce..918c41e6b 100755 --- a/setup.sh +++ b/setup.sh @@ -252,7 +252,9 @@ function install() { echo "-------------------------" echo "Run the bot !" echo "-------------------------" - echo "You can now use the bot by executing 'source .env/bin/activate; freqtrade trade'." + echo "You can now use the bot by executing 'source .env/bin/activate; freqtrade '." + echo "You can see the list of available bot subcommands by executing 'source .env/bin/activate; freqtrade --help'." + echo "You verify that freqtrade is installed successfully by running 'source .env/bin/activate; freqtrade --version'." } function plot() { diff --git a/tests/conftest.py b/tests/conftest.py index 64d0cd5ee..d95475b8c 100644 --- a/tests/conftest.py +++ b/tests/conftest.py @@ -166,6 +166,52 @@ def patch_get_signal(freqtrade: FreqtradeBot, value=(True, False)) -> None: freqtrade.exchange.refresh_latest_ohlcv = lambda p: None +def create_mock_trades(fee): + """ + Create some fake trades ... + """ + # Simulate dry_run entries + trade = Trade( + pair='ETH/BTC', + stake_amount=0.001, + amount=123.0, + fee_open=fee.return_value, + fee_close=fee.return_value, + open_rate=0.123, + exchange='bittrex', + open_order_id='dry_run_buy_12345' + ) + Trade.session.add(trade) + + trade = Trade( + pair='ETC/BTC', + stake_amount=0.001, + amount=123.0, + fee_open=fee.return_value, + fee_close=fee.return_value, + open_rate=0.123, + close_rate=0.128, + close_profit=0.005, + exchange='bittrex', + is_open=False, + open_order_id='dry_run_sell_12345' + ) + Trade.session.add(trade) + + # Simulate prod entry + trade = Trade( + pair='ETC/BTC', + stake_amount=0.001, + amount=123.0, + fee_open=fee.return_value, + fee_close=fee.return_value, + open_rate=0.123, + exchange='bittrex', + open_order_id='prod_buy_12345' + ) + Trade.session.add(trade) + + @pytest.fixture(autouse=True) def patch_coingekko(mocker) -> None: """ @@ -693,6 +739,31 @@ def shitcoinmarkets(markets): "future": False, "active": True }, + 'ADAHALF/USDT': { + "percentage": True, + "tierBased": False, + "taker": 0.001, + "maker": 0.001, + "precision": { + "base": 8, + "quote": 8, + "amount": 2, + "price": 4 + }, + "limits": { + }, + "id": "ADAHALFUSDT", + "symbol": "ADAHALF/USDT", + "base": "ADAHALF", + "quote": "USDT", + "baseId": "ADAHALF", + "quoteId": "USDT", + "info": {}, + "type": "spot", + "spot": True, + "future": False, + "active": True + }, }) return shitmarkets @@ -712,6 +783,7 @@ def limit_buy_order(): 'datetime': arrow.utcnow().isoformat(), 'price': 0.00001099, 'amount': 90.99181073, + 'filled': 90.99181073, 'remaining': 0.0, 'status': 'closed' } @@ -727,6 +799,7 @@ def market_buy_order(): 'datetime': arrow.utcnow().isoformat(), 'price': 0.00004099, 'amount': 91.99181073, + 'filled': 91.99181073, 'remaining': 0.0, 'status': 'closed' } @@ -742,6 +815,7 @@ def market_sell_order(): 'datetime': arrow.utcnow().isoformat(), 'price': 0.00004173, 'amount': 91.99181073, + 'filled': 91.99181073, 'remaining': 0.0, 'status': 'closed' } @@ -757,6 +831,7 @@ def limit_buy_order_old(): 'datetime': str(arrow.utcnow().shift(minutes=-601).datetime), 'price': 0.00001099, 'amount': 90.99181073, + 'filled': 0.0, 'remaining': 90.99181073, 'status': 'open' } @@ -772,6 +847,7 @@ def limit_sell_order_old(): 'datetime': arrow.utcnow().shift(minutes=-601).isoformat(), 'price': 0.00001099, 'amount': 90.99181073, + 'filled': 0.0, 'remaining': 90.99181073, 'status': 'open' } @@ -787,6 +863,7 @@ def limit_buy_order_old_partial(): 'datetime': arrow.utcnow().shift(minutes=-601).isoformat(), 'price': 0.00001099, 'amount': 90.99181073, + 'filled': 23.0, 'remaining': 67.99181073, 'status': 'open' } @@ -810,6 +887,7 @@ def limit_sell_order(): 'datetime': arrow.utcnow().isoformat(), 'price': 0.00001173, 'amount': 90.99181073, + 'filled': 90.99181073, 'remaining': 0.0, 'status': 'closed' } @@ -1190,6 +1268,29 @@ def tickers(): "quoteVolume": 323652.075405, "info": {} }, + # Example of leveraged pair with incomplete info + "ADAHALF/USDT": { + "symbol": "ADAHALF/USDT", + "timestamp": 1580469388244, + "datetime": "2020-01-31T11:16:28.244Z", + "high": None, + "low": None, + "bid": 0.7305, + "bidVolume": None, + "ask": 0.7342, + "askVolume": None, + "vwap": None, + "open": None, + "close": None, + "last": None, + "previousClose": None, + "change": None, + "percentage": 2.628, + "average": None, + "baseVolume": 0.0, + "quoteVolume": 0.0, + "info": {} + }, }) diff --git a/tests/data/test_btanalysis.py b/tests/data/test_btanalysis.py index 463e5ae36..4da2acc5e 100644 --- a/tests/data/test_btanalysis.py +++ b/tests/data/test_btanalysis.py @@ -15,7 +15,7 @@ from freqtrade.data.btanalysis import (BT_DATA_COLUMNS, load_backtest_data, load_trades, load_trades_from_db) from freqtrade.data.history import load_data, load_pair_history -from tests.test_persistence import create_mock_trades +from tests.conftest import create_mock_trades def test_load_backtest_data(testdatadir): @@ -191,3 +191,28 @@ def test_calculate_max_drawdown(testdatadir): assert low == Timestamp('2018-01-30 04:45:00', tz='UTC') with pytest.raises(ValueError, match='Trade dataframe empty.'): drawdown, h, low = calculate_max_drawdown(DataFrame()) + + +def test_calculate_max_drawdown2(): + values = [0.011580, 0.010048, 0.011340, 0.012161, 0.010416, 0.010009, 0.020024, + -0.024662, -0.022350, 0.020496, -0.029859, -0.030511, 0.010041, 0.010872, + -0.025782, 0.010400, 0.012374, 0.012467, 0.114741, 0.010303, 0.010088, + -0.033961, 0.010680, 0.010886, -0.029274, 0.011178, 0.010693, 0.010711] + + dates = [Arrow(2020, 1, 1).shift(days=i) for i in range(len(values))] + df = DataFrame(zip(values, dates), columns=['profit', 'open_time']) + # sort by profit and reset index + df = df.sort_values('profit').reset_index(drop=True) + df1 = df.copy() + drawdown, h, low = calculate_max_drawdown(df, date_col='open_time', value_col='profit') + # Ensure df has not been altered. + assert df.equals(df1) + + assert isinstance(drawdown, float) + # High must be before low + assert h < low + assert drawdown == 0.091755 + + df = DataFrame(zip(values[:5], dates[:5]), columns=['profit', 'open_time']) + with pytest.raises(ValueError, match='No losing trade, therefore no drawdown.'): + calculate_max_drawdown(df, date_col='open_time', value_col='profit') diff --git a/tests/exchange/test_exchange.py b/tests/exchange/test_exchange.py index 8d8930f66..3c92612a0 100644 --- a/tests/exchange/test_exchange.py +++ b/tests/exchange/test_exchange.py @@ -253,6 +253,32 @@ def test_price_to_precision(default_conf, mocker, price, precision_mode, precisi assert pytest.approx(exchange.price_to_precision(pair, price)) == expected +@pytest.mark.parametrize("price,precision_mode,precision,expected", [ + (2.34559, 2, 4, 0.0001), + (2.34559, 2, 5, 0.00001), + (2.34559, 2, 3, 0.001), + (2.9999, 2, 3, 0.001), + (200.0511, 2, 3, 0.001), + # Tests for Tick_size + (2.34559, 4, 0.0001, 0.0001), + (2.34559, 4, 0.00001, 0.00001), + (2.34559, 4, 0.0025, 0.0025), + (2.9909, 4, 0.0025, 0.0025), + (234.43, 4, 0.5, 0.5), + (234.43, 4, 0.0025, 0.0025), + (234.43, 4, 0.00013, 0.00013), + +]) +def test_price_get_one_pip(default_conf, mocker, price, precision_mode, precision, expected): + markets = PropertyMock(return_value={'ETH/BTC': {'precision': {'price': precision}}}) + exchange = get_patched_exchange(mocker, default_conf, id="binance") + mocker.patch('freqtrade.exchange.Exchange.markets', markets) + mocker.patch('freqtrade.exchange.Exchange.precisionMode', + PropertyMock(return_value=precision_mode)) + pair = 'ETH/BTC' + assert pytest.approx(exchange.price_get_one_pip(pair, price)) == expected + + def test_set_sandbox(default_conf, mocker): """ Test working scenario @@ -1702,7 +1728,69 @@ def test_get_historic_trades_notsupported(default_conf, mocker, caplog, exchange def test_cancel_order_dry_run(default_conf, mocker, exchange_name): default_conf['dry_run'] = True exchange = get_patched_exchange(mocker, default_conf, id=exchange_name) - assert exchange.cancel_order(order_id='123', pair='TKN/BTC') is None + assert exchange.cancel_order(order_id='123', pair='TKN/BTC') == {} + + +@pytest.mark.parametrize("exchange_name", EXCHANGES) +@pytest.mark.parametrize("order,result", [ + ({'status': 'closed', 'filled': 10}, False), + ({'status': 'closed', 'filled': 0.0}, True), + ({'status': 'canceled', 'filled': 0.0}, True), + ({'status': 'canceled', 'filled': 10.0}, False), + ({'status': 'unknown', 'filled': 10.0}, False), + ({'result': 'testest123'}, False), + ]) +def test_check_order_canceled_empty(mocker, default_conf, exchange_name, order, result): + exchange = get_patched_exchange(mocker, default_conf, id=exchange_name) + assert exchange.check_order_canceled_empty(order) == result + + +@pytest.mark.parametrize("exchange_name", EXCHANGES) +@pytest.mark.parametrize("order,result", [ + ({'status': 'closed', 'amount': 10, 'fee': {}}, True), + ({'status': 'closed', 'amount': 0.0, 'fee': {}}, True), + ({'status': 'canceled', 'amount': 0.0, 'fee': {}}, True), + ({'status': 'canceled', 'amount': 10.0}, False), + ({'amount': 10.0, 'fee': {}}, False), + ({'result': 'testest123'}, False), + ('hello_world', False), +]) +def test_is_cancel_order_result_suitable(mocker, default_conf, exchange_name, order, result): + exchange = get_patched_exchange(mocker, default_conf, id=exchange_name) + assert exchange.is_cancel_order_result_suitable(order) == result + + +@pytest.mark.parametrize("exchange_name", EXCHANGES) +@pytest.mark.parametrize("corder,call_corder,call_forder", [ + ({'status': 'closed', 'amount': 10, 'fee': {}}, 1, 0), + ({'amount': 10, 'fee': {}}, 1, 1), +]) +def test_cancel_order_with_result(default_conf, mocker, exchange_name, corder, + call_corder, call_forder): + default_conf['dry_run'] = False + api_mock = MagicMock() + api_mock.cancel_order = MagicMock(return_value=corder) + api_mock.fetch_order = MagicMock(return_value={}) + exchange = get_patched_exchange(mocker, default_conf, api_mock, id=exchange_name) + res = exchange.cancel_order_with_result('1234', 'ETH/BTC', 1234) + assert isinstance(res, dict) + assert api_mock.cancel_order.call_count == call_corder + assert api_mock.fetch_order.call_count == call_forder + + +@pytest.mark.parametrize("exchange_name", EXCHANGES) +def test_cancel_order_with_result_error(default_conf, mocker, exchange_name, caplog): + default_conf['dry_run'] = False + api_mock = MagicMock() + api_mock.cancel_order = MagicMock(side_effect=ccxt.InvalidOrder("Did not find order")) + api_mock.fetch_order = MagicMock(side_effect=ccxt.InvalidOrder("Did not find order")) + exchange = get_patched_exchange(mocker, default_conf, api_mock, id=exchange_name) + + res = exchange.cancel_order_with_result('1234', 'ETH/BTC', 1541) + assert isinstance(res, dict) + assert log_has("Could not cancel order 1234.", caplog) + assert log_has("Could not fetch cancelled order 1234.", caplog) + assert res['amount'] == 1541 # Ensure that if not dry_run, we should call API diff --git a/tests/optimize/test_optimize_reports.py b/tests/optimize/test_optimize_reports.py index f19668459..e0782146a 100644 --- a/tests/optimize/test_optimize_reports.py +++ b/tests/optimize/test_optimize_reports.py @@ -160,10 +160,14 @@ def test_backtest_record(default_conf, fee, mocker): # reset test to test with strategy name names = [] records = [] - results['Strat'] = pd.DataFrame() + results['Strat'] = results['DefStrat'] + results['Strat2'] = results['DefStrat'] store_backtest_result(Path("backtest-result.json"), results) - # Assert file_dump_json was only called once - assert names == [Path('backtest-result-DefStrat.json')] + assert names == [ + Path('backtest-result-DefStrat.json'), + Path('backtest-result-Strat.json'), + Path('backtest-result-Strat2.json'), + ] records = records[0] # Ensure records are of correct type assert len(records) == 4 diff --git a/tests/pairlist/test_pairlist.py b/tests/pairlist/test_pairlist.py index 1ce1151b7..7dfe8bcca 100644 --- a/tests/pairlist/test_pairlist.py +++ b/tests/pairlist/test_pairlist.py @@ -46,6 +46,28 @@ def static_pl_conf(whitelist_conf): return whitelist_conf +def test_log_on_refresh(mocker, static_pl_conf, markets, tickers): + mocker.patch.multiple('freqtrade.exchange.Exchange', + markets=PropertyMock(return_value=markets), + exchange_has=MagicMock(return_value=True), + get_tickers=tickers + ) + freqtrade = get_patched_freqtradebot(mocker, static_pl_conf) + logmock = MagicMock() + # Assign starting whitelist + pl = freqtrade.pairlists._pairlists[0] + pl.log_on_refresh(logmock, 'Hello world') + assert logmock.call_count == 1 + pl.log_on_refresh(logmock, 'Hello world') + assert logmock.call_count == 1 + assert pl._log_cache.currsize == 1 + assert ('Hello world',) in pl._log_cache._Cache__data + + pl.log_on_refresh(logmock, 'Hello world2') + assert logmock.call_count == 2 + assert pl._log_cache.currsize == 2 + + def test_load_pairlist_noexist(mocker, markets, default_conf): bot = get_patched_freqtradebot(mocker, default_conf) mocker.patch('freqtrade.exchange.Exchange.markets', PropertyMock(return_value=markets)) @@ -141,7 +163,7 @@ def test_VolumePairList_refresh_empty(mocker, markets_empty, whitelist_conf): ([{"method": "VolumePairList", "number_assets": 5, "sort_key": "bidVolume"}], "BTC", ['HOT/BTC', 'FUEL/BTC', 'XRP/BTC', 'LTC/BTC', 'TKN/BTC']), ([{"method": "VolumePairList", "number_assets": 5, "sort_key": "quoteVolume"}], - "USDT", ['ETH/USDT', 'NANO/USDT']), + "USDT", ['ETH/USDT', 'NANO/USDT', 'ADAHALF/USDT']), # No pair for ETH ... ([{"method": "VolumePairList", "number_assets": 5, "sort_key": "quoteVolume"}], "ETH", []), @@ -155,6 +177,10 @@ def test_VolumePairList_refresh_empty(mocker, markets_empty, whitelist_conf): ([{"method": "VolumePairList", "number_assets": 5, "sort_key": "quoteVolume"}, {"method": "PriceFilter", "low_price_ratio": 0.03}], "BTC", ['ETH/BTC', 'TKN/BTC', 'LTC/BTC', 'XRP/BTC']), + # PriceFilter and VolumePairList + ([{"method": "VolumePairList", "number_assets": 5, "sort_key": "quoteVolume"}, + {"method": "PriceFilter", "low_price_ratio": 0.03}], + "USDT", ['ETH/USDT', 'NANO/USDT']), # Hot is removed by precision_filter, Fuel by low_price_filter. ([{"method": "VolumePairList", "number_assets": 6, "sort_key": "quoteVolume"}, {"method": "PrecisionFilter"}, @@ -199,7 +225,9 @@ def test_VolumePairList_whitelist_gen(mocker, whitelist_conf, shitcoinmarkets, t assert log_has_re(r'^Removed .* from whitelist, because stop price .* ' r'would be <= stop limit.*', caplog) if pairlist['method'] == 'PriceFilter': - assert log_has_re(r'^Removed .* from whitelist, because 1 unit is .*%$', caplog) + assert (log_has_re(r'^Removed .* from whitelist, because 1 unit is .*%$', caplog) or + log_has_re(r"^Removed .* from whitelist, because ticker\['last'\] is empty.*", + caplog)) def test_gen_pair_whitelist_not_supported(mocker, default_conf, tickers) -> None: diff --git a/tests/rpc/test_rpc.py b/tests/rpc/test_rpc.py index 47ffb771b..d2af4bd87 100644 --- a/tests/rpc/test_rpc.py +++ b/tests/rpc/test_rpc.py @@ -13,7 +13,7 @@ from freqtrade.persistence import Trade from freqtrade.rpc import RPC, RPCException from freqtrade.rpc.fiat_convert import CryptoToFiatConverter from freqtrade.state import State -from tests.conftest import get_patched_freqtradebot, patch_get_signal +from tests.conftest import get_patched_freqtradebot, patch_get_signal, create_mock_trades # Functions for recurrent object patching @@ -49,6 +49,18 @@ def test_rpc_trade_status(default_conf, ticker, fee, mocker) -> None: 'base_currency': 'BTC', 'open_date': ANY, 'open_date_hum': ANY, + 'is_open': ANY, + 'fee_open': ANY, + 'fee_close': ANY, + 'open_rate_requested': ANY, + 'open_trade_price': ANY, + 'close_rate_requested': ANY, + 'sell_reason': ANY, + 'min_rate': ANY, + 'max_rate': ANY, + 'strategy': ANY, + 'ticker_interval': ANY, + 'open_order_id': ANY, 'close_date': None, 'close_date_hum': None, 'open_rate': 1.098e-05, @@ -76,6 +88,18 @@ def test_rpc_trade_status(default_conf, ticker, fee, mocker) -> None: 'base_currency': 'BTC', 'open_date': ANY, 'open_date_hum': ANY, + 'is_open': ANY, + 'fee_open': ANY, + 'fee_close': ANY, + 'open_rate_requested': ANY, + 'open_trade_price': ANY, + 'close_rate_requested': ANY, + 'sell_reason': ANY, + 'min_rate': ANY, + 'max_rate': ANY, + 'strategy': ANY, + 'ticker_interval': ANY, + 'open_order_id': ANY, 'close_date': None, 'close_date_hum': None, 'open_rate': 1.098e-05, @@ -187,6 +211,32 @@ def test_rpc_daily_profit(default_conf, update, ticker, fee, rpc._rpc_daily_profit(0, stake_currency, fiat_display_currency) +def test_rpc_trade_history(mocker, default_conf, markets, fee): + mocker.patch('freqtrade.rpc.telegram.Telegram', MagicMock()) + mocker.patch.multiple( + 'freqtrade.exchange.Exchange', + markets=PropertyMock(return_value=markets) + ) + + freqtradebot = get_patched_freqtradebot(mocker, default_conf) + create_mock_trades(fee) + rpc = RPC(freqtradebot) + rpc._fiat_converter = CryptoToFiatConverter() + trades = rpc._rpc_trade_history(2) + assert len(trades['trades']) == 2 + assert trades['trades_count'] == 2 + assert isinstance(trades['trades'][0], dict) + assert isinstance(trades['trades'][1], dict) + + trades = rpc._rpc_trade_history(0) + assert len(trades['trades']) == 3 + assert trades['trades_count'] == 3 + # The first trade is for ETH ... sorting is descending + assert trades['trades'][-1]['pair'] == 'ETH/BTC' + assert trades['trades'][0]['pair'] == 'ETC/BTC' + assert trades['trades'][1]['pair'] == 'ETC/BTC' + + def test_rpc_trade_statistics(default_conf, ticker, ticker_sell_up, fee, limit_buy_order, limit_sell_order, mocker) -> None: mocker.patch.multiple( diff --git a/tests/rpc/test_rpc_apiserver.py b/tests/rpc/test_rpc_apiserver.py index e0abd886d..6548790cb 100644 --- a/tests/rpc/test_rpc_apiserver.py +++ b/tests/rpc/test_rpc_apiserver.py @@ -13,7 +13,7 @@ from freqtrade.__init__ import __version__ from freqtrade.persistence import Trade from freqtrade.rpc.api_server import BASE_URI, ApiServer from freqtrade.state import State -from tests.conftest import get_patched_freqtradebot, log_has, patch_get_signal +from tests.conftest import get_patched_freqtradebot, log_has, patch_get_signal, create_mock_trades _TEST_USER = "FreqTrader" _TEST_PASS = "SuperSecurePassword1!" @@ -302,6 +302,30 @@ def test_api_daily(botclient, mocker, ticker, fee, markets): assert rc.json[0][0] == str(datetime.utcnow().date()) +def test_api_trades(botclient, mocker, ticker, fee, markets): + ftbot, client = botclient + patch_get_signal(ftbot, (True, False)) + mocker.patch.multiple( + 'freqtrade.exchange.Exchange', + markets=PropertyMock(return_value=markets) + ) + rc = client_get(client, f"{BASE_URI}/trades") + assert_response(rc) + assert len(rc.json) == 2 + assert rc.json['trades_count'] == 0 + + create_mock_trades(fee) + + rc = client_get(client, f"{BASE_URI}/trades") + assert_response(rc) + assert len(rc.json['trades']) == 3 + assert rc.json['trades_count'] == 3 + rc = client_get(client, f"{BASE_URI}/trades?limit=2") + assert_response(rc) + assert len(rc.json['trades']) == 2 + assert rc.json['trades_count'] == 2 + + def test_api_edge_disabled(botclient, mocker, ticker, fee, markets): ftbot, client = botclient patch_get_signal(ftbot, (True, False)) @@ -444,7 +468,21 @@ def test_api_status(botclient, mocker, ticker, fee, markets): 'stake_amount': 0.001, 'stop_loss': 0.0, 'stop_loss_pct': None, - 'trade_id': 1}] + 'trade_id': 1, + 'close_rate_requested': None, + 'current_rate': 1.099e-05, + 'fee_close': 0.0025, + 'fee_open': 0.0025, + 'open_date': ANY, + 'is_open': True, + 'max_rate': 0.0, + 'min_rate': None, + 'open_order_id': ANY, + 'open_rate_requested': 1.098e-05, + 'open_trade_price': 0.0010025, + 'sell_reason': None, + 'strategy': 'DefaultStrategy', + 'ticker_interval': 5}] def test_api_version(botclient): @@ -533,7 +571,21 @@ def test_api_forcebuy(botclient, mocker, fee): 'stake_amount': 1, 'stop_loss': None, 'stop_loss_pct': None, - 'trade_id': None} + 'trade_id': None, + 'close_profit': None, + 'close_rate_requested': None, + 'fee_close': 0.0025, + 'fee_open': 0.0025, + 'is_open': False, + 'max_rate': None, + 'min_rate': None, + 'open_order_id': '123456', + 'open_rate_requested': None, + 'open_trade_price': 0.2460546025, + 'sell_reason': None, + 'strategy': None, + 'ticker_interval': None + } def test_api_forcesell(botclient, mocker, ticker, fee, markets): diff --git a/tests/rpc/test_rpc_telegram.py b/tests/rpc/test_rpc_telegram.py index d769016c4..bbc961763 100644 --- a/tests/rpc/test_rpc_telegram.py +++ b/tests/rpc/test_rpc_telegram.py @@ -1316,18 +1316,20 @@ def test_send_msg_sell_cancel_notification(default_conf, mocker) -> None: 'type': RPCMessageType.SELL_CANCEL_NOTIFICATION, 'exchange': 'Binance', 'pair': 'KEY/ETH', + 'reason': 'Cancelled on exchange' }) assert msg_mock.call_args[0][0] \ - == ('*Binance:* Cancelling Open Sell Order for KEY/ETH') + == ('*Binance:* Cancelling Open Sell Order for KEY/ETH. Reason: Cancelled on exchange') msg_mock.reset_mock() telegram.send_msg({ 'type': RPCMessageType.SELL_CANCEL_NOTIFICATION, 'exchange': 'Binance', 'pair': 'KEY/ETH', + 'reason': 'timeout' }) assert msg_mock.call_args[0][0] \ - == ('*Binance:* Cancelling Open Sell Order for KEY/ETH') + == ('*Binance:* Cancelling Open Sell Order for KEY/ETH. Reason: timeout') # Reset singleton function to avoid random breaks telegram._fiat_converter.convert_amount = old_convamount diff --git a/tests/strategy/test_interface.py b/tests/strategy/test_interface.py index 949dda4a0..dd6b11a06 100644 --- a/tests/strategy/test_interface.py +++ b/tests/strategy/test_interface.py @@ -4,46 +4,53 @@ import logging from unittest.mock import MagicMock import arrow +import pytest from pandas import DataFrame from freqtrade.configuration import TimeRange from freqtrade.data.history import load_data +from freqtrade.exceptions import StrategyError from freqtrade.persistence import Trade from freqtrade.resolvers import StrategyResolver +from freqtrade.strategy.strategy_wrapper import strategy_safe_wrapper +from tests.conftest import get_patched_exchange, log_has, log_has_re + from .strats.default_strategy import DefaultStrategy -from tests.conftest import get_patched_exchange, log_has # Avoid to reinit the same object again and again _STRATEGY = DefaultStrategy(config={}) -def test_returns_latest_buy_signal(mocker, default_conf, ohlcv_history): - mocker.patch.object( - _STRATEGY, '_analyze_ticker_internal', - return_value=DataFrame([{'buy': 1, 'sell': 0, 'date': arrow.utcnow()}]) - ) - assert _STRATEGY.get_signal('ETH/BTC', '5m', ohlcv_history) == (True, False) +def test_returns_latest_signal(mocker, default_conf, ohlcv_history): + ohlcv_history.loc[1, 'date'] = arrow.utcnow() + # Take a copy to correctly modify the call + mocked_history = ohlcv_history.copy() + mocked_history['sell'] = 0 + mocked_history['buy'] = 0 + mocked_history.loc[1, 'sell'] = 1 mocker.patch.object( _STRATEGY, '_analyze_ticker_internal', - return_value=DataFrame([{'buy': 0, 'sell': 1, 'date': arrow.utcnow()}]) - ) - assert _STRATEGY.get_signal('ETH/BTC', '5m', ohlcv_history) == (False, True) - - -def test_returns_latest_sell_signal(mocker, default_conf, ohlcv_history): - mocker.patch.object( - _STRATEGY, '_analyze_ticker_internal', - return_value=DataFrame([{'sell': 1, 'buy': 0, 'date': arrow.utcnow()}]) + return_value=mocked_history ) assert _STRATEGY.get_signal('ETH/BTC', '5m', ohlcv_history) == (False, True) + mocked_history.loc[1, 'sell'] = 0 + mocked_history.loc[1, 'buy'] = 1 mocker.patch.object( _STRATEGY, '_analyze_ticker_internal', - return_value=DataFrame([{'sell': 0, 'buy': 1, 'date': arrow.utcnow()}]) + return_value=mocked_history ) assert _STRATEGY.get_signal('ETH/BTC', '5m', ohlcv_history) == (True, False) + mocked_history.loc[1, 'sell'] = 0 + mocked_history.loc[1, 'buy'] = 0 + + mocker.patch.object( + _STRATEGY, '_analyze_ticker_internal', + return_value=mocked_history + ) + assert _STRATEGY.get_signal('ETH/BTC', '5m', ohlcv_history) == (False, False) def test_get_signal_empty(default_conf, mocker, caplog): @@ -65,7 +72,7 @@ def test_get_signal_exception_valueerror(default_conf, mocker, caplog, ohlcv_his ) assert (False, False) == _STRATEGY.get_signal('foo', default_conf['ticker_interval'], ohlcv_history) - assert log_has('Unable to analyze candle (OHLCV) data for pair foo: xyz', caplog) + assert log_has_re(r'Strategy caused the following exception: xyz.*', caplog) def test_get_signal_empty_dataframe(default_conf, mocker, caplog, ohlcv_history): @@ -74,26 +81,74 @@ def test_get_signal_empty_dataframe(default_conf, mocker, caplog, ohlcv_history) _STRATEGY, '_analyze_ticker_internal', return_value=DataFrame([]) ) + mocker.patch.object(_STRATEGY, 'assert_df') + assert (False, False) == _STRATEGY.get_signal('xyz', default_conf['ticker_interval'], ohlcv_history) assert log_has('Empty dataframe for pair xyz', caplog) def test_get_signal_old_dataframe(default_conf, mocker, caplog, ohlcv_history): - caplog.set_level(logging.INFO) # default_conf defines a 5m interval. we check interval * 2 + 5m # this is necessary as the last candle is removed (partial candles) by default - oldtime = arrow.utcnow().shift(minutes=-16) - ticks = DataFrame([{'buy': 1, 'date': oldtime}]) + ohlcv_history.loc[1, 'date'] = arrow.utcnow().shift(minutes=-16) + # Take a copy to correctly modify the call + mocked_history = ohlcv_history.copy() + mocked_history['sell'] = 0 + mocked_history['buy'] = 0 + mocked_history.loc[1, 'buy'] = 1 + + caplog.set_level(logging.INFO) mocker.patch.object( _STRATEGY, '_analyze_ticker_internal', - return_value=DataFrame(ticks) + return_value=mocked_history ) + mocker.patch.object(_STRATEGY, 'assert_df') assert (False, False) == _STRATEGY.get_signal('xyz', default_conf['ticker_interval'], ohlcv_history) assert log_has('Outdated history for pair xyz. Last tick is 16 minutes old', caplog) +def test_assert_df_raise(default_conf, mocker, caplog, ohlcv_history): + # default_conf defines a 5m interval. we check interval * 2 + 5m + # this is necessary as the last candle is removed (partial candles) by default + ohlcv_history.loc[1, 'date'] = arrow.utcnow().shift(minutes=-16) + # Take a copy to correctly modify the call + mocked_history = ohlcv_history.copy() + mocked_history['sell'] = 0 + mocked_history['buy'] = 0 + mocked_history.loc[1, 'buy'] = 1 + + caplog.set_level(logging.INFO) + mocker.patch.object( + _STRATEGY, 'assert_df', + side_effect=StrategyError('Dataframe returned...') + ) + assert (False, False) == _STRATEGY.get_signal('xyz', default_conf['ticker_interval'], + ohlcv_history) + assert log_has('Unable to analyze candle (OHLCV) data for pair xyz: Dataframe returned...', + caplog) + + +def test_assert_df(default_conf, mocker, ohlcv_history): + # Ensure it's running when passed correctly + _STRATEGY.assert_df(ohlcv_history, len(ohlcv_history), + ohlcv_history.loc[1, 'close'], ohlcv_history.loc[1, 'date']) + + with pytest.raises(StrategyError, match=r"Dataframe returned from strategy.*length\."): + _STRATEGY.assert_df(ohlcv_history, len(ohlcv_history) + 1, + ohlcv_history.loc[1, 'close'], ohlcv_history.loc[1, 'date']) + + with pytest.raises(StrategyError, + match=r"Dataframe returned from strategy.*last close price\."): + _STRATEGY.assert_df(ohlcv_history, len(ohlcv_history), + ohlcv_history.loc[1, 'close'] + 0.01, ohlcv_history.loc[1, 'date']) + with pytest.raises(StrategyError, + match=r"Dataframe returned from strategy.*last date\."): + _STRATEGY.assert_df(ohlcv_history, len(ohlcv_history), + ohlcv_history.loc[1, 'close'], ohlcv_history.loc[0, 'date']) + + def test_get_signal_handles_exceptions(mocker, default_conf): exchange = get_patched_exchange(mocker, default_conf) mocker.patch.object( @@ -114,6 +169,19 @@ def test_ohlcvdata_to_dataframe(default_conf, testdatadir) -> None: assert len(processed['UNITTEST/BTC']) == 102 # partial candle was removed +def test_ohlcvdata_to_dataframe_copy(mocker, default_conf, testdatadir) -> None: + default_conf.update({'strategy': 'DefaultStrategy'}) + strategy = StrategyResolver.load_strategy(default_conf) + aimock = mocker.patch('freqtrade.strategy.interface.IStrategy.advise_indicators') + timerange = TimeRange.parse_timerange('1510694220-1510700340') + data = load_data(testdatadir, '1m', ['UNITTEST/BTC'], timerange=timerange, + fill_up_missing=True) + strategy.ohlcvdata_to_dataframe(data) + assert aimock.call_count == 1 + # Ensure that a copy of the dataframe is passed to advice_indicators + assert aimock.call_args_list[0][0][0] is not data + + def test_min_roi_reached(default_conf, fee) -> None: # Use list to confirm sequence does not matter @@ -322,3 +390,38 @@ def test_is_pair_locked(default_conf): pair = 'ETH/BTC' strategy.unlock_pair(pair) assert not strategy.is_pair_locked(pair) + + +@pytest.mark.parametrize('error', [ + ValueError, KeyError, Exception, +]) +def test_strategy_safe_wrapper_error(caplog, error): + def failing_method(): + raise error('This is an error.') + + def working_method(argumentpassedin): + return argumentpassedin + + with pytest.raises(StrategyError, match=r'This is an error.'): + strategy_safe_wrapper(failing_method, message='DeadBeef')() + + assert log_has_re(r'DeadBeef.*', caplog) + ret = strategy_safe_wrapper(failing_method, message='DeadBeef', default_retval=True)() + + assert isinstance(ret, bool) + assert ret + + +@pytest.mark.parametrize('value', [ + 1, 22, 55, True, False, {'a': 1, 'b': '112'}, + [1, 2, 3, 4], (4, 2, 3, 6) +]) +def test_strategy_safe_wrapper(value): + + def working_method(argumentpassedin): + return argumentpassedin + + ret = strategy_safe_wrapper(working_method, message='DeadBeef')(value) + + assert type(ret) == type(value) + assert ret == value diff --git a/tests/test_directory_operations.py b/tests/test_directory_operations.py index 889338a64..71c91549f 100644 --- a/tests/test_directory_operations.py +++ b/tests/test_directory_operations.py @@ -25,7 +25,7 @@ def test_create_userdata_dir(mocker, default_conf, caplog) -> None: md = mocker.patch.object(Path, 'mkdir', MagicMock()) x = create_userdata_dir('/tmp/bar', create_dir=True) - assert md.call_count == 8 + assert md.call_count == 9 assert md.call_args[1]['parents'] is False assert log_has(f'Created user-data directory: {Path("/tmp/bar")}', caplog) assert isinstance(x, Path) diff --git a/tests/test_freqtradebot.py b/tests/test_freqtradebot.py index e37270bd3..6f2ce9f3c 100644 --- a/tests/test_freqtradebot.py +++ b/tests/test_freqtradebot.py @@ -1592,13 +1592,13 @@ def test_exit_positions_exception(mocker, default_conf, limit_buy_order, caplog) mocker.patch('freqtrade.exchange.Exchange.get_order', return_value=limit_buy_order) trade = MagicMock() - trade.open_order_id = '123' + trade.open_order_id = None trade.open_fee = 0.001 trades = [trade] # Test raise of DependencyException exception mocker.patch( - 'freqtrade.freqtradebot.FreqtradeBot.update_trade_state', + 'freqtrade.freqtradebot.FreqtradeBot.handle_trade', side_effect=DependencyException() ) n = freqtrade.exit_positions(trades) @@ -1939,8 +1939,10 @@ def test_close_trade(default_conf, ticker, limit_buy_order, limit_sell_order, freqtrade.handle_trade(trade) -def test_check_handle_timedout_buy(default_conf, ticker, limit_buy_order_old, open_trade, - fee, mocker) -> None: +def test_check_handle_timedout_buy_usercustom(default_conf, ticker, limit_buy_order_old, open_trade, + fee, mocker) -> None: + default_conf["unfilledtimeout"] = {"buy": 1400, "sell": 30} + rpc_mock = patch_RPCManager(mocker) cancel_order_mock = MagicMock(return_value=limit_buy_order_old) patch_exchange(mocker) @@ -1955,6 +1957,52 @@ def test_check_handle_timedout_buy(default_conf, ticker, limit_buy_order_old, op Trade.session.add(open_trade) + # Return false - trade remains open + freqtrade.strategy.check_buy_timeout = MagicMock(return_value=False) + freqtrade.check_handle_timedout() + assert cancel_order_mock.call_count == 0 + trades = Trade.query.filter(Trade.open_order_id.is_(open_trade.open_order_id)).all() + nb_trades = len(trades) + assert nb_trades == 1 + assert freqtrade.strategy.check_buy_timeout.call_count == 1 + + # Raise Keyerror ... (no impact on trade) + freqtrade.strategy.check_buy_timeout = MagicMock(side_effect=KeyError) + freqtrade.check_handle_timedout() + assert cancel_order_mock.call_count == 0 + trades = Trade.query.filter(Trade.open_order_id.is_(open_trade.open_order_id)).all() + nb_trades = len(trades) + assert nb_trades == 1 + assert freqtrade.strategy.check_buy_timeout.call_count == 1 + + freqtrade.strategy.check_buy_timeout = MagicMock(return_value=True) + # Trade should be closed since the function returns true + freqtrade.check_handle_timedout() + assert cancel_order_mock.call_count == 1 + assert rpc_mock.call_count == 1 + trades = Trade.query.filter(Trade.open_order_id.is_(open_trade.open_order_id)).all() + nb_trades = len(trades) + assert nb_trades == 0 + assert freqtrade.strategy.check_buy_timeout.call_count == 1 + + +def test_check_handle_timedout_buy(default_conf, ticker, limit_buy_order_old, open_trade, + fee, mocker) -> None: + rpc_mock = patch_RPCManager(mocker) + cancel_order_mock = MagicMock(return_value=limit_buy_order_old) + patch_exchange(mocker) + mocker.patch.multiple( + 'freqtrade.exchange.Exchange', + fetch_ticker=ticker, + get_order=MagicMock(return_value=limit_buy_order_old), + cancel_order_with_result=cancel_order_mock, + get_fee=fee + ) + freqtrade = FreqtradeBot(default_conf) + + Trade.session.add(open_trade) + + freqtrade.strategy.check_buy_timeout = MagicMock(return_value=False) # check it does cancel buy orders over the time limit freqtrade.check_handle_timedout() assert cancel_order_mock.call_count == 1 @@ -1962,6 +2010,8 @@ def test_check_handle_timedout_buy(default_conf, ticker, limit_buy_order_old, op trades = Trade.query.filter(Trade.open_order_id.is_(open_trade.open_order_id)).all() nb_trades = len(trades) assert nb_trades == 0 + # Custom user buy-timeout is never called + assert freqtrade.strategy.check_buy_timeout.call_count == 0 def test_check_handle_cancelled_buy(default_conf, ticker, limit_buy_order_old, open_trade, @@ -1970,7 +2020,7 @@ def test_check_handle_cancelled_buy(default_conf, ticker, limit_buy_order_old, o rpc_mock = patch_RPCManager(mocker) cancel_order_mock = MagicMock() patch_exchange(mocker) - limit_buy_order_old.update({"status": "canceled"}) + limit_buy_order_old.update({"status": "canceled", 'filled': 0.0}) mocker.patch.multiple( 'freqtrade.exchange.Exchange', fetch_ticker=ticker, @@ -2018,6 +2068,51 @@ def test_check_handle_timedout_buy_exception(default_conf, ticker, limit_buy_ord assert nb_trades == 1 +def test_check_handle_timedout_sell_usercustom(default_conf, ticker, limit_sell_order_old, mocker, + open_trade) -> None: + default_conf["unfilledtimeout"] = {"buy": 1440, "sell": 1440} + rpc_mock = patch_RPCManager(mocker) + cancel_order_mock = MagicMock() + patch_exchange(mocker) + mocker.patch.multiple( + 'freqtrade.exchange.Exchange', + fetch_ticker=ticker, + get_order=MagicMock(return_value=limit_sell_order_old), + cancel_order=cancel_order_mock + ) + freqtrade = FreqtradeBot(default_conf) + + open_trade.open_date = arrow.utcnow().shift(hours=-5).datetime + open_trade.close_date = arrow.utcnow().shift(minutes=-601).datetime + open_trade.is_open = False + + Trade.session.add(open_trade) + + freqtrade.strategy.check_sell_timeout = MagicMock(return_value=False) + # Return false - No impact + freqtrade.check_handle_timedout() + assert cancel_order_mock.call_count == 0 + assert rpc_mock.call_count == 0 + assert open_trade.is_open is False + assert freqtrade.strategy.check_sell_timeout.call_count == 1 + + freqtrade.strategy.check_sell_timeout = MagicMock(side_effect=KeyError) + # Return Error - No impact + freqtrade.check_handle_timedout() + assert cancel_order_mock.call_count == 0 + assert rpc_mock.call_count == 0 + assert open_trade.is_open is False + assert freqtrade.strategy.check_sell_timeout.call_count == 1 + + # Return True - sells! + freqtrade.strategy.check_sell_timeout = MagicMock(return_value=True) + freqtrade.check_handle_timedout() + assert cancel_order_mock.call_count == 1 + assert rpc_mock.call_count == 1 + assert open_trade.is_open is True + assert freqtrade.strategy.check_sell_timeout.call_count == 1 + + def test_check_handle_timedout_sell(default_conf, ticker, limit_sell_order_old, mocker, open_trade) -> None: rpc_mock = patch_RPCManager(mocker) @@ -2037,11 +2132,14 @@ def test_check_handle_timedout_sell(default_conf, ticker, limit_sell_order_old, Trade.session.add(open_trade) + freqtrade.strategy.check_sell_timeout = MagicMock(return_value=False) # check it does cancel sell orders over the time limit freqtrade.check_handle_timedout() assert cancel_order_mock.call_count == 1 assert rpc_mock.call_count == 1 assert open_trade.is_open is True + # Custom user sell-timeout is never called + assert freqtrade.strategy.check_sell_timeout.call_count == 0 def test_check_handle_cancelled_sell(default_conf, ticker, limit_sell_order_old, open_trade, @@ -2049,13 +2147,13 @@ def test_check_handle_cancelled_sell(default_conf, ticker, limit_sell_order_old, """ Handle sell order cancelled on exchange""" rpc_mock = patch_RPCManager(mocker) cancel_order_mock = MagicMock() - limit_sell_order_old.update({"status": "canceled"}) + limit_sell_order_old.update({"status": "canceled", 'filled': 0.0}) patch_exchange(mocker) mocker.patch.multiple( 'freqtrade.exchange.Exchange', fetch_ticker=ticker, get_order=MagicMock(return_value=limit_sell_order_old), - cancel_order=cancel_order_mock + cancel_order_with_result=cancel_order_mock ) freqtrade = FreqtradeBot(default_conf) @@ -2082,7 +2180,7 @@ def test_check_handle_timedout_partial(default_conf, ticker, limit_buy_order_old 'freqtrade.exchange.Exchange', fetch_ticker=ticker, get_order=MagicMock(return_value=limit_buy_order_old_partial), - cancel_order=cancel_order_mock + cancel_order_with_result=cancel_order_mock ) freqtrade = FreqtradeBot(default_conf) @@ -2109,7 +2207,7 @@ def test_check_handle_timedout_partial_fee(default_conf, ticker, open_trade, cap 'freqtrade.exchange.Exchange', fetch_ticker=ticker, get_order=MagicMock(return_value=limit_buy_order_old_partial), - cancel_order=cancel_order_mock, + cancel_order_with_result=cancel_order_mock, get_trades_for_order=MagicMock(return_value=trades_for_order), ) freqtrade = FreqtradeBot(default_conf) @@ -2129,7 +2227,7 @@ def test_check_handle_timedout_partial_fee(default_conf, ticker, open_trade, cap assert rpc_mock.call_count == 2 trades = Trade.query.filter(Trade.open_order_id.is_(open_trade.open_order_id)).all() assert len(trades) == 1 - # Verify that tradehas been updated + # Verify that trade has been updated assert trades[0].amount == (limit_buy_order_old_partial['amount'] - limit_buy_order_old_partial['remaining']) - 0.0001 assert trades[0].open_order_id is None @@ -2146,7 +2244,7 @@ def test_check_handle_timedout_partial_except(default_conf, ticker, open_trade, 'freqtrade.exchange.Exchange', fetch_ticker=ticker, get_order=MagicMock(return_value=limit_buy_order_old_partial), - cancel_order=cancel_order_mock, + cancel_order_with_result=cancel_order_mock, get_trades_for_order=MagicMock(return_value=trades_for_order), ) mocker.patch('freqtrade.freqtradebot.FreqtradeBot.get_real_amount', @@ -2168,7 +2266,7 @@ def test_check_handle_timedout_partial_except(default_conf, ticker, open_trade, assert rpc_mock.call_count == 2 trades = Trade.query.filter(Trade.open_order_id.is_(open_trade.open_order_id)).all() assert len(trades) == 1 - # Verify that tradehas been updated + # Verify that trade has been updated assert trades[0].amount == (limit_buy_order_old_partial['amount'] - limit_buy_order_old_partial['remaining']) @@ -2204,14 +2302,11 @@ def test_check_handle_timedout_exception(default_conf, ticker, open_trade, mocke caplog) -def test_handle_timedout_limit_buy(mocker, default_conf, limit_buy_order) -> None: +def test_handle_timedout_limit_buy(mocker, caplog, default_conf, limit_buy_order) -> None: patch_RPCManager(mocker) patch_exchange(mocker) cancel_order_mock = MagicMock(return_value=limit_buy_order) - mocker.patch.multiple( - 'freqtrade.exchange.Exchange', - cancel_order=cancel_order_mock - ) + mocker.patch('freqtrade.exchange.Exchange.cancel_order_with_result', cancel_order_mock) freqtrade = FreqtradeBot(default_conf) @@ -2227,9 +2322,13 @@ def test_handle_timedout_limit_buy(mocker, default_conf, limit_buy_order) -> Non assert not freqtrade.handle_timedout_limit_buy(trade, limit_buy_order) assert cancel_order_mock.call_count == 1 + mocker.patch('freqtrade.exchange.Exchange.cancel_order', side_effect=InvalidOrderException) + assert not freqtrade.handle_timedout_limit_buy(trade, limit_buy_order) + @pytest.mark.parametrize('cancelorder', [ {}, + {'remaining': None}, 'String Return value', 123 ]) @@ -2276,7 +2375,8 @@ def test_handle_timedout_limit_sell(mocker, default_conf) -> None: assert freqtrade.handle_timedout_limit_sell(trade, order) assert cancel_order_mock.call_count == 1 order['amount'] = 2 - assert not freqtrade.handle_timedout_limit_sell(trade, order) + assert (freqtrade.handle_timedout_limit_sell(trade, order) + == 'partially filled - keeping order open') # Assert cancel_order was not called (callcount remains unchanged) assert cancel_order_mock.call_count == 1 @@ -2499,6 +2599,7 @@ def test_execute_sell_with_stoploss_on_exchange(default_conf, ticker, fee, ticke assert trade trades = [trade] + freqtrade.check_handle_timedout() freqtrade.exit_positions(trades) # Increase the price and sell it @@ -2544,8 +2645,11 @@ def test_may_execute_sell_after_stoploss_on_exchange_hit(default_conf, ticker, f # Create some test data freqtrade.enter_positions() + freqtrade.check_handle_timedout() trade = Trade.query.first() trades = [trade] + assert trade.stoploss_order_id is None + freqtrade.exit_positions(trades) assert trade assert trade.stoploss_order_id == '123' diff --git a/tests/test_misc.py b/tests/test_misc.py index c1e23926b..9fd6164d5 100644 --- a/tests/test_misc.py +++ b/tests/test_misc.py @@ -9,7 +9,9 @@ import pytest from freqtrade.data.converter import ohlcv_to_dataframe from freqtrade.misc import (datesarray_to_datetimearray, file_dump_json, file_load_json, format_ms_time, pair_to_filename, - plural, shorten_date) + plural, render_template, + render_template_with_fallback, safe_value_fallback, + shorten_date) def test_shorten_date() -> None: @@ -93,6 +95,27 @@ def test_format_ms_time() -> None: assert format_ms_time(date_in_epoch_ms) == res.astimezone(None).strftime('%Y-%m-%dT%H:%M:%S') +def test_safe_value_fallback(): + dict1 = {'keya': None, 'keyb': 2, 'keyc': 5, 'keyd': None} + dict2 = {'keya': 20, 'keyb': None, 'keyc': 6, 'keyd': None} + assert safe_value_fallback(dict1, dict2, 'keya', 'keya') == 20 + assert safe_value_fallback(dict2, dict1, 'keya', 'keya') == 20 + + assert safe_value_fallback(dict1, dict2, 'keyb', 'keyb') == 2 + assert safe_value_fallback(dict2, dict1, 'keyb', 'keyb') == 2 + + assert safe_value_fallback(dict1, dict2, 'keyc', 'keyc') == 5 + assert safe_value_fallback(dict2, dict1, 'keyc', 'keyc') == 6 + + assert safe_value_fallback(dict1, dict2, 'keyd', 'keyd') is None + assert safe_value_fallback(dict2, dict1, 'keyd', 'keyd') is None + assert safe_value_fallback(dict2, dict1, 'keyd', 'keyd', 1234) == 1234 + + assert safe_value_fallback(dict1, dict2, 'keyNo', 'keyNo') is None + assert safe_value_fallback(dict2, dict1, 'keyNo', 'keyNo') is None + assert safe_value_fallback(dict2, dict1, 'keyNo', 'keyNo', 1234) == 1234 + + def test_plural() -> None: assert plural(0, "page") == "pages" assert plural(0.0, "page") == "pages" @@ -123,3 +146,17 @@ def test_plural() -> None: assert plural(1.5, "ox", "oxen") == "oxen" assert plural(-0.5, "ox", "oxen") == "oxen" assert plural(-1.5, "ox", "oxen") == "oxen" + + +def test_render_template_fallback(mocker): + from jinja2.exceptions import TemplateNotFound + with pytest.raises(TemplateNotFound): + val = render_template( + templatefile='subtemplates/indicators_does-not-exist.j2',) + + val = render_template_with_fallback( + templatefile='subtemplates/indicators_does-not-exist.j2', + templatefallbackfile='subtemplates/indicators_minimal.j2', + ) + assert isinstance(val, str) + assert 'if self.dp' in val diff --git a/tests/test_persistence.py b/tests/test_persistence.py index 991922cba..ceac24356 100644 --- a/tests/test_persistence.py +++ b/tests/test_persistence.py @@ -9,53 +9,7 @@ from sqlalchemy import create_engine from freqtrade import constants from freqtrade.exceptions import OperationalException from freqtrade.persistence import Trade, clean_dry_run_db, init -from tests.conftest import log_has - - -def create_mock_trades(fee): - """ - Create some fake trades ... - """ - # Simulate dry_run entries - trade = Trade( - pair='ETH/BTC', - stake_amount=0.001, - amount=123.0, - fee_open=fee.return_value, - fee_close=fee.return_value, - open_rate=0.123, - exchange='bittrex', - open_order_id='dry_run_buy_12345' - ) - Trade.session.add(trade) - - trade = Trade( - pair='ETC/BTC', - stake_amount=0.001, - amount=123.0, - fee_open=fee.return_value, - fee_close=fee.return_value, - open_rate=0.123, - close_rate=0.128, - close_profit=0.005, - exchange='bittrex', - is_open=False, - open_order_id='dry_run_sell_12345' - ) - Trade.session.add(trade) - - # Simulate prod entry - trade = Trade( - pair='ETC/BTC', - stake_amount=0.001, - amount=123.0, - fee_open=fee.return_value, - fee_close=fee.return_value, - open_rate=0.123, - exchange='bittrex', - open_order_id='prod_buy_12345' - ) - Trade.session.add(trade) +from tests.conftest import log_has, create_mock_trades def test_init_create_session(default_conf): @@ -777,18 +731,31 @@ def test_to_json(default_conf, fee): assert result == {'trade_id': None, 'pair': 'ETH/BTC', + 'is_open': None, 'open_date_hum': '2 hours ago', 'open_date': trade.open_date.strftime("%Y-%m-%d %H:%M:%S"), + 'open_order_id': 'dry_run_buy_12345', 'close_date_hum': None, 'close_date': None, 'open_rate': 0.123, + 'open_rate_requested': None, + 'open_trade_price': 15.1668225, + 'fee_close': 0.0025, + 'fee_open': 0.0025, 'close_rate': None, + 'close_rate_requested': None, 'amount': 123.0, 'stake_amount': 0.001, + 'close_profit': None, + 'sell_reason': None, 'stop_loss': None, 'stop_loss_pct': None, 'initial_stop_loss': None, - 'initial_stop_loss_pct': None} + 'initial_stop_loss_pct': None, + 'min_rate': None, + 'max_rate': None, + 'strategy': None, + 'ticker_interval': None} # Simulate dry_run entries trade = Trade( @@ -819,7 +786,20 @@ def test_to_json(default_conf, fee): 'stop_loss': None, 'stop_loss_pct': None, 'initial_stop_loss': None, - 'initial_stop_loss_pct': None} + 'initial_stop_loss_pct': None, + 'close_profit': None, + 'close_rate_requested': None, + 'fee_close': 0.0025, + 'fee_open': 0.0025, + 'is_open': None, + 'max_rate': None, + 'min_rate': None, + 'open_order_id': None, + 'open_rate_requested': None, + 'open_trade_price': 12.33075, + 'sell_reason': None, + 'strategy': None, + 'ticker_interval': None} def test_stoploss_reinitialization(default_conf, fee): diff --git a/tests/test_plotting.py b/tests/test_plotting.py index a5c965429..0258b94d1 100644 --- a/tests/test_plotting.py +++ b/tests/test_plotting.py @@ -266,7 +266,7 @@ def test_generate_profit_graph(testdatadir): filename = testdatadir / "backtest-result_test.json" trades = load_backtest_data(filename) timerange = TimeRange.parse_timerange("20180110-20180112") - pairs = ["TRX/BTC", "ADA/BTC"] + pairs = ["TRX/BTC", "XLM/BTC"] trades = trades[trades['close_time'] < pd.Timestamp('2018-01-12', tz='UTC')] data = history.load_data(datadir=testdatadir, @@ -292,7 +292,7 @@ def test_generate_profit_graph(testdatadir): profit = find_trace_in_fig_data(figure.data, "Profit") assert isinstance(profit, go.Scatter) - profit = find_trace_in_fig_data(figure.data, "Max drawdown 0.00%") + profit = find_trace_in_fig_data(figure.data, "Max drawdown 10.45%") assert isinstance(profit, go.Scatter) for pair in pairs: diff --git a/user_data/logs/.gitkeep b/user_data/logs/.gitkeep new file mode 100644 index 000000000..e69de29bb