diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 3c511f44d..e15059f56 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -11,7 +11,7 @@ Few pointers for contributions: - Create your PR against the `develop` branch, not `master`. - New features need to contain unit tests and must be PEP8 conformant (max-line-length = 100). -If you are unsure, discuss the feature on our [Slack](https://join.slack.com/t/highfrequencybot/shared_invite/enQtMjQ5NTM0OTYzMzY3LWMxYzE3M2MxNDdjMGM3ZTYwNzFjMGIwZGRjNTc3ZGU3MGE3NzdmZGMwNmU3NDM5ZTNmM2Y3NjRiNzk4NmM4OGE) +If you are unsure, discuss the feature on our [Slack](https://join.slack.com/t/highfrequencybot/shared_invite/enQtNjU5ODcwNjI1MDU3LWEyODBiNzkzNzcyNzU0MWYyYzE5NjIyOTQxMzBmMGUxOTIzM2YyN2Y4NWY1YTEwZDgwYTRmMzE2NmM5ZmY2MTg) or in a [issue](https://github.com/freqtrade/freqtrade/issues) before a PR. ## Getting started diff --git a/README.md b/README.md index 98dad1d2e..240b4f917 100644 --- a/README.md +++ b/README.md @@ -141,7 +141,7 @@ Accounts having BNB accounts use this to pay for fees - if your first trade happ For any questions not covered by the documentation or for further information about the bot, we encourage you to join our slack channel. -- [Click here to join Slack channel](https://join.slack.com/t/highfrequencybot/shared_invite/enQtMjQ5NTM0OTYzMzY3LWMxYzE3M2MxNDdjMGM3ZTYwNzFjMGIwZGRjNTc3ZGU3MGE3NzdmZGMwNmU3NDM5ZTNmM2Y3NjRiNzk4NmM4OGE). +- [Click here to join Slack channel](https://join.slack.com/t/highfrequencybot/shared_invite/enQtNjU5ODcwNjI1MDU3LWEyODBiNzkzNzcyNzU0MWYyYzE5NjIyOTQxMzBmMGUxOTIzM2YyN2Y4NWY1YTEwZDgwYTRmMzE2NmM5ZmY2MTg). ### [Bugs / Issues](https://github.com/freqtrade/freqtrade/issues?q=is%3Aissue) @@ -172,7 +172,7 @@ to understand the requirements before sending your pull-requests. Coding is not a neccessity to contribute - maybe start with improving our documentation? Issues labeled [good first issue](https://github.com/freqtrade/freqtrade/labels/good%20first%20issue) can be good first contributions, and will help get you familiar with the codebase. -**Note** before starting any major new feature work, *please open an issue describing what you are planning to do* or talk to us on [Slack](https://join.slack.com/t/highfrequencybot/shared_invite/enQtMjQ5NTM0OTYzMzY3LWMxYzE3M2MxNDdjMGM3ZTYwNzFjMGIwZGRjNTc3ZGU3MGE3NzdmZGMwNmU3NDM5ZTNmM2Y3NjRiNzk4NmM4OGE). This will ensure that interested parties can give valuable feedback on the feature, and let others know that you are working on it. +**Note** before starting any major new feature work, *please open an issue describing what you are planning to do* or talk to us on [Slack](https://join.slack.com/t/highfrequencybot/shared_invite/enQtNjU5ODcwNjI1MDU3LWEyODBiNzkzNzcyNzU0MWYyYzE5NjIyOTQxMzBmMGUxOTIzM2YyN2Y4NWY1YTEwZDgwYTRmMzE2NmM5ZmY2MTg). This will ensure that interested parties can give valuable feedback on the feature, and let others know that you are working on it. **Important:** Always create your PR against the `develop` branch, not `master`. diff --git a/docs/backtesting.md b/docs/backtesting.md index a25d3c1d5..5a25bc255 100644 --- a/docs/backtesting.md +++ b/docs/backtesting.md @@ -123,11 +123,12 @@ python scripts/download_backtest_data.py --exchange binance This will download ticker data for all the currency pairs you defined in `pairs.json`. -- To use a different folder than the exchange specific default, use `--export user_data/data/some_directory`. +- To use a different folder than the exchange specific default, use `--datadir user_data/data/some_directory`. - To change the exchange used to download the tickers, use `--exchange`. Default is `bittrex`. - To use `pairs.json` from some other folder, use `--pairs-file some_other_dir/pairs.json`. - To download ticker data for only 10 days, use `--days 10`. - Use `--timeframes` to specify which tickers to download. Default is `--timeframes 1m 5m` which will download 1-minute and 5-minute tickers. +- To use exchange, timeframe and list of pairs as defined in your configuration file, use the `-c/--config` option. With this, the script uses the whitelist defined in the config as the list of currency pairs to download data for and does not require the pairs.json file. You can combine `-c/--config` with other options. For help about backtesting usage, please refer to [Backtesting commands](#backtesting-commands). diff --git a/docs/bot-usage.md b/docs/bot-usage.md index cb98e1ea5..b215d7b7c 100644 --- a/docs/bot-usage.md +++ b/docs/bot-usage.md @@ -26,7 +26,8 @@ optional arguments: --version show program's version number and exit -c PATH, --config PATH Specify configuration file (default: None). Multiple - --config options may be used. + --config options may be used. Can be set to '-' to + read config from stdin. -d PATH, --datadir PATH Path to backtest data. -s NAME, --strategy NAME diff --git a/docs/developer.md b/docs/developer.md index 74535234d..cf6b5d2cd 100644 --- a/docs/developer.md +++ b/docs/developer.md @@ -2,7 +2,7 @@ This page is intended for developers of FreqTrade, people who want to contribute to the FreqTrade codebase or documentation, or people who want to understand the source code of the application they're running. -All contributions, bug reports, bug fixes, documentation improvements, enhancements and ideas are welcome. We [track issues](https://github.com/freqtrade/freqtrade/issues) on [GitHub](https://github.com) and also have a dev channel in [slack](https://join.slack.com/t/highfrequencybot/shared_invite/enQtMjQ5NTM0OTYzMzY3LWMxYzE3M2MxNDdjMGM3ZTYwNzFjMGIwZGRjNTc3ZGU3MGE3NzdmZGMwNmU3NDM5ZTNmM2Y3NjRiNzk4NmM4OGE) where you can ask questions. +All contributions, bug reports, bug fixes, documentation improvements, enhancements and ideas are welcome. We [track issues](https://github.com/freqtrade/freqtrade/issues) on [GitHub](https://github.com) and also have a dev channel in [slack](https://join.slack.com/t/highfrequencybot/shared_invite/enQtNjU5ODcwNjI1MDU3LWEyODBiNzkzNzcyNzU0MWYyYzE5NjIyOTQxMzBmMGUxOTIzM2YyN2Y4NWY1YTEwZDgwYTRmMzE2NmM5ZmY2MTg) where you can ask questions. ## Documentation diff --git a/docs/hyperopt.md b/docs/hyperopt.md index 79ea4771b..15b02b56f 100644 --- a/docs/hyperopt.md +++ b/docs/hyperopt.md @@ -12,7 +12,7 @@ and still take a long time. ## Prepare Hyperopting Before we start digging into Hyperopt, we recommend you to take a look at -an example hyperopt file located into [user_data/hyperopts/](https://github.com/freqtrade/freqtrade/blob/develop/user_data/hyperopts/test_hyperopt.py) +an example hyperopt file located into [user_data/hyperopts/](https://github.com/freqtrade/freqtrade/blob/develop/user_data/hyperopts/sample_hyperopt.py) Configuring hyperopt is similar to writing your own strategy, and many tasks will be similar and a lot of code can be copied across from the strategy. @@ -71,6 +71,11 @@ Place the corresponding settings into the following methods The configuration and rules are the same than for buy signals. To avoid naming collisions in the search-space, please prefix all sell-spaces with `sell-`. +#### Using ticker-interval as part of the Strategy + +The Strategy exposes the ticker-interval as `self.ticker_interval`. The same value is available as class-attribute `HyperoptName.ticker_interval`. +In the case of the linked sample-value this would be `SampleHyperOpts.ticker_interval`. + ## Solving a Mystery Let's say you are curious: should you use MACD crossings or lower Bollinger diff --git a/docs/index.md b/docs/index.md index 9fbc0519c..63d6be75e 100644 --- a/docs/index.md +++ b/docs/index.md @@ -64,7 +64,7 @@ To run this bot we recommend you a cloud instance with a minimum of: Help / Slack For any questions not covered by the documentation or for further information about the bot, we encourage you to join our Slack channel. -Click [here](https://join.slack.com/t/highfrequencybot/shared_invite/enQtMjQ5NTM0OTYzMzY3LWMxYzE3M2MxNDdjMGM3ZTYwNzFjMGIwZGRjNTc3ZGU3MGE3NzdmZGMwNmU3NDM5ZTNmM2Y3NjRiNzk4NmM4OGE) to join Slack channel. +Click [here](https://join.slack.com/t/highfrequencybot/shared_invite/enQtNjU5ODcwNjI1MDU3LWEyODBiNzkzNzcyNzU0MWYyYzE5NjIyOTQxMzBmMGUxOTIzM2YyN2Y4NWY1YTEwZDgwYTRmMzE2NmM5ZmY2MTg) to join Slack channel. ## Ready to try? diff --git a/docs/installation.md b/docs/installation.md index d215dc8d6..f0c536ade 100644 --- a/docs/installation.md +++ b/docs/installation.md @@ -6,7 +6,7 @@ This page explains how to prepare your environment for running the bot. Before running your bot in production you will need to setup few external API. In production mode, the bot will require valid Exchange API -credentials. We also reccomend a [Telegram bot](telegram-usage.md#setup-your-telegram-bot) (optional but recommended). +credentials. We also recommend a [Telegram bot](telegram-usage.md#setup-your-telegram-bot) (optional but recommended). - [Setup your exchange account](#setup-your-exchange-account) diff --git a/docs/strategy-customization.md b/docs/strategy-customization.md index fd9760bda..57c646aed 100644 --- a/docs/strategy-customization.md +++ b/docs/strategy-customization.md @@ -410,7 +410,7 @@ To get additional Ideas for strategies, head over to our [strategy repository](h Feel free to use any of them as inspiration for your own strategies. We're happy to accept Pull Requests containing new Strategies to that repo. -We also got a *strategy-sharing* channel in our [Slack community](https://join.slack.com/t/highfrequencybot/shared_invite/enQtMjQ5NTM0OTYzMzY3LWMxYzE3M2MxNDdjMGM3ZTYwNzFjMGIwZGRjNTc3ZGU3MGE3NzdmZGMwNmU3NDM5ZTNmM2Y3NjRiNzk4NmM4OGE) which is a great place to get and/or share ideas. +We also got a *strategy-sharing* channel in our [Slack community](https://join.slack.com/t/highfrequencybot/shared_invite/enQtNjU5ODcwNjI1MDU3LWEyODBiNzkzNzcyNzU0MWYyYzE5NjIyOTQxMzBmMGUxOTIzM2YyN2Y4NWY1YTEwZDgwYTRmMzE2NmM5ZmY2MTg) which is a great place to get and/or share ideas. ## Next step diff --git a/docs/webhook-config.md b/docs/webhook-config.md index 811b57f9b..112f8a77e 100644 --- a/docs/webhook-config.md +++ b/docs/webhook-config.md @@ -43,6 +43,7 @@ Possible parameters are: * `stake_amount` * `stake_currency` * `fiat_currency` +* `order_type` ### Webhooksell @@ -61,6 +62,7 @@ Possible parameters are: * `stake_currency` * `fiat_currency` * `sell_reason` +* `order_type` ### Webhookstatus diff --git a/freqtrade/arguments.py b/freqtrade/arguments.py index 89b587c6f..aa1a3e6da 100644 --- a/freqtrade/arguments.py +++ b/freqtrade/arguments.py @@ -33,7 +33,8 @@ class Arguments(object): self.parser = argparse.ArgumentParser(description=description) def _load_args(self) -> None: - self.common_args_parser() + self.common_options() + self.main_options() self._build_subcommands() def get_parsed_arg(self) -> argparse.Namespace: @@ -47,7 +48,7 @@ class Arguments(object): return self.parsed_arg - def parse_args(self) -> argparse.Namespace: + def parse_args(self, no_default_config: bool = False) -> argparse.Namespace: """ Parses given arguments and returns an argparse Namespace instance. """ @@ -55,123 +56,121 @@ class Arguments(object): # Workaround issue in argparse with action='append' and default value # (see https://bugs.python.org/issue16399) - if parsed_arg.config is None: + if not no_default_config and parsed_arg.config is None: parsed_arg.config = [constants.DEFAULT_CONFIG] return parsed_arg - def common_args_parser(self) -> None: + def common_options(self) -> None: """ - Parses given common arguments and returns them as a parsed object. + Parses arguments that are common for the main Freqtrade, all subcommands and scripts. """ - self.parser.add_argument( + parser = self.parser + + parser.add_argument( '-v', '--verbose', help='Verbose mode (-vv for more, -vvv to get all messages).', action='count', dest='loglevel', default=0, ) - self.parser.add_argument( + parser.add_argument( '--logfile', help='Log to the file specified', dest='logfile', - type=str, - metavar='FILE' + metavar='FILE', ) - self.parser.add_argument( + parser.add_argument( '--version', action='version', version=f'%(prog)s {__version__}' ) - self.parser.add_argument( + parser.add_argument( '-c', '--config', - help='Specify configuration file (default: %(default)s). ' - 'Multiple --config options may be used.', + help=f'Specify configuration file (default: {constants.DEFAULT_CONFIG}). ' + f'Multiple --config options may be used. ' + f'Can be set to `-` to read config from stdin.', dest='config', action='append', - type=str, metavar='PATH', ) - self.parser.add_argument( + parser.add_argument( '-d', '--datadir', help='Path to backtest data.', dest='datadir', - default=None, - type=str, metavar='PATH', ) - self.parser.add_argument( + + def main_options(self) -> None: + """ + Parses arguments for the main Freqtrade. + """ + parser = self.parser + + parser.add_argument( '-s', '--strategy', help='Specify strategy class name (default: %(default)s).', dest='strategy', default='DefaultStrategy', - type=str, metavar='NAME', ) - self.parser.add_argument( + parser.add_argument( '--strategy-path', help='Specify additional strategy lookup path.', dest='strategy_path', - type=str, metavar='PATH', ) - self.parser.add_argument( + parser.add_argument( '--dynamic-whitelist', - help='Dynamically generate and update whitelist' - ' based on 24h BaseVolume (default: %(const)s).' - ' DEPRECATED.', + help='Dynamically generate and update whitelist ' + 'based on 24h BaseVolume (default: %(const)s). ' + 'DEPRECATED.', dest='dynamic_whitelist', const=constants.DYNAMIC_WHITELIST, type=int, metavar='INT', nargs='?', ) - self.parser.add_argument( + parser.add_argument( '--db-url', - help='Override trades database URL, this is useful if dry_run is enabled' - ' or in custom deployments (default: %(default)s).', + help=f'Override trades database URL, this is useful if dry_run is enabled ' + f'or in custom deployments (default: {constants.DEFAULT_DB_DRYRUN_URL}.', dest='db_url', - type=str, metavar='PATH', ) - self.parser.add_argument( + parser.add_argument( '--sd-notify', help='Notify systemd service manager.', action='store_true', dest='sd_notify', ) - @staticmethod - def optimizer_shared_options(parser: argparse.ArgumentParser) -> None: + def common_optimize_options(self, subparser: argparse.ArgumentParser = None) -> None: """ - Parses given common arguments for Backtesting, Edge and Hyperopt modules. + Parses arguments common for Backtesting, Edge and Hyperopt modules. :param parser: - :return: """ + parser = subparser or self.parser + parser.add_argument( '-i', '--ticker-interval', help='Specify ticker interval (1m, 5m, 30m, 1h, 1d).', dest='ticker_interval', - type=str, ) parser.add_argument( '--timerange', help='Specify what timerange of data to use.', - default=None, - type=str, dest='timerange', ) parser.add_argument( '--max_open_trades', help='Specify max_open_trades to use.', - default=None, type=int, dest='max_open_trades', ) parser.add_argument( '--stake_amount', help='Specify stake_amount.', - default=None, type=float, dest='stake_amount', ) @@ -184,11 +183,12 @@ class Arguments(object): dest='refresh_pairs', ) - @staticmethod - def backtesting_options(parser: argparse.ArgumentParser) -> None: + def backtesting_options(self, subparser: argparse.ArgumentParser = None) -> None: """ Parses given arguments for Backtesting module. """ + parser = subparser or self.parser + parser.add_argument( '--eps', '--enable-position-stacking', help='Allow buying the same pair multiple times (position stacking).', @@ -224,47 +224,44 @@ class Arguments(object): '--export', help='Export backtest results, argument are: trades. ' 'Example --export=trades', - type=str, - default=None, dest='export', ) parser.add_argument( '--export-filename', - help='Save backtest results to this filename \ - requires --export to be set as well\ - Example --export-filename=user_data/backtest_data/backtest_today.json\ - (default: %(default)s)', - type=str, + help='Save backtest results to this filename ' + 'requires --export to be set as well. ' + 'Example --export-filename=user_data/backtest_data/backtest_today.json ' + '(default: %(default)s)', default=os.path.join('user_data', 'backtest_data', 'backtest-result.json'), dest='exportfilename', metavar='PATH', ) - @staticmethod - def edge_options(parser: argparse.ArgumentParser) -> None: + def edge_options(self, subparser: argparse.ArgumentParser = None) -> None: """ Parses given arguments for Edge module. """ + parser = subparser or self.parser + parser.add_argument( '--stoplosses', help='Defines a range of stoploss against which edge will assess the strategy ' - 'the format is "min,max,step" (without any space).' - 'example: --stoplosses=-0.01,-0.1,-0.001', - type=str, + 'the format is "min,max,step" (without any space). ' + 'Example: --stoplosses=-0.01,-0.1,-0.001', dest='stoploss_range', ) - @staticmethod - def hyperopt_options(parser: argparse.ArgumentParser) -> None: + def hyperopt_options(self, subparser: argparse.ArgumentParser = None) -> None: """ Parses given arguments for Hyperopt module. """ + parser = subparser or self.parser + parser.add_argument( '--customhyperopt', help='Specify hyperopt class name (default: %(default)s).', dest='hyperopt', default=constants.DEFAULT_HYPEROPT, - type=str, metavar='NAME', ) parser.add_argument( @@ -292,8 +289,8 @@ class Arguments(object): ) parser.add_argument( '-s', '--spaces', - help='Specify which parameters to hyperopt. Space separate list. \ - Default: %(default)s.', + help='Specify which parameters to hyperopt. Space separate list. ' + 'Default: %(default)s.', choices=['all', 'buy', 'sell', 'roi', 'stoploss'], default='all', nargs='+', @@ -321,7 +318,6 @@ class Arguments(object): '--random-state', help='Set random state to some positive integer for reproducible hyperopt results.', dest='hyperopt_random_state', - default=None, type=Arguments.check_int_positive, metavar='INT', ) @@ -335,33 +331,55 @@ class Arguments(object): metavar='INT', ) + def list_exchanges_options(self, subparser: argparse.ArgumentParser = None) -> None: + """ + Parses given arguments for the list-exchanges command. + """ + parser = subparser or self.parser + + parser.add_argument( + '-1', '--one-column', + help='Print exchanges in one column', + action='store_true', + dest='print_one_column', + ) + def _build_subcommands(self) -> None: """ - Builds and attaches all subcommands + Builds and attaches all subcommands. :return: None """ from freqtrade.optimize import start_backtesting, start_hyperopt, start_edge + from freqtrade.utils import start_list_exchanges subparsers = self.parser.add_subparsers(dest='subparser') # Add backtesting subcommand backtesting_cmd = subparsers.add_parser('backtesting', help='Backtesting module.') backtesting_cmd.set_defaults(func=start_backtesting) - self.optimizer_shared_options(backtesting_cmd) + self.common_optimize_options(backtesting_cmd) self.backtesting_options(backtesting_cmd) # Add edge subcommand edge_cmd = subparsers.add_parser('edge', help='Edge module.') edge_cmd.set_defaults(func=start_edge) - self.optimizer_shared_options(edge_cmd) + self.common_optimize_options(edge_cmd) self.edge_options(edge_cmd) # Add hyperopt subcommand hyperopt_cmd = subparsers.add_parser('hyperopt', help='Hyperopt module.') hyperopt_cmd.set_defaults(func=start_hyperopt) - self.optimizer_shared_options(hyperopt_cmd) + self.common_optimize_options(hyperopt_cmd) self.hyperopt_options(hyperopt_cmd) + # Add list-exchanges subcommand + list_exchanges_cmd = subparsers.add_parser( + 'list-exchanges', + help='Print available exchanges.' + ) + list_exchanges_cmd.set_defaults(func=start_list_exchanges) + self.list_exchanges_options(list_exchanges_cmd) + @staticmethod def parse_timerange(text: Optional[str]) -> TimeRange: """ @@ -416,78 +434,85 @@ class Arguments(object): ) return uint - def scripts_options(self) -> None: + def common_scripts_options(self, subparser: argparse.ArgumentParser = None) -> None: """ - Parses given arguments for scripts. + Parses arguments common for scripts. """ - self.parser.add_argument( + parser = subparser or self.parser + + parser.add_argument( '-p', '--pairs', help='Show profits for only this pairs. Pairs are comma-separated.', dest='pairs', - default=None ) - def testdata_dl_options(self) -> None: + def download_data_options(self) -> None: """ - Parses given arguments for testdata download + Parses given arguments for testdata download script """ - self.parser.add_argument( + parser = self.parser + + parser.add_argument( '--pairs-file', help='File containing a list of pairs to download.', dest='pairs_file', - default=None, - metavar='PATH', + metavar='FILE', ) - - self.parser.add_argument( - '--export', - help='Export files to given dir.', - dest='export', - default=None, - metavar='PATH', - ) - - self.parser.add_argument( - '-c', '--config', - help='Specify configuration file (default: %(default)s). ' - 'Multiple --config options may be used.', - dest='config', - action='append', - type=str, - metavar='PATH', - ) - - self.parser.add_argument( + parser.add_argument( '--days', help='Download data for given number of days.', dest='days', - type=int, + type=Arguments.check_int_positive, metavar='INT', - default=None ) - - self.parser.add_argument( + parser.add_argument( '--exchange', - help='Exchange name (default: %(default)s). Only valid if no config is provided.', + help=f'Exchange name (default: {constants.DEFAULT_EXCHANGE}). ' + f'Only valid if no config is provided.', dest='exchange', - type=str, - default='bittrex' ) - - self.parser.add_argument( + parser.add_argument( '-t', '--timeframes', - help='Specify which tickers to download. Space separated list. \ - Default: %(default)s.', + help=f'Specify which tickers to download. Space separated list. ' + f'Default: {constants.DEFAULT_DOWNLOAD_TICKER_INTERVALS}.', choices=['1m', '3m', '5m', '15m', '30m', '1h', '2h', '4h', '6h', '8h', '12h', '1d', '3d', '1w'], - default=['1m', '5m'], nargs='+', dest='timeframes', ) - - self.parser.add_argument( + parser.add_argument( '--erase', help='Clean all existing data for the selected exchange/pairs/timeframes.', dest='erase', action='store_true' ) + + def plot_dataframe_options(self) -> None: + """ + Parses given arguments for plot dataframe script + """ + parser = self.parser + + parser.add_argument( + '--indicators1', + help='Set indicators from your strategy you want in the first row of the graph. ' + 'Separate them with a coma. E.g: ema3,ema5 (default: %(default)s)', + default='sma,ema3,ema5', + dest='indicators1', + ) + + parser.add_argument( + '--indicators2', + help='Set indicators from your strategy you want in the third row of the graph. ' + 'Separate them with a coma. E.g: fastd,fastk (default: %(default)s)', + default='macd,macdsignal', + dest='indicators2', + ) + parser.add_argument( + '--plot-limit', + help='Specify tick limit for plotting - too high values cause huge files - ' + 'Default: %(default)s', + dest='plot_limit', + default=750, + type=int, + ) diff --git a/freqtrade/configuration.py b/freqtrade/configuration.py index c19580c36..b2c35c977 100644 --- a/freqtrade/configuration.py +++ b/freqtrade/configuration.py @@ -13,7 +13,8 @@ from jsonschema import Draft4Validator, validators from jsonschema.exceptions import ValidationError, best_match from freqtrade import OperationalException, constants -from freqtrade.exchange import is_exchange_supported, supported_exchanges +from freqtrade.exchange import (is_exchange_bad, is_exchange_available, + is_exchange_officially_supported, available_exchanges) from freqtrade.misc import deep_merge_dicts from freqtrade.state import RunMode @@ -33,13 +34,17 @@ def set_loggers(log_level: int = 0) -> None: logging.getLogger('telegram').setLevel(logging.INFO) -def _extend_with_default(validator_class): - validate_properties = validator_class.VALIDATORS["properties"] +def _extend_validator(validator_class): + """ + Extended validator for the Freqtrade configuration JSON Schema. + Currently it only handles defaults for subschemas. + """ + validate_properties = validator_class.VALIDATORS['properties'] def set_defaults(validator, properties, instance, schema): for prop, subschema in properties.items(): - if "default" in subschema: - instance.setdefault(prop, subschema["default"]) + if 'default' in subschema: + instance.setdefault(prop, subschema['default']) for error in validate_properties( validator, properties, instance, schema, @@ -47,11 +52,11 @@ def _extend_with_default(validator_class): yield error return validators.extend( - validator_class, {"properties": set_defaults}, + validator_class, {'properties': set_defaults} ) -ValidatorWithDefaults = _extend_with_default(Draft4Validator) +FreqtradeValidator = _extend_validator(Draft4Validator) class Configuration(object): @@ -74,6 +79,7 @@ class Configuration(object): # Now expecting a list of config filenames here, not a string for path in self.args.config: logger.info('Using config: %s ...', path) + # Merge config options, overwriting old values config = deep_merge_dicts(self._load_config_file(path), config) @@ -97,6 +103,9 @@ class Configuration(object): # Load Optimize configurations config = self._load_optimize_config(config) + # Add plotting options if available + config = self._load_plot_config(config) + # Set runmode if not self.runmode: # Handle real mode, infer dry/live from config @@ -113,7 +122,8 @@ class Configuration(object): :return: configuration as dictionary """ try: - with open(path) as file: + # Read config from stdin if requested in the options + with open(path) if path != '-' else sys.stdin as file: conf = json.load(file) except FileNotFoundError: raise OperationalException( @@ -122,12 +132,11 @@ class Configuration(object): return conf - def _load_common_config(self, config: Dict[str, Any]) -> Dict[str, Any]: + def _load_logging_config(self, config: Dict[str, Any]) -> None: """ - Extract information for sys.argv and load common configuration - :return: configuration as dictionary + Extract information for sys.argv and load logging configuration: + the --loglevel, --logfile options """ - # Log level if 'loglevel' in self.args and self.args.loglevel: config.update({'verbosity': self.args.loglevel}) @@ -153,6 +162,13 @@ class Configuration(object): set_loggers(config['verbosity']) logger.info('Verbosity set to %s', config['verbosity']) + def _load_common_config(self, config: Dict[str, Any]) -> Dict[str, Any]: + """ + Extract information for sys.argv and load common configuration + :return: configuration as dictionary + """ + self._load_logging_config(config) + # Support for sd_notify if self.args.sd_notify: config['internals'].update({'sd_notify': True}) @@ -228,6 +244,17 @@ class Configuration(object): else: logger.info(logstring.format(config[argname])) + def _load_datadir_config(self, config: Dict[str, Any]) -> None: + """ + Extract information for sys.argv and load datadir configuration: + the --datadir option + """ + if 'datadir' in self.args and self.args.datadir: + config.update({'datadir': self._create_datadir(config, self.args.datadir)}) + else: + config.update({'datadir': self._create_datadir(config, None)}) + logger.info('Using data folder: %s ...', config.get('datadir')) + def _load_optimize_config(self, config: Dict[str, Any]) -> Dict[str, Any]: """ Extract information for sys.argv and load Optimize configuration @@ -263,11 +290,7 @@ class Configuration(object): self._args_to_config(config, argname='timerange', logstring='Parameter --timerange detected: {} ...') - if 'datadir' in self.args and self.args.datadir: - config.update({'datadir': self._create_datadir(config, self.args.datadir)}) - else: - config.update({'datadir': self._create_datadir(config, None)}) - logger.info('Using data folder: %s ...', config.get('datadir')) + self._load_datadir_config(config) self._args_to_config(config, argname='refresh_pairs', logstring='Parameter -r/--refresh-pairs-cached detected ...') @@ -318,6 +341,26 @@ class Configuration(object): return config + def _load_plot_config(self, config: Dict[str, Any]) -> Dict[str, Any]: + """ + Extract information for sys.argv Plotting configuration + :return: configuration as dictionary + """ + + self._args_to_config(config, argname='pairs', + logstring='Using pairs {}') + + self._args_to_config(config, argname='indicators1', + logstring='Using indicators1: {}') + + self._args_to_config(config, argname='indicators2', + logstring='Using indicators2: {}') + + self._args_to_config(config, argname='plot_limit', + logstring='Limiting plot to: {}') + + return config + def _validate_config_schema(self, conf: Dict[str, Any]) -> Dict[str, Any]: """ Validate the configuration follow the Config Schema @@ -325,7 +368,7 @@ class Configuration(object): :return: Returns the config if valid, otherwise throw an exception """ try: - ValidatorWithDefaults(constants.CONF_SCHEMA).validate(conf) + FreqtradeValidator(constants.CONF_SCHEMA).validate(conf) return conf except ValidationError as exception: logger.critical( @@ -375,22 +418,40 @@ class Configuration(object): return self.config - def check_exchange(self, config: Dict[str, Any]) -> bool: + def check_exchange(self, config: Dict[str, Any], check_for_bad: bool = True) -> bool: """ Check if the exchange name in the config file is supported by Freqtrade - :return: True or raised an exception if the exchange if not supported + :param check_for_bad: if True, check the exchange against the list of known 'bad' + exchanges + :return: False if exchange is 'bad', i.e. is known to work with the bot with + critical issues or does not work at all, crashes, etc. True otherwise. + raises an exception if the exchange if not supported by ccxt + and thus is not known for the Freqtrade at all. """ + logger.info("Checking exchange...") + exchange = config.get('exchange', {}).get('name').lower() - if not is_exchange_supported(exchange): - - exception_msg = f'Exchange "{exchange}" not supported.\n' \ - f'The following exchanges are supported: ' \ - f'{", ".join(supported_exchanges())}' - - logger.critical(exception_msg) + if not is_exchange_available(exchange): raise OperationalException( - exception_msg + f'Exchange "{exchange}" is not supported by ccxt ' + f'and therefore not available for the bot.\n' + f'The following exchanges are supported by ccxt: ' + f'{", ".join(available_exchanges())}' ) - logger.debug('Exchange "%s" supported', exchange) + if check_for_bad and is_exchange_bad(exchange): + logger.warning(f'Exchange "{exchange}" is known to not work with the bot yet. ' + f'Use it only for development and testing purposes.') + return False + + if is_exchange_officially_supported(exchange): + logger.info(f'Exchange "{exchange}" is officially supported ' + f'by the Freqtrade development team.') + else: + logger.warning(f'Exchange "{exchange}" is supported by ccxt ' + f'and therefore available for the bot but not officially supported ' + f'by the Freqtrade development team. ' + f'It may work flawlessly (please report back) or have serious issues. ' + f'Use it at your own discretion.') + return True diff --git a/freqtrade/constants.py b/freqtrade/constants.py index 4772952fc..7a487fcc7 100644 --- a/freqtrade/constants.py +++ b/freqtrade/constants.py @@ -4,6 +4,7 @@ bot constants """ DEFAULT_CONFIG = 'config.json' +DEFAULT_EXCHANGE = 'bittrex' DYNAMIC_WHITELIST = 20 # pairs PROCESS_THROTTLE_SECS = 5 # sec DEFAULT_TICKER_INTERVAL = 5 # min @@ -21,6 +22,7 @@ ORDERTYPE_POSSIBILITIES = ['limit', 'market'] ORDERTIF_POSSIBILITIES = ['gtc', 'fok', 'ioc'] AVAILABLE_PAIRLISTS = ['StaticPairList', 'VolumePairList'] DRY_RUN_WALLET = 999.9 +DEFAULT_DOWNLOAD_TICKER_INTERVALS = '1m 5m' TICKER_INTERVALS = [ '1m', '3m', '5m', '15m', '30m', diff --git a/freqtrade/data/btanalysis.py b/freqtrade/data/btanalysis.py index 6fce4361b..f78ca3fa8 100644 --- a/freqtrade/data/btanalysis.py +++ b/freqtrade/data/btanalysis.py @@ -1,12 +1,18 @@ """ Helpers when analyzing backtest data """ +import logging from pathlib import Path import numpy as np import pandas as pd +import pytz +from freqtrade import persistence from freqtrade.misc import json_load +from freqtrade.persistence import Trade + +logger = logging.getLogger(__name__) # must align with columns in backtest.py BT_DATA_COLUMNS = ["pair", "profitperc", "open_time", "close_time", "index", "duration", @@ -17,7 +23,7 @@ def load_backtest_data(filename) -> pd.DataFrame: """ Load backtest data file. :param filename: pathlib.Path object, or string pointing to the file. - :return a dataframe with the analysis results + :return: a dataframe with the analysis results """ if isinstance(filename, str): filename = Path(filename) @@ -65,3 +71,48 @@ def evaluate_result_multi(results: pd.DataFrame, freq: str, max_open_trades: int df2 = df2.set_index('date') df_final = df2.resample(freq)[['pair']].count() return df_final[df_final['pair'] > max_open_trades] + + +def load_trades(db_url: str = None, exportfilename: str = None) -> pd.DataFrame: + """ + Load trades, either from a DB (using dburl) or via a backtest export file. + :param db_url: Sqlite url (default format sqlite:///tradesv3.dry-run.sqlite) + :param exportfilename: Path to a file exported from backtesting + :return: Dataframe containing Trades + """ + timeZone = pytz.UTC + + trades: pd.DataFrame = pd.DataFrame([], columns=BT_DATA_COLUMNS) + + if db_url: + persistence.init(db_url, clean_open_orders=False) + columns = ["pair", "profit", "open_time", "close_time", + "open_rate", "close_rate", "duration"] + + for x in Trade.query.all(): + logger.info("date: {}".format(x.open_date)) + + trades = pd.DataFrame([(t.pair, t.calc_profit(), + t.open_date.replace(tzinfo=timeZone), + t.close_date.replace(tzinfo=timeZone) if t.close_date else None, + t.open_rate, t.close_rate, + t.close_date.timestamp() - t.open_date.timestamp() + if t.close_date else None) + for t in Trade.query.all()], + columns=columns) + + elif exportfilename: + + trades = load_backtest_data(Path(exportfilename)) + + return trades + + +def extract_trades_of_period(dataframe: pd.DataFrame, trades: pd.DataFrame) -> pd.DataFrame: + """ + Compare trades and backtested pair DataFrames to get trades performed on backtested period + :return: the DataFrame of a trades of period + """ + trades = trades.loc[(trades['open_time'] >= dataframe.iloc[0]['date']) & + (trades['close_time'] <= dataframe.iloc[-1]['date'])] + return trades diff --git a/freqtrade/data/history.py b/freqtrade/data/history.py index 655034b33..2a0d9b15e 100644 --- a/freqtrade/data/history.py +++ b/freqtrade/data/history.py @@ -63,7 +63,7 @@ def load_tickerdata_file( timerange: Optional[TimeRange] = None) -> Optional[list]: """ Load a pair from file, either .json.gz or .json - :return tickerlist or None if unsuccesful + :return: tickerlist or None if unsuccesful """ filename = pair_data_filename(datadir, pair, ticker_interval) pairdata = misc.file_load_json(filename) diff --git a/freqtrade/exchange/__init__.py b/freqtrade/exchange/__init__.py index 3c90e69ee..5c58320f6 100644 --- a/freqtrade/exchange/__init__.py +++ b/freqtrade/exchange/__init__.py @@ -1,6 +1,8 @@ from freqtrade.exchange.exchange import Exchange # noqa: F401 -from freqtrade.exchange.exchange import (is_exchange_supported, # noqa: F401 - supported_exchanges) +from freqtrade.exchange.exchange import (is_exchange_bad, # noqa: F401 + is_exchange_available, + is_exchange_officially_supported, + available_exchanges) from freqtrade.exchange.exchange import (timeframe_to_seconds, # noqa: F401 timeframe_to_minutes, timeframe_to_msecs) diff --git a/freqtrade/exchange/exchange.py b/freqtrade/exchange/exchange.py index db264d1bc..a65294091 100644 --- a/freqtrade/exchange/exchange.py +++ b/freqtrade/exchange/exchange.py @@ -156,8 +156,8 @@ class Exchange(object): # Find matching class for the given exchange name name = exchange_config['name'] - if not is_exchange_supported(name, ccxt_module): - raise OperationalException(f'Exchange {name} is not supported') + if not is_exchange_available(name, ccxt_module): + raise OperationalException(f'Exchange {name} is not supported by ccxt') ex_config = { 'apiKey': exchange_config.get('key'), @@ -722,11 +722,19 @@ class Exchange(object): raise OperationalException(e) -def is_exchange_supported(exchange: str, ccxt_module=None) -> bool: - return exchange in supported_exchanges(ccxt_module) +def is_exchange_bad(exchange: str) -> bool: + return exchange in ['bitmex'] -def supported_exchanges(ccxt_module=None) -> List[str]: +def is_exchange_available(exchange: str, ccxt_module=None) -> bool: + return exchange in available_exchanges(ccxt_module) + + +def is_exchange_officially_supported(exchange: str) -> bool: + return exchange in ['bittrex', 'binance'] + + +def available_exchanges(ccxt_module=None) -> List[str]: return ccxt_module.exchanges if ccxt_module is not None else ccxt.exchanges diff --git a/freqtrade/freqtradebot.py b/freqtrade/freqtradebot.py index 471e9d218..b6fc005dd 100644 --- a/freqtrade/freqtradebot.py +++ b/freqtrade/freqtradebot.py @@ -53,8 +53,7 @@ class FreqtradeBot(object): self.rpc: RPCManager = RPCManager(self) - exchange_name = self.config.get('exchange', {}).get('name').title() - self.exchange = ExchangeResolver(exchange_name, self.config).exchange + self.exchange = ExchangeResolver(self.config['exchange']['name'], self.config).exchange self.wallets = Wallets(self.config, self.exchange) self.dataprovider = DataProvider(self.config, self.exchange) @@ -205,19 +204,19 @@ class FreqtradeBot(object): else: stake_amount = self.config['stake_amount'] - avaliable_amount = self.wallets.get_free(self.config['stake_currency']) + available_amount = self.wallets.get_free(self.config['stake_currency']) if stake_amount == constants.UNLIMITED_STAKE_AMOUNT: open_trades = len(Trade.get_open_trades()) if open_trades >= self.config['max_open_trades']: logger.warning('Can\'t open a new trade: max number of trades is reached') return None - return avaliable_amount / (self.config['max_open_trades'] - open_trades) + return available_amount / (self.config['max_open_trades'] - open_trades) # Check if stake_amount is fulfilled - if avaliable_amount < stake_amount: + if available_amount < stake_amount: raise DependencyException( - f"Available balance({avaliable_amount} {self.config['stake_currency']}) is " + f"Available balance({available_amount} {self.config['stake_currency']}) is " f"lower than stake amount({stake_amount} {self.config['stake_currency']})" ) @@ -345,8 +344,8 @@ class FreqtradeBot(object): return False amount = stake_amount / buy_limit_requested - - order = self.exchange.buy(pair=pair, ordertype=self.strategy.order_types['buy'], + order_type = self.strategy.order_types['buy'] + order = self.exchange.buy(pair=pair, ordertype=order_type, amount=amount, rate=buy_limit_requested, time_in_force=time_in_force) order_id = order['id'] @@ -356,7 +355,6 @@ class FreqtradeBot(object): buy_limit_filled_price = buy_limit_requested if order_status == 'expired' or order_status == 'rejected': - order_type = self.strategy.order_types['buy'] order_tif = self.strategy.order_time_in_force['buy'] # return false if the order is not filled @@ -390,6 +388,7 @@ class FreqtradeBot(object): 'exchange': self.exchange.name.capitalize(), 'pair': pair_s, 'limit': buy_limit_filled_price, + 'order_type': order_type, 'stake_amount': stake_amount, 'stake_currency': stake_currency, 'fiat_currency': fiat_currency @@ -691,13 +690,22 @@ class FreqtradeBot(object): # cancelling the current stoploss on exchange first logger.info('Trailing stoploss: cancelling current stoploss on exchange (id:{%s})' 'in order to add another one ...', order['id']) - if self.exchange.cancel_order(order['id'], trade.pair): + try: + self.exchange.cancel_order(order['id'], trade.pair) + except InvalidOrderException: + logger.exception(f"Could not cancel stoploss order {order['id']} " + f"for pair {trade.pair}") + + try: # creating the new one stoploss_order_id = self.exchange.stoploss_limit( pair=trade.pair, amount=trade.amount, stop_price=trade.stop_loss, rate=trade.stop_loss * 0.99 )['id'] trade.stoploss_order_id = str(stoploss_order_id) + except DependencyException: + logger.exception(f"Could create trailing stoploss order " + f"for pair {trade.pair}.") def check_sell(self, trade: Trade, sell_rate: float, buy: bool, sell: bool) -> bool: if self.edge: @@ -843,7 +851,10 @@ class FreqtradeBot(object): # First cancelling stoploss on exchange ... if self.strategy.order_types.get('stoploss_on_exchange') and trade.stoploss_order_id: - self.exchange.cancel_order(trade.stoploss_order_id, trade.pair) + try: + self.exchange.cancel_order(trade.stoploss_order_id, trade.pair) + except InvalidOrderException: + logger.exception(f"Could not cancel stoploss order {trade.stoploss_order_id}") # Execute sell and update trade record order_id = self.exchange.sell(pair=str(trade.pair), @@ -875,6 +886,7 @@ class FreqtradeBot(object): 'pair': trade.pair, 'gain': gain, 'limit': trade.close_rate_requested, + 'order_type': self.strategy.order_types['sell'], 'amount': trade.amount, 'open_rate': trade.open_rate, 'current_rate': current_rate, diff --git a/freqtrade/optimize/__init__.py b/freqtrade/optimize/__init__.py index 475aaa82f..8b548eefe 100644 --- a/freqtrade/optimize/__init__.py +++ b/freqtrade/optimize/__init__.py @@ -5,8 +5,9 @@ from typing import Any, Dict from filelock import FileLock, Timeout from freqtrade import DependencyException, constants -from freqtrade.configuration import Configuration from freqtrade.state import RunMode +from freqtrade.utils import setup_utils_configuration + logger = logging.getLogger(__name__) @@ -17,12 +18,7 @@ def setup_configuration(args: Namespace, method: RunMode) -> Dict[str, Any]: :param args: Cli args from Arguments() :return: Configuration """ - configuration = Configuration(args, method) - config = configuration.load_config() - - # Ensure we do not use Exchange credentials - config['exchange']['key'] = '' - config['exchange']['secret'] = '' + config = setup_utils_configuration(args, method) if method == RunMode.BACKTEST: if config['stake_amount'] == constants.UNLIMITED_STAKE_AMOUNT: diff --git a/freqtrade/optimize/backtesting.py b/freqtrade/optimize/backtesting.py index bb72cbada..923119591 100644 --- a/freqtrade/optimize/backtesting.py +++ b/freqtrade/optimize/backtesting.py @@ -63,8 +63,7 @@ class Backtesting(object): self.config['dry_run'] = True self.strategylist: List[IStrategy] = [] - exchange_name = self.config.get('exchange', {}).get('name').title() - self.exchange = ExchangeResolver(exchange_name, self.config).exchange + self.exchange = ExchangeResolver(self.config['exchange']['name'], self.config).exchange self.fee = self.exchange.get_fee() if self.config.get('runmode') != RunMode.HYPEROPT: diff --git a/freqtrade/optimize/edge_cli.py b/freqtrade/optimize/edge_cli.py index 8232c79c9..231493e4d 100644 --- a/freqtrade/optimize/edge_cli.py +++ b/freqtrade/optimize/edge_cli.py @@ -6,6 +6,7 @@ This module contains the edge backtesting interface import logging from typing import Dict, Any from tabulate import tabulate +from freqtrade import constants from freqtrade.edge import Edge from freqtrade.arguments import Arguments @@ -32,6 +33,7 @@ class EdgeCli(object): self.config['exchange']['secret'] = '' self.config['exchange']['password'] = '' self.config['exchange']['uid'] = '' + self.config['stake_amount'] = constants.UNLIMITED_STAKE_AMOUNT self.config['dry_run'] = True self.exchange = Exchange(self.config) self.strategy = StrategyResolver(self.config).strategy diff --git a/freqtrade/optimize/hyperopt_interface.py b/freqtrade/optimize/hyperopt_interface.py index 622de3015..08823ece0 100644 --- a/freqtrade/optimize/hyperopt_interface.py +++ b/freqtrade/optimize/hyperopt_interface.py @@ -20,6 +20,7 @@ class IHyperOpt(ABC): stoploss -> float: optimal stoploss designed for the strategy ticker_interval -> int: value of the ticker interval to use for the strategy """ + ticker_interval: str @staticmethod @abstractmethod diff --git a/freqtrade/plot/__init__.py b/freqtrade/plot/__init__.py new file mode 100644 index 000000000..e69de29bb diff --git a/freqtrade/plot/plotting.py b/freqtrade/plot/plotting.py new file mode 100644 index 000000000..94c0830bf --- /dev/null +++ b/freqtrade/plot/plotting.py @@ -0,0 +1,221 @@ +import logging +from typing import List + +import pandas as pd +from pathlib import Path + +logger = logging.getLogger(__name__) + + +try: + from plotly import tools + from plotly.offline import plot + import plotly.graph_objs as go +except ImportError: + logger.exception("Module plotly not found \n Please install using `pip install plotly`") + exit(1) + + +def generate_row(fig, row, indicators: List[str], data: pd.DataFrame) -> tools.make_subplots: + """ + Generator all the indicator selected by the user for a specific row + :param fig: Plot figure to append to + :param row: row number for this plot + :param indicators: List of indicators present in the dataframe + :param data: candlestick DataFrame + """ + for indicator in indicators: + if indicator in data: + # TODO: Figure out why scattergl causes problems + scattergl = go.Scatter( + x=data['date'], + y=data[indicator].values, + mode='lines', + name=indicator + ) + fig.append_trace(scattergl, row, 1) + else: + logger.info( + 'Indicator "%s" ignored. Reason: This indicator is not found ' + 'in your strategy.', + indicator + ) + + return fig + + +def plot_trades(fig, trades: pd.DataFrame): + """ + Plot trades to "fig" + """ + # Trades can be empty + if trades is not None and len(trades) > 0: + trade_buys = go.Scatter( + x=trades["open_time"], + y=trades["open_rate"], + mode='markers', + name='trade_buy', + marker=dict( + symbol='square-open', + size=11, + line=dict(width=2), + color='green' + ) + ) + # Create description for sell summarizing the trade + desc = trades.apply(lambda row: f"{round(row['profitperc'], 3)}%, {row['sell_reason']}, " + f"{row['duration']}min", + axis=1) + trade_sells = go.Scatter( + x=trades["close_time"], + y=trades["close_rate"], + text=desc, + mode='markers', + name='trade_sell', + marker=dict( + symbol='square-open', + size=11, + line=dict(width=2), + color='red' + ) + ) + fig.append_trace(trade_buys, 1, 1) + fig.append_trace(trade_sells, 1, 1) + return fig + + +def generate_graph( + pair: str, + data: pd.DataFrame, + trades: pd.DataFrame = None, + indicators1: List[str] = [], + indicators2: List[str] = [], +) -> go.Figure: + """ + Generate the graph from the data generated by Backtesting or from DB + Volume will always be ploted in row2, so Row 1 and 3 are to our disposal for custom indicators + :param pair: Pair to Display on the graph + :param data: OHLCV DataFrame containing indicators and buy/sell signals + :param trades: All trades created + :param indicators1: List containing Main plot indicators + :param indicators2: List containing Sub plot indicators + :return: None + """ + + # Define the graph + fig = tools.make_subplots( + rows=3, + cols=1, + shared_xaxes=True, + row_width=[1, 1, 4], + vertical_spacing=0.0001, + ) + fig['layout'].update(title=pair) + fig['layout']['yaxis1'].update(title='Price') + fig['layout']['yaxis2'].update(title='Volume') + fig['layout']['yaxis3'].update(title='Other') + fig['layout']['xaxis']['rangeslider'].update(visible=False) + + # Common information + candles = go.Candlestick( + x=data.date, + open=data.open, + high=data.high, + low=data.low, + close=data.close, + name='Price' + ) + fig.append_trace(candles, 1, 1) + + if 'buy' in data.columns: + df_buy = data[data['buy'] == 1] + if len(df_buy) > 0: + buys = go.Scatter( + x=df_buy.date, + y=df_buy.close, + mode='markers', + name='buy', + marker=dict( + symbol='triangle-up-dot', + size=9, + line=dict(width=1), + color='green', + ) + ) + fig.append_trace(buys, 1, 1) + else: + logger.warning("No buy-signals found.") + + if 'sell' in data.columns: + df_sell = data[data['sell'] == 1] + if len(df_sell) > 0: + sells = go.Scatter( + x=df_sell.date, + y=df_sell.close, + mode='markers', + name='sell', + marker=dict( + symbol='triangle-down-dot', + size=9, + line=dict(width=1), + color='red', + ) + ) + fig.append_trace(sells, 1, 1) + else: + logger.warning("No sell-signals found.") + + if 'bb_lowerband' in data and 'bb_upperband' in data: + bb_lower = go.Scattergl( + x=data.date, + y=data.bb_lowerband, + name='BB lower', + line={'color': 'rgba(255,255,255,0)'}, + ) + bb_upper = go.Scattergl( + x=data.date, + y=data.bb_upperband, + name='BB upper', + fill="tonexty", + fillcolor="rgba(0,176,246,0.2)", + line={'color': 'rgba(255,255,255,0)'}, + ) + fig.append_trace(bb_lower, 1, 1) + fig.append_trace(bb_upper, 1, 1) + + # Add indicators to main plot + fig = generate_row(fig=fig, row=1, indicators=indicators1, data=data) + + fig = plot_trades(fig, trades) + + # Volume goes to row 2 + volume = go.Bar( + x=data['date'], + y=data['volume'], + name='Volume' + ) + fig.append_trace(volume, 2, 1) + + # Add indicators to seperate row + fig = generate_row(fig=fig, row=3, indicators=indicators2, data=data) + + return fig + + +def generate_plot_file(fig, pair, ticker_interval) -> None: + """ + Generate a plot html file from pre populated fig plotly object + :param fig: Plotly Figure to plot + :param pair: Pair to plot (used as filename and Plot title) + :param ticker_interval: Used as part of the filename + :return: None + """ + logger.info('Generate plot file for %s', pair) + + pair_name = pair.replace("/", "_") + file_name = 'freqtrade-plot-' + pair_name + '-' + ticker_interval + '.html' + + Path("user_data/plots").mkdir(parents=True, exist_ok=True) + + plot(fig, filename=str(Path('user_data/plots').joinpath(file_name)), + auto_open=False) diff --git a/freqtrade/resolvers/exchange_resolver.py b/freqtrade/resolvers/exchange_resolver.py index 8d1845c71..25a86dd0e 100644 --- a/freqtrade/resolvers/exchange_resolver.py +++ b/freqtrade/resolvers/exchange_resolver.py @@ -22,6 +22,7 @@ class ExchangeResolver(IResolver): Load the custom class from config parameter :param config: configuration dictionary """ + exchange_name = exchange_name.title() try: self.exchange = self._load_exchange(exchange_name, kwargs={'config': config}) except ImportError: diff --git a/freqtrade/resolvers/hyperopt_resolver.py b/freqtrade/resolvers/hyperopt_resolver.py index e7683bc78..9333bb09a 100644 --- a/freqtrade/resolvers/hyperopt_resolver.py +++ b/freqtrade/resolvers/hyperopt_resolver.py @@ -32,6 +32,9 @@ class HyperOptResolver(IResolver): hyperopt_name = config.get('hyperopt') or DEFAULT_HYPEROPT self.hyperopt = self._load_hyperopt(hyperopt_name, extra_dir=config.get('hyperopt_path')) + # Assign ticker_interval to be used in hyperopt + self.hyperopt.__class__.ticker_interval = str(config['ticker_interval']) + if not hasattr(self.hyperopt, 'populate_buy_trend'): logger.warning("Custom Hyperopt does not provide populate_buy_trend. " "Using populate_buy_trend from DefaultStrategy.") diff --git a/freqtrade/rpc/telegram.py b/freqtrade/rpc/telegram.py index 497c117ac..3eb060074 100644 --- a/freqtrade/rpc/telegram.py +++ b/freqtrade/rpc/telegram.py @@ -132,7 +132,7 @@ class Telegram(RPC): msg['stake_amount_fiat'] = 0 message = ("*{exchange}:* Buying {pair}\n" - "with limit `{limit:.8f}\n" + "at rate `{limit:.8f}\n" "({stake_amount:.6f} {stake_currency}").format(**msg) if msg.get('fiat_currency', None): @@ -144,7 +144,7 @@ class Telegram(RPC): msg['profit_percent'] = round(msg['profit_percent'] * 100, 2) message = ("*{exchange}:* Selling {pair}\n" - "*Limit:* `{limit:.8f}`\n" + "*Rate:* `{limit:.8f}`\n" "*Amount:* `{amount:.8f}`\n" "*Open Rate:* `{open_rate:.8f}`\n" "*Current Rate:* `{current_rate:.8f}`\n" diff --git a/freqtrade/strategy/interface.py b/freqtrade/strategy/interface.py index db266d95f..949a88b91 100644 --- a/freqtrade/strategy/interface.py +++ b/freqtrade/strategy/interface.py @@ -158,7 +158,7 @@ class IStrategy(ABC): """ Parses the given ticker history and returns a populated DataFrame add several TA indicators and buy signal to it - :return DataFrame with ticker data and indicator data + :return: DataFrame with ticker data and indicator data """ pair = str(metadata.get('pair')) @@ -308,14 +308,16 @@ class IStrategy(ABC): if trailing_stop: # trailing stoploss handling - sl_offset = self.config.get('trailing_stop_positive_offset') or 0.0 tsl_only_offset = self.config.get('trailing_only_offset_is_reached', False) + # Make sure current_profit is calculated using high for backtesting. + high_profit = current_profit if not high else trade.calc_profit_percent(high) + # Don't update stoploss if trailing_only_offset_is_reached is true. - if not (tsl_only_offset and current_profit < sl_offset): + if not (tsl_only_offset and high_profit < sl_offset): # Specific handling for trailing_stop_positive - if 'trailing_stop_positive' in self.config and current_profit > sl_offset: + if 'trailing_stop_positive' in self.config and high_profit > sl_offset: # Ignore mypy error check in configuration that this is a float stop_loss_value = self.config.get('trailing_stop_positive') # type: ignore logger.debug(f"using positive stop loss: {stop_loss_value} " @@ -349,7 +351,7 @@ class IStrategy(ABC): """ Based an earlier trade and current price and ROI configuration, decides whether bot should sell. Requires current_profit to be in percent!! - :return True if bot should sell at current rate + :return: True if bot should sell at current rate """ # Check if time matches and current rate is above threshold @@ -378,6 +380,7 @@ class IStrategy(ABC): :param metadata: Additional information, like the currently traded pair :return: a Dataframe with all mandatory indicators for the strategies """ + logger.debug(f"Populating indicators for pair {metadata.get('pair')}.") if self._populate_fun_len == 2: warnings.warn("deprecated - check out the Sample strategy to see " "the current function headers!", DeprecationWarning) @@ -393,6 +396,7 @@ class IStrategy(ABC): :param pair: Additional information, like the currently traded pair :return: DataFrame with buy column """ + logger.debug(f"Populating buy signals for pair {metadata.get('pair')}.") if self._buy_fun_len == 2: warnings.warn("deprecated - check out the Sample strategy to see " "the current function headers!", DeprecationWarning) @@ -408,6 +412,7 @@ class IStrategy(ABC): :param pair: Additional information, like the currently traded pair :return: DataFrame with sell column """ + logger.debug(f"Populating sell signals for pair {metadata.get('pair')}.") if self._sell_fun_len == 2: warnings.warn("deprecated - check out the Sample strategy to see " "the current function headers!", DeprecationWarning) diff --git a/freqtrade/tests/conftest.py b/freqtrade/tests/conftest.py index 808d128ad..eb2a8600f 100644 --- a/freqtrade/tests/conftest.py +++ b/freqtrade/tests/conftest.py @@ -5,6 +5,8 @@ import re from copy import deepcopy from datetime import datetime from functools import reduce +from pathlib import Path +from typing import List from unittest.mock import MagicMock, PropertyMock import arrow @@ -12,6 +14,7 @@ import pytest from telegram import Chat, Message, Update from freqtrade import constants, persistence +from freqtrade.arguments import Arguments from freqtrade.data.converter import parse_ticker_dataframe from freqtrade.edge import Edge, PairInfo from freqtrade.exchange import Exchange @@ -36,6 +39,10 @@ def log_has_re(line, logs): False) +def get_args(args) -> List[str]: + return Arguments(args, '').get_parsed_arg() + + def patch_exchange(mocker, api_mock=None, id='bittrex') -> None: mocker.patch('freqtrade.exchange.Exchange._load_markets', MagicMock(return_value={})) mocker.patch('freqtrade.exchange.Exchange.validate_pairs', MagicMock()) @@ -54,7 +61,7 @@ def get_patched_exchange(mocker, config, api_mock=None, id='bittrex') -> Exchang patch_exchange(mocker, api_mock, id) config["exchange"]["name"] = id try: - exchange = ExchangeResolver(id.title(), config).exchange + exchange = ExchangeResolver(id, config).exchange except ImportError: exchange = Exchange(config) return exchange @@ -104,11 +111,23 @@ def patch_freqtradebot(mocker, config) -> None: def get_patched_freqtradebot(mocker, config) -> FreqtradeBot: + """ + This function patches _init_modules() to not call dependencies + :param mocker: a Mocker object to apply patches + :param config: Config to pass to the bot + :return: FreqtradeBot + """ patch_freqtradebot(mocker, config) return FreqtradeBot(config) def get_patched_worker(mocker, config) -> Worker: + """ + This function patches _init_modules() to not call dependencies + :param mocker: a Mocker object to apply patches + :param config: Config to pass to the bot + :return: Worker + """ patch_freqtradebot(mocker, config) return Worker(args=None, config=config) @@ -145,6 +164,11 @@ def patch_coinmarketcap(mocker) -> None: ) +@pytest.fixture(scope='function') +def init_persistence(default_conf): + persistence.init(default_conf['db_url'], default_conf['dry_run']) + + @pytest.fixture(scope="function") def default_conf(): """ Returns validated configuration suitable for most tests """ @@ -854,9 +878,9 @@ def tickers(): @pytest.fixture def result(): - with open('freqtrade/tests/testdata/UNITTEST_BTC-1m.json') as data_file: - return parse_ticker_dataframe(json.load(data_file), '1m', - pair="UNITTEST/BTC", fill_missing=True) + with Path('freqtrade/tests/testdata/UNITTEST_BTC-1m.json').open('r') as data_file: + return parse_ticker_dataframe(json.load(data_file), '1m', pair="UNITTEST/BTC", + fill_missing=True) # FIX: # Create an fixture/function diff --git a/freqtrade/tests/data/test_btanalysis.py b/freqtrade/tests/data/test_btanalysis.py index dd7cbe0d9..6fa529394 100644 --- a/freqtrade/tests/data/test_btanalysis.py +++ b/freqtrade/tests/data/test_btanalysis.py @@ -1,8 +1,15 @@ -import pytest -from pandas import DataFrame +from unittest.mock import MagicMock -from freqtrade.data.btanalysis import BT_DATA_COLUMNS, load_backtest_data -from freqtrade.data.history import make_testdata_path +from arrow import Arrow +import pytest +from pandas import DataFrame, to_datetime + +from freqtrade.arguments import TimeRange +from freqtrade.data.btanalysis import (BT_DATA_COLUMNS, + extract_trades_of_period, + load_backtest_data, load_trades) +from freqtrade.data.history import load_pair_history, make_testdata_path +from freqtrade.tests.test_persistence import create_mock_trades def test_load_backtest_data(): @@ -19,3 +26,59 @@ def test_load_backtest_data(): with pytest.raises(ValueError, match=r"File .* does not exist\."): load_backtest_data(str("filename") + "nofile") + + +def test_load_trades_file(default_conf, fee, mocker): + # Real testing of load_backtest_data is done in test_load_backtest_data + lbt = mocker.patch("freqtrade.data.btanalysis.load_backtest_data", MagicMock()) + filename = make_testdata_path(None) / "backtest-result_test.json" + load_trades(db_url=None, exportfilename=filename) + assert lbt.call_count == 1 + + +@pytest.mark.usefixtures("init_persistence") +def test_load_trades_db(default_conf, fee, mocker): + + create_mock_trades(fee) + # remove init so it does not init again + init_mock = mocker.patch('freqtrade.persistence.init', MagicMock()) + + trades = load_trades(db_url=default_conf['db_url'], exportfilename=None) + assert init_mock.call_count == 1 + assert len(trades) == 3 + assert isinstance(trades, DataFrame) + assert "pair" in trades.columns + assert "open_time" in trades.columns + + +def test_extract_trades_of_period(): + pair = "UNITTEST/BTC" + timerange = TimeRange(None, 'line', 0, -1000) + + data = load_pair_history(pair=pair, ticker_interval='1m', + datadir=None, timerange=timerange) + + # timerange = 2017-11-14 06:07 - 2017-11-14 22:58:00 + trades = DataFrame( + {'pair': [pair, pair, pair, pair], + 'profit_percent': [0.0, 0.1, -0.2, -0.5], + 'profit_abs': [0.0, 1, -2, -5], + 'open_time': to_datetime([Arrow(2017, 11, 13, 15, 40, 0).datetime, + Arrow(2017, 11, 14, 9, 41, 0).datetime, + Arrow(2017, 11, 14, 14, 20, 0).datetime, + Arrow(2017, 11, 15, 3, 40, 0).datetime, + ], utc=True + ), + 'close_time': to_datetime([Arrow(2017, 11, 13, 16, 40, 0).datetime, + Arrow(2017, 11, 14, 10, 41, 0).datetime, + Arrow(2017, 11, 14, 15, 25, 0).datetime, + Arrow(2017, 11, 15, 3, 55, 0).datetime, + ], utc=True) + }) + trades1 = extract_trades_of_period(data, trades) + # First and last trade are dropped as they are out of range + assert len(trades1) == 2 + assert trades1.iloc[0].open_time == Arrow(2017, 11, 14, 9, 41, 0).datetime + assert trades1.iloc[0].close_time == Arrow(2017, 11, 14, 10, 41, 0).datetime + assert trades1.iloc[-1].open_time == Arrow(2017, 11, 14, 14, 20, 0).datetime + assert trades1.iloc[-1].close_time == Arrow(2017, 11, 14, 15, 25, 0).datetime diff --git a/freqtrade/tests/exchange/test_exchange.py b/freqtrade/tests/exchange/test_exchange.py index f0dc96626..48a8538a9 100644 --- a/freqtrade/tests/exchange/test_exchange.py +++ b/freqtrade/tests/exchange/test_exchange.py @@ -124,14 +124,14 @@ def test_exchange_resolver(default_conf, mocker, caplog): caplog.record_tuples) caplog.clear() - exchange = ExchangeResolver('Kraken', default_conf).exchange + exchange = ExchangeResolver('kraken', default_conf).exchange assert isinstance(exchange, Exchange) assert isinstance(exchange, Kraken) assert not isinstance(exchange, Binance) assert not log_has_re(r"No .* specific subclass found. Using the generic class instead.", caplog.record_tuples) - exchange = ExchangeResolver('Binance', default_conf).exchange + exchange = ExchangeResolver('binance', default_conf).exchange assert isinstance(exchange, Exchange) assert isinstance(exchange, Binance) assert not isinstance(exchange, Kraken) diff --git a/freqtrade/tests/optimize/__init__.py b/freqtrade/tests/optimize/__init__.py index 457113cb7..41500051f 100644 --- a/freqtrade/tests/optimize/__init__.py +++ b/freqtrade/tests/optimize/__init__.py @@ -29,6 +29,10 @@ class BTContainer(NamedTuple): trades: List[BTrade] profit_perc: float trailing_stop: bool = False + trailing_only_offset_is_reached: bool = False + trailing_stop_positive: float = None + trailing_stop_positive_offset: float = 0.0 + use_sell_signal: bool = False def _get_frame_time_from_offset(offset): diff --git a/freqtrade/tests/optimize/test_backtest_detail.py b/freqtrade/tests/optimize/test_backtest_detail.py index 32c6bd09b..402e22391 100644 --- a/freqtrade/tests/optimize/test_backtest_detail.py +++ b/freqtrade/tests/optimize/test_backtest_detail.py @@ -14,6 +14,21 @@ from freqtrade.tests.optimize import (BTContainer, BTrade, _get_frame_time_from_offset, tests_ticker_interval) +# Test 0 Sell signal sell +# Test with Stop-loss at 1% +# TC0: Sell signal in candle 3 +tc0 = BTContainer(data=[ + # D O H L C V B S + [0, 5000, 5025, 4975, 4987, 6172, 1, 0], + [1, 5000, 5025, 4975, 4987, 6172, 0, 0], # enter trade (signal on last candle) + [2, 4987, 5012, 4986, 4600, 6172, 0, 0], # exit with stoploss hit + [3, 5010, 5000, 4980, 5010, 6172, 0, 1], + [4, 5010, 4987, 4977, 4995, 6172, 0, 0], + [5, 4995, 4995, 4995, 4950, 6172, 0, 0]], + stop_loss=-0.01, roi=1, profit_perc=0.002, use_sell_signal=True, + trades=[BTrade(sell_reason=SellType.SELL_SIGNAL, open_tick=1, close_tick=4)] +) + # Test 1 Minus 8% Close # Test with Stop-loss at 1% # TC1: Stop-Loss Triggered 1% loss @@ -146,7 +161,7 @@ tc8 = BTContainer(data=[ # Test 9 - trailing_stop should raise - high and low in same candle. # Candle Data for test 9 # Set stop-loss at 10%, ROI at 10% (should not apply) -# TC9: Trailing stoploss - stoploss should be adjusted candle 2 +# TC9: Trailing stoploss - stoploss should be adjusted candle 3 tc9 = BTContainer(data=[ # D O H L C V B S [0, 5000, 5050, 4950, 5000, 6172, 1, 0], @@ -158,7 +173,59 @@ tc9 = BTContainer(data=[ trades=[BTrade(sell_reason=SellType.TRAILING_STOP_LOSS, open_tick=1, close_tick=3)] ) +# Test 10 - trailing_stop should raise so candle 3 causes a stoploss +# without applying trailing_stop_positive since stoploss_offset is at 10%. +# Set stop-loss at 10%, ROI at 10% (should not apply) +# TC10: Trailing stoploss - stoploss should be adjusted candle 2 +tc10 = BTContainer(data=[ + # D O H L C V B S + [0, 5000, 5050, 4950, 5000, 6172, 1, 0], + [1, 5000, 5050, 4950, 5100, 6172, 0, 0], + [2, 5100, 5251, 5100, 5100, 6172, 0, 0], + [3, 4850, 5050, 4650, 4750, 6172, 0, 0], + [4, 4750, 4950, 4350, 4750, 6172, 0, 0]], + stop_loss=-0.10, roi=0.10, profit_perc=-0.1, trailing_stop=True, + trailing_only_offset_is_reached=True, trailing_stop_positive_offset=0.10, + trailing_stop_positive=0.03, + trades=[BTrade(sell_reason=SellType.STOP_LOSS, open_tick=1, close_tick=4)] +) + +# Test 11 - trailing_stop should raise so candle 3 causes a stoploss +# applying a positive trailing stop of 3% since stop_positive_offset is reached. +# Set stop-loss at 10%, ROI at 10% (should not apply) +# TC11: Trailing stoploss - stoploss should be adjusted candle 2, +tc11 = BTContainer(data=[ + # D O H L C V B S + [0, 5000, 5050, 4950, 5000, 6172, 1, 0], + [1, 5000, 5050, 4950, 5100, 6172, 0, 0], + [2, 5100, 5251, 5100, 5100, 6172, 0, 0], + [3, 4850, 5050, 4650, 4750, 6172, 0, 0], + [4, 4750, 4950, 4350, 4750, 6172, 0, 0]], + stop_loss=-0.10, roi=0.10, profit_perc=0.019, trailing_stop=True, + trailing_only_offset_is_reached=True, trailing_stop_positive_offset=0.05, + trailing_stop_positive=0.03, + trades=[BTrade(sell_reason=SellType.TRAILING_STOP_LOSS, open_tick=1, close_tick=3)] +) + +# Test 12 - trailing_stop should raise in candle 2 and cause a stoploss in the same candle +# applying a positive trailing stop of 3% since stop_positive_offset is reached. +# Set stop-loss at 10%, ROI at 10% (should not apply) +# TC12: Trailing stoploss - stoploss should be adjusted candle 2, +tc12 = BTContainer(data=[ + # D O H L C V B S + [0, 5000, 5050, 4950, 5000, 6172, 1, 0], + [1, 5000, 5050, 4950, 5100, 6172, 0, 0], + [2, 5100, 5251, 4650, 5100, 6172, 0, 0], + [3, 4850, 5050, 4650, 4750, 6172, 0, 0], + [4, 4750, 4950, 4350, 4750, 6172, 0, 0]], + stop_loss=-0.10, roi=0.10, profit_perc=0.019, trailing_stop=True, + trailing_only_offset_is_reached=True, trailing_stop_positive_offset=0.05, + trailing_stop_positive=0.03, + trades=[BTrade(sell_reason=SellType.TRAILING_STOP_LOSS, open_tick=1, close_tick=2)] +) + TESTS = [ + tc0, tc1, tc2, tc3, @@ -168,6 +235,9 @@ TESTS = [ tc7, tc8, tc9, + tc10, + tc11, + tc12, ] @@ -180,6 +250,13 @@ def test_backtest_results(default_conf, fee, mocker, caplog, data) -> None: default_conf["minimal_roi"] = {"0": data.roi} default_conf["ticker_interval"] = tests_ticker_interval default_conf["trailing_stop"] = data.trailing_stop + default_conf["trailing_only_offset_is_reached"] = data.trailing_only_offset_is_reached + # Only add this to configuration If it's necessary + if data.trailing_stop_positive: + default_conf["trailing_stop_positive"] = data.trailing_stop_positive + default_conf["trailing_stop_positive_offset"] = data.trailing_stop_positive_offset + default_conf["experimental"] = {"use_sell_signal": data.use_sell_signal} + mocker.patch("freqtrade.exchange.Exchange.get_fee", MagicMock(return_value=0.0)) patch_exchange(mocker) frame = _build_backtest_dataframe(data.data) diff --git a/freqtrade/tests/optimize/test_backtesting.py b/freqtrade/tests/optimize/test_backtesting.py index 98e8808a8..28568f20c 100644 --- a/freqtrade/tests/optimize/test_backtesting.py +++ b/freqtrade/tests/optimize/test_backtesting.py @@ -3,7 +3,6 @@ import json import math import random -from typing import List from unittest.mock import MagicMock import numpy as np @@ -12,7 +11,7 @@ import pytest from arrow import Arrow from freqtrade import DependencyException, constants -from freqtrade.arguments import Arguments, TimeRange +from freqtrade.arguments import TimeRange from freqtrade.data import history from freqtrade.data.btanalysis import evaluate_result_multi from freqtrade.data.converter import parse_ticker_dataframe @@ -23,11 +22,7 @@ from freqtrade.optimize.backtesting import Backtesting from freqtrade.state import RunMode from freqtrade.strategy.default_strategy import DefaultStrategy from freqtrade.strategy.interface import SellType -from freqtrade.tests.conftest import log_has, log_has_re, patch_exchange - - -def get_args(args) -> List[str]: - return Arguments(args, '').get_parsed_arg() +from freqtrade.tests.conftest import get_args, log_has, log_has_re, patch_exchange def trim_dictlist(dict_list, num): diff --git a/freqtrade/tests/optimize/test_edge_cli.py b/freqtrade/tests/optimize/test_edge_cli.py index 5d16b0f2d..6b527543f 100644 --- a/freqtrade/tests/optimize/test_edge_cli.py +++ b/freqtrade/tests/optimize/test_edge_cli.py @@ -2,19 +2,13 @@ # pragma pylint: disable=protected-access, too-many-lines, invalid-name, too-many-arguments import json -from typing import List from unittest.mock import MagicMock -from freqtrade.arguments import Arguments from freqtrade.edge import PairInfo -from freqtrade.optimize import start_edge, setup_configuration +from freqtrade.optimize import setup_configuration, start_edge from freqtrade.optimize.edge_cli import EdgeCli from freqtrade.state import RunMode -from freqtrade.tests.conftest import log_has, log_has_re, patch_exchange - - -def get_args(args) -> List[str]: - return Arguments(args, '').get_parsed_arg() +from freqtrade.tests.conftest import get_args, log_has, log_has_re, patch_exchange def test_setup_configuration_without_arguments(mocker, default_conf, caplog) -> None: @@ -117,8 +111,10 @@ def test_start(mocker, fee, edge_conf, caplog) -> None: def test_edge_init(mocker, edge_conf) -> None: patch_exchange(mocker) + edge_conf['stake_amount'] = 20 edge_cli = EdgeCli(edge_conf) assert edge_cli.config == edge_conf + assert edge_cli.config['stake_amount'] == 'unlimited' assert callable(edge_cli.edge.calculate) diff --git a/freqtrade/tests/optimize/test_hyperopt.py b/freqtrade/tests/optimize/test_hyperopt.py index 2c601e0fa..c3d6d0076 100644 --- a/freqtrade/tests/optimize/test_hyperopt.py +++ b/freqtrade/tests/optimize/test_hyperopt.py @@ -16,8 +16,7 @@ from freqtrade.optimize.hyperopt import Hyperopt, HYPEROPT_LOCKFILE from freqtrade.optimize import setup_configuration, start_hyperopt from freqtrade.resolvers.hyperopt_resolver import HyperOptResolver from freqtrade.state import RunMode -from freqtrade.tests.conftest import log_has, log_has_re, patch_exchange -from freqtrade.tests.optimize.test_backtesting import get_args +from freqtrade.tests.conftest import get_args, log_has, log_has_re, patch_exchange @pytest.fixture(scope='function') @@ -168,6 +167,7 @@ def test_hyperoptresolver(mocker, default_conf, caplog) -> None: "Using populate_sell_trend from DefaultStrategy.", caplog.record_tuples) assert log_has("Custom Hyperopt does not provide populate_buy_trend. " "Using populate_buy_trend from DefaultStrategy.", caplog.record_tuples) + assert hasattr(x, "ticker_interval") def test_start(mocker, default_conf, caplog) -> None: diff --git a/freqtrade/tests/rpc/test_rpc_telegram.py b/freqtrade/tests/rpc/test_rpc_telegram.py index 46ef15f56..b34e214af 100644 --- a/freqtrade/tests/rpc/test_rpc_telegram.py +++ b/freqtrade/tests/rpc/test_rpc_telegram.py @@ -756,6 +756,7 @@ def test_forcesell_handle(default_conf, update, ticker, fee, 'gain': 'profit', 'limit': 1.172e-05, 'amount': 90.99181073703367, + 'order_type': 'limit', 'open_rate': 1.099e-05, 'current_rate': 1.172e-05, 'profit_amount': 6.126e-05, @@ -810,6 +811,7 @@ def test_forcesell_down_handle(default_conf, update, ticker, fee, 'gain': 'loss', 'limit': 1.044e-05, 'amount': 90.99181073703367, + 'order_type': 'limit', 'open_rate': 1.099e-05, 'current_rate': 1.044e-05, 'profit_amount': -5.492e-05, @@ -855,6 +857,7 @@ def test_forcesell_all_handle(default_conf, update, ticker, fee, markets, mocker 'gain': 'loss', 'limit': 1.098e-05, 'amount': 90.99181073703367, + 'order_type': 'limit', 'open_rate': 1.099e-05, 'current_rate': 1.098e-05, 'profit_amount': -5.91e-06, @@ -1188,6 +1191,7 @@ def test_send_msg_buy_notification(default_conf, mocker) -> None: 'exchange': 'Bittrex', 'pair': 'ETH/BTC', 'limit': 1.099e-05, + 'order_type': 'limit', 'stake_amount': 0.001, 'stake_amount_fiat': 0.0, 'stake_currency': 'BTC', @@ -1195,7 +1199,7 @@ def test_send_msg_buy_notification(default_conf, mocker) -> None: }) assert msg_mock.call_args[0][0] \ == '*Bittrex:* Buying ETH/BTC\n' \ - 'with limit `0.00001099\n' \ + 'at rate `0.00001099\n' \ '(0.001000 BTC,0.000 USD)`' @@ -1217,6 +1221,7 @@ def test_send_msg_sell_notification(default_conf, mocker) -> None: 'gain': 'loss', 'limit': 3.201e-05, 'amount': 1333.3333333333335, + 'order_type': 'market', 'open_rate': 7.5e-05, 'current_rate': 3.201e-05, 'profit_amount': -0.05746268, @@ -1227,7 +1232,7 @@ def test_send_msg_sell_notification(default_conf, mocker) -> None: }) assert msg_mock.call_args[0][0] \ == ('*Binance:* Selling KEY/ETH\n' - '*Limit:* `0.00003201`\n' + '*Rate:* `0.00003201`\n' '*Amount:* `1333.33333333`\n' '*Open Rate:* `0.00007500`\n' '*Current Rate:* `0.00003201`\n' @@ -1242,6 +1247,7 @@ def test_send_msg_sell_notification(default_conf, mocker) -> None: 'gain': 'loss', 'limit': 3.201e-05, 'amount': 1333.3333333333335, + 'order_type': 'market', 'open_rate': 7.5e-05, 'current_rate': 3.201e-05, 'profit_amount': -0.05746268, @@ -1251,7 +1257,7 @@ def test_send_msg_sell_notification(default_conf, mocker) -> None: }) assert msg_mock.call_args[0][0] \ == ('*Binance:* Selling KEY/ETH\n' - '*Limit:* `0.00003201`\n' + '*Rate:* `0.00003201`\n' '*Amount:* `1333.33333333`\n' '*Open Rate:* `0.00007500`\n' '*Current Rate:* `0.00003201`\n' @@ -1339,6 +1345,7 @@ def test_send_msg_buy_notification_no_fiat(default_conf, mocker) -> None: 'exchange': 'Bittrex', 'pair': 'ETH/BTC', 'limit': 1.099e-05, + 'order_type': 'limit', 'stake_amount': 0.001, 'stake_amount_fiat': 0.0, 'stake_currency': 'BTC', @@ -1346,7 +1353,7 @@ def test_send_msg_buy_notification_no_fiat(default_conf, mocker) -> None: }) assert msg_mock.call_args[0][0] \ == '*Bittrex:* Buying ETH/BTC\n' \ - 'with limit `0.00001099\n' \ + 'at rate `0.00001099\n' \ '(0.001000 BTC)`' @@ -1367,6 +1374,7 @@ def test_send_msg_sell_notification_no_fiat(default_conf, mocker) -> None: 'gain': 'loss', 'limit': 3.201e-05, 'amount': 1333.3333333333335, + 'order_type': 'limit', 'open_rate': 7.5e-05, 'current_rate': 3.201e-05, 'profit_amount': -0.05746268, @@ -1377,7 +1385,7 @@ def test_send_msg_sell_notification_no_fiat(default_conf, mocker) -> None: }) assert msg_mock.call_args[0][0] \ == '*Binance:* Selling KEY/ETH\n' \ - '*Limit:* `0.00003201`\n' \ + '*Rate:* `0.00003201`\n' \ '*Amount:* `1333.33333333`\n' \ '*Open Rate:* `0.00007500`\n' \ '*Current Rate:* `0.00003201`\n' \ diff --git a/freqtrade/tests/rpc/test_rpc_webhook.py b/freqtrade/tests/rpc/test_rpc_webhook.py index da7aec0a6..a2dcd9b31 100644 --- a/freqtrade/tests/rpc/test_rpc_webhook.py +++ b/freqtrade/tests/rpc/test_rpc_webhook.py @@ -74,6 +74,7 @@ def test_send_msg(default_conf, mocker): 'gain': "profit", 'limit': 0.005, 'amount': 0.8, + 'order_type': 'limit', 'open_rate': 0.004, 'current_rate': 0.005, 'profit_amount': 0.001, @@ -126,6 +127,7 @@ def test_exception_send_msg(default_conf, mocker, caplog): 'exchange': 'Bittrex', 'pair': 'ETH/BTC', 'limit': 0.005, + 'order_type': 'limit', 'stake_amount': 0.8, 'stake_amount_fiat': 500, 'stake_currency': 'BTC', diff --git a/freqtrade/tests/strategy/test_strategy.py b/freqtrade/tests/strategy/test_strategy.py index 2ed2567f9..15d1c18ef 100644 --- a/freqtrade/tests/strategy/test_strategy.py +++ b/freqtrade/tests/strategy/test_strategy.py @@ -63,27 +63,22 @@ def test_search_strategy(): def test_load_strategy(result): resolver = StrategyResolver({'strategy': 'TestStrategy'}) - metadata = {'pair': 'ETH/BTC'} - assert 'adx' in resolver.strategy.advise_indicators(result, metadata=metadata) + assert 'adx' in resolver.strategy.advise_indicators(result, {'pair': 'ETH/BTC'}) def test_load_strategy_byte64(result): with open("freqtrade/tests/strategy/test_strategy.py", "r") as file: encoded_string = urlsafe_b64encode(file.read().encode("utf-8")).decode("utf-8") resolver = StrategyResolver({'strategy': 'TestStrategy:{}'.format(encoded_string)}) - assert 'adx' in resolver.strategy.advise_indicators(result, 'ETH/BTC') + assert 'adx' in resolver.strategy.advise_indicators(result, {'pair': 'ETH/BTC'}) def test_load_strategy_invalid_directory(result, caplog): resolver = StrategyResolver() - extra_dir = path.join('some', 'path') + extra_dir = Path.cwd() / 'some/path' resolver._load_strategy('TestStrategy', config={}, extra_dir=extra_dir) - assert ( - 'freqtrade.resolvers.strategy_resolver', - logging.WARNING, - 'Path "{}" does not exist'.format(extra_dir), - ) in caplog.record_tuples + assert log_has_re(r'Path .*' + r'some.*path.*' + r'.* does not exist', caplog.record_tuples) assert 'adx' in resolver.strategy.advise_indicators(result, {'pair': 'ETH/BTC'}) @@ -371,7 +366,7 @@ def test_deprecate_populate_indicators(result): with warnings.catch_warnings(record=True) as w: # Cause all warnings to always be triggered. warnings.simplefilter("always") - indicators = resolver.strategy.advise_indicators(result, 'ETH/BTC') + indicators = resolver.strategy.advise_indicators(result, {'pair': 'ETH/BTC'}) assert len(w) == 1 assert issubclass(w[-1].category, DeprecationWarning) assert "deprecated - check out the Sample strategy to see the current function headers!" \ @@ -380,7 +375,7 @@ def test_deprecate_populate_indicators(result): with warnings.catch_warnings(record=True) as w: # Cause all warnings to always be triggered. warnings.simplefilter("always") - resolver.strategy.advise_buy(indicators, 'ETH/BTC') + resolver.strategy.advise_buy(indicators, {'pair': 'ETH/BTC'}) assert len(w) == 1 assert issubclass(w[-1].category, DeprecationWarning) assert "deprecated - check out the Sample strategy to see the current function headers!" \ @@ -389,7 +384,7 @@ def test_deprecate_populate_indicators(result): with warnings.catch_warnings(record=True) as w: # Cause all warnings to always be triggered. warnings.simplefilter("always") - resolver.strategy.advise_sell(indicators, 'ETH_BTC') + resolver.strategy.advise_sell(indicators, {'pair': 'ETH_BTC'}) assert len(w) == 1 assert issubclass(w[-1].category, DeprecationWarning) assert "deprecated - check out the Sample strategy to see the current function headers!" \ diff --git a/freqtrade/tests/test_arguments.py b/freqtrade/tests/test_arguments.py index ecd108b5e..d9292bdb5 100644 --- a/freqtrade/tests/test_arguments.py +++ b/freqtrade/tests/test_arguments.py @@ -47,9 +47,9 @@ def test_parse_args_verbose() -> None: assert args.loglevel == 1 -def test_scripts_options() -> None: +def test_common_scripts_options() -> None: arguments = Arguments(['-p', 'ETH/BTC'], '') - arguments.scripts_options() + arguments.common_scripts_options() args = arguments.get_parsed_arg() assert args.pairs == 'ETH/BTC' @@ -170,22 +170,40 @@ def test_parse_args_hyperopt_custom() -> None: assert call_args.func is not None -def test_testdata_dl_options() -> None: +def test_download_data_options() -> None: args = [ '--pairs-file', 'file_with_pairs', - '--export', 'export/folder', + '--datadir', 'datadir/folder', '--days', '30', '--exchange', 'binance' ] arguments = Arguments(args, '') - arguments.testdata_dl_options() + arguments.common_options() + arguments.download_data_options() args = arguments.parse_args() assert args.pairs_file == 'file_with_pairs' - assert args.export == 'export/folder' + assert args.datadir == 'datadir/folder' assert args.days == 30 assert args.exchange == 'binance' +def test_plot_dataframe_options() -> None: + args = [ + '--indicators1', 'sma10,sma100', + '--indicators2', 'macd,fastd,fastk', + '--plot-limit', '30', + '-p', 'UNITTEST/BTC', + ] + arguments = Arguments(args, '') + arguments.common_scripts_options() + arguments.plot_dataframe_options() + pargs = arguments.parse_args(True) + assert pargs.indicators1 == "sma10,sma100" + assert pargs.indicators2 == "macd,fastd,fastk" + assert pargs.plot_limit == 30 + assert pargs.pairs == "UNITTEST/BTC" + + def test_check_int_positive() -> None: assert Arguments.check_int_positive("3") == 3 diff --git a/freqtrade/tests/test_configuration.py b/freqtrade/tests/test_configuration.py index aee0dfadd..38f17fbea 100644 --- a/freqtrade/tests/test_configuration.py +++ b/freqtrade/tests/test_configuration.py @@ -15,7 +15,7 @@ from freqtrade.arguments import Arguments from freqtrade.configuration import Configuration, set_loggers from freqtrade.constants import DEFAULT_DB_DRYRUN_URL, DEFAULT_DB_PROD_URL from freqtrade.state import RunMode -from freqtrade.tests.conftest import log_has +from freqtrade.tests.conftest import log_has, log_has_re @pytest.fixture(scope="function") @@ -470,21 +470,52 @@ def test_hyperopt_with_arguments(mocker, default_conf, caplog) -> None: def test_check_exchange(default_conf, caplog) -> None: configuration = Configuration(Namespace()) - # Test a valid exchange + # Test an officially supported by Freqtrade team exchange default_conf.get('exchange').update({'name': 'BITTREX'}) assert configuration.check_exchange(default_conf) + assert log_has_re(r"Exchange .* is officially supported by the Freqtrade development team\.", + caplog.record_tuples) + caplog.clear() - # Test a valid exchange + # Test an officially supported by Freqtrade team exchange default_conf.get('exchange').update({'name': 'binance'}) assert configuration.check_exchange(default_conf) + assert log_has_re(r"Exchange .* is officially supported by the Freqtrade development team\.", + caplog.record_tuples) + caplog.clear() - # Test a invalid exchange + # Test an available exchange, supported by ccxt + default_conf.get('exchange').update({'name': 'kraken'}) + assert configuration.check_exchange(default_conf) + assert log_has_re(r"Exchange .* is supported by ccxt and .* not officially supported " + r"by the Freqtrade development team\. .*", + caplog.record_tuples) + caplog.clear() + + # Test a 'bad' exchange, which known to have serious problems + default_conf.get('exchange').update({'name': 'bitmex'}) + assert not configuration.check_exchange(default_conf) + assert log_has_re(r"Exchange .* is known to not work with the bot yet\. " + r"Use it only for development and testing purposes\.", + caplog.record_tuples) + caplog.clear() + + # Test a 'bad' exchange with check_for_bad=False + default_conf.get('exchange').update({'name': 'bitmex'}) + assert configuration.check_exchange(default_conf, False) + assert log_has_re(r"Exchange .* is supported by ccxt and .* not officially supported " + r"by the Freqtrade development team\. .*", + caplog.record_tuples) + caplog.clear() + + # Test an invalid exchange default_conf.get('exchange').update({'name': 'unknown_exchange'}) configuration.config = default_conf with pytest.raises( OperationalException, - match=r'.*Exchange "unknown_exchange" not supported.*' + match=r'.*Exchange "unknown_exchange" is not supported by ccxt ' + r'and therefore not available for the bot.*' ): configuration.check_exchange(default_conf) diff --git a/freqtrade/tests/test_freqtradebot.py b/freqtrade/tests/test_freqtradebot.py index 6566e4036..87b344853 100644 --- a/freqtrade/tests/test_freqtradebot.py +++ b/freqtrade/tests/test_freqtradebot.py @@ -19,47 +19,13 @@ from freqtrade.persistence import Trade from freqtrade.rpc import RPCMessageType from freqtrade.state import State from freqtrade.strategy.interface import SellCheckTuple, SellType -from freqtrade.tests.conftest import (log_has, log_has_re, patch_edge, - patch_exchange, patch_get_signal, - patch_wallet) +from freqtrade.tests.conftest import (get_patched_freqtradebot, + get_patched_worker, log_has, log_has_re, + patch_edge, patch_exchange, + patch_get_signal, patch_wallet) from freqtrade.worker import Worker -# Functions for recurrent object patching -def patch_freqtradebot(mocker, config) -> None: - """ - This function patches _init_modules() to not call dependencies - :param mocker: a Mocker object to apply patches - :param config: Config to pass to the bot - :return: None - """ - mocker.patch('freqtrade.freqtradebot.RPCManager', MagicMock()) - mocker.patch('freqtrade.freqtradebot.persistence.init', MagicMock()) - patch_exchange(mocker) - - -def get_patched_freqtradebot(mocker, config) -> FreqtradeBot: - """ - This function patches _init_modules() to not call dependencies - :param mocker: a Mocker object to apply patches - :param config: Config to pass to the bot - :return: FreqtradeBot - """ - patch_freqtradebot(mocker, config) - return FreqtradeBot(config) - - -def get_patched_worker(mocker, config) -> Worker: - """ - This function patches _init_modules() to not call dependencies - :param mocker: a Mocker object to apply patches - :param config: Config to pass to the bot - :return: Worker - """ - patch_freqtradebot(mocker, config) - return Worker(args=None, config=config) - - def patch_RPCManager(mocker) -> MagicMock: """ This function mock RPC manager to avoid repeating this code in almost every tests @@ -1176,6 +1142,77 @@ def test_handle_stoploss_on_exchange_trailing(mocker, default_conf, fee, caplog, stop_price=0.00002344 * 0.95) +def test_handle_stoploss_on_exchange_trailing_error(mocker, default_conf, fee, caplog, + markets, limit_buy_order, + limit_sell_order) -> None: + # When trailing stoploss is set + stoploss_limit = MagicMock(return_value={'id': 13434334}) + patch_exchange(mocker) + + mocker.patch.multiple( + 'freqtrade.exchange.Exchange', + get_ticker=MagicMock(return_value={ + 'bid': 0.00001172, + 'ask': 0.00001173, + 'last': 0.00001172 + }), + buy=MagicMock(return_value={'id': limit_buy_order['id']}), + sell=MagicMock(return_value={'id': limit_sell_order['id']}), + get_fee=fee, + markets=PropertyMock(return_value=markets), + stoploss_limit=stoploss_limit + ) + + # enabling TSL + default_conf['trailing_stop'] = True + + freqtrade = get_patched_freqtradebot(mocker, default_conf) + # enabling stoploss on exchange + freqtrade.strategy.order_types['stoploss_on_exchange'] = True + + # setting stoploss + freqtrade.strategy.stoploss = -0.05 + + # setting stoploss_on_exchange_interval to 60 seconds + freqtrade.strategy.order_types['stoploss_on_exchange_interval'] = 60 + patch_get_signal(freqtrade) + freqtrade.create_trade() + trade = Trade.query.first() + trade.is_open = True + trade.open_order_id = None + trade.stoploss_order_id = "abcd" + trade.stop_loss = 0.2 + trade.stoploss_last_update = arrow.utcnow().shift(minutes=-601).datetime.replace(tzinfo=None) + + stoploss_order_hanging = { + 'id': "abcd", + 'status': 'open', + 'type': 'stop_loss_limit', + 'price': 3, + 'average': 2, + 'info': { + 'stopPrice': '0.1' + } + } + mocker.patch('freqtrade.exchange.Exchange.cancel_order', side_effect=InvalidOrderException()) + mocker.patch('freqtrade.exchange.Exchange.get_order', stoploss_order_hanging) + freqtrade.handle_trailing_stoploss_on_exchange(trade, stoploss_order_hanging) + assert log_has_re(r"Could not cancel stoploss order abcd for pair ETH/BTC.*", + caplog.record_tuples) + + # Still try to create order + assert stoploss_limit.call_count == 1 + + # Fail creating stoploss order + caplog.clear() + cancel_mock = mocker.patch("freqtrade.exchange.Exchange.cancel_order", MagicMock()) + mocker.patch("freqtrade.exchange.Exchange.stoploss_limit", side_effect=DependencyException()) + freqtrade.handle_trailing_stoploss_on_exchange(trade, stoploss_order_hanging) + assert cancel_mock.call_count == 1 + assert log_has_re(r"Could create trailing stoploss order for pair ETH/BTC\..*", + caplog.record_tuples) + + def test_tsl_on_exchange_compatible_with_edge(mocker, edge_conf, fee, caplog, markets, limit_buy_order, limit_sell_order) -> None: @@ -1994,6 +2031,7 @@ def test_execute_sell_up(default_conf, ticker, fee, ticker_sell_up, markets, moc 'gain': 'profit', 'limit': 1.172e-05, 'amount': 90.99181073703367, + 'order_type': 'limit', 'open_rate': 1.099e-05, 'current_rate': 1.172e-05, 'profit_amount': 6.126e-05, @@ -2040,6 +2078,7 @@ def test_execute_sell_down(default_conf, ticker, fee, ticker_sell_down, markets, 'gain': 'loss', 'limit': 1.044e-05, 'amount': 90.99181073703367, + 'order_type': 'limit', 'open_rate': 1.099e-05, 'current_rate': 1.044e-05, 'profit_amount': -5.492e-05, @@ -2094,6 +2133,7 @@ def test_execute_sell_down_stoploss_on_exchange_dry_run(default_conf, ticker, fe 'gain': 'loss', 'limit': 1.08801e-05, 'amount': 90.99181073703367, + 'order_type': 'limit', 'open_rate': 1.099e-05, 'current_rate': 1.044e-05, 'profit_amount': -1.498e-05, @@ -2105,6 +2145,36 @@ def test_execute_sell_down_stoploss_on_exchange_dry_run(default_conf, ticker, fe } == last_msg +def test_execute_sell_sloe_cancel_exception(mocker, default_conf, ticker, fee, + markets, caplog) -> None: + freqtrade = get_patched_freqtradebot(mocker, default_conf) + mocker.patch('freqtrade.exchange.Exchange.cancel_order', side_effect=InvalidOrderException()) + sellmock = MagicMock() + mocker.patch.multiple( + 'freqtrade.exchange.Exchange', + _load_markets=MagicMock(return_value={}), + get_ticker=ticker, + get_fee=fee, + markets=PropertyMock(return_value=markets), + sell=sellmock + ) + + freqtrade.strategy.order_types['stoploss_on_exchange'] = True + patch_get_signal(freqtrade) + freqtrade.create_trade() + + trade = Trade.query.first() + Trade.session = MagicMock() + + freqtrade.config['dry_run'] = False + trade.stoploss_order_id = "abcd" + + freqtrade.execute_sell(trade=trade, limit=1234, + sell_reason=SellType.STOP_LOSS) + assert sellmock.call_count == 1 + assert log_has('Could not cancel stoploss order abcd', caplog.record_tuples) + + def test_execute_sell_with_stoploss_on_exchange(default_conf, ticker, fee, ticker_sell_up, markets, mocker) -> None: @@ -2265,6 +2335,7 @@ def test_execute_sell_without_conf_sell_up(default_conf, ticker, fee, 'gain': 'profit', 'limit': 1.172e-05, 'amount': 90.99181073703367, + 'order_type': 'limit', 'open_rate': 1.099e-05, 'current_rate': 1.172e-05, 'profit_amount': 6.126e-05, @@ -2312,6 +2383,7 @@ def test_execute_sell_without_conf_sell_down(default_conf, ticker, fee, 'gain': 'loss', 'limit': 1.044e-05, 'amount': 90.99181073703367, + 'order_type': 'limit', 'open_rate': 1.099e-05, 'current_rate': 1.044e-05, 'profit_amount': -5.492e-05, diff --git a/freqtrade/tests/test_persistence.py b/freqtrade/tests/test_persistence.py index bb00fa8f4..32425ef7b 100644 --- a/freqtrade/tests/test_persistence.py +++ b/freqtrade/tests/test_persistence.py @@ -11,9 +11,48 @@ from freqtrade.persistence import Trade, clean_dry_run_db, init from freqtrade.tests.conftest import log_has -@pytest.fixture(scope='function') -def init_persistence(default_conf): - init(default_conf['db_url'], default_conf['dry_run']) +def create_mock_trades(fee): + """ + Create some fake trades ... + """ + # Simulate dry_run entries + trade = Trade( + pair='ETH/BTC', + stake_amount=0.001, + amount=123.0, + fee_open=fee.return_value, + fee_close=fee.return_value, + open_rate=0.123, + exchange='bittrex', + open_order_id='dry_run_buy_12345' + ) + Trade.session.add(trade) + + trade = Trade( + pair='ETC/BTC', + stake_amount=0.001, + amount=123.0, + fee_open=fee.return_value, + fee_close=fee.return_value, + open_rate=0.123, + exchange='bittrex', + is_open=False, + open_order_id='dry_run_sell_12345' + ) + Trade.session.add(trade) + + # Simulate prod entry + trade = Trade( + pair='ETC/BTC', + stake_amount=0.001, + amount=123.0, + fee_open=fee.return_value, + fee_close=fee.return_value, + open_rate=0.123, + exchange='bittrex', + open_order_id='prod_buy_12345' + ) + Trade.session.add(trade) def test_init_create_session(default_conf): @@ -671,45 +710,7 @@ def test_adjust_min_max_rates(fee): @pytest.mark.usefixtures("init_persistence") def test_get_open(default_conf, fee): - # Simulate dry_run entries - trade = Trade( - pair='ETH/BTC', - stake_amount=0.001, - amount=123.0, - fee_open=fee.return_value, - fee_close=fee.return_value, - open_rate=0.123, - exchange='bittrex', - open_order_id='dry_run_buy_12345' - ) - Trade.session.add(trade) - - trade = Trade( - pair='ETC/BTC', - stake_amount=0.001, - amount=123.0, - fee_open=fee.return_value, - fee_close=fee.return_value, - open_rate=0.123, - exchange='bittrex', - is_open=False, - open_order_id='dry_run_sell_12345' - ) - Trade.session.add(trade) - - # Simulate prod entry - trade = Trade( - pair='ETC/BTC', - stake_amount=0.001, - amount=123.0, - fee_open=fee.return_value, - fee_close=fee.return_value, - open_rate=0.123, - exchange='bittrex', - open_order_id='prod_buy_12345' - ) - Trade.session.add(trade) - + create_mock_trades(fee) assert len(Trade.get_open_trades()) == 2 diff --git a/freqtrade/tests/test_plotting.py b/freqtrade/tests/test_plotting.py new file mode 100644 index 000000000..15ab698d8 --- /dev/null +++ b/freqtrade/tests/test_plotting.py @@ -0,0 +1,188 @@ + +from unittest.mock import MagicMock + +from plotly import tools +import plotly.graph_objs as go +from copy import deepcopy + +from freqtrade.arguments import TimeRange +from freqtrade.data import history +from freqtrade.data.btanalysis import load_backtest_data +from freqtrade.plot.plotting import (generate_graph, generate_plot_file, + generate_row, plot_trades) +from freqtrade.strategy.default_strategy import DefaultStrategy +from freqtrade.tests.conftest import log_has, log_has_re + + +def fig_generating_mock(fig, *args, **kwargs): + """ Return Fig - used to mock generate_row and plot_trades""" + return fig + + +def find_trace_in_fig_data(data, search_string: str): + matches = filter(lambda x: x.name == search_string, data) + return next(matches) + + +def generage_empty_figure(): + return tools.make_subplots( + rows=3, + cols=1, + shared_xaxes=True, + row_width=[1, 1, 4], + vertical_spacing=0.0001, + ) + + +def test_generate_row(default_conf, caplog): + pair = "UNITTEST/BTC" + timerange = TimeRange(None, 'line', 0, -1000) + + data = history.load_pair_history(pair=pair, ticker_interval='1m', + datadir=None, timerange=timerange) + indicators1 = ["ema10"] + indicators2 = ["macd"] + + # Generate buy/sell signals and indicators + strat = DefaultStrategy(default_conf) + data = strat.analyze_ticker(data, {'pair': pair}) + fig = generage_empty_figure() + + # Row 1 + fig1 = generate_row(fig=deepcopy(fig), row=1, indicators=indicators1, data=data) + figure = fig1.layout.figure + ema10 = find_trace_in_fig_data(figure.data, "ema10") + assert isinstance(ema10, go.Scatter) + assert ema10.yaxis == "y" + + fig2 = generate_row(fig=deepcopy(fig), row=3, indicators=indicators2, data=data) + figure = fig2.layout.figure + macd = find_trace_in_fig_data(figure.data, "macd") + assert isinstance(macd, go.Scatter) + assert macd.yaxis == "y3" + + # No indicator found + fig3 = generate_row(fig=deepcopy(fig), row=3, indicators=['no_indicator'], data=data) + assert fig == fig3 + assert log_has_re(r'Indicator "no_indicator" ignored\..*', caplog.record_tuples) + + +def test_plot_trades(): + fig1 = generage_empty_figure() + # nothing happens when no trades are available + fig = plot_trades(fig1, None) + assert fig == fig1 + pair = "ADA/BTC" + filename = history.make_testdata_path(None) / "backtest-result_test.json" + trades = load_backtest_data(filename) + trades = trades.loc[trades['pair'] == pair] + + fig = plot_trades(fig, trades) + figure = fig1.layout.figure + + # Check buys - color, should be in first graph, ... + trade_buy = find_trace_in_fig_data(figure.data, "trade_buy") + assert isinstance(trade_buy, go.Scatter) + assert trade_buy.yaxis == 'y' + assert len(trades) == len(trade_buy.x) + assert trade_buy.marker.color == 'green' + + trade_sell = find_trace_in_fig_data(figure.data, "trade_sell") + assert isinstance(trade_sell, go.Scatter) + assert trade_sell.yaxis == 'y' + assert len(trades) == len(trade_sell.x) + assert trade_sell.marker.color == 'red' + + +def test_generate_graph_no_signals_no_trades(default_conf, mocker, caplog): + row_mock = mocker.patch('freqtrade.plot.plotting.generate_row', + MagicMock(side_effect=fig_generating_mock)) + trades_mock = mocker.patch('freqtrade.plot.plotting.plot_trades', + MagicMock(side_effect=fig_generating_mock)) + + pair = "UNITTEST/BTC" + timerange = TimeRange(None, 'line', 0, -1000) + data = history.load_pair_history(pair=pair, ticker_interval='1m', + datadir=None, timerange=timerange) + data['buy'] = 0 + data['sell'] = 0 + + indicators1 = [] + indicators2 = [] + fig = generate_graph(pair=pair, data=data, trades=None, + indicators1=indicators1, indicators2=indicators2) + assert isinstance(fig, go.Figure) + assert fig.layout.title.text == pair + figure = fig.layout.figure + + assert len(figure.data) == 2 + # Candlesticks are plotted first + candles = find_trace_in_fig_data(figure.data, "Price") + assert isinstance(candles, go.Candlestick) + + volume = find_trace_in_fig_data(figure.data, "Volume") + assert isinstance(volume, go.Bar) + + assert row_mock.call_count == 2 + assert trades_mock.call_count == 1 + + assert log_has("No buy-signals found.", caplog.record_tuples) + assert log_has("No sell-signals found.", caplog.record_tuples) + + +def test_generate_graph_no_trades(default_conf, mocker): + row_mock = mocker.patch('freqtrade.plot.plotting.generate_row', + MagicMock(side_effect=fig_generating_mock)) + trades_mock = mocker.patch('freqtrade.plot.plotting.plot_trades', + MagicMock(side_effect=fig_generating_mock)) + pair = 'UNITTEST/BTC' + timerange = TimeRange(None, 'line', 0, -1000) + data = history.load_pair_history(pair=pair, ticker_interval='1m', + datadir=None, timerange=timerange) + + # Generate buy/sell signals and indicators + strat = DefaultStrategy(default_conf) + data = strat.analyze_ticker(data, {'pair': pair}) + + indicators1 = [] + indicators2 = [] + fig = generate_graph(pair=pair, data=data, trades=None, + indicators1=indicators1, indicators2=indicators2) + assert isinstance(fig, go.Figure) + assert fig.layout.title.text == pair + figure = fig.layout.figure + + assert len(figure.data) == 6 + # Candlesticks are plotted first + candles = find_trace_in_fig_data(figure.data, "Price") + assert isinstance(candles, go.Candlestick) + + volume = find_trace_in_fig_data(figure.data, "Volume") + assert isinstance(volume, go.Bar) + + buy = find_trace_in_fig_data(figure.data, "buy") + assert isinstance(buy, go.Scatter) + # All buy-signals should be plotted + assert int(data.buy.sum()) == len(buy.x) + + sell = find_trace_in_fig_data(figure.data, "sell") + assert isinstance(sell, go.Scatter) + # All buy-signals should be plotted + assert int(data.sell.sum()) == len(sell.x) + + assert find_trace_in_fig_data(figure.data, "BB lower") + assert find_trace_in_fig_data(figure.data, "BB upper") + + assert row_mock.call_count == 2 + assert trades_mock.call_count == 1 + + +def test_generate_plot_file(mocker, caplog): + fig = generage_empty_figure() + plot_mock = mocker.patch("freqtrade.plot.plotting.plot", MagicMock()) + generate_plot_file(fig, "UNITTEST/BTC", "5m") + + assert plot_mock.call_count == 1 + assert plot_mock.call_args[0][0] == fig + assert (plot_mock.call_args_list[0][1]['filename'] + == "user_data/plots/freqtrade-plot-UNITTEST_BTC-5m.html") diff --git a/freqtrade/tests/test_utils.py b/freqtrade/tests/test_utils.py new file mode 100644 index 000000000..a12b709d7 --- /dev/null +++ b/freqtrade/tests/test_utils.py @@ -0,0 +1,42 @@ +from freqtrade.utils import setup_utils_configuration, start_list_exchanges +from freqtrade.tests.conftest import get_args +from freqtrade.state import RunMode + +import re + + +def test_setup_utils_configuration(): + args = [ + '--config', 'config.json.example', + ] + + config = setup_utils_configuration(get_args(args), RunMode.OTHER) + assert "exchange" in config + assert config['exchange']['dry_run'] is True + assert config['exchange']['key'] == '' + assert config['exchange']['secret'] == '' + + +def test_list_exchanges(capsys): + + args = [ + "list-exchanges", + ] + + start_list_exchanges(get_args(args)) + captured = capsys.readouterr() + assert re.match(r"Exchanges supported by ccxt and available.*", captured.out) + assert re.match(r".*binance,.*", captured.out) + assert re.match(r".*bittrex,.*", captured.out) + + # Test with --one-column + args = [ + "list-exchanges", + "--one-column", + ] + + start_list_exchanges(get_args(args)) + captured = capsys.readouterr() + assert not re.match(r"Exchanges supported by ccxt and available.*", captured.out) + assert re.search(r"^binance$", captured.out, re.MULTILINE) + assert re.search(r"^bittrex$", captured.out, re.MULTILINE) diff --git a/freqtrade/utils.py b/freqtrade/utils.py new file mode 100644 index 000000000..d550ef43c --- /dev/null +++ b/freqtrade/utils.py @@ -0,0 +1,41 @@ +import logging +from argparse import Namespace +from typing import Any, Dict + +from freqtrade.configuration import Configuration +from freqtrade.exchange import available_exchanges +from freqtrade.state import RunMode + + +logger = logging.getLogger(__name__) + + +def setup_utils_configuration(args: Namespace, method: RunMode) -> Dict[str, Any]: + """ + Prepare the configuration for utils subcommands + :param args: Cli args from Arguments() + :return: Configuration + """ + configuration = Configuration(args, method) + config = configuration.load_config() + + config['exchange']['dry_run'] = True + # Ensure we do not use Exchange credentials + config['exchange']['key'] = '' + config['exchange']['secret'] = '' + + return config + + +def start_list_exchanges(args: Namespace) -> None: + """ + Print available exchanges + :param args: Cli args from Arguments() + :return: None + """ + + if args.print_one_column: + print('\n'.join(available_exchanges())) + else: + print(f"Exchanges supported by ccxt and available for Freqtrade: " + f"{', '.join(available_exchanges())}") diff --git a/requirements-common.txt b/requirements-common.txt index 8b44e9d28..ed3f052a9 100644 --- a/requirements-common.txt +++ b/requirements-common.txt @@ -1,6 +1,6 @@ # requirements without requirements installable via conda # mainly used for Raspberry pi installs -ccxt==1.18.667 +ccxt==1.18.725 SQLAlchemy==1.3.4 python-telegram-bot==11.1.0 arrow==0.14.2 diff --git a/requirements-dev.txt b/requirements-dev.txt index 315033847..ce05b47f0 100644 --- a/requirements-dev.txt +++ b/requirements-dev.txt @@ -1,12 +1,13 @@ # Include all requirements to run the bot. -r requirements.txt +-r requirements-plot.txt flake8==3.7.7 flake8-type-annotations==0.1.0 flake8-tidy-imports==2.0.0 -pytest==4.6.2 +pytest==4.6.3 pytest-mock==1.10.4 pytest-asyncio==0.10.0 pytest-cov==2.7.1 -coveralls==1.8.0 +coveralls==1.8.1 mypy==0.701 diff --git a/scripts/download_backtest_data.py b/scripts/download_backtest_data.py index 42b305778..dd4627c14 100755 --- a/scripts/download_backtest_data.py +++ b/scripts/download_backtest_data.py @@ -1,55 +1,67 @@ #!/usr/bin/env python3 """ -This script generates json data +This script generates json files with pairs history data """ +import arrow import json import sys from pathlib import Path -import arrow -from typing import Any, Dict +from typing import Any, Dict, List -from freqtrade.arguments import Arguments -from freqtrade.arguments import TimeRange -from freqtrade.exchange import Exchange +from freqtrade.arguments import Arguments, TimeRange +from freqtrade.configuration import Configuration from freqtrade.data.history import download_pair_history -from freqtrade.configuration import Configuration, set_loggers +from freqtrade.exchange import Exchange from freqtrade.misc import deep_merge_dicts import logging -logging.basicConfig( - level=logging.INFO, - format='%(asctime)s - %(name)s - %(levelname)s - %(message)s', -) -set_loggers(0) + +logger = logging.getLogger('download_backtest_data') DEFAULT_DL_PATH = 'user_data/data' -arguments = Arguments(sys.argv[1:], 'download utility') -arguments.testdata_dl_options() -args = arguments.parse_args() +arguments = Arguments(sys.argv[1:], 'Download backtest data') +arguments.common_options() +arguments.download_data_options() -timeframes = args.timeframes +# Do not read the default config if config is not specified +# in the command line options explicitely +args = arguments.parse_args(no_default_config=True) + +# Use bittrex as default exchange +exchange_name = args.exchange or 'bittrex' + +pairs: List = [] + +configuration = Configuration(args) +config: Dict[str, Any] = {} if args.config: - configuration = Configuration(args) - - config: Dict[str, Any] = {} # Now expecting a list of config filenames here, not a string for path in args.config: - print(f"Using config: {path}...") + logger.info(f"Using config: {path}...") # Merge config options, overwriting old values config = deep_merge_dicts(configuration._load_config_file(path), config) config['stake_currency'] = '' # Ensure we do not use Exchange credentials + config['exchange']['dry_run'] = True config['exchange']['key'] = '' config['exchange']['secret'] = '' + + pairs = config['exchange']['pair_whitelist'] + + if config.get('ticker_interval'): + timeframes = args.timeframes or [config.get('ticker_interval')] + else: + timeframes = args.timeframes or ['1m', '5m'] + else: config = { 'stake_currency': '', 'dry_run': True, 'exchange': { - 'name': args.exchange, + 'name': exchange_name, 'key': '', 'secret': '', 'pair_whitelist': [], @@ -59,56 +71,72 @@ else: } } } + timeframes = args.timeframes or ['1m', '5m'] +configuration._load_logging_config(config) -dl_path = Path(DEFAULT_DL_PATH).joinpath(config['exchange']['name']) -if args.export: - dl_path = Path(args.export) +if args.config and args.exchange: + logger.warning("The --exchange option is ignored, " + "using exchange settings from the configuration file.") -if not dl_path.is_dir(): - sys.exit(f'Directory {dl_path} does not exist.') +# Check if the exchange set by the user is supported +configuration.check_exchange(config) + +configuration._load_datadir_config(config) + +dl_path = Path(config['datadir']) pairs_file = Path(args.pairs_file) if args.pairs_file else dl_path.joinpath('pairs.json') -if not pairs_file.exists(): - sys.exit(f'No pairs file found with path {pairs_file}.') -with pairs_file.open() as file: - PAIRS = list(set(json.load(file))) +if not pairs or args.pairs_file: + logger.info(f'Reading pairs file "{pairs_file}".') + # Download pairs from the pairs file if no config is specified + # or if pairs file is specified explicitely + if not pairs_file.exists(): + sys.exit(f'No pairs file found with path "{pairs_file}".') -PAIRS.sort() + with pairs_file.open() as file: + pairs = list(set(json.load(file))) + pairs.sort() timerange = TimeRange() if args.days: time_since = arrow.utcnow().shift(days=-args.days).strftime("%Y%m%d") timerange = arguments.parse_timerange(f'{time_since}-') +logger.info(f'About to download pairs: {pairs}, intervals: {timeframes} to {dl_path}') -print(f'About to download pairs: {PAIRS} to {dl_path}') - -# Init exchange -exchange = Exchange(config) pairs_not_available = [] -for pair in PAIRS: - if pair not in exchange._api.markets: - pairs_not_available.append(pair) - print(f"skipping pair {pair}") - continue - for ticker_interval in timeframes: - pair_print = pair.replace('/', '_') - filename = f'{pair_print}-{ticker_interval}.json' - dl_file = dl_path.joinpath(filename) - if args.erase and dl_file.exists(): - print(f'Deleting existing data for pair {pair}, interval {ticker_interval}') - dl_file.unlink() +try: + # Init exchange + exchange = Exchange(config) - print(f'downloading pair {pair}, interval {ticker_interval}') - download_pair_history(datadir=dl_path, exchange=exchange, - pair=pair, - ticker_interval=ticker_interval, - timerange=timerange) + for pair in pairs: + if pair not in exchange._api.markets: + pairs_not_available.append(pair) + logger.info(f"Skipping pair {pair}...") + continue + for ticker_interval in timeframes: + pair_print = pair.replace('/', '_') + filename = f'{pair_print}-{ticker_interval}.json' + dl_file = dl_path.joinpath(filename) + if args.erase and dl_file.exists(): + logger.info( + f'Deleting existing data for pair {pair}, interval {ticker_interval}.') + dl_file.unlink() + logger.info(f'Downloading pair {pair}, interval {ticker_interval}.') + download_pair_history(datadir=dl_path, exchange=exchange, + pair=pair, ticker_interval=str(ticker_interval), + timerange=timerange) -if pairs_not_available: - print(f"Pairs [{','.join(pairs_not_available)}] not availble.") +except KeyboardInterrupt: + sys.exit("SIGINT received, aborting ...") + +finally: + if pairs_not_available: + logger.info( + f"Pairs [{','.join(pairs_not_available)}] not available " + f"on exchange {config['exchange']['name']}.") diff --git a/scripts/plot_dataframe.py b/scripts/plot_dataframe.py index 4f8ffb32b..3792233de 100755 --- a/scripts/plot_dataframe.py +++ b/scripts/plot_dataframe.py @@ -26,141 +26,21 @@ Example of usage: """ import logging import sys -from argparse import Namespace from pathlib import Path from typing import Any, Dict, List import pandas as pd -import plotly.graph_objs as go -import pytz -from plotly import tools -from plotly.offline import plot -from freqtrade import persistence -from freqtrade.arguments import Arguments, TimeRange +from freqtrade.arguments import Arguments from freqtrade.data import history -from freqtrade.data.btanalysis import BT_DATA_COLUMNS, load_backtest_data -from freqtrade.exchange import Exchange +from freqtrade.data.btanalysis import load_trades, extract_trades_of_period from freqtrade.optimize import setup_configuration -from freqtrade.persistence import Trade -from freqtrade.resolvers import StrategyResolver +from freqtrade.plot.plotting import (generate_graph, + generate_plot_file) +from freqtrade.resolvers import ExchangeResolver, StrategyResolver from freqtrade.state import RunMode logger = logging.getLogger(__name__) -_CONF: Dict[str, Any] = {} - -timeZone = pytz.UTC - - -def load_trades(args: Namespace, pair: str, timerange: TimeRange) -> pd.DataFrame: - trades: pd.DataFrame = pd.DataFrame() - if args.db_url: - persistence.init(args.db_url, clean_open_orders=False) - - columns = ["pair", "profit", "open_time", "close_time", - "open_rate", "close_rate", "duration"] - - for x in Trade.query.all(): - print("date: {}".format(x.open_date)) - - trades = pd.DataFrame([(t.pair, t.calc_profit(), - t.open_date.replace(tzinfo=timeZone), - t.close_date.replace(tzinfo=timeZone) if t.close_date else None, - t.open_rate, t.close_rate, - t.close_date.timestamp() - t.open_date.timestamp() - if t.close_date else None) - for t in Trade.query.filter(Trade.pair.is_(pair)).all()], - columns=columns) - - elif args.exportfilename: - - file = Path(args.exportfilename) - if file.exists(): - trades = load_backtest_data(file) - - else: - trades = pd.DataFrame([], columns=BT_DATA_COLUMNS) - - return trades - - -def generate_plot_file(fig, pair, ticker_interval, is_last) -> None: - """ - Generate a plot html file from pre populated fig plotly object - :return: None - """ - logger.info('Generate plot file for %s', pair) - - pair_name = pair.replace("/", "_") - file_name = 'freqtrade-plot-' + pair_name + '-' + ticker_interval + '.html' - - Path("user_data/plots").mkdir(parents=True, exist_ok=True) - - plot(fig, filename=str(Path('user_data/plots').joinpath(file_name)), auto_open=False) - if is_last: - plot(fig, filename=str(Path('user_data').joinpath('freqtrade-plot.html')), auto_open=False) - - -def get_trading_env(args: Namespace): - """ - Initalize freqtrade Exchange and Strategy, split pairs recieved in parameter - :return: Strategy - """ - global _CONF - - # Load the configuration - _CONF.update(setup_configuration(args, RunMode.BACKTEST)) - print(_CONF) - - pairs = args.pairs.split(',') - if pairs is None: - logger.critical('Parameter --pairs mandatory;. E.g --pairs ETH/BTC,XRP/BTC') - exit() - - # Load the strategy - try: - strategy = StrategyResolver(_CONF).strategy - exchange = Exchange(_CONF) - except AttributeError: - logger.critical( - 'Impossible to load the strategy. Please check the file "user_data/strategies/%s.py"', - args.strategy - ) - exit() - - return [strategy, exchange, pairs] - - -def get_tickers_data(strategy, exchange, pairs: List[str], args): - """ - Get tickers data for each pairs on live or local, option defined in args - :return: dictinnary of tickers. output format: {'pair': tickersdata} - """ - - ticker_interval = strategy.ticker_interval - timerange = Arguments.parse_timerange(args.timerange) - - tickers = history.load_data( - datadir=Path(str(_CONF.get("datadir"))), - pairs=pairs, - ticker_interval=ticker_interval, - refresh_pairs=_CONF.get('refresh_pairs', False), - timerange=timerange, - exchange=Exchange(_CONF), - live=args.live, - ) - - # No ticker found, impossible to download, len mismatch - for pair, data in tickers.copy().items(): - logger.debug("checking tickers data of pair: %s", pair) - logger.debug("data.empty: %s", data.empty) - logger.debug("len(data): %s", len(data)) - if data.empty: - del tickers[pair] - logger.info( - 'An issue occured while retreiving datas of %s pair, please retry ' - 'using -l option for live or --refresh-pairs-cached', pair) - return tickers def generate_dataframe(strategy, tickers, pair) -> pd.DataFrame: @@ -177,211 +57,7 @@ def generate_dataframe(strategy, tickers, pair) -> pd.DataFrame: return dataframe -def extract_trades_of_period(dataframe, trades) -> pd.DataFrame: - """ - Compare trades and backtested pair DataFrames to get trades performed on backtested period - :return: the DataFrame of a trades of period - """ - trades = trades.loc[trades['open_time'] >= dataframe.iloc[0]['date']] - return trades - - -def generate_graph( - pair: str, - trades: pd.DataFrame, - data: pd.DataFrame, - indicators1: str, - indicators2: str - ) -> tools.make_subplots: - """ - Generate the graph from the data generated by Backtesting or from DB - :param pair: Pair to Display on the graph - :param trades: All trades created - :param data: Dataframe - :indicators1: String Main plot indicators - :indicators2: String Sub plot indicators - :return: None - """ - - # Define the graph - fig = tools.make_subplots( - rows=3, - cols=1, - shared_xaxes=True, - row_width=[1, 1, 4], - vertical_spacing=0.0001, - ) - fig['layout'].update(title=pair) - fig['layout']['yaxis1'].update(title='Price') - fig['layout']['yaxis2'].update(title='Volume') - fig['layout']['yaxis3'].update(title='Other') - fig['layout']['xaxis']['rangeslider'].update(visible=False) - - # Common information - candles = go.Candlestick( - x=data.date, - open=data.open, - high=data.high, - low=data.low, - close=data.close, - name='Price' - ) - - df_buy = data[data['buy'] == 1] - buys = go.Scattergl( - x=df_buy.date, - y=df_buy.close, - mode='markers', - name='buy', - marker=dict( - symbol='triangle-up-dot', - size=9, - line=dict(width=1), - color='green', - ) - ) - df_sell = data[data['sell'] == 1] - sells = go.Scattergl( - x=df_sell.date, - y=df_sell.close, - mode='markers', - name='sell', - marker=dict( - symbol='triangle-down-dot', - size=9, - line=dict(width=1), - color='red', - ) - ) - - trade_buys = go.Scattergl( - x=trades["open_time"], - y=trades["open_rate"], - mode='markers', - name='trade_buy', - marker=dict( - symbol='square-open', - size=11, - line=dict(width=2), - color='green' - ) - ) - trade_sells = go.Scattergl( - x=trades["close_time"], - y=trades["close_rate"], - mode='markers', - name='trade_sell', - marker=dict( - symbol='square-open', - size=11, - line=dict(width=2), - color='red' - ) - ) - - # Row 1 - fig.append_trace(candles, 1, 1) - - if 'bb_lowerband' in data and 'bb_upperband' in data: - bb_lower = go.Scatter( - x=data.date, - y=data.bb_lowerband, - name='BB lower', - line={'color': 'rgba(255,255,255,0)'}, - ) - bb_upper = go.Scatter( - x=data.date, - y=data.bb_upperband, - name='BB upper', - fill="tonexty", - fillcolor="rgba(0,176,246,0.2)", - line={'color': 'rgba(255,255,255,0)'}, - ) - fig.append_trace(bb_lower, 1, 1) - fig.append_trace(bb_upper, 1, 1) - - fig = generate_row(fig=fig, row=1, raw_indicators=indicators1, data=data) - fig.append_trace(buys, 1, 1) - fig.append_trace(sells, 1, 1) - fig.append_trace(trade_buys, 1, 1) - fig.append_trace(trade_sells, 1, 1) - - # Row 2 - volume = go.Bar( - x=data['date'], - y=data['volume'], - name='Volume' - ) - fig.append_trace(volume, 2, 1) - - # Row 3 - fig = generate_row(fig=fig, row=3, raw_indicators=indicators2, data=data) - - return fig - - -def generate_row(fig, row, raw_indicators, data) -> tools.make_subplots: - """ - Generator all the indicator selected by the user for a specific row - """ - for indicator in raw_indicators.split(','): - if indicator in data: - scattergl = go.Scattergl( - x=data['date'], - y=data[indicator], - name=indicator - ) - fig.append_trace(scattergl, row, 1) - else: - logger.info( - 'Indicator "%s" ignored. Reason: This indicator is not found ' - 'in your strategy.', - indicator - ) - - return fig - - -def plot_parse_args(args: List[str]) -> Namespace: - """ - Parse args passed to the script - :param args: Cli arguments - :return: args: Array with all arguments - """ - arguments = Arguments(args, 'Graph dataframe') - arguments.scripts_options() - arguments.parser.add_argument( - '--indicators1', - help='Set indicators from your strategy you want in the first row of the graph. Separate ' - 'them with a coma. E.g: ema3,ema5 (default: %(default)s)', - type=str, - default='sma,ema3,ema5', - dest='indicators1', - ) - - arguments.parser.add_argument( - '--indicators2', - help='Set indicators from your strategy you want in the third row of the graph. Separate ' - 'them with a coma. E.g: fastd,fastk (default: %(default)s)', - type=str, - default='macd,macdsignal', - dest='indicators2', - ) - arguments.parser.add_argument( - '--plot-limit', - help='Specify tick limit for plotting - too high values cause huge files - ' - 'Default: %(default)s', - dest='plot_limit', - default=750, - type=int, - ) - arguments.common_args_parser() - arguments.optimizer_shared_options(arguments.parser) - arguments.backtesting_options(arguments.parser) - return arguments.parse_args() - - -def analyse_and_plot_pairs(args: Namespace): +def analyse_and_plot_pairs(config: Dict[str, Any]): """ From arguments provided in cli: -Initialise backtest env @@ -392,12 +68,28 @@ def analyse_and_plot_pairs(args: Namespace): -Generate plot files :return: None """ - strategy, exchange, pairs = get_trading_env(args) + exchange = ExchangeResolver(config.get('exchange', {}).get('name'), config).exchange + + strategy = StrategyResolver(config).strategy + if "pairs" in config: + pairs = config["pairs"].split(',') + else: + pairs = config["exchange"]["pair_whitelist"] + # Set timerange to use - timerange = Arguments.parse_timerange(args.timerange) + timerange = Arguments.parse_timerange(config["timerange"]) ticker_interval = strategy.ticker_interval - tickers = get_tickers_data(strategy, exchange, pairs, args) + tickers = history.load_data( + datadir=Path(str(config.get("datadir"))), + pairs=pairs, + ticker_interval=config['ticker_interval'], + refresh_pairs=config.get('refresh_pairs', False), + timerange=timerange, + exchange=exchange, + live=config.get("live", False), + ) + pair_counter = 0 for pair, data in tickers.items(): pair_counter += 1 @@ -406,23 +98,44 @@ def analyse_and_plot_pairs(args: Namespace): tickers[pair] = data dataframe = generate_dataframe(strategy, tickers, pair) - trades = load_trades(args, pair, timerange) + trades = load_trades(db_url=config["db_url"], + exportfilename=config["exportfilename"]) + trades = trades.loc[trades['pair'] == pair] trades = extract_trades_of_period(dataframe, trades) fig = generate_graph( pair=pair, - trades=trades, data=dataframe, - indicators1=args.indicators1, - indicators2=args.indicators2 + trades=trades, + indicators1=config["indicators1"].split(","), + indicators2=config["indicators2"].split(",") ) - is_last = (False, True)[pair_counter == len(tickers)] - generate_plot_file(fig, pair, ticker_interval, is_last) + generate_plot_file(fig, pair, ticker_interval) logger.info('End of ploting process %s plots generated', pair_counter) +def plot_parse_args(args: List[str]) -> Dict[str, Any]: + """ + Parse args passed to the script + :param args: Cli arguments + :return: args: Array with all arguments + """ + arguments = Arguments(args, 'Graph dataframe') + arguments.common_options() + arguments.main_options() + arguments.common_optimize_options() + arguments.backtesting_options() + arguments.common_scripts_options() + arguments.plot_dataframe_options() + parsed_args = arguments.parse_args() + + # Load the configuration + config = setup_configuration(parsed_args, RunMode.BACKTEST) + return config + + def main(sysargv: List[str]) -> None: """ This function will initiate the bot and start the trading loop. diff --git a/scripts/plot_profit.py b/scripts/plot_profit.py index 5f7d42c87..fd98c120c 100755 --- a/scripts/plot_profit.py +++ b/scripts/plot_profit.py @@ -206,10 +206,11 @@ def plot_parse_args(args: List[str]) -> Namespace: :return: args: Array with all arguments """ arguments = Arguments(args, 'Graph profits') - arguments.scripts_options() - arguments.common_args_parser() - arguments.optimizer_shared_options(arguments.parser) - arguments.backtesting_options(arguments.parser) + arguments.common_options() + arguments.main_options() + arguments.common_optimize_options() + arguments.backtesting_options() + arguments.common_scripts_options() return arguments.parse_args() diff --git a/scripts/rest_client.py b/scripts/rest_client.py index 2261fba0b..a46b3ebfb 100755 --- a/scripts/rest_client.py +++ b/scripts/rest_client.py @@ -65,14 +65,14 @@ class FtRestClient(): def start(self): """ Start the bot if it's in stopped state. - :returns: json object + :return: json object """ return self._post("start") def stop(self): """ Stop the bot. Use start to restart - :returns: json object + :return: json object """ return self._post("stop") @@ -80,77 +80,77 @@ class FtRestClient(): """ Stop buying (but handle sells gracefully). use reload_conf to reset - :returns: json object + :return: json object """ return self._post("stopbuy") def reload_conf(self): """ Reload configuration - :returns: json object + :return: json object """ return self._post("reload_conf") def balance(self): """ Get the account balance - :returns: json object + :return: json object """ return self._get("balance") def count(self): """ Returns the amount of open trades - :returns: json object + :return: json object """ return self._get("count") def daily(self, days=None): """ Returns the amount of open trades - :returns: json object + :return: json object """ return self._get("daily", params={"timescale": days} if days else None) def edge(self): """ Returns information about edge - :returns: json object + :return: json object """ return self._get("edge") def profit(self): """ Returns the profit summary - :returns: json object + :return: json object """ return self._get("profit") def performance(self): """ Returns the performance of the different coins - :returns: json object + :return: json object """ return self._get("performance") def status(self): """ Get the status of open trades - :returns: json object + :return: json object """ return self._get("status") def version(self): """ Returns the version of the bot - :returns: json object containing the version + :return: json object containing the version """ return self._get("version") def whitelist(self): """ Show the current whitelist - :returns: json object + :return: json object """ return self._get("whitelist") @@ -158,7 +158,7 @@ class FtRestClient(): """ Show the current blacklist :param add: List of coins to add (example: "BNB/BTC") - :returns: json object + :return: json object """ if not args: return self._get("blacklist") @@ -170,7 +170,7 @@ class FtRestClient(): Buy an asset :param pair: Pair to buy (ETH/BTC) :param price: Optional - price to buy - :returns: json object of the trade + :return: json object of the trade """ data = {"pair": pair, "price": price @@ -181,7 +181,7 @@ class FtRestClient(): """ Force-sell a trade :param tradeid: Id of the trade (can be received via status command) - :returns: json object + :return: json object """ return self._post("forcesell", data={"tradeid": tradeid}) diff --git a/user_data/strategies/test_strategy.py b/user_data/strategies/test_strategy.py index 2415e43eb..d8ff790b2 100644 --- a/user_data/strategies/test_strategy.py +++ b/user_data/strategies/test_strategy.py @@ -44,8 +44,8 @@ class TestStrategy(IStrategy): # trailing stoploss trailing_stop = False - trailing_stop_positive = 0.01 - trailing_stop_positive_offset = 0.0 # Disabled / not configured + # trailing_stop_positive = 0.01 + # trailing_stop_positive_offset = 0.0 # Disabled / not configured # Optimal ticker interval for the strategy ticker_interval = '5m'