diff --git a/README.md b/README.md index 045d6b624..d9e537382 100644 --- a/README.md +++ b/README.md @@ -136,8 +136,8 @@ to understand the requirements before sending your pull-requests. ### Bot commands ```bash -usage: main.py [-h] [-c PATH] [-v] [--version] [--dynamic-whitelist [INT]] - [--dry-run-db] +usage: main.py [-h] [-v] [--version] [-c PATH] [--dry-run-db] [--datadir PATH] + [--dynamic-whitelist [INT]] {backtesting,hyperopt} ... Simple High Frequency Trading Bot for crypto currencies @@ -149,16 +149,17 @@ positional arguments: optional arguments: -h, --help show this help message and exit - -c PATH, --config PATH - specify configuration file (default: config.json) -v, --verbose be verbose --version show program's version number and exit - --dynamic-whitelist [INT] - dynamically generate and update whitelist based on 24h - BaseVolume (Default 20 currencies) + -c PATH, --config PATH + specify configuration file (default: config.json) --dry-run-db Force dry run to use a local DB "tradesv3.dry_run.sqlite" instead of memory DB. Work only if dry_run is enabled. + --datadir PATH path to backtest data (default freqdata/tests/testdata + --dynamic-whitelist [INT] + dynamically generate and update whitelist based on 24h + BaseVolume (Default 20 currencies) ``` More details on: - [How to run the bot](https://github.com/gcarq/freqtrade/blob/develop/docs/bot-usage.md#bot-commands) diff --git a/docs/backtesting.md b/docs/backtesting.md index 8574d9dc2..8b12b04e8 100644 --- a/docs/backtesting.md +++ b/docs/backtesting.md @@ -14,7 +14,7 @@ real data. This is what we call Backtesting will use the crypto-currencies (pair) from your config file and load static tickers located in -[/freqtrade/tests/testdata](https://github.com/gcarq/freqtrade/tree/develop/freqtrade/tests/testdata). +[/freqtrade/tests/testdata](https://github.com/gcarq/freqtrade/tree/develop/freqtrade/tests/testdata). If the 5 min and 1 min ticker for the crypto-currencies to test is not already in the `testdata` folder, backtesting will download them automatically. Testdata files will not be updated until you specify it. @@ -51,15 +51,49 @@ python3 ./freqtrade/main.py backtesting --realistic-simulation --live python3 ./freqtrade/main.py backtesting --datadir freqtrade/tests/testdata-20180101 ``` +**Exporting trades to file** +```bash +freqtrade backtesting --export trades +``` + +**Running backtest with smaller testset** +Use the `--timerange` argument to change how much of the testset +you want to use. The last N ticks/timeframes will be used. + +Example: +```bash +python3 ./freqtrade/main.py backtesting --timerange=-200 +``` + +***Advanced use of timerange*** +Doing `--timerange=-200` will get the last 200 timeframes +from your inputdata. You can also specify specific dates, +or a range span indexed by start and stop. + +The full timerange specification: +- Use last 123 tickframes of data: `--timerange=-123` +- Use first 123 tickframes of data: `--timerange=123-` +- Use tickframes from line 123 through 456: `--timerange=123-456` + + +Incoming feature, not implemented yet: +- `--timerange=-20180131` +- `--timerange=20180101-` +- `--timerange=20180101-20181231` + + +**Update testdata directory** To update your testdata directory, or download into another testdata directory: ```bash -mkdir freqtrade/tests/testdata-20180113 -cp freqtrade/tests/testdata/pairs.json freqtrade/tests/testdata-20180113 -cd freqtrade/tests/testdata-20180113 +mkdir -p user_data/data/testdata-20180113 +cp freqtrade/tests/testdata/pairs.json user_data/data-20180113 +cd user_data/data-20180113 +``` Possibly edit pairs.json file to include/exclude pairs -python download_backtest_data.py -p pairs.json +```bash +python freqtrade/tests/testdata/download_backtest_data.py -p pairs.json ``` The script will read your pairs.json file, and download ticker data diff --git a/docs/configuration.md b/docs/configuration.md index 63fad1e32..7a41644ee 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -30,6 +30,7 @@ The table below will list all configuration parameters. | `exchange.pair_whitelist` | [] | No | List of currency to use by the bot. Can be overrided with `--dynamic-whitelist` param. | `exchange.pair_blacklist` | [] | No | List of currency the bot must avoid. Useful when using `--dynamic-whitelist` param. | `experimental.use_sell_signal` | false | No | Use your sell strategy in addition of the `minimal_roi`. +| `experimental.sell_profit_only` | false | No | waits until you have made a positive profit before taking a sell decision. | `telegram.enabled` | true | Yes | Enable or not the usage of Telegram. | `telegram.token` | token | No | Your Telegram bot token. Only required is `enable` is `true`. | `telegram.chat_id` | chat_id | No | Your personal Telegram account id. Only required is `enable` is `true`. diff --git a/docs/hyperopt.md b/docs/hyperopt.md index 24a9dbc51..af564f0b6 100644 --- a/docs/hyperopt.md +++ b/docs/hyperopt.md @@ -168,6 +168,16 @@ If you would like to learn parameters using an alternate ticke-data that you have on-disk, use the --datadir PATH option. Default hyperopt will use data from directory freqtrade/tests/testdata. +### Running hyperopt with smaller testset + +Use the --timeperiod argument to change how much of the testset +you want to use. The last N ticks/timeframes will be used. +Example: + +```bash +python3 ./freqtrade/main.py hyperopt --timeperiod -200 +``` + ### Hyperopt with MongoDB Hyperopt with MongoDB, is like Hyperopt under steroids. As you saw by executing the previous command is the execution takes a long time. diff --git a/docs/plotting.md b/docs/plotting.md new file mode 100644 index 000000000..598443e12 --- /dev/null +++ b/docs/plotting.md @@ -0,0 +1,48 @@ +# Plotting +This page explains how to plot prices, indicator, profits. + +## Table of Contents +- [Plot price and indicators](#plot-price-and-indicators) +- [Plot profit](#plot-profit) + +## Plot price and indicators +Usage for the price plotter: +script/plot_dataframe.py [-h] [-p pair] + +Example +``` +python script/plot_dataframe.py -p BTC_ETH,BTC_LTC +``` + +The -p pair argument, can be used to specify what +pair you would like to plot. + + +## Plot profit + +The profit plotter show a picture with three plots: +1) Average closing price for all pairs +2) The summarized profit made by backtesting. + Note that this is not the real-world profit, but + more of an estimate. +3) Each pair individually profit + +The first graph is good to get a grip of how the overall market +progresses. + +The second graph will show how you algorithm works or doesnt. +Perhaps you want an algorithm that steadily makes small profits, +or one that acts less seldom, but makes big swings. + +The third graph can be useful to spot outliers, events in pairs +that makes profit spikes. + +Usage for the profit plotter: +script/plot_profit.py [-h] [-p pair] [--datadir directory] [--ticker_interval num] + +The -p pair argument, can be used to plot a single pair + +Example +``` +python python scripts/plot_profit.py --datadir ../freqtrade/freqtrade/tests/testdata-20171221/ -p BTC_LTC +``` diff --git a/freqtrade/analyze.py b/freqtrade/analyze.py index 4a2a46a5d..cee5d175a 100644 --- a/freqtrade/analyze.py +++ b/freqtrade/analyze.py @@ -281,36 +281,36 @@ def analyze_ticker(ticker_history: List[Dict]) -> DataFrame: return dataframe -def get_signal(pair: str, signal: SignalType, interval: int) -> bool: +def get_signal(pair: str, interval: int) -> (bool, bool): """ Calculates current signal based several technical analysis indicators :param pair: pair in format BTC_ANT or BTC-ANT - :return: True if pair is good for buying, False otherwise + :return: (True, False) if pair is good for buying and not for selling """ ticker_hist = get_ticker_history(pair, interval) if not ticker_hist: logger.warning('Empty ticker history for pair %s', pair) - return False + return (False, False) try: dataframe = analyze_ticker(ticker_hist) except ValueError as ex: logger.warning('Unable to analyze ticker for pair %s: %s', pair, str(ex)) - return False + return (False, False) except Exception as ex: logger.exception('Unexpected error when analyzing ticker for pair %s: %s', pair, str(ex)) - return False + return (False, False) if dataframe.empty: - return False + return (False, False) latest = dataframe.iloc[-1] # Check if dataframe is out of date signal_date = arrow.get(latest['date']) if signal_date < arrow.now() - timedelta(minutes=10): - return False + return (False, False) - result = latest[signal.value] == 1 - logger.debug('%s_trigger: %s (pair=%s, signal=%s)', signal.value, latest['date'], pair, result) - return result + (buy, sell) = latest[SignalType.BUY.value] == 1, latest[SignalType.SELL.value] == 1 + logger.debug('trigger: %s (pair=%s) buy=%s sell=%s', latest['date'], pair, str(buy), str(sell)) + return (buy, sell) diff --git a/freqtrade/main.py b/freqtrade/main.py index 8906aceb7..34bc72658 100755 --- a/freqtrade/main.py +++ b/freqtrade/main.py @@ -14,7 +14,7 @@ from cachetools import cached, TTLCache from freqtrade import (DependencyException, OperationalException, __version__, exchange, persistence, rpc) -from freqtrade.analyze import SignalType, get_signal +from freqtrade.analyze import get_signal from freqtrade.fiat_convert import CryptoToFiatConverter from freqtrade.misc import (State, get_state, load_config, parse_args, throttle, update_state) @@ -129,9 +129,17 @@ def check_handle_timedout(timeoutvalue: int) -> None: timeoutthreashold = arrow.utcnow().shift(minutes=-timeoutvalue).datetime for trade in Trade.query.filter(Trade.open_order_id.isnot(None)).all(): - order = exchange.get_order(trade.open_order_id) + try: + order = exchange.get_order(trade.open_order_id) + except requests.exceptions.RequestException: + logger.info('Cannot query order for %s due to %s', trade, traceback.format_exc()) + continue ordertime = arrow.get(order['opened']) + # Check if trade is still actually open + if int(order['remaining']) == 0: + continue + if order['type'] == "LIMIT_BUY" and ordertime < timeoutthreashold: # Buy timeout - cancel order exchange.cancel_order(trade.open_order_id) @@ -140,6 +148,8 @@ def check_handle_timedout(timeoutvalue: int) -> None: Trade.session.delete(trade) Trade.session.flush() logger.info('Buy order timeout for %s.', trade) + rpc.send_msg('*Timeout:* Unfilled buy order for {} cancelled'.format( + trade.pair.replace('_', '/'))) else: # if trade is partially complete, edit the stake details for the trade # and close the order @@ -147,6 +157,8 @@ def check_handle_timedout(timeoutvalue: int) -> None: trade.stake_amount = trade.amount * trade.open_rate trade.open_order_id = None logger.info('Partial buy order timeout for %s.', trade) + rpc.send_msg('*Timeout:* Remaining buy order for {} cancelled'.format( + trade.pair.replace('_', '/'))) elif order['type'] == "LIMIT_SELL" and ordertime < timeoutthreashold: # Sell timeout - cancel order and update trade if order['remaining'] == order['amount']: @@ -157,6 +169,8 @@ def check_handle_timedout(timeoutvalue: int) -> None: trade.close_date = None trade.is_open = True trade.open_order_id = None + rpc.send_msg('*Timeout:* Unfilled sell order for {} cancelled'.format( + trade.pair.replace('_', '/'))) logger.info('Sell order timeout for %s.', trade) return True else: @@ -247,24 +261,27 @@ def handle_trade(trade: Trade, interval: int) -> bool: logger.debug('Handling %s ...', trade) current_rate = exchange.get_ticker(trade.pair)['bid'] - # Check if minimal roi has been reached - if min_roi_reached(trade, current_rate, datetime.utcnow()): + (buy, sell) = (False, False) + + if _CONF.get('experimental', {}).get('use_sell_signal'): + (buy, sell) = get_signal(trade.pair) + + # Check if minimal roi has been reached and no longer in buy conditions (avoiding a fee) + if not buy and min_roi_reached(trade, current_rate, datetime.utcnow()): logger.debug('Executing sell due to ROI ...') execute_sell(trade, current_rate) return True - # Experimental: Check if sell signal has been enabled and triggered - if _CONF.get('experimental', {}).get('use_sell_signal'): - # Experimental: Check if the trade is profitable before selling it (avoid selling at loss) - if _CONF.get('experimental', {}).get('sell_profit_only'): - logger.debug('Checking if trade is profitable ...') - if trade.calc_profit(rate=current_rate) <= 0: - return False - logger.debug('Checking sell_signal ...') - if get_signal(trade.pair, SignalType.SELL, interval): - logger.debug('Executing sell due to sell signal ...') - execute_sell(trade, current_rate) - return True + # Experimental: Check if the trade is profitable before selling it (avoid selling at loss) + if _CONF.get('experimental', {}).get('sell_profit_only', False): + logger.debug('Checking if trade is profitable ...') + if not buy and trade.calc_profit(rate=current_rate) <= 0: + return False + + if sell and not buy: + logger.debug('Executing sell due to sell signal ...') + execute_sell(trade, current_rate) + return True return False @@ -305,7 +322,8 @@ def create_trade(stake_amount: float, interval: int) -> bool: # Pick pair based on StochRSI buy signals for _pair in whitelist: - if get_signal(_pair, SignalType.BUY, interval): + (buy, sell) = get_signal(_pair) + if buy and not sell: pair = _pair break else: diff --git a/freqtrade/misc.py b/freqtrade/misc.py index c97e4635f..a91062e4e 100644 --- a/freqtrade/misc.py +++ b/freqtrade/misc.py @@ -4,6 +4,7 @@ import json import logging import time import os +import re from typing import Any, Callable, Dict, List from jsonschema import Draft4Validator, validate @@ -115,6 +116,14 @@ def common_args_parser(description: str): type=str, metavar='PATH', ) + parser.add_argument( + '--datadir', + help='path to backtest data (default freqdata/tests/testdata)', + dest='datadir', + default=os.path.join('freqtrade', 'tests', 'testdata'), + type=str, + metavar='PATH', + ) return parser @@ -131,14 +140,6 @@ def parse_args(args: List[str], description: str): action='store_true', dest='dry_run_db', ) - parser.add_argument( - '-dd', '--datadir', - help='path to backtest data (default freqdata/tests/testdata', - dest='datadir', - default=os.path.join('freqtrade', 'tests', 'testdata'), - type=str, - metavar='PATH', - ) parser.add_argument( '--dynamic-whitelist', help='dynamically generate and update whitelist \ @@ -154,6 +155,113 @@ def parse_args(args: List[str], description: str): return parser.parse_args(args) +def backtesting_options(parser: argparse.ArgumentParser) -> None: + parser.add_argument( + '-l', '--live', + action='store_true', + dest='live', + help='using live data', + ) + parser.add_argument( + '-i', '--ticker-interval', + help='specify ticker interval in minutes (1, 5, 30, 60, 1440)', + dest='ticker_interval', + default=5, + type=int, + metavar='INT', + ) + parser.add_argument( + '--realistic-simulation', + help='uses max_open_trades from config to simulate real world limitations', + action='store_true', + dest='realistic_simulation', + ) + parser.add_argument( + '-r', '--refresh-pairs-cached', + help='refresh the pairs files in tests/testdata with the latest data from Bittrex. \ + Use it if you want to run your backtesting with up-to-date data.', + action='store_true', + dest='refresh_pairs', + ) + parser.add_argument( + '--export', + help='Export backtest results, argument are: trades\ + Example --export=trades', + type=str, + default=None, + dest='export', + ) + parser.add_argument( + '--timerange', + help='Specify what timerange of data to use.', + default=None, + type=str, + dest='timerange', + ) + + +def hyperopt_options(parser: argparse.ArgumentParser) -> None: + parser.add_argument( + '-e', '--epochs', + help='specify number of epochs (default: 100)', + dest='epochs', + default=100, + type=int, + metavar='INT', + ) + parser.add_argument( + '--use-mongodb', + help='parallelize evaluations with mongodb (requires mongod in PATH)', + dest='mongodb', + action='store_true', + ) + parser.add_argument( + '-i', '--ticker-interval', + help='specify ticker interval in minutes (default: 5)', + dest='ticker_interval', + default=5, + type=int, + metavar='INT', + ) + parser.add_argument( + '--timerange', + help='Specify what timerange of data to use.', + default=None, + type=str, + dest='timerange', + ) + + +def parse_timerange(text): + if text is None: + return None + syntax = [('^-(\d{8})$', (None, 'date')), + ('^(\d{8})-$', ('date', None)), + ('^(\d{8})-(\d{8})$', ('date', 'date')), + ('^(-\d+)$', (None, 'line')), + ('^(\d+)-$', ('line', None)), + ('^(\d+)-(\d+)$', ('index', 'index'))] + for rex, stype in syntax: + # Apply the regular expression to text + m = re.match(rex, text) + if m: # Regex has matched + rvals = m.groups() + n = 0 + start = None + stop = None + if stype[0]: + start = rvals[n] + if stype[0] != 'date': + start = int(start) + n += 1 + if stype[1]: + stop = rvals[n] + if stype[1] != 'date': + stop = int(stop) + return (stype, start, stop) + raise Exception('Incorrect syntax for timerange "%s"' % text) + + def build_subcommands(parser: argparse.ArgumentParser) -> None: """ Builds and attaches all subcommands """ from freqtrade.optimize import backtesting, hyperopt @@ -163,59 +271,12 @@ def build_subcommands(parser: argparse.ArgumentParser) -> None: # Add backtesting subcommand backtesting_cmd = subparsers.add_parser('backtesting', help='backtesting module') backtesting_cmd.set_defaults(func=backtesting.start) - backtesting_cmd.add_argument( - '-l', '--live', - action='store_true', - dest='live', - help='using live data', - ) - backtesting_cmd.add_argument( - '-i', '--ticker-interval', - help='specify ticker interval in minutes (1, 5, 30, 60, 1440)', - dest='ticker_interval', - default=5, - type=int, - metavar='INT', - ) - backtesting_cmd.add_argument( - '--realistic-simulation', - help='uses max_open_trades from config to simulate real world limitations', - action='store_true', - dest='realistic_simulation', - ) - backtesting_cmd.add_argument( - '-r', '--refresh-pairs-cached', - help='refresh the pairs files in tests/testdata with the latest data from Bittrex. \ - Use it if you want to run your backtesting with up-to-date data.', - action='store_true', - dest='refresh_pairs', - ) + backtesting_options(backtesting_cmd) # Add hyperopt subcommand hyperopt_cmd = subparsers.add_parser('hyperopt', help='hyperopt module') hyperopt_cmd.set_defaults(func=hyperopt.start) - hyperopt_cmd.add_argument( - '-e', '--epochs', - help='specify number of epochs (default: 100)', - dest='epochs', - default=100, - type=int, - metavar='INT', - ) - hyperopt_cmd.add_argument( - '--use-mongodb', - help='parallelize evaluations with mongodb (requires mongod in PATH)', - dest='mongodb', - action='store_true', - ) - hyperopt_cmd.add_argument( - '-i', '--ticker-interval', - help='specify ticker interval in minutes (default: 5)', - dest='ticker_interval', - default=5, - type=int, - metavar='INT', - ) + hyperopt_options(hyperopt_cmd) # Required json-schema for user specified config diff --git a/freqtrade/optimize/__init__.py b/freqtrade/optimize/__init__.py index 3a22e6a5a..0ac808424 100644 --- a/freqtrade/optimize/__init__.py +++ b/freqtrade/optimize/__init__.py @@ -8,11 +8,25 @@ from pandas import DataFrame from freqtrade.exchange import get_ticker_history from freqtrade.optimize.hyperopt_conf import hyperopt_optimize_conf from freqtrade.analyze import populate_indicators, parse_ticker_dataframe +from freqtrade import misc logger = logging.getLogger(__name__) -def load_tickerdata_file(datadir, pair, ticker_interval): +def trim_tickerlist(tickerlist, timerange): + (stype, start, stop) = timerange + if stype == (None, 'line'): + return tickerlist[stop:] + elif stype == ('line', None): + return tickerlist[0:start] + elif stype == ('index', 'index'): + return tickerlist[start:stop] + else: + return tickerlist + + +def load_tickerdata_file(datadir, pair, ticker_interval, + timerange=None): """ Load a pair from file, :return dict OR empty if unsuccesful @@ -30,11 +44,13 @@ def load_tickerdata_file(datadir, pair, ticker_interval): # Read the file, load the json with open(file) as tickerdata: pairdata = json.load(tickerdata) + if timerange: + pairdata = trim_tickerlist(pairdata, timerange) return pairdata def load_data(datadir: str, ticker_interval: int, pairs: Optional[List[str]] = None, - refresh_pairs: Optional[bool] = False) -> Dict[str, List]: + refresh_pairs: Optional[bool] = False, timerange=None) -> Dict[str, List]: """ Loads ticker history data for the given parameters :param ticker_interval: ticker interval in minutes @@ -51,16 +67,21 @@ def load_data(datadir: str, ticker_interval: int, pairs: Optional[List[str]] = N download_pairs(datadir, _pairs, ticker_interval) for pair in _pairs: - pairdata = load_tickerdata_file(datadir, pair, ticker_interval) + pairdata = load_tickerdata_file(datadir, pair, ticker_interval, timerange=timerange) if not pairdata: # download the tickerdata from exchange download_backtesting_testdata(datadir, pair=pair, interval=ticker_interval) # and retry reading the pair - pairdata = load_tickerdata_file(datadir, pair, ticker_interval) + pairdata = load_tickerdata_file(datadir, pair, ticker_interval, timerange=timerange) result[pair] = pairdata return result +def tickerdata_to_dataframe(data): + preprocessed = preprocess(data) + return preprocessed + + def preprocess(tickerdata: Dict[str, List]) -> Dict[str, DataFrame]: """Creates a dataframe and populates indicators for given ticker data""" return {pair: populate_indicators(parse_ticker_dataframe(pair_data)) @@ -126,7 +147,6 @@ def download_backtesting_testdata(datadir: str, pair: str, interval: int = 5) -> logger.debug("New End: {}".format(data[-1:][0]['T'])) data = sorted(data, key=lambda data: data['T']) - with open(filename, "wt") as fp: - json.dump(data, fp) + misc.file_dump_json(filename, data) return True diff --git a/freqtrade/optimize/backtesting.py b/freqtrade/optimize/backtesting.py index 2ae88f559..935d6f6c7 100644 --- a/freqtrade/optimize/backtesting.py +++ b/freqtrade/optimize/backtesting.py @@ -13,7 +13,6 @@ from freqtrade import exchange from freqtrade.analyze import populate_buy_trend, populate_sell_trend from freqtrade.exchange import Bittrex from freqtrade.main import min_roi_reached -from freqtrade.optimize import preprocess from freqtrade.persistence import Trade logger = logging.getLogger(__name__) @@ -67,17 +66,60 @@ def generate_text_table( return tabulate(tabular_data, headers=headers, floatfmt=floatfmt) -def backtest(stake_amount: float, processed: Dict[str, DataFrame], - max_open_trades: int = 0, realistic: bool = True, sell_profit_only: bool = False, - stoploss: int = -1.00, use_sell_signal: bool = False) -> DataFrame: +def get_trade_entry(pair, row, ticker, trade_count_lock, args): + stake_amount = args['stake_amount'] + max_open_trades = args.get('max_open_trades', 0) + sell_profit_only = args.get('sell_profit_only', False) + stoploss = args.get('stoploss', -1) + use_sell_signal = args.get('use_sell_signal', False) + trade = Trade(open_rate=row.close, + open_date=row.date, + stake_amount=stake_amount, + amount=stake_amount / row.open, + fee=exchange.get_fee() + ) + + # calculate win/lose forwards from buy point + sell_subset = ticker[row.Index + 1:][['close', 'date', 'sell']] + for row2 in sell_subset.itertuples(index=True): + if max_open_trades > 0: + # Increase trade_count_lock for every iteration + trade_count_lock[row2.date] = trade_count_lock.get(row2.date, 0) + 1 + + current_profit_percent = trade.calc_profit_percent(rate=row2.close) + if (sell_profit_only and current_profit_percent < 0): + continue + if min_roi_reached(trade, row2.close, row2.date) or \ + (row2.sell == 1 and use_sell_signal) or \ + current_profit_percent <= stoploss: + current_profit_btc = trade.calc_profit(rate=row2.close) + return row2, (pair, + current_profit_percent, + current_profit_btc, + row2.Index - row.Index, + current_profit_btc > 0, + current_profit_btc < 0 + ) + + +def backtest(args) -> DataFrame: """ Implements backtesting functionality - :param stake_amount: btc amount to use for each trade - :param processed: a processed dictionary with format {pair, data} - :param max_open_trades: maximum number of concurrent trades (default: 0, disabled) - :param realistic: do we try to simulate realistic trades? (default: True) + :param args: a dict containing: + stake_amount: btc amount to use for each trade + processed: a processed dictionary with format {pair, data} + max_open_trades: maximum number of concurrent trades (default: 0, disabled) + realistic: do we try to simulate realistic trades? (default: True) + sell_profit_only: sell if profit only + use_sell_signal: act on sell-signal + stoploss: use stoploss :return: DataFrame """ + processed = args['processed'] + max_open_trades = args.get('max_open_trades', 0) + realistic = args.get('realistic', True) + record = args.get('record', None) + records = [] trades = [] trade_count_lock: dict = {} exchange._API = Bittrex({'key': '', 'secret': ''}) @@ -100,41 +142,25 @@ def backtest(stake_amount: float, processed: Dict[str, DataFrame], # Increase lock trade_count_lock[row.date] = trade_count_lock.get(row.date, 0) + 1 - trade = Trade( - open_rate=row.close, - open_date=row.date, - stake_amount=stake_amount, - amount=stake_amount / row.open, - fee=exchange.get_fee() - ) - - # calculate win/lose forwards from buy point - sell_subset = ticker[row.Index + 1:][['close', 'date', 'sell']] - for row2 in sell_subset.itertuples(index=True): - if max_open_trades > 0: - # Increase trade_count_lock for every iteration - trade_count_lock[row2.date] = trade_count_lock.get(row2.date, 0) + 1 - - current_profit_percent = trade.calc_profit_percent(rate=row2.close) - if (sell_profit_only and current_profit_percent < 0): - continue - if min_roi_reached(trade, row2.close, row2.date) or \ - (row2.sell == 1 and use_sell_signal) or \ - current_profit_percent <= stoploss: - current_profit_btc = trade.calc_profit(rate=row2.close) - lock_pair_until = row2.Index - - trades.append( - ( - pair, - current_profit_percent, - current_profit_btc, - row2.Index - row.Index, - current_profit_btc > 0, - current_profit_btc < 0 - ) - ) - break + ret = get_trade_entry(pair, row, ticker, + trade_count_lock, args) + if ret: + row2, trade_entry = ret + lock_pair_until = row2.Index + trades.append(trade_entry) + if record: + # Note, need to be json.dump friendly + # record a tuple of pair, current_profit_percent, + # entry-date, duration + records.append((pair, trade_entry[1], + row.date.strftime('%s'), + row2.date.strftime('%s'), + row.Index, trade_entry[3])) + # For now export inside backtest(), maybe change so that backtest() + # returns a tuple like: (dataframe, records, logs, etc) + if record and record.find('trades') >= 0: + logger.info('Dumping backtest results') + misc.file_dump_json('backtest-result.json', records) labels = ['currency', 'profit_percent', 'profit_BTC', 'duration', 'profit', 'loss'] return DataFrame.from_records(trades, columns=labels) @@ -167,6 +193,10 @@ def start(args): logger.info('Using stake_currency: %s ...', config['stake_currency']) logger.info('Using stake_amount: %s ...', config['stake_amount']) + timerange = misc.parse_timerange(args.timerange) + data = optimize.load_data(args.datadir, pairs=pairs, ticker_interval=args.ticker_interval, + refresh_pairs=args.refresh_pairs, + timerange=timerange) max_open_trades = 0 if args.realistic_simulation: logger.info('Using max_open_trades: %s ...', config['max_open_trades']) @@ -176,21 +206,22 @@ def start(args): from freqtrade import main main._CONF = config - preprocessed = preprocess(data) + preprocessed = optimize.tickerdata_to_dataframe(data) # Print timeframe min_date, max_date = get_timeframe(preprocessed) logger.info('Measuring data from %s up to %s ...', min_date.isoformat(), max_date.isoformat()) - # Execute backtest and print results - results = backtest( - stake_amount=config['stake_amount'], - processed=preprocessed, - max_open_trades=max_open_trades, - realistic=args.realistic_simulation, - sell_profit_only=config.get('experimental', {}).get('sell_profit_only', False), - stoploss=config.get('stoploss'), - use_sell_signal=config.get('experimental', {}).get('use_sell_signal', False) - ) + sell_profit_only = config.get('experimental', {}).get('sell_profit_only', False) + use_sell_signal = config.get('experimental', {}).get('use_sell_signal', False) + results = backtest({'stake_amount': config['stake_amount'], + 'processed': preprocessed, + 'max_open_trades': max_open_trades, + 'realistic': args.realistic_simulation, + 'sell_profit_only': sell_profit_only, + 'use_sell_signal': use_sell_signal, + 'stoploss': config.get('stoploss'), + 'record': args.export + }) logger.info( '\n==================================== BACKTESTING REPORT ====================================\n%s', # noqa generate_text_table(data, results, config['stake_currency'], args.ticker_interval) diff --git a/freqtrade/optimize/hyperopt.py b/freqtrade/optimize/hyperopt.py index 959e64206..b9780c13a 100644 --- a/freqtrade/optimize/hyperopt.py +++ b/freqtrade/optimize/hyperopt.py @@ -15,7 +15,7 @@ from hyperopt import STATUS_FAIL, STATUS_OK, Trials, fmin, hp, space_eval, tpe from hyperopt.mongoexp import MongoTrials from pandas import DataFrame -from freqtrade import main # noqa +from freqtrade import main, misc # noqa from freqtrade import exchange, optimize from freqtrade.exchange import Bittrex from freqtrade.misc import load_config @@ -164,7 +164,9 @@ def optimizer(params): from freqtrade.optimize import backtesting backtesting.populate_buy_trend = buy_strategy_generator(params) - results = backtest(OPTIMIZE_CONFIG['stake_amount'], PROCESSED, stoploss=params['stoploss']) + results = backtest({'stake_amount': OPTIMIZE_CONFIG['stake_amount'], + 'processed': PROCESSED, + 'stoploss': params['stoploss']}) result_explanation = format_results(results) total_profit = results.profit_percent.sum() @@ -273,8 +275,11 @@ def start(args): logger.info('Using config: %s ...', args.config) config = load_config(args.config) pairs = config['exchange']['pair_whitelist'] - PROCESSED = optimize.preprocess(optimize.load_data( - args.datadir, pairs=pairs, ticker_interval=args.ticker_interval)) + timerange = misc.parse_timerange(args.timerange) + data = optimize.load_data(args.datadir, pairs=pairs, + ticker_interval=args.ticker_interval, + timerange=timerange) + PROCESSED = optimize.tickerdata_to_dataframe(data) if args.mongodb: logger.info('Using mongodb ...') diff --git a/freqtrade/rpc/telegram.py b/freqtrade/rpc/telegram.py index 12e701a1c..0fdc734f4 100644 --- a/freqtrade/rpc/telegram.py +++ b/freqtrade/rpc/telegram.py @@ -241,20 +241,27 @@ def _daily(bot: Bot, update: Update) -> None: .order_by(Trade.close_date)\ .all() curdayprofit = sum(trade.calc_profit() for trade in trades) - profit_days[profitday] = format(curdayprofit, '.8f') + profit_days[profitday] = { + 'amount': format(curdayprofit, '.8f'), + 'trades': len(trades) + } stats = [ [ key, - '{value:.8f} {symbol}'.format(value=float(value), symbol=_CONF['stake_currency']), + '{value:.8f} {symbol}'.format( + value=float(value['amount']), + symbol=_CONF['stake_currency'] + ), '{value:.3f} {symbol}'.format( value=_FIAT_CONVERT.convert_amount( - value, + value['amount'], _CONF['stake_currency'], _CONF['fiat_display_currency'] ), symbol=_CONF['fiat_display_currency'] - ) + ), + '{value} trade{s}'.format(value=value['trades'], s='' if value['trades'] < 2 else 's'), ] for key, value in profit_days.items() ] @@ -262,7 +269,8 @@ def _daily(bot: Bot, update: Update) -> None: headers=[ 'Day', 'Profit {}'.format(_CONF['stake_currency']), - 'Profit {}'.format(_CONF['fiat_display_currency']) + 'Profit {}'.format(_CONF['fiat_display_currency']), + '# Trades' ], tablefmt='simple') diff --git a/freqtrade/tests/optimize/test_backtesting.py b/freqtrade/tests/optimize/test_backtesting.py index 5f899a48a..2872df83f 100644 --- a/freqtrade/tests/optimize/test_backtesting.py +++ b/freqtrade/tests/optimize/test_backtesting.py @@ -11,6 +11,13 @@ from freqtrade.optimize.backtesting import backtest, generate_text_table, get_ti import freqtrade.optimize.backtesting as backtesting +def trim_dictlist(dl, num): + new = {} + for pair, pair_data in dl.items(): + new[pair] = pair_data[num:] + return new + + def test_generate_text_table(): results = pd.DataFrame( { @@ -43,8 +50,11 @@ def test_backtest(default_conf, mocker): exchange._API = Bittrex({'key': '', 'secret': ''}) data = optimize.load_data(None, ticker_interval=5, pairs=['BTC_ETH']) - results = backtest(default_conf['stake_amount'], - optimize.preprocess(data), 10, True) + data = trim_dictlist(data, -200) + results = backtest({'stake_amount': default_conf['stake_amount'], + 'processed': optimize.preprocess(data), + 'max_open_trades': 10, + 'realistic': True}) assert not results.empty @@ -54,21 +64,17 @@ def test_backtest_1min_ticker_interval(default_conf, mocker): # Run a backtesting for an exiting 5min ticker_interval data = optimize.load_data(None, ticker_interval=1, pairs=['BTC_UNITEST']) - results = backtest(default_conf['stake_amount'], - optimize.preprocess(data), 1, True) + data = trim_dictlist(data, -200) + results = backtest({'stake_amount': default_conf['stake_amount'], + 'processed': optimize.preprocess(data), + 'max_open_trades': 1, + 'realistic': True}) assert not results.empty -def trim_dictlist(dl, num): - new = {} - for pair, pair_data in dl.items(): - new[pair] = pair_data[num:] - return new - - def load_data_test(what): - data = optimize.load_data(None, ticker_interval=1, pairs=['BTC_UNITEST']) - data = trim_dictlist(data, -100) + timerange = ((None, 'line'), None, -100) + data = optimize.load_data(None, ticker_interval=1, pairs=['BTC_UNITEST'], timerange=timerange) pair = data['BTC_UNITEST'] datalen = len(pair) # Depending on the what parameter we now adjust the @@ -113,7 +119,10 @@ def simple_backtest(config, contour, num_results): data = load_data_test(contour) processed = optimize.preprocess(data) assert isinstance(processed, dict) - results = backtest(config['stake_amount'], processed, 1, True) + results = backtest({'stake_amount': config['stake_amount'], + 'processed': processed, + 'max_open_trades': 1, + 'realistic': True}) # results :: assert len(results) == num_results @@ -125,8 +134,11 @@ def simple_backtest(config, contour, num_results): def test_backtest2(default_conf, mocker): mocker.patch.dict('freqtrade.main._CONF', default_conf) data = optimize.load_data(None, ticker_interval=5, pairs=['BTC_ETH']) - results = backtest(default_conf['stake_amount'], - optimize.preprocess(data), 10, True) + data = trim_dictlist(data, -200) + results = backtest({'stake_amount': default_conf['stake_amount'], + 'processed': optimize.preprocess(data), + 'max_open_trades': 10, + 'realistic': True}) assert not results.empty @@ -149,10 +161,10 @@ def test_backtest_pricecontours(default_conf, mocker): simple_backtest(default_conf, contour, numres) -def mocked_load_data(datadir, pairs=[], ticker_interval=0, refresh_pairs=False): - tickerdata = optimize.load_tickerdata_file(datadir, 'BTC_UNITEST', 1) +def mocked_load_data(datadir, pairs=[], ticker_interval=0, refresh_pairs=False, timerange=None): + tickerdata = optimize.load_tickerdata_file(datadir, 'BTC_UNITEST', 1, timerange=timerange) pairdata = {'BTC_UNITEST': tickerdata} - return trim_dictlist(pairdata, -100) + return pairdata def test_backtest_start(default_conf, mocker, caplog): @@ -166,6 +178,8 @@ def test_backtest_start(default_conf, mocker, caplog): args.level = 10 args.live = False args.datadir = None + args.export = None + args.timerange = '-100' # needed due to MagicMock malleability backtesting.start(args) # check the logs, that will contain the backtest result exists = ['Using max_open_trades: 1 ...', diff --git a/freqtrade/tests/optimize/test_hyperopt.py b/freqtrade/tests/optimize/test_hyperopt.py index 4bb5c8c0b..6e21cde76 100644 --- a/freqtrade/tests/optimize/test_hyperopt.py +++ b/freqtrade/tests/optimize/test_hyperopt.py @@ -54,6 +54,7 @@ def create_trials(mocker): def test_start_calls_fmin(mocker): trials = create_trials(mocker) + mocker.patch('freqtrade.optimize.tickerdata_to_dataframe') mocker.patch('freqtrade.optimize.hyperopt.TRIALS', return_value=trials) mocker.patch('freqtrade.optimize.hyperopt.sorted', return_value=trials.results) @@ -61,7 +62,8 @@ def test_start_calls_fmin(mocker): mocker.patch('freqtrade.optimize.load_data') mock_fmin = mocker.patch('freqtrade.optimize.hyperopt.fmin', return_value={}) - args = mocker.Mock(epochs=1, config='config.json.example', mongodb=False) + args = mocker.Mock(epochs=1, config='config.json.example', mongodb=False, + timerange=None) start(args) mock_fmin.assert_called_once() @@ -70,11 +72,12 @@ def test_start_calls_fmin(mocker): def test_start_uses_mongotrials(mocker): mock_mongotrials = mocker.patch('freqtrade.optimize.hyperopt.MongoTrials', return_value=create_trials(mocker)) - mocker.patch('freqtrade.optimize.preprocess') + mocker.patch('freqtrade.optimize.tickerdata_to_dataframe') mocker.patch('freqtrade.optimize.load_data') mocker.patch('freqtrade.optimize.hyperopt.fmin', return_value={}) - args = mocker.Mock(epochs=1, config='config.json.example', mongodb=True) + args = mocker.Mock(epochs=1, config='config.json.example', mongodb=True, + timerange=None) start(args) mock_mongotrials.assert_called_once() @@ -125,11 +128,12 @@ def test_fmin_best_results(mocker, caplog): } mocker.patch('freqtrade.optimize.hyperopt.MongoTrials', return_value=create_trials(mocker)) - mocker.patch('freqtrade.optimize.preprocess') + mocker.patch('freqtrade.optimize.tickerdata_to_dataframe') mocker.patch('freqtrade.optimize.load_data') mocker.patch('freqtrade.optimize.hyperopt.fmin', return_value=fmin_result) - args = mocker.Mock(epochs=1, config='config.json.example') + args = mocker.Mock(epochs=1, config='config.json.example', + timerange=None) start(args) exists = [ @@ -147,11 +151,12 @@ def test_fmin_best_results(mocker, caplog): def test_fmin_throw_value_error(mocker, caplog): mocker.patch('freqtrade.optimize.hyperopt.MongoTrials', return_value=create_trials(mocker)) - mocker.patch('freqtrade.optimize.preprocess') + mocker.patch('freqtrade.optimize.tickerdata_to_dataframe') mocker.patch('freqtrade.optimize.load_data') mocker.patch('freqtrade.optimize.hyperopt.fmin', side_effect=ValueError()) - args = mocker.Mock(epochs=1, config='config.json.example') + args = mocker.Mock(epochs=1, config='config.json.example', + timerange=None) start(args) exists = [ @@ -185,7 +190,8 @@ def test_resuming_previous_hyperopt_results_succeeds(mocker): return_value={}) args = mocker.Mock(epochs=1, config='config.json.example', - mongodb=False) + mongodb=False, + timerange=None) start(args) diff --git a/freqtrade/tests/optimize/test_optimize.py b/freqtrade/tests/optimize/test_optimize.py index 66c0dc820..282016b45 100644 --- a/freqtrade/tests/optimize/test_optimize.py +++ b/freqtrade/tests/optimize/test_optimize.py @@ -129,22 +129,25 @@ def test_download_pairs(default_conf, ticker_history, mocker): _backup_file(file1_5) _backup_file(file2_1) _backup_file(file2_5) + + assert os.path.isfile(file1_1) is False + assert os.path.isfile(file2_1) is False assert download_pairs(None, pairs=['BTC-MEME', 'BTC-CFI'], ticker_interval=1) is True assert os.path.isfile(file1_1) is True - assert os.path.isfile(file1_5) is False assert os.path.isfile(file2_1) is True - assert os.path.isfile(file2_5) is False # clean files freshly downloaded _clean_test_file(file1_1) _clean_test_file(file2_1) + + assert os.path.isfile(file1_5) is False + assert os.path.isfile(file2_5) is False assert download_pairs(None, pairs=['BTC-MEME', 'BTC-CFI'], ticker_interval=5) is True - assert os.path.isfile(file1_1) is False + assert os.path.isfile(file1_5) is True - assert os.path.isfile(file2_1) is False assert os.path.isfile(file2_5) is True # clean files freshly downloaded @@ -199,3 +202,11 @@ def test_load_tickerdata_file(): assert not load_tickerdata_file(None, 'BTC_UNITEST', 7) tickerdata = load_tickerdata_file(None, 'BTC_UNITEST', 1) assert _btc_unittest_length == len(tickerdata) + + +def test_tickerdata_to_dataframe(): + timerange = ((None, 'line'), None, -100) + tick = load_tickerdata_file(None, 'BTC_UNITEST', 1, timerange=timerange) + tickerlist = {'BTC_UNITEST': tick} + data = optimize.tickerdata_to_dataframe(tickerlist) + assert 100 == len(data['BTC_UNITEST']) diff --git a/freqtrade/tests/rpc/test_rpc_telegram.py b/freqtrade/tests/rpc/test_rpc_telegram.py index 2e77d6c90..99b6270e0 100644 --- a/freqtrade/tests/rpc/test_rpc_telegram.py +++ b/freqtrade/tests/rpc/test_rpc_telegram.py @@ -77,7 +77,7 @@ def test_authorized_only_exception(default_conf, mocker): def test_status_handle(default_conf, update, ticker, mocker): mocker.patch.dict('freqtrade.main._CONF', default_conf) - mocker.patch('freqtrade.main.get_signal', side_effect=lambda s, t, i: True) + mocker.patch('freqtrade.main.get_signal', side_effect=lambda s: (True, False)) msg_mock = MagicMock() mocker.patch('freqtrade.main.rpc.send_msg', MagicMock()) mocker.patch.multiple('freqtrade.rpc.telegram', @@ -112,7 +112,7 @@ def test_status_handle(default_conf, update, ticker, mocker): def test_status_table_handle(default_conf, update, ticker, mocker): mocker.patch.dict('freqtrade.main._CONF', default_conf) - mocker.patch('freqtrade.main.get_signal', side_effect=lambda s, t, i: True) + mocker.patch('freqtrade.main.get_signal', side_effect=lambda s: (True, False)) msg_mock = MagicMock() mocker.patch('freqtrade.main.rpc.send_msg', MagicMock()) mocker.patch.multiple( @@ -154,7 +154,7 @@ def test_status_table_handle(default_conf, update, ticker, mocker): def test_profit_handle( default_conf, update, ticker, ticker_sell_up, limit_buy_order, limit_sell_order, mocker): mocker.patch.dict('freqtrade.main._CONF', default_conf) - mocker.patch('freqtrade.main.get_signal', side_effect=lambda s, t, i: True) + mocker.patch('freqtrade.main.get_signal', side_effect=lambda s: (True, False)) msg_mock = MagicMock() mocker.patch('freqtrade.main.rpc.send_msg', MagicMock()) mocker.patch.multiple('freqtrade.rpc.telegram', @@ -210,7 +210,7 @@ def test_profit_handle( def test_forcesell_handle(default_conf, update, ticker, ticker_sell_up, mocker): mocker.patch.dict('freqtrade.main._CONF', default_conf) - mocker.patch('freqtrade.main.get_signal', side_effect=lambda s, t, i: True) + mocker.patch('freqtrade.main.get_signal', side_effect=lambda s: (True, False)) rpc_mock = mocker.patch('freqtrade.main.rpc.send_msg', MagicMock()) mocker.patch.multiple('freqtrade.rpc.telegram', _CONF=default_conf, @@ -247,7 +247,7 @@ def test_forcesell_handle(default_conf, update, ticker, ticker_sell_up, mocker): def test_forcesell_down_handle(default_conf, update, ticker, ticker_sell_down, mocker): mocker.patch.dict('freqtrade.main._CONF', default_conf) - mocker.patch('freqtrade.main.get_signal', side_effect=lambda s, t, i: True) + mocker.patch('freqtrade.main.get_signal', side_effect=lambda s: (True, False)) rpc_mock = mocker.patch('freqtrade.main.rpc.send_msg', MagicMock()) mocker.patch.multiple('freqtrade.rpc.telegram', _CONF=default_conf, @@ -308,7 +308,7 @@ def test_exec_forcesell_open_orders(default_conf, ticker, mocker): def test_forcesell_all_handle(default_conf, update, ticker, mocker): mocker.patch.dict('freqtrade.main._CONF', default_conf) - mocker.patch('freqtrade.main.get_signal', side_effect=lambda s, t, i: True) + mocker.patch('freqtrade.main.get_signal', side_effect=lambda s: (True, False)) rpc_mock = mocker.patch('freqtrade.main.rpc.send_msg', MagicMock()) mocker.patch.multiple('freqtrade.rpc.telegram', _CONF=default_conf, @@ -339,7 +339,7 @@ def test_forcesell_all_handle(default_conf, update, ticker, mocker): def test_forcesell_handle_invalid(default_conf, update, mocker): mocker.patch.dict('freqtrade.main._CONF', default_conf) - mocker.patch('freqtrade.main.get_signal', side_effect=lambda s, t, i: True) + mocker.patch('freqtrade.main.get_signal', side_effect=lambda s: (True, True)) msg_mock = MagicMock() mocker.patch.multiple('freqtrade.rpc.telegram', _CONF=default_conf, @@ -376,7 +376,7 @@ def test_forcesell_handle_invalid(default_conf, update, mocker): def test_performance_handle( default_conf, update, ticker, limit_buy_order, limit_sell_order, mocker): mocker.patch.dict('freqtrade.main._CONF', default_conf) - mocker.patch('freqtrade.main.get_signal', side_effect=lambda s, t, i: True) + mocker.patch('freqtrade.main.get_signal', side_effect=lambda s: (True, False)) msg_mock = MagicMock() mocker.patch('freqtrade.main.rpc.send_msg', MagicMock()) mocker.patch.multiple('freqtrade.rpc.telegram', @@ -410,7 +410,7 @@ def test_performance_handle( def test_daily_handle( default_conf, update, ticker, limit_buy_order, limit_sell_order, mocker): mocker.patch.dict('freqtrade.main._CONF', default_conf) - mocker.patch('freqtrade.main.get_signal', side_effect=lambda s, t, i: True) + mocker.patch('freqtrade.main.get_signal', side_effect=lambda s: (True, False)) msg_mock = MagicMock() mocker.patch('freqtrade.main.rpc.send_msg', MagicMock()) mocker.patch.multiple('freqtrade.rpc.telegram', @@ -448,6 +448,28 @@ def test_daily_handle( assert str(datetime.utcnow().date()) in msg_mock.call_args_list[0][0][0] assert str(' 0.00006217 BTC') in msg_mock.call_args_list[0][0][0] assert str(' 0.933 USD') in msg_mock.call_args_list[0][0][0] + assert str(' 1 trade') in msg_mock.call_args_list[0][0][0] + assert str(' 0 trade') in msg_mock.call_args_list[0][0][0] + + # Reset msg_mock + msg_mock.reset_mock() + # Add two other trades + create_trade(0.001, int(default_conf['ticker_interval'])) + create_trade(0.001, int(default_conf['ticker_interval'])) + + trades = Trade.query.all() + for trade in trades: + trade.update(limit_buy_order) + trade.update(limit_sell_order) + trade.close_date = datetime.utcnow() + trade.is_open = False + + update.message.text = '/daily 1' + + _daily(bot=MagicMock(), update=update) + assert str(' 0.00018651 BTC') in msg_mock.call_args_list[0][0][0] + assert str(' 2.798 USD') in msg_mock.call_args_list[0][0][0] + assert str(' 3 trades') in msg_mock.call_args_list[0][0][0] # Try invalid data msg_mock.reset_mock() @@ -460,7 +482,7 @@ def test_daily_handle( def test_count_handle(default_conf, update, ticker, mocker): mocker.patch.dict('freqtrade.main._CONF', default_conf) - mocker.patch('freqtrade.main.get_signal', side_effect=lambda s, t, i: True) + mocker.patch('freqtrade.main.get_signal', side_effect=lambda s: (True, False)) msg_mock = MagicMock() mocker.patch.multiple( 'freqtrade.rpc.telegram', @@ -492,7 +514,7 @@ def test_count_handle(default_conf, update, ticker, mocker): def test_performance_handle_invalid(default_conf, update, mocker): mocker.patch.dict('freqtrade.main._CONF', default_conf) - mocker.patch('freqtrade.main.get_signal', side_effect=lambda s, t, i: True) + mocker.patch('freqtrade.main.get_signal', side_effect=lambda s: (True, True)) msg_mock = MagicMock() mocker.patch.multiple('freqtrade.rpc.telegram', _CONF=default_conf, diff --git a/freqtrade/tests/test_analyze.py b/freqtrade/tests/test_analyze.py index 5ff313897..e5c625e78 100644 --- a/freqtrade/tests/test_analyze.py +++ b/freqtrade/tests/test_analyze.py @@ -6,7 +6,7 @@ import arrow import pytest from pandas import DataFrame -from freqtrade.analyze import (SignalType, get_signal, parse_ticker_dataframe, +from freqtrade.analyze import (get_signal, parse_ticker_dataframe, populate_buy_trend, populate_indicators, populate_sell_trend) @@ -40,30 +40,30 @@ def test_returns_latest_buy_signal(mocker): mocker.patch('freqtrade.analyze.get_ticker_history', return_value=MagicMock()) mocker.patch( 'freqtrade.analyze.analyze_ticker', - return_value=DataFrame([{'buy': 1, 'date': arrow.utcnow()}]) + return_value=DataFrame([{'buy': 1, 'sell': 0, 'date': arrow.utcnow()}]) ) - assert get_signal('BTC-ETH', SignalType.BUY, 5) + assert get_signal('BTC-ETH', 5) == (True, False) mocker.patch( 'freqtrade.analyze.analyze_ticker', - return_value=DataFrame([{'buy': 0, 'date': arrow.utcnow()}]) + return_value=DataFrame([{'buy': 0, 'sell': 1, 'date': arrow.utcnow()}]) ) - assert not get_signal('BTC-ETH', SignalType.BUY, 5) + assert get_signal('BTC-ETH',5) == (False, True) def test_returns_latest_sell_signal(mocker): mocker.patch('freqtrade.analyze.get_ticker_history', return_value=MagicMock()) mocker.patch( 'freqtrade.analyze.analyze_ticker', - return_value=DataFrame([{'sell': 1, 'date': arrow.utcnow()}]) + return_value=DataFrame([{'sell': 1, 'buy': 0, 'date': arrow.utcnow()}]) ) - assert get_signal('BTC-ETH', SignalType.SELL, 5) + assert get_signal('BTC-ETH', 5) == (False, True) mocker.patch( 'freqtrade.analyze.analyze_ticker', - return_value=DataFrame([{'sell': 0, 'date': arrow.utcnow()}]) + return_value=DataFrame([{'sell': 0, 'buy': 1, 'date': arrow.utcnow()}]) ) - assert not get_signal('BTC-ETH', SignalType.SELL, 5) + assert get_signal('BTC-ETH', 5) == (True, False) def test_get_signal_handles_exceptions(mocker): @@ -71,4 +71,4 @@ def test_get_signal_handles_exceptions(mocker): mocker.patch('freqtrade.analyze.analyze_ticker', side_effect=Exception('invalid ticker history ')) - assert not get_signal('BTC-ETH', SignalType.BUY, 5) + assert get_signal('BTC-ETH', 5) == (False, False) diff --git a/freqtrade/tests/test_main.py b/freqtrade/tests/test_main.py index 19ac8d1b5..0ce22db30 100644 --- a/freqtrade/tests/test_main.py +++ b/freqtrade/tests/test_main.py @@ -10,7 +10,6 @@ from sqlalchemy import create_engine import freqtrade.main as main from freqtrade import DependencyException, OperationalException -from freqtrade.analyze import SignalType from freqtrade.exchange import Exchanges from freqtrade.main import (_process, check_handle_timedout, create_trade, execute_sell, get_target_bid, handle_trade, init) @@ -52,7 +51,7 @@ def test_main_start_hyperopt(mocker): def test_process_trade_creation(default_conf, ticker, limit_buy_order, health, mocker): mocker.patch.dict('freqtrade.main._CONF', default_conf) mocker.patch.multiple('freqtrade.rpc', init=MagicMock(), send_msg=MagicMock()) - mocker.patch('freqtrade.main.get_signal', side_effect=lambda s, t, i: True) + mocker.patch('freqtrade.main.get_signal', side_effect=lambda s: (True, False)) mocker.patch.multiple('freqtrade.main.exchange', validate_pairs=MagicMock(), get_ticker=ticker, @@ -82,7 +81,7 @@ def test_process_trade_creation(default_conf, ticker, limit_buy_order, health, m def test_process_exchange_failures(default_conf, ticker, health, mocker): mocker.patch.dict('freqtrade.main._CONF', default_conf) mocker.patch.multiple('freqtrade.rpc', init=MagicMock(), send_msg=MagicMock()) - mocker.patch('freqtrade.main.get_signal', side_effect=lambda s, t, i: True) + mocker.patch('freqtrade.main.get_signal', side_effect=lambda s: (True, False)) sleep_mock = mocker.patch('time.sleep', side_effect=lambda _: None) mocker.patch.multiple('freqtrade.main.exchange', validate_pairs=MagicMock(), @@ -99,7 +98,7 @@ def test_process_operational_exception(default_conf, ticker, health, mocker): msg_mock = MagicMock() mocker.patch.dict('freqtrade.main._CONF', default_conf) mocker.patch.multiple('freqtrade.rpc', init=MagicMock(), send_msg=msg_mock) - mocker.patch('freqtrade.main.get_signal', side_effect=lambda s, t, i: True) + mocker.patch('freqtrade.main.get_signal', side_effect=lambda s: (True, False)) mocker.patch.multiple('freqtrade.main.exchange', validate_pairs=MagicMock(), get_ticker=ticker, @@ -117,8 +116,7 @@ def test_process_operational_exception(default_conf, ticker, health, mocker): def test_process_trade_handling(default_conf, ticker, limit_buy_order, health, mocker): mocker.patch.dict('freqtrade.main._CONF', default_conf) mocker.patch.multiple('freqtrade.rpc', init=MagicMock(), send_msg=MagicMock()) - mocker.patch('freqtrade.main.get_signal', - side_effect=lambda *args: False if args[1] == SignalType.SELL else True) + mocker.patch('freqtrade.main.get_signal', side_effect=lambda s: (True, False)) mocker.patch.multiple('freqtrade.main.exchange', validate_pairs=MagicMock(), get_ticker=ticker, @@ -140,7 +138,7 @@ def test_process_trade_handling(default_conf, ticker, limit_buy_order, health, m def test_create_trade(default_conf, ticker, limit_buy_order, mocker): mocker.patch.dict('freqtrade.main._CONF', default_conf) - mocker.patch('freqtrade.main.get_signal', side_effect=lambda s, t, i: True) + mocker.patch('freqtrade.main.get_signal', side_effect=lambda s: (True, False)) mocker.patch.multiple('freqtrade.rpc', init=MagicMock(), send_msg=MagicMock()) mocker.patch.multiple('freqtrade.main.exchange', validate_pairs=MagicMock(), @@ -171,7 +169,7 @@ def test_create_trade(default_conf, ticker, limit_buy_order, mocker): def test_create_trade_minimal_amount(default_conf, ticker, mocker): mocker.patch.dict('freqtrade.main._CONF', default_conf) mocker.patch.multiple('freqtrade.rpc', init=MagicMock(), send_msg=MagicMock()) - mocker.patch('freqtrade.main.get_signal', side_effect=lambda s, t, i: True) + mocker.patch('freqtrade.main.get_signal', side_effect=lambda s: (True, False)) buy_mock = mocker.patch( 'freqtrade.main.exchange.buy', MagicMock(return_value='mocked_limit_buy') ) @@ -187,7 +185,7 @@ def test_create_trade_minimal_amount(default_conf, ticker, mocker): def test_create_trade_no_stake_amount(default_conf, ticker, mocker): mocker.patch.dict('freqtrade.main._CONF', default_conf) - mocker.patch('freqtrade.main.get_signal', side_effect=lambda s, t, i: True) + mocker.patch('freqtrade.main.get_signal', side_effect=lambda s: (True, False)) mocker.patch.multiple('freqtrade.rpc', init=MagicMock(), send_msg=MagicMock()) mocker.patch.multiple('freqtrade.main.exchange', validate_pairs=MagicMock(), @@ -200,7 +198,7 @@ def test_create_trade_no_stake_amount(default_conf, ticker, mocker): def test_create_trade_no_pairs(default_conf, ticker, mocker): mocker.patch.dict('freqtrade.main._CONF', default_conf) - mocker.patch('freqtrade.main.get_signal', side_effect=lambda s, t, i: True) + mocker.patch('freqtrade.main.get_signal', side_effect=lambda s: (True, False)) mocker.patch.multiple('freqtrade.rpc', init=MagicMock(), send_msg=MagicMock()) mocker.patch.multiple('freqtrade.main.exchange', validate_pairs=MagicMock(), @@ -216,7 +214,7 @@ def test_create_trade_no_pairs(default_conf, ticker, mocker): def test_create_trade_no_pairs_after_blacklist(default_conf, ticker, mocker): mocker.patch.dict('freqtrade.main._CONF', default_conf) - mocker.patch('freqtrade.main.get_signal', side_effect=lambda s, t, i: True) + mocker.patch('freqtrade.main.get_signal', side_effect=lambda s: (True, False)) mocker.patch.multiple('freqtrade.rpc', init=MagicMock(), send_msg=MagicMock()) mocker.patch.multiple('freqtrade.main.exchange', validate_pairs=MagicMock(), @@ -233,7 +231,7 @@ def test_create_trade_no_pairs_after_blacklist(default_conf, ticker, mocker): def test_handle_trade(default_conf, limit_buy_order, limit_sell_order, mocker): mocker.patch.dict('freqtrade.main._CONF', default_conf) - mocker.patch('freqtrade.main.get_signal', side_effect=lambda s, t, i: True) + mocker.patch('freqtrade.main.get_signal', side_effect=lambda s: (True, False)) mocker.patch.multiple('freqtrade.rpc', init=MagicMock(), send_msg=MagicMock()) mocker.patch.multiple('freqtrade.main.exchange', validate_pairs=MagicMock(), @@ -256,6 +254,7 @@ def test_handle_trade(default_conf, limit_buy_order, limit_sell_order, mocker): trade.update(limit_buy_order) assert trade.is_open is True + mocker.patch('freqtrade.main.get_signal', side_effect=lambda s: (False, True)) assert handle_trade(trade, int(default_conf['ticker_interval'])) is True assert trade.open_order_id == 'mocked_limit_sell' @@ -268,11 +267,57 @@ def test_handle_trade(default_conf, limit_buy_order, limit_sell_order, mocker): assert trade.close_date is not None +def test_handle_overlpapping_signals(default_conf, ticker, mocker, caplog): + default_conf.update({'experimental': {'use_sell_signal': True}}) + mocker.patch.dict('freqtrade.main._CONF', default_conf) + + mocker.patch('freqtrade.main.get_signal', side_effect=lambda s: (True, True)) + mocker.patch.multiple('freqtrade.rpc', init=MagicMock(), send_msg=MagicMock()) + mocker.patch.multiple('freqtrade.main.exchange', + validate_pairs=MagicMock(), + get_ticker=ticker, + buy=MagicMock(return_value='mocked_limit_buy')) + mocker.patch('freqtrade.main.min_roi_reached', return_value=False) + + init(default_conf, create_engine('sqlite://')) + create_trade(0.001, int(default_conf['ticker_interval'])) + + # Buy and Sell triggering, so doing nothing ... + trades = Trade.query.all() + assert len(trades) == 0 + + # Buy is triggering, so buying ... + mocker.patch('freqtrade.main.get_signal', side_effect=lambda s: (True, False)) + create_trade(0.001, int(default_conf['ticker_interval'])) + trades = Trade.query.all() + assert len(trades) == 1 + assert trades[0].is_open is True + + # Buy and Sell are not triggering, so doing nothing ... + mocker.patch('freqtrade.main.get_signal', side_effect=lambda s: (False, False)) + assert handle_trade(trades[0], int(default_conf['ticker_interval'])) is False + trades = Trade.query.all() + assert len(trades) == 1 + assert trades[0].is_open is True + + # Buy and Sell are triggering, so doing nothing ... + mocker.patch('freqtrade.main.get_signal', side_effect=lambda s: (True, True)) + assert handle_trade(trades[0], int(default_conf['ticker_interval'])) is False + trades = Trade.query.all() + assert len(trades) == 1 + assert trades[0].is_open is True + + # Sell is triggering, guess what : we are Selling! + mocker.patch('freqtrade.main.get_signal', side_effect=lambda s: (False, True)) + trades = Trade.query.all() + assert handle_trade(trades[0], int(default_conf['ticker_interval'])) is True + + def test_handle_trade_roi(default_conf, ticker, mocker, caplog): default_conf.update({'experimental': {'use_sell_signal': True}}) mocker.patch.dict('freqtrade.main._CONF', default_conf) - mocker.patch('freqtrade.main.get_signal', side_effect=lambda s, t, i: True) + mocker.patch('freqtrade.main.get_signal', side_effect=lambda s: (True, False)) mocker.patch.multiple('freqtrade.rpc', init=MagicMock(), send_msg=MagicMock()) mocker.patch.multiple('freqtrade.main.exchange', validate_pairs=MagicMock(), @@ -291,13 +336,12 @@ def test_handle_trade_roi(default_conf, ticker, mocker, caplog): # we might just want to check if we are in a sell condition without # executing # if ROI is reached we must sell - mocker.patch('freqtrade.main.get_signal', side_effect=lambda s, t, i: False) + mocker.patch('freqtrade.main.get_signal', side_effect=lambda s: (False, True)) assert handle_trade(trade, interval=int(default_conf['ticker_interval'])) assert ('freqtrade', logging.DEBUG, 'Executing sell due to ROI ...') in caplog.record_tuples # if ROI is reached we must sell even if sell-signal is not signalled - mocker.patch('freqtrade.main.get_signal', side_effect=lambda s, t, i: True) + mocker.patch('freqtrade.main.get_signal', side_effect=lambda s: (False, True)) assert handle_trade(trade, interval=int(default_conf['ticker_interval'])) - assert ('freqtrade', logging.DEBUG, 'Executing sell due to ROI ...') in caplog.record_tuples @@ -305,7 +349,7 @@ def test_handle_trade_experimental(default_conf, ticker, mocker, caplog): default_conf.update({'experimental': {'use_sell_signal': True}}) mocker.patch.dict('freqtrade.main._CONF', default_conf) - mocker.patch('freqtrade.main.get_signal', side_effect=lambda s, t, i: True) + mocker.patch('freqtrade.main.get_signal', side_effect=lambda s: (True, False)) mocker.patch.multiple('freqtrade.rpc', init=MagicMock(), send_msg=MagicMock()) mocker.patch.multiple('freqtrade.main.exchange', validate_pairs=MagicMock(), @@ -319,11 +363,10 @@ def test_handle_trade_experimental(default_conf, ticker, mocker, caplog): trade = Trade.query.first() trade.is_open = True - mocker.patch('freqtrade.main.get_signal', side_effect=lambda s, t, i: False) + mocker.patch('freqtrade.main.get_signal', side_effect=lambda s: (False, False)) value_returned = handle_trade(trade, int(default_conf['ticker_interval'])) - assert ('freqtrade', logging.DEBUG, 'Checking sell_signal ...') in caplog.record_tuples assert value_returned is False - mocker.patch('freqtrade.main.get_signal', side_effect=lambda s, t, i: True) + mocker.patch('freqtrade.main.get_signal', side_effect=lambda s: (False, True)) assert handle_trade(trade, int(default_conf['ticker_interval'])) s = 'Executing sell due to sell signal ...' assert ('freqtrade', logging.DEBUG, s) in caplog.record_tuples @@ -331,7 +374,7 @@ def test_handle_trade_experimental(default_conf, ticker, mocker, caplog): def test_close_trade(default_conf, ticker, limit_buy_order, limit_sell_order, mocker): mocker.patch.dict('freqtrade.main._CONF', default_conf) - mocker.patch('freqtrade.main.get_signal', side_effect=lambda s, t, i: True) + mocker.patch('freqtrade.main.get_signal', side_effect=lambda s: (True, False)) mocker.patch.multiple('freqtrade.rpc', init=MagicMock(), send_msg=MagicMock()) mocker.patch.multiple('freqtrade.main.exchange', validate_pairs=MagicMock(), @@ -356,7 +399,8 @@ def test_close_trade(default_conf, ticker, limit_buy_order, limit_sell_order, mo def test_check_handle_timedout_buy(default_conf, ticker, limit_buy_order_old, mocker): mocker.patch.dict('freqtrade.main._CONF', default_conf) cancel_order_mock = MagicMock() - mocker.patch.multiple('freqtrade.rpc', init=MagicMock(), send_msg=MagicMock()) + mocker.patch('freqtrade.rpc.init', MagicMock()) + rpc_mock = mocker.patch('freqtrade.main.rpc.send_msg', MagicMock()) mocker.patch.multiple('freqtrade.main.exchange', validate_pairs=MagicMock(), get_ticker=ticker, @@ -381,6 +425,7 @@ def test_check_handle_timedout_buy(default_conf, ticker, limit_buy_order_old, mo # check it does cancel buy orders over the time limit check_handle_timedout(600) assert cancel_order_mock.call_count == 1 + assert rpc_mock.call_count == 1 trades = Trade.query.filter(Trade.open_order_id.is_(trade_buy.open_order_id)).all() assert len(trades) == 0 @@ -388,7 +433,8 @@ def test_check_handle_timedout_buy(default_conf, ticker, limit_buy_order_old, mo def test_check_handle_timedout_sell(default_conf, ticker, limit_sell_order_old, mocker): mocker.patch.dict('freqtrade.main._CONF', default_conf) cancel_order_mock = MagicMock() - mocker.patch.multiple('freqtrade.rpc', init=MagicMock(), send_msg=MagicMock()) + mocker.patch('freqtrade.rpc.init', MagicMock()) + rpc_mock = mocker.patch('freqtrade.main.rpc.send_msg', MagicMock()) mocker.patch.multiple('freqtrade.main.exchange', validate_pairs=MagicMock(), get_ticker=ticker, @@ -414,6 +460,7 @@ def test_check_handle_timedout_sell(default_conf, ticker, limit_sell_order_old, # check it does cancel sell orders over the time limit check_handle_timedout(600) assert cancel_order_mock.call_count == 1 + assert rpc_mock.call_count == 1 assert trade_sell.is_open is True @@ -421,7 +468,8 @@ def test_check_handle_timedout_partial(default_conf, ticker, limit_buy_order_old mocker): mocker.patch.dict('freqtrade.main._CONF', default_conf) cancel_order_mock = MagicMock() - mocker.patch.multiple('freqtrade.rpc', init=MagicMock(), send_msg=MagicMock()) + mocker.patch('freqtrade.rpc.init', MagicMock()) + rpc_mock = mocker.patch('freqtrade.main.rpc.send_msg', MagicMock()) mocker.patch.multiple('freqtrade.main.exchange', validate_pairs=MagicMock(), get_ticker=ticker, @@ -447,6 +495,7 @@ def test_check_handle_timedout_partial(default_conf, ticker, limit_buy_order_old # note this is for a partially-complete buy order check_handle_timedout(600) assert cancel_order_mock.call_count == 1 + assert rpc_mock.call_count == 1 trades = Trade.query.filter(Trade.open_order_id.is_(trade_buy.open_order_id)).all() assert len(trades) == 1 assert trades[0].amount == 23.0 @@ -470,7 +519,7 @@ def test_balance_bigger_last_ask(mocker): def test_execute_sell_up(default_conf, ticker, ticker_sell_up, mocker): mocker.patch.dict('freqtrade.main._CONF', default_conf) - mocker.patch('freqtrade.main.get_signal', side_effect=lambda s, t, i: True) + mocker.patch('freqtrade.main.get_signal', side_effect=lambda s: (True, False)) mocker.patch('freqtrade.rpc.init', MagicMock()) rpc_mock = mocker.patch('freqtrade.main.rpc.send_msg', MagicMock()) mocker.patch.multiple('freqtrade.main.exchange', @@ -503,7 +552,7 @@ def test_execute_sell_up(default_conf, ticker, ticker_sell_up, mocker): def test_execute_sell_down(default_conf, ticker, ticker_sell_down, mocker): mocker.patch.dict('freqtrade.main._CONF', default_conf) - mocker.patch('freqtrade.main.get_signal', side_effect=lambda s, t, i: True) + mocker.patch('freqtrade.main.get_signal', side_effect=lambda s: (True, False)) mocker.patch('freqtrade.rpc.init', MagicMock()) rpc_mock = mocker.patch('freqtrade.main.rpc.send_msg', MagicMock()) mocker.patch.multiple('freqtrade.rpc.telegram', @@ -540,7 +589,7 @@ def test_execute_sell_down(default_conf, ticker, ticker_sell_down, mocker): def test_execute_sell_without_conf_sell_down(default_conf, ticker, ticker_sell_down, mocker): mocker.patch.dict('freqtrade.main._CONF', default_conf) - mocker.patch('freqtrade.main.get_signal', side_effect=lambda s, t, i: True) + mocker.patch('freqtrade.main.get_signal', side_effect=lambda s,: (True, False)) mocker.patch('freqtrade.rpc.init', MagicMock()) rpc_mock = mocker.patch('freqtrade.main.rpc.send_msg', MagicMock()) mocker.patch.multiple('freqtrade.main.exchange', @@ -572,7 +621,7 @@ def test_execute_sell_without_conf_sell_down(default_conf, ticker, ticker_sell_d def test_execute_sell_without_conf_sell_up(default_conf, ticker, ticker_sell_up, mocker): mocker.patch.dict('freqtrade.main._CONF', default_conf) - mocker.patch('freqtrade.main.get_signal', side_effect=lambda s, t, i: True) + mocker.patch('freqtrade.main.get_signal', side_effect=lambda s: (True, False)) mocker.patch('freqtrade.rpc.init', MagicMock()) rpc_mock = mocker.patch('freqtrade.main.rpc.send_msg', MagicMock()) mocker.patch.multiple('freqtrade.main.exchange', @@ -609,7 +658,7 @@ def test_sell_profit_only_enable_profit(default_conf, limit_buy_order, mocker): mocker.patch.dict('freqtrade.main._CONF', default_conf) mocker.patch('freqtrade.main.min_roi_reached', return_value=False) - mocker.patch('freqtrade.main.get_signal', side_effect=lambda s, t, i: True) + mocker.patch('freqtrade.main.get_signal', side_effect=lambda s: (True, False)) mocker.patch.multiple('freqtrade.rpc', init=MagicMock(), send_msg=MagicMock()) mocker.patch.multiple('freqtrade.main.exchange', validate_pairs=MagicMock(), @@ -625,6 +674,7 @@ def test_sell_profit_only_enable_profit(default_conf, limit_buy_order, mocker): trade = Trade.query.first() trade.update(limit_buy_order) + mocker.patch('freqtrade.main.get_signal', side_effect=lambda s: (False, True)) assert handle_trade(trade, int(default_conf['ticker_interval'])) is True @@ -636,7 +686,7 @@ def test_sell_profit_only_disable_profit(default_conf, limit_buy_order, mocker): mocker.patch.dict('freqtrade.main._CONF', default_conf) mocker.patch('freqtrade.main.min_roi_reached', return_value=False) - mocker.patch('freqtrade.main.get_signal', side_effect=lambda s, t, i: True) + mocker.patch('freqtrade.main.get_signal', side_effect=lambda s: (True, False)) mocker.patch.multiple('freqtrade.rpc', init=MagicMock(), send_msg=MagicMock()) mocker.patch.multiple('freqtrade.main.exchange', validate_pairs=MagicMock(), @@ -652,6 +702,7 @@ def test_sell_profit_only_disable_profit(default_conf, limit_buy_order, mocker): trade = Trade.query.first() trade.update(limit_buy_order) + mocker.patch('freqtrade.main.get_signal', side_effect=lambda s: (False, True)) assert handle_trade(trade, int(default_conf['ticker_interval'])) is True @@ -663,7 +714,7 @@ def test_sell_profit_only_enable_loss(default_conf, limit_buy_order, mocker): mocker.patch.dict('freqtrade.main._CONF', default_conf) mocker.patch('freqtrade.main.min_roi_reached', return_value=False) - mocker.patch('freqtrade.main.get_signal', side_effect=lambda s, t, i: True) + mocker.patch('freqtrade.main.get_signal', side_effect=lambda s: (True, False)) mocker.patch.multiple('freqtrade.rpc', init=MagicMock(), send_msg=MagicMock()) mocker.patch.multiple('freqtrade.main.exchange', validate_pairs=MagicMock(), @@ -679,6 +730,7 @@ def test_sell_profit_only_enable_loss(default_conf, limit_buy_order, mocker): trade = Trade.query.first() trade.update(limit_buy_order) + mocker.patch('freqtrade.main.get_signal', side_effect=lambda s: (False, True)) assert handle_trade(trade, int(default_conf['ticker_interval'])) is False @@ -690,7 +742,7 @@ def test_sell_profit_only_disable_loss(default_conf, limit_buy_order, mocker): mocker.patch.dict('freqtrade.main._CONF', default_conf) mocker.patch('freqtrade.main.min_roi_reached', return_value=False) - mocker.patch('freqtrade.main.get_signal', side_effect=lambda s, t, i: True) + mocker.patch('freqtrade.main.get_signal', side_effect=lambda s: (True, False)) mocker.patch.multiple('freqtrade.rpc', init=MagicMock(), send_msg=MagicMock()) mocker.patch.multiple('freqtrade.main.exchange', validate_pairs=MagicMock(), @@ -706,4 +758,5 @@ def test_sell_profit_only_disable_loss(default_conf, limit_buy_order, mocker): trade = Trade.query.first() trade.update(limit_buy_order) + mocker.patch('freqtrade.main.get_signal', side_effect=lambda s: (False, True)) assert handle_trade(trade, int(default_conf['ticker_interval'])) is True diff --git a/freqtrade/tests/test_misc.py b/freqtrade/tests/test_misc.py index 63cfba627..fb768b89c 100644 --- a/freqtrade/tests/test_misc.py +++ b/freqtrade/tests/test_misc.py @@ -5,10 +5,11 @@ import time from copy import deepcopy import pytest +from unittest.mock import MagicMock from jsonschema import ValidationError from freqtrade.misc import (common_args_parser, load_config, parse_args, - throttle) + throttle, file_dump_json, parse_timerange) def test_throttle(): @@ -133,6 +134,21 @@ def test_parse_args_hyperopt_custom(mocker): assert call_args.func is not None +def test_file_dump_json(default_conf, mocker): + file_open = mocker.patch('freqtrade.misc.open', MagicMock()) + json_dump = mocker.patch('json.dump', MagicMock()) + file_dump_json('somefile', [1, 2, 3]) + assert file_open.call_count == 1 + assert json_dump.call_count == 1 + + +def test_parse_timerange_incorrect(): + assert ((None, 'line'), None, -200) == parse_timerange('-200') + assert (('line', None), 200, None) == parse_timerange('200-') + with pytest.raises(Exception, match=r'Incorrect syntax.*'): + parse_timerange('-') + + def test_load_config(default_conf, mocker): file_mock = mocker.patch('freqtrade.misc.open', mocker.mock_open( read_data=json.dumps(default_conf) diff --git a/requirements.txt b/requirements.txt index d650efea8..d37312268 100644 --- a/requirements.txt +++ b/requirements.txt @@ -1,7 +1,7 @@ python-bittrex==0.2.2 SQLAlchemy==1.2.1 python-telegram-bot==9.0.0 -arrow==0.12.0 +arrow==0.12.1 cachetools==2.0.1 requests==2.18.4 urllib3==1.22 @@ -11,7 +11,7 @@ scikit-learn==0.19.1 scipy==1.0.0 jsonschema==2.6.0 numpy==1.14.0 -TA-Lib==0.4.15 +TA-Lib==0.4.16 pytest==3.3.2 pytest-mock==1.6.3 pytest-cov==2.5.1 diff --git a/scripts/plot_dataframe.py b/scripts/plot_dataframe.py index dc869c27a..d2c32d754 100755 --- a/scripts/plot_dataframe.py +++ b/scripts/plot_dataframe.py @@ -28,7 +28,7 @@ def plot_parse_args(args ): return parser.parse_args(args) -def plot_analyzed_dataframe(args) -> None: +def plot_analyzed_dataframe(args): """ Calls analyze() and plots the returned dataframe :param pair: pair as str diff --git a/scripts/plot_profit.py b/scripts/plot_profit.py new file mode 100755 index 000000000..e8bdbee5c --- /dev/null +++ b/scripts/plot_profit.py @@ -0,0 +1,155 @@ +#!/usr/bin/env python3 + +import sys +import argparse +import json +import matplotlib.pyplot as plt +import numpy as np + +import freqtrade.optimize as optimize +import freqtrade.misc as misc +import freqtrade.exchange as exchange +import freqtrade.analyze as analyze + + +def plot_parse_args(args ): + parser = misc.common_args_parser('Graph utility') + # FIX: perhaps delete those backtesting options that are not feasible (shows up in -h) + misc.backtesting_options(parser) + parser.add_argument( + '-p', '--pair', + help = 'Show profits for only this pairs. Pairs are comma-separated.', + dest = 'pair', + default = None + ) + return parser.parse_args(args) + + +# data:: [ pair, profit-%, enter, exit, time, duration] +# data:: ['BTC_XMR', 0.00537847, '1511176800', '1511178000', 5057, 1] +# FIX: make use of the enter/exit dates to insert the +# profit more precisely into the pg array +def make_profit_array(data, px, filter_pairs=[]): + pg = np.zeros(px) + # Go through the trades + # and make an total profit + # array + for trade in data: + pair = trade[0] + if filter_pairs and pair not in filter_pairs: + continue + profit = trade[1] + tim = trade[4] + dur = trade[5] + pg[tim+dur-1] += profit + + # rewrite the pg array to go from + # total profits at each timeframe + # to accumulated profits + pa = 0 + for x in range(0,len(pg)): + p = pg[x] # Get current total percent + pa += p # Add to the accumulated percent + pg[x] = pa # write back to save memory + + return pg + + +def plot_profit(args) -> None: + """ + Plots the total profit for all pairs. + Note, the profit calculation isn't realistic. + But should be somewhat proportional, and therefor useful + in helping out to find a good algorithm. + """ + + # We need to use the same pairs, same tick_interval + # and same timeperiod as used in backtesting + # to match the tickerdata against the profits-results + + filter_pairs = args.pair + + config = misc.load_config(args.config) + pairs = config['exchange']['pair_whitelist'] + if filter_pairs: + filter_pairs = filter_pairs.split(',') + pairs = list(set(pairs) & set(filter_pairs)) + print('Filter, keep pairs %s' % pairs) + + tickers = optimize.load_data(args.datadir, pairs=pairs, + ticker_interval=args.ticker_interval, + refresh_pairs=False) + dataframes = optimize.preprocess(tickers) + + # Make an average close price of all the pairs that was involved. + # this could be useful to gauge the overall market trend + + # FIX: since the dataframes are of unequal length, + # andor has different dates, we need to merge them + # But we dont have the date information in the + # backtesting results, this is needed to match the dates + # For now, assume the dataframes are aligned. + max_x = 0 + for pair, pair_data in dataframes.items(): + n = len(pair_data['close']) + max_x = max(max_x, n) + # if max_x != n: + # raise Exception('Please rerun script. Input data has different lengths %s' + # %('Different pair length: %s <=> %s' %(max_x, n))) + print('max_x: %s' %(max_x)) + + # We are essentially saying: + # array <- sum dataframes[*]['close'] / num_items dataframes + # FIX: there should be some onliner numpy/panda for this + avgclose = np.zeros(max_x) + num = 0 + for pair, pair_data in dataframes.items(): + close = pair_data['close'] + maxprice = max(close) # Normalize price to [0,1] + print('Pair %s has length %s' %(pair, len(close))) + for x in range(0, len(close)): + avgclose[x] += close[x] / maxprice + # avgclose += close + num += 1 + avgclose /= num + + # Load the profits results + # And make an profits-growth array + + filename = 'backtest-result.json' + with open(filename) as file: + data = json.load(file) + pg = make_profit_array(data, max_x, filter_pairs) + + # + # Plot the pairs average close prices, and total profit growth + # + + fig, (ax1, ax2, ax3) = plt.subplots(3, sharex=True) + fig.suptitle('total profit') + ax1.plot(avgclose, label='avgclose') + ax2.plot(pg, label='profit') + ax1.legend(loc='upper left') + ax2.legend(loc='upper left') + + # FIX if we have one line pair in paris + # then skip the plotting of the third graph, + # or change what we plot + # In third graph, we plot each profit separately + for pair in pairs: + pg = make_profit_array(data, max_x, pair) + ax3.plot(pg, label=pair) + ax3.legend(loc='upper left') + # black background to easier see multiple colors + ax3.set_facecolor('black') + + # Fine-tune figure; make subplots close to each other and hide x ticks for + # all but bottom plot. + fig.subplots_adjust(hspace=0) + plt.setp([a.get_xticklabels() for a in fig.axes[:-1]], visible=False) + plt.show() + + +if __name__ == '__main__': + args = plot_parse_args(sys.argv[1:]) + plot_profit(args)