From 732281bca05320ef35325c5e972dbea4f28163d2 Mon Sep 17 00:00:00 2001 From: Rob Moggach Date: Mon, 8 Jan 2018 20:27:41 -0800 Subject: [PATCH 01/44] public git URL --- docs/installation.md | 42 +++++++++++++++++++++--------------------- 1 file changed, 21 insertions(+), 21 deletions(-) diff --git a/docs/installation.md b/docs/installation.md index 9ea6265ba..88e6bf7e2 100644 --- a/docs/installation.md +++ b/docs/installation.md @@ -1,7 +1,7 @@ # Install the bot -This page explains how to prepare your environment for running the bot. -To understand how to set up the bot please read the Bot -[Bot configuration](https://github.com/gcarq/freqtrade/blob/develop/docs/configuration.md) +This page explains how to prepare your environment for running the bot. +To understand how to set up the bot please read the Bot +[Bot configuration](https://github.com/gcarq/freqtrade/blob/develop/docs/configuration.md) page. ## Table of Contents @@ -21,14 +21,14 @@ Start by downloading Docker for your platform: - [Windows](https://www.docker.com/products/docker#/windows) - [Linux](https://www.docker.com/products/docker#/linux) -Once you have Docker installed, simply create the config file -(e.g. `config.json`) and then create a Docker image for `freqtrade` +Once you have Docker installed, simply create the config file +(e.g. `config.json`) and then create a Docker image for `freqtrade` using the Dockerfile in this repo. ### 1. Prepare the bot -1. Clone the git +1. Clone the git ```bash -git clone git@github.com:gcarq/freqtrade.git +git clone https://github.com/gcarq/freqtrade.git ``` 2. (Optional) Checkout the develop branch ```bash @@ -42,7 +42,7 @@ cd freqtrade ```bash cp config.json.example config.json ``` -To edit the config please refer to the [Bot Configuration](https://github.com/gcarq/freqtrade/blob/develop/docs/configuration.md) page +To edit the config please refer to the [Bot Configuration](https://github.com/gcarq/freqtrade/blob/develop/docs/configuration.md) page 5. Create your DB file (Optional, the bot will create it if it is missing) ```bash # For Production @@ -60,7 +60,7 @@ docker build -t freqtrade . For security reasons, your configuration file will not be included in the image, you will need to bind mount it. It is also advised to bind mount -a sqlite database file (see the "5. Run a restartable docker image" +a sqlite database file (see the "5. Run a restartable docker image" section) to keep it between updates. ### 3. Verify the docker image @@ -100,7 +100,7 @@ docker run -d \ -v ~/.freqtrade/tradesv3.sqlite:/freqtrade/tradesv3.sqlite \ freqtrade ``` -If you are using `dry_run=True` it's not necessary to mount +If you are using `dry_run=True` it's not necessary to mount `tradesv3.sqlite`, but you can mount `tradesv3.dryrun.sqlite` if you plan to use the dry run mode with the param `--dry-run-db`. @@ -116,14 +116,14 @@ docker stop freqtrade docker start freqtrade ``` -You do not need to rebuild the image for configuration changes, it will +You do not need to rebuild the image for configuration changes, it will suffice to edit `config.json` and restart the container. # Linux / MacOS ## 1. Requirements Click each one for install guide: -- [Python 3.6.x](http://docs.python-guide.org/en/latest/starting/installation/), +- [Python 3.6.x](http://docs.python-guide.org/en/latest/starting/installation/), note the bot was not tested on Python >= 3.7.x - [pip](https://pip.pypa.io/en/stable/installing/) - [git](https://git-scm.com/book/en/v2/Getting-Started-Installing-Git) @@ -142,7 +142,7 @@ sudo apt-get update sudo apt-get install python3.6 python3.6-venv build-essential autoconf libtool pkg-config make wget git ``` -**2.1.2. Install TA-LIB** +**2.1.2. Install TA-LIB** Official webpage: https://mrjbq7.github.io/ta-lib/install.html ``` wget http://prdownloads.sourceforge.net/ta-lib/ta-lib-0.4.0-src.tar.gz @@ -155,7 +155,7 @@ cd .. rm -rf ./ta-lib* ``` -**2.1.3. [Optional] Install MongoDB** +**2.1.3. [Optional] Install MongoDB** Install MongoDB if you plan to optimize your strategy with Hyperopt. ```bash @@ -176,7 +176,7 @@ If you are on a different Linux OS you maybe have to adapt things like: brew install python3 git wget ``` -**2.3.2. [Optional] Install MongoDB** +**2.3.2. [Optional] Install MongoDB** Install MongoDB if you plan to optimize your strategy with Hyperopt. ```bash curl -O https://fastdl.mongodb.org/osx/mongodb-osx-ssl-x86_64-3.4.10.tgz @@ -188,15 +188,15 @@ export PATH=/env/mongodb/bin:$PATH ## 3. Clone the repo The following steps are made for Linux/mac environment -1. Clone the git `git clone git@github.com:gcarq/freqtrade.git` -2. (Optional) Checkout the develop branch `git checkout develop` +1. Clone the git `git clone https://github.com/gcarq/freqtrade.git` +2. (Optional) Checkout the develop branch `git checkout develop` ## 4. Prepare the bot ```bash cd freqtrade cp config.json.example config.json ``` -To edit the config please refer to [Bot Configuration](https://github.com/gcarq/freqtrade/blob/develop/docs/configuration.md) +To edit the config please refer to [Bot Configuration](https://github.com/gcarq/freqtrade/blob/develop/docs/configuration.md) ## 5. Setup your virtual env ```bash @@ -215,7 +215,7 @@ python3.6 ./freqtrade/main.py -c config.json ``` ### Advanced Linux -**systemd service file** +**systemd service file** Copy `./freqtrade.service` to your systemd user directory (usually `~/.config/systemd/user`) and update `WorkingDirectory` and `ExecStart` to match your setup. After that you can start the daemon with: @@ -224,7 +224,7 @@ systemctl --user start freqtrade ``` # Windows -We do recommend Windows users to use [Docker](#docker) this will work +We do recommend Windows users to use [Docker](#docker) this will work much easier and smoother (also safer). ```cmd @@ -242,5 +242,5 @@ much easier and smoother (also safer). *Thanks [Owdr](https://github.com/Owdr) for the commands. Source: [Issue #222](https://github.com/gcarq/freqtrade/issues/222)* ## Next step -Now you have an environment ready, the next step is to +Now you have an environment ready, the next step is to [configure your bot](https://github.com/gcarq/freqtrade/blob/develop/docs/configuration.md). From feca87345f241bc906d8f40a578c8a5ec79a0756 Mon Sep 17 00:00:00 2001 From: kryofly Date: Wed, 10 Jan 2018 23:00:40 +0100 Subject: [PATCH 02/44] refactor --- freqtrade/optimize/__init__.py | 5 +++++ freqtrade/optimize/backtesting.py | 9 ++++----- freqtrade/optimize/hyperopt.py | 4 ++-- 3 files changed, 11 insertions(+), 7 deletions(-) diff --git a/freqtrade/optimize/__init__.py b/freqtrade/optimize/__init__.py index 2d73c3215..aff4c372d 100644 --- a/freqtrade/optimize/__init__.py +++ b/freqtrade/optimize/__init__.py @@ -61,6 +61,11 @@ def load_data(datadir: str, ticker_interval: int = 5, pairs: Optional[List[str]] return result +def tickerdata_to_dataframe(data): + preprocessed = preprocess(data) + return preprocessed + + def preprocess(tickerdata: Dict[str, List]) -> Dict[str, DataFrame]: """Creates a dataframe and populates indicators for given ticker data""" return {pair: populate_indicators(parse_ticker_dataframe(pair_data)) diff --git a/freqtrade/optimize/backtesting.py b/freqtrade/optimize/backtesting.py index 315f960d8..5521ee98f 100644 --- a/freqtrade/optimize/backtesting.py +++ b/freqtrade/optimize/backtesting.py @@ -13,7 +13,6 @@ from freqtrade.analyze import populate_buy_trend, populate_sell_trend from freqtrade.exchange import Bittrex from freqtrade.main import min_roi_reached import freqtrade.misc as misc -from freqtrade.optimize import preprocess import freqtrade.optimize as optimize from freqtrade.persistence import Trade @@ -162,12 +161,12 @@ def start(args): data[pair] = exchange.get_ticker_history(pair, args.ticker_interval) else: logger.info('Using local backtesting data (using whitelist in given config) ...') - data = optimize.load_data(args.datadir, pairs=pairs, ticker_interval=args.ticker_interval, - refresh_pairs=args.refresh_pairs) - logger.info('Using stake_currency: %s ...', config['stake_currency']) logger.info('Using stake_amount: %s ...', config['stake_amount']) + data = optimize.load_data(args.datadir, pairs=pairs, ticker_interval=args.ticker_interval, + refresh_pairs=args.refresh_pairs) + max_open_trades = 0 if args.realistic_simulation: logger.info('Using max_open_trades: %s ...', config['max_open_trades']) @@ -177,7 +176,7 @@ def start(args): from freqtrade import main main._CONF = config - preprocessed = preprocess(data) + preprocessed = optimize.tickerdata_to_dataframe(data) # Print timeframe min_date, max_date = get_timeframe(preprocessed) logger.info('Measuring data from %s up to %s ...', min_date.isoformat(), max_date.isoformat()) diff --git a/freqtrade/optimize/hyperopt.py b/freqtrade/optimize/hyperopt.py index cf46b96ad..b5490d72e 100644 --- a/freqtrade/optimize/hyperopt.py +++ b/freqtrade/optimize/hyperopt.py @@ -230,8 +230,8 @@ def start(args): logger.info('Using config: %s ...', args.config) config = load_config(args.config) pairs = config['exchange']['pair_whitelist'] - PROCESSED = optimize.preprocess(optimize.load_data( - args.datadir, pairs=pairs, ticker_interval=args.ticker_interval)) + data = optimize.load_data(args.datadir, pairs=pairs, ticker_interval=args.ticker_interval) + PROCESSED = optimize.tickerdata_to_dataframe(data) if args.mongodb: logger.info('Using mongodb ...') From b0f3fd7ffb434c472a61fadb421f46839e3d8f2a Mon Sep 17 00:00:00 2001 From: kryofly Date: Wed, 10 Jan 2018 23:03:05 +0100 Subject: [PATCH 03/44] timeperiod argument to backtesting and hyperopt --- freqtrade/misc.py | 14 ++++++++++++++ freqtrade/optimize/__init__.py | 11 ++++++++++- freqtrade/optimize/backtesting.py | 2 +- freqtrade/optimize/hyperopt.py | 2 +- freqtrade/tests/optimize/test_backtesting.py | 14 ++++---------- freqtrade/tests/optimize/test_hyperopt.py | 8 ++++---- freqtrade/tests/optimize/test_optimize.py | 7 +++++++ 7 files changed, 41 insertions(+), 17 deletions(-) diff --git a/freqtrade/misc.py b/freqtrade/misc.py index afc4334e8..97e1eca36 100644 --- a/freqtrade/misc.py +++ b/freqtrade/misc.py @@ -183,6 +183,13 @@ def build_subcommands(parser: argparse.ArgumentParser) -> None: action='store_true', dest='refresh_pairs', ) + backtesting_cmd.add_argument( + '-tp', '--timeperiod', + help='Use the last N ticks of data.', + default=None, + type=int, + dest='timeperiod', + ) # Add hyperopt subcommand hyperopt_cmd = subparsers.add_parser('hyperopt', help='hyperopt module') @@ -209,6 +216,13 @@ def build_subcommands(parser: argparse.ArgumentParser) -> None: type=int, metavar='INT', ) + hyperopt_cmd.add_argument( + '-tp', '--timeperiod', + help='Use the last N ticks of data.', + default=None, + type=int, + dest='timeperiod', + ) # Required json-schema for user specified config diff --git a/freqtrade/optimize/__init__.py b/freqtrade/optimize/__init__.py index aff4c372d..3f6a98ac1 100644 --- a/freqtrade/optimize/__init__.py +++ b/freqtrade/optimize/__init__.py @@ -12,6 +12,13 @@ from freqtrade.analyze import populate_indicators, parse_ticker_dataframe logger = logging.getLogger(__name__) +def trim_tickerlist(dl, num): + new = {} + for pair, pair_data in dl.items(): + new[pair] = pair_data[num:] + return new + + def load_tickerdata_file(datadir, pair, ticker_interval): """ Load a pair from file, @@ -61,7 +68,9 @@ def load_data(datadir: str, ticker_interval: int = 5, pairs: Optional[List[str]] return result -def tickerdata_to_dataframe(data): +def tickerdata_to_dataframe(data, timeperiod=None): + if timeperiod: + data = trim_tickerlist(data, timeperiod) preprocessed = preprocess(data) return preprocessed diff --git a/freqtrade/optimize/backtesting.py b/freqtrade/optimize/backtesting.py index 5521ee98f..35b2c6d27 100644 --- a/freqtrade/optimize/backtesting.py +++ b/freqtrade/optimize/backtesting.py @@ -176,7 +176,7 @@ def start(args): from freqtrade import main main._CONF = config - preprocessed = optimize.tickerdata_to_dataframe(data) + preprocessed = optimize.tickerdata_to_dataframe(data, timeperiod=args.timeperiod) # Print timeframe min_date, max_date = get_timeframe(preprocessed) logger.info('Measuring data from %s up to %s ...', min_date.isoformat(), max_date.isoformat()) diff --git a/freqtrade/optimize/hyperopt.py b/freqtrade/optimize/hyperopt.py index b5490d72e..a556258d0 100644 --- a/freqtrade/optimize/hyperopt.py +++ b/freqtrade/optimize/hyperopt.py @@ -231,7 +231,7 @@ def start(args): config = load_config(args.config) pairs = config['exchange']['pair_whitelist'] data = optimize.load_data(args.datadir, pairs=pairs, ticker_interval=args.ticker_interval) - PROCESSED = optimize.tickerdata_to_dataframe(data) + PROCESSED = optimize.tickerdata_to_dataframe(data, timeperiod=args.timeperiod) if args.mongodb: logger.info('Using mongodb ...') diff --git a/freqtrade/tests/optimize/test_backtesting.py b/freqtrade/tests/optimize/test_backtesting.py index 5f899a48a..d4172d983 100644 --- a/freqtrade/tests/optimize/test_backtesting.py +++ b/freqtrade/tests/optimize/test_backtesting.py @@ -6,7 +6,7 @@ import pandas as pd from unittest.mock import MagicMock from freqtrade import exchange, optimize from freqtrade.exchange import Bittrex -from freqtrade.optimize import preprocess +from freqtrade.optimize import preprocess, trim_tickerlist from freqtrade.optimize.backtesting import backtest, generate_text_table, get_timeframe import freqtrade.optimize.backtesting as backtesting @@ -59,16 +59,9 @@ def test_backtest_1min_ticker_interval(default_conf, mocker): assert not results.empty -def trim_dictlist(dl, num): - new = {} - for pair, pair_data in dl.items(): - new[pair] = pair_data[num:] - return new - - def load_data_test(what): data = optimize.load_data(None, ticker_interval=1, pairs=['BTC_UNITEST']) - data = trim_dictlist(data, -100) + data = trim_tickerlist(data, -100) pair = data['BTC_UNITEST'] datalen = len(pair) # Depending on the what parameter we now adjust the @@ -152,7 +145,7 @@ def test_backtest_pricecontours(default_conf, mocker): def mocked_load_data(datadir, pairs=[], ticker_interval=0, refresh_pairs=False): tickerdata = optimize.load_tickerdata_file(datadir, 'BTC_UNITEST', 1) pairdata = {'BTC_UNITEST': tickerdata} - return trim_dictlist(pairdata, -100) + return trim_tickerlist(pairdata, -100) def test_backtest_start(default_conf, mocker, caplog): @@ -166,6 +159,7 @@ def test_backtest_start(default_conf, mocker, caplog): args.level = 10 args.live = False args.datadir = None + args.timeperiod = None # needed due to MagicMock malleability backtesting.start(args) # check the logs, that will contain the backtest result exists = ['Using max_open_trades: 1 ...', diff --git a/freqtrade/tests/optimize/test_hyperopt.py b/freqtrade/tests/optimize/test_hyperopt.py index a309af7fe..f8feda0f8 100644 --- a/freqtrade/tests/optimize/test_hyperopt.py +++ b/freqtrade/tests/optimize/test_hyperopt.py @@ -37,7 +37,7 @@ def create_trials(mocker): def test_start_calls_fmin(mocker): mocker.patch('freqtrade.optimize.hyperopt.Trials', return_value=create_trials(mocker)) - mocker.patch('freqtrade.optimize.preprocess') + mocker.patch('freqtrade.optimize.tickerdata_to_dataframe') mocker.patch('freqtrade.optimize.load_data') mock_fmin = mocker.patch('freqtrade.optimize.hyperopt.fmin', return_value={}) @@ -50,7 +50,7 @@ def test_start_calls_fmin(mocker): def test_start_uses_mongotrials(mocker): mock_mongotrials = mocker.patch('freqtrade.optimize.hyperopt.MongoTrials', return_value=create_trials(mocker)) - mocker.patch('freqtrade.optimize.preprocess') + mocker.patch('freqtrade.optimize.tickerdata_to_dataframe') mocker.patch('freqtrade.optimize.load_data') mocker.patch('freqtrade.optimize.hyperopt.fmin', return_value={}) @@ -104,7 +104,7 @@ def test_fmin_best_results(mocker, caplog): } mocker.patch('freqtrade.optimize.hyperopt.MongoTrials', return_value=create_trials(mocker)) - mocker.patch('freqtrade.optimize.preprocess') + mocker.patch('freqtrade.optimize.tickerdata_to_dataframe') mocker.patch('freqtrade.optimize.load_data') mocker.patch('freqtrade.optimize.hyperopt.fmin', return_value=fmin_result) @@ -126,7 +126,7 @@ def test_fmin_best_results(mocker, caplog): def test_fmin_throw_value_error(mocker, caplog): mocker.patch('freqtrade.optimize.hyperopt.MongoTrials', return_value=create_trials(mocker)) - mocker.patch('freqtrade.optimize.preprocess') + mocker.patch('freqtrade.optimize.tickerdata_to_dataframe') mocker.patch('freqtrade.optimize.load_data') mocker.patch('freqtrade.optimize.hyperopt.fmin', side_effect=ValueError()) diff --git a/freqtrade/tests/optimize/test_optimize.py b/freqtrade/tests/optimize/test_optimize.py index a5892f278..07e9a46b6 100644 --- a/freqtrade/tests/optimize/test_optimize.py +++ b/freqtrade/tests/optimize/test_optimize.py @@ -174,3 +174,10 @@ def test_load_tickerdata_file(): assert not load_tickerdata_file(None, 'BTC_UNITEST', 7) tickerdata = load_tickerdata_file(None, 'BTC_UNITEST', 1) assert _btc_unittest_length == len(tickerdata) + + +def test_tickerdata_to_dataframe(): + tick = load_tickerdata_file(None, 'BTC_UNITEST', 1) + tickerlist = {'BTC_UNITEST': tick} + data = optimize.tickerdata_to_dataframe(tickerlist, timeperiod=-100) + assert 100 == len(data['BTC_UNITEST']) From 94883202b8e108ab0e5244e866638dc2d579a019 Mon Sep 17 00:00:00 2001 From: kryofly Date: Thu, 11 Jan 2018 00:14:36 +0100 Subject: [PATCH 04/44] docs: --timeperiod argument --- docs/backtesting.md | 10 ++++++++++ docs/hyperopt.md | 10 ++++++++++ 2 files changed, 20 insertions(+) diff --git a/docs/backtesting.md b/docs/backtesting.md index c426e2b5c..31745877f 100644 --- a/docs/backtesting.md +++ b/docs/backtesting.md @@ -51,6 +51,16 @@ python3 ./freqtrade/main.py backtesting --realistic-simulation --live python3 ./freqtrade/main.py backtesting --datadir freqtrade/tests/testdata-20180101 ``` +**Running backtest with smaller testset** +Use the --timeperiod argument to change how much of the testset +you want to use. The last N ticks/timeframes will be used. +Example: + +```bash +python3 ./freqtrade/main.py backtesting --timeperiod -200 +``` + + For help about backtesting usage, please refer to [Backtesting commands](#backtesting-commands). diff --git a/docs/hyperopt.md b/docs/hyperopt.md index 24a9dbc51..af564f0b6 100644 --- a/docs/hyperopt.md +++ b/docs/hyperopt.md @@ -168,6 +168,16 @@ If you would like to learn parameters using an alternate ticke-data that you have on-disk, use the --datadir PATH option. Default hyperopt will use data from directory freqtrade/tests/testdata. +### Running hyperopt with smaller testset + +Use the --timeperiod argument to change how much of the testset +you want to use. The last N ticks/timeframes will be used. +Example: + +```bash +python3 ./freqtrade/main.py hyperopt --timeperiod -200 +``` + ### Hyperopt with MongoDB Hyperopt with MongoDB, is like Hyperopt under steroids. As you saw by executing the previous command is the execution takes a long time. From 46a1a2de101cb9f35d83cdfc6794e868a9312409 Mon Sep 17 00:00:00 2001 From: pyup-bot Date: Thu, 11 Jan 2018 20:53:26 +0100 Subject: [PATCH 05/44] Update ta-lib from 0.4.14 to 0.4.15 --- requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/requirements.txt b/requirements.txt index 8a3c8b9cf..e860baedd 100644 --- a/requirements.txt +++ b/requirements.txt @@ -11,7 +11,7 @@ scikit-learn==0.19.1 scipy==1.0.0 jsonschema==2.6.0 numpy==1.14.0 -TA-Lib==0.4.14 +TA-Lib==0.4.15 pytest==3.3.2 pytest-mock==1.6.3 pytest-cov==2.5.1 From 39c6e5263a422588adda63cdb358b947b5cbf988 Mon Sep 17 00:00:00 2001 From: Gerald Lonlas Date: Thu, 11 Jan 2018 21:09:04 -0800 Subject: [PATCH 06/44] Fix plot_dataframe.py --- scripts/plot_dataframe.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scripts/plot_dataframe.py b/scripts/plot_dataframe.py index ce636a4b5..f07033637 100755 --- a/scripts/plot_dataframe.py +++ b/scripts/plot_dataframe.py @@ -10,7 +10,7 @@ from freqtrade.misc import common_args_parser def plot_parse_args(args ): - parser = common_args_parser(args, 'Graph utility') + parser = common_args_parser(description='Graph utility') parser.add_argument( '-p', '--pair', help = 'What currency pair', From a26cb4bc6b3b438df1f606c1ba9c3a7fa59b6ac6 Mon Sep 17 00:00:00 2001 From: pyup-bot Date: Fri, 12 Jan 2018 11:08:23 +0100 Subject: [PATCH 07/44] Update pymarketcap from 3.3.145 to 3.3.147 --- requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/requirements.txt b/requirements.txt index 8a3c8b9cf..52f4e7f7e 100644 --- a/requirements.txt +++ b/requirements.txt @@ -19,7 +19,7 @@ hyperopt==0.1 # do not upgrade networkx before this is fixed https://github.com/hyperopt/hyperopt/issues/325 networkx==1.11 tabulate==0.8.2 -pymarketcap==3.3.145 +pymarketcap==3.3.147 # Required for plotting data #matplotlib==2.1.0 From 48432abff1cb7e9316e4e944c9f71a68611c4dc3 Mon Sep 17 00:00:00 2001 From: kryofly Date: Fri, 12 Jan 2018 19:48:52 +0100 Subject: [PATCH 08/44] remove two-letter options --- freqtrade/misc.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/freqtrade/misc.py b/freqtrade/misc.py index 8dceee5c3..47b1705b3 100644 --- a/freqtrade/misc.py +++ b/freqtrade/misc.py @@ -127,7 +127,7 @@ def parse_args(args: List[str], description: str): dest='dry_run_db', ) parser.add_argument( - '-dd', '--datadir', + '--datadir', help='path to backtest data (default freqdata/tests/testdata', dest='datadir', default=os.path.join('freqtrade', 'tests', 'testdata'), @@ -186,7 +186,7 @@ def build_subcommands(parser: argparse.ArgumentParser) -> None: dest='refresh_pairs', ) backtesting_cmd.add_argument( - '-tp', '--timeperiod', + '--timeperiod', help='Use the last N ticks of data.', default=None, type=int, From d48d2d08dfff665c90dc083100948e2c552e1550 Mon Sep 17 00:00:00 2001 From: Rob Moggach Date: Fri, 12 Jan 2018 18:36:12 -0800 Subject: [PATCH 09/44] cleaned up installation docs --- docs/installation.md | 286 +++++++++++++++++++++++++++---------------- 1 file changed, 178 insertions(+), 108 deletions(-) diff --git a/docs/installation.md b/docs/installation.md index 88e6bf7e2..695ebe6d1 100644 --- a/docs/installation.md +++ b/docs/installation.md @@ -1,98 +1,118 @@ -# Install the bot +# Installation + This page explains how to prepare your environment for running the bot. -To understand how to set up the bot please read the Bot -[Bot configuration](https://github.com/gcarq/freqtrade/blob/develop/docs/configuration.md) -page. + +To understand how to set up the bot please read the [Bot Configuration](https://github.com/gcarq/freqtrade/blob/develop/docs/configuration.md) page. + + ## Table of Contents -- [Docker Automatic Installation](#docker) -- [Linux or Mac manual Installation](#linux--mac) + +* [Docker Automatic Installation](#docker) +* [Linux or Mac Manual Installation](#linux--mac) - [Linux - Ubuntu 16.04](#21-linux---ubuntu-1604) - [Linux - Other distro](#22-linux---other-distro) - - [MacOS installation](#23-macos-installation) + - [MacOS Installation](#23-macos-installation) - [Advanced Linux ](#advanced-linux) -- [Windows manual Installation](#windows) +* [Windows manual Installation](#windows) -# Docker -## Easy installation +------ + +## Automatic Installation - Docker + Start by downloading Docker for your platform: -- [Mac](https://www.docker.com/products/docker#/mac) -- [Windows](https://www.docker.com/products/docker#/windows) -- [Linux](https://www.docker.com/products/docker#/linux) -Once you have Docker installed, simply create the config file -(e.g. `config.json`) and then create a Docker image for `freqtrade` -using the Dockerfile in this repo. +* [Mac](https://www.docker.com/products/docker#/mac) +* [Windows](https://www.docker.com/products/docker#/windows) +* [Linux](https://www.docker.com/products/docker#/linux) + +Once you have Docker installed, simply create the config file (e.g. `config.json`) and then create a Docker image for `freqtrade` using the Dockerfile in this repo. + + +### 1. Prepare the Bot + +#### 1.1. Clone the git repository -### 1. Prepare the bot -1. Clone the git ```bash git clone https://github.com/gcarq/freqtrade.git ``` -2. (Optional) Checkout the develop branch + +#### 1.2. (Optional) Checkout the develop branch + ```bash git checkout develop ``` -3. Go into the new directory + +#### 1.3. Go into the new directory + ```bash cd freqtrade ``` -4. Copy `config.sample` to `config.json` + +#### 1.4. Copy `config.json.example` to `config.json` + ```bash -cp config.json.example config.json +cat config.json.example >> config.json ``` -To edit the config please refer to the [Bot Configuration](https://github.com/gcarq/freqtrade/blob/develop/docs/configuration.md) page -5. Create your DB file (Optional, the bot will create it if it is missing) + +> To edit the config please refer to the [Bot Configuration](https://github.com/gcarq/freqtrade/blob/develop/docs/configuration.md) page. + +#### 1.5. Create your database file *(optional - the bot will create it if it is missing)* + ```bash # For Production touch tradesv3.sqlite -# For Dry-run +# For Dry-Run touch tradesv3.dryrun.sqlite ``` -### 2. Build the docker image + +### 2. Build the Docker image + ```bash cd freqtrade docker build -t freqtrade . ``` -For security reasons, your configuration file will not be included in the -image, you will need to bind mount it. It is also advised to bind mount -a sqlite database file (see the "5. Run a restartable docker image" -section) to keep it between updates. +For security reasons, your configuration file will not be included in the image, you will need to bind mount it. It is also advised to bind mount an SQLite database file (see the "5. Run a restartable docker image" section) to keep it between updates. -### 3. Verify the docker image -After build process you can verify that the image was created with: -``` + +### 3. Verify the Docker image + +After the build process you can verify that the image was created with: + +```bash docker images ``` -### 4. Run the docker image -You can run a one-off container that is immediately deleted upon exiting with -the following command (config.json must be in the current working directory): -``` +### 4. Run the Docker image + +You can run a one-off container that is immediately deleted upon exiting with the following command (`config.json` must be in the current working directory): + +```bash docker run --rm -v `pwd`/config.json:/freqtrade/config.json -it freqtrade ``` -In this example, the database will be created inside the docker instance -and will be lost when you will refresh your image. +In this example, the database will be created inside the docker instance and will be lost when you will refresh your image. + ### 5. Run a restartable docker image -To run a restartable instance in the background (feel free to place your -configuration and database files wherever it feels comfortable on your -filesystem). -**5.1. Move your config file and database** +To run a restartable instance in the background (feel free to place your configuration and database files wherever it feels comfortable on your filesystem). + +#### 5.1. Move your config file and database + ```bash mkdir ~/.freqtrade mv config.json ~/.freqtrade mv tradesv3.sqlite ~/.freqtrade ``` -**5.2. Run the docker image** +#### 5.2. Run the docker image + ```bash docker run -d \ --name freqtrade \ @@ -100,12 +120,12 @@ docker run -d \ -v ~/.freqtrade/tradesv3.sqlite:/freqtrade/tradesv3.sqlite \ freqtrade ``` -If you are using `dry_run=True` it's not necessary to mount -`tradesv3.sqlite`, but you can mount `tradesv3.dryrun.sqlite` if you -plan to use the dry run mode with the param `--dry-run-db`. + +If you are using `dry_run=True` it's not necessary to mount `tradesv3.sqlite`, but you can mount `tradesv3.dryrun.sqlite` if you plan to use the dry run mode with the param `--dry-run-db`. ### 6. Monitor your Docker instance + You can then use the following commands to monitor and manage your container: ```bash @@ -116,35 +136,42 @@ docker stop freqtrade docker start freqtrade ``` -You do not need to rebuild the image for configuration changes, it will -suffice to edit `config.json` and restart the container. +You do not need to rebuild the image for configuration changes, it will suffice to edit `config.json` and restart the container. -# Linux / MacOS -## 1. Requirements +------ + + +## Custom Installation + +We've included/collected install instructions for Ubuntu 16.04, MacOS, and Windows. These are guidelines and your success may vary with other distros. + +### Requirements + Click each one for install guide: -- [Python 3.6.x](http://docs.python-guide.org/en/latest/starting/installation/), -note the bot was not tested on Python >= 3.7.x -- [pip](https://pip.pypa.io/en/stable/installing/) -- [git](https://git-scm.com/book/en/v2/Getting-Started-Installing-Git) -- [virtualenv](https://virtualenv.pypa.io/en/stable/installation/) (Recommended) -- [TA-Lib](https://mrjbq7.github.io/ta-lib/install.html) -## 2. First install required packages -This bot require Python 3.6 and TA-LIB +* [Python 3.6.x](http://docs.python-guide.org/en/latest/starting/installation/), note the bot was not tested on Python >= 3.7.x +* [pip](https://pip.pypa.io/en/stable/installing/) +* [git](https://git-scm.com/book/en/v2/Getting-Started-Installing-Git) +* [virtualenv](https://virtualenv.pypa.io/en/stable/installation/) (Recommended) +* [TA-Lib](https://mrjbq7.github.io/ta-lib/install.html) -### 2.1 Linux - Ubuntu 16.04 -**2.1.1. Install Python 3.6, Git, and wget** +### Linux - Ubuntu 16.04 + +#### 1. Install Python 3.6, Git, and wget + ```bash sudo add-apt-repository ppa:jonathonf/python-3.6 sudo apt-get update sudo apt-get install python3.6 python3.6-venv build-essential autoconf libtool pkg-config make wget git ``` -**2.1.2. Install TA-LIB** +#### 2. Install TA-Lib + Official webpage: https://mrjbq7.github.io/ta-lib/install.html -``` + +```bash wget http://prdownloads.sourceforge.net/ta-lib/ta-lib-0.4.0-src.tar.gz tar xvzf ta-lib-0.4.0-src.tar.gz cd ta-lib @@ -155,29 +182,58 @@ cd .. rm -rf ./ta-lib* ``` -**2.1.3. [Optional] Install MongoDB** +#### 3. [Optional] Install MongoDB + Install MongoDB if you plan to optimize your strategy with Hyperopt. ```bash sudo apt-get install mongodb-org ``` -Complete tutorial on [Digital Ocean: How to Install MongoDB on Ubuntu 16.04](https://www.digitalocean.com/community/tutorials/how-to-install-mongodb-on-ubuntu-16-04) -### 2.2. Linux - Other distro -If you are on a different Linux OS you maybe have to adapt things like: +> Complete tutorial from Digital Ocean: [How to Install MongoDB on Ubuntu 16.04](https://www.digitalocean.com/community/tutorials/how-to-install-mongodb-on-ubuntu-16-04). -- package manager (for example yum instead of apt-get) -- package names +#### 4. Install FreqTrade -### 2.3. MacOS installation +Clone the git repository: -**2.3.1. Install Python 3.6, git and wget** ```bash -brew install python3 git wget +git clone https://github.com/gcarq/freqtrade.git ``` -**2.3.2. [Optional] Install MongoDB** +Optionally checkout the develop branch: + +```bash +git checkout develop +``` + +#### 5. Configure `freqtrade` as a `systemd` service + +From the freqtrade repo... copy `freqtrade.service` to your systemd user directory (usually `~/.config/systemd/user`) and update `WorkingDirectory` and `ExecStart` to match your setup. + +After that you can start the daemon with: +```bash +systemctl --user start freqtrade +``` + +For this to be persistent (run when user is logged out) you'll need to enable `linger` for your freqtrade user. + +```bash +sudo loginctl enable-linger "$USER" +``` + + +### MacOS Installation + +#### 1. Install Python 3.6, git, wget and ta-lib + +```bash +brew install python3 git wget ta-lib +``` + +#### 2. [Optional] Install MongoDB + Install MongoDB if you plan to optimize your strategy with Hyperopt. + ```bash curl -O https://fastdl.mongodb.org/osx/mongodb-osx-ssl-x86_64-3.4.10.tgz tar -zxvf mongodb-osx-ssl-x86_64-3.4.10.tgz @@ -186,46 +242,27 @@ cp -R -n mongodb-osx-x86_64-3.4.10/ /env/mongodb export PATH=/env/mongodb/bin:$PATH ``` -## 3. Clone the repo -The following steps are made for Linux/mac environment -1. Clone the git `git clone https://github.com/gcarq/freqtrade.git` -2. (Optional) Checkout the develop branch `git checkout develop` +#### 3. Install FreqTrade -## 4. Prepare the bot -```bash -cd freqtrade -cp config.json.example config.json -``` -To edit the config please refer to [Bot Configuration](https://github.com/gcarq/freqtrade/blob/develop/docs/configuration.md) - -## 5. Setup your virtual env -```bash -python3.6 -m venv .env -source .env/bin/activate -pip3.6 install -r requirements.txt -pip3.6 install -e . -``` - -## 6. Run the bot -If this is the first time you run the bot, ensure you are running it -in Dry-run `"dry_run": true,` otherwise it will start to buy and sell coins. +Clone the git repository: ```bash -python3.6 ./freqtrade/main.py -c config.json +git clone https://github.com/gcarq/freqtrade.git ``` -### Advanced Linux -**systemd service file** -Copy `./freqtrade.service` to your systemd user directory (usually `~/.config/systemd/user`) -and update `WorkingDirectory` and `ExecStart` to match your setup. -After that you can start the daemon with: +Optionally checkout the develop branch: + ```bash -systemctl --user start freqtrade +git checkout develop ``` -# Windows -We do recommend Windows users to use [Docker](#docker) this will work -much easier and smoother (also safer). + +### Windows + +We recommend that Windows users use [Docker](#docker) as this will work +much easier and smoother (also more secure). + +## Install freqtrade ```cmd #copy paste config.json to \path\freqtrade-develop\freqtrade @@ -239,8 +276,41 @@ much easier and smoother (also safer). >cd freqtrade >python main.py ``` + *Thanks [Owdr](https://github.com/Owdr) for the commands. Source: [Issue #222](https://github.com/gcarq/freqtrade/issues/222)* -## Next step -Now you have an environment ready, the next step is to -[configure your bot](https://github.com/gcarq/freqtrade/blob/develop/docs/configuration.md). + +------ + + +## First Step + +```bash +cd freqtrade +cp config.json.example config.json +``` + +> *To edit the config please refer to [Bot Configuration](https://github.com/gcarq/freqtrade/blob/develop/docs/configuration.md).* + + +### 5. Setup your Python virtual environment (virtualenv) +```bash +python3.6 -m venv .env +source .env/bin/activate +pip3.6 install -r requirements.txt +pip3.6 install -e . +``` + +### 6. Run the Bot +If this is the first time you run the bot, ensure you are running it +in Dry-run `"dry_run": true,` otherwise it will start to buy and sell coins. + +```bash +python3.6 ./freqtrade/main.py -c config.json +``` + +------ + +## Next Step +Now you have an environment ready, the next step is +[Bot Configuration](https://github.com/gcarq/freqtrade/blob/develop/docs/configuration.md)... From 25e021d4b465fb4cf731289abb42818f4d5874f7 Mon Sep 17 00:00:00 2001 From: Rob Moggach Date: Fri, 12 Jan 2018 21:32:09 -0800 Subject: [PATCH 10/44] installation docs update --- docs/installation.md | 53 ++++++++++++++++++++++++-------------------- 1 file changed, 29 insertions(+), 24 deletions(-) diff --git a/docs/installation.md b/docs/installation.md index 695ebe6d1..9ac8a99dc 100644 --- a/docs/installation.md +++ b/docs/installation.md @@ -8,15 +8,16 @@ To understand how to set up the bot please read the [Bot Configuration](https:// ## Table of Contents -* [Docker Automatic Installation](#docker) -* [Linux or Mac Manual Installation](#linux--mac) - - [Linux - Ubuntu 16.04](#21-linux---ubuntu-1604) - - [Linux - Other distro](#22-linux---other-distro) - - [MacOS Installation](#23-macos-installation) - - [Advanced Linux ](#advanced-linux) -* [Windows manual Installation](#windows) - +* [Table of Contents](#table-of-contents) +* [Automatic Installation - Docker](#automatic-installation-docker) +* [Custom Installation](#custom-installation) + - [Requirements](#requirements) + - [Linux - Ubuntu 16.04](#linux-ubuntu-1604) + - [MacOS](#macos) + - [Windows](#windows) +* [First Steps](#first-step) + ------ ## Automatic Installation - Docker @@ -60,11 +61,14 @@ cat config.json.example >> config.json #### 1.5. Create your database file *(optional - the bot will create it if it is missing)* -```bash -# For Production -touch tradesv3.sqlite -# For Dry-Run +Production +```bash +touch tradesv3.sqlite +```` + +Dry-Run +```bash touch tradesv3.dryrun.sqlite ``` @@ -222,7 +226,7 @@ sudo loginctl enable-linger "$USER" ``` -### MacOS Installation +### MacOS #### 1. Install Python 3.6, git, wget and ta-lib @@ -262,10 +266,11 @@ git checkout develop We recommend that Windows users use [Docker](#docker) as this will work much easier and smoother (also more secure). -## Install freqtrade +#### 1. Install freqtrade + +copy paste `config.json` to ``\path\freqtrade-develop\freqtrade` ```cmd -#copy paste config.json to \path\freqtrade-develop\freqtrade >cd \path\freqtrade-develop >python -m venv .env >cd .env\Scripts @@ -277,13 +282,15 @@ much easier and smoother (also more secure). >python main.py ``` -*Thanks [Owdr](https://github.com/Owdr) for the commands. Source: [Issue #222](https://github.com/gcarq/freqtrade/issues/222)* +> Thanks [Owdr](https://github.com/Owdr) for the commands. Source: [Issue #222](https://github.com/gcarq/freqtrade/issues/222) ------ -## First Step +## First Steps + +### 1. Initialize the configuration ```bash cd freqtrade @@ -293,7 +300,8 @@ cp config.json.example config.json > *To edit the config please refer to [Bot Configuration](https://github.com/gcarq/freqtrade/blob/develop/docs/configuration.md).* -### 5. Setup your Python virtual environment (virtualenv) +### 2. Setup your Python virtual environment (virtualenv) + ```bash python3.6 -m venv .env source .env/bin/activate @@ -301,16 +309,13 @@ pip3.6 install -r requirements.txt pip3.6 install -e . ``` -### 6. Run the Bot -If this is the first time you run the bot, ensure you are running it -in Dry-run `"dry_run": true,` otherwise it will start to buy and sell coins. +### 3. Run the Bot + +If this is the first time you run the bot, ensure you are running it in Dry-run `"dry_run": true,` otherwise it will start to buy and sell coins. ```bash python3.6 ./freqtrade/main.py -c config.json ``` ------- - -## Next Step Now you have an environment ready, the next step is [Bot Configuration](https://github.com/gcarq/freqtrade/blob/develop/docs/configuration.md)... From 3087ca0823a688f329a9ccbe265656db0048bccb Mon Sep 17 00:00:00 2001 From: Gerald Lonlas Date: Fri, 12 Jan 2018 22:56:39 -0800 Subject: [PATCH 11/44] Update freqtrade version --- freqtrade/__init__.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/freqtrade/__init__.py b/freqtrade/__init__.py index a190ca117..d1671e4c6 100644 --- a/freqtrade/__init__.py +++ b/freqtrade/__init__.py @@ -1,5 +1,5 @@ """ FreqTrade bot """ -__version__ = '0.14.3' +__version__ = '0.15.1' class DependencyException(BaseException): From f7a44d1cecd2c01188660c4e6244e06c21d3a872 Mon Sep 17 00:00:00 2001 From: Jean-Baptiste LE STANG Date: Sat, 13 Jan 2018 09:50:02 +0100 Subject: [PATCH 12/44] Fixing the ticker analysis with null value --- freqtrade/exchange/bittrex.py | 6 ++---- freqtrade/tests/exchange/test_exchange_bittrex.py | 13 ++++++++++++- 2 files changed, 14 insertions(+), 5 deletions(-) diff --git a/freqtrade/exchange/bittrex.py b/freqtrade/exchange/bittrex.py index a5fd68016..e6cacbd4e 100644 --- a/freqtrade/exchange/bittrex.py +++ b/freqtrade/exchange/bittrex.py @@ -123,10 +123,8 @@ class Bittrex(Exchange): message=data['message'], pair=pair)) - if not data.get('result') \ - or not data['result'].get('Bid') \ - or not data['result'].get('Ask') \ - or not data['result'].get('Last'): + if not data.get('result') or\ + not all(key in data.get('result', {}) for key in ['Bid', 'Ask', 'Last']): raise ContentDecodingError('{message} params=({pair})'.format( message='Got invalid response from bittrex', pair=pair)) diff --git a/freqtrade/tests/exchange/test_exchange_bittrex.py b/freqtrade/tests/exchange/test_exchange_bittrex.py index e01e7fa02..4302f9f5f 100644 --- a/freqtrade/tests/exchange/test_exchange_bittrex.py +++ b/freqtrade/tests/exchange/test_exchange_bittrex.py @@ -212,7 +212,18 @@ def test_exchange_bittrex_get_ticker_bad(): wb = make_wrap_bittrex() fb = FakeBittrex() fb.result = {'success': True, - 'result': {'Bid': 1}} # incomplete result + 'result': {'Bid': 1, 'Ask': 0}} # incomplete result + + with pytest.raises(ContentDecodingError, match=r'.*Got invalid response from bittrex params.*'): + wb.get_ticker('BTC_ETH') + fb.result = {'success': False, + 'message': 'gone bad' + } + with pytest.raises(btx.OperationalException, match=r'.*gone bad.*'): + wb.get_ticker('BTC_ETH') + + fb.result = {'success': True, + 'result': {}} # incomplete result with pytest.raises(ContentDecodingError, match=r'.*Got invalid response from bittrex params.*'): wb.get_ticker('BTC_ETH') fb.result = {'success': False, From e5b27baa590cf72fe2a8131837c064bf829874c2 Mon Sep 17 00:00:00 2001 From: pyup-bot Date: Sat, 13 Jan 2018 13:38:23 +0100 Subject: [PATCH 13/44] Update pymarketcap from 3.3.147 to 3.3.148 --- requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/requirements.txt b/requirements.txt index 80cc78e17..2bc86ad61 100644 --- a/requirements.txt +++ b/requirements.txt @@ -19,7 +19,7 @@ hyperopt==0.1 # do not upgrade networkx before this is fixed https://github.com/hyperopt/hyperopt/issues/325 networkx==1.11 tabulate==0.8.2 -pymarketcap==3.3.147 +pymarketcap==3.3.148 # Required for plotting data #matplotlib==2.1.0 From 3277e491f18835fd7697bc4a4467277b5ba27703 Mon Sep 17 00:00:00 2001 From: kryofly Date: Sat, 13 Jan 2018 17:39:36 +0100 Subject: [PATCH 14/44] support download for multiple testdata sets --- docs/backtesting.md | 15 +++++++ freqtrade/misc.py | 5 +++ .../tests/testdata/download_backtest_data.py | 41 +++++++++++-------- freqtrade/tests/testdata/pairs.json | 23 +++++++++++ 4 files changed, 68 insertions(+), 16 deletions(-) create mode 100644 freqtrade/tests/testdata/pairs.json diff --git a/docs/backtesting.md b/docs/backtesting.md index c426e2b5c..8574d9dc2 100644 --- a/docs/backtesting.md +++ b/docs/backtesting.md @@ -51,6 +51,21 @@ python3 ./freqtrade/main.py backtesting --realistic-simulation --live python3 ./freqtrade/main.py backtesting --datadir freqtrade/tests/testdata-20180101 ``` +To update your testdata directory, or download into another testdata directory: +```bash +mkdir freqtrade/tests/testdata-20180113 +cp freqtrade/tests/testdata/pairs.json freqtrade/tests/testdata-20180113 +cd freqtrade/tests/testdata-20180113 + +Possibly edit pairs.json file to include/exclude pairs + +python download_backtest_data.py -p pairs.json +``` + +The script will read your pairs.json file, and download ticker data +into the current working directory. + + For help about backtesting usage, please refer to [Backtesting commands](#backtesting-commands). diff --git a/freqtrade/misc.py b/freqtrade/misc.py index 3d70f6b25..979174f8d 100644 --- a/freqtrade/misc.py +++ b/freqtrade/misc.py @@ -15,6 +15,11 @@ from freqtrade import __version__ logger = logging.getLogger(__name__) +def file_dump_json(filename, data): + with open(filename, 'w') as fp: + json.dump(data, fp) + + class State(enum.Enum): RUNNING = 0 STOPPED = 1 diff --git a/freqtrade/tests/testdata/download_backtest_data.py b/freqtrade/tests/testdata/download_backtest_data.py index 37cd4c95f..0cb545b3a 100755 --- a/freqtrade/tests/testdata/download_backtest_data.py +++ b/freqtrade/tests/testdata/download_backtest_data.py @@ -1,29 +1,38 @@ #!/usr/bin/env python3 """This script generate json data from bittrex""" +import sys import json -from os import path from freqtrade import exchange from freqtrade.exchange import Bittrex +from freqtrade import misc -PAIRS = [ - 'BTC_BCC', 'BTC_ETH', 'BTC_MER', 'BTC_POWR', 'BTC_ETC', - 'BTC_OK', 'BTC_NEO', 'BTC_EMC2', 'BTC_DASH', 'BTC_LSK', - 'BTC_LTC', 'BTC_XZC', 'BTC_OMG', 'BTC_STRAT', 'BTC_XRP', - 'BTC_QTUM', 'BTC_WAVES', 'BTC_VTC', 'BTC_XLM', 'BTC_MCO' -] -TICKER_INTERVAL = 5 # ticker interval in minutes (currently implemented: 1 and 5) -OUTPUT_DIR = path.dirname(path.realpath(__file__)) +parser = misc.common_args_parser('download utility') +parser.add_argument( + '-p', '--pair', + help='JSON file containing pairs to download', + dest='pair', + default=None +) +args = parser.parse_args(sys.argv[1:]) + +TICKER_INTERVALS = [1, 5] # ticker interval in minutes (currently implemented: 1 and 5) +PAIRS = [] + +if args.pair: + with open(args.pair) as file: + PAIRS = json.load(file) +PAIRS = list(set(PAIRS)) + +print('About to download pairs:', PAIRS) # Init Bittrex exchange exchange._API = Bittrex({'key': '', 'secret': ''}) for pair in PAIRS: - data = exchange.get_ticker_history(pair, TICKER_INTERVAL) - filename = path.join(OUTPUT_DIR, '{}-{}.json'.format( - pair, - TICKER_INTERVAL, - )) - with open(filename, 'w') as fp: - json.dump(data, fp) + for tick_interval in TICKER_INTERVALS: + print('downloading pair %s, interval %s' % (pair, tick_interval)) + data = exchange.get_ticker_history(pair, tick_interval) + filename = '{}-{}.json'.format(pair, tick_interval) + misc.file_dump_json(filename, data) diff --git a/freqtrade/tests/testdata/pairs.json b/freqtrade/tests/testdata/pairs.json new file mode 100644 index 000000000..c3e339b4c --- /dev/null +++ b/freqtrade/tests/testdata/pairs.json @@ -0,0 +1,23 @@ +[ + "BTC_ADA", + "BTC_BAT", + "BTC_DASH", + "BTC_ETC", + "BTC_ETH", + "BTC_GBYTE", + "BTC_LSK", + "BTC_LTC", + "BTC_NEO", + "BTC_NXT", + "BTC_POWR", + "BTC_STORJ", + "BTC_QTUM", + "BTC_WAVES", + "BTC_VTC", + "BTC_XLM", + "BTC_XMR", + "BTC_XVG", + "BTC_XRP", + "BTC_ZEC" +] + From 344843d802241f26877c0820ea5e3b7e6a869f7e Mon Sep 17 00:00:00 2001 From: Gerald Lonlas Date: Sat, 13 Jan 2018 22:54:22 -0800 Subject: [PATCH 15/44] Update doc: 'cp' becomes 'cp -n', and add more FAQ questions --- docs/faq.md | 64 +++++++++++++++++++++++++++++++++++++----- docs/installation.md | 4 +-- docs/sql_cheatsheet.md | 12 ++++++++ 3 files changed, 71 insertions(+), 9 deletions(-) diff --git a/docs/faq.md b/docs/faq.md index 58929e15c..b3f15613a 100644 --- a/docs/faq.md +++ b/docs/faq.md @@ -2,20 +2,70 @@ #### I have waited 5 minutes, why hasn't the bot made any trades yet?! -Depending on the buy strategy, the amount of whitelisted coins, the situation of the market etc, it can take up to hours to find good entry position for a trade. Be patient! +Depending on the buy strategy, the amount of whitelisted coins, the +situation of the market etc, it can take up to hours to find good entry +position for a trade. Be patient! #### I have made 12 trades already, why is my total profit negative?! -I understand your disappointment but unfortunately 12 trades is just not enough to say anything. If you run backtesting, you can see that our current algorithm does leave you on the plus side, but that is after thousands of trades and even there, you will be left with losses on specific coins that you have traded tens if not hundreds of times. We of course constantly aim to improve the bot but it will _always_ be a gamble, which should leave you with modest wins on monthly basis but you can't say much from few trades. +I understand your disappointment but unfortunately 12 trades is just +not enough to say anything. If you run backtesting, you can see that our +current algorithm does leave you on the plus side, but that is after +thousands of trades and even there, you will be left with losses on +specific coins that you have traded tens if not hundreds of times. We +of course constantly aim to improve the bot but it will _always_ be a +gamble, which should leave you with modest wins on monthly basis but +you can't say much from few trades. -#### I’d like to change the stake amount. Can I just stop the bot with /stop and then change the config.json and run it again? +#### I’d like to change the stake amount. Can I just stop the bot with +/stop and then change the config.json and run it again? -Not quite. Trades are persisted to a database but the configuration is currently only read when the bot is killed and restarted. `/stop` more like pauses. You can stop your bot, adjust settings and start it again. +Not quite. Trades are persisted to a database but the configuration is +currently only read when the bot is killed and restarted. `/stop` more +like pauses. You can stop your bot, adjust settings and start it again. #### I want to improve the bot with a new strategy -That's great. We have a nice backtesting and hyperoptimizing setup. See the tutorial [[here|Testing-new-strategies-with-Hyperopt]]. +That's great. We have a nice backtesting and hyperoptimizing setup. See +the tutorial [here|Testing-new-strategies-with-Hyperopt](https://github.com/gcarq/freqtrade/blob/develop/docs/bot-usage.md#hyperopt-commands). -#### Is there a setting to only SELL the coins being held and not perform anymore BUYS? +#### Is there a setting to only SELL the coins being held and not +perform anymore BUYS? + +You can use the `/forcesell all` command from Telegram. + +### How many epoch do I need to get a good Hyperopt result? +Per default Hyperopts without `-e` or `--epochs` parameter will only +run 100 epochs, means 100 evals of your triggers, guards, .... Too few +to find a great result (unless if you are very lucky), so you probably +have to run it for 10.000 or more. But it will take an eternity to +compute. + +We recommend you to run it at least 10.000 epochs: +```bash +python3 ./freqtrade/main.py hyperopt -e 10000 +``` + +or if you want intermediate result to see +```bash +for i in {1..100}; do python3 ./freqtrade/main.py hyperopt -e 100; done +``` + +#### Why it is so long to run hyperopt? +Finding a great Hyperopt results takes time. + +If you wonder why it takes a while to find great hyperopt results + +This answer was written during the under the release 0.15.1, when we had +: +- 8 triggers +- 9 guards: let's say we evaluate even 10 values from each +- 1 stoploss calculation: let's say we want 10 values from that too to +be evaluated + +The following calculation is still very rough and not very precise +but it will give the idea. With only these triggers and guards there is +already 8*10^9*10 evaluations. A roughly total of 80 billion evals. +Did you run 100 000 evals? Congrats, you've done roughly 1 / 100 000 th +of the search space. -You can use the `/forcesell all` command from Telegram. \ No newline at end of file diff --git a/docs/installation.md b/docs/installation.md index 30431345b..59e0663ae 100644 --- a/docs/installation.md +++ b/docs/installation.md @@ -40,7 +40,7 @@ cd freqtrade ``` 4. Copy `config.sample` to `config.json` ```bash -cp config.json.example config.json +cp -n config.json.example config.json ``` To edit the config please refer to the [Bot Configuration](https://github.com/gcarq/freqtrade/blob/develop/docs/configuration.md) page 5. Create your DB file (Optional, the bot will create it if it is missing) @@ -194,7 +194,7 @@ The following steps are made for Linux/mac environment ## 4. Prepare the bot ```bash cd freqtrade -cp config.json.example config.json +cp -n config.json.example config.json ``` To edit the config please refer to [Bot Configuration](https://github.com/gcarq/freqtrade/blob/develop/docs/configuration.md) diff --git a/docs/sql_cheatsheet.md b/docs/sql_cheatsheet.md index 1edbcde5b..065f264f1 100644 --- a/docs/sql_cheatsheet.md +++ b/docs/sql_cheatsheet.md @@ -67,6 +67,18 @@ SET is_open=0, close_date='2017-12-20 03:08:45.103418', close_rate=0.19638016, c WHERE id=31; ``` +## Insert manually a new trade + +```sql +INSERT +INTO trades (exchange, pair, is_open, fee, open_rate, stake_amount, amount, open_date) +VALUES ('BITTREX', 'BTC_', 1, 0.0025, , , , '') +``` + +**Example:** +```sql +INSERT INTO trades (exchange, pair, is_open, fee, open_rate, stake_amount, amount, open_date) VALUES ('BITTREX', 'BTC_ETC', 1, 0.0025, 0.00258580, 0.002, 0.7715262081, '2017-11-28 12:44:24.000000') +``` ## Fix wrong fees in the table If your DB was created before From 92241baadee5b41651d087d2d659c9e1e688ecd5 Mon Sep 17 00:00:00 2001 From: Janne Sinivirta Date: Sun, 14 Jan 2018 13:09:39 +0200 Subject: [PATCH 16/44] log the loss value --- freqtrade/optimize/hyperopt.py | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/freqtrade/optimize/hyperopt.py b/freqtrade/optimize/hyperopt.py index 71ddd33c6..19c48371c 100644 --- a/freqtrade/optimize/hyperopt.py +++ b/freqtrade/optimize/hyperopt.py @@ -133,10 +133,11 @@ def log_results(results): if results['loss'] < CURRENT_BEST_LOSS: CURRENT_BEST_LOSS = results['loss'] - logger.info('{:5d}/{}: {}'.format( + logger.info('{:5d}/{}: {}. Loss {:.5f}'.format( results['current_tries'], results['total_tries'], - results['result'])) + results['result'], + results['loss'])) else: print('.', end='') sys.stdout.flush() From f1e176d35c77d3772732532978007409d2d2adb3 Mon Sep 17 00:00:00 2001 From: Janne Sinivirta Date: Sun, 14 Jan 2018 13:10:25 +0200 Subject: [PATCH 17/44] log total profit in percentages also --- freqtrade/optimize/hyperopt.py | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/freqtrade/optimize/hyperopt.py b/freqtrade/optimize/hyperopt.py index 19c48371c..97c6a82f5 100644 --- a/freqtrade/optimize/hyperopt.py +++ b/freqtrade/optimize/hyperopt.py @@ -191,12 +191,13 @@ def optimizer(params): def format_results(results: DataFrame): return ('{:6d} trades. Avg profit {: 5.2f}%. ' - 'Total profit {: 11.8f} BTC. Avg duration {:5.1f} mins.').format( + 'Total profit {: 11.8f} BTC ({:.4f}Σ%). Avg duration {:5.1f} mins.').format( len(results.index), results.profit_percent.mean() * 100.0, results.profit_BTC.sum(), + results.profit_percent.sum(), results.duration.mean() * 5, - ) + ) def buy_strategy_generator(params): From ec7bfba8dfcc85fe47282b76d39dc994ddcf47f3 Mon Sep 17 00:00:00 2001 From: Janne Sinivirta Date: Sun, 14 Jan 2018 13:11:19 +0200 Subject: [PATCH 18/44] add comment about checking the new total profit logging --- freqtrade/optimize/hyperopt.py | 1 + 1 file changed, 1 insertion(+) diff --git a/freqtrade/optimize/hyperopt.py b/freqtrade/optimize/hyperopt.py index 97c6a82f5..3235a10e6 100644 --- a/freqtrade/optimize/hyperopt.py +++ b/freqtrade/optimize/hyperopt.py @@ -41,6 +41,7 @@ MAX_ACCEPTED_TRADE_DURATION = 240 # this is expexted avg profit * expected trade count # for example 3.5%, 1100 trades, EXPECTED_MAX_PROFIT = 3.85 +# check that the reported Σ% values do not exceed this! EXPECTED_MAX_PROFIT = 3.85 # Configuration and data used by hyperopt From b5cd9dab26e19fd8b45f3822ef0dcb21acc44da8 Mon Sep 17 00:00:00 2001 From: Rob Moggach Date: Sun, 14 Jan 2018 12:25:30 -0500 Subject: [PATCH 19/44] change cat to cp --- docs/installation.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/installation.md b/docs/installation.md index 9ac8a99dc..65dea9595 100644 --- a/docs/installation.md +++ b/docs/installation.md @@ -54,7 +54,7 @@ cd freqtrade #### 1.4. Copy `config.json.example` to `config.json` ```bash -cat config.json.example >> config.json +cp -n config.json.example config.json ``` > To edit the config please refer to the [Bot Configuration](https://github.com/gcarq/freqtrade/blob/develop/docs/configuration.md) page. From dd9ab5264df2fdf9d26b132ab7c38b52eae8b2cb Mon Sep 17 00:00:00 2001 From: Anton Ermak Date: Sat, 13 Jan 2018 14:58:23 +0700 Subject: [PATCH 20/44] Estimated BTC and fiat value for balance --- freqtrade/rpc/telegram.py | 20 +++++++++++++++++++- 1 file changed, 19 insertions(+), 1 deletion(-) diff --git a/freqtrade/rpc/telegram.py b/freqtrade/rpc/telegram.py index 009714682..a5c07d75c 100644 --- a/freqtrade/rpc/telegram.py +++ b/freqtrade/rpc/telegram.py @@ -382,13 +382,31 @@ def _balance(bot: Bot, update: Update) -> None: if not balances: output = '`All balances are zero.`' + total = 0.0 for currency in balances: - output += """*Currency*: {Currency} + coin = currency['Currency'] + if coin == 'BTC': + currency["Rate"] = 1.0 + else: + currency["Rate"] = exchange.get_ticker('BTC_' + coin, False)['bid'] + currency['BTC'] = currency["Rate"] * currency["Balance"] + total = total + currency['BTC'] + output += """{Currency}: *Available*: {Available} *Balance*: {Balance} *Pending*: {Pending} +*Est. BTC*: {BTC: .8f} """.format(**currency) + + symbol = _CONF['fiat_display_currency'] + value = _FIAT_CONVERT.convert_amount( + total, 'BTC', symbol + ) + output += """*Estimated Value*: +*BTC*: {0: .8f} +*{1}*: {2: .2f} +""".format(total, symbol, value) send_msg(output) From 5db04b15e721400594b5789312fd22abeafae81c Mon Sep 17 00:00:00 2001 From: Anton Ermak Date: Mon, 15 Jan 2018 12:07:01 +0700 Subject: [PATCH 21/44] Balance Estimated BTC - fix test --- freqtrade/rpc/telegram.py | 2 +- freqtrade/tests/rpc/test_rpc_telegram.py | 4 ++++ 2 files changed, 5 insertions(+), 1 deletion(-) diff --git a/freqtrade/rpc/telegram.py b/freqtrade/rpc/telegram.py index a5c07d75c..12e701a1c 100644 --- a/freqtrade/rpc/telegram.py +++ b/freqtrade/rpc/telegram.py @@ -391,7 +391,7 @@ def _balance(bot: Bot, update: Update) -> None: currency["Rate"] = exchange.get_ticker('BTC_' + coin, False)['bid'] currency['BTC'] = currency["Rate"] * currency["Balance"] total = total + currency['BTC'] - output += """{Currency}: + output += """*Currency*: {Currency} *Available*: {Available} *Balance*: {Balance} *Pending*: {Pending} diff --git a/freqtrade/tests/rpc/test_rpc_telegram.py b/freqtrade/tests/rpc/test_rpc_telegram.py index 428c544c1..58bf0154b 100644 --- a/freqtrade/tests/rpc/test_rpc_telegram.py +++ b/freqtrade/tests/rpc/test_rpc_telegram.py @@ -606,11 +606,15 @@ def test_balance_handle(default_conf, update, mocker): send_msg=msg_mock) mocker.patch.multiple('freqtrade.main.exchange', get_balances=MagicMock(return_value=mock_balance)) + mocker.patch.multiple('freqtrade.fiat_convert.Pymarketcap', + ticker=MagicMock(return_value={'price_usd': 15000.0}), + _cache_symbols=MagicMock(return_value={'BTC': 1})) _balance(bot=MagicMock(), update=update) assert msg_mock.call_count == 1 assert '*Currency*: BTC' in msg_mock.call_args_list[0][0][0] assert 'Balance' in msg_mock.call_args_list[0][0][0] + assert 'Est. BTC' in msg_mock.call_args_list[0][0][0] def test_help_handle(default_conf, update, mocker): From 50462fdb0084854ebfe3848f8fbb06b74cee83e7 Mon Sep 17 00:00:00 2001 From: pyup-bot Date: Mon, 15 Jan 2018 16:32:27 +0100 Subject: [PATCH 22/44] Update sqlalchemy from 1.2.0 to 1.2.1 --- requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/requirements.txt b/requirements.txt index 2bc86ad61..d650efea8 100644 --- a/requirements.txt +++ b/requirements.txt @@ -1,5 +1,5 @@ python-bittrex==0.2.2 -SQLAlchemy==1.2.0 +SQLAlchemy==1.2.1 python-telegram-bot==9.0.0 arrow==0.12.0 cachetools==2.0.1 From 71bb348698e5f7de928ed6232fbae55fca789a73 Mon Sep 17 00:00:00 2001 From: kryofly Date: Mon, 15 Jan 2018 21:49:06 +0100 Subject: [PATCH 23/44] rename --timeperiod to --timerange --- docs/backtesting.md | 4 ++-- freqtrade/misc.py | 8 ++++---- freqtrade/optimize/__init__.py | 6 +++--- freqtrade/optimize/backtesting.py | 2 +- freqtrade/optimize/hyperopt.py | 2 +- freqtrade/tests/optimize/test_backtesting.py | 2 +- freqtrade/tests/optimize/test_optimize.py | 2 +- 7 files changed, 13 insertions(+), 13 deletions(-) diff --git a/docs/backtesting.md b/docs/backtesting.md index 800420ea6..7984e67bc 100644 --- a/docs/backtesting.md +++ b/docs/backtesting.md @@ -52,12 +52,12 @@ python3 ./freqtrade/main.py backtesting --datadir freqtrade/tests/testdata-20180 ``` **Running backtest with smaller testset** -Use the --timeperiod argument to change how much of the testset +Use the --timerange argument to change how much of the testset you want to use. The last N ticks/timeframes will be used. Example: ```bash -python3 ./freqtrade/main.py backtesting --timeperiod -200 +python3 ./freqtrade/main.py backtesting --timerange -200 ``` **Update testdata directory diff --git a/freqtrade/misc.py b/freqtrade/misc.py index 9c2bce84f..c019509d6 100644 --- a/freqtrade/misc.py +++ b/freqtrade/misc.py @@ -191,11 +191,11 @@ def build_subcommands(parser: argparse.ArgumentParser) -> None: dest='refresh_pairs', ) backtesting_cmd.add_argument( - '--timeperiod', + '--timerange', help='Use the last N ticks of data.', default=None, type=int, - dest='timeperiod', + dest='timerange', ) # Add hyperopt subcommand @@ -224,11 +224,11 @@ def build_subcommands(parser: argparse.ArgumentParser) -> None: metavar='INT', ) hyperopt_cmd.add_argument( - '-tp', '--timeperiod', + '-tp', '--timerange', help='Use the last N ticks of data.', default=None, type=int, - dest='timeperiod', + dest='timerange', ) diff --git a/freqtrade/optimize/__init__.py b/freqtrade/optimize/__init__.py index 3f6a98ac1..54d339b73 100644 --- a/freqtrade/optimize/__init__.py +++ b/freqtrade/optimize/__init__.py @@ -68,9 +68,9 @@ def load_data(datadir: str, ticker_interval: int = 5, pairs: Optional[List[str]] return result -def tickerdata_to_dataframe(data, timeperiod=None): - if timeperiod: - data = trim_tickerlist(data, timeperiod) +def tickerdata_to_dataframe(data, timerange=None): + if timerange: + data = trim_tickerlist(data, timerange) preprocessed = preprocess(data) return preprocessed diff --git a/freqtrade/optimize/backtesting.py b/freqtrade/optimize/backtesting.py index afcd0c539..5b8afc267 100644 --- a/freqtrade/optimize/backtesting.py +++ b/freqtrade/optimize/backtesting.py @@ -175,7 +175,7 @@ def start(args): from freqtrade import main main._CONF = config - preprocessed = optimize.tickerdata_to_dataframe(data, timeperiod=args.timeperiod) + preprocessed = optimize.tickerdata_to_dataframe(data, timerange=args.timerange) # Print timeframe min_date, max_date = get_timeframe(preprocessed) logger.info('Measuring data from %s up to %s ...', min_date.isoformat(), max_date.isoformat()) diff --git a/freqtrade/optimize/hyperopt.py b/freqtrade/optimize/hyperopt.py index 15019e16b..abba7c35f 100644 --- a/freqtrade/optimize/hyperopt.py +++ b/freqtrade/optimize/hyperopt.py @@ -260,7 +260,7 @@ def start(args): config = load_config(args.config) pairs = config['exchange']['pair_whitelist'] data = optimize.load_data(args.datadir, pairs=pairs, ticker_interval=args.ticker_interval) - PROCESSED = optimize.tickerdata_to_dataframe(data, timeperiod=args.timeperiod) + PROCESSED = optimize.tickerdata_to_dataframe(data, timerange=args.timerange) if args.mongodb: logger.info('Using mongodb ...') diff --git a/freqtrade/tests/optimize/test_backtesting.py b/freqtrade/tests/optimize/test_backtesting.py index d4172d983..c6bcdd713 100644 --- a/freqtrade/tests/optimize/test_backtesting.py +++ b/freqtrade/tests/optimize/test_backtesting.py @@ -159,7 +159,7 @@ def test_backtest_start(default_conf, mocker, caplog): args.level = 10 args.live = False args.datadir = None - args.timeperiod = None # needed due to MagicMock malleability + args.timerange = None # needed due to MagicMock malleability backtesting.start(args) # check the logs, that will contain the backtest result exists = ['Using max_open_trades: 1 ...', diff --git a/freqtrade/tests/optimize/test_optimize.py b/freqtrade/tests/optimize/test_optimize.py index 6decb1414..bfbc99745 100644 --- a/freqtrade/tests/optimize/test_optimize.py +++ b/freqtrade/tests/optimize/test_optimize.py @@ -179,5 +179,5 @@ def test_load_tickerdata_file(): def test_tickerdata_to_dataframe(): tick = load_tickerdata_file(None, 'BTC_UNITEST', 1) tickerlist = {'BTC_UNITEST': tick} - data = optimize.tickerdata_to_dataframe(tickerlist, timeperiod=-100) + data = optimize.tickerdata_to_dataframe(tickerlist, timerange=-100) assert 100 == len(data['BTC_UNITEST']) From 0e58ab7e012a684d9d2aefa4e15aefe6e27153b2 Mon Sep 17 00:00:00 2001 From: kryofly Date: Mon, 15 Jan 2018 22:25:02 +0100 Subject: [PATCH 24/44] more advanced use of --timerange --- docs/backtesting.md | 15 ++++++- freqtrade/misc.py | 41 +++++++++++++++++--- freqtrade/optimize/__init__.py | 34 +++++++++------- freqtrade/optimize/backtesting.py | 7 ++-- freqtrade/optimize/hyperopt.py | 9 +++-- freqtrade/tests/optimize/test_backtesting.py | 14 +++---- freqtrade/tests/optimize/test_hyperopt.py | 15 ++++--- freqtrade/tests/optimize/test_optimize.py | 5 ++- freqtrade/tests/test_misc.py | 9 ++++- 9 files changed, 109 insertions(+), 40 deletions(-) diff --git a/docs/backtesting.md b/docs/backtesting.md index 7984e67bc..9829a10c8 100644 --- a/docs/backtesting.md +++ b/docs/backtesting.md @@ -57,9 +57,22 @@ you want to use. The last N ticks/timeframes will be used. Example: ```bash -python3 ./freqtrade/main.py backtesting --timerange -200 +python3 ./freqtrade/main.py backtesting --timerange=-200 ``` +***Advanced use of timerange*** + Doing --timerange=-200 will get the last 200 timeframes + from your inputdata. You can also specify specific dates, + or a range span indexed by start and stop. + The full timerange specification: + Not implemented yet! --timerange=-20180131 + Not implemented yet! --timerange=20180101- + Not implemented yet! --timerange=20180101-20181231 + Last 123 tickframes of data: --timerange=-123 + First 123 tickframes of data: --timerange=123- + Tickframes from line 123 through 456: --timerange=123-456 + + **Update testdata directory To update your testdata directory, or download into another testdata directory: ```bash diff --git a/freqtrade/misc.py b/freqtrade/misc.py index c019509d6..a9aeee80e 100644 --- a/freqtrade/misc.py +++ b/freqtrade/misc.py @@ -4,6 +4,7 @@ import json import logging import time import os +import re from typing import Any, Callable, Dict, List from jsonschema import Draft4Validator, validate @@ -192,9 +193,9 @@ def build_subcommands(parser: argparse.ArgumentParser) -> None: ) backtesting_cmd.add_argument( '--timerange', - help='Use the last N ticks of data.', + help='Specify what timerange of data to use.', default=None, - type=int, + type=str, dest='timerange', ) @@ -224,14 +225,44 @@ def build_subcommands(parser: argparse.ArgumentParser) -> None: metavar='INT', ) hyperopt_cmd.add_argument( - '-tp', '--timerange', - help='Use the last N ticks of data.', + '--timerange', + help='Specify what timerange of data to use.', default=None, - type=int, + type=str, dest='timerange', ) +def parse_timerange(text): + if text is None: + return None + syntax = [('^-(\d{8})$', (None, 'date')), + ('^(\d{8})-$', ('date', None)), + ('^(\d{8})-(\d{8})$', ('date', 'date')), + ('^(-\d+)$', (None, 'line')), + ('^(\d+)-$', ('line', None)), + ('^(\d+)-(\d+)$', ('index', 'index'))] + for rex, stype in syntax: + # Apply the regular expression to text + m = re.match(rex, text) + if m: # Regex has matched + rvals = m.groups() + n = 0 + start = None + stop = None + if stype[0]: + start = rvals[n] + if stype[0] != 'date': + start = int(start) + n += 1 + if stype[1]: + stop = rvals[n] + if stype[1] != 'date': + stop = int(stop) + return (stype, start, stop) + raise Exception('Incorrect syntax for timerange "%s"' % text) + + # Required json-schema for user specified config CONF_SCHEMA = { 'type': 'object', diff --git a/freqtrade/optimize/__init__.py b/freqtrade/optimize/__init__.py index 54d339b73..40269db46 100644 --- a/freqtrade/optimize/__init__.py +++ b/freqtrade/optimize/__init__.py @@ -12,14 +12,20 @@ from freqtrade.analyze import populate_indicators, parse_ticker_dataframe logger = logging.getLogger(__name__) -def trim_tickerlist(dl, num): - new = {} - for pair, pair_data in dl.items(): - new[pair] = pair_data[num:] - return new +def trim_tickerlist(tickerlist, timerange): + (stype, start, stop) = timerange + if stype == (None, 'line'): + return tickerlist[stop:] + elif stype == ('line', None): + return tickerlist[0:start] + elif stype == ('index', 'index'): + return tickerlist[start:stop] + else: + return tickerlist -def load_tickerdata_file(datadir, pair, ticker_interval): +def load_tickerdata_file(datadir, pair, ticker_interval, + timerange=None): """ Load a pair from file, :return dict OR empty if unsuccesful @@ -37,11 +43,15 @@ def load_tickerdata_file(datadir, pair, ticker_interval): # Read the file, load the json with open(file) as tickerdata: pairdata = json.load(tickerdata) + if timerange: + pairdata = trim_tickerlist(pairdata, timerange) return pairdata -def load_data(datadir: str, ticker_interval: int = 5, pairs: Optional[List[str]] = None, - refresh_pairs: Optional[bool] = False) -> Dict[str, List]: +def load_data(datadir: str, ticker_interval: int = 5, + pairs: Optional[List[str]] = None, + refresh_pairs: Optional[bool] = False, + timerange=None) -> Dict[str, List]: """ Loads ticker history data for the given parameters :param ticker_interval: ticker interval in minutes @@ -58,19 +68,17 @@ def load_data(datadir: str, ticker_interval: int = 5, pairs: Optional[List[str]] download_pairs(datadir, _pairs) for pair in _pairs: - pairdata = load_tickerdata_file(datadir, pair, ticker_interval) + pairdata = load_tickerdata_file(datadir, pair, ticker_interval, timerange=timerange) if not pairdata: # download the tickerdata from exchange download_backtesting_testdata(datadir, pair=pair, interval=ticker_interval) # and retry reading the pair - pairdata = load_tickerdata_file(datadir, pair, ticker_interval) + pairdata = load_tickerdata_file(datadir, pair, ticker_interval, timerange=timerange) result[pair] = pairdata return result -def tickerdata_to_dataframe(data, timerange=None): - if timerange: - data = trim_tickerlist(data, timerange) +def tickerdata_to_dataframe(data): preprocessed = preprocess(data) return preprocessed diff --git a/freqtrade/optimize/backtesting.py b/freqtrade/optimize/backtesting.py index 5b8afc267..4f3d4bb24 100644 --- a/freqtrade/optimize/backtesting.py +++ b/freqtrade/optimize/backtesting.py @@ -163,9 +163,10 @@ def start(args): logger.info('Using stake_currency: %s ...', config['stake_currency']) logger.info('Using stake_amount: %s ...', config['stake_amount']) + timerange = misc.parse_timerange(args.timerange) data = optimize.load_data(args.datadir, pairs=pairs, ticker_interval=args.ticker_interval, - refresh_pairs=args.refresh_pairs) - + refresh_pairs=args.refresh_pairs, + timerange=timerange) max_open_trades = 0 if args.realistic_simulation: logger.info('Using max_open_trades: %s ...', config['max_open_trades']) @@ -175,7 +176,7 @@ def start(args): from freqtrade import main main._CONF = config - preprocessed = optimize.tickerdata_to_dataframe(data, timerange=args.timerange) + preprocessed = optimize.tickerdata_to_dataframe(data) # Print timeframe min_date, max_date = get_timeframe(preprocessed) logger.info('Measuring data from %s up to %s ...', min_date.isoformat(), max_date.isoformat()) diff --git a/freqtrade/optimize/hyperopt.py b/freqtrade/optimize/hyperopt.py index abba7c35f..7a1744dc0 100644 --- a/freqtrade/optimize/hyperopt.py +++ b/freqtrade/optimize/hyperopt.py @@ -15,7 +15,7 @@ from hyperopt import STATUS_FAIL, STATUS_OK, Trials, fmin, hp, space_eval, tpe from hyperopt.mongoexp import MongoTrials from pandas import DataFrame -from freqtrade import main # noqa +from freqtrade import main, misc # noqa from freqtrade import exchange, optimize from freqtrade.exchange import Bittrex from freqtrade.misc import load_config @@ -259,8 +259,11 @@ def start(args): logger.info('Using config: %s ...', args.config) config = load_config(args.config) pairs = config['exchange']['pair_whitelist'] - data = optimize.load_data(args.datadir, pairs=pairs, ticker_interval=args.ticker_interval) - PROCESSED = optimize.tickerdata_to_dataframe(data, timerange=args.timerange) + timerange = misc.parse_timerange(args.timerange) + data = optimize.load_data(args.datadir, pairs=pairs, + ticker_interval=args.ticker_interval, + timerange=timerange) + PROCESSED = optimize.tickerdata_to_dataframe(data) if args.mongodb: logger.info('Using mongodb ...') diff --git a/freqtrade/tests/optimize/test_backtesting.py b/freqtrade/tests/optimize/test_backtesting.py index c6bcdd713..92170a184 100644 --- a/freqtrade/tests/optimize/test_backtesting.py +++ b/freqtrade/tests/optimize/test_backtesting.py @@ -6,7 +6,7 @@ import pandas as pd from unittest.mock import MagicMock from freqtrade import exchange, optimize from freqtrade.exchange import Bittrex -from freqtrade.optimize import preprocess, trim_tickerlist +from freqtrade.optimize import preprocess from freqtrade.optimize.backtesting import backtest, generate_text_table, get_timeframe import freqtrade.optimize.backtesting as backtesting @@ -60,8 +60,8 @@ def test_backtest_1min_ticker_interval(default_conf, mocker): def load_data_test(what): - data = optimize.load_data(None, ticker_interval=1, pairs=['BTC_UNITEST']) - data = trim_tickerlist(data, -100) + timerange = ((None, 'line'), None, -100) + data = optimize.load_data(None, ticker_interval=1, pairs=['BTC_UNITEST'], timerange=timerange) pair = data['BTC_UNITEST'] datalen = len(pair) # Depending on the what parameter we now adjust the @@ -142,10 +142,10 @@ def test_backtest_pricecontours(default_conf, mocker): simple_backtest(default_conf, contour, numres) -def mocked_load_data(datadir, pairs=[], ticker_interval=0, refresh_pairs=False): - tickerdata = optimize.load_tickerdata_file(datadir, 'BTC_UNITEST', 1) +def mocked_load_data(datadir, pairs=[], ticker_interval=0, refresh_pairs=False, timerange=None): + tickerdata = optimize.load_tickerdata_file(datadir, 'BTC_UNITEST', 1, timerange=timerange) pairdata = {'BTC_UNITEST': tickerdata} - return trim_tickerlist(pairdata, -100) + return pairdata def test_backtest_start(default_conf, mocker, caplog): @@ -159,7 +159,7 @@ def test_backtest_start(default_conf, mocker, caplog): args.level = 10 args.live = False args.datadir = None - args.timerange = None # needed due to MagicMock malleability + args.timerange = '-100' # needed due to MagicMock malleability backtesting.start(args) # check the logs, that will contain the backtest result exists = ['Using max_open_trades: 1 ...', diff --git a/freqtrade/tests/optimize/test_hyperopt.py b/freqtrade/tests/optimize/test_hyperopt.py index ac7b198e4..1c4cceee5 100644 --- a/freqtrade/tests/optimize/test_hyperopt.py +++ b/freqtrade/tests/optimize/test_hyperopt.py @@ -62,7 +62,8 @@ def test_start_calls_fmin(mocker): mocker.patch('freqtrade.optimize.load_data') mock_fmin = mocker.patch('freqtrade.optimize.hyperopt.fmin', return_value={}) - args = mocker.Mock(epochs=1, config='config.json.example', mongodb=False) + args = mocker.Mock(epochs=1, config='config.json.example', mongodb=False, + timerange=None) start(args) mock_fmin.assert_called_once() @@ -75,7 +76,8 @@ def test_start_uses_mongotrials(mocker): mocker.patch('freqtrade.optimize.load_data') mocker.patch('freqtrade.optimize.hyperopt.fmin', return_value={}) - args = mocker.Mock(epochs=1, config='config.json.example', mongodb=True) + args = mocker.Mock(epochs=1, config='config.json.example', mongodb=True, + timerange=None) start(args) mock_mongotrials.assert_called_once() @@ -129,7 +131,8 @@ def test_fmin_best_results(mocker, caplog): mocker.patch('freqtrade.optimize.load_data') mocker.patch('freqtrade.optimize.hyperopt.fmin', return_value=fmin_result) - args = mocker.Mock(epochs=1, config='config.json.example') + args = mocker.Mock(epochs=1, config='config.json.example', + timerange=None) start(args) exists = [ @@ -151,7 +154,8 @@ def test_fmin_throw_value_error(mocker, caplog): mocker.patch('freqtrade.optimize.load_data') mocker.patch('freqtrade.optimize.hyperopt.fmin', side_effect=ValueError()) - args = mocker.Mock(epochs=1, config='config.json.example') + args = mocker.Mock(epochs=1, config='config.json.example', + timerange=None) start(args) exists = [ @@ -185,7 +189,8 @@ def test_resuming_previous_hyperopt_results_succeeds(mocker): return_value={}) args = mocker.Mock(epochs=1, config='config.json.example', - mongodb=False) + mongodb=False, + timerange=None) start(args) diff --git a/freqtrade/tests/optimize/test_optimize.py b/freqtrade/tests/optimize/test_optimize.py index bfbc99745..61325797b 100644 --- a/freqtrade/tests/optimize/test_optimize.py +++ b/freqtrade/tests/optimize/test_optimize.py @@ -177,7 +177,8 @@ def test_load_tickerdata_file(): def test_tickerdata_to_dataframe(): - tick = load_tickerdata_file(None, 'BTC_UNITEST', 1) + timerange = ((None, 'line'), None, -100) + tick = load_tickerdata_file(None, 'BTC_UNITEST', 1, timerange=timerange) tickerlist = {'BTC_UNITEST': tick} - data = optimize.tickerdata_to_dataframe(tickerlist, timerange=-100) + data = optimize.tickerdata_to_dataframe(tickerlist) assert 100 == len(data['BTC_UNITEST']) diff --git a/freqtrade/tests/test_misc.py b/freqtrade/tests/test_misc.py index 63cfba627..74f611f5f 100644 --- a/freqtrade/tests/test_misc.py +++ b/freqtrade/tests/test_misc.py @@ -8,7 +8,7 @@ import pytest from jsonschema import ValidationError from freqtrade.misc import (common_args_parser, load_config, parse_args, - throttle) + throttle, parse_timerange) def test_throttle(): @@ -133,6 +133,13 @@ def test_parse_args_hyperopt_custom(mocker): assert call_args.func is not None +def test_parse_timerange_incorrect(): + assert ((None, 'line'), None, -200) == parse_timerange('-200') + assert (('line', None), 200, None) == parse_timerange('200-') + with pytest.raises(Exception, match=r'Incorrect syntax.*'): + parse_timerange('-') + + def test_load_config(default_conf, mocker): file_mock = mocker.patch('freqtrade.misc.open', mocker.mock_open( read_data=json.dumps(default_conf) From 01e10014bb07d53d4c19b1406cb9f3bce6508909 Mon Sep 17 00:00:00 2001 From: Stephen Dade Date: Tue, 16 Jan 2018 21:55:15 +1100 Subject: [PATCH 25/44] Order timeouts - added exception catching and rpc messaging --- freqtrade/main.py | 15 ++++++++++++++- freqtrade/tests/test_main.py | 12 +++++++++--- 2 files changed, 23 insertions(+), 4 deletions(-) diff --git a/freqtrade/main.py b/freqtrade/main.py index c404d6c11..796fa8515 100755 --- a/freqtrade/main.py +++ b/freqtrade/main.py @@ -129,9 +129,16 @@ def check_handle_timedout(timeoutvalue: int) -> None: timeoutthreashold = arrow.utcnow().shift(minutes=-timeoutvalue).datetime for trade in Trade.query.filter(Trade.open_order_id.isnot(None)).all(): - order = exchange.get_order(trade.open_order_id) + try: + order = exchange.get_order(trade.open_order_id) + except OperationalException: + continue ordertime = arrow.get(order['opened']) + # Check if trade is still actually open + if int(order['remaining']) == 0: + continue + if order['type'] == "LIMIT_BUY" and ordertime < timeoutthreashold: # Buy timeout - cancel order exchange.cancel_order(trade.open_order_id) @@ -140,6 +147,8 @@ def check_handle_timedout(timeoutvalue: int) -> None: Trade.session.delete(trade) Trade.session.flush() logger.info('Buy order timeout for %s.', trade) + rpc.send_msg('*Timeout:* Unfilled buy order for {} cancelled'.format( + trade.pair.replace('_', '/'))) else: # if trade is partially complete, edit the stake details for the trade # and close the order @@ -147,6 +156,8 @@ def check_handle_timedout(timeoutvalue: int) -> None: trade.stake_amount = trade.amount * trade.open_rate trade.open_order_id = None logger.info('Partial buy order timeout for %s.', trade) + rpc.send_msg('*Timeout:* Remaining buy order for {} cancelled'.format( + trade.pair.replace('_', '/'))) elif order['type'] == "LIMIT_SELL" and ordertime < timeoutthreashold: # Sell timeout - cancel order and update trade if order['remaining'] == order['amount']: @@ -157,6 +168,8 @@ def check_handle_timedout(timeoutvalue: int) -> None: trade.close_date = None trade.is_open = True trade.open_order_id = None + rpc.send_msg('*Timeout:* Unfilled sell order for {} cancelled'.format( + trade.pair.replace('_', '/'))) logger.info('Sell order timeout for %s.', trade) return True else: diff --git a/freqtrade/tests/test_main.py b/freqtrade/tests/test_main.py index 97bef2257..95eb0ad82 100644 --- a/freqtrade/tests/test_main.py +++ b/freqtrade/tests/test_main.py @@ -355,7 +355,8 @@ def test_close_trade(default_conf, ticker, limit_buy_order, limit_sell_order, mo def test_check_handle_timedout_buy(default_conf, ticker, limit_buy_order_old, mocker): mocker.patch.dict('freqtrade.main._CONF', default_conf) cancel_order_mock = MagicMock() - mocker.patch.multiple('freqtrade.rpc', init=MagicMock(), send_msg=MagicMock()) + mocker.patch('freqtrade.rpc.init', MagicMock()) + rpc_mock = mocker.patch('freqtrade.main.rpc.send_msg', MagicMock()) mocker.patch.multiple('freqtrade.main.exchange', validate_pairs=MagicMock(), get_ticker=ticker, @@ -380,6 +381,7 @@ def test_check_handle_timedout_buy(default_conf, ticker, limit_buy_order_old, mo # check it does cancel buy orders over the time limit check_handle_timedout(600) assert cancel_order_mock.call_count == 1 + assert rpc_mock.call_count == 1 trades = Trade.query.filter(Trade.open_order_id.is_(trade_buy.open_order_id)).all() assert len(trades) == 0 @@ -387,7 +389,8 @@ def test_check_handle_timedout_buy(default_conf, ticker, limit_buy_order_old, mo def test_check_handle_timedout_sell(default_conf, ticker, limit_sell_order_old, mocker): mocker.patch.dict('freqtrade.main._CONF', default_conf) cancel_order_mock = MagicMock() - mocker.patch.multiple('freqtrade.rpc', init=MagicMock(), send_msg=MagicMock()) + mocker.patch('freqtrade.rpc.init', MagicMock()) + rpc_mock = mocker.patch('freqtrade.main.rpc.send_msg', MagicMock()) mocker.patch.multiple('freqtrade.main.exchange', validate_pairs=MagicMock(), get_ticker=ticker, @@ -413,6 +416,7 @@ def test_check_handle_timedout_sell(default_conf, ticker, limit_sell_order_old, # check it does cancel sell orders over the time limit check_handle_timedout(600) assert cancel_order_mock.call_count == 1 + assert rpc_mock.call_count == 1 assert trade_sell.is_open is True @@ -420,7 +424,8 @@ def test_check_handle_timedout_partial(default_conf, ticker, limit_buy_order_old mocker): mocker.patch.dict('freqtrade.main._CONF', default_conf) cancel_order_mock = MagicMock() - mocker.patch.multiple('freqtrade.rpc', init=MagicMock(), send_msg=MagicMock()) + mocker.patch('freqtrade.rpc.init', MagicMock()) + rpc_mock = mocker.patch('freqtrade.main.rpc.send_msg', MagicMock()) mocker.patch.multiple('freqtrade.main.exchange', validate_pairs=MagicMock(), get_ticker=ticker, @@ -446,6 +451,7 @@ def test_check_handle_timedout_partial(default_conf, ticker, limit_buy_order_old # note this is for a partially-complete buy order check_handle_timedout(600) assert cancel_order_mock.call_count == 1 + assert rpc_mock.call_count == 1 trades = Trade.query.filter(Trade.open_order_id.is_(trade_buy.open_order_id)).all() assert len(trades) == 1 assert trades[0].amount == 23.0 From 38fe7ec7cdae4703cb96588067ea419e10f7acfa Mon Sep 17 00:00:00 2001 From: Janne Sinivirta Date: Tue, 16 Jan 2018 16:35:48 +0200 Subject: [PATCH 26/44] adjust default target values for hyperopt --- freqtrade/optimize/hyperopt.py | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/freqtrade/optimize/hyperopt.py b/freqtrade/optimize/hyperopt.py index 3235a10e6..02a64fa74 100644 --- a/freqtrade/optimize/hyperopt.py +++ b/freqtrade/optimize/hyperopt.py @@ -30,19 +30,19 @@ logging.getLogger('hyperopt.tpe').setLevel(logging.WARNING) logger = logging.getLogger(__name__) # set TARGET_TRADES to suit your number concurrent trades so its realistic to 20days of data -TARGET_TRADES = 1100 +TARGET_TRADES = 600 TOTAL_TRIES = 0 _CURRENT_TRIES = 0 CURRENT_BEST_LOSS = 100 # max average trade duration in minutes # if eval ends with higher value, we consider it a failed eval -MAX_ACCEPTED_TRADE_DURATION = 240 +MAX_ACCEPTED_TRADE_DURATION = 300 # this is expexted avg profit * expected trade count # for example 3.5%, 1100 trades, EXPECTED_MAX_PROFIT = 3.85 # check that the reported Σ% values do not exceed this! -EXPECTED_MAX_PROFIT = 3.85 +EXPECTED_MAX_PROFIT = 3.0 # Configuration and data used by hyperopt PROCESSED = None # optimize.preprocess(optimize.load_data()) From 501be8a3bc2df259765bedf6ad6f3fd6f66cd4a8 Mon Sep 17 00:00:00 2001 From: Janne Sinivirta Date: Tue, 16 Jan 2018 16:36:50 +0200 Subject: [PATCH 27/44] adjust the hyperopt objective function to emphasize profit and allow more variation in trade counts --- freqtrade/optimize/hyperopt.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/freqtrade/optimize/hyperopt.py b/freqtrade/optimize/hyperopt.py index 02a64fa74..f8a494a0a 100644 --- a/freqtrade/optimize/hyperopt.py +++ b/freqtrade/optimize/hyperopt.py @@ -146,9 +146,9 @@ def log_results(results): def calculate_loss(total_profit: float, trade_count: int, trade_duration: float): """ objective function, returns smaller number for more optimal results """ - trade_loss = 1 - 0.35 * exp(-(trade_count - TARGET_TRADES) ** 2 / 10 ** 5.2) + trade_loss = 1 - 0.25 * exp(-(trade_count - TARGET_TRADES) ** 2 / 10 ** 5.8) profit_loss = max(0, 1 - total_profit / EXPECTED_MAX_PROFIT) - duration_loss = min(trade_duration / MAX_ACCEPTED_TRADE_DURATION, 1) + duration_loss = 0.7 + 0.3 * min(trade_duration / MAX_ACCEPTED_TRADE_DURATION, 1) return trade_loss + profit_loss + duration_loss From 3e1a70bbb2510b2aec9aa8b0ac54976a1d29d9cb Mon Sep 17 00:00:00 2001 From: Janne Sinivirta Date: Tue, 16 Jan 2018 10:47:34 +0200 Subject: [PATCH 28/44] enable correct bollinger bands --- freqtrade/analyze.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/freqtrade/analyze.py b/freqtrade/analyze.py index f85c46248..ac8279603 100644 --- a/freqtrade/analyze.py +++ b/freqtrade/analyze.py @@ -114,11 +114,11 @@ def populate_indicators(dataframe: DataFrame) -> DataFrame: dataframe['blower'] = ta.BBANDS(dataframe, nbdevup=2, nbdevdn=2)['lowerband'] """ # Bollinger bands + """ bollinger = qtpylib.bollinger_bands(qtpylib.typical_price(dataframe), window=20, stds=2) dataframe['bb_lowerband'] = bollinger['lower'] dataframe['bb_middleband'] = bollinger['mid'] dataframe['bb_upperband'] = bollinger['upper'] - """ # EMA - Exponential Moving Average dataframe['ema5'] = ta.EMA(dataframe, timeperiod=5) From 99260735ae9202918579e58b8d54193aa7a41a21 Mon Sep 17 00:00:00 2001 From: Janne Sinivirta Date: Tue, 16 Jan 2018 10:48:42 +0200 Subject: [PATCH 29/44] remove broken bbands trigger from hyperopt. add two working bbands triggers --- freqtrade/optimize/hyperopt.py | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/freqtrade/optimize/hyperopt.py b/freqtrade/optimize/hyperopt.py index f8a494a0a..9afea7f79 100644 --- a/freqtrade/optimize/hyperopt.py +++ b/freqtrade/optimize/hyperopt.py @@ -96,6 +96,7 @@ SPACE = { ]), 'trigger': hp.choice('trigger', [ {'type': 'lower_bb'}, + {'type': 'lower_bb_tema'}, {'type': 'faststoch10'}, {'type': 'ao_cross_zero'}, {'type': 'ema5_cross_ema10'}, @@ -227,7 +228,8 @@ def buy_strategy_generator(params): # TRIGGERS triggers = { - 'lower_bb': dataframe['tema'] <= dataframe['blower'], + 'lower_bb': (dataframe['close'] < dataframe['bb_lowerband']), + 'lower_bb_tema': (dataframe['tema'] < dataframe['bb_lowerband']), 'faststoch10': (crossed_above(dataframe['fastd'], 10.0)), 'ao_cross_zero': (crossed_above(dataframe['ao'], 0.0)), 'ema5_cross_ema10': (crossed_above(dataframe['ema5'], dataframe['ema10'])), From fadac5fe4abe89442b80e3cd73b502e9808c6377 Mon Sep 17 00:00:00 2001 From: Janne Sinivirta Date: Tue, 16 Jan 2018 11:13:12 +0200 Subject: [PATCH 30/44] remove too aggressive trigger --- freqtrade/optimize/hyperopt.py | 2 -- 1 file changed, 2 deletions(-) diff --git a/freqtrade/optimize/hyperopt.py b/freqtrade/optimize/hyperopt.py index 9afea7f79..a21549922 100644 --- a/freqtrade/optimize/hyperopt.py +++ b/freqtrade/optimize/hyperopt.py @@ -102,7 +102,6 @@ SPACE = { {'type': 'ema5_cross_ema10'}, {'type': 'macd_cross_signal'}, {'type': 'sar_reversal'}, - {'type': 'stochf_cross'}, {'type': 'ht_sine'}, ]), 'stoploss': hp.uniform('stoploss', -0.5, -0.02), @@ -235,7 +234,6 @@ def buy_strategy_generator(params): 'ema5_cross_ema10': (crossed_above(dataframe['ema5'], dataframe['ema10'])), 'macd_cross_signal': (crossed_above(dataframe['macd'], dataframe['macdsignal'])), 'sar_reversal': (crossed_above(dataframe['close'], dataframe['sar'])), - 'stochf_cross': (crossed_above(dataframe['fastk'], dataframe['fastd'])), 'ht_sine': (crossed_above(dataframe['htleadsine'], dataframe['htsine'])), } conditions.append(triggers.get(params['trigger']['type'])) From dc01807b3ca608139e40259fbe1cc0f3b867f75c Mon Sep 17 00:00:00 2001 From: Janne Sinivirta Date: Tue, 16 Jan 2018 11:16:08 +0200 Subject: [PATCH 31/44] switch ema5 trigger to ema3 cross trigger --- freqtrade/analyze.py | 1 + freqtrade/optimize/hyperopt.py | 4 ++-- 2 files changed, 3 insertions(+), 2 deletions(-) diff --git a/freqtrade/analyze.py b/freqtrade/analyze.py index ac8279603..ecb0b2b57 100644 --- a/freqtrade/analyze.py +++ b/freqtrade/analyze.py @@ -121,6 +121,7 @@ def populate_indicators(dataframe: DataFrame) -> DataFrame: dataframe['bb_upperband'] = bollinger['upper'] # EMA - Exponential Moving Average + dataframe['ema3'] = ta.EMA(dataframe, timeperiod=3) dataframe['ema5'] = ta.EMA(dataframe, timeperiod=5) dataframe['ema10'] = ta.EMA(dataframe, timeperiod=10) dataframe['ema50'] = ta.EMA(dataframe, timeperiod=50) diff --git a/freqtrade/optimize/hyperopt.py b/freqtrade/optimize/hyperopt.py index a21549922..9b71900c5 100644 --- a/freqtrade/optimize/hyperopt.py +++ b/freqtrade/optimize/hyperopt.py @@ -99,7 +99,7 @@ SPACE = { {'type': 'lower_bb_tema'}, {'type': 'faststoch10'}, {'type': 'ao_cross_zero'}, - {'type': 'ema5_cross_ema10'}, + {'type': 'ema3_cross_ema10'}, {'type': 'macd_cross_signal'}, {'type': 'sar_reversal'}, {'type': 'ht_sine'}, @@ -231,7 +231,7 @@ def buy_strategy_generator(params): 'lower_bb_tema': (dataframe['tema'] < dataframe['bb_lowerband']), 'faststoch10': (crossed_above(dataframe['fastd'], 10.0)), 'ao_cross_zero': (crossed_above(dataframe['ao'], 0.0)), - 'ema5_cross_ema10': (crossed_above(dataframe['ema5'], dataframe['ema10'])), + 'ema3_cross_ema10': (crossed_above(dataframe['ema3'], dataframe['ema10'])), 'macd_cross_signal': (crossed_above(dataframe['macd'], dataframe['macdsignal'])), 'sar_reversal': (crossed_above(dataframe['close'], dataframe['sar'])), 'ht_sine': (crossed_above(dataframe['htleadsine'], dataframe['htsine'])), From ce963aae5822c9c6ad5b2a222657a9468956205c Mon Sep 17 00:00:00 2001 From: Janne Sinivirta Date: Tue, 16 Jan 2018 13:31:45 +0200 Subject: [PATCH 32/44] add macd < 0 guard to hyperopt --- freqtrade/optimize/hyperopt.py | 6 ++++++ freqtrade/tests/optimize/test_hyperopt.py | 3 ++- 2 files changed, 8 insertions(+), 1 deletion(-) diff --git a/freqtrade/optimize/hyperopt.py b/freqtrade/optimize/hyperopt.py index 9b71900c5..0fdd2fc01 100644 --- a/freqtrade/optimize/hyperopt.py +++ b/freqtrade/optimize/hyperopt.py @@ -58,6 +58,10 @@ main._CONF = OPTIMIZE_CONFIG SPACE = { + 'macd_below_zero': hp.choice('macd_below_zero', [ + {'enabled': False}, + {'enabled': True} + ]), 'mfi': hp.choice('mfi', [ {'enabled': False}, {'enabled': True, 'value': hp.quniform('mfi-value', 5, 25, 1)} @@ -207,6 +211,8 @@ def buy_strategy_generator(params): # GUARDS AND TRENDS if params['uptrend_long_ema']['enabled']: conditions.append(dataframe['ema50'] > dataframe['ema100']) + if params['macd_below_zero']['enabled']: + conditions.append(dataframe['macd'] < 0) if params['uptrend_short_ema']['enabled']: conditions.append(dataframe['ema5'] > dataframe['ema10']) if params['mfi']['enabled']: diff --git a/freqtrade/tests/optimize/test_hyperopt.py b/freqtrade/tests/optimize/test_hyperopt.py index 3e03d26c0..4bb5c8c0b 100644 --- a/freqtrade/tests/optimize/test_hyperopt.py +++ b/freqtrade/tests/optimize/test_hyperopt.py @@ -107,6 +107,7 @@ def test_no_log_if_loss_does_not_improve(mocker): def test_fmin_best_results(mocker, caplog): fmin_result = { + "macd_below_zero": 0, "adx": 1, "adx-value": 15.0, "fastd": 1, @@ -136,7 +137,7 @@ def test_fmin_best_results(mocker, caplog): '"adx": {\n "enabled": true,\n "value": 15.0\n },', '"green_candle": {\n "enabled": true\n },', '"mfi": {\n "enabled": false\n },', - '"trigger": {\n "type": "ao_cross_zero"\n },', + '"trigger": {\n "type": "faststoch10"\n },', '"stoploss": -0.1', ] From 8896b39231e43886f20f0d5f34cbf85a6481c4a9 Mon Sep 17 00:00:00 2001 From: Janne Sinivirta Date: Tue, 16 Jan 2018 13:53:30 +0200 Subject: [PATCH 33/44] add heikenashi reversal bullish trigger to hyperopt --- freqtrade/analyze.py | 2 -- freqtrade/optimize/hyperopt.py | 3 +++ 2 files changed, 3 insertions(+), 2 deletions(-) diff --git a/freqtrade/analyze.py b/freqtrade/analyze.py index ecb0b2b57..3d3f9b5cc 100644 --- a/freqtrade/analyze.py +++ b/freqtrade/analyze.py @@ -211,14 +211,12 @@ def populate_indicators(dataframe: DataFrame) -> DataFrame: # Chart type # ------------------------------------ - """ # Heikinashi stategy heikinashi = qtpylib.heikinashi(dataframe) dataframe['ha_open'] = heikinashi['open'] dataframe['ha_close'] = heikinashi['close'] dataframe['ha_high'] = heikinashi['high'] dataframe['ha_low'] = heikinashi['low'] - """ return dataframe diff --git a/freqtrade/optimize/hyperopt.py b/freqtrade/optimize/hyperopt.py index 0fdd2fc01..cb681cf36 100644 --- a/freqtrade/optimize/hyperopt.py +++ b/freqtrade/optimize/hyperopt.py @@ -107,6 +107,7 @@ SPACE = { {'type': 'macd_cross_signal'}, {'type': 'sar_reversal'}, {'type': 'ht_sine'}, + {'type': 'heiken_reversal_bull'}, ]), 'stoploss': hp.uniform('stoploss', -0.5, -0.02), } @@ -241,6 +242,8 @@ def buy_strategy_generator(params): 'macd_cross_signal': (crossed_above(dataframe['macd'], dataframe['macdsignal'])), 'sar_reversal': (crossed_above(dataframe['close'], dataframe['sar'])), 'ht_sine': (crossed_above(dataframe['htleadsine'], dataframe['htsine'])), + 'heiken_reversal_bull': (crossed_above(dataframe['ha_close'], dataframe['ha_open'])) & + (dataframe['ha_low'] == dataframe['ha_open']), } conditions.append(triggers.get(params['trigger']['type'])) From c670ccfd376b23c798a01cf26b0250eccaaeb8d5 Mon Sep 17 00:00:00 2001 From: Janne Sinivirta Date: Tue, 16 Jan 2018 18:51:22 +0200 Subject: [PATCH 34/44] add trigger +DI crossed above -DI --- freqtrade/analyze.py | 2 ++ freqtrade/optimize/hyperopt.py | 2 ++ 2 files changed, 4 insertions(+) diff --git a/freqtrade/analyze.py b/freqtrade/analyze.py index 3d3f9b5cc..0d01576df 100644 --- a/freqtrade/analyze.py +++ b/freqtrade/analyze.py @@ -74,6 +74,8 @@ def populate_indicators(dataframe: DataFrame) -> DataFrame: # Plus Directional Indicator / Movement dataframe['plus_dm'] = ta.PLUS_DM(dataframe) dataframe['plus_di'] = ta.PLUS_DI(dataframe) + dataframe['minus_di'] = ta.MINUS_DI(dataframe) + """ # ROC dataframe['roc'] = ta.ROC(dataframe) diff --git a/freqtrade/optimize/hyperopt.py b/freqtrade/optimize/hyperopt.py index cb681cf36..959e64206 100644 --- a/freqtrade/optimize/hyperopt.py +++ b/freqtrade/optimize/hyperopt.py @@ -108,6 +108,7 @@ SPACE = { {'type': 'sar_reversal'}, {'type': 'ht_sine'}, {'type': 'heiken_reversal_bull'}, + {'type': 'di_cross'}, ]), 'stoploss': hp.uniform('stoploss', -0.5, -0.02), } @@ -244,6 +245,7 @@ def buy_strategy_generator(params): 'ht_sine': (crossed_above(dataframe['htleadsine'], dataframe['htsine'])), 'heiken_reversal_bull': (crossed_above(dataframe['ha_close'], dataframe['ha_open'])) & (dataframe['ha_low'] == dataframe['ha_open']), + 'di_cross': (crossed_above(dataframe['plus_di'], dataframe['minus_di'])), } conditions.append(triggers.get(params['trigger']['type'])) From 12ffbf5047ffbdb00a7c94e7b1b4e9ee44902567 Mon Sep 17 00:00:00 2001 From: toto Date: Tue, 16 Jan 2018 20:22:15 +0100 Subject: [PATCH 35/44] - get_signal to return both SELL and BUY signal - _process modified so that we do not sell if we would buy afterwards - execute_sell modified so that that min_roi_reached is not executed if we would buy afterwards Veuillez saisir le message de validation pour vos modifications. Les lignes --- freqtrade/analyze.py | 20 ++++----- freqtrade/main.py | 35 ++++++++------- freqtrade/tests/rpc/test_rpc_telegram.py | 22 ++++----- freqtrade/tests/test_main.py | 57 +++++++++++++----------- 4 files changed, 71 insertions(+), 63 deletions(-) diff --git a/freqtrade/analyze.py b/freqtrade/analyze.py index f85c46248..936af1e29 100644 --- a/freqtrade/analyze.py +++ b/freqtrade/analyze.py @@ -280,36 +280,36 @@ def analyze_ticker(ticker_history: List[Dict]) -> DataFrame: return dataframe -def get_signal(pair: str, signal: SignalType) -> bool: +def get_signal(pair: str) -> (bool, bool): """ Calculates current signal based several technical analysis indicators :param pair: pair in format BTC_ANT or BTC-ANT - :return: True if pair is good for buying, False otherwise + :return: (True, False) if pair is good for buying and not for selling """ ticker_hist = get_ticker_history(pair) if not ticker_hist: logger.warning('Empty ticker history for pair %s', pair) - return False + return (False, False) try: dataframe = analyze_ticker(ticker_hist) except ValueError as ex: logger.warning('Unable to analyze ticker for pair %s: %s', pair, str(ex)) - return False + return (False, False) except Exception as ex: logger.exception('Unexpected error when analyzing ticker for pair %s: %s', pair, str(ex)) - return False + return (False, False) if dataframe.empty: - return False + return (False, False) latest = dataframe.iloc[-1] # Check if dataframe is out of date signal_date = arrow.get(latest['date']) if signal_date < arrow.now() - timedelta(minutes=10): - return False + return (False, False) - result = latest[signal.value] == 1 - logger.debug('%s_trigger: %s (pair=%s, signal=%s)', signal.value, latest['date'], pair, result) - return result + (buy, sell) = latest[SignalType.BUY] == 1, latest[SignalType.SELL] == 1 + logger.debug('%trigger: %s (pair=%s, buy=%s sell=%s)', latest['date'], pair, buy, sell) + return (buy, sell) diff --git a/freqtrade/main.py b/freqtrade/main.py index c404d6c11..1640c7836 100755 --- a/freqtrade/main.py +++ b/freqtrade/main.py @@ -14,7 +14,7 @@ from cachetools import cached, TTLCache from freqtrade import (DependencyException, OperationalException, __version__, exchange, persistence, rpc) -from freqtrade.analyze import SignalType, get_signal +from freqtrade.analyze import get_signal from freqtrade.fiat_convert import CryptoToFiatConverter from freqtrade.misc import (State, get_state, load_config, parse_args, throttle, update_state) @@ -247,24 +247,28 @@ def handle_trade(trade: Trade) -> bool: logger.debug('Handling %s ...', trade) current_rate = exchange.get_ticker(trade.pair)['bid'] - # Check if minimal roi has been reached - if min_roi_reached(trade, current_rate, datetime.utcnow()): + (buy, sell) = (False, False) + + if _CONF.get('experimental', {}).get('use_sell_signal'): + (buy, sell) = get_signal(trade.pair) + + # Check if minimal roi has been reached and no longer in buy conditions (avoiding a fee) + if not buy and min_roi_reached(trade, current_rate, datetime.utcnow()): logger.debug('Executing sell due to ROI ...') execute_sell(trade, current_rate) return True + # Experimental: Check if the trade is profitable before selling it (avoid selling at loss) + if _CONF.get('experimental', {}).get('sell_profit_only'): + logger.debug('Checking if trade is profitable ...') + if not buy and trade.calc_profit(rate=current_rate) <= 0: + return False + # Experimental: Check if sell signal has been enabled and triggered - if _CONF.get('experimental', {}).get('use_sell_signal'): - # Experimental: Check if the trade is profitable before selling it (avoid selling at loss) - if _CONF.get('experimental', {}).get('sell_profit_only'): - logger.debug('Checking if trade is profitable ...') - if trade.calc_profit(rate=current_rate) <= 0: - return False - logger.debug('Checking sell_signal ...') - if get_signal(trade.pair, SignalType.SELL): - logger.debug('Executing sell due to sell signal ...') - execute_sell(trade, current_rate) - return True + if sell and not buy: + logger.debug('Executing sell due to sell signal ...') + execute_sell(trade, current_rate) + return True return False @@ -305,7 +309,8 @@ def create_trade(stake_amount: float) -> bool: # Pick pair based on StochRSI buy signals for _pair in whitelist: - if get_signal(_pair, SignalType.BUY): + (buy, sell) = get_signal(_pair) + if buy and not sell: pair = _pair break else: diff --git a/freqtrade/tests/rpc/test_rpc_telegram.py b/freqtrade/tests/rpc/test_rpc_telegram.py index 58bf0154b..992693248 100644 --- a/freqtrade/tests/rpc/test_rpc_telegram.py +++ b/freqtrade/tests/rpc/test_rpc_telegram.py @@ -77,7 +77,7 @@ def test_authorized_only_exception(default_conf, mocker): def test_status_handle(default_conf, update, ticker, mocker): mocker.patch.dict('freqtrade.main._CONF', default_conf) - mocker.patch('freqtrade.main.get_signal', side_effect=lambda s, t: True) + mocker.patch('freqtrade.main.get_signal', side_effect=lambda s: (True, False)) msg_mock = MagicMock() mocker.patch('freqtrade.main.rpc.send_msg', MagicMock()) mocker.patch.multiple('freqtrade.rpc.telegram', @@ -112,7 +112,7 @@ def test_status_handle(default_conf, update, ticker, mocker): def test_status_table_handle(default_conf, update, ticker, mocker): mocker.patch.dict('freqtrade.main._CONF', default_conf) - mocker.patch('freqtrade.main.get_signal', side_effect=lambda s, t: True) + mocker.patch('freqtrade.main.get_signal', side_effect=lambda s: (True, False)) msg_mock = MagicMock() mocker.patch('freqtrade.main.rpc.send_msg', MagicMock()) mocker.patch.multiple( @@ -154,7 +154,7 @@ def test_status_table_handle(default_conf, update, ticker, mocker): def test_profit_handle( default_conf, update, ticker, ticker_sell_up, limit_buy_order, limit_sell_order, mocker): mocker.patch.dict('freqtrade.main._CONF', default_conf) - mocker.patch('freqtrade.main.get_signal', side_effect=lambda s, t: True) + mocker.patch('freqtrade.main.get_signal', side_effect=lambda s: (True, False)) msg_mock = MagicMock() mocker.patch('freqtrade.main.rpc.send_msg', MagicMock()) mocker.patch.multiple('freqtrade.rpc.telegram', @@ -210,7 +210,7 @@ def test_profit_handle( def test_forcesell_handle(default_conf, update, ticker, ticker_sell_up, mocker): mocker.patch.dict('freqtrade.main._CONF', default_conf) - mocker.patch('freqtrade.main.get_signal', side_effect=lambda s, t: True) + mocker.patch('freqtrade.main.get_signal', side_effect=lambda s: (True, False)) rpc_mock = mocker.patch('freqtrade.main.rpc.send_msg', MagicMock()) mocker.patch.multiple('freqtrade.rpc.telegram', _CONF=default_conf, @@ -247,7 +247,7 @@ def test_forcesell_handle(default_conf, update, ticker, ticker_sell_up, mocker): def test_forcesell_down_handle(default_conf, update, ticker, ticker_sell_down, mocker): mocker.patch.dict('freqtrade.main._CONF', default_conf) - mocker.patch('freqtrade.main.get_signal', side_effect=lambda s, t: True) + mocker.patch('freqtrade.main.get_signal', side_effect=lambda s: (True, False)) rpc_mock = mocker.patch('freqtrade.main.rpc.send_msg', MagicMock()) mocker.patch.multiple('freqtrade.rpc.telegram', _CONF=default_conf, @@ -308,7 +308,7 @@ def test_exec_forcesell_open_orders(default_conf, ticker, mocker): def test_forcesell_all_handle(default_conf, update, ticker, mocker): mocker.patch.dict('freqtrade.main._CONF', default_conf) - mocker.patch('freqtrade.main.get_signal', side_effect=lambda s, t: True) + mocker.patch('freqtrade.main.get_signal', side_effect=lambda s: (True, False)) rpc_mock = mocker.patch('freqtrade.main.rpc.send_msg', MagicMock()) mocker.patch.multiple('freqtrade.rpc.telegram', _CONF=default_conf, @@ -339,7 +339,7 @@ def test_forcesell_all_handle(default_conf, update, ticker, mocker): def test_forcesell_handle_invalid(default_conf, update, mocker): mocker.patch.dict('freqtrade.main._CONF', default_conf) - mocker.patch('freqtrade.main.get_signal', side_effect=lambda s, t: True) + mocker.patch('freqtrade.main.get_signal', side_effect=lambda s: (True, True)) msg_mock = MagicMock() mocker.patch.multiple('freqtrade.rpc.telegram', _CONF=default_conf, @@ -376,7 +376,7 @@ def test_forcesell_handle_invalid(default_conf, update, mocker): def test_performance_handle( default_conf, update, ticker, limit_buy_order, limit_sell_order, mocker): mocker.patch.dict('freqtrade.main._CONF', default_conf) - mocker.patch('freqtrade.main.get_signal', side_effect=lambda s, t: True) + mocker.patch('freqtrade.main.get_signal', side_effect=lambda s: (True, False)) msg_mock = MagicMock() mocker.patch('freqtrade.main.rpc.send_msg', MagicMock()) mocker.patch.multiple('freqtrade.rpc.telegram', @@ -410,7 +410,7 @@ def test_performance_handle( def test_daily_handle( default_conf, update, ticker, limit_buy_order, limit_sell_order, mocker): mocker.patch.dict('freqtrade.main._CONF', default_conf) - mocker.patch('freqtrade.main.get_signal', side_effect=lambda s, t: True) + mocker.patch('freqtrade.main.get_signal', side_effect=lambda s: (True, False)) msg_mock = MagicMock() mocker.patch('freqtrade.main.rpc.send_msg', MagicMock()) mocker.patch.multiple('freqtrade.rpc.telegram', @@ -460,7 +460,7 @@ def test_daily_handle( def test_count_handle(default_conf, update, ticker, mocker): mocker.patch.dict('freqtrade.main._CONF', default_conf) - mocker.patch('freqtrade.main.get_signal', side_effect=lambda s, t: True) + mocker.patch('freqtrade.main.get_signal', side_effect=lambda s: (True, False)) msg_mock = MagicMock() mocker.patch.multiple( 'freqtrade.rpc.telegram', @@ -492,7 +492,7 @@ def test_count_handle(default_conf, update, ticker, mocker): def test_performance_handle_invalid(default_conf, update, mocker): mocker.patch.dict('freqtrade.main._CONF', default_conf) - mocker.patch('freqtrade.main.get_signal', side_effect=lambda s, t: True) + mocker.patch('freqtrade.main.get_signal', side_effect=lambda s: (True, True)) msg_mock = MagicMock() mocker.patch.multiple('freqtrade.rpc.telegram', _CONF=default_conf, diff --git a/freqtrade/tests/test_main.py b/freqtrade/tests/test_main.py index 97bef2257..c99727135 100644 --- a/freqtrade/tests/test_main.py +++ b/freqtrade/tests/test_main.py @@ -10,7 +10,6 @@ from sqlalchemy import create_engine import freqtrade.main as main from freqtrade import DependencyException, OperationalException -from freqtrade.analyze import SignalType from freqtrade.exchange import Exchanges from freqtrade.main import (_process, check_handle_timedout, create_trade, execute_sell, get_target_bid, handle_trade, init) @@ -52,7 +51,7 @@ def test_main_start_hyperopt(mocker): def test_process_trade_creation(default_conf, ticker, limit_buy_order, health, mocker): mocker.patch.dict('freqtrade.main._CONF', default_conf) mocker.patch.multiple('freqtrade.rpc', init=MagicMock(), send_msg=MagicMock()) - mocker.patch('freqtrade.main.get_signal', side_effect=lambda s, t: True) + mocker.patch('freqtrade.main.get_signal', side_effect=lambda s: (True, False)) mocker.patch.multiple('freqtrade.main.exchange', validate_pairs=MagicMock(), get_ticker=ticker, @@ -82,7 +81,7 @@ def test_process_trade_creation(default_conf, ticker, limit_buy_order, health, m def test_process_exchange_failures(default_conf, ticker, health, mocker): mocker.patch.dict('freqtrade.main._CONF', default_conf) mocker.patch.multiple('freqtrade.rpc', init=MagicMock(), send_msg=MagicMock()) - mocker.patch('freqtrade.main.get_signal', side_effect=lambda s, t: True) + mocker.patch('freqtrade.main.get_signal', side_effect=lambda s: (True, False)) sleep_mock = mocker.patch('time.sleep', side_effect=lambda _: None) mocker.patch.multiple('freqtrade.main.exchange', validate_pairs=MagicMock(), @@ -99,7 +98,7 @@ def test_process_operational_exception(default_conf, ticker, health, mocker): msg_mock = MagicMock() mocker.patch.dict('freqtrade.main._CONF', default_conf) mocker.patch.multiple('freqtrade.rpc', init=MagicMock(), send_msg=msg_mock) - mocker.patch('freqtrade.main.get_signal', side_effect=lambda s, t: True) + mocker.patch('freqtrade.main.get_signal', side_effect=lambda s: (True, False)) mocker.patch.multiple('freqtrade.main.exchange', validate_pairs=MagicMock(), get_ticker=ticker, @@ -117,8 +116,7 @@ def test_process_operational_exception(default_conf, ticker, health, mocker): def test_process_trade_handling(default_conf, ticker, limit_buy_order, health, mocker): mocker.patch.dict('freqtrade.main._CONF', default_conf) mocker.patch.multiple('freqtrade.rpc', init=MagicMock(), send_msg=MagicMock()) - mocker.patch('freqtrade.main.get_signal', - side_effect=lambda *args: False if args[1] == SignalType.SELL else True) + mocker.patch('freqtrade.main.get_signal', side_effect=lambda s: (True, False)) mocker.patch.multiple('freqtrade.main.exchange', validate_pairs=MagicMock(), get_ticker=ticker, @@ -140,7 +138,7 @@ def test_process_trade_handling(default_conf, ticker, limit_buy_order, health, m def test_create_trade(default_conf, ticker, limit_buy_order, mocker): mocker.patch.dict('freqtrade.main._CONF', default_conf) - mocker.patch('freqtrade.main.get_signal', side_effect=lambda s, t: True) + mocker.patch('freqtrade.main.get_signal', side_effect=lambda s: (True, False)) mocker.patch.multiple('freqtrade.rpc', init=MagicMock(), send_msg=MagicMock()) mocker.patch.multiple('freqtrade.main.exchange', validate_pairs=MagicMock(), @@ -171,7 +169,7 @@ def test_create_trade(default_conf, ticker, limit_buy_order, mocker): def test_create_trade_minimal_amount(default_conf, ticker, mocker): mocker.patch.dict('freqtrade.main._CONF', default_conf) mocker.patch.multiple('freqtrade.rpc', init=MagicMock(), send_msg=MagicMock()) - mocker.patch('freqtrade.main.get_signal', side_effect=lambda s, t: True) + mocker.patch('freqtrade.main.get_signal', side_effect=lambda s: (True, False)) buy_mock = mocker.patch( 'freqtrade.main.exchange.buy', MagicMock(return_value='mocked_limit_buy') ) @@ -187,7 +185,7 @@ def test_create_trade_minimal_amount(default_conf, ticker, mocker): def test_create_trade_no_stake_amount(default_conf, ticker, mocker): mocker.patch.dict('freqtrade.main._CONF', default_conf) - mocker.patch('freqtrade.main.get_signal', side_effect=lambda s, t: True) + mocker.patch('freqtrade.main.get_signal', side_effect=lambda s: (True, False)) mocker.patch.multiple('freqtrade.rpc', init=MagicMock(), send_msg=MagicMock()) mocker.patch.multiple('freqtrade.main.exchange', validate_pairs=MagicMock(), @@ -200,7 +198,7 @@ def test_create_trade_no_stake_amount(default_conf, ticker, mocker): def test_create_trade_no_pairs(default_conf, ticker, mocker): mocker.patch.dict('freqtrade.main._CONF', default_conf) - mocker.patch('freqtrade.main.get_signal', side_effect=lambda s, t: True) + mocker.patch('freqtrade.main.get_signal', side_effect=lambda s: (True, False)) mocker.patch.multiple('freqtrade.rpc', init=MagicMock(), send_msg=MagicMock()) mocker.patch.multiple('freqtrade.main.exchange', validate_pairs=MagicMock(), @@ -216,7 +214,7 @@ def test_create_trade_no_pairs(default_conf, ticker, mocker): def test_create_trade_no_pairs_after_blacklist(default_conf, ticker, mocker): mocker.patch.dict('freqtrade.main._CONF', default_conf) - mocker.patch('freqtrade.main.get_signal', side_effect=lambda s, t: True) + mocker.patch('freqtrade.main.get_signal', side_effect=lambda s: (True, False)) mocker.patch.multiple('freqtrade.rpc', init=MagicMock(), send_msg=MagicMock()) mocker.patch.multiple('freqtrade.main.exchange', validate_pairs=MagicMock(), @@ -233,7 +231,7 @@ def test_create_trade_no_pairs_after_blacklist(default_conf, ticker, mocker): def test_handle_trade(default_conf, limit_buy_order, limit_sell_order, mocker): mocker.patch.dict('freqtrade.main._CONF', default_conf) - mocker.patch('freqtrade.main.get_signal', side_effect=lambda s, t: True) + mocker.patch('freqtrade.main.get_signal', side_effect=lambda s: (True, False)) mocker.patch.multiple('freqtrade.rpc', init=MagicMock(), send_msg=MagicMock()) mocker.patch.multiple('freqtrade.main.exchange', validate_pairs=MagicMock(), @@ -256,6 +254,7 @@ def test_handle_trade(default_conf, limit_buy_order, limit_sell_order, mocker): trade.update(limit_buy_order) assert trade.is_open is True + mocker.patch('freqtrade.main.get_signal', side_effect=lambda s: (False, True)) handle_trade(trade) assert trade.open_order_id == 'mocked_limit_sell' @@ -272,7 +271,7 @@ def test_handle_trade_roi(default_conf, ticker, mocker, caplog): default_conf.update({'experimental': {'use_sell_signal': True}}) mocker.patch.dict('freqtrade.main._CONF', default_conf) - mocker.patch('freqtrade.main.get_signal', side_effect=lambda s, t: True) + mocker.patch('freqtrade.main.get_signal', side_effect=lambda s: (True, False)) mocker.patch.multiple('freqtrade.rpc', init=MagicMock(), send_msg=MagicMock()) mocker.patch.multiple('freqtrade.main.exchange', validate_pairs=MagicMock(), @@ -291,11 +290,11 @@ def test_handle_trade_roi(default_conf, ticker, mocker, caplog): # we might just want to check if we are in a sell condition without # executing # if ROI is reached we must sell - mocker.patch('freqtrade.main.get_signal', side_effect=lambda s, t: False) + mocker.patch('freqtrade.main.get_signal', side_effect=lambda s: (False, True)) assert handle_trade(trade) assert ('freqtrade', logging.DEBUG, 'Executing sell due to ROI ...') in caplog.record_tuples # if ROI is reached we must sell even if sell-signal is not signalled - mocker.patch('freqtrade.main.get_signal', side_effect=lambda s, t: True) + mocker.patch('freqtrade.main.get_signal', side_effect=lambda s: (False, True)) assert handle_trade(trade) assert ('freqtrade', logging.DEBUG, 'Executing sell due to ROI ...') in caplog.record_tuples @@ -304,7 +303,7 @@ def test_handle_trade_experimental(default_conf, ticker, mocker, caplog): default_conf.update({'experimental': {'use_sell_signal': True}}) mocker.patch.dict('freqtrade.main._CONF', default_conf) - mocker.patch('freqtrade.main.get_signal', side_effect=lambda s, t: True) + mocker.patch('freqtrade.main.get_signal', side_effect=lambda s: (True, False)) mocker.patch.multiple('freqtrade.rpc', init=MagicMock(), send_msg=MagicMock()) mocker.patch.multiple('freqtrade.main.exchange', validate_pairs=MagicMock(), @@ -318,11 +317,10 @@ def test_handle_trade_experimental(default_conf, ticker, mocker, caplog): trade = Trade.query.first() trade.is_open = True - mocker.patch('freqtrade.main.get_signal', side_effect=lambda s, t: False) + mocker.patch('freqtrade.main.get_signal', side_effect=lambda s: (False, False)) value_returned = handle_trade(trade) - assert ('freqtrade', logging.DEBUG, 'Checking sell_signal ...') in caplog.record_tuples assert value_returned is False - mocker.patch('freqtrade.main.get_signal', side_effect=lambda s, t: True) + mocker.patch('freqtrade.main.get_signal', side_effect=lambda s: (False, True)) assert handle_trade(trade) s = 'Executing sell due to sell signal ...' assert ('freqtrade', logging.DEBUG, s) in caplog.record_tuples @@ -330,7 +328,7 @@ def test_handle_trade_experimental(default_conf, ticker, mocker, caplog): def test_close_trade(default_conf, ticker, limit_buy_order, limit_sell_order, mocker): mocker.patch.dict('freqtrade.main._CONF', default_conf) - mocker.patch('freqtrade.main.get_signal', side_effect=lambda s, t: True) + mocker.patch('freqtrade.main.get_signal', side_effect=lambda s: (True, False)) mocker.patch.multiple('freqtrade.rpc', init=MagicMock(), send_msg=MagicMock()) mocker.patch.multiple('freqtrade.main.exchange', validate_pairs=MagicMock(), @@ -469,7 +467,7 @@ def test_balance_bigger_last_ask(mocker): def test_execute_sell_up(default_conf, ticker, ticker_sell_up, mocker): mocker.patch.dict('freqtrade.main._CONF', default_conf) - mocker.patch('freqtrade.main.get_signal', side_effect=lambda s, t: True) + mocker.patch('freqtrade.main.get_signal', side_effect=lambda s: (True, False)) mocker.patch('freqtrade.rpc.init', MagicMock()) rpc_mock = mocker.patch('freqtrade.main.rpc.send_msg', MagicMock()) mocker.patch.multiple('freqtrade.main.exchange', @@ -502,7 +500,7 @@ def test_execute_sell_up(default_conf, ticker, ticker_sell_up, mocker): def test_execute_sell_down(default_conf, ticker, ticker_sell_down, mocker): mocker.patch.dict('freqtrade.main._CONF', default_conf) - mocker.patch('freqtrade.main.get_signal', side_effect=lambda s, t: True) + mocker.patch('freqtrade.main.get_signal', side_effect=lambda s: (True, False)) mocker.patch('freqtrade.rpc.init', MagicMock()) rpc_mock = mocker.patch('freqtrade.main.rpc.send_msg', MagicMock()) mocker.patch.multiple('freqtrade.rpc.telegram', @@ -539,7 +537,7 @@ def test_execute_sell_down(default_conf, ticker, ticker_sell_down, mocker): def test_execute_sell_without_conf(default_conf, ticker, ticker_sell_up, mocker): mocker.patch.dict('freqtrade.main._CONF', default_conf) - mocker.patch('freqtrade.main.get_signal', side_effect=lambda s, t: True) + mocker.patch('freqtrade.main.get_signal', side_effect=lambda s: (True, False)) mocker.patch('freqtrade.rpc.init', MagicMock()) rpc_mock = mocker.patch('freqtrade.main.rpc.send_msg', MagicMock()) mocker.patch.multiple('freqtrade.main.exchange', @@ -576,7 +574,7 @@ def test_sell_profit_only_enable_profit(default_conf, limit_buy_order, mocker): mocker.patch.dict('freqtrade.main._CONF', default_conf) mocker.patch('freqtrade.main.min_roi_reached', return_value=False) - mocker.patch('freqtrade.main.get_signal', side_effect=lambda s, t: True) + mocker.patch('freqtrade.main.get_signal', side_effect=lambda s: (True, False)) mocker.patch.multiple('freqtrade.rpc', init=MagicMock(), send_msg=MagicMock()) mocker.patch.multiple('freqtrade.main.exchange', validate_pairs=MagicMock(), @@ -592,6 +590,7 @@ def test_sell_profit_only_enable_profit(default_conf, limit_buy_order, mocker): trade = Trade.query.first() trade.update(limit_buy_order) + mocker.patch('freqtrade.main.get_signal', side_effect=lambda s: (False, True)) assert handle_trade(trade) is True @@ -603,7 +602,7 @@ def test_sell_profit_only_disable_profit(default_conf, limit_buy_order, mocker): mocker.patch.dict('freqtrade.main._CONF', default_conf) mocker.patch('freqtrade.main.min_roi_reached', return_value=False) - mocker.patch('freqtrade.main.get_signal', side_effect=lambda s, t: True) + mocker.patch('freqtrade.main.get_signal', side_effect=lambda s: (True, False)) mocker.patch.multiple('freqtrade.rpc', init=MagicMock(), send_msg=MagicMock()) mocker.patch.multiple('freqtrade.main.exchange', validate_pairs=MagicMock(), @@ -619,6 +618,7 @@ def test_sell_profit_only_disable_profit(default_conf, limit_buy_order, mocker): trade = Trade.query.first() trade.update(limit_buy_order) + mocker.patch('freqtrade.main.get_signal', side_effect=lambda s: (False, True)) assert handle_trade(trade) is True @@ -630,7 +630,7 @@ def test_sell_profit_only_enable_loss(default_conf, limit_buy_order, mocker): mocker.patch.dict('freqtrade.main._CONF', default_conf) mocker.patch('freqtrade.main.min_roi_reached', return_value=False) - mocker.patch('freqtrade.main.get_signal', side_effect=lambda s, t: True) + mocker.patch('freqtrade.main.get_signal', side_effect=lambda s: (True, False)) mocker.patch.multiple('freqtrade.rpc', init=MagicMock(), send_msg=MagicMock()) mocker.patch.multiple('freqtrade.main.exchange', validate_pairs=MagicMock(), @@ -646,6 +646,7 @@ def test_sell_profit_only_enable_loss(default_conf, limit_buy_order, mocker): trade = Trade.query.first() trade.update(limit_buy_order) + mocker.patch('freqtrade.main.get_signal', side_effect=lambda s: (False, True)) assert handle_trade(trade) is False @@ -657,7 +658,7 @@ def test_sell_profit_only_disable_loss(default_conf, limit_buy_order, mocker): mocker.patch.dict('freqtrade.main._CONF', default_conf) mocker.patch('freqtrade.main.min_roi_reached', return_value=False) - mocker.patch('freqtrade.main.get_signal', side_effect=lambda s, t: True) + mocker.patch('freqtrade.main.get_signal', side_effect=lambda s: (True, False)) mocker.patch.multiple('freqtrade.rpc', init=MagicMock(), send_msg=MagicMock()) mocker.patch.multiple('freqtrade.main.exchange', validate_pairs=MagicMock(), @@ -673,4 +674,6 @@ def test_sell_profit_only_disable_loss(default_conf, limit_buy_order, mocker): trade = Trade.query.first() trade.update(limit_buy_order) + + mocker.patch('freqtrade.main.get_signal', side_effect=lambda s: (False, True)) assert handle_trade(trade) is True From 6dd48fb820c1f1cdfa2773bc75493b1717c6baa1 Mon Sep 17 00:00:00 2001 From: toto Date: Tue, 16 Jan 2018 21:18:43 +0100 Subject: [PATCH 36/44] Adding unitest --- freqtrade/analyze.py | 4 +-- freqtrade/tests/test_analyze.py | 18 ++++++------- freqtrade/tests/test_main.py | 46 +++++++++++++++++++++++++++++++++ 3 files changed, 57 insertions(+), 11 deletions(-) diff --git a/freqtrade/analyze.py b/freqtrade/analyze.py index 936af1e29..80bc50445 100644 --- a/freqtrade/analyze.py +++ b/freqtrade/analyze.py @@ -310,6 +310,6 @@ def get_signal(pair: str) -> (bool, bool): if signal_date < arrow.now() - timedelta(minutes=10): return (False, False) - (buy, sell) = latest[SignalType.BUY] == 1, latest[SignalType.SELL] == 1 - logger.debug('%trigger: %s (pair=%s, buy=%s sell=%s)', latest['date'], pair, buy, sell) + (buy, sell) = latest[SignalType.BUY.value] == 1, latest[SignalType.SELL.value] == 1 + logger.debug('trigger: %s (pair=%s) buy=%s sell=%s', latest['date'], pair, str(buy), str(sell)) return (buy, sell) diff --git a/freqtrade/tests/test_analyze.py b/freqtrade/tests/test_analyze.py index d1afc4200..c913606fd 100644 --- a/freqtrade/tests/test_analyze.py +++ b/freqtrade/tests/test_analyze.py @@ -40,30 +40,30 @@ def test_returns_latest_buy_signal(mocker): mocker.patch('freqtrade.analyze.get_ticker_history', return_value=MagicMock()) mocker.patch( 'freqtrade.analyze.analyze_ticker', - return_value=DataFrame([{'buy': 1, 'date': arrow.utcnow()}]) + return_value=DataFrame([{'buy': 1, 'sell': 0, 'date': arrow.utcnow()}]) ) - assert get_signal('BTC-ETH', SignalType.BUY) + assert get_signal('BTC-ETH') == (True, False) mocker.patch( 'freqtrade.analyze.analyze_ticker', - return_value=DataFrame([{'buy': 0, 'date': arrow.utcnow()}]) + return_value=DataFrame([{'buy': 0, 'sell':1, 'date': arrow.utcnow()}]) ) - assert not get_signal('BTC-ETH', SignalType.BUY) + assert get_signal('BTC-ETH') == (False, True) def test_returns_latest_sell_signal(mocker): mocker.patch('freqtrade.analyze.get_ticker_history', return_value=MagicMock()) mocker.patch( 'freqtrade.analyze.analyze_ticker', - return_value=DataFrame([{'sell': 1, 'date': arrow.utcnow()}]) + return_value=DataFrame([{'sell': 1, 'buy': 0, 'date': arrow.utcnow()}]) ) - assert get_signal('BTC-ETH', SignalType.SELL) + assert get_signal('BTC-ETH') == (False, True) mocker.patch( 'freqtrade.analyze.analyze_ticker', - return_value=DataFrame([{'sell': 0, 'date': arrow.utcnow()}]) + return_value=DataFrame([{'sell': 0, 'buy': 1, 'date': arrow.utcnow()}]) ) - assert not get_signal('BTC-ETH', SignalType.SELL) + assert get_signal('BTC-ETH') == (True,False) def test_get_signal_handles_exceptions(mocker): @@ -71,4 +71,4 @@ def test_get_signal_handles_exceptions(mocker): mocker.patch('freqtrade.analyze.analyze_ticker', side_effect=Exception('invalid ticker history ')) - assert not get_signal('BTC-ETH', SignalType.BUY) + assert get_signal('BTC-ETH') == (False,False) diff --git a/freqtrade/tests/test_main.py b/freqtrade/tests/test_main.py index c99727135..103ebb70b 100644 --- a/freqtrade/tests/test_main.py +++ b/freqtrade/tests/test_main.py @@ -267,6 +267,52 @@ def test_handle_trade(default_conf, limit_buy_order, limit_sell_order, mocker): assert trade.close_date is not None +def test_handle_overlpapping_signals(default_conf, ticker, mocker, caplog): + default_conf.update({'experimental': {'use_sell_signal': True}}) + mocker.patch.dict('freqtrade.main._CONF', default_conf) + + mocker.patch('freqtrade.main.get_signal', side_effect=lambda s: (True, True)) + mocker.patch.multiple('freqtrade.rpc', init=MagicMock(), send_msg=MagicMock()) + mocker.patch.multiple('freqtrade.main.exchange', + validate_pairs=MagicMock(), + get_ticker=ticker, + buy=MagicMock(return_value='mocked_limit_buy')) + mocker.patch('freqtrade.main.min_roi_reached', return_value=False) + + init(default_conf, create_engine('sqlite://')) + create_trade(0.001) + + # Buy and Sell triggering, so doing nothing ... + trades = Trade.query.all() + assert len(trades) == 0 + + # Buy is triggering, so buying ... + mocker.patch('freqtrade.main.get_signal', side_effect=lambda s: (True, False)) + create_trade(0.001) + trades = Trade.query.all() + assert len(trades) == 1 + assert trades[0].is_open is True + + # Buy and Sell are not triggering, so doing nothing ... + mocker.patch('freqtrade.main.get_signal', side_effect=lambda s: (False, False)) + assert handle_trade(trades[0]) is False + trades = Trade.query.all() + assert len(trades) == 1 + assert trades[0].is_open is True + + # Buy and Sell are triggering, so doing nothing ... + mocker.patch('freqtrade.main.get_signal', side_effect=lambda s: (True, True)) + assert handle_trade(trades[0]) is False + trades = Trade.query.all() + assert len(trades) == 1 + assert trades[0].is_open is True + + # Sell is triggering, guess what : we are Selling! + mocker.patch('freqtrade.main.get_signal', side_effect=lambda s: (False, True)) + trades = Trade.query.all() + assert handle_trade(trades[0]) is True + + def test_handle_trade_roi(default_conf, ticker, mocker, caplog): default_conf.update({'experimental': {'use_sell_signal': True}}) mocker.patch.dict('freqtrade.main._CONF', default_conf) From 572303963733d4a6d63aef25b587791f0ab9a03c Mon Sep 17 00:00:00 2001 From: toto Date: Tue, 16 Jan 2018 21:21:43 +0100 Subject: [PATCH 37/44] fXXXXXXk8 --- freqtrade/tests/test_analyze.py | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/freqtrade/tests/test_analyze.py b/freqtrade/tests/test_analyze.py index c913606fd..b8cd2f6e3 100644 --- a/freqtrade/tests/test_analyze.py +++ b/freqtrade/tests/test_analyze.py @@ -6,7 +6,7 @@ import arrow import pytest from pandas import DataFrame -from freqtrade.analyze import (SignalType, get_signal, parse_ticker_dataframe, +from freqtrade.analyze import (get_signal, parse_ticker_dataframe, populate_buy_trend, populate_indicators, populate_sell_trend) @@ -46,7 +46,7 @@ def test_returns_latest_buy_signal(mocker): mocker.patch( 'freqtrade.analyze.analyze_ticker', - return_value=DataFrame([{'buy': 0, 'sell':1, 'date': arrow.utcnow()}]) + return_value=DataFrame([{'buy': 0, 'sell': 1, 'date': arrow.utcnow()}]) ) assert get_signal('BTC-ETH') == (False, True) @@ -63,7 +63,7 @@ def test_returns_latest_sell_signal(mocker): 'freqtrade.analyze.analyze_ticker', return_value=DataFrame([{'sell': 0, 'buy': 1, 'date': arrow.utcnow()}]) ) - assert get_signal('BTC-ETH') == (True,False) + assert get_signal('BTC-ETH') == (True, False) def test_get_signal_handles_exceptions(mocker): @@ -71,4 +71,4 @@ def test_get_signal_handles_exceptions(mocker): mocker.patch('freqtrade.analyze.analyze_ticker', side_effect=Exception('invalid ticker history ')) - assert get_signal('BTC-ETH') == (False,False) + assert get_signal('BTC-ETH') == (False, False) From 04be438b352232b21377fe3c244fd66b0f2b7999 Mon Sep 17 00:00:00 2001 From: Stephen Dade Date: Wed, 17 Jan 2018 19:51:27 +1100 Subject: [PATCH 38/44] Better exception handling for check_handle_timedout --- freqtrade/main.py | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/freqtrade/main.py b/freqtrade/main.py index 796fa8515..01f3d6e16 100755 --- a/freqtrade/main.py +++ b/freqtrade/main.py @@ -131,7 +131,8 @@ def check_handle_timedout(timeoutvalue: int) -> None: for trade in Trade.query.filter(Trade.open_order_id.isnot(None)).all(): try: order = exchange.get_order(trade.open_order_id) - except OperationalException: + except requests.exceptions.RequestException: + logger.info('Cannot query order for %s due to %s', trade, traceback.format_exc()) continue ordertime = arrow.get(order['opened']) From 0d709847ee5babd35439e812a46c8aa093714288 Mon Sep 17 00:00:00 2001 From: Jean-Baptiste LE STANG Date: Wed, 17 Jan 2018 11:31:26 +0100 Subject: [PATCH 39/44] Fixing the doc and and the default value of sell_profit_only to False --- docs/configuration.md | 1 + freqtrade/main.py | 2 +- 2 files changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/configuration.md b/docs/configuration.md index 384775765..7aade1e59 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -29,6 +29,7 @@ The table below will list all configuration parameters. | `exchange.pair_whitelist` | [] | No | List of currency to use by the bot. Can be overrided with `--dynamic-whitelist` param. | `exchange.pair_blacklist` | [] | No | List of currency the bot must avoid. Useful when using `--dynamic-whitelist` param. | `experimental.use_sell_signal` | false | No | Use your sell strategy in addition of the `minimal_roi`. +| `experimental.sell_profit_only` | false | No | waits until you have made a positive profit before taking a sell decision. | `telegram.enabled` | true | Yes | Enable or not the usage of Telegram. | `telegram.token` | token | No | Your Telegram bot token. Only required is `enable` is `true`. | `telegram.chat_id` | chat_id | No | Your personal Telegram account id. Only required is `enable` is `true`. diff --git a/freqtrade/main.py b/freqtrade/main.py index 1640c7836..e692b7c6a 100755 --- a/freqtrade/main.py +++ b/freqtrade/main.py @@ -259,7 +259,7 @@ def handle_trade(trade: Trade) -> bool: return True # Experimental: Check if the trade is profitable before selling it (avoid selling at loss) - if _CONF.get('experimental', {}).get('sell_profit_only'): + if _CONF.get('experimental', {}).get('sell_profit_only', False): logger.debug('Checking if trade is profitable ...') if not buy and trade.calc_profit(rate=current_rate) <= 0: return False From 423b25146737abb1ae37aa1c6fab9499f64ec2ca Mon Sep 17 00:00:00 2001 From: kryofly Date: Wed, 17 Jan 2018 18:19:39 +0100 Subject: [PATCH 40/44] tests: speed up backtests --- freqtrade/tests/optimize/test_backtesting.py | 17 ++++++++++------- 1 file changed, 10 insertions(+), 7 deletions(-) diff --git a/freqtrade/tests/optimize/test_backtesting.py b/freqtrade/tests/optimize/test_backtesting.py index 5f899a48a..4ef4b1606 100644 --- a/freqtrade/tests/optimize/test_backtesting.py +++ b/freqtrade/tests/optimize/test_backtesting.py @@ -11,6 +11,13 @@ from freqtrade.optimize.backtesting import backtest, generate_text_table, get_ti import freqtrade.optimize.backtesting as backtesting +def trim_dictlist(dl, num): + new = {} + for pair, pair_data in dl.items(): + new[pair] = pair_data[num:] + return new + + def test_generate_text_table(): results = pd.DataFrame( { @@ -43,6 +50,7 @@ def test_backtest(default_conf, mocker): exchange._API = Bittrex({'key': '', 'secret': ''}) data = optimize.load_data(None, ticker_interval=5, pairs=['BTC_ETH']) + data = trim_dictlist(data, -200) results = backtest(default_conf['stake_amount'], optimize.preprocess(data), 10, True) assert not results.empty @@ -54,18 +62,12 @@ def test_backtest_1min_ticker_interval(default_conf, mocker): # Run a backtesting for an exiting 5min ticker_interval data = optimize.load_data(None, ticker_interval=1, pairs=['BTC_UNITEST']) + data = trim_dictlist(data, -200) results = backtest(default_conf['stake_amount'], optimize.preprocess(data), 1, True) assert not results.empty -def trim_dictlist(dl, num): - new = {} - for pair, pair_data in dl.items(): - new[pair] = pair_data[num:] - return new - - def load_data_test(what): data = optimize.load_data(None, ticker_interval=1, pairs=['BTC_UNITEST']) data = trim_dictlist(data, -100) @@ -125,6 +127,7 @@ def simple_backtest(config, contour, num_results): def test_backtest2(default_conf, mocker): mocker.patch.dict('freqtrade.main._CONF', default_conf) data = optimize.load_data(None, ticker_interval=5, pairs=['BTC_ETH']) + data = trim_dictlist(data, -200) results = backtest(default_conf['stake_amount'], optimize.preprocess(data), 10, True) assert not results.empty From fb34fe8c9a3c174d13916b367cd1c6bd83f68d9f Mon Sep 17 00:00:00 2001 From: pyup-bot Date: Wed, 17 Jan 2018 23:08:30 +0100 Subject: [PATCH 41/44] Update ta-lib from 0.4.15 to 0.4.16 --- requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/requirements.txt b/requirements.txt index d650efea8..cb24b327a 100644 --- a/requirements.txt +++ b/requirements.txt @@ -11,7 +11,7 @@ scikit-learn==0.19.1 scipy==1.0.0 jsonschema==2.6.0 numpy==1.14.0 -TA-Lib==0.4.15 +TA-Lib==0.4.16 pytest==3.3.2 pytest-mock==1.6.3 pytest-cov==2.5.1 From 9a48e3b8670448384affa816205a7d882e2c469d Mon Sep 17 00:00:00 2001 From: pyup-bot Date: Fri, 19 Jan 2018 01:33:33 +0100 Subject: [PATCH 42/44] Update arrow from 0.12.0 to 0.12.1 --- requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/requirements.txt b/requirements.txt index cb24b327a..d37312268 100644 --- a/requirements.txt +++ b/requirements.txt @@ -1,7 +1,7 @@ python-bittrex==0.2.2 SQLAlchemy==1.2.1 python-telegram-bot==9.0.0 -arrow==0.12.0 +arrow==0.12.1 cachetools==2.0.1 requests==2.18.4 urllib3==1.22 From 861e065d083b0f2134941bcfc33545883f44d477 Mon Sep 17 00:00:00 2001 From: Gerald Lonlas Date: Thu, 18 Jan 2018 21:07:55 -0800 Subject: [PATCH 43/44] Fix markdown mistakes in backtesting doc --- docs/backtesting.md | 46 +++++++++++++++++++++++++-------------------- 1 file changed, 26 insertions(+), 20 deletions(-) diff --git a/docs/backtesting.md b/docs/backtesting.md index 9829a10c8..7fc0366e6 100644 --- a/docs/backtesting.md +++ b/docs/backtesting.md @@ -14,7 +14,7 @@ real data. This is what we call Backtesting will use the crypto-currencies (pair) from your config file and load static tickers located in -[/freqtrade/tests/testdata](https://github.com/gcarq/freqtrade/tree/develop/freqtrade/tests/testdata). +[/freqtrade/tests/testdata](https://github.com/gcarq/freqtrade/tree/develop/freqtrade/tests/testdata). If the 5 min and 1 min ticker for the crypto-currencies to test is not already in the `testdata` folder, backtesting will download them automatically. Testdata files will not be updated until you specify it. @@ -51,38 +51,44 @@ python3 ./freqtrade/main.py backtesting --realistic-simulation --live python3 ./freqtrade/main.py backtesting --datadir freqtrade/tests/testdata-20180101 ``` -**Running backtest with smaller testset** -Use the --timerange argument to change how much of the testset +**Running backtest with smaller testset** +Use the `--timerange` argument to change how much of the testset you want to use. The last N ticks/timeframes will be used. -Example: +Example: ```bash python3 ./freqtrade/main.py backtesting --timerange=-200 ``` -***Advanced use of timerange*** - Doing --timerange=-200 will get the last 200 timeframes - from your inputdata. You can also specify specific dates, - or a range span indexed by start and stop. - The full timerange specification: - Not implemented yet! --timerange=-20180131 - Not implemented yet! --timerange=20180101- - Not implemented yet! --timerange=20180101-20181231 - Last 123 tickframes of data: --timerange=-123 - First 123 tickframes of data: --timerange=123- - Tickframes from line 123 through 456: --timerange=123-456 +***Advanced use of timerange*** +Doing `--timerange=-200` will get the last 200 timeframes +from your inputdata. You can also specify specific dates, +or a range span indexed by start and stop. + +The full timerange specification: +- Use last 123 tickframes of data: `--timerange=-123` +- Use first 123 tickframes of data: `--timerange=123-` +- Use tickframes from line 123 through 456: `--timerange=123-456` -**Update testdata directory +Incoming feature, not implemented yet: +- `--timerange=-20180131` +- `--timerange=20180101-` +- `--timerange=20180101-20181231` + + +**Update testdata directory** To update your testdata directory, or download into another testdata directory: ```bash -mkdir freqtrade/tests/testdata-20180113 -cp freqtrade/tests/testdata/pairs.json freqtrade/tests/testdata-20180113 -cd freqtrade/tests/testdata-20180113 +mkdir -p user_data/data/testdata-20180113 +cp freqtrade/tests/testdata/pairs.json user_data/data-20180113 +cd user_data/data-20180113 +``` Possibly edit pairs.json file to include/exclude pairs -python download_backtest_data.py -p pairs.json +```bash +python freqtrade/tests/testdata/download_backtest_data.py -p pairs.json ``` The script will read your pairs.json file, and download ticker data From ddc1b7cd495e0cbee8d2725be9b62002fe1466fd Mon Sep 17 00:00:00 2001 From: Gerald Lonlas Date: Thu, 18 Jan 2018 21:15:20 -0800 Subject: [PATCH 44/44] Update bot commands in README.md --- README.md | 15 ++++++++------- 1 file changed, 8 insertions(+), 7 deletions(-) diff --git a/README.md b/README.md index 045d6b624..d9e537382 100644 --- a/README.md +++ b/README.md @@ -136,8 +136,8 @@ to understand the requirements before sending your pull-requests. ### Bot commands ```bash -usage: main.py [-h] [-c PATH] [-v] [--version] [--dynamic-whitelist [INT]] - [--dry-run-db] +usage: main.py [-h] [-v] [--version] [-c PATH] [--dry-run-db] [--datadir PATH] + [--dynamic-whitelist [INT]] {backtesting,hyperopt} ... Simple High Frequency Trading Bot for crypto currencies @@ -149,16 +149,17 @@ positional arguments: optional arguments: -h, --help show this help message and exit - -c PATH, --config PATH - specify configuration file (default: config.json) -v, --verbose be verbose --version show program's version number and exit - --dynamic-whitelist [INT] - dynamically generate and update whitelist based on 24h - BaseVolume (Default 20 currencies) + -c PATH, --config PATH + specify configuration file (default: config.json) --dry-run-db Force dry run to use a local DB "tradesv3.dry_run.sqlite" instead of memory DB. Work only if dry_run is enabled. + --datadir PATH path to backtest data (default freqdata/tests/testdata + --dynamic-whitelist [INT] + dynamically generate and update whitelist based on 24h + BaseVolume (Default 20 currencies) ``` More details on: - [How to run the bot](https://github.com/gcarq/freqtrade/blob/develop/docs/bot-usage.md#bot-commands)