diff --git a/README.md b/README.md index 045d6b624..d9e537382 100644 --- a/README.md +++ b/README.md @@ -136,8 +136,8 @@ to understand the requirements before sending your pull-requests. ### Bot commands ```bash -usage: main.py [-h] [-c PATH] [-v] [--version] [--dynamic-whitelist [INT]] - [--dry-run-db] +usage: main.py [-h] [-v] [--version] [-c PATH] [--dry-run-db] [--datadir PATH] + [--dynamic-whitelist [INT]] {backtesting,hyperopt} ... Simple High Frequency Trading Bot for crypto currencies @@ -149,16 +149,17 @@ positional arguments: optional arguments: -h, --help show this help message and exit - -c PATH, --config PATH - specify configuration file (default: config.json) -v, --verbose be verbose --version show program's version number and exit - --dynamic-whitelist [INT] - dynamically generate and update whitelist based on 24h - BaseVolume (Default 20 currencies) + -c PATH, --config PATH + specify configuration file (default: config.json) --dry-run-db Force dry run to use a local DB "tradesv3.dry_run.sqlite" instead of memory DB. Work only if dry_run is enabled. + --datadir PATH path to backtest data (default freqdata/tests/testdata + --dynamic-whitelist [INT] + dynamically generate and update whitelist based on 24h + BaseVolume (Default 20 currencies) ``` More details on: - [How to run the bot](https://github.com/gcarq/freqtrade/blob/develop/docs/bot-usage.md#bot-commands) diff --git a/docs/backtesting.md b/docs/backtesting.md index c426e2b5c..7fc0366e6 100644 --- a/docs/backtesting.md +++ b/docs/backtesting.md @@ -14,7 +14,7 @@ real data. This is what we call Backtesting will use the crypto-currencies (pair) from your config file and load static tickers located in -[/freqtrade/tests/testdata](https://github.com/gcarq/freqtrade/tree/develop/freqtrade/tests/testdata). +[/freqtrade/tests/testdata](https://github.com/gcarq/freqtrade/tree/develop/freqtrade/tests/testdata). If the 5 min and 1 min ticker for the crypto-currencies to test is not already in the `testdata` folder, backtesting will download them automatically. Testdata files will not be updated until you specify it. @@ -51,6 +51,50 @@ python3 ./freqtrade/main.py backtesting --realistic-simulation --live python3 ./freqtrade/main.py backtesting --datadir freqtrade/tests/testdata-20180101 ``` +**Running backtest with smaller testset** +Use the `--timerange` argument to change how much of the testset +you want to use. The last N ticks/timeframes will be used. + +Example: +```bash +python3 ./freqtrade/main.py backtesting --timerange=-200 +``` + +***Advanced use of timerange*** +Doing `--timerange=-200` will get the last 200 timeframes +from your inputdata. You can also specify specific dates, +or a range span indexed by start and stop. + +The full timerange specification: +- Use last 123 tickframes of data: `--timerange=-123` +- Use first 123 tickframes of data: `--timerange=123-` +- Use tickframes from line 123 through 456: `--timerange=123-456` + + +Incoming feature, not implemented yet: +- `--timerange=-20180131` +- `--timerange=20180101-` +- `--timerange=20180101-20181231` + + +**Update testdata directory** +To update your testdata directory, or download into another testdata directory: +```bash +mkdir -p user_data/data/testdata-20180113 +cp freqtrade/tests/testdata/pairs.json user_data/data-20180113 +cd user_data/data-20180113 +``` + +Possibly edit pairs.json file to include/exclude pairs + +```bash +python freqtrade/tests/testdata/download_backtest_data.py -p pairs.json +``` + +The script will read your pairs.json file, and download ticker data +into the current working directory. + + For help about backtesting usage, please refer to [Backtesting commands](#backtesting-commands). diff --git a/docs/configuration.md b/docs/configuration.md index 384775765..7aade1e59 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -29,6 +29,7 @@ The table below will list all configuration parameters. | `exchange.pair_whitelist` | [] | No | List of currency to use by the bot. Can be overrided with `--dynamic-whitelist` param. | `exchange.pair_blacklist` | [] | No | List of currency the bot must avoid. Useful when using `--dynamic-whitelist` param. | `experimental.use_sell_signal` | false | No | Use your sell strategy in addition of the `minimal_roi`. +| `experimental.sell_profit_only` | false | No | waits until you have made a positive profit before taking a sell decision. | `telegram.enabled` | true | Yes | Enable or not the usage of Telegram. | `telegram.token` | token | No | Your Telegram bot token. Only required is `enable` is `true`. | `telegram.chat_id` | chat_id | No | Your personal Telegram account id. Only required is `enable` is `true`. diff --git a/docs/faq.md b/docs/faq.md index 58929e15c..b3f15613a 100644 --- a/docs/faq.md +++ b/docs/faq.md @@ -2,20 +2,70 @@ #### I have waited 5 minutes, why hasn't the bot made any trades yet?! -Depending on the buy strategy, the amount of whitelisted coins, the situation of the market etc, it can take up to hours to find good entry position for a trade. Be patient! +Depending on the buy strategy, the amount of whitelisted coins, the +situation of the market etc, it can take up to hours to find good entry +position for a trade. Be patient! #### I have made 12 trades already, why is my total profit negative?! -I understand your disappointment but unfortunately 12 trades is just not enough to say anything. If you run backtesting, you can see that our current algorithm does leave you on the plus side, but that is after thousands of trades and even there, you will be left with losses on specific coins that you have traded tens if not hundreds of times. We of course constantly aim to improve the bot but it will _always_ be a gamble, which should leave you with modest wins on monthly basis but you can't say much from few trades. +I understand your disappointment but unfortunately 12 trades is just +not enough to say anything. If you run backtesting, you can see that our +current algorithm does leave you on the plus side, but that is after +thousands of trades and even there, you will be left with losses on +specific coins that you have traded tens if not hundreds of times. We +of course constantly aim to improve the bot but it will _always_ be a +gamble, which should leave you with modest wins on monthly basis but +you can't say much from few trades. -#### I’d like to change the stake amount. Can I just stop the bot with /stop and then change the config.json and run it again? +#### I’d like to change the stake amount. Can I just stop the bot with +/stop and then change the config.json and run it again? -Not quite. Trades are persisted to a database but the configuration is currently only read when the bot is killed and restarted. `/stop` more like pauses. You can stop your bot, adjust settings and start it again. +Not quite. Trades are persisted to a database but the configuration is +currently only read when the bot is killed and restarted. `/stop` more +like pauses. You can stop your bot, adjust settings and start it again. #### I want to improve the bot with a new strategy -That's great. We have a nice backtesting and hyperoptimizing setup. See the tutorial [[here|Testing-new-strategies-with-Hyperopt]]. +That's great. We have a nice backtesting and hyperoptimizing setup. See +the tutorial [here|Testing-new-strategies-with-Hyperopt](https://github.com/gcarq/freqtrade/blob/develop/docs/bot-usage.md#hyperopt-commands). -#### Is there a setting to only SELL the coins being held and not perform anymore BUYS? +#### Is there a setting to only SELL the coins being held and not +perform anymore BUYS? + +You can use the `/forcesell all` command from Telegram. + +### How many epoch do I need to get a good Hyperopt result? +Per default Hyperopts without `-e` or `--epochs` parameter will only +run 100 epochs, means 100 evals of your triggers, guards, .... Too few +to find a great result (unless if you are very lucky), so you probably +have to run it for 10.000 or more. But it will take an eternity to +compute. + +We recommend you to run it at least 10.000 epochs: +```bash +python3 ./freqtrade/main.py hyperopt -e 10000 +``` + +or if you want intermediate result to see +```bash +for i in {1..100}; do python3 ./freqtrade/main.py hyperopt -e 100; done +``` + +#### Why it is so long to run hyperopt? +Finding a great Hyperopt results takes time. + +If you wonder why it takes a while to find great hyperopt results + +This answer was written during the under the release 0.15.1, when we had +: +- 8 triggers +- 9 guards: let's say we evaluate even 10 values from each +- 1 stoploss calculation: let's say we want 10 values from that too to +be evaluated + +The following calculation is still very rough and not very precise +but it will give the idea. With only these triggers and guards there is +already 8*10^9*10 evaluations. A roughly total of 80 billion evals. +Did you run 100 000 evals? Congrats, you've done roughly 1 / 100 000 th +of the search space. -You can use the `/forcesell all` command from Telegram. \ No newline at end of file diff --git a/docs/hyperopt.md b/docs/hyperopt.md index 24a9dbc51..af564f0b6 100644 --- a/docs/hyperopt.md +++ b/docs/hyperopt.md @@ -168,6 +168,16 @@ If you would like to learn parameters using an alternate ticke-data that you have on-disk, use the --datadir PATH option. Default hyperopt will use data from directory freqtrade/tests/testdata. +### Running hyperopt with smaller testset + +Use the --timeperiod argument to change how much of the testset +you want to use. The last N ticks/timeframes will be used. +Example: + +```bash +python3 ./freqtrade/main.py hyperopt --timeperiod -200 +``` + ### Hyperopt with MongoDB Hyperopt with MongoDB, is like Hyperopt under steroids. As you saw by executing the previous command is the execution takes a long time. diff --git a/docs/installation.md b/docs/installation.md index 30431345b..c48f170c6 100644 --- a/docs/installation.md +++ b/docs/installation.md @@ -1,98 +1,119 @@ -# Install the bot +# Installation + This page explains how to prepare your environment for running the bot. -To understand how to set up the bot please read the Bot -[Bot configuration](https://github.com/gcarq/freqtrade/blob/develop/docs/configuration.md) -page. + +To understand how to set up the bot please read the [Bot Configuration](https://github.com/gcarq/freqtrade/blob/develop/docs/configuration.md) page. ## Table of Contents -- [Docker Automatic Installation](#docker) -- [Linux or Mac manual Installation](#linux--mac) - - [Linux - Ubuntu 16.04](#21-linux---ubuntu-1604) - - [Linux - Other distro](#22-linux---other-distro) - - [MacOS installation](#23-macos-installation) - - [Advanced Linux ](#advanced-linux) -- [Windows manual Installation](#windows) -# Docker +* [Table of Contents](#table-of-contents) +* [Automatic Installation - Docker](#automatic-installation-docker) +* [Custom Installation](#custom-installation) + - [Requirements](#requirements) + - [Linux - Ubuntu 16.04](#linux-ubuntu-1604) + - [MacOS](#macos) + - [Windows](#windows) +* [First Steps](#first-step) + + +------ + +## Automatic Installation - Docker -## Easy installation Start by downloading Docker for your platform: -- [Mac](https://www.docker.com/products/docker#/mac) -- [Windows](https://www.docker.com/products/docker#/windows) -- [Linux](https://www.docker.com/products/docker#/linux) -Once you have Docker installed, simply create the config file -(e.g. `config.json`) and then create a Docker image for `freqtrade` -using the Dockerfile in this repo. +* [Mac](https://www.docker.com/products/docker#/mac) +* [Windows](https://www.docker.com/products/docker#/windows) +* [Linux](https://www.docker.com/products/docker#/linux) + +Once you have Docker installed, simply create the config file (e.g. `config.json`) and then create a Docker image for `freqtrade` using the Dockerfile in this repo. + + +### 1. Prepare the Bot + +#### 1.1. Clone the git repository -### 1. Prepare the bot -1. Clone the git ```bash git clone https://github.com/gcarq/freqtrade.git ``` -2. (Optional) Checkout the develop branch + +#### 1.2. (Optional) Checkout the develop branch + ```bash git checkout develop ``` -3. Go into the new directory + +#### 1.3. Go into the new directory + ```bash cd freqtrade ``` -4. Copy `config.sample` to `config.json` -```bash -cp config.json.example config.json -``` -To edit the config please refer to the [Bot Configuration](https://github.com/gcarq/freqtrade/blob/develop/docs/configuration.md) page -5. Create your DB file (Optional, the bot will create it if it is missing) -```bash -# For Production -touch tradesv3.sqlite -# For Dry-run +#### 1.4. Copy `config.json.example` to `config.json` + +```bash +cp -n config.json.example config.json +``` + +> To edit the config please refer to the [Bot Configuration](https://github.com/gcarq/freqtrade/blob/develop/docs/configuration.md) page. + +#### 1.5. Create your database file *(optional - the bot will create it if it is missing)* + +Production +```bash +touch tradesv3.sqlite +```` + +Dry-Run +```bash touch tradesv3.dryrun.sqlite ``` -### 2. Build the docker image + +### 2. Build the Docker image + ```bash cd freqtrade docker build -t freqtrade . ``` -For security reasons, your configuration file will not be included in the -image, you will need to bind mount it. It is also advised to bind mount -a sqlite database file (see the "5. Run a restartable docker image" -section) to keep it between updates. +For security reasons, your configuration file will not be included in the image, you will need to bind mount it. It is also advised to bind mount an SQLite database file (see the "5. Run a restartable docker image" section) to keep it between updates. -### 3. Verify the docker image -After build process you can verify that the image was created with: -``` + +### 3. Verify the Docker image + +After the build process you can verify that the image was created with: + +```bash docker images ``` -### 4. Run the docker image -You can run a one-off container that is immediately deleted upon exiting with -the following command (config.json must be in the current working directory): -``` +### 4. Run the Docker image + +You can run a one-off container that is immediately deleted upon exiting with the following command (`config.json` must be in the current working directory): + +```bash docker run --rm -v `pwd`/config.json:/freqtrade/config.json -it freqtrade ``` -In this example, the database will be created inside the docker instance -and will be lost when you will refresh your image. +In this example, the database will be created inside the docker instance and will be lost when you will refresh your image. + ### 5. Run a restartable docker image -To run a restartable instance in the background (feel free to place your -configuration and database files wherever it feels comfortable on your -filesystem). -**5.1. Move your config file and database** +To run a restartable instance in the background (feel free to place your configuration and database files wherever it feels comfortable on your filesystem). + +#### 5.1. Move your config file and database + ```bash mkdir ~/.freqtrade mv config.json ~/.freqtrade mv tradesv3.sqlite ~/.freqtrade ``` -**5.2. Run the docker image** +#### 5.2. Run the docker image + ```bash docker run -d \ --name freqtrade \ @@ -100,12 +121,11 @@ docker run -d \ -v ~/.freqtrade/tradesv3.sqlite:/freqtrade/tradesv3.sqlite \ freqtrade ``` -If you are using `dry_run=True` it's not necessary to mount -`tradesv3.sqlite`, but you can mount `tradesv3.dryrun.sqlite` if you -plan to use the dry run mode with the param `--dry-run-db`. +If you are using `dry_run=True` it's not necessary to mount `tradesv3.sqlite`, but you can mount `tradesv3.dryrun.sqlite` if you plan to use the dry run mode with the param `--dry-run-db`. ### 6. Monitor your Docker instance + You can then use the following commands to monitor and manage your container: ```bash @@ -116,35 +136,39 @@ docker stop freqtrade docker start freqtrade ``` -You do not need to rebuild the image for configuration changes, it will -suffice to edit `config.json` and restart the container. +You do not need to rebuild the image for configuration changes, it will suffice to edit `config.json` and restart the container. +------ + +## Custom Installation + +We've included/collected install instructions for Ubuntu 16.04, MacOS, and Windows. These are guidelines and your success may vary with other distros. + +### Requirements -# Linux / MacOS -## 1. Requirements Click each one for install guide: -- [Python 3.6.x](http://docs.python-guide.org/en/latest/starting/installation/), -note the bot was not tested on Python >= 3.7.x -- [pip](https://pip.pypa.io/en/stable/installing/) -- [git](https://git-scm.com/book/en/v2/Getting-Started-Installing-Git) -- [virtualenv](https://virtualenv.pypa.io/en/stable/installation/) (Recommended) -- [TA-Lib](https://mrjbq7.github.io/ta-lib/install.html) +* [Python 3.6.x](http://docs.python-guide.org/en/latest/starting/installation/), note the bot was not tested on Python >= 3.7.x +* [pip](https://pip.pypa.io/en/stable/installing/) +* [git](https://git-scm.com/book/en/v2/Getting-Started-Installing-Git) +* [virtualenv](https://virtualenv.pypa.io/en/stable/installation/) (Recommended) +* [TA-Lib](https://mrjbq7.github.io/ta-lib/install.html) -## 2. First install required packages -This bot require Python 3.6 and TA-LIB -### 2.1 Linux - Ubuntu 16.04 +### Linux - Ubuntu 16.04 + +#### 1. Install Python 3.6, Git, and wget -**2.1.1. Install Python 3.6, Git, and wget** ```bash sudo add-apt-repository ppa:jonathonf/python-3.6 sudo apt-get update sudo apt-get install python3.6 python3.6-venv python3.6-dev build-essential autoconf libtool pkg-config make wget git ``` -**2.1.2. Install TA-LIB** +#### 2. Install TA-Lib + Official webpage: https://mrjbq7.github.io/ta-lib/install.html -``` + +```bash wget http://prdownloads.sourceforge.net/ta-lib/ta-lib-0.4.0-src.tar.gz tar xvzf ta-lib-0.4.0-src.tar.gz cd ta-lib @@ -155,29 +179,58 @@ cd .. rm -rf ./ta-lib* ``` -**2.1.3. [Optional] Install MongoDB** +#### 3. [Optional] Install MongoDB + Install MongoDB if you plan to optimize your strategy with Hyperopt. ```bash sudo apt-get install mongodb-org ``` -Complete tutorial on [Digital Ocean: How to Install MongoDB on Ubuntu 16.04](https://www.digitalocean.com/community/tutorials/how-to-install-mongodb-on-ubuntu-16-04) -### 2.2. Linux - Other distro -If you are on a different Linux OS you maybe have to adapt things like: +> Complete tutorial from Digital Ocean: [How to Install MongoDB on Ubuntu 16.04](https://www.digitalocean.com/community/tutorials/how-to-install-mongodb-on-ubuntu-16-04). -- package manager (for example yum instead of apt-get) -- package names +#### 4. Install FreqTrade -### 2.3. MacOS installation +Clone the git repository: -**2.3.1. Install Python 3.6, git and wget** ```bash -brew install python3 git wget +git clone https://github.com/gcarq/freqtrade.git ``` -**2.3.2. [Optional] Install MongoDB** +Optionally checkout the develop branch: + +```bash +git checkout develop +``` + +#### 5. Configure `freqtrade` as a `systemd` service + +From the freqtrade repo... copy `freqtrade.service` to your systemd user directory (usually `~/.config/systemd/user`) and update `WorkingDirectory` and `ExecStart` to match your setup. + +After that you can start the daemon with: +```bash +systemctl --user start freqtrade +``` + +For this to be persistent (run when user is logged out) you'll need to enable `linger` for your freqtrade user. + +```bash +sudo loginctl enable-linger "$USER" +``` + + +### MacOS + +#### 1. Install Python 3.6, git, wget and ta-lib + +```bash +brew install python3 git wget ta-lib +``` + +#### 2. [Optional] Install MongoDB + Install MongoDB if you plan to optimize your strategy with Hyperopt. + ```bash curl -O https://fastdl.mongodb.org/osx/mongodb-osx-ssl-x86_64-3.4.10.tgz tar -zxvf mongodb-osx-ssl-x86_64-3.4.10.tgz @@ -186,49 +239,30 @@ cp -R -n mongodb-osx-x86_64-3.4.10/ /env/mongodb export PATH=/env/mongodb/bin:$PATH ``` -## 3. Clone the repo -The following steps are made for Linux/mac environment -1. Clone the git `git clone https://github.com/gcarq/freqtrade.git` -2. (Optional) Checkout the develop branch `git checkout develop` +#### 3. Install FreqTrade -## 4. Prepare the bot -```bash -cd freqtrade -cp config.json.example config.json -``` -To edit the config please refer to [Bot Configuration](https://github.com/gcarq/freqtrade/blob/develop/docs/configuration.md) - -## 5. Setup your virtual env -```bash -python3.6 -m venv .env -source .env/bin/activate -pip3.6 install -r requirements.txt -pip3.6 install -e . -``` - -## 6. Run the bot -If this is the first time you run the bot, ensure you are running it -in Dry-run `"dry_run": true,` otherwise it will start to buy and sell coins. +Clone the git repository: ```bash -python3.6 ./freqtrade/main.py -c config.json +git clone https://github.com/gcarq/freqtrade.git ``` -### Advanced Linux -**systemd service file** -Copy `./freqtrade.service` to your systemd user directory (usually `~/.config/systemd/user`) -and update `WorkingDirectory` and `ExecStart` to match your setup. -After that you can start the daemon with: +Optionally checkout the develop branch: + ```bash -systemctl --user start freqtrade +git checkout develop ``` -# Windows -We do recommend Windows users to use [Docker](#docker) this will work -much easier and smoother (also safer). +### Windows + +We recommend that Windows users use [Docker](#docker) as this will work +much easier and smoother (also more secure). + +#### 1. Install freqtrade + +copy paste `config.json` to ``\path\freqtrade-develop\freqtrade` ```cmd -#copy paste config.json to \path\freqtrade-develop\freqtrade >cd \path\freqtrade-develop >python -m venv .env >cd .env\Scripts @@ -239,8 +273,41 @@ much easier and smoother (also safer). >cd freqtrade >python main.py ``` -*Thanks [Owdr](https://github.com/Owdr) for the commands. Source: [Issue #222](https://github.com/gcarq/freqtrade/issues/222)* -## Next step -Now you have an environment ready, the next step is to -[configure your bot](https://github.com/gcarq/freqtrade/blob/develop/docs/configuration.md). +> Thanks [Owdr](https://github.com/Owdr) for the commands. Source: [Issue #222](https://github.com/gcarq/freqtrade/issues/222) + + +------ + + +## First Steps + +### 1. Initialize the configuration + +```bash +cd freqtrade +cp config.json.example config.json +``` + +> *To edit the config please refer to [Bot Configuration](https://github.com/gcarq/freqtrade/blob/develop/docs/configuration.md).* + + +### 2. Setup your Python virtual environment (virtualenv) + +```bash +python3.6 -m venv .env +source .env/bin/activate +pip3.6 install -r requirements.txt +pip3.6 install -e . +``` + +### 3. Run the Bot + +If this is the first time you run the bot, ensure you are running it in Dry-run `"dry_run": true,` otherwise it will start to buy and sell coins. + +```bash +python3.6 ./freqtrade/main.py -c config.json +``` + +Now you have an environment ready, the next step is +[Bot Configuration](https://github.com/gcarq/freqtrade/blob/develop/docs/configuration.md)... diff --git a/docs/sql_cheatsheet.md b/docs/sql_cheatsheet.md index 1edbcde5b..065f264f1 100644 --- a/docs/sql_cheatsheet.md +++ b/docs/sql_cheatsheet.md @@ -67,6 +67,18 @@ SET is_open=0, close_date='2017-12-20 03:08:45.103418', close_rate=0.19638016, c WHERE id=31; ``` +## Insert manually a new trade + +```sql +INSERT +INTO trades (exchange, pair, is_open, fee, open_rate, stake_amount, amount, open_date) +VALUES ('BITTREX', 'BTC_', 1, 0.0025, , , , '') +``` + +**Example:** +```sql +INSERT INTO trades (exchange, pair, is_open, fee, open_rate, stake_amount, amount, open_date) VALUES ('BITTREX', 'BTC_ETC', 1, 0.0025, 0.00258580, 0.002, 0.7715262081, '2017-11-28 12:44:24.000000') +``` ## Fix wrong fees in the table If your DB was created before diff --git a/freqtrade/__init__.py b/freqtrade/__init__.py index a190ca117..d1671e4c6 100644 --- a/freqtrade/__init__.py +++ b/freqtrade/__init__.py @@ -1,5 +1,5 @@ """ FreqTrade bot """ -__version__ = '0.14.3' +__version__ = '0.15.1' class DependencyException(BaseException): diff --git a/freqtrade/analyze.py b/freqtrade/analyze.py index 43a87134e..0c3c69c8c 100644 --- a/freqtrade/analyze.py +++ b/freqtrade/analyze.py @@ -74,6 +74,8 @@ def populate_indicators(dataframe: DataFrame) -> DataFrame: # Plus Directional Indicator / Movement dataframe['plus_dm'] = ta.PLUS_DM(dataframe) dataframe['plus_di'] = ta.PLUS_DI(dataframe) + dataframe['minus_di'] = ta.MINUS_DI(dataframe) + """ # ROC dataframe['roc'] = ta.ROC(dataframe) @@ -114,13 +116,14 @@ def populate_indicators(dataframe: DataFrame) -> DataFrame: dataframe['blower'] = ta.BBANDS(dataframe, nbdevup=2, nbdevdn=2)['lowerband'] """ # Bollinger bands + """ bollinger = qtpylib.bollinger_bands(qtpylib.typical_price(dataframe), window=20, stds=2) dataframe['bb_lowerband'] = bollinger['lower'] dataframe['bb_middleband'] = bollinger['mid'] dataframe['bb_upperband'] = bollinger['upper'] - """ # EMA - Exponential Moving Average + dataframe['ema3'] = ta.EMA(dataframe, timeperiod=3) dataframe['ema5'] = ta.EMA(dataframe, timeperiod=5) dataframe['ema10'] = ta.EMA(dataframe, timeperiod=10) dataframe['ema50'] = ta.EMA(dataframe, timeperiod=50) @@ -210,14 +213,12 @@ def populate_indicators(dataframe: DataFrame) -> DataFrame: # Chart type # ------------------------------------ - """ # Heikinashi stategy heikinashi = qtpylib.heikinashi(dataframe) dataframe['ha_open'] = heikinashi['open'] dataframe['ha_close'] = heikinashi['close'] dataframe['ha_high'] = heikinashi['high'] dataframe['ha_low'] = heikinashi['low'] - """ return dataframe @@ -280,29 +281,38 @@ def analyze_ticker(ticker_history: List[Dict]) -> DataFrame: return dataframe -def get_signal(pair: str, signal: SignalType) -> bool: +# FIX: 20180109, there could be some confusion because we will make a +# boolean result (execute the action or not depending on the signal). +# But the above checks can also return False, and we hide that. +# 20180119 Update to above fix, after an code update we now return +# a tuple (buy, sell). We could take advantage of this +# To distinguish an error from an non-signal situation (False, False) +# by just returning False. +# In short, if we return False it is error, If a tuple we +# get the signal situation. +def get_signal(pair: str) -> (bool, bool): """ Calculates current signal based several technical analysis indicators :param pair: pair in format BTC_ANT or BTC-ANT - :return: True if pair is good for buying, False otherwise + :return: (True, False) if pair is good for buying and not for selling """ ticker_hist = get_ticker_history(pair) if not ticker_hist: logger.warning('Empty ticker history for pair %s', pair) - return False + return (False, False) try: dataframe = analyze_ticker(ticker_hist) except ValueError as ex: logger.warning('Unable to analyze ticker for pair %s: %s', pair, str(ex)) - return False + return (False, False) except Exception as ex: logger.exception('Unexpected error when analyzing ticker for pair %s: %s', pair, str(ex)) - return False + return (False, False) if dataframe.empty: logger.warning('Empty dataframe for pair %s', pair) - return False + return (False, False) latest = dataframe.iloc[-1] @@ -310,11 +320,8 @@ def get_signal(pair: str, signal: SignalType) -> bool: signal_date = arrow.get(latest['date']) if signal_date < arrow.now() - timedelta(minutes=10): logger.warning('Too old dataframe for pair %s', pair) - return False + return (False, False) - # FIX: 20180109, there could be some confusion because we will make a - # boolean result (execute the action or not depending on the signal). - # But the above checks can also return False, and we hide that. - result = latest[signal.value] == 1 - logger.debug('%s_trigger: %s (pair=%s, signal=%s)', signal.value, latest['date'], pair, result) - return result + (buy, sell) = latest[SignalType.BUY.value] == 1, latest[SignalType.SELL.value] == 1 + logger.debug('trigger: %s (pair=%s) buy=%s sell=%s', latest['date'], pair, str(buy), str(sell)) + return (buy, sell) diff --git a/freqtrade/exchange/bittrex.py b/freqtrade/exchange/bittrex.py index a5fd68016..e6cacbd4e 100644 --- a/freqtrade/exchange/bittrex.py +++ b/freqtrade/exchange/bittrex.py @@ -123,10 +123,8 @@ class Bittrex(Exchange): message=data['message'], pair=pair)) - if not data.get('result') \ - or not data['result'].get('Bid') \ - or not data['result'].get('Ask') \ - or not data['result'].get('Last'): + if not data.get('result') or\ + not all(key in data.get('result', {}) for key in ['Bid', 'Ask', 'Last']): raise ContentDecodingError('{message} params=({pair})'.format( message='Got invalid response from bittrex', pair=pair)) diff --git a/freqtrade/main.py b/freqtrade/main.py index 918a0dcc9..fe20ed218 100755 --- a/freqtrade/main.py +++ b/freqtrade/main.py @@ -14,7 +14,7 @@ from cachetools import cached, TTLCache from freqtrade import (DependencyException, OperationalException, __version__, exchange, persistence, rpc) -from freqtrade.analyze import SignalType, get_signal +from freqtrade.analyze import get_signal from freqtrade.fiat_convert import CryptoToFiatConverter from freqtrade.misc import (State, get_state, load_config, parse_args, throttle, update_state) @@ -155,6 +155,8 @@ def handle_timedout_limit_buy(trade, order): # check_handle_timedout will flush afterwards Trade.session.flush() logger.info('Buy order timeout for %s.', trade) + rpc.send_msg('*Timeout:* Unfilled buy order for {} cancelled'.format( + trade.pair.replace('_', '/'))) return True else: # if trade is partially complete, edit the stake details for the trade @@ -163,6 +165,8 @@ def handle_timedout_limit_buy(trade, order): trade.stake_amount = trade.amount * trade.open_rate trade.open_order_id = None logger.info('Partial buy order timeout for %s.', trade) + rpc.send_msg('*Timeout:* Remaining buy order for {} cancelled'.format( + trade.pair.replace('_', '/'))) return False @@ -180,6 +184,8 @@ def handle_timedout_limit_sell(trade, order): trade.close_date = None trade.is_open = True trade.open_order_id = None + rpc.send_msg('*Timeout:* Unfilled sell order for {} cancelled'.format( + trade.pair.replace('_', '/'))) logger.info('Sell order timeout for %s.', trade) return True else: @@ -196,20 +202,21 @@ def check_handle_timedout(timeoutvalue: int) -> None: timeoutthreashold = arrow.utcnow().shift(minutes=-timeoutvalue).datetime for trade in Trade.query.filter(Trade.open_order_id.isnot(None)).all(): - order = exchange.get_order(trade.open_order_id) + try: + order = exchange.get_order(trade.open_order_id) + except requests.exceptions.RequestException: + logger.info('Cannot query order for %s due to %s', trade, traceback.format_exc()) + continue ordertime = arrow.get(order['opened']) + # Check if trade is still actually open + if int(order['remaining']) == 0: + continue + if order['type'] == "LIMIT_BUY" and ordertime < timeoutthreashold: handle_timedout_limit_buy(trade, order) - elif order['type'] == "LIMIT_SELL" and ordertime < timeoutthreashold: - if handle_timedout_limit_sell(trade, order): - # BUG? if there is more trades that are - # timed out, shouldn't we collect and - # then return all of them? - # Also the function signature is return None. - # But we return True here. - return True + handle_timedout_limit_sell(trade, order) def execute_sell(trade: Trade, limit: float) -> None: @@ -295,24 +302,28 @@ def handle_trade(trade: Trade) -> bool: logger.debug('Handling %s ...', trade) current_rate = exchange.get_ticker(trade.pair)['bid'] - # Check if minimal roi has been reached - if min_roi_reached(trade, current_rate, datetime.utcnow()): + (buy, sell) = (False, False) + + if _CONF.get('experimental', {}).get('use_sell_signal'): + (buy, sell) = get_signal(trade.pair) + + # Check if minimal roi has been reached and no longer in buy conditions (avoiding a fee) + if not buy and min_roi_reached(trade, current_rate, datetime.utcnow()): logger.debug('Executing sell due to ROI ...') execute_sell(trade, current_rate) return True + # Experimental: Check if the trade is profitable before selling it (avoid selling at loss) + if _CONF.get('experimental', {}).get('sell_profit_only', False): + logger.debug('Checking if trade is profitable ...') + if not buy and trade.calc_profit(rate=current_rate) <= 0: + return False + # Experimental: Check if sell signal has been enabled and triggered - if _CONF.get('experimental', {}).get('use_sell_signal'): - # Experimental: Check if the trade is profitable before selling it (avoid selling at loss) - if _CONF.get('experimental', {}).get('sell_profit_only'): - logger.debug('Checking if trade is profitable ...') - if trade.calc_profit(rate=current_rate) <= 0: - return False - logger.debug('Checking sell_signal ...') - if get_signal(trade.pair, SignalType.SELL): - logger.debug('Executing sell due to sell signal ...') - execute_sell(trade, current_rate) - return True + if sell and not buy: + logger.debug('Executing sell due to sell signal ...') + execute_sell(trade, current_rate) + return True return False @@ -353,7 +364,8 @@ def create_trade(stake_amount: float) -> bool: # Pick pair based on StochRSI buy signals for _pair in whitelist: - if get_signal(_pair, SignalType.BUY): + (buy, sell) = get_signal(_pair) + if buy and not sell: pair = _pair break else: diff --git a/freqtrade/misc.py b/freqtrade/misc.py index 3d70f6b25..a9aeee80e 100644 --- a/freqtrade/misc.py +++ b/freqtrade/misc.py @@ -4,6 +4,7 @@ import json import logging import time import os +import re from typing import Any, Callable, Dict, List from jsonschema import Draft4Validator, validate @@ -15,6 +16,11 @@ from freqtrade import __version__ logger = logging.getLogger(__name__) +def file_dump_json(filename, data): + with open(filename, 'w') as fp: + json.dump(data, fp) + + class State(enum.Enum): RUNNING = 0 STOPPED = 1 @@ -127,7 +133,7 @@ def parse_args(args: List[str], description: str): dest='dry_run_db', ) parser.add_argument( - '-dd', '--datadir', + '--datadir', help='path to backtest data (default freqdata/tests/testdata', dest='datadir', default=os.path.join('freqtrade', 'tests', 'testdata'), @@ -185,6 +191,13 @@ def build_subcommands(parser: argparse.ArgumentParser) -> None: action='store_true', dest='refresh_pairs', ) + backtesting_cmd.add_argument( + '--timerange', + help='Specify what timerange of data to use.', + default=None, + type=str, + dest='timerange', + ) # Add hyperopt subcommand hyperopt_cmd = subparsers.add_parser('hyperopt', help='hyperopt module') @@ -211,6 +224,43 @@ def build_subcommands(parser: argparse.ArgumentParser) -> None: type=int, metavar='INT', ) + hyperopt_cmd.add_argument( + '--timerange', + help='Specify what timerange of data to use.', + default=None, + type=str, + dest='timerange', + ) + + +def parse_timerange(text): + if text is None: + return None + syntax = [('^-(\d{8})$', (None, 'date')), + ('^(\d{8})-$', ('date', None)), + ('^(\d{8})-(\d{8})$', ('date', 'date')), + ('^(-\d+)$', (None, 'line')), + ('^(\d+)-$', ('line', None)), + ('^(\d+)-(\d+)$', ('index', 'index'))] + for rex, stype in syntax: + # Apply the regular expression to text + m = re.match(rex, text) + if m: # Regex has matched + rvals = m.groups() + n = 0 + start = None + stop = None + if stype[0]: + start = rvals[n] + if stype[0] != 'date': + start = int(start) + n += 1 + if stype[1]: + stop = rvals[n] + if stype[1] != 'date': + stop = int(stop) + return (stype, start, stop) + raise Exception('Incorrect syntax for timerange "%s"' % text) # Required json-schema for user specified config diff --git a/freqtrade/optimize/__init__.py b/freqtrade/optimize/__init__.py index d76334778..c33a3641c 100644 --- a/freqtrade/optimize/__init__.py +++ b/freqtrade/optimize/__init__.py @@ -12,7 +12,20 @@ from freqtrade.analyze import populate_indicators, parse_ticker_dataframe logger = logging.getLogger(__name__) -def load_tickerdata_file(datadir, pair, ticker_interval): +def trim_tickerlist(tickerlist, timerange): + (stype, start, stop) = timerange + if stype == (None, 'line'): + return tickerlist[stop:] + elif stype == ('line', None): + return tickerlist[0:start] + elif stype == ('index', 'index'): + return tickerlist[start:stop] + else: + return tickerlist + + +def load_tickerdata_file(datadir, pair, ticker_interval, + timerange=None): """ Load a pair from file, :return dict OR empty if unsuccesful @@ -30,11 +43,15 @@ def load_tickerdata_file(datadir, pair, ticker_interval): # Read the file, load the json with open(file) as tickerdata: pairdata = json.load(tickerdata) + if timerange: + pairdata = trim_tickerlist(pairdata, timerange) return pairdata -def load_data(datadir: str, ticker_interval: int = 5, pairs: Optional[List[str]] = None, - refresh_pairs: Optional[bool] = False) -> Dict[str, List]: +def load_data(datadir: str, ticker_interval: int = 5, + pairs: Optional[List[str]] = None, + refresh_pairs: Optional[bool] = False, + timerange=None) -> Dict[str, List]: """ Loads ticker history data for the given parameters :param ticker_interval: ticker interval in minutes @@ -51,16 +68,21 @@ def load_data(datadir: str, ticker_interval: int = 5, pairs: Optional[List[str]] download_pairs(datadir, _pairs) for pair in _pairs: - pairdata = load_tickerdata_file(datadir, pair, ticker_interval) + pairdata = load_tickerdata_file(datadir, pair, ticker_interval, timerange=timerange) if not pairdata: # download the tickerdata from exchange download_backtesting_testdata(datadir, pair=pair, interval=ticker_interval) # and retry reading the pair - pairdata = load_tickerdata_file(datadir, pair, ticker_interval) + pairdata = load_tickerdata_file(datadir, pair, ticker_interval, timerange=timerange) result[pair] = pairdata return result +def tickerdata_to_dataframe(data): + preprocessed = preprocess(data) + return preprocessed + + def preprocess(tickerdata: Dict[str, List]) -> Dict[str, DataFrame]: """Creates a dataframe and populates indicators for given ticker data""" return {pair: populate_indicators(parse_ticker_dataframe(pair_data)) diff --git a/freqtrade/optimize/backtesting.py b/freqtrade/optimize/backtesting.py index 6d600b303..4f3d4bb24 100644 --- a/freqtrade/optimize/backtesting.py +++ b/freqtrade/optimize/backtesting.py @@ -13,7 +13,6 @@ from freqtrade import exchange from freqtrade.analyze import populate_buy_trend, populate_sell_trend from freqtrade.exchange import Bittrex from freqtrade.main import min_roi_reached -from freqtrade.optimize import preprocess from freqtrade.persistence import Trade logger = logging.getLogger(__name__) @@ -161,12 +160,13 @@ def start(args): data[pair] = exchange.get_ticker_history(pair, args.ticker_interval) else: logger.info('Using local backtesting data (using whitelist in given config) ...') - data = optimize.load_data(args.datadir, pairs=pairs, ticker_interval=args.ticker_interval, - refresh_pairs=args.refresh_pairs) - logger.info('Using stake_currency: %s ...', config['stake_currency']) logger.info('Using stake_amount: %s ...', config['stake_amount']) + timerange = misc.parse_timerange(args.timerange) + data = optimize.load_data(args.datadir, pairs=pairs, ticker_interval=args.ticker_interval, + refresh_pairs=args.refresh_pairs, + timerange=timerange) max_open_trades = 0 if args.realistic_simulation: logger.info('Using max_open_trades: %s ...', config['max_open_trades']) @@ -176,7 +176,7 @@ def start(args): from freqtrade import main main._CONF = config - preprocessed = preprocess(data) + preprocessed = optimize.tickerdata_to_dataframe(data) # Print timeframe min_date, max_date = get_timeframe(preprocessed) logger.info('Measuring data from %s up to %s ...', min_date.isoformat(), max_date.isoformat()) diff --git a/freqtrade/optimize/hyperopt.py b/freqtrade/optimize/hyperopt.py index 71ddd33c6..b98646090 100644 --- a/freqtrade/optimize/hyperopt.py +++ b/freqtrade/optimize/hyperopt.py @@ -15,7 +15,7 @@ from hyperopt import STATUS_FAIL, STATUS_OK, Trials, fmin, hp, space_eval, tpe from hyperopt.mongoexp import MongoTrials from pandas import DataFrame -from freqtrade import main # noqa +from freqtrade import main, misc # noqa from freqtrade import exchange, optimize from freqtrade.exchange import Bittrex from freqtrade.misc import load_config @@ -30,18 +30,19 @@ logging.getLogger('hyperopt.tpe').setLevel(logging.WARNING) logger = logging.getLogger(__name__) # set TARGET_TRADES to suit your number concurrent trades so its realistic to 20days of data -TARGET_TRADES = 1100 +TARGET_TRADES = 600 TOTAL_TRIES = 0 _CURRENT_TRIES = 0 CURRENT_BEST_LOSS = 100 # max average trade duration in minutes # if eval ends with higher value, we consider it a failed eval -MAX_ACCEPTED_TRADE_DURATION = 240 +MAX_ACCEPTED_TRADE_DURATION = 300 # this is expexted avg profit * expected trade count # for example 3.5%, 1100 trades, EXPECTED_MAX_PROFIT = 3.85 -EXPECTED_MAX_PROFIT = 3.85 +# check that the reported Σ% values do not exceed this! +EXPECTED_MAX_PROFIT = 3.0 # Configuration and data used by hyperopt PROCESSED = None # optimize.preprocess(optimize.load_data()) @@ -57,6 +58,10 @@ main._CONF = OPTIMIZE_CONFIG SPACE = { + 'macd_below_zero': hp.choice('macd_below_zero', [ + {'enabled': False}, + {'enabled': True} + ]), 'mfi': hp.choice('mfi', [ {'enabled': False}, {'enabled': True, 'value': hp.quniform('mfi-value', 5, 25, 1)} @@ -95,13 +100,15 @@ SPACE = { ]), 'trigger': hp.choice('trigger', [ {'type': 'lower_bb'}, + {'type': 'lower_bb_tema'}, {'type': 'faststoch10'}, {'type': 'ao_cross_zero'}, - {'type': 'ema5_cross_ema10'}, + {'type': 'ema3_cross_ema10'}, {'type': 'macd_cross_signal'}, {'type': 'sar_reversal'}, - {'type': 'stochf_cross'}, {'type': 'ht_sine'}, + {'type': 'heiken_reversal_bull'}, + {'type': 'di_cross'}, ]), 'stoploss': hp.uniform('stoploss', -0.5, -0.02), } @@ -133,10 +140,11 @@ def log_results(results): if results['loss'] < CURRENT_BEST_LOSS: CURRENT_BEST_LOSS = results['loss'] - logger.info('{:5d}/{}: {}'.format( + logger.info('{:5d}/{}: {}. Loss {:.5f}'.format( results['current_tries'], results['total_tries'], - results['result'])) + results['result'], + results['loss'])) else: print('.', end='') sys.stdout.flush() @@ -144,9 +152,9 @@ def log_results(results): def calculate_loss(total_profit: float, trade_count: int, trade_duration: float): """ objective function, returns smaller number for more optimal results """ - trade_loss = 1 - 0.35 * exp(-(trade_count - TARGET_TRADES) ** 2 / 10 ** 5.2) + trade_loss = 1 - 0.25 * exp(-(trade_count - TARGET_TRADES) ** 2 / 10 ** 5.8) profit_loss = max(0, 1 - total_profit / EXPECTED_MAX_PROFIT) - duration_loss = min(trade_duration / MAX_ACCEPTED_TRADE_DURATION, 1) + duration_loss = 0.7 + 0.3 * min(trade_duration / MAX_ACCEPTED_TRADE_DURATION, 1) return trade_loss + profit_loss + duration_loss @@ -190,12 +198,13 @@ def optimizer(params): def format_results(results: DataFrame): return ('{:6d} trades. Avg profit {: 5.2f}%. ' - 'Total profit {: 11.8f} BTC. Avg duration {:5.1f} mins.').format( + 'Total profit {: 11.8f} BTC ({:.4f}Σ%). Avg duration {:5.1f} mins.').format( len(results.index), results.profit_percent.mean() * 100.0, results.profit_BTC.sum(), + results.profit_percent.sum(), results.duration.mean() * 5, - ) + ) def buy_strategy_generator(params): @@ -204,6 +213,8 @@ def buy_strategy_generator(params): # GUARDS AND TRENDS if params['uptrend_long_ema']['enabled']: conditions.append(dataframe['ema50'] > dataframe['ema100']) + if params['macd_below_zero']['enabled']: + conditions.append(dataframe['macd'] < 0) if params['uptrend_short_ema']['enabled']: conditions.append(dataframe['ema5'] > dataframe['ema10']) if params['mfi']['enabled']: @@ -224,14 +235,17 @@ def buy_strategy_generator(params): # TRIGGERS triggers = { - 'lower_bb': dataframe['tema'] <= dataframe['blower'], + 'lower_bb': (dataframe['close'] < dataframe['bb_lowerband']), + 'lower_bb_tema': (dataframe['tema'] < dataframe['bb_lowerband']), 'faststoch10': (crossed_above(dataframe['fastd'], 10.0)), 'ao_cross_zero': (crossed_above(dataframe['ao'], 0.0)), - 'ema5_cross_ema10': (crossed_above(dataframe['ema5'], dataframe['ema10'])), + 'ema3_cross_ema10': (crossed_above(dataframe['ema3'], dataframe['ema10'])), 'macd_cross_signal': (crossed_above(dataframe['macd'], dataframe['macdsignal'])), 'sar_reversal': (crossed_above(dataframe['close'], dataframe['sar'])), - 'stochf_cross': (crossed_above(dataframe['fastk'], dataframe['fastd'])), 'ht_sine': (crossed_above(dataframe['htleadsine'], dataframe['htsine'])), + 'heiken_reversal_bull': (crossed_above(dataframe['ha_close'], dataframe['ha_open'])) & + (dataframe['ha_low'] == dataframe['ha_open']), + 'di_cross': (crossed_above(dataframe['plus_di'], dataframe['minus_di'])), } conditions.append(triggers.get(params['trigger']['type'])) @@ -259,8 +273,11 @@ def start(args): logger.info('Using config: %s ...', args.config) config = load_config(args.config) pairs = config['exchange']['pair_whitelist'] - PROCESSED = optimize.preprocess(optimize.load_data( - args.datadir, pairs=pairs, ticker_interval=args.ticker_interval)) + timerange = misc.parse_timerange(args.timerange) + data = optimize.load_data(args.datadir, pairs=pairs, + ticker_interval=args.ticker_interval, + timerange=timerange) + PROCESSED = optimize.tickerdata_to_dataframe(data) if args.mongodb: logger.info('Using mongodb ...') diff --git a/freqtrade/rpc/telegram.py b/freqtrade/rpc/telegram.py index 009714682..12e701a1c 100644 --- a/freqtrade/rpc/telegram.py +++ b/freqtrade/rpc/telegram.py @@ -382,13 +382,31 @@ def _balance(bot: Bot, update: Update) -> None: if not balances: output = '`All balances are zero.`' + total = 0.0 for currency in balances: + coin = currency['Currency'] + if coin == 'BTC': + currency["Rate"] = 1.0 + else: + currency["Rate"] = exchange.get_ticker('BTC_' + coin, False)['bid'] + currency['BTC'] = currency["Rate"] * currency["Balance"] + total = total + currency['BTC'] output += """*Currency*: {Currency} *Available*: {Available} *Balance*: {Balance} *Pending*: {Pending} +*Est. BTC*: {BTC: .8f} """.format(**currency) + + symbol = _CONF['fiat_display_currency'] + value = _FIAT_CONVERT.convert_amount( + total, 'BTC', symbol + ) + output += """*Estimated Value*: +*BTC*: {0: .8f} +*{1}*: {2: .2f} +""".format(total, symbol, value) send_msg(output) diff --git a/freqtrade/tests/exchange/test_exchange_bittrex.py b/freqtrade/tests/exchange/test_exchange_bittrex.py index 35d242b55..3931836c1 100644 --- a/freqtrade/tests/exchange/test_exchange_bittrex.py +++ b/freqtrade/tests/exchange/test_exchange_bittrex.py @@ -212,7 +212,18 @@ def test_exchange_bittrex_get_ticker_bad(): wb = make_wrap_bittrex() fb = FakeBittrex() fb.result = {'success': True, - 'result': {'Bid': 1}} # incomplete result + 'result': {'Bid': 1, 'Ask': 0}} # incomplete result + + with pytest.raises(ContentDecodingError, match=r'.*Got invalid response from bittrex params.*'): + wb.get_ticker('BTC_ETH') + fb.result = {'success': False, + 'message': 'gone bad' + } + with pytest.raises(btx.OperationalException, match=r'.*gone bad.*'): + wb.get_ticker('BTC_ETH') + + fb.result = {'success': True, + 'result': {}} # incomplete result with pytest.raises(ContentDecodingError, match=r'.*Got invalid response from bittrex params.*'): wb.get_ticker('BTC_ETH') fb.result = {'success': False, diff --git a/freqtrade/tests/optimize/test_backtesting.py b/freqtrade/tests/optimize/test_backtesting.py index 5f899a48a..c570801c5 100644 --- a/freqtrade/tests/optimize/test_backtesting.py +++ b/freqtrade/tests/optimize/test_backtesting.py @@ -11,6 +11,13 @@ from freqtrade.optimize.backtesting import backtest, generate_text_table, get_ti import freqtrade.optimize.backtesting as backtesting +def trim_dictlist(dl, num): + new = {} + for pair, pair_data in dl.items(): + new[pair] = pair_data[num:] + return new + + def test_generate_text_table(): results = pd.DataFrame( { @@ -43,6 +50,7 @@ def test_backtest(default_conf, mocker): exchange._API = Bittrex({'key': '', 'secret': ''}) data = optimize.load_data(None, ticker_interval=5, pairs=['BTC_ETH']) + data = trim_dictlist(data, -200) results = backtest(default_conf['stake_amount'], optimize.preprocess(data), 10, True) assert not results.empty @@ -54,21 +62,15 @@ def test_backtest_1min_ticker_interval(default_conf, mocker): # Run a backtesting for an exiting 5min ticker_interval data = optimize.load_data(None, ticker_interval=1, pairs=['BTC_UNITEST']) + data = trim_dictlist(data, -200) results = backtest(default_conf['stake_amount'], optimize.preprocess(data), 1, True) assert not results.empty -def trim_dictlist(dl, num): - new = {} - for pair, pair_data in dl.items(): - new[pair] = pair_data[num:] - return new - - def load_data_test(what): - data = optimize.load_data(None, ticker_interval=1, pairs=['BTC_UNITEST']) - data = trim_dictlist(data, -100) + timerange = ((None, 'line'), None, -100) + data = optimize.load_data(None, ticker_interval=1, pairs=['BTC_UNITEST'], timerange=timerange) pair = data['BTC_UNITEST'] datalen = len(pair) # Depending on the what parameter we now adjust the @@ -125,6 +127,7 @@ def simple_backtest(config, contour, num_results): def test_backtest2(default_conf, mocker): mocker.patch.dict('freqtrade.main._CONF', default_conf) data = optimize.load_data(None, ticker_interval=5, pairs=['BTC_ETH']) + data = trim_dictlist(data, -200) results = backtest(default_conf['stake_amount'], optimize.preprocess(data), 10, True) assert not results.empty @@ -149,10 +152,10 @@ def test_backtest_pricecontours(default_conf, mocker): simple_backtest(default_conf, contour, numres) -def mocked_load_data(datadir, pairs=[], ticker_interval=0, refresh_pairs=False): - tickerdata = optimize.load_tickerdata_file(datadir, 'BTC_UNITEST', 1) +def mocked_load_data(datadir, pairs=[], ticker_interval=0, refresh_pairs=False, timerange=None): + tickerdata = optimize.load_tickerdata_file(datadir, 'BTC_UNITEST', 1, timerange=timerange) pairdata = {'BTC_UNITEST': tickerdata} - return trim_dictlist(pairdata, -100) + return pairdata def test_backtest_start(default_conf, mocker, caplog): @@ -166,6 +169,7 @@ def test_backtest_start(default_conf, mocker, caplog): args.level = 10 args.live = False args.datadir = None + args.timerange = '-100' # needed due to MagicMock malleability backtesting.start(args) # check the logs, that will contain the backtest result exists = ['Using max_open_trades: 1 ...', diff --git a/freqtrade/tests/optimize/test_hyperopt.py b/freqtrade/tests/optimize/test_hyperopt.py index 3e03d26c0..6e21cde76 100644 --- a/freqtrade/tests/optimize/test_hyperopt.py +++ b/freqtrade/tests/optimize/test_hyperopt.py @@ -54,6 +54,7 @@ def create_trials(mocker): def test_start_calls_fmin(mocker): trials = create_trials(mocker) + mocker.patch('freqtrade.optimize.tickerdata_to_dataframe') mocker.patch('freqtrade.optimize.hyperopt.TRIALS', return_value=trials) mocker.patch('freqtrade.optimize.hyperopt.sorted', return_value=trials.results) @@ -61,7 +62,8 @@ def test_start_calls_fmin(mocker): mocker.patch('freqtrade.optimize.load_data') mock_fmin = mocker.patch('freqtrade.optimize.hyperopt.fmin', return_value={}) - args = mocker.Mock(epochs=1, config='config.json.example', mongodb=False) + args = mocker.Mock(epochs=1, config='config.json.example', mongodb=False, + timerange=None) start(args) mock_fmin.assert_called_once() @@ -70,11 +72,12 @@ def test_start_calls_fmin(mocker): def test_start_uses_mongotrials(mocker): mock_mongotrials = mocker.patch('freqtrade.optimize.hyperopt.MongoTrials', return_value=create_trials(mocker)) - mocker.patch('freqtrade.optimize.preprocess') + mocker.patch('freqtrade.optimize.tickerdata_to_dataframe') mocker.patch('freqtrade.optimize.load_data') mocker.patch('freqtrade.optimize.hyperopt.fmin', return_value={}) - args = mocker.Mock(epochs=1, config='config.json.example', mongodb=True) + args = mocker.Mock(epochs=1, config='config.json.example', mongodb=True, + timerange=None) start(args) mock_mongotrials.assert_called_once() @@ -107,6 +110,7 @@ def test_no_log_if_loss_does_not_improve(mocker): def test_fmin_best_results(mocker, caplog): fmin_result = { + "macd_below_zero": 0, "adx": 1, "adx-value": 15.0, "fastd": 1, @@ -124,11 +128,12 @@ def test_fmin_best_results(mocker, caplog): } mocker.patch('freqtrade.optimize.hyperopt.MongoTrials', return_value=create_trials(mocker)) - mocker.patch('freqtrade.optimize.preprocess') + mocker.patch('freqtrade.optimize.tickerdata_to_dataframe') mocker.patch('freqtrade.optimize.load_data') mocker.patch('freqtrade.optimize.hyperopt.fmin', return_value=fmin_result) - args = mocker.Mock(epochs=1, config='config.json.example') + args = mocker.Mock(epochs=1, config='config.json.example', + timerange=None) start(args) exists = [ @@ -136,7 +141,7 @@ def test_fmin_best_results(mocker, caplog): '"adx": {\n "enabled": true,\n "value": 15.0\n },', '"green_candle": {\n "enabled": true\n },', '"mfi": {\n "enabled": false\n },', - '"trigger": {\n "type": "ao_cross_zero"\n },', + '"trigger": {\n "type": "faststoch10"\n },', '"stoploss": -0.1', ] @@ -146,11 +151,12 @@ def test_fmin_best_results(mocker, caplog): def test_fmin_throw_value_error(mocker, caplog): mocker.patch('freqtrade.optimize.hyperopt.MongoTrials', return_value=create_trials(mocker)) - mocker.patch('freqtrade.optimize.preprocess') + mocker.patch('freqtrade.optimize.tickerdata_to_dataframe') mocker.patch('freqtrade.optimize.load_data') mocker.patch('freqtrade.optimize.hyperopt.fmin', side_effect=ValueError()) - args = mocker.Mock(epochs=1, config='config.json.example') + args = mocker.Mock(epochs=1, config='config.json.example', + timerange=None) start(args) exists = [ @@ -184,7 +190,8 @@ def test_resuming_previous_hyperopt_results_succeeds(mocker): return_value={}) args = mocker.Mock(epochs=1, config='config.json.example', - mongodb=False) + mongodb=False, + timerange=None) start(args) diff --git a/freqtrade/tests/optimize/test_optimize.py b/freqtrade/tests/optimize/test_optimize.py index e325e021c..4af5ed828 100644 --- a/freqtrade/tests/optimize/test_optimize.py +++ b/freqtrade/tests/optimize/test_optimize.py @@ -189,3 +189,11 @@ def test_init(mocker): conf = {'exchange': {'pair_whitelist': []}} mocker.patch('freqtrade.optimize.hyperopt_optimize_conf', return_value=conf) assert {} == optimize.load_data('', pairs=[], refresh_pairs=True) + + +def test_tickerdata_to_dataframe(): + timerange = ((None, 'line'), None, -100) + tick = load_tickerdata_file(None, 'BTC_UNITEST', 1, timerange=timerange) + tickerlist = {'BTC_UNITEST': tick} + data = optimize.tickerdata_to_dataframe(tickerlist) + assert 100 == len(data['BTC_UNITEST']) diff --git a/freqtrade/tests/rpc/test_rpc_telegram.py b/freqtrade/tests/rpc/test_rpc_telegram.py index 428c544c1..992693248 100644 --- a/freqtrade/tests/rpc/test_rpc_telegram.py +++ b/freqtrade/tests/rpc/test_rpc_telegram.py @@ -77,7 +77,7 @@ def test_authorized_only_exception(default_conf, mocker): def test_status_handle(default_conf, update, ticker, mocker): mocker.patch.dict('freqtrade.main._CONF', default_conf) - mocker.patch('freqtrade.main.get_signal', side_effect=lambda s, t: True) + mocker.patch('freqtrade.main.get_signal', side_effect=lambda s: (True, False)) msg_mock = MagicMock() mocker.patch('freqtrade.main.rpc.send_msg', MagicMock()) mocker.patch.multiple('freqtrade.rpc.telegram', @@ -112,7 +112,7 @@ def test_status_handle(default_conf, update, ticker, mocker): def test_status_table_handle(default_conf, update, ticker, mocker): mocker.patch.dict('freqtrade.main._CONF', default_conf) - mocker.patch('freqtrade.main.get_signal', side_effect=lambda s, t: True) + mocker.patch('freqtrade.main.get_signal', side_effect=lambda s: (True, False)) msg_mock = MagicMock() mocker.patch('freqtrade.main.rpc.send_msg', MagicMock()) mocker.patch.multiple( @@ -154,7 +154,7 @@ def test_status_table_handle(default_conf, update, ticker, mocker): def test_profit_handle( default_conf, update, ticker, ticker_sell_up, limit_buy_order, limit_sell_order, mocker): mocker.patch.dict('freqtrade.main._CONF', default_conf) - mocker.patch('freqtrade.main.get_signal', side_effect=lambda s, t: True) + mocker.patch('freqtrade.main.get_signal', side_effect=lambda s: (True, False)) msg_mock = MagicMock() mocker.patch('freqtrade.main.rpc.send_msg', MagicMock()) mocker.patch.multiple('freqtrade.rpc.telegram', @@ -210,7 +210,7 @@ def test_profit_handle( def test_forcesell_handle(default_conf, update, ticker, ticker_sell_up, mocker): mocker.patch.dict('freqtrade.main._CONF', default_conf) - mocker.patch('freqtrade.main.get_signal', side_effect=lambda s, t: True) + mocker.patch('freqtrade.main.get_signal', side_effect=lambda s: (True, False)) rpc_mock = mocker.patch('freqtrade.main.rpc.send_msg', MagicMock()) mocker.patch.multiple('freqtrade.rpc.telegram', _CONF=default_conf, @@ -247,7 +247,7 @@ def test_forcesell_handle(default_conf, update, ticker, ticker_sell_up, mocker): def test_forcesell_down_handle(default_conf, update, ticker, ticker_sell_down, mocker): mocker.patch.dict('freqtrade.main._CONF', default_conf) - mocker.patch('freqtrade.main.get_signal', side_effect=lambda s, t: True) + mocker.patch('freqtrade.main.get_signal', side_effect=lambda s: (True, False)) rpc_mock = mocker.patch('freqtrade.main.rpc.send_msg', MagicMock()) mocker.patch.multiple('freqtrade.rpc.telegram', _CONF=default_conf, @@ -308,7 +308,7 @@ def test_exec_forcesell_open_orders(default_conf, ticker, mocker): def test_forcesell_all_handle(default_conf, update, ticker, mocker): mocker.patch.dict('freqtrade.main._CONF', default_conf) - mocker.patch('freqtrade.main.get_signal', side_effect=lambda s, t: True) + mocker.patch('freqtrade.main.get_signal', side_effect=lambda s: (True, False)) rpc_mock = mocker.patch('freqtrade.main.rpc.send_msg', MagicMock()) mocker.patch.multiple('freqtrade.rpc.telegram', _CONF=default_conf, @@ -339,7 +339,7 @@ def test_forcesell_all_handle(default_conf, update, ticker, mocker): def test_forcesell_handle_invalid(default_conf, update, mocker): mocker.patch.dict('freqtrade.main._CONF', default_conf) - mocker.patch('freqtrade.main.get_signal', side_effect=lambda s, t: True) + mocker.patch('freqtrade.main.get_signal', side_effect=lambda s: (True, True)) msg_mock = MagicMock() mocker.patch.multiple('freqtrade.rpc.telegram', _CONF=default_conf, @@ -376,7 +376,7 @@ def test_forcesell_handle_invalid(default_conf, update, mocker): def test_performance_handle( default_conf, update, ticker, limit_buy_order, limit_sell_order, mocker): mocker.patch.dict('freqtrade.main._CONF', default_conf) - mocker.patch('freqtrade.main.get_signal', side_effect=lambda s, t: True) + mocker.patch('freqtrade.main.get_signal', side_effect=lambda s: (True, False)) msg_mock = MagicMock() mocker.patch('freqtrade.main.rpc.send_msg', MagicMock()) mocker.patch.multiple('freqtrade.rpc.telegram', @@ -410,7 +410,7 @@ def test_performance_handle( def test_daily_handle( default_conf, update, ticker, limit_buy_order, limit_sell_order, mocker): mocker.patch.dict('freqtrade.main._CONF', default_conf) - mocker.patch('freqtrade.main.get_signal', side_effect=lambda s, t: True) + mocker.patch('freqtrade.main.get_signal', side_effect=lambda s: (True, False)) msg_mock = MagicMock() mocker.patch('freqtrade.main.rpc.send_msg', MagicMock()) mocker.patch.multiple('freqtrade.rpc.telegram', @@ -460,7 +460,7 @@ def test_daily_handle( def test_count_handle(default_conf, update, ticker, mocker): mocker.patch.dict('freqtrade.main._CONF', default_conf) - mocker.patch('freqtrade.main.get_signal', side_effect=lambda s, t: True) + mocker.patch('freqtrade.main.get_signal', side_effect=lambda s: (True, False)) msg_mock = MagicMock() mocker.patch.multiple( 'freqtrade.rpc.telegram', @@ -492,7 +492,7 @@ def test_count_handle(default_conf, update, ticker, mocker): def test_performance_handle_invalid(default_conf, update, mocker): mocker.patch.dict('freqtrade.main._CONF', default_conf) - mocker.patch('freqtrade.main.get_signal', side_effect=lambda s, t: True) + mocker.patch('freqtrade.main.get_signal', side_effect=lambda s: (True, True)) msg_mock = MagicMock() mocker.patch.multiple('freqtrade.rpc.telegram', _CONF=default_conf, @@ -606,11 +606,15 @@ def test_balance_handle(default_conf, update, mocker): send_msg=msg_mock) mocker.patch.multiple('freqtrade.main.exchange', get_balances=MagicMock(return_value=mock_balance)) + mocker.patch.multiple('freqtrade.fiat_convert.Pymarketcap', + ticker=MagicMock(return_value={'price_usd': 15000.0}), + _cache_symbols=MagicMock(return_value={'BTC': 1})) _balance(bot=MagicMock(), update=update) assert msg_mock.call_count == 1 assert '*Currency*: BTC' in msg_mock.call_args_list[0][0][0] assert 'Balance' in msg_mock.call_args_list[0][0][0] + assert 'Est. BTC' in msg_mock.call_args_list[0][0][0] def test_help_handle(default_conf, update, mocker): diff --git a/freqtrade/tests/test_analyze.py b/freqtrade/tests/test_analyze.py index 9e4693205..70815a280 100644 --- a/freqtrade/tests/test_analyze.py +++ b/freqtrade/tests/test_analyze.py @@ -8,7 +8,7 @@ import datetime import pytest from pandas import DataFrame -from freqtrade.analyze import (SignalType, get_signal, parse_ticker_dataframe, +from freqtrade.analyze import (get_signal, parse_ticker_dataframe, populate_buy_trend, populate_indicators, populate_sell_trend) @@ -42,35 +42,35 @@ def test_returns_latest_buy_signal(mocker): mocker.patch('freqtrade.analyze.get_ticker_history', return_value=MagicMock()) mocker.patch( 'freqtrade.analyze.analyze_ticker', - return_value=DataFrame([{'buy': 1, 'date': arrow.utcnow()}]) + return_value=DataFrame([{'buy': 1, 'sell': 0, 'date': arrow.utcnow()}]) ) - assert get_signal('BTC-ETH', SignalType.BUY) + assert get_signal('BTC-ETH') == (True, False) mocker.patch( 'freqtrade.analyze.analyze_ticker', - return_value=DataFrame([{'buy': 0, 'date': arrow.utcnow()}]) + return_value=DataFrame([{'buy': 0, 'sell': 1, 'date': arrow.utcnow()}]) ) - assert not get_signal('BTC-ETH', SignalType.BUY) + assert get_signal('BTC-ETH') == (False, True) def test_returns_latest_sell_signal(mocker): mocker.patch('freqtrade.analyze.get_ticker_history', return_value=MagicMock()) mocker.patch( 'freqtrade.analyze.analyze_ticker', - return_value=DataFrame([{'sell': 1, 'date': arrow.utcnow()}]) + return_value=DataFrame([{'sell': 1, 'buy': 0, 'date': arrow.utcnow()}]) ) - assert get_signal('BTC-ETH', SignalType.SELL) + assert get_signal('BTC-ETH') == (False, True) mocker.patch( 'freqtrade.analyze.analyze_ticker', - return_value=DataFrame([{'sell': 0, 'date': arrow.utcnow()}]) + return_value=DataFrame([{'sell': 0, 'buy': 1, 'date': arrow.utcnow()}]) ) - assert not get_signal('BTC-ETH', SignalType.SELL) + assert get_signal('BTC-ETH') == (True, False) def test_get_signal_empty(mocker, caplog): mocker.patch('freqtrade.analyze.get_ticker_history', return_value=None) - assert not get_signal('foo', SignalType.BUY) + assert (False, False) == get_signal('foo') assert tt.log_has('Empty ticker history for pair foo', caplog.record_tuples) @@ -79,17 +79,15 @@ def test_get_signal_execption_valueerror(mocker, caplog): mocker.patch('freqtrade.analyze.get_ticker_history', return_value=1) mocker.patch('freqtrade.analyze.analyze_ticker', side_effect=ValueError('xyz')) - assert not get_signal('foo', SignalType.BUY) + assert (False, False) == get_signal('foo') assert tt.log_has('Unable to analyze ticker for pair foo: xyz', caplog.record_tuples) -# This error should never occur becase analyze_ticker is run first, -# and that function can only add columns, it cant delete all rows from the dataframe def test_get_signal_empty_dataframe(mocker, caplog): mocker.patch('freqtrade.analyze.get_ticker_history', return_value=1) mocker.patch('freqtrade.analyze.analyze_ticker', return_value=DataFrame([])) - assert not get_signal('xyz', SignalType.BUY) + assert (False, False) == get_signal('xyz') assert tt.log_has('Empty dataframe for pair xyz', caplog.record_tuples) @@ -100,7 +98,7 @@ def test_get_signal_old_dataframe(mocker, caplog): oldtime = arrow.utcnow() - datetime.timedelta(minutes=11) ticks = DataFrame([{'buy': 1, 'date': oldtime}]) mocker.patch('freqtrade.analyze.analyze_ticker', return_value=DataFrame(ticks)) - assert not get_signal('xyz', SignalType.BUY) + assert (False, False) == get_signal('xyz') assert tt.log_has('Too old dataframe for pair xyz', caplog.record_tuples) @@ -110,4 +108,4 @@ def test_get_signal_handles_exceptions(mocker): mocker.patch('freqtrade.analyze.analyze_ticker', side_effect=Exception('invalid ticker history ')) - assert not get_signal('BTC-ETH', SignalType.BUY) + assert get_signal('BTC-ETH') == (False, False) diff --git a/freqtrade/tests/test_main.py b/freqtrade/tests/test_main.py index 7b93c623a..8e1872677 100644 --- a/freqtrade/tests/test_main.py +++ b/freqtrade/tests/test_main.py @@ -11,7 +11,6 @@ from sqlalchemy import create_engine import freqtrade.main as main from freqtrade import DependencyException, OperationalException -from freqtrade.analyze import SignalType from freqtrade.exchange import Exchanges from freqtrade.main import (_process, check_handle_timedout, create_trade, execute_sell, get_target_bid, handle_trade, init) @@ -91,7 +90,7 @@ def test_process_maybe_execute_buy_exception(default_conf, mocker, caplog): def test_process_trade_creation(default_conf, ticker, limit_buy_order, health, mocker): mocker.patch.dict('freqtrade.main._CONF', default_conf) mocker.patch.multiple('freqtrade.rpc', init=MagicMock(), send_msg=MagicMock()) - mocker.patch('freqtrade.main.get_signal', side_effect=lambda s, t: True) + mocker.patch('freqtrade.main.get_signal', side_effect=lambda s: (True, False)) mocker.patch.multiple('freqtrade.main.exchange', validate_pairs=MagicMock(), get_ticker=ticker, @@ -121,7 +120,7 @@ def test_process_trade_creation(default_conf, ticker, limit_buy_order, health, m def test_process_exchange_failures(default_conf, ticker, health, mocker): mocker.patch.dict('freqtrade.main._CONF', default_conf) mocker.patch.multiple('freqtrade.rpc', init=MagicMock(), send_msg=MagicMock()) - mocker.patch('freqtrade.main.get_signal', side_effect=lambda s, t: True) + mocker.patch('freqtrade.main.get_signal', side_effect=lambda s: (True, False)) sleep_mock = mocker.patch('time.sleep', side_effect=lambda _: None) mocker.patch.multiple('freqtrade.main.exchange', validate_pairs=MagicMock(), @@ -138,7 +137,7 @@ def test_process_operational_exception(default_conf, ticker, health, mocker): msg_mock = MagicMock() mocker.patch.dict('freqtrade.main._CONF', default_conf) mocker.patch.multiple('freqtrade.rpc', init=MagicMock(), send_msg=msg_mock) - mocker.patch('freqtrade.main.get_signal', side_effect=lambda s, t: True) + mocker.patch('freqtrade.main.get_signal', side_effect=lambda s: (True, False)) mocker.patch.multiple('freqtrade.main.exchange', validate_pairs=MagicMock(), get_ticker=ticker, @@ -156,8 +155,7 @@ def test_process_operational_exception(default_conf, ticker, health, mocker): def test_process_trade_handling(default_conf, ticker, limit_buy_order, health, mocker): mocker.patch.dict('freqtrade.main._CONF', default_conf) mocker.patch.multiple('freqtrade.rpc', init=MagicMock(), send_msg=MagicMock()) - mocker.patch('freqtrade.main.get_signal', - side_effect=lambda *args: False if args[1] == SignalType.SELL else True) + mocker.patch('freqtrade.main.get_signal', side_effect=lambda s: (True, False)) mocker.patch.multiple('freqtrade.main.exchange', validate_pairs=MagicMock(), get_ticker=ticker, @@ -179,7 +177,7 @@ def test_process_trade_handling(default_conf, ticker, limit_buy_order, health, m def test_create_trade(default_conf, ticker, limit_buy_order, mocker): mocker.patch.dict('freqtrade.main._CONF', default_conf) - mocker.patch('freqtrade.main.get_signal', side_effect=lambda s, t: True) + mocker.patch('freqtrade.main.get_signal', side_effect=lambda s: (True, False)) mocker.patch.multiple('freqtrade.rpc', init=MagicMock(), send_msg=MagicMock()) mocker.patch.multiple('freqtrade.main.exchange', validate_pairs=MagicMock(), @@ -210,7 +208,7 @@ def test_create_trade(default_conf, ticker, limit_buy_order, mocker): def test_create_trade_minimal_amount(default_conf, ticker, mocker): mocker.patch.dict('freqtrade.main._CONF', default_conf) mocker.patch.multiple('freqtrade.rpc', init=MagicMock(), send_msg=MagicMock()) - mocker.patch('freqtrade.main.get_signal', side_effect=lambda s, t: True) + mocker.patch('freqtrade.main.get_signal', side_effect=lambda s: (True, False)) buy_mock = mocker.patch( 'freqtrade.main.exchange.buy', MagicMock(return_value='mocked_limit_buy') ) @@ -226,7 +224,7 @@ def test_create_trade_minimal_amount(default_conf, ticker, mocker): def test_create_trade_no_stake_amount(default_conf, ticker, mocker): mocker.patch.dict('freqtrade.main._CONF', default_conf) - mocker.patch('freqtrade.main.get_signal', side_effect=lambda s, t: True) + mocker.patch('freqtrade.main.get_signal', side_effect=lambda s: (True, False)) mocker.patch.multiple('freqtrade.rpc', init=MagicMock(), send_msg=MagicMock()) mocker.patch.multiple('freqtrade.main.exchange', validate_pairs=MagicMock(), @@ -239,7 +237,7 @@ def test_create_trade_no_stake_amount(default_conf, ticker, mocker): def test_create_trade_no_pairs(default_conf, ticker, mocker): mocker.patch.dict('freqtrade.main._CONF', default_conf) - mocker.patch('freqtrade.main.get_signal', side_effect=lambda s, t: True) + mocker.patch('freqtrade.main.get_signal', side_effect=lambda s: (True, False)) mocker.patch.multiple('freqtrade.rpc', init=MagicMock(), send_msg=MagicMock()) mocker.patch.multiple('freqtrade.main.exchange', validate_pairs=MagicMock(), @@ -255,7 +253,7 @@ def test_create_trade_no_pairs(default_conf, ticker, mocker): def test_create_trade_no_pairs_after_blacklist(default_conf, ticker, mocker): mocker.patch.dict('freqtrade.main._CONF', default_conf) - mocker.patch('freqtrade.main.get_signal', side_effect=lambda s, t: True) + mocker.patch('freqtrade.main.get_signal', side_effect=lambda s: (True, False)) mocker.patch.multiple('freqtrade.rpc', init=MagicMock(), send_msg=MagicMock()) mocker.patch.multiple('freqtrade.main.exchange', validate_pairs=MagicMock(), @@ -273,7 +271,7 @@ def test_create_trade_no_pairs_after_blacklist(default_conf, ticker, mocker): def test_create_trade_no_signal(default_conf, ticker, mocker): default_conf['dry_run'] = True mocker.patch.dict('freqtrade.main._CONF', default_conf) - mocker.patch('freqtrade.main.get_signal', MagicMock(return_value=False)) + mocker.patch('freqtrade.main.get_signal', MagicMock(return_value=(False, False))) mocker.patch.multiple('freqtrade.exchange', get_ticker_history=MagicMock(return_value=20)) mocker.patch.multiple('freqtrade.main.exchange', @@ -286,7 +284,7 @@ def test_create_trade_no_signal(default_conf, ticker, mocker): def test_handle_trade(default_conf, limit_buy_order, limit_sell_order, mocker): mocker.patch.dict('freqtrade.main._CONF', default_conf) - mocker.patch('freqtrade.main.get_signal', side_effect=lambda s, t: True) + mocker.patch('freqtrade.main.get_signal', side_effect=lambda s: (True, False)) mocker.patch.multiple('freqtrade.rpc', init=MagicMock(), send_msg=MagicMock()) mocker.patch.multiple('freqtrade.main.exchange', validate_pairs=MagicMock(), @@ -309,6 +307,7 @@ def test_handle_trade(default_conf, limit_buy_order, limit_sell_order, mocker): trade.update(limit_buy_order) assert trade.is_open is True + mocker.patch('freqtrade.main.get_signal', side_effect=lambda s: (False, True)) handle_trade(trade) assert trade.open_order_id == 'mocked_limit_sell' @@ -321,11 +320,57 @@ def test_handle_trade(default_conf, limit_buy_order, limit_sell_order, mocker): assert trade.close_date is not None +def test_handle_overlpapping_signals(default_conf, ticker, mocker, caplog): + default_conf.update({'experimental': {'use_sell_signal': True}}) + mocker.patch.dict('freqtrade.main._CONF', default_conf) + + mocker.patch('freqtrade.main.get_signal', side_effect=lambda s: (True, True)) + mocker.patch.multiple('freqtrade.rpc', init=MagicMock(), send_msg=MagicMock()) + mocker.patch.multiple('freqtrade.main.exchange', + validate_pairs=MagicMock(), + get_ticker=ticker, + buy=MagicMock(return_value='mocked_limit_buy')) + mocker.patch('freqtrade.main.min_roi_reached', return_value=False) + + init(default_conf, create_engine('sqlite://')) + create_trade(0.001) + + # Buy and Sell triggering, so doing nothing ... + trades = Trade.query.all() + assert len(trades) == 0 + + # Buy is triggering, so buying ... + mocker.patch('freqtrade.main.get_signal', side_effect=lambda s: (True, False)) + create_trade(0.001) + trades = Trade.query.all() + assert len(trades) == 1 + assert trades[0].is_open is True + + # Buy and Sell are not triggering, so doing nothing ... + mocker.patch('freqtrade.main.get_signal', side_effect=lambda s: (False, False)) + assert handle_trade(trades[0]) is False + trades = Trade.query.all() + assert len(trades) == 1 + assert trades[0].is_open is True + + # Buy and Sell are triggering, so doing nothing ... + mocker.patch('freqtrade.main.get_signal', side_effect=lambda s: (True, True)) + assert handle_trade(trades[0]) is False + trades = Trade.query.all() + assert len(trades) == 1 + assert trades[0].is_open is True + + # Sell is triggering, guess what : we are Selling! + mocker.patch('freqtrade.main.get_signal', side_effect=lambda s: (False, True)) + trades = Trade.query.all() + assert handle_trade(trades[0]) is True + + def test_handle_trade_roi(default_conf, ticker, mocker, caplog): default_conf.update({'experimental': {'use_sell_signal': True}}) mocker.patch.dict('freqtrade.main._CONF', default_conf) - mocker.patch('freqtrade.main.get_signal', side_effect=lambda s, t: True) + mocker.patch('freqtrade.main.get_signal', side_effect=lambda s: (True, False)) mocker.patch.multiple('freqtrade.rpc', init=MagicMock(), send_msg=MagicMock()) mocker.patch.multiple('freqtrade.main.exchange', validate_pairs=MagicMock(), @@ -344,11 +389,11 @@ def test_handle_trade_roi(default_conf, ticker, mocker, caplog): # we might just want to check if we are in a sell condition without # executing # if ROI is reached we must sell - mocker.patch('freqtrade.main.get_signal', side_effect=lambda s, t: False) + mocker.patch('freqtrade.main.get_signal', side_effect=lambda s: (False, True)) assert handle_trade(trade) assert ('freqtrade', logging.DEBUG, 'Executing sell due to ROI ...') in caplog.record_tuples # if ROI is reached we must sell even if sell-signal is not signalled - mocker.patch('freqtrade.main.get_signal', side_effect=lambda s, t: True) + mocker.patch('freqtrade.main.get_signal', side_effect=lambda s: (False, True)) assert handle_trade(trade) assert ('freqtrade', logging.DEBUG, 'Executing sell due to ROI ...') in caplog.record_tuples @@ -357,7 +402,7 @@ def test_handle_trade_experimental(default_conf, ticker, mocker, caplog): default_conf.update({'experimental': {'use_sell_signal': True}}) mocker.patch.dict('freqtrade.main._CONF', default_conf) - mocker.patch('freqtrade.main.get_signal', side_effect=lambda s, t: True) + mocker.patch('freqtrade.main.get_signal', side_effect=lambda s: (True, False)) mocker.patch.multiple('freqtrade.rpc', init=MagicMock(), send_msg=MagicMock()) mocker.patch.multiple('freqtrade.main.exchange', validate_pairs=MagicMock(), @@ -371,11 +416,10 @@ def test_handle_trade_experimental(default_conf, ticker, mocker, caplog): trade = Trade.query.first() trade.is_open = True - mocker.patch('freqtrade.main.get_signal', side_effect=lambda s, t: False) + mocker.patch('freqtrade.main.get_signal', side_effect=lambda s: (False, False)) value_returned = handle_trade(trade) - assert ('freqtrade', logging.DEBUG, 'Checking sell_signal ...') in caplog.record_tuples assert value_returned is False - mocker.patch('freqtrade.main.get_signal', side_effect=lambda s, t: True) + mocker.patch('freqtrade.main.get_signal', side_effect=lambda s: (False, True)) assert handle_trade(trade) s = 'Executing sell due to sell signal ...' assert ('freqtrade', logging.DEBUG, s) in caplog.record_tuples @@ -383,7 +427,7 @@ def test_handle_trade_experimental(default_conf, ticker, mocker, caplog): def test_close_trade(default_conf, ticker, limit_buy_order, limit_sell_order, mocker): mocker.patch.dict('freqtrade.main._CONF', default_conf) - mocker.patch('freqtrade.main.get_signal', side_effect=lambda s, t: True) + mocker.patch('freqtrade.main.get_signal', side_effect=lambda s: (True, False)) mocker.patch.multiple('freqtrade.rpc', init=MagicMock(), send_msg=MagicMock()) mocker.patch.multiple('freqtrade.main.exchange', validate_pairs=MagicMock(), @@ -408,7 +452,8 @@ def test_close_trade(default_conf, ticker, limit_buy_order, limit_sell_order, mo def test_check_handle_timedout_buy(default_conf, ticker, limit_buy_order_old, mocker): mocker.patch.dict('freqtrade.main._CONF', default_conf) cancel_order_mock = MagicMock() - mocker.patch.multiple('freqtrade.rpc', init=MagicMock(), send_msg=MagicMock()) + mocker.patch('freqtrade.rpc.init', MagicMock()) + rpc_mock = mocker.patch('freqtrade.main.rpc.send_msg', MagicMock()) mocker.patch.multiple('freqtrade.main.exchange', validate_pairs=MagicMock(), get_ticker=ticker, @@ -433,6 +478,7 @@ def test_check_handle_timedout_buy(default_conf, ticker, limit_buy_order_old, mo # check it does cancel buy orders over the time limit check_handle_timedout(600) assert cancel_order_mock.call_count == 1 + assert rpc_mock.call_count == 1 trades = Trade.query.filter(Trade.open_order_id.is_(trade_buy.open_order_id)).all() assert len(trades) == 0 @@ -454,7 +500,8 @@ def test_handle_timedout_limit_buy(default_conf, mocker): def test_check_handle_timedout_sell(default_conf, ticker, limit_sell_order_old, mocker): mocker.patch.dict('freqtrade.main._CONF', default_conf) cancel_order_mock = MagicMock() - mocker.patch.multiple('freqtrade.rpc', init=MagicMock(), send_msg=MagicMock()) + mocker.patch('freqtrade.rpc.init', MagicMock()) + rpc_mock = mocker.patch('freqtrade.main.rpc.send_msg', MagicMock()) mocker.patch.multiple('freqtrade.main.exchange', validate_pairs=MagicMock(), get_ticker=ticker, @@ -480,6 +527,7 @@ def test_check_handle_timedout_sell(default_conf, ticker, limit_sell_order_old, # check it does cancel sell orders over the time limit check_handle_timedout(600) assert cancel_order_mock.call_count == 1 + assert rpc_mock.call_count == 1 assert trade_sell.is_open is True @@ -501,7 +549,8 @@ def test_check_handle_timedout_partial(default_conf, ticker, limit_buy_order_old mocker): mocker.patch.dict('freqtrade.main._CONF', default_conf) cancel_order_mock = MagicMock() - mocker.patch.multiple('freqtrade.rpc', init=MagicMock(), send_msg=MagicMock()) + mocker.patch('freqtrade.rpc.init', MagicMock()) + rpc_mock = mocker.patch('freqtrade.main.rpc.send_msg', MagicMock()) mocker.patch.multiple('freqtrade.main.exchange', validate_pairs=MagicMock(), get_ticker=ticker, @@ -527,6 +576,7 @@ def test_check_handle_timedout_partial(default_conf, ticker, limit_buy_order_old # note this is for a partially-complete buy order check_handle_timedout(600) assert cancel_order_mock.call_count == 1 + assert rpc_mock.call_count == 1 trades = Trade.query.filter(Trade.open_order_id.is_(trade_buy.open_order_id)).all() assert len(trades) == 1 assert trades[0].amount == 23.0 @@ -550,7 +600,7 @@ def test_balance_bigger_last_ask(mocker): def test_execute_sell_up(default_conf, ticker, ticker_sell_up, mocker): mocker.patch.dict('freqtrade.main._CONF', default_conf) - mocker.patch('freqtrade.main.get_signal', side_effect=lambda s, t: True) + mocker.patch('freqtrade.main.get_signal', side_effect=lambda s: (True, False)) mocker.patch('freqtrade.rpc.init', MagicMock()) rpc_mock = mocker.patch('freqtrade.main.rpc.send_msg', MagicMock()) mocker.patch.multiple('freqtrade.main.exchange', @@ -583,7 +633,7 @@ def test_execute_sell_up(default_conf, ticker, ticker_sell_up, mocker): def test_execute_sell_down(default_conf, ticker, ticker_sell_down, mocker): mocker.patch.dict('freqtrade.main._CONF', default_conf) - mocker.patch('freqtrade.main.get_signal', side_effect=lambda s, t: True) + mocker.patch('freqtrade.main.get_signal', side_effect=lambda s: (True, False)) mocker.patch('freqtrade.rpc.init', MagicMock()) rpc_mock = mocker.patch('freqtrade.main.rpc.send_msg', MagicMock()) mocker.patch.multiple('freqtrade.rpc.telegram', @@ -620,7 +670,7 @@ def test_execute_sell_down(default_conf, ticker, ticker_sell_down, mocker): def test_execute_sell_without_conf(default_conf, ticker, ticker_sell_up, mocker): mocker.patch.dict('freqtrade.main._CONF', default_conf) - mocker.patch('freqtrade.main.get_signal', side_effect=lambda s, t: True) + mocker.patch('freqtrade.main.get_signal', side_effect=lambda s: (True, False)) mocker.patch('freqtrade.rpc.init', MagicMock()) rpc_mock = mocker.patch('freqtrade.main.rpc.send_msg', MagicMock()) mocker.patch.multiple('freqtrade.main.exchange', @@ -657,7 +707,7 @@ def test_sell_profit_only_enable_profit(default_conf, limit_buy_order, mocker): mocker.patch.dict('freqtrade.main._CONF', default_conf) mocker.patch('freqtrade.main.min_roi_reached', return_value=False) - mocker.patch('freqtrade.main.get_signal', side_effect=lambda s, t: True) + mocker.patch('freqtrade.main.get_signal', side_effect=lambda s: (True, False)) mocker.patch.multiple('freqtrade.rpc', init=MagicMock(), send_msg=MagicMock()) mocker.patch.multiple('freqtrade.main.exchange', validate_pairs=MagicMock(), @@ -673,6 +723,7 @@ def test_sell_profit_only_enable_profit(default_conf, limit_buy_order, mocker): trade = Trade.query.first() trade.update(limit_buy_order) + mocker.patch('freqtrade.main.get_signal', side_effect=lambda s: (False, True)) assert handle_trade(trade) is True @@ -684,7 +735,7 @@ def test_sell_profit_only_disable_profit(default_conf, limit_buy_order, mocker): mocker.patch.dict('freqtrade.main._CONF', default_conf) mocker.patch('freqtrade.main.min_roi_reached', return_value=False) - mocker.patch('freqtrade.main.get_signal', side_effect=lambda s, t: True) + mocker.patch('freqtrade.main.get_signal', side_effect=lambda s: (True, False)) mocker.patch.multiple('freqtrade.rpc', init=MagicMock(), send_msg=MagicMock()) mocker.patch.multiple('freqtrade.main.exchange', validate_pairs=MagicMock(), @@ -700,6 +751,7 @@ def test_sell_profit_only_disable_profit(default_conf, limit_buy_order, mocker): trade = Trade.query.first() trade.update(limit_buy_order) + mocker.patch('freqtrade.main.get_signal', side_effect=lambda s: (False, True)) assert handle_trade(trade) is True @@ -711,7 +763,7 @@ def test_sell_profit_only_enable_loss(default_conf, limit_buy_order, mocker): mocker.patch.dict('freqtrade.main._CONF', default_conf) mocker.patch('freqtrade.main.min_roi_reached', return_value=False) - mocker.patch('freqtrade.main.get_signal', side_effect=lambda s, t: True) + mocker.patch('freqtrade.main.get_signal', side_effect=lambda s: (True, False)) mocker.patch.multiple('freqtrade.rpc', init=MagicMock(), send_msg=MagicMock()) mocker.patch.multiple('freqtrade.main.exchange', validate_pairs=MagicMock(), @@ -727,6 +779,7 @@ def test_sell_profit_only_enable_loss(default_conf, limit_buy_order, mocker): trade = Trade.query.first() trade.update(limit_buy_order) + mocker.patch('freqtrade.main.get_signal', side_effect=lambda s: (False, True)) assert handle_trade(trade) is False @@ -738,7 +791,7 @@ def test_sell_profit_only_disable_loss(default_conf, limit_buy_order, mocker): mocker.patch.dict('freqtrade.main._CONF', default_conf) mocker.patch('freqtrade.main.min_roi_reached', return_value=False) - mocker.patch('freqtrade.main.get_signal', side_effect=lambda s, t: True) + mocker.patch('freqtrade.main.get_signal', side_effect=lambda s: (True, False)) mocker.patch.multiple('freqtrade.rpc', init=MagicMock(), send_msg=MagicMock()) mocker.patch.multiple('freqtrade.main.exchange', validate_pairs=MagicMock(), @@ -754,4 +807,6 @@ def test_sell_profit_only_disable_loss(default_conf, limit_buy_order, mocker): trade = Trade.query.first() trade.update(limit_buy_order) + + mocker.patch('freqtrade.main.get_signal', side_effect=lambda s: (False, True)) assert handle_trade(trade) is True diff --git a/freqtrade/tests/test_misc.py b/freqtrade/tests/test_misc.py index 63cfba627..74f611f5f 100644 --- a/freqtrade/tests/test_misc.py +++ b/freqtrade/tests/test_misc.py @@ -8,7 +8,7 @@ import pytest from jsonschema import ValidationError from freqtrade.misc import (common_args_parser, load_config, parse_args, - throttle) + throttle, parse_timerange) def test_throttle(): @@ -133,6 +133,13 @@ def test_parse_args_hyperopt_custom(mocker): assert call_args.func is not None +def test_parse_timerange_incorrect(): + assert ((None, 'line'), None, -200) == parse_timerange('-200') + assert (('line', None), 200, None) == parse_timerange('200-') + with pytest.raises(Exception, match=r'Incorrect syntax.*'): + parse_timerange('-') + + def test_load_config(default_conf, mocker): file_mock = mocker.patch('freqtrade.misc.open', mocker.mock_open( read_data=json.dumps(default_conf) diff --git a/freqtrade/tests/testdata/download_backtest_data.py b/freqtrade/tests/testdata/download_backtest_data.py index 37cd4c95f..0cb545b3a 100755 --- a/freqtrade/tests/testdata/download_backtest_data.py +++ b/freqtrade/tests/testdata/download_backtest_data.py @@ -1,29 +1,38 @@ #!/usr/bin/env python3 """This script generate json data from bittrex""" +import sys import json -from os import path from freqtrade import exchange from freqtrade.exchange import Bittrex +from freqtrade import misc -PAIRS = [ - 'BTC_BCC', 'BTC_ETH', 'BTC_MER', 'BTC_POWR', 'BTC_ETC', - 'BTC_OK', 'BTC_NEO', 'BTC_EMC2', 'BTC_DASH', 'BTC_LSK', - 'BTC_LTC', 'BTC_XZC', 'BTC_OMG', 'BTC_STRAT', 'BTC_XRP', - 'BTC_QTUM', 'BTC_WAVES', 'BTC_VTC', 'BTC_XLM', 'BTC_MCO' -] -TICKER_INTERVAL = 5 # ticker interval in minutes (currently implemented: 1 and 5) -OUTPUT_DIR = path.dirname(path.realpath(__file__)) +parser = misc.common_args_parser('download utility') +parser.add_argument( + '-p', '--pair', + help='JSON file containing pairs to download', + dest='pair', + default=None +) +args = parser.parse_args(sys.argv[1:]) + +TICKER_INTERVALS = [1, 5] # ticker interval in minutes (currently implemented: 1 and 5) +PAIRS = [] + +if args.pair: + with open(args.pair) as file: + PAIRS = json.load(file) +PAIRS = list(set(PAIRS)) + +print('About to download pairs:', PAIRS) # Init Bittrex exchange exchange._API = Bittrex({'key': '', 'secret': ''}) for pair in PAIRS: - data = exchange.get_ticker_history(pair, TICKER_INTERVAL) - filename = path.join(OUTPUT_DIR, '{}-{}.json'.format( - pair, - TICKER_INTERVAL, - )) - with open(filename, 'w') as fp: - json.dump(data, fp) + for tick_interval in TICKER_INTERVALS: + print('downloading pair %s, interval %s' % (pair, tick_interval)) + data = exchange.get_ticker_history(pair, tick_interval) + filename = '{}-{}.json'.format(pair, tick_interval) + misc.file_dump_json(filename, data) diff --git a/freqtrade/tests/testdata/pairs.json b/freqtrade/tests/testdata/pairs.json new file mode 100644 index 000000000..c3e339b4c --- /dev/null +++ b/freqtrade/tests/testdata/pairs.json @@ -0,0 +1,23 @@ +[ + "BTC_ADA", + "BTC_BAT", + "BTC_DASH", + "BTC_ETC", + "BTC_ETH", + "BTC_GBYTE", + "BTC_LSK", + "BTC_LTC", + "BTC_NEO", + "BTC_NXT", + "BTC_POWR", + "BTC_STORJ", + "BTC_QTUM", + "BTC_WAVES", + "BTC_VTC", + "BTC_XLM", + "BTC_XMR", + "BTC_XVG", + "BTC_XRP", + "BTC_ZEC" +] + diff --git a/requirements.txt b/requirements.txt index 8a3c8b9cf..d37312268 100644 --- a/requirements.txt +++ b/requirements.txt @@ -1,7 +1,7 @@ python-bittrex==0.2.2 -SQLAlchemy==1.2.0 +SQLAlchemy==1.2.1 python-telegram-bot==9.0.0 -arrow==0.12.0 +arrow==0.12.1 cachetools==2.0.1 requests==2.18.4 urllib3==1.22 @@ -11,7 +11,7 @@ scikit-learn==0.19.1 scipy==1.0.0 jsonschema==2.6.0 numpy==1.14.0 -TA-Lib==0.4.14 +TA-Lib==0.4.16 pytest==3.3.2 pytest-mock==1.6.3 pytest-cov==2.5.1 @@ -19,7 +19,7 @@ hyperopt==0.1 # do not upgrade networkx before this is fixed https://github.com/hyperopt/hyperopt/issues/325 networkx==1.11 tabulate==0.8.2 -pymarketcap==3.3.145 +pymarketcap==3.3.148 # Required for plotting data #matplotlib==2.1.0 diff --git a/scripts/plot_dataframe.py b/scripts/plot_dataframe.py index ce636a4b5..f07033637 100755 --- a/scripts/plot_dataframe.py +++ b/scripts/plot_dataframe.py @@ -10,7 +10,7 @@ from freqtrade.misc import common_args_parser def plot_parse_args(args ): - parser = common_args_parser(args, 'Graph utility') + parser = common_args_parser(description='Graph utility') parser.add_argument( '-p', '--pair', help = 'What currency pair',