Merge branch 'develop' into develop

.gitignore

@@ -85,3 +85,5 @@ target/
 .venv
 .idea
 .vscode
+
+hyperopt_trials.pickle

.pylintrc

@@ -1,10 +1,10 @@
 [MASTER]
-extension-pkg-whitelist=numpy,talib
+extension-pkg-whitelist=numpy,talib,talib.abstract
 
 [BASIC]
 good-names=logger
 ignore=vendor
 
 [TYPECHECK]
-ignored-modules=numpy,talib
+ignored-modules=numpy,talib,talib.abstract
 

docs/backtesting.md

@@ -51,6 +51,21 @@ python3 ./freqtrade/main.py backtesting --realistic-simulation --live
 python3 ./freqtrade/main.py backtesting --datadir freqtrade/tests/testdata-20180101
 ```
 
+To update your testdata directory, or download into another testdata directory:
+```bash
+mkdir freqtrade/tests/testdata-20180113
+cp freqtrade/tests/testdata/pairs.json freqtrade/tests/testdata-20180113
+cd freqtrade/tests/testdata-20180113
+
+Possibly edit pairs.json file to include/exclude pairs
+
+python download_backtest_data.py -p pairs.json
+```
+
+The script will read your pairs.json file, and download ticker data
+into the current working directory.
+
+
 For help about backtesting usage, please refer to 
 [Backtesting commands](#backtesting-commands).
 

docs/faq.md

@@ -2,20 +2,70 @@
 
 #### I have waited 5 minutes, why hasn't the bot made any trades yet?!
 
-Depending on the buy strategy, the amount of whitelisted coins, the situation of the market etc, it can take up to hours to find good entry position for a trade. Be patient!
+Depending on the buy strategy, the amount of whitelisted coins, the 
+situation of the market etc, it can take up to hours to find good entry 
+position for a trade. Be patient!
 
 #### I have made 12 trades already, why is my total profit negative?!
 
-I understand your disappointment but unfortunately 12 trades is just not enough to say anything. If you run backtesting, you can see that our current algorithm does leave you on the plus side, but that is after thousands of trades and even there, you will be left with losses on specific coins that you have traded tens if not hundreds of times. We of course constantly aim to improve the bot but it will _always_ be a gamble, which should leave you with modest wins on monthly basis but you can't say much from few trades.
+I understand your disappointment but unfortunately 12 trades is just 
+not enough to say anything. If you run backtesting, you can see that our 
+current algorithm does leave you on the plus side, but that is after 
+thousands of trades and even there, you will be left with losses on 
+specific coins that you have traded tens if not hundreds of times. We 
+of course constantly aim to improve the bot but it will _always_ be a 
+gamble, which should leave you with modest wins on monthly basis but 
+you can't say much from few trades.
 
-#### I’d like to change the stake amount. Can I just stop the bot with /stop and then change the config.json and run it again?
+#### I’d like to change the stake amount. Can I just stop the bot with 
+/stop and then change the config.json and run it again?
 
-Not quite. Trades are persisted to a database but the configuration is currently only read when the bot is killed and restarted. `/stop` more like pauses. You can stop your bot, adjust settings and start it again.
+Not quite. Trades are persisted to a database but the configuration is 
+currently only read when the bot is killed and restarted. `/stop` more 
+like pauses. You can stop your bot, adjust settings and start it again.
 
 #### I want to improve the bot with a new strategy
 
-That's great. We have a nice backtesting and hyperoptimizing setup. See the tutorial [[here|Testing-new-strategies-with-Hyperopt]].
+That's great. We have a nice backtesting and hyperoptimizing setup. See 
+the tutorial [here|Testing-new-strategies-with-Hyperopt](https://github.com/gcarq/freqtrade/blob/develop/docs/bot-usage.md#hyperopt-commands).
 
-#### Is there a setting to only SELL the coins being held and not perform anymore BUYS?
+#### Is there a setting to only SELL the coins being held and not 
+perform anymore BUYS?
+
+You can use the `/forcesell all` command from Telegram. 
+
+### How many epoch do I need to get a good Hyperopt result?
+Per default Hyperopts without `-e` or `--epochs` parameter will only 
+run 100 epochs, means 100 evals of your triggers, guards, .... Too few 
+to find a great result (unless if you are very lucky), so you probably 
+have to run it for 10.000 or more. But it will take an eternity to 
+compute.
+
+We recommend you to run it at least 10.000 epochs:
+```bash
+python3 ./freqtrade/main.py hyperopt -e 10000
+```
+
+or if you want intermediate result to see
+```bash
+for i in {1..100}; do python3 ./freqtrade/main.py hyperopt -e 100; done
+```
+
+#### Why it is so long to run hyperopt?
+Finding a great Hyperopt results takes time. 
+
+If you wonder why it takes a while to find great hyperopt results
+
+This answer was written during the under the release 0.15.1, when we had
+:
+- 8 triggers
+- 9 guards: let's say we evaluate even 10 values from each
+- 1 stoploss calculation: let's say we want 10 values from that too to 
+be evaluated
+
+The following calculation is still very rough and not very precise
+but it will give the idea. With only these triggers and guards there is 
+already 8*10^9*10 evaluations. A roughly total of 80 billion evals. 
+Did you run 100 000 evals? Congrats, you've done roughly 1 / 100 000 th 
+of the search space.
 
-You can use the `/forcesell all` command from Telegram.

docs/installation.md

@@ -4,8 +4,6 @@ This page explains how to prepare your environment for running the bot.
 
 To understand how to set up the bot please read the [Bot Configuration](https://github.com/gcarq/freqtrade/blob/develop/docs/configuration.md) page.
 
-
-
 ## Table of Contents
 
 * [Table of Contents](#table-of-contents)
@@ -61,7 +59,6 @@ cp -n config.json.example config.json
 
 #### 1.5. Create your database file *(optional - the bot will create it if it is missing)*
 
-
 Production
 ```bash
 touch tradesv3.sqlite
@@ -127,7 +124,6 @@ docker run -d \
 
 If you are using `dry_run=True` it's not necessary to mount `tradesv3.sqlite`, but you can mount `tradesv3.dryrun.sqlite` if you plan to use the dry run mode with the param `--dry-run-db`.
 
-
 ### 6. Monitor your Docker instance
 
 You can then use the following commands to monitor and manage your container:
@@ -142,10 +138,8 @@ docker start freqtrade
 
 You do not need to rebuild the image for configuration changes, it will suffice to edit `config.json` and restart the container.
 
-
 ------
 
-
 ## Custom Installation
 
 We've included/collected install instructions for Ubuntu 16.04, MacOS, and Windows. These are guidelines and your success may vary with other distros.
@@ -153,7 +147,6 @@ We've included/collected install instructions for Ubuntu 16.04, MacOS, and Windo
 ### Requirements
 
 Click each one for install guide:
-
 * [Python 3.6.x](http://docs.python-guide.org/en/latest/starting/installation/), note the bot was not tested on Python >= 3.7.x
 * [pip](https://pip.pypa.io/en/stable/installing/)
 * [git](https://git-scm.com/book/en/v2/Getting-Started-Installing-Git)
@@ -168,7 +161,7 @@ Click each one for install guide:
 ```bash
 sudo add-apt-repository ppa:jonathonf/python-3.6
 sudo apt-get update
-sudo apt-get install python3.6 python3.6-venv build-essential autoconf libtool pkg-config make wget git
+sudo apt-get install python3.6 python3.6-venv python3.6-dev build-essential autoconf libtool pkg-config make wget git
 ```
 
 #### 2. Install TA-Lib
@@ -260,7 +253,6 @@ Optionally checkout the develop branch:
 git checkout develop
 ```
 
-
 ### Windows
 
 We recommend that Windows users use [Docker](#docker) as this will work

docs/sql_cheatsheet.md

@@ -67,6 +67,18 @@ SET is_open=0, close_date='2017-12-20 03:08:45.103418', close_rate=0.19638016, c
 WHERE id=31;
 ```
 
+## Insert manually a new trade
+
+```sql
+INSERT 
+INTO trades (exchange, pair, is_open, fee, open_rate, stake_amount, amount, open_date) 
+VALUES ('BITTREX', 'BTC_<COIN>', 1, 0.0025, <open_rate>, <stake_amount>, <amount>, '<datetime>')
+```
+
+**Example:**
+```sql
+INSERT INTO trades (exchange, pair, is_open, fee, open_rate, stake_amount, amount, open_date) VALUES ('BITTREX', 'BTC_ETC', 1, 0.0025, 0.00258580, 0.002, 0.7715262081, '2017-11-28 12:44:24.000000')
+```
 
 ## Fix wrong fees in the table
 If your DB was created before 

freqtrade/__init__.py

@@ -1,5 +1,5 @@
 """ FreqTrade bot """
-__version__ = '0.14.3'
+__version__ = '0.15.1'
 
 
 class DependencyException(BaseException):

freqtrade/analyze.py

@@ -4,14 +4,14 @@ Functions to analyze ticker data with indicators and produce buy and sell signal
 import logging
 from datetime import timedelta
 from enum import Enum
-from typing import List, Dict
+from typing import Dict, List
 
 import arrow
 import talib.abstract as ta
 from pandas import DataFrame, to_datetime
 
-from freqtrade.exchange import get_ticker_history
 import freqtrade.vendor.qtpylib.indicators as qtpylib
+from freqtrade.exchange import get_ticker_history
 
 logger = logging.getLogger(__name__)
 

freqtrade/exchange/bittrex.py

@@ -1,7 +1,9 @@
 import logging
-from typing import List, Dict, Optional
+import requests
+from typing import Dict, List, Optional
 
-from bittrex.bittrex import Bittrex as _Bittrex, API_V2_0, API_V1_1
+from bittrex.bittrex import Bittrex as _Bittrex
+from bittrex.bittrex import API_V1_1, API_V2_0
 from requests.exceptions import ContentDecodingError
 
 from freqtrade import OperationalException
@@ -13,6 +15,20 @@ _API: _Bittrex = None
 _API_V2: _Bittrex = None
 _EXCHANGE_CONF: dict = {}
 
+# API socket timeout
+API_TIMEOUT = 60
+
+
+def custom_requests(request_url, apisign):
+    """
+    Set timeout for requests
+    """
+    return requests.get(
+        request_url,
+        headers={"apisign": apisign},
+        timeout=API_TIMEOUT
+    ).json()
+
 
 class Bittrex(Exchange):
     """
@@ -31,12 +47,14 @@ class Bittrex(Exchange):
             api_secret=_EXCHANGE_CONF['secret'],
             calls_per_second=1,
             api_version=API_V1_1,
+            dispatch=custom_requests
         )
         _API_V2 = _Bittrex(
             api_key=_EXCHANGE_CONF['key'],
             api_secret=_EXCHANGE_CONF['secret'],
             calls_per_second=1,
             api_version=API_V2_0,
+            dispatch=custom_requests
         )
         self.cached_ticker = {}
 
@@ -105,10 +123,8 @@ class Bittrex(Exchange):
                     message=data['message'],
                     pair=pair))
 
-            if not data.get('result') \
+            if not data.get('result') or\
-                    or not data['result'].get('Bid') \
+                    not all(key in data.get('result', {}) for key in ['Bid', 'Ask', 'Last']):
-                    or not data['result'].get('Ask') \
-                    or not data['result'].get('Last'):
                 raise ContentDecodingError('{message} params=({pair})'.format(
                     message='Got invalid response from bittrex',
                     pair=pair))

freqtrade/exchange/interface.py

@@ -1,5 +1,5 @@
 from abc import ABC, abstractmethod
-from typing import List, Dict, Optional
+from typing import Dict, List, Optional
 
 
 class Exchange(ABC):

freqtrade/fiat_convert.py

@@ -1,5 +1,6 @@
 import logging
 import time
+
 from pymarketcap import Pymarketcap
 
 logger = logging.getLogger(__name__)

freqtrade/main.py

@@ -5,20 +5,20 @@ import logging
 import sys
 import time
 import traceback
-import arrow
 from datetime import datetime
-from typing import Dict, Optional, List
+from typing import Dict, List, Optional
 
+import arrow
 import requests
 from cachetools import cached, TTLCache
 
-from freqtrade import __version__, exchange, persistence, rpc, DependencyException, \
+from freqtrade import (DependencyException, OperationalException, __version__,
-    OperationalException
+                       exchange, persistence, rpc)
-from freqtrade.analyze import get_signal, SignalType
+from freqtrade.analyze import SignalType, get_signal
-from freqtrade.misc import State, get_state, update_state, parse_args, throttle, \
-    load_config
-from freqtrade.persistence import Trade
 from freqtrade.fiat_convert import CryptoToFiatConverter
+from freqtrade.misc import (State, get_state, load_config, parse_args,
+                            throttle, update_state)
+from freqtrade.persistence import Trade
 
 logger = logging.getLogger('freqtrade')
 
@@ -179,11 +179,11 @@ def execute_sell(trade: Trade, limit: float) -> None:
     profit_trade = trade.calc_profit(rate=limit)
 
     message = '*{exchange}:* Selling [{pair}]({pair_url}) with limit `{limit:.8f}`'.format(
-                    exchange=trade.exchange,
+        exchange=trade.exchange,
-                    pair=trade.pair.replace('_', '/'),
+        pair=trade.pair.replace('_', '/'),
-                    pair_url=exchange.get_pair_detail_url(trade.pair),
+        pair_url=exchange.get_pair_detail_url(trade.pair),
-                    limit=limit
+        limit=limit
-                )
+    )
 
     # For regular case, when the configuration exists
     if 'stake_currency' in _CONF and 'fiat_display_currency' in _CONF:
@@ -195,12 +195,12 @@ def execute_sell(trade: Trade, limit: float) -> None:
         )
         message += '` ({gain}: {profit_percent:.2f}%, {profit_coin:.8f} {coin}`' \
                    '` / {profit_fiat:.3f} {fiat})`'.format(
-                        gain="profit" if fmt_exp_profit > 0 else "loss",
+                       gain="profit" if fmt_exp_profit > 0 else "loss",
-                        profit_percent=fmt_exp_profit,
+                       profit_percent=fmt_exp_profit,
-                        profit_coin=profit_trade,
+                       profit_coin=profit_trade,
-                        coin=_CONF['stake_currency'],
+                       coin=_CONF['stake_currency'],
-                        profit_fiat=profit_fiat,
+                       profit_fiat=profit_fiat,
-                        fiat=_CONF['fiat_display_currency'],
+                       fiat=_CONF['fiat_display_currency'],
                    )
     # Because telegram._forcesell does not have the configuration
     # Ignore the FIAT value and does not show the stake_currency as well

freqtrade/misc.py

@@ -3,10 +3,11 @@ import enum
 import json
 import logging
 import time
-from typing import Any, Callable, List, Dict
+import os
+from typing import Any, Callable, Dict, List
 
-from jsonschema import validate, Draft4Validator
+from jsonschema import Draft4Validator, validate
-from jsonschema.exceptions import best_match, ValidationError
+from jsonschema.exceptions import ValidationError, best_match
 from wrapt import synchronized
 
 from freqtrade import __version__
@@ -14,6 +15,11 @@ from freqtrade import __version__
 logger = logging.getLogger(__name__)
 
 
+def file_dump_json(filename, data):
+    with open(filename, 'w') as fp:
+        json.dump(data, fp)
+
+
 class State(enum.Enum):
     RUNNING = 0
     STOPPED = 1
@@ -57,8 +63,8 @@ def load_config(path: str) -> Dict:
     try:
         validate(conf, CONF_SCHEMA)
         return conf
-    except ValidationError as ex:
+    except ValidationError as exception:
-        logger.fatal('Invalid configuration. See config.json.example. Reason: %s', ex)
+        logger.fatal('Invalid configuration. See config.json.example. Reason: %s', exception)
         raise ValidationError(
             best_match(Draft4Validator(CONF_SCHEMA).iter_errors(conf)).message
         )
@@ -81,7 +87,7 @@ def throttle(func: Callable[..., Any], min_secs: float, *args, **kwargs) -> Any:
     return result
 
 
-def parse_args_common(args: List[str], description: str):
+def common_args_parser(description: str):
     """
     Parses given common arguments and returns them as a parsed object.
     """
@@ -117,11 +123,11 @@ def parse_args(args: List[str], description: str):
     Parses given arguments and returns an argparse Namespace instance.
     Returns None if a sub command has been selected and executed.
     """
-    parser = parse_args_common(args, description)
+    parser = common_args_parser(description)
     parser.add_argument(
         '--dry-run-db',
-        help='Force dry run to use a local DB "tradesv3.dry_run.sqlite" instead of memory DB. Work only if dry_run is \
+        help='Force dry run to use a local DB "tradesv3.dry_run.sqlite" \
-             enabled.',  # noqa
+             instead of memory DB. Work only if dry_run is enabled.',
         action='store_true',
         dest='dry_run_db',
     )
@@ -129,13 +135,14 @@ def parse_args(args: List[str], description: str):
         '-dd', '--datadir',
         help='path to backtest data (default freqdata/tests/testdata',
         dest='datadir',
-        default='freqtrade/tests/testdata',
+        default=os.path.join('freqtrade', 'tests', 'testdata'),
         type=str,
         metavar='PATH',
     )
     parser.add_argument(
         '--dynamic-whitelist',
-        help='dynamically generate and update whitelist based on 24h BaseVolume (Default 20 currencies)',  # noqa
+        help='dynamically generate and update whitelist \
+             based on 24h BaseVolume (Default 20 currencies)',  # noqa
         dest='dynamic_whitelist',
         const=20,
         type=int,

freqtrade/optimize/backtesting.py

@@ -1,20 +1,19 @@
 # pragma pylint: disable=missing-docstring,W0212
 
-
 import logging
-from typing import Tuple, Dict
+from typing import Dict, Tuple
 
 import arrow
 from pandas import DataFrame, Series
 from tabulate import tabulate
 
+import freqtrade.misc as misc
+import freqtrade.optimize as optimize
 from freqtrade import exchange
 from freqtrade.analyze import populate_buy_trend, populate_sell_trend
 from freqtrade.exchange import Bittrex
 from freqtrade.main import min_roi_reached
-import freqtrade.misc as misc
 from freqtrade.optimize import preprocess
-import freqtrade.optimize as optimize
 from freqtrade.persistence import Trade
 
 logger = logging.getLogger(__name__)
@@ -122,20 +121,20 @@ def backtest(stake_amount: float, processed: Dict[str, DataFrame],
                 if min_roi_reached(trade, row2.close, row2.date) or \
                     (row2.sell == 1 and use_sell_signal) or \
                         current_profit_percent <= stoploss:
-                        current_profit_btc = trade.calc_profit(rate=row2.close)
+                    current_profit_btc = trade.calc_profit(rate=row2.close)
-                        lock_pair_until = row2.Index
+                    lock_pair_until = row2.Index
 
-                        trades.append(
+                    trades.append(
-                            (
+                        (
-                                pair,
+                            pair,
-                                current_profit_percent,
+                            current_profit_percent,
-                                current_profit_btc,
+                            current_profit_btc,
-                                row2.Index - row.Index,
+                            row2.Index - row.Index,
-                                current_profit_btc > 0,
+                            current_profit_btc > 0,
-                                current_profit_btc < 0
+                            current_profit_btc < 0
-                            )
                         )
-                        break
+                    )
+                    break
     labels = ['currency', 'profit_percent', 'profit_BTC', 'duration', 'profit', 'loss']
     return DataFrame.from_records(trades, columns=labels)
 
@@ -193,6 +192,6 @@ def start(args):
         use_sell_signal=config.get('experimental', {}).get('use_sell_signal', False)
     )
     logger.info(
-        '\n==================================== BACKTESTING REPORT ====================================\n%s', # noqa
+        '\n==================================== BACKTESTING REPORT ====================================\n%s',  # noqa
         generate_text_table(data, results, config['stake_currency'], args.ticker_interval)
     )

freqtrade/optimize/hyperopt.py

@@ -4,14 +4,18 @@
 import json
 import logging
 import sys
+import pickle
+import signal
+import os
 from functools import reduce
 from math import exp
 from operator import itemgetter
 
-from hyperopt import fmin, tpe, hp, Trials, STATUS_OK, STATUS_FAIL, space_eval
+from hyperopt import STATUS_FAIL, STATUS_OK, Trials, fmin, hp, space_eval, tpe
 from hyperopt.mongoexp import MongoTrials
 from pandas import DataFrame
 
+from freqtrade import main  # noqa
 from freqtrade import exchange, optimize
 from freqtrade.exchange import Bittrex
 from freqtrade.misc import load_config
@@ -27,7 +31,7 @@ logger = logging.getLogger(__name__)
 
 # set TARGET_TRADES to suit your number concurrent trades so its realistic to 20days of data
 TARGET_TRADES = 1100
-TOTAL_TRIES = None
+TOTAL_TRIES = 0
 _CURRENT_TRIES = 0
 CURRENT_BEST_LOSS = 100
 
@@ -43,6 +47,10 @@ EXPECTED_MAX_PROFIT = 3.85
 PROCESSED = None  # optimize.preprocess(optimize.load_data())
 OPTIMIZE_CONFIG = hyperopt_optimize_conf()
 
+# Hyperopt Trials
+TRIALS_FILE = os.path.join('freqtrade', 'optimize', 'hyperopt_trials.pickle')
+TRIALS = Trials()
+
 # Monkey patch config
 from freqtrade import main  # noqa
 main._CONF = OPTIMIZE_CONFIG
@@ -95,10 +103,30 @@ SPACE = {
         {'type': 'stochf_cross'},
         {'type': 'ht_sine'},
     ]),
-    'stoploss': hp.quniform('stoploss', -30, -2, 1),
+    'stoploss': hp.uniform('stoploss', -0.5, -0.02),
 }
 
 
+def save_trials(trials, trials_path=TRIALS_FILE):
+    """Save hyperopt trials to file"""
+    logger.info('Saving Trials to \'{}\''.format(trials_path))
+    pickle.dump(trials, open(trials_path, 'wb'))
+
+
+def read_trials(trials_path=TRIALS_FILE):
+    """Read hyperopt trials file"""
+    logger.info('Reading Trials from \'{}\''.format(trials_path))
+    trials = pickle.load(open(trials_path, 'rb'))
+    os.remove(trials_path)
+    return trials
+
+
+def log_trials_result(trials):
+    vals = json.dumps(trials.best_trial['misc']['vals'], indent=4)
+    results = trials.best_trial['result']['result']
+    logger.info('Best result:\n%s\nwith values:\n%s', results, vals)
+
+
 def log_results(results):
     """ log results if it is better than any previous evaluation """
     global CURRENT_BEST_LOSS
@@ -167,7 +195,7 @@ def format_results(results: DataFrame):
                 results.profit_percent.mean() * 100.0,
                 results.profit_BTC.sum(),
                 results.duration.mean() * 5,
-            )
+    )
 
 
 def buy_strategy_generator(params):
@@ -216,7 +244,8 @@ def buy_strategy_generator(params):
 
 
 def start(args):
-    global TOTAL_TRIES, PROCESSED, SPACE
+    global TOTAL_TRIES, PROCESSED, SPACE, TRIALS, _CURRENT_TRIES
+
     TOTAL_TRIES = args.epochs
 
     exchange._API = Bittrex({'key': '', 'secret': ''})
@@ -238,9 +267,19 @@ def start(args):
         logger.info('Start scripts/start-mongodb.sh and start-hyperopt-worker.sh manually!')
 
         db_name = 'freqtrade_hyperopt'
-        trials = MongoTrials('mongo://127.0.0.1:1234/{}/jobs'.format(db_name), exp_key='exp1')
+        TRIALS = MongoTrials('mongo://127.0.0.1:1234/{}/jobs'.format(db_name), exp_key='exp1')
     else:
-        trials = Trials()
+        logger.info('Preparing Trials..')
+        signal.signal(signal.SIGINT, signal_handler)
+        # read trials file if we have one
+        if os.path.exists(TRIALS_FILE):
+            TRIALS = read_trials()
+
+            _CURRENT_TRIES = len(TRIALS.results)
+            TOTAL_TRIES = TOTAL_TRIES + _CURRENT_TRIES
+            logger.info(
+                'Continuing with trials. Current: {}, Total: {}'
+                .format(_CURRENT_TRIES, TOTAL_TRIES))
 
     try:
         best_parameters = fmin(
@@ -248,10 +287,10 @@ def start(args):
             space=SPACE,
             algo=tpe.suggest,
             max_evals=TOTAL_TRIES,
-            trials=trials
+            trials=TRIALS
         )
 
-        results = sorted(trials.results, key=itemgetter('loss'))
+        results = sorted(TRIALS.results, key=itemgetter('loss'))
         best_result = results[0]['result']
 
     except ValueError:
@@ -265,3 +304,15 @@ def start(args):
 
     logger.info('Best parameters:\n%s', json.dumps(best_parameters, indent=4))
     logger.info('Best Result:\n%s', best_result)
+
+    # Store trials result to file to resume next time
+    save_trials(TRIALS)
+
+
+def signal_handler(sig, frame):
+    """Hyperopt SIGINT handler"""
+    logger.info('Hyperopt received {}'.format(signal.Signals(sig).name))
+
+    save_trials(TRIALS)
+    log_trials_result(TRIALS)
+    sys.exit(0)

freqtrade/optimize/hyperopt_conf.py

@@ -15,10 +15,10 @@ def hyperopt_optimize_conf() -> dict:
         'stake_currency': 'BTC',
         'stake_amount': 0.01,
         "minimal_roi": {
-            '40':  0.0,
+            '40': 0.0,
-            '30':  0.01,
+            '30': 0.01,
-            '20':  0.02,
+            '20': 0.02,
-            '0':  0.04,
+            '0': 0.04,
         },
         'stoploss': -0.10,
         "bid_strategy": {

freqtrade/persistence.py

@@ -1,10 +1,11 @@
 import logging
 from datetime import datetime
 from decimal import Decimal, getcontext
-from typing import Optional, Dict
+from typing import Dict, Optional
 
 import arrow
-from sqlalchemy import Boolean, Column, DateTime, Float, Integer, String, create_engine
+from sqlalchemy import (Boolean, Column, DateTime, Float, Integer, String,
+                        create_engine)
 from sqlalchemy.engine import Engine
 from sqlalchemy.ext.declarative import declarative_base
 from sqlalchemy.orm.scoping import scoped_session

freqtrade/rpc/telegram.py

@@ -1,21 +1,21 @@
 import logging
 import re
+from datetime import datetime, timedelta
 from decimal import Decimal
-from datetime import timedelta, datetime
+from typing import Any, Callable
-from typing import Callable, Any
 
 import arrow
 from pandas import DataFrame
 from sqlalchemy import and_, func, text
 from tabulate import tabulate
-from telegram import ParseMode, Bot, Update, ReplyKeyboardMarkup
+from telegram import Bot, ParseMode, ReplyKeyboardMarkup, Update
 from telegram.error import NetworkError, TelegramError
 from telegram.ext import CommandHandler, Updater
 
-from freqtrade import exchange, __version__
+from freqtrade import __version__, exchange
-from freqtrade.misc import get_state, State, update_state
-from freqtrade.persistence import Trade
 from freqtrade.fiat_convert import CryptoToFiatConverter
+from freqtrade.misc import State, get_state, update_state
+from freqtrade.persistence import Trade
 
 # Remove noisy log messages
 logging.getLogger('requests.packages.urllib3').setLevel(logging.INFO)
@@ -255,7 +255,7 @@ def _daily(bot: Bot, update: Update) -> None:
                 ),
                 symbol=_CONF['fiat_display_currency']
             )
-         ]
+        ]
         for key, value in profit_days.items()
     ]
     stats = tabulate(stats,

freqtrade/tests/conftest.py

@@ -2,10 +2,10 @@
 from datetime import datetime
 from unittest.mock import MagicMock
 
-import pytest
 import arrow
+import pytest
 from jsonschema import validate
-from telegram import Message, Chat, Update
+from telegram import Chat, Message, Update
 
 from freqtrade.misc import CONF_SCHEMA
 
@@ -20,10 +20,10 @@ def default_conf():
         "fiat_display_currency": "USD",
         "dry_run": True,
         "minimal_roi": {
-            "40":  0.0,
+            "40": 0.0,
-            "30":  0.01,
+            "30": 0.01,
-            "20":  0.02,
+            "20": 0.02,
-            "0":  0.04
+            "0": 0.04
         },
         "stoploss": -0.10,
         "unfilledtimeout": 600,

freqtrade/tests/exchange/test_exchange_bittrex.py

@@ -1,9 +1,325 @@
 # pragma pylint: disable=missing-docstring,C0103
 
 import pytest
+from unittest.mock import MagicMock
 from requests.exceptions import ContentDecodingError
 
-from freqtrade.exchange import Bittrex
+from freqtrade.exchange.bittrex import Bittrex
+import freqtrade.exchange.bittrex as btx
+
+
+# Eat this flake8
+# +------------------+
+# |  bittrex.Bittrex |
+# +------------------+
+#         |
+#       (mock Fake_bittrex)
+#         |
+# +-----------------------------+
+# |  freqtrade.exchange.Bittrex |
+# +-----------------------------+
+# Call into Bittrex will flow up to the
+# external package bittrex.Bittrex.
+# By inserting a mock, we redirect those
+# calls.
+# The faked bittrex API is called just 'fb'
+# The freqtrade.exchange.Bittrex is a
+# wrapper, and is called 'wb'
+
+
+def _stub_config():
+    return {'key': '',
+            'secret': ''}
+
+
+class FakeBittrex():
+    def __init__(self, success=True):
+        self.success = True  # Believe in yourself
+        self.result = None
+        self.get_ticker_call_count = 0
+        # This is really ugly, doing side-effect during instance creation
+        # But we're allowed to in testing-code
+        btx._API = MagicMock()
+        btx._API.buy_limit = self.fake_buysell_limit
+        btx._API.sell_limit = self.fake_buysell_limit
+        btx._API.get_balance = self.fake_get_balance
+        btx._API.get_balances = self.fake_get_balances
+        btx._API.get_ticker = self.fake_get_ticker
+        btx._API.get_order = self.fake_get_order
+        btx._API.cancel = self.fake_cancel_order
+        btx._API.get_markets = self.fake_get_markets
+        btx._API.get_market_summaries = self.fake_get_market_summaries
+        btx._API_V2 = MagicMock()
+        btx._API_V2.get_candles = self.fake_get_candles
+        btx._API_V2.get_wallet_health = self.fake_get_wallet_health
+
+    def fake_buysell_limit(self, pair, amount, limit):
+        return {'success': self.success,
+                'result': {'uuid': '1234'},
+                'message': 'barter'}
+
+    def fake_get_balance(self, cur):
+        return {'success': self.success,
+                'result': {'Balance': 1234},
+                'message': 'unbalanced'}
+
+    def fake_get_balances(self):
+        return {'success': self.success,
+                'result': [{'BTC_ETH': 1234}],
+                'message': 'no balances'}
+
+    def fake_get_ticker(self, pair):
+        self.get_ticker_call_count += 1
+        return self.result or {'success': self.success,
+                               'result': {'Bid': 1, 'Ask': 1, 'Last': 1},
+                               'message': 'NO_API_RESPONSE'}
+
+    def fake_get_candles(self, pair, interval):
+        return self.result or {'success': self.success,
+                               'result': [{'C': 0, 'V': 0, 'O': 0, 'H': 0, 'L': 0, 'T': 0}],
+                               'message': 'candles lit'}
+
+    def fake_get_order(self, uuid):
+        return {'success': self.success,
+                'result': {'OrderUuid': 'ABC123',
+                           'Type': 'Type',
+                           'Exchange': 'BTC_ETH',
+                           'Opened': True,
+                           'PricePerUnit': 1,
+                           'Quantity': 1,
+                           'QuantityRemaining': 1,
+                           'Closed': True
+                           },
+                'message': 'lost'}
+
+    def fake_cancel_order(self, uuid):
+        return self.result or {'success': self.success,
+                               'message': 'no such order'}
+
+    def fake_get_markets(self):
+        return self.result or {'success': self.success,
+                               'message': 'market gone',
+                               'result': [{'MarketName': '-_'}]}
+
+    def fake_get_market_summaries(self):
+        return self.result or {'success': self.success,
+                               'message': 'no summary',
+                               'result': ['sum']}
+
+    def fake_get_wallet_health(self):
+        return self.result or {'success': self.success,
+                               'message': 'bad health',
+                               'result': [{'Health': {'Currency': 'BTC_ETH',
+                                                      'IsActive': True,
+                                                      'LastChecked': 0},
+                                           'Currency': {'Notice': True}}]}
+
+
+# The freqtrade.exchange.bittrex is called wrap_bittrex
+# to not confuse naming with bittrex.bittrex
+def make_wrap_bittrex():
+    conf = _stub_config()
+    wb = btx.Bittrex(conf)
+    return wb
+
+
+def test_exchange_bittrex_class():
+    conf = _stub_config()
+    b = Bittrex(conf)
+    assert isinstance(b, Bittrex)
+    slots = dir(b)
+    for name in ['fee', 'buy', 'sell', 'get_balance', 'get_balances',
+                 'get_ticker', 'get_ticker_history', 'get_order',
+                 'cancel_order', 'get_pair_detail_url', 'get_markets',
+                 'get_market_summaries', 'get_wallet_health']:
+        assert name in slots
+        # FIX: ensure that the slot is also a method in the class
+        # getattr(b, name) => bound method Bittrex.buy
+        # type(getattr(b, name)) => class 'method'
+
+
+def test_exchange_bittrex_fee():
+    fee = Bittrex.fee.__get__(Bittrex)
+    assert fee >= 0 and fee < 0.1  # Fee is 0-10 %
+
+
+def test_exchange_bittrex_buy_good(mocker):
+    wb = make_wrap_bittrex()
+    fb = FakeBittrex()
+    uuid = wb.buy('BTC_ETH', 1, 1)
+    assert uuid == fb.fake_buysell_limit(1, 2, 3)['result']['uuid']
+
+    fb.success = False
+    with pytest.raises(btx.OperationalException, match=r'barter.*'):
+        wb.buy('BAD', 1, 1)
+
+
+def test_exchange_bittrex_sell_good(mocker):
+    wb = make_wrap_bittrex()
+    fb = FakeBittrex()
+    uuid = wb.sell('BTC_ETH', 1, 1)
+    assert uuid == fb.fake_buysell_limit(1, 2, 3)['result']['uuid']
+
+    fb.success = False
+    with pytest.raises(btx.OperationalException, match=r'barter.*'):
+        uuid = wb.sell('BAD', 1, 1)
+
+
+def test_exchange_bittrex_get_balance(mocker):
+    wb = make_wrap_bittrex()
+    fb = FakeBittrex()
+    bal = wb.get_balance('BTC_ETH')
+    assert bal == fb.fake_get_balance(1)['result']['Balance']
+
+    fb.success = False
+    with pytest.raises(btx.OperationalException, match=r'unbalanced'):
+        wb.get_balance('BTC_ETH')
+
+
+def test_exchange_bittrex_get_balances():
+    wb = make_wrap_bittrex()
+    fb = FakeBittrex()
+    bals = wb.get_balances()
+    assert bals == fb.fake_get_balances()['result']
+
+    fb.success = False
+    with pytest.raises(btx.OperationalException, match=r'no balances'):
+        wb.get_balances()
+
+
+def test_exchange_bittrex_get_ticker():
+    wb = make_wrap_bittrex()
+    fb = FakeBittrex()
+
+    # Poll ticker, which updates the cache
+    tick = wb.get_ticker('BTC_ETH')
+    for x in ['bid', 'ask', 'last']:
+        assert x in tick
+    # Ensure the side-effect was made (update the ticker cache)
+    assert 'BTC_ETH' in wb.cached_ticker.keys()
+
+    # taint the cache, so we can recognize the cache wall utilized
+    wb.cached_ticker['BTC_ETH']['bid'] = 1234
+    # Poll again, getting the cached result
+    fb.get_ticker_call_count = 0
+    tick = wb.get_ticker('BTC_ETH', False)
+    # Ensure the result was from the cache, and that we didn't call exchange
+    assert wb.cached_ticker['BTC_ETH']['bid'] == 1234
+    assert fb.get_ticker_call_count == 0
+
+
+def test_exchange_bittrex_get_ticker_bad():
+    wb = make_wrap_bittrex()
+    fb = FakeBittrex()
+    fb.result = {'success': True,
+                 'result': {'Bid': 1, 'Ask': 0}}  # incomplete result
+
+    with pytest.raises(ContentDecodingError, match=r'.*Got invalid response from bittrex params.*'):
+        wb.get_ticker('BTC_ETH')
+    fb.result = {'success': False,
+                 'message': 'gone bad'
+                 }
+    with pytest.raises(btx.OperationalException, match=r'.*gone bad.*'):
+        wb.get_ticker('BTC_ETH')
+
+    fb.result = {'success': True,
+                 'result': {}}  # incomplete result
+    with pytest.raises(ContentDecodingError, match=r'.*Got invalid response from bittrex params.*'):
+        wb.get_ticker('BTC_ETH')
+    fb.result = {'success': False,
+                 'message': 'gone bad'
+                 }
+    with pytest.raises(btx.OperationalException, match=r'.*gone bad.*'):
+        wb.get_ticker('BTC_ETH')
+
+
+def test_exchange_bittrex_get_ticker_history_one():
+    wb = make_wrap_bittrex()
+    FakeBittrex()
+    assert wb.get_ticker_history('BTC_ETH', 1)
+
+
+def test_exchange_bittrex_get_ticker_history():
+    wb = make_wrap_bittrex()
+    fb = FakeBittrex()
+    assert wb.get_ticker_history('BTC_ETH', 5)
+    with pytest.raises(ValueError, match=r'.*Cannot parse tick_interval.*'):
+        wb.get_ticker_history('BTC_ETH', 2)
+
+    fb.success = False
+    with pytest.raises(btx.OperationalException, match=r'candles lit.*'):
+        wb.get_ticker_history('BTC_ETH', 5)
+
+    fb.success = True
+    with pytest.raises(ContentDecodingError, match=r'.*Got invalid response from bittrex.*'):
+        fb.result = {'bad': 0}
+        wb.get_ticker_history('BTC_ETH', 5)
+
+    with pytest.raises(ContentDecodingError, match=r'.*Required property C not present.*'):
+        fb.result = {'success': True,
+                     'result': [{'V': 0, 'O': 0, 'H': 0, 'L': 0, 'T': 0}],  # close is missing
+                     'message': 'candles lit'}
+        wb.get_ticker_history('BTC_ETH', 5)
+
+
+def test_exchange_bittrex_get_order():
+    wb = make_wrap_bittrex()
+    fb = FakeBittrex()
+    order = wb.get_order('someUUID')
+    assert order['id'] == 'ABC123'
+    fb.success = False
+    with pytest.raises(btx.OperationalException, match=r'lost'):
+        wb.get_order('someUUID')
+
+
+def test_exchange_bittrex_cancel_order():
+    wb = make_wrap_bittrex()
+    fb = FakeBittrex()
+    wb.cancel_order('someUUID')
+    with pytest.raises(btx.OperationalException, match=r'no such order'):
+        fb.success = False
+        wb.cancel_order('someUUID')
+    # Note: this can be a bug in exchange.bittrex._validate_response
+    with pytest.raises(KeyError):
+        fb.result = {'success': False}  # message is missing!
+        wb.cancel_order('someUUID')
+    with pytest.raises(btx.OperationalException, match=r'foo'):
+        fb.result = {'success': False, 'message': 'foo'}
+        wb.cancel_order('someUUID')
+
+
+def test_exchange_get_pair_detail_url():
+    wb = make_wrap_bittrex()
+    assert wb.get_pair_detail_url('BTC_ETH')
+
+
+def test_exchange_get_markets():
+    wb = make_wrap_bittrex()
+    fb = FakeBittrex()
+    x = wb.get_markets()
+    assert x == ['__']
+    with pytest.raises(btx.OperationalException, match=r'market gone'):
+        fb.success = False
+        wb.get_markets()
+
+
+def test_exchange_get_market_summaries():
+    wb = make_wrap_bittrex()
+    fb = FakeBittrex()
+    assert ['sum'] == wb.get_market_summaries()
+    with pytest.raises(btx.OperationalException, match=r'no summary'):
+        fb.success = False
+        wb.get_market_summaries()
+
+
+def test_exchange_get_wallet_health():
+    wb = make_wrap_bittrex()
+    fb = FakeBittrex()
+    x = wb.get_wallet_health()
+    assert x[0]['Currency'] == 'BTC_ETH'
+    with pytest.raises(btx.OperationalException, match=r'bad health'):
+        fb.success = False
+        wb.get_wallet_health()
 
 
 def test_validate_response_success():

freqtrade/tests/optimize/test_hyperopt.py

@@ -1,7 +1,6 @@
 # pragma pylint: disable=missing-docstring,W0212,C0103
-
 from freqtrade.optimize.hyperopt import calculate_loss, TARGET_TRADES, EXPECTED_MAX_PROFIT, start, \
-    log_results
+    log_results, save_trials, read_trials
 
 
 def test_loss_calculation_prefer_correct_trade_count():
@@ -27,16 +26,37 @@ def test_loss_calculation_has_limited_profit():
 
 
 def create_trials(mocker):
+    """
+    When creating trials, mock the hyperopt Trials so that *by default*
+      - we don't create any pickle'd files in the filesystem
+      - we might have a pickle'd file so make sure that we return
+        false when looking for it
+    """
+    mocker.patch('freqtrade.optimize.hyperopt.TRIALS_FILE',
+                 return_value='freqtrade/tests/optimize/ut_trials.pickle')
+    mocker.patch('freqtrade.optimize.hyperopt.os.path.exists',
+                 return_value=False)
+    mocker.patch('freqtrade.optimize.hyperopt.save_trials',
+                 return_value=None)
+    mocker.patch('freqtrade.optimize.hyperopt.read_trials',
+                 return_value=None)
+    mocker.patch('freqtrade.optimize.hyperopt.os.remove',
+                 return_value=True)
     return mocker.Mock(
         results=[{
             'loss': 1,
-            'result': 'foo'
+            'result': 'foo',
-        }]
+            'status': 'ok'
+        }],
+        best_trial={'misc': {'vals': {'adx': 999}}}
     )
 
 
 def test_start_calls_fmin(mocker):
-    mocker.patch('freqtrade.optimize.hyperopt.Trials', return_value=create_trials(mocker))
+    trials = create_trials(mocker)
+    mocker.patch('freqtrade.optimize.hyperopt.TRIALS', return_value=trials)
+    mocker.patch('freqtrade.optimize.hyperopt.sorted',
+                 return_value=trials.results)
     mocker.patch('freqtrade.optimize.preprocess')
     mocker.patch('freqtrade.optimize.load_data')
     mock_fmin = mocker.patch('freqtrade.optimize.hyperopt.fmin', return_value={})
@@ -87,20 +107,20 @@ def test_no_log_if_loss_does_not_improve(mocker):
 
 def test_fmin_best_results(mocker, caplog):
     fmin_result = {
-      "adx": 1,
+        "adx": 1,
-      "adx-value": 15.0,
+        "adx-value": 15.0,
-      "fastd": 1,
+        "fastd": 1,
-      "fastd-value": 40.0,
+        "fastd-value": 40.0,
-      "green_candle": 1,
+        "green_candle": 1,
-      "mfi": 0,
+        "mfi": 0,
-      "over_sar": 0,
+        "over_sar": 0,
-      "rsi": 1,
+        "rsi": 1,
-      "rsi-value": 37.0,
+        "rsi-value": 37.0,
-      "trigger": 2,
+        "trigger": 2,
-      "uptrend_long_ema": 1,
+        "uptrend_long_ema": 1,
-      "uptrend_short_ema": 0,
+        "uptrend_short_ema": 0,
-      "uptrend_sma": 0,
+        "uptrend_sma": 0,
-      "stoploss": -10,
+        "stoploss": -0.1,
     }
 
     mocker.patch('freqtrade.optimize.hyperopt.MongoTrials', return_value=create_trials(mocker))
@@ -117,7 +137,7 @@ def test_fmin_best_results(mocker, caplog):
         '"green_candle": {\n        "enabled": true\n    },',
         '"mfi": {\n        "enabled": false\n    },',
         '"trigger": {\n        "type": "ao_cross_zero"\n    },',
-        '"stoploss": -10.0',
+        '"stoploss": -0.1',
     ]
 
     for line in exists:
@@ -141,3 +161,63 @@ def test_fmin_throw_value_error(mocker, caplog):
 
     for line in exists:
         assert line in caplog.text
+
+
+def test_resuming_previous_hyperopt_results_succeeds(mocker):
+    import freqtrade.optimize.hyperopt as hyperopt
+    trials = create_trials(mocker)
+    mocker.patch('freqtrade.optimize.hyperopt.TRIALS',
+                 return_value=trials)
+    mocker.patch('freqtrade.optimize.hyperopt.os.path.exists',
+                 return_value=True)
+    mocker.patch('freqtrade.optimize.hyperopt.len',
+                 return_value=len(trials.results))
+    mock_read = mocker.patch('freqtrade.optimize.hyperopt.read_trials',
+                             return_value=trials)
+    mock_save = mocker.patch('freqtrade.optimize.hyperopt.save_trials',
+                             return_value=None)
+    mocker.patch('freqtrade.optimize.hyperopt.sorted',
+                 return_value=trials.results)
+    mocker.patch('freqtrade.optimize.preprocess')
+    mocker.patch('freqtrade.optimize.load_data')
+    mocker.patch('freqtrade.optimize.hyperopt.fmin',
+                 return_value={})
+    args = mocker.Mock(epochs=1,
+                       config='config.json.example',
+                       mongodb=False)
+
+    start(args)
+
+    mock_read.assert_called_once()
+    mock_save.assert_called_once()
+
+    current_tries = hyperopt._CURRENT_TRIES
+    total_tries = hyperopt.TOTAL_TRIES
+
+    assert current_tries == len(trials.results)
+    assert total_tries == (current_tries + len(trials.results))
+
+
+def test_save_trials_saves_trials(mocker):
+    trials = create_trials(mocker)
+    mock_dump = mocker.patch('freqtrade.optimize.hyperopt.pickle.dump',
+                             return_value=None)
+    trials_path = mocker.patch('freqtrade.optimize.hyperopt.TRIALS_FILE',
+                               return_value='ut_trials.pickle')
+    mocker.patch('freqtrade.optimize.hyperopt.open',
+                 return_value=trials_path)
+    save_trials(trials, trials_path)
+
+    mock_dump.assert_called_once_with(trials, trials_path)
+
+
+def test_read_trials_returns_trials_file(mocker):
+    trials = create_trials(mocker)
+    mock_load = mocker.patch('freqtrade.optimize.hyperopt.pickle.load',
+                             return_value=trials)
+    mock_open = mocker.patch('freqtrade.optimize.hyperopt.open',
+                             return_value=mock_load)
+
+    assert read_trials() == trials
+    mock_open.assert_called_once()
+    mock_load.assert_called_once()

freqtrade/tests/optimize/test_optimize.py

@@ -6,7 +6,7 @@ from shutil import copyfile
 from freqtrade import exchange, optimize
 from freqtrade.exchange import Bittrex
 from freqtrade.optimize.__init__ import make_testdata_path, download_pairs,\
-     download_backtesting_testdata, load_tickerdata_file
+    download_backtesting_testdata, load_tickerdata_file
 
 # Change this if modifying BTC_UNITEST testdatafile
 _btc_unittest_length = 13681

freqtrade/tests/test_acl_pair.py

@@ -7,17 +7,17 @@ from freqtrade.main import refresh_whitelist, gen_pair_whitelist
 
 def whitelist_conf():
     return {
-        "stake_currency": "BTC",
+        'stake_currency': 'BTC',
-        "exchange": {
+        'exchange': {
-            "pair_whitelist": [
+            'pair_whitelist': [
-                "BTC_ETH",
+                'BTC_ETH',
-                "BTC_TKN",
+                'BTC_TKN',
-                "BTC_TRST",
+                'BTC_TRST',
-                "BTC_SWT",
+                'BTC_SWT',
-                "BTC_BCC"
+                'BTC_BCC'
             ],
-            "pair_blacklist": [
+            'pair_blacklist': [
-                "BTC_BLK"
+                'BTC_BLK'
             ],
         },
     }
@@ -25,52 +25,51 @@ def whitelist_conf():
 
 def get_market_summaries():
     return [{
-         "MarketName": "BTC-TKN",
+        'MarketName': 'BTC-TKN',
-         "High": 0.00000919,
+        'High': 0.00000919,
-         "Low": 0.00000820,
+        'Low': 0.00000820,
-         "Volume": 74339.61396015,
+        'Volume': 74339.61396015,
-         "Last": 0.00000820,
+        'Last': 0.00000820,
-         "BaseVolume": 1664,
+        'BaseVolume': 1664,
-         "TimeStamp": "2014-07-09T07:19:30.15",
+        'TimeStamp': '2014-07-09T07:19:30.15',
-         "Bid": 0.00000820,
+        'Bid': 0.00000820,
-         "Ask": 0.00000831,
+        'Ask': 0.00000831,
-         "OpenBuyOrders": 15,
+        'OpenBuyOrders': 15,
-         "OpenSellOrders": 15,
+        'OpenSellOrders': 15,
-         "PrevDay": 0.00000821,
+        'PrevDay': 0.00000821,
-         "Created": "2014-03-20T06:00:00",
+        'Created': '2014-03-20T06:00:00',
-         "DisplayMarketName": ""
+        'DisplayMarketName': ''
-      }, {
+    }, {
-         "MarketName": "BTC-ETH",
+        'MarketName': 'BTC-ETH',
-         "High": 0.00000072,
+        'High': 0.00000072,
-         "Low": 0.00000001,
+        'Low': 0.00000001,
-         "Volume": 166340678.42280999,
+        'Volume': 166340678.42280999,
-         "Last": 0.00000005,
+        'Last': 0.00000005,
-         "BaseVolume": 42,
+        'BaseVolume': 42,
-         "TimeStamp": "2014-07-09T07:21:40.51",
+        'TimeStamp': '2014-07-09T07:21:40.51',
-         "Bid": 0.00000004,
+        'Bid': 0.00000004,
-         "Ask": 0.00000005,
+        'Ask': 0.00000005,
-         "OpenBuyOrders": 18,
+        'OpenBuyOrders': 18,
-         "OpenSellOrders": 18,
+        'OpenSellOrders': 18,
-         "PrevDay": 0.00000002,
+        'PrevDay': 0.00000002,
-         "Created": "2014-05-30T07:57:49.637",
+        'Created': '2014-05-30T07:57:49.637',
-         "DisplayMarketName": ""
+        'DisplayMarketName': ''
-      }, {
+    }, {
-         "MarketName": "BTC-BLK",
+        'MarketName': 'BTC-BLK',
-         "High": 0.00000072,
+        'High': 0.00000072,
-         "Low": 0.00000001,
+        'Low': 0.00000001,
-         "Volume": 166340678.42280999,
+        'Volume': 166340678.42280999,
-         "Last": 0.00000005,
+        'Last': 0.00000005,
-         "BaseVolume": 3,
+        'BaseVolume': 3,
-         "TimeStamp": "2014-07-09T07:21:40.51",
+        'TimeStamp': '2014-07-09T07:21:40.51',
-         "Bid": 0.00000004,
+        'Bid': 0.00000004,
-         "Ask": 0.00000005,
+        'Ask': 0.00000005,
-         "OpenBuyOrders": 18,
+        'OpenBuyOrders': 18,
-         "OpenSellOrders": 18,
+        'OpenSellOrders': 18,
-         "PrevDay": 0.00000002,
+        'PrevDay': 0.00000002,
-         "Created": "2014-05-30T07:57:49.637",
+        'Created': '2014-05-30T07:57:49.637',
-         "DisplayMarketName": ""
+        'DisplayMarketName': ''
-      }
+    }]
-    ]
 
 
 def get_health():
@@ -95,7 +94,8 @@ def test_refresh_market_pair_not_in_whitelist(mocker):
     mocker.patch.dict('freqtrade.main._CONF', conf)
     mocker.patch.multiple('freqtrade.main.exchange',
                           get_wallet_health=get_health)
-    refreshedwhitelist = refresh_whitelist(conf['exchange']['pair_whitelist'] + ['BTC_XXX'])
+    refreshedwhitelist = refresh_whitelist(
+        conf['exchange']['pair_whitelist'] + ['BTC_XXX'])
     # List ordered by BaseVolume
     whitelist = ['BTC_ETH', 'BTC_TKN']
     # Ensure all except those in whitelist are removed
@@ -123,7 +123,8 @@ def test_refresh_whitelist_dynamic(mocker):
                           get_market_summaries=get_market_summaries)
     # argument: use the whitelist dynamically by exchange-volume
     whitelist = ['BTC_TKN', 'BTC_ETH']
-    refreshedwhitelist = refresh_whitelist(gen_pair_whitelist(conf['stake_currency']))
+    refreshedwhitelist = refresh_whitelist(
+        gen_pair_whitelist(conf['stake_currency']))
     assert whitelist == refreshedwhitelist
 
 

freqtrade/tests/test_analyze.py

@@ -6,8 +6,9 @@ import arrow
 import pytest
 from pandas import DataFrame
 
-from freqtrade.analyze import parse_ticker_dataframe, populate_buy_trend, populate_indicators, \
+from freqtrade.analyze import (SignalType, get_signal, parse_ticker_dataframe,
-    get_signal, SignalType, populate_sell_trend
+                               populate_buy_trend, populate_indicators,
+                               populate_sell_trend)
 
 
 @pytest.fixture

freqtrade/tests/test_dataframe.py

@@ -1,7 +1,7 @@
 import pandas
 
-from freqtrade import analyze
 import freqtrade.optimize
+from freqtrade import analyze
 
 _pairs = ['BTC_ETH']
 

freqtrade/tests/test_fiat_convert.py

@@ -1,10 +1,11 @@
 # pragma pylint: disable=missing-docstring, too-many-arguments, too-many-ancestors, C0103
 
 import time
-import pytest
 from unittest.mock import MagicMock
 
-from freqtrade.fiat_convert import CryptoToFiatConverter, CryptoFiat
+import pytest
+
+from freqtrade.fiat_convert import CryptoFiat, CryptoToFiatConverter
 
 
 def test_pair_convertion_object():

freqtrade/tests/test_main.py

@@ -1,28 +1,27 @@
 # pragma pylint: disable=missing-docstring,C0103
 import copy
+import logging
 from unittest.mock import MagicMock
 
+import arrow
 import pytest
 import requests
-import logging
-import arrow
 from sqlalchemy import create_engine
 
+import freqtrade.main as main
 from freqtrade import DependencyException, OperationalException
 from freqtrade.analyze import SignalType
 from freqtrade.exchange import Exchanges
-from freqtrade.main import create_trade, handle_trade, init, \
+from freqtrade.main import (_process, check_handle_timedout, create_trade,
-    get_target_bid, _process, execute_sell, check_handle_timedout
+                            execute_sell, get_target_bid, handle_trade, init)
-from freqtrade.misc import get_state, State
+from freqtrade.misc import State, get_state
 from freqtrade.persistence import Trade
-import freqtrade.main as main
 
 
-# Test that main() can start backtesting or hyperopt.
-# and also ensure we can pass some specific arguments
-# argument parsing is done in test_misc.py
-
 def test_parse_args_backtesting(mocker):
+    """ Test that main() can start backtesting or hyperopt.
+        and also ensure we can pass some specific arguments
+        argument parsing is done in test_misc.py """
     backtesting_mock = mocker.patch(
         'freqtrade.optimize.backtesting.start', MagicMock())
     with pytest.raises(SystemExit, match=r'0'):
@@ -269,7 +268,7 @@ def test_handle_trade(default_conf, limit_buy_order, limit_sell_order, mocker):
     assert trade.close_date is not None
 
 
-def test_handle_trade_roi(default_conf, ticker, limit_buy_order, mocker, caplog):
+def test_handle_trade_roi(default_conf, ticker, mocker, caplog):
     default_conf.update({'experimental': {'use_sell_signal': True}})
     mocker.patch.dict('freqtrade.main._CONF', default_conf)
 
@@ -301,7 +300,7 @@ def test_handle_trade_roi(default_conf, ticker, limit_buy_order, mocker, caplog)
     assert ('freqtrade', logging.DEBUG, 'Executing sell due to ROI ...') in caplog.record_tuples
 
 
-def test_handle_trade_experimental(default_conf, ticker, limit_buy_order, mocker, caplog):
+def test_handle_trade_experimental(default_conf, ticker, mocker, caplog):
     default_conf.update({'experimental': {'use_sell_signal': True}})
     mocker.patch.dict('freqtrade.main._CONF', default_conf)
 
@@ -353,7 +352,7 @@ def test_close_trade(default_conf, ticker, limit_buy_order, limit_sell_order, mo
         handle_trade(trade)
 
 
-def test_check_handle_timedout_buy(default_conf, ticker, health, limit_buy_order_old, mocker):
+def test_check_handle_timedout_buy(default_conf, ticker, limit_buy_order_old, mocker):
     mocker.patch.dict('freqtrade.main._CONF', default_conf)
     cancel_order_mock = MagicMock()
     mocker.patch.multiple('freqtrade.rpc', init=MagicMock(), send_msg=MagicMock())
@@ -385,7 +384,7 @@ def test_check_handle_timedout_buy(default_conf, ticker, health, limit_buy_order
     assert len(trades) == 0
 
 
-def test_check_handle_timedout_sell(default_conf, ticker, health, limit_sell_order_old, mocker):
+def test_check_handle_timedout_sell(default_conf, ticker, limit_sell_order_old, mocker):
     mocker.patch.dict('freqtrade.main._CONF', default_conf)
     cancel_order_mock = MagicMock()
     mocker.patch.multiple('freqtrade.rpc', init=MagicMock(), send_msg=MagicMock())
@@ -418,7 +417,7 @@ def test_check_handle_timedout_sell(default_conf, ticker, health, limit_sell_ord
 
 
 def test_check_handle_timedout_partial(default_conf, ticker, limit_buy_order_old_partial,
-                                       health, mocker):
+                                       mocker):
     mocker.patch.dict('freqtrade.main._CONF', default_conf)
     cancel_order_mock = MagicMock()
     mocker.patch.multiple('freqtrade.rpc', init=MagicMock(), send_msg=MagicMock())
@@ -624,54 +623,54 @@ def test_sell_profit_only_disable_profit(default_conf, limit_buy_order, mocker):
 
 
 def test_sell_profit_only_enable_loss(default_conf, limit_buy_order, mocker):
-        default_conf['experimental'] = {
+    default_conf['experimental'] = {
-            'use_sell_signal': True,
+        'use_sell_signal': True,
-            'sell_profit_only': True,
+        'sell_profit_only': True,
-        }
+    }
 
-        mocker.patch.dict('freqtrade.main._CONF', default_conf)
+    mocker.patch.dict('freqtrade.main._CONF', default_conf)
-        mocker.patch('freqtrade.main.min_roi_reached', return_value=False)
+    mocker.patch('freqtrade.main.min_roi_reached', return_value=False)
-        mocker.patch('freqtrade.main.get_signal', side_effect=lambda s, t: True)
+    mocker.patch('freqtrade.main.get_signal', side_effect=lambda s, t: True)
-        mocker.patch.multiple('freqtrade.rpc', init=MagicMock(), send_msg=MagicMock())
+    mocker.patch.multiple('freqtrade.rpc', init=MagicMock(), send_msg=MagicMock())
-        mocker.patch.multiple('freqtrade.main.exchange',
+    mocker.patch.multiple('freqtrade.main.exchange',
-                              validate_pairs=MagicMock(),
+                          validate_pairs=MagicMock(),
-                              get_ticker=MagicMock(return_value={
+                          get_ticker=MagicMock(return_value={
-                                  'bid': 0.00000172,
+                              'bid': 0.00000172,
-                                  'ask': 0.00000173,
+                              'ask': 0.00000173,
-                                  'last': 0.00000172
+                              'last': 0.00000172
-                              }),
+                          }),
-                              buy=MagicMock(return_value='mocked_limit_buy'))
+                          buy=MagicMock(return_value='mocked_limit_buy'))
 
-        init(default_conf, create_engine('sqlite://'))
+    init(default_conf, create_engine('sqlite://'))
-        create_trade(0.001)
+    create_trade(0.001)
 
-        trade = Trade.query.first()
+    trade = Trade.query.first()
-        trade.update(limit_buy_order)
+    trade.update(limit_buy_order)
-        assert handle_trade(trade) is False
+    assert handle_trade(trade) is False
 
 
 def test_sell_profit_only_disable_loss(default_conf, limit_buy_order, mocker):
-        default_conf['experimental'] = {
+    default_conf['experimental'] = {
-            'use_sell_signal': True,
+        'use_sell_signal': True,
-            'sell_profit_only': False,
+        'sell_profit_only': False,
-        }
+    }
 
-        mocker.patch.dict('freqtrade.main._CONF', default_conf)
+    mocker.patch.dict('freqtrade.main._CONF', default_conf)
-        mocker.patch('freqtrade.main.min_roi_reached', return_value=False)
+    mocker.patch('freqtrade.main.min_roi_reached', return_value=False)
-        mocker.patch('freqtrade.main.get_signal', side_effect=lambda s, t: True)
+    mocker.patch('freqtrade.main.get_signal', side_effect=lambda s, t: True)
-        mocker.patch.multiple('freqtrade.rpc', init=MagicMock(), send_msg=MagicMock())
+    mocker.patch.multiple('freqtrade.rpc', init=MagicMock(), send_msg=MagicMock())
-        mocker.patch.multiple('freqtrade.main.exchange',
+    mocker.patch.multiple('freqtrade.main.exchange',
-                              validate_pairs=MagicMock(),
+                          validate_pairs=MagicMock(),
-                              get_ticker=MagicMock(return_value={
+                          get_ticker=MagicMock(return_value={
-                                  'bid': 0.00000172,
+                              'bid': 0.00000172,
-                                  'ask': 0.00000173,
+                              'ask': 0.00000173,
-                                  'last': 0.00000172
+                              'last': 0.00000172
-                              }),
+                          }),
-                              buy=MagicMock(return_value='mocked_limit_buy'))
+                          buy=MagicMock(return_value='mocked_limit_buy'))
 
-        init(default_conf, create_engine('sqlite://'))
+    init(default_conf, create_engine('sqlite://'))
-        create_trade(0.001)
+    create_trade(0.001)
 
-        trade = Trade.query.first()
+    trade = Trade.query.first()
-        trade.update(limit_buy_order)
+    trade.update(limit_buy_order)
-        assert handle_trade(trade) is True
+    assert handle_trade(trade) is True

freqtrade/tests/test_misc.py

@@ -1,14 +1,14 @@
 # pragma pylint: disable=missing-docstring,C0103
+import argparse
 import json
 import time
-import argparse
 from copy import deepcopy
 
 import pytest
 from jsonschema import ValidationError
 
-from freqtrade.misc import throttle, parse_args, load_config,\
+from freqtrade.misc import (common_args_parser, load_config, parse_args,
-     parse_args_common
+                            throttle)
 
 
 def test_throttle():
@@ -39,12 +39,10 @@ def test_throttle_with_assets():
     assert result == -1
 
 
-# Parse common command-line-arguments
+# Parse common command-line-arguments. Used for all tools
-# used for all tools
-
 
 def test_parse_args_none():
-    args = parse_args_common([], '')
+    args = common_args_parser('')
     assert isinstance(args, argparse.ArgumentParser)
 
 
@@ -87,12 +85,12 @@ def test_parse_args_invalid():
 
 def test_parse_args_dynamic_whitelist():
     args = parse_args(['--dynamic-whitelist'], '')
-    assert args.dynamic_whitelist is 20
+    assert args.dynamic_whitelist == 20
 
 
 def test_parse_args_dynamic_whitelist_10():
     args = parse_args(['--dynamic-whitelist', '10'], '')
-    assert args.dynamic_whitelist is 10
+    assert args.dynamic_whitelist == 10
 
 
 def test_parse_args_dynamic_whitelist_invalid_values():

freqtrade/tests/test_persistence.py

@@ -1,9 +1,10 @@
 # pragma pylint: disable=missing-docstring
+import os
+
 import pytest
 
-import os
 from freqtrade.exchange import Exchanges
-from freqtrade.persistence import init, Trade
+from freqtrade.persistence import Trade, init
 
 
 def test_init_create_session(default_conf, mocker):

freqtrade/tests/testdata/download_backtest_data.py

@@ -1,29 +1,38 @@
 #!/usr/bin/env python3
 
 """This script generate json data from bittrex"""
+import sys
 import json
-from os import path
 
 from freqtrade import exchange
 from freqtrade.exchange import Bittrex
+from freqtrade import misc
 
-PAIRS = [
+parser = misc.common_args_parser('download utility')
-    'BTC_BCC', 'BTC_ETH', 'BTC_MER', 'BTC_POWR', 'BTC_ETC',
+parser.add_argument(
-    'BTC_OK', 'BTC_NEO', 'BTC_EMC2', 'BTC_DASH', 'BTC_LSK',
+        '-p', '--pair',
-    'BTC_LTC', 'BTC_XZC', 'BTC_OMG', 'BTC_STRAT', 'BTC_XRP',
+        help='JSON file containing pairs to download',
-    'BTC_QTUM', 'BTC_WAVES', 'BTC_VTC', 'BTC_XLM', 'BTC_MCO'
+        dest='pair',
-]
+        default=None
-TICKER_INTERVAL = 5  # ticker interval in minutes (currently implemented: 1 and 5)
+)
-OUTPUT_DIR = path.dirname(path.realpath(__file__))
+args = parser.parse_args(sys.argv[1:])
+
+TICKER_INTERVALS = [1, 5]  # ticker interval in minutes (currently implemented: 1 and 5)
+PAIRS = []
+
+if args.pair:
+    with open(args.pair) as file:
+        PAIRS = json.load(file)
+PAIRS = list(set(PAIRS))
+
+print('About to download pairs:', PAIRS)
 
 # Init Bittrex exchange
 exchange._API = Bittrex({'key': '', 'secret': ''})
 
 for pair in PAIRS:
-    data = exchange.get_ticker_history(pair, TICKER_INTERVAL)
+    for tick_interval in TICKER_INTERVALS:
-    filename = path.join(OUTPUT_DIR, '{}-{}.json'.format(
+        print('downloading pair %s, interval %s' % (pair, tick_interval))
-        pair,
+        data = exchange.get_ticker_history(pair, tick_interval)
-        TICKER_INTERVAL,
+        filename = '{}-{}.json'.format(pair, tick_interval)
-    ))
+        misc.file_dump_json(filename, data)
-    with open(filename, 'w') as fp:
-        json.dump(data, fp)

freqtrade/tests/testdata/pairs.json

@@ -0,0 +1,23 @@
+[
+ "BTC_ADA",
+ "BTC_BAT",
+ "BTC_DASH",
+ "BTC_ETC",
+ "BTC_ETH",
+ "BTC_GBYTE",
+ "BTC_LSK",
+ "BTC_LTC",
+ "BTC_NEO",
+ "BTC_NXT",
+ "BTC_POWR",
+ "BTC_STORJ",
+ "BTC_QTUM",
+ "BTC_WAVES",
+ "BTC_VTC",
+ "BTC_XLM",
+ "BTC_XMR",
+ "BTC_XVG",
+ "BTC_XRP",
+ "BTC_ZEC"
+]
+

requirements.txt

@@ -11,7 +11,7 @@ scikit-learn==0.19.1
 scipy==1.0.0
 jsonschema==2.6.0
 numpy==1.14.0
-TA-Lib==0.4.14
+TA-Lib==0.4.15
 pytest==3.3.2
 pytest-mock==1.6.3
 pytest-cov==2.5.1
@@ -19,7 +19,7 @@ hyperopt==0.1
 # do not upgrade networkx before this is fixed https://github.com/hyperopt/hyperopt/issues/325
 networkx==1.11
 tabulate==0.8.2
-pymarketcap==3.3.145
+pymarketcap==3.3.148
 
 # Required for plotting data
 #matplotlib==2.1.0

scripts/plot_dataframe.py

@@ -6,11 +6,11 @@ import matplotlib  # Install PYQT5 manually if you want to test this helper func
 matplotlib.use("Qt5Agg")
 import matplotlib.pyplot as plt
 from freqtrade import exchange, analyze
-from freqtrade.misc import parse_args_common
+from freqtrade.misc import common_args_parser
 
 
 def plot_parse_args(args ):
-    parser = parse_args_common(args, 'Graph utility')
+    parser = common_args_parser(description='Graph utility')
     parser.add_argument(
         '-p', '--pair',
         help = 'What currency pair',