adelorenzo/stable
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Merge pull request #920 from freqtrade/hyperopt-strip

Browse Source 
Remove mongodb from Hyperopt
This commit is contained in:
Janne Sinivirta 2018-06-16 10:33:44 +03:00 committed by GitHub
commit 0347ce21fd
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
15 changed files with 23 additions and 270 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

3
.travis.yml
View File

@ -16,16 +16,15 @@ install:
- pip install --upgrade flake8 coveralls pytest-random-order mypy
- pip install -r requirements.txt
- pip install -e .
- cp config.json.example config.json
jobs:
include:
- script:
- pytest --cov=freqtrade --cov-config=.coveragerc freqtrade/tests/
- coveralls
- script:
- cp config.json.example config.json
- python freqtrade/main.py --datadir freqtrade/tests/testdata backtesting
- script:
- cp config.json.example config.json
- python freqtrade/main.py --datadir freqtrade/tests/testdata hyperopt -e 5
- script: flake8 freqtrade
- script: mypy freqtrade

12
docs/bot-usage.md
View File

@ -160,13 +160,12 @@ the parameter `-l` or `--live`.
## Hyperopt commands
It is possible to use hyperopt for trading strategy optimization.
Hyperopt uses an internal json config return by `hyperopt_optimize_conf()`
located in `freqtrade/optimize/hyperopt_conf.py`.
To optimize your strategy, you can use hyperopt parameter hyperoptimization
to find optimal parameter values for your stategy.
```
usage: main.py hyperopt [-h] [-i TICKER_INTERVAL] [--realistic-simulation]
[--timerange TIMERANGE] [-e INT] [--use-mongodb]
[--timerange TIMERANGE] [-e INT]
[-s {all,buy,roi,stoploss} [{all,buy,roi,stoploss} ...]]
optional arguments:
@ -176,11 +175,8 @@ optional arguments:
--realistic-simulation
uses max_open_trades from config to simulate real
world limitations
--timerange TIMERANGE
specify what timerange of data to use.
--timerange TIMERANGE specify what timerange of data to use.
-e INT, --epochs INT specify number of epochs (default: 100)
--use-mongodb parallelize evaluations with mongodb (requires mongod
in PATH)
-s {all,buy,roi,stoploss} [{all,buy,roi,stoploss} ...], --spaces {all,buy,roi,stoploss} [{all,buy,roi,stoploss} ...]
Specify which parameters to hyperopt. Space separate
list. Default: all

36
docs/hyperopt.md
View File

@ -9,7 +9,6 @@ parameters with Hyperopt.
- [Advanced Hyperopt notions](#advanced-notions)
- [Understand the Guards and Triggers](#understand-the-guards-and-triggers)
- [Execute Hyperopt](#execute-hyperopt)
- [Hyperopt with MongoDB](#hyperopt-with-mongoDB)
- [Understand the hyperopts result](#understand-the-backtesting-result)
## Prepare Hyperopt
@ -194,41 +193,6 @@ Legal values are:
- `stoploss`: search for the best stoploss value
- space-separated list of any of the above values for example `--spaces roi stoploss`
### Hyperopt with MongoDB
Hyperopt with MongoDB, is like Hyperopt under steroids. As you saw by
executing the previous command is the execution takes a long time.
To accelerate it you can use hyperopt with MongoDB.
To run hyperopt with MongoDb you will need 3 terminals.
**Terminal 1: Start MongoDB**
```bash
cd <freqtrade>
source .env/bin/activate
python3 scripts/start-mongodb.py
```
**Terminal 2: Start Hyperopt worker**
```bash
cd <freqtrade>
source .env/bin/activate
python3 scripts/start-hyperopt-worker.py
```
**Terminal 3: Start Hyperopt with MongoDB**
```bash
cd <freqtrade>
source .env/bin/activate
python3 ./freqtrade/main.py -c config.json hyperopt --use-mongodb
```
**Re-run an Hyperopt**
To re-run Hyperopt you have to delete the existing MongoDB table.
```bash
cd <freqtrade>
rm -rf .hyperopt/mongodb/
```
## Understand the hyperopts result
Once Hyperopt is completed you can use the result to adding new buy
signal. Given following result from hyperopt:

28
docs/installation.md
View File

@ -225,17 +225,7 @@ cd ..
rm -rf ./ta-lib*
```
#### 3. [Optional] Install MongoDB
Install MongoDB if you plan to optimize your strategy with Hyperopt.
```bash
sudo apt-get install mongodb-org
```
> Complete tutorial from Digital Ocean: [How to Install MongoDB on Ubuntu 16.04](https://www.digitalocean.com/community/tutorials/how-to-install-mongodb-on-ubuntu-16-04).
#### 4. Install FreqTrade
#### 3. Install FreqTrade
Clone the git repository:
@ -243,7 +233,7 @@ Clone the git repository:
git clone https://github.com/freqtrade/freqtrade.git
```
#### 5. Configure `freqtrade` as a `systemd` service
#### 4. Configure `freqtrade` as a `systemd` service
From the freqtrade repo... copy `freqtrade.service` to your systemd user directory (usually `~/.config/systemd/user`) and update `WorkingDirectory` and `ExecStart` to match your setup.
@ -267,19 +257,7 @@ sudo loginctl enable-linger "$USER"
brew install python3 git wget ta-lib
```
#### 2. [Optional] Install MongoDB
Install MongoDB if you plan to optimize your strategy with Hyperopt.
```bash
curl -O https://fastdl.mongodb.org/osx/mongodb-osx-ssl-x86_64-3.4.10.tgz
tar -zxvf mongodb-osx-ssl-x86_64-3.4.10.tgz
mkdir -p <path_freqtrade>/env/mongodb
cp -R -n mongodb-osx-x86_64-3.4.10/ <path_freqtrade>/env/mongodb
export PATH=<path_freqtrade>/env/mongodb/bin:$PATH
```
#### 3. Install FreqTrade
#### 2. Install FreqTrade
Clone the git repository:

6
freqtrade/arguments.py
View File

@ -203,12 +203,6 @@ class Arguments(object):
type=int,
metavar='INT',
)
parser.add_argument(
'--use-mongodb',
help='parallelize evaluations with mongodb (requires mongod in PATH)',
dest='mongodb',
action='store_true',
)
parser.add_argument(
'-s', '--spaces',
help='Specify which parameters to hyperopt. Space separate list. \

5
freqtrade/configuration.py
View File

@ -188,11 +188,6 @@ class Configuration(object):
logger.info('Parameter --epochs detected ...')
logger.info('Will run Hyperopt with for %s epochs ...', config.get('epochs'))
# If --mongodb is used we add it to the configuration
if 'mongodb' in self.args and self.args.mongodb:
config.update({'mongodb': self.args.mongodb})
logger.info('Parameter --use-mongodb detected ...')
# If --spaces is used we add it to the configuration
if 'spaces' in self.args and self.args.spaces:
config.update({'spaces': self.args.spaces})

10
freqtrade/optimize/__init__.py
View File

@ -11,8 +11,6 @@ from freqtrade import misc, constants
from freqtrade.exchange import get_ticker_history
from freqtrade.arguments import TimeRange
from user_data.hyperopt_conf import hyperopt_optimize_conf
logger = logging.getLogger(__name__)
@ -83,7 +81,7 @@ def load_tickerdata_file(
def load_data(datadir: str,
ticker_interval: str,
pairs: Optional[List[str]] = None,
pairs: List[str],
refresh_pairs: Optional[bool] = False,
timerange: TimeRange = TimeRange(None, None, 0, 0)) -> Dict[str, List]:
"""
@ -92,14 +90,12 @@ def load_data(datadir: str,
"""
result = {}
_pairs = pairs or hyperopt_optimize_conf()['exchange']['pair_whitelist']
# If the user force the refresh of pairs
if refresh_pairs:
logger.info('Download data for all pairs and store them in %s', datadir)
download_pairs(datadir, _pairs, ticker_interval, timerange=timerange)
download_pairs(datadir, pairs, ticker_interval, timerange=timerange)
for pair in _pairs:
for pair in pairs:
pairdata = load_tickerdata_file(datadir, pair, ticker_interval, timerange=timerange)
if pairdata:
result[pair] = pairdata

42
freqtrade/optimize/hyperopt.py
View File

@ -19,7 +19,6 @@ from typing import Dict, Any, Callable, Optional
import numpy
import talib.abstract as ta
from hyperopt import STATUS_FAIL, STATUS_OK, Trials, fmin, hp, space_eval, tpe
from hyperopt.mongoexp import MongoTrials
from pandas import DataFrame
import freqtrade.vendor.qtpylib.indicators as qtpylib
@ -27,7 +26,6 @@ from freqtrade.arguments import Arguments
from freqtrade.configuration import Configuration
from freqtrade.optimize import load_data
from freqtrade.optimize.backtesting import Backtesting
from user_data.hyperopt_conf import hyperopt_optimize_conf
logger = logging.getLogger(__name__)
@ -508,32 +506,20 @@ class Hyperopt(Backtesting):
self.analyze.populate_indicators = Hyperopt.populate_indicators # type: ignore
self.processed = self.tickerdata_to_dataframe(data)
if self.config.get('mongodb'):
logger.info('Using mongodb ...')
logger.info('Preparing Trials..')
signal.signal(signal.SIGINT, self.signal_handler)
# read trials file if we have one
if os.path.exists(self.trials_file) and os.path.getsize(self.trials_file) > 0:
self.trials = self.read_trials()
self.current_tries = len(self.trials.results)
self.total_tries += self.current_tries
logger.info(
'Start scripts/start-mongodb.sh and start-hyperopt-worker.sh manually!'
'Continuing with trials. Current: %d, Total: %d',
self.current_tries,
self.total_tries
)
db_name = 'freqtrade_hyperopt'
self.trials = MongoTrials(
arg='mongo://127.0.0.1:1234/{}/jobs'.format(db_name),
exp_key='exp1'
)
else:
logger.info('Preparing Trials..')
signal.signal(signal.SIGINT, self.signal_handler)
# read trials file if we have one
if os.path.exists(self.trials_file) and os.path.getsize(self.trials_file) > 0:
self.trials = self.read_trials()
self.current_tries = len(self.trials.results)
self.total_tries += self.current_tries
logger.info(
'Continuing with trials. Current: %d, Total: %d',
self.current_tries,
self.total_tries
)
try:
best_parameters = fmin(
fn=self.generate_optimizer,
@ -589,18 +575,14 @@ def start(args: Namespace) -> None:
"""
# Remove noisy log messages
logging.getLogger('hyperopt.mongoexp').setLevel(logging.WARNING)
logging.getLogger('hyperopt.tpe').setLevel(logging.WARNING)
# Initialize configuration
# Monkey patch the configuration with hyperopt_conf.py
configuration = Configuration(args)
logger.info('Starting freqtrade in Hyperopt mode')
config = configuration.load_config()
optimize_config = hyperopt_optimize_conf()
config = configuration._load_common_config(optimize_config)
config = configuration._load_backtesting_config(config)
config = configuration._load_hyperopt_config(config)
config['exchange']['key'] = ''
config['exchange']['secret'] = ''

38
freqtrade/tests/optimize/test_hyperopt.py
View File

@ -23,8 +23,6 @@ def init_hyperopt(default_conf, mocker):
global _HYPEROPT_INITIALIZED, _HYPEROPT
if not _HYPEROPT_INITIALIZED:
mocker.patch('freqtrade.exchange.validate_pairs', MagicMock(return_value=True))
mocker.patch('freqtrade.optimize.hyperopt.hyperopt_optimize_conf',
MagicMock(return_value=default_conf))
mocker.patch('freqtrade.exchange.validate_pairs', MagicMock())
_HYPEROPT = Hyperopt(default_conf)
_HYPEROPT_INITIALIZED = True
@ -64,8 +62,6 @@ def test_start(mocker, default_conf, caplog) -> None:
"""
start_mock = MagicMock()
mocker.patch('freqtrade.optimize.hyperopt.Hyperopt.start', start_mock)
mocker.patch('freqtrade.optimize.hyperopt.hyperopt_optimize_conf',
MagicMock(return_value=default_conf))
mocker.patch('freqtrade.freqtradebot.exchange.validate_pairs', MagicMock())
args = [
@ -182,7 +178,6 @@ def test_fmin_best_results(mocker, init_hyperopt, default_conf, caplog) -> None:
mocker.patch('freqtrade.optimize.hyperopt.load_data', MagicMock())
mocker.patch('freqtrade.optimize.hyperopt.fmin', return_value=fmin_result)
mocker.patch('freqtrade.optimize.hyperopt.hyperopt_optimize_conf', return_value=conf)
mocker.patch('freqtrade.freqtradebot.exchange.validate_pairs', MagicMock())
StrategyResolver({'strategy': 'DefaultStrategy'})
@ -227,7 +222,6 @@ def test_fmin_throw_value_error(mocker, init_hyperopt, default_conf, caplog) ->
conf.update({'epochs': 1})
conf.update({'timerange': None})
conf.update({'spaces': 'all'})
mocker.patch('freqtrade.optimize.hyperopt.hyperopt_optimize_conf', return_value=conf)
mocker.patch('freqtrade.freqtradebot.exchange.validate_pairs', MagicMock())
StrategyResolver({'strategy': 'DefaultStrategy'})
@ -253,7 +247,6 @@ def test_resuming_previous_hyperopt_results_succeeds(mocker, init_hyperopt, defa
conf = deepcopy(default_conf)
conf.update({'config': 'config.json.example'})
conf.update({'epochs': 1})
conf.update({'mongodb': False})
conf.update({'timerange': None})
conf.update({'spaces': 'all'})
@ -270,7 +263,6 @@ def test_resuming_previous_hyperopt_results_succeeds(mocker, init_hyperopt, defa
mocker.patch('freqtrade.optimize.hyperopt.sorted', return_value=trials.results)
mocker.patch('freqtrade.optimize.hyperopt.load_data', MagicMock())
mocker.patch('freqtrade.optimize.hyperopt.fmin', return_value={})
mocker.patch('freqtrade.optimize.hyperopt.hyperopt_optimize_conf', return_value=conf)
mocker.patch('freqtrade.exchange.validate_pairs', MagicMock())
StrategyResolver({'strategy': 'DefaultStrategy'})
@ -348,7 +340,6 @@ def test_start_calls_fmin(mocker, init_hyperopt, default_conf) -> None:
conf = deepcopy(default_conf)
conf.update({'config': 'config.json.example'})
conf.update({'epochs': 1})
conf.update({'mongodb': False})
conf.update({'timerange': None})
conf.update({'spaces': 'all'})
@ -360,35 +351,6 @@ def test_start_calls_fmin(mocker, init_hyperopt, default_conf) -> None:
mock_fmin.assert_called_once()
def test_start_uses_mongotrials(mocker, init_hyperopt, default_conf) -> None:
mocker.patch('freqtrade.optimize.hyperopt.load_data', MagicMock())
mock_fmin = mocker.patch('freqtrade.optimize.hyperopt.fmin', return_value={})
mock_mongotrials = mocker.patch(
'freqtrade.optimize.hyperopt.MongoTrials',
return_value=create_trials(mocker)
)
conf = deepcopy(default_conf)
conf.update({'config': 'config.json.example'})
conf.update({'epochs': 1})
conf.update({'mongodb': True})
conf.update({'timerange': None})
conf.update({'spaces': 'all'})
mocker.patch('freqtrade.optimize.hyperopt.hyperopt_optimize_conf', return_value=conf)
mocker.patch('freqtrade.freqtradebot.exchange.validate_pairs', MagicMock())
hyperopt = Hyperopt(conf)
hyperopt.tickerdata_to_dataframe = MagicMock()
hyperopt.start()
mock_mongotrials.assert_called_once()
mock_fmin.assert_called_once()
# test log_trials_result
# test buy_strategy_generator def populate_buy_trend
# test optimizer if 'ro_t1' in params
def test_format_results(init_hyperopt):
"""
Test Hyperopt.format_results()

16
freqtrade/tests/optimize/test_hyperopt_config.py
View File

@ -1,16 +0,0 @@
# pragma pylint: disable=missing-docstring,W0212
from user_data.hyperopt_conf import hyperopt_optimize_conf
def test_hyperopt_optimize_conf():
hyperopt_conf = hyperopt_optimize_conf()
assert "max_open_trades" in hyperopt_conf
assert "stake_currency" in hyperopt_conf
assert "stake_amount" in hyperopt_conf
assert "minimal_roi" in hyperopt_conf
assert "stoploss" in hyperopt_conf
assert "bid_strategy" in hyperopt_conf
assert "exchange" in hyperopt_conf
assert "pair_whitelist" in hyperopt_conf['exchange']

2
freqtrade/tests/optimize/test_optimize.py
View File

@ -326,8 +326,6 @@ def test_load_tickerdata_file() -> None:
def test_init(default_conf, mocker) -> None:
conf = {'exchange': {'pair_whitelist': []}}
mocker.patch('freqtrade.optimize.hyperopt_optimize_conf', return_value=conf)
assert {} == optimize.load_data(
'',
pairs=[],

5
freqtrade/tests/test_configuration.py
View File

@ -310,7 +310,6 @@ def test_hyperopt_with_arguments(mocker, default_conf, caplog) -> None:
arglist = [
'hyperopt',
'--epochs', '10',
'--use-mongodb',
'--spaces', 'all',
]
@ -324,10 +323,6 @@ def test_hyperopt_with_arguments(mocker, default_conf, caplog) -> None:
assert log_has('Parameter --epochs detected ...', caplog.record_tuples)
assert log_has('Will run Hyperopt with for 10 epochs ...', caplog.record_tuples)
assert 'mongodb' in config
assert config['mongodb'] is True
assert log_has('Parameter --use-mongodb detected ...', caplog.record_tuples)
assert 'spaces' in config
assert config['spaces'] == ['all']
assert log_has('Parameter -s/--spaces detected: [\'all\']', caplog.record_tuples)

27
scripts/start-hyperopt-worker.py
View File

@ -1,27 +0,0 @@
#!/usr/bin/env python3
import multiprocessing
import os
import subprocess
PROC_COUNT = multiprocessing.cpu_count() - 1
DB_NAME = 'freqtrade_hyperopt'
WORK_DIR = os.path.join(
os.path.sep,
os.path.abspath(os.path.dirname(__file__)),
'..', '.hyperopt', 'worker'
)
if not os.path.exists(WORK_DIR):
os.makedirs(WORK_DIR)
# Spawn workers
command = [
'hyperopt-mongo-worker',
'--mongo=127.0.0.1:1234/{}'.format(DB_NAME),
'--poll-interval=0.1',
'--workdir={}'.format(WORK_DIR),
]
processes = [subprocess.Popen(command) for i in range(PROC_COUNT)]
# Join all workers
for proc in processes:
proc.wait()

21
scripts/start-mongodb.py
View File

@ -1,21 +0,0 @@
#!/usr/bin/env python3
import os
import subprocess
DB_PATH = os.path.join(
os.path.sep,
os.path.abspath(os.path.dirname(__file__)),
'..', '.hyperopt', 'mongodb'
)
if not os.path.exists(DB_PATH):
os.makedirs(DB_PATH)
subprocess.Popen([
'mongod',
'--bind_ip=127.0.0.1',
'--port=1234',
'--nohttpinterface',
'--dbpath={}'.format(DB_PATH),
]).wait()

42
user_data/hyperopt_conf.py
View File

@ -1,42 +0,0 @@
"""
File that contains the configuration for Hyperopt
"""
def hyperopt_optimize_conf() -> dict:
"""
This function is used to define which parameters Hyperopt must used.
The "pair_whitelist" is only used is your are using Hyperopt with MongoDB,
without MongoDB, Hyperopt will use the pair your have set in your config file.
:return:
"""
return {
'max_open_trades': 3,
'stake_currency': 'BTC',
'stake_amount': 0.01,
"minimal_roi": {
'40': 0.0,
'30': 0.01,
'20': 0.02,
'0': 0.04,
},
'stoploss': -0.10,
"bid_strategy": {
"ask_last_balance": 0.0
},
"exchange": {
"name": "bittrex",
"pair_whitelist": [
"ETH/BTC",
"LTC/BTC",
"ETC/BTC",
"DASH/BTC",
"ZEC/BTC",
"XLM/BTC",
"NXT/BTC",
"POWR/BTC",
"ADA/BTC",
"XMR/BTC"
]
}
}