adelorenzo/stable
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

whitelist conflict resolved with develop branch

Browse Source
This commit is contained in:
misagh 2018-11-02 18:59:31 +01:00
commit 080ecae332
16 changed files with 351 additions and 468 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

64
docs/hyperopt.md
View File

@ -1,4 +1,5 @@
# Hyperopt
This page explains how to tune your strategy by finding the optimal
parameters, a process called hyperparameter optimization. The bot uses several
algorithms included in the `scikit-optimize` package to accomplish this. The
@ -8,17 +9,20 @@ and still take a long time.
*Note:* Hyperopt will crash when used with only 1 CPU Core as found out in [Issue #1133](https://github.com/freqtrade/freqtrade/issues/1133)
## Table of Contents
- [Prepare your Hyperopt](#prepare-hyperopt)
- [Configure your Guards and Triggers](#configure-your-guards-and-triggers)
- [Solving a Mystery](#solving-a-mystery)
- [Adding New Indicators](#adding-new-indicators)
- [Execute Hyperopt](#execute-hyperopt)
- [Understand the hyperopts result](#understand-the-backtesting-result)
- [Understand the hyperopt result](#understand-the-hyperopt-result)
## Prepare Hyperopting
We recommend you start by taking a look at `hyperopt.py` file located in [freqtrade/optimize](https://github.com/freqtrade/freqtrade/blob/develop/freqtrade/optimize/hyperopt.py)
### Configure your Guards and Triggers
There are two places you need to change to add a new buy strategy for testing:
- Inside [populate_buy_trend()](https://github.com/freqtrade/freqtrade/blob/develop/freqtrade/optimize/hyperopt.py#L231-L264).
- Inside [hyperopt_space()](https://github.com/freqtrade/freqtrade/blob/develop/freqtrade/optimize/hyperopt.py#L213-L224)
@ -113,11 +117,12 @@ When you want to test an indicator that isn't used by the bot currently, remembe
add it to the `populate_indicators()` method in `hyperopt.py`.
## Execute Hyperopt
Once you have updated your hyperopt configuration you can run it.
Because hyperopt tries a lot of combination to find the best parameters
it will take time you will have the result (more than 30 mins).
We strongly recommend to use `screen` to prevent any connection loss.
Once you have updated your hyperopt configuration you can run it.
Because hyperopt tries a lot of combinations to find the best parameters it will take time you will have the result (more than 30 mins).
We strongly recommend to use `screen` or `tmux` to prevent any connection loss.
```bash
python3 ./freqtrade/main.py -c config.json hyperopt -e 5000
```
@ -126,11 +131,13 @@ The `-e` flag will set how many evaluations hyperopt will do. We recommend
running at least several thousand evaluations.
### Execute Hyperopt with Different Ticker-Data Source
If you would like to hyperopt parameters using an alternate ticker data that
you have on-disk, use the `--datadir PATH` option. Default hyperopt will
use data from directory `user_data/data`.
### Running Hyperopt with Smaller Testset
Use the `--timerange` argument to change how much of the testset
you want to use. The last N ticks/timeframes will be used.
Example:
@ -140,6 +147,7 @@ python3 ./freqtrade/main.py hyperopt --timerange -200
```
### Running Hyperopt with Smaller Search Space
Use the `--spaces` argument to limit the search space used by hyperopt.
Letting Hyperopt optimize everything is a huuuuge search space. Often it
might make more sense to start by just searching for initial buy algorithm.
@ -154,7 +162,8 @@ Legal values are:
- `stoploss`: search for the best stoploss value
- space-separated list of any of the above values for example `--spaces roi stoploss`
## Understand the Hyperopts Result
## Understand the Hyperopt Result
Once Hyperopt is completed you can use the result to create a new strategy.
Given the following result from hyperopt:
@ -166,22 +175,24 @@ with values:
```
You should understand this result like:
- The buy trigger that worked best was `bb_lower`.
- You should not use ADX because `adx-enabled: False`)
- You should **consider** using the RSI indicator (`rsi-enabled: True` and the best value is `29.0` (`rsi-value: 29.0`)
You have to look inside your strategy file into `buy_strategy_generator()`
method, what those values match to.
method, what those values match to.
So for example you had `rsi-value: 29.0` so we would look
at `rsi`-block, that translates to the following code block:
So for example you had `rsi-value: 29.0` so we would look at `rsi`-block, that translates to the following code block:
```
(dataframe['rsi'] < 29.0)
```
Translating your whole hyperopt result as the new buy-signal
would then look like:
```
```python
def populate_buy_trend(self, dataframe: DataFrame) -> DataFrame:
dataframe.loc[
(
@ -192,6 +203,39 @@ def populate_buy_trend(self, dataframe: DataFrame) -> DataFrame:
return dataframe
```
### Understand Hyperopt ROI results
If you are optimizing ROI, you're result will look as follows and include a ROI table.
```
Best result:
135 trades. Avg profit 0.57%. Total profit 0.03871918 BTC (0.7722Σ%). Avg duration 180.4 mins.
with values:
{'adx-value': 44, 'rsi-value': 29, 'adx-enabled': False, 'rsi-enabled': True, 'trigger': 'bb_lower', 'roi_t1': 40, 'roi_t2': 57, 'roi_t3': 21, 'roi_p1': 0.03634636907306948, 'roi_p2': 0.055237357937802885, 'roi_p3': 0.015163796015548354, 'stoploss': -0.37996664668703606}
ROI table:
{0: 0.10674752302642071, 21: 0.09158372701087236, 78: 0.03634636907306948, 118: 0}
```
This would translate to the following ROI table:
``` python
minimal_roi = {
"118": 0,
"78": 0.0363463,
"21": 0.0915,
"0": 0.106
}
```
### Validate backtest result
Once the optimized strategy has been implemented into your strategy, you should backtest this strategy to make sure everything is working as expected.
To archive the same results (number of trades, ...) than during hyperopt, please use the command line flag `--disable-max-market-positions`.
This setting is the default for hyperopt for speed reasons. You can overwrite this in the configuration by setting `"position_stacking"=false` or by changing the relevant line in your hyperopt file [here](https://github.com/freqtrade/freqtrade/blob/develop/freqtrade/optimize/hyperopt.py#L283).
Dry/live runs will **NOT** use position stacking - therefore it does make sense to also validate the strategy without this as it's closer to reality.
## Next Step
Now you have a perfect bot and want to control it from Telegram. Your
next step is to learn the [Telegram usage](https://github.com/freqtrade/freqtrade/blob/develop/docs/telegram-usage.md).

2
freqtrade/edge/__init__.py
View File

@ -38,7 +38,7 @@ class Edge():
self.strategy: IStrategy = StrategyResolver(self.config).strategy
self.ticker_interval = self.strategy.ticker_interval
self.tickerdata_to_dataframe = self.strategy.tickerdata_to_dataframe
self.get_timeframe = Backtesting.get_timeframe
self.get_timeframe = optimize.get_timeframe
self.advise_sell = self.strategy.advise_sell
self.advise_buy = self.strategy.advise_buy

2
freqtrade/exchange/__init__.py
View File

@ -375,6 +375,8 @@ class Exchange(object):
def get_ticker(self, pair: str, refresh: Optional[bool] = True) -> dict:
if refresh or pair not in self._cached_ticker.keys():
try:
if pair not in self._api.markets:
raise DependencyException(f"Pair {pair} not available")
data = self._api.fetch_ticker(pair)
try:
self._cached_ticker[pair] = {

26
freqtrade/freqtradebot.py
View File

@ -61,6 +61,7 @@ class FreqtradeBot(object):
if self.config.get('edge', {}).get('enabled', False):
self.edge = Edge(self.config, self.exchange)
self.active_pair_whitelist: List[str] = self.config['exchange']['pair_whitelist']
self._init_modules()
def _init_modules(self) -> None:
@ -114,11 +115,8 @@ class FreqtradeBot(object):
constants.PROCESS_THROTTLE_SECS
)
nb_assets = self.config.get('dynamic_whitelist', None)
self._throttle(func=self._process,
min_secs=min_secs,
nb_assets=nb_assets)
min_secs=min_secs)
return state
def _startup_messages(self) -> None:
@ -169,15 +167,15 @@ class FreqtradeBot(object):
time.sleep(duration)
return result
def _process(self, nb_assets: Optional[int] = 0) -> bool:
def _process(self) -> bool:
"""
Queries the persistence layer for open trades and handles them,
otherwise a new trade is created.
:param: nb_assets: the maximum number of pairs to be traded at the same time
:return: True if one or more trades has been created or closed, False otherwise
"""
state_changed = False
try:
nb_assets = self.config.get('dynamic_whitelist', None)
# Refresh whitelist based on wallet maintenance
sanitized_list = self._refresh_whitelist(
self._gen_pair_whitelist(
@ -186,8 +184,7 @@ class FreqtradeBot(object):
)
# Keep only the subsets of pairs wanted (up to nb_assets)
final_list = sanitized_list[:nb_assets] if nb_assets else sanitized_list
self.config['exchange']['pair_whitelist'] = final_list
self.active_pair_whitelist = sanitized_list[:nb_assets] if nb_assets else sanitized_list
# Calculating Edge positiong
# Should be called before refresh_tickers
@ -197,11 +194,20 @@ class FreqtradeBot(object):
self.edge.calculate()
# Refreshing candles
self.exchange.refresh_tickers(final_list, self.strategy.ticker_interval)
self.exchange.refresh_tickers(self.active_pair_whitelist, self.strategy.ticker_interval)
# Query trades from persistence layer
trades = Trade.query.filter(Trade.is_open.is_(True)).all()
# Extend active-pair whitelist with pairs from open trades
# ensures that tickers are downloaded for open trades
self.active_pair_whitelist.extend([trade.pair for trade in trades
if trade.pair not in self.active_pair_whitelist])
# Refreshing candles
self.exchange.refresh_tickers(self.active_pair_whitelist, self.strategy.ticker_interval)
# First process current opened trades
for trade in trades:
state_changed |= self.process_maybe_execute_sell(trade)
@ -389,7 +395,7 @@ class FreqtradeBot(object):
:return: True if a trade object has been created and persisted, False otherwise
"""
interval = self.strategy.ticker_interval
whitelist = copy.deepcopy(self.config['exchange']['pair_whitelist'])
whitelist = copy.deepcopy(self.active_pair_whitelist)
# Remove currently opened and latest pairs from whitelist
for trade in Trade.query.filter(Trade.is_open.is_(True)).all():

48
freqtrade/optimize/__init__.py
View File

@ -10,8 +10,12 @@ except ImportError:
_UJSON = False
import logging
import os
from datetime import datetime
from typing import Optional, List, Dict, Tuple, Any
import operator
import arrow
from pandas import DataFrame
from freqtrade import misc, constants, OperationalException
from freqtrade.exchange import Exchange
@ -59,6 +63,42 @@ def trim_tickerlist(tickerlist: List[Dict], timerange: TimeRange) -> List[Dict]:
return tickerlist[start_index:stop_index]
def get_timeframe(data: Dict[str, DataFrame]) -> Tuple[arrow.Arrow, arrow.Arrow]:
"""
Get the maximum timeframe for the given backtest data
:param data: dictionary with preprocessed backtesting data
:return: tuple containing min_date, max_date
"""
timeframe = [
(arrow.get(frame['date'].min()), arrow.get(frame['date'].max()))
for frame in data.values()
]
return min(timeframe, key=operator.itemgetter(0))[0], \
max(timeframe, key=operator.itemgetter(1))[1]
def validate_backtest_data(data: Dict[str, DataFrame], min_date: datetime,
max_date: datetime, ticker_interval_mins: int) -> bool:
"""
Validates preprocessed backtesting data for missing values and shows warnings about it that.
:param data: dictionary with preprocessed backtesting data
:param min_date: start-date of the data
:param max_date: end-date of the data
:param ticker_interval_mins: ticker interval in minutes
"""
# total difference in minutes / interval-minutes
expected_frames = int((max_date - min_date).total_seconds() // 60 // ticker_interval_mins)
found_missing = False
for pair, df in data.items():
dflen = len(df)
if dflen < expected_frames:
found_missing = True
logger.warning("%s has missing frames: expected %s, got %s, that's %s missing values",
pair, expected_frames, dflen, expected_frames - dflen)
return found_missing
def load_tickerdata_file(
datadir: str, pair: str,
ticker_interval: str,
@ -113,6 +153,14 @@ def load_data(datadir: str,
for pair in pairs:
pairdata = load_tickerdata_file(datadir, pair, ticker_interval, timerange=timerange)
if pairdata:
if timerange.starttype == 'date' and pairdata[0][0] > timerange.startts * 1000:
logger.warning('Missing data at start for pair %s, data starts at %s',
pair,
arrow.get(pairdata[0][0] // 1000).strftime('%Y-%m-%d %H:%M:%S'))
if timerange.stoptype == 'date' and pairdata[-1][0] < timerange.stopts * 1000:
logger.warning('Missing data at end for pair %s, data ends at %s',
pair,
arrow.get(pairdata[-1][0] // 1000).strftime('%Y-%m-%d %H:%M:%S'))
result[pair] = pairdata
else:
logger.warning(

27
freqtrade/optimize/backtesting.py
View File

@ -4,14 +4,12 @@
This module contains the backtesting logic
"""
import logging
import operator
from argparse import Namespace
from copy import deepcopy
from datetime import datetime, timedelta
from pathlib import Path
from typing import Any, Dict, List, NamedTuple, Optional, Tuple
from typing import Any, Dict, List, NamedTuple, Optional
import arrow
from pandas import DataFrame
from tabulate import tabulate
@ -88,24 +86,9 @@ class Backtesting(object):
"""
self.strategy = strategy
self.ticker_interval = self.config.get('ticker_interval')
self.tickerdata_to_dataframe = strategy.tickerdata_to_dataframe
self.advise_buy = strategy.advise_buy
self.advise_sell = strategy.advise_sell
@staticmethod
def get_timeframe(data: Dict[str, DataFrame]) -> Tuple[arrow.Arrow, arrow.Arrow]:
"""
Get the maximum timeframe for the given backtest data
:param data: dictionary with preprocessed backtesting data
:return: tuple containing min_date, max_date
"""
timeframe = [
(arrow.get(frame['date'].min()), arrow.get(frame['date'].max()))
for frame in data.values()
]
return min(timeframe, key=operator.itemgetter(0))[0], \
max(timeframe, key=operator.itemgetter(1))[1]
def _generate_text_table(self, data: Dict[str, Dict], results: DataFrame,
skip_nan: bool = False) -> str:
"""
@ -371,10 +354,12 @@ class Backtesting(object):
self._set_strategy(strat)
# need to reprocess data every time to populate signals
preprocessed = self.tickerdata_to_dataframe(data)
preprocessed = self.strategy.tickerdata_to_dataframe(data)
# Print timeframe
min_date, max_date = self.get_timeframe(preprocessed)
min_date, max_date = optimize.get_timeframe(preprocessed)
# Validate dataframe for missing values
optimize.validate_backtest_data(preprocessed, min_date, max_date,
constants.TICKER_INTERVAL_MINUTES[self.ticker_interval])
logger.info(
'Measuring data from %s up to %s (%s days)..',
min_date.isoformat(),

2
freqtrade/optimize/hyperopt.py
View File

@ -352,7 +352,7 @@ class Hyperopt(Backtesting):
if self.has_space('buy'):
self.strategy.advise_indicators = Hyperopt.populate_indicators # type: ignore
dump(self.tickerdata_to_dataframe(data), TICKERDATA_PICKLE)
dump(self.strategy.tickerdata_to_dataframe(data), TICKERDATA_PICKLE)
self.exchange = None # type: ignore
self.load_previous_results()

21
freqtrade/rpc/rpc.py
View File

@ -10,10 +10,10 @@ from typing import Dict, Any, List, Optional
import arrow
import sqlalchemy as sql
from numpy import mean, nan_to_num
from numpy import mean, nan_to_num, NAN
from pandas import DataFrame
from freqtrade import TemporaryError
from freqtrade import TemporaryError, DependencyException
from freqtrade.fiat_convert import CryptoToFiatConverter
from freqtrade.misc import shorten_date
from freqtrade.persistence import Trade
@ -93,7 +93,10 @@ class RPC(object):
if trade.open_order_id:
order = self._freqtrade.exchange.get_order(trade.open_order_id, trade.pair)
# calculate profit and send message to user
current_rate = self._freqtrade.exchange.get_ticker(trade.pair, False)['bid']
try:
current_rate = self._freqtrade.exchange.get_ticker(trade.pair, False)['bid']
except DependencyException:
current_rate = NAN
current_profit = trade.calc_profit_percent(current_rate)
fmt_close_profit = (f'{round(trade.close_profit * 100, 2):.2f}%'
if trade.close_profit else None)
@ -122,7 +125,10 @@ class RPC(object):
trades_list = []
for trade in trades:
# calculate profit and send message to user
current_rate = self._freqtrade.exchange.get_ticker(trade.pair, False)['bid']
try:
current_rate = self._freqtrade.exchange.get_ticker(trade.pair, False)['bid']
except DependencyException:
current_rate = NAN
trade_perc = (100 * trade.calc_profit_percent(current_rate))
trades_list.append([
trade.id,
@ -207,7 +213,10 @@ class RPC(object):
profit_closed_percent.append(profit_percent)
else:
# Get current rate
current_rate = self._freqtrade.exchange.get_ticker(trade.pair, False)['bid']
try:
current_rate = self._freqtrade.exchange.get_ticker(trade.pair, False)['bid']
except DependencyException:
current_rate = NAN
profit_percent = trade.calc_profit_percent(rate=current_rate)
profit_all_coin.append(
@ -275,7 +284,7 @@ class RPC(object):
rate = 1.0 / self._freqtrade.exchange.get_ticker('BTC/USDT', False)['bid']
else:
rate = self._freqtrade.exchange.get_ticker(coin + '/BTC', False)['bid']
except TemporaryError:
except (TemporaryError, DependencyException):
continue
est_btc: float = rate * balance['total']
total = total + est_btc

4
freqtrade/tests/exchange/test_exchange.py
View File

@ -572,6 +572,7 @@ def test_get_ticker(default_conf, mocker):
'last': 0.0001,
}
api_mock.fetch_ticker = MagicMock(return_value=tick)
api_mock.markets = {'ETH/BTC': {}}
exchange = get_patched_exchange(mocker, default_conf, api_mock)
# retrieve original ticker
ticker = exchange.get_ticker(pair='ETH/BTC')
@ -614,6 +615,9 @@ def test_get_ticker(default_conf, mocker):
exchange = get_patched_exchange(mocker, default_conf, api_mock)
exchange.get_ticker(pair='ETH/BTC', refresh=True)
with pytest.raises(DependencyException, match=r'Pair XRP/ETH not available'):
exchange.get_ticker(pair='XRP/ETH', refresh=True)
def test_get_history(default_conf, mocker, caplog):
exchange = get_patched_exchange(mocker, default_conf)

42
freqtrade/tests/optimize/test_backtesting.py
View File

@ -89,7 +89,7 @@ def simple_backtest(config, contour, num_results, mocker) -> None:
backtesting = Backtesting(config)
data = load_data_test(contour)
processed = backtesting.tickerdata_to_dataframe(data)
processed = backtesting.strategy.tickerdata_to_dataframe(data)
assert isinstance(processed, dict)
results = backtesting.backtest(
{
@ -119,13 +119,13 @@ def _load_pair_as_ticks(pair, tickfreq):
# FIX: fixturize this?
def _make_backtest_conf(mocker, conf=None, pair='UNITTEST/BTC', record=None):
data = optimize.load_data(None, ticker_interval='8m', pairs=[pair])
data = optimize.load_data(None, ticker_interval='1m', pairs=[pair])
data = trim_dictlist(data, -201)
patch_exchange(mocker)
backtesting = Backtesting(conf)
return {
'stake_amount': conf['stake_amount'],
'processed': backtesting.tickerdata_to_dataframe(data),
'processed': backtesting.strategy.tickerdata_to_dataframe(data),
'max_open_trades': 10,
'position_stacking': False,
'record': record
@ -313,7 +313,7 @@ def test_backtesting_init(mocker, default_conf) -> None:
backtesting = Backtesting(default_conf)
assert backtesting.config == default_conf
assert backtesting.ticker_interval == '5m'
assert callable(backtesting.tickerdata_to_dataframe)
assert callable(backtesting.strategy.tickerdata_to_dataframe)
assert callable(backtesting.advise_buy)
assert callable(backtesting.advise_sell)
get_fee.assert_called()
@ -327,7 +327,7 @@ def test_tickerdata_to_dataframe(default_conf, mocker) -> None:
tickerlist = {'UNITTEST/BTC': tick}
backtesting = Backtesting(default_conf)
data = backtesting.tickerdata_to_dataframe(tickerlist)
data = backtesting.strategy.tickerdata_to_dataframe(tickerlist)
assert len(data['UNITTEST/BTC']) == 99
# Load strategy to compare the result between Backtesting function and strategy are the same
@ -336,22 +336,6 @@ def test_tickerdata_to_dataframe(default_conf, mocker) -> None:
assert data['UNITTEST/BTC'].equals(data2['UNITTEST/BTC'])
def test_get_timeframe(default_conf, mocker) -> None:
patch_exchange(mocker)
backtesting = Backtesting(default_conf)
data = backtesting.tickerdata_to_dataframe(
optimize.load_data(
None,
ticker_interval='1m',
pairs=['UNITTEST/BTC']
)
)
min_date, max_date = backtesting.get_timeframe(data)
assert min_date.isoformat() == '2017-11-04T23:02:00+00:00'
assert max_date.isoformat() == '2017-11-14T22:58:00+00:00'
def test_generate_text_table(default_conf, mocker):
patch_exchange(mocker)
backtesting = Backtesting(default_conf)
@ -451,21 +435,21 @@ def test_generate_text_table_strategyn(default_conf, mocker):
def test_backtesting_start(default_conf, mocker, caplog) -> None:
def get_timeframe(input1, input2):
def get_timeframe(input1):
return Arrow(2017, 11, 14, 21, 17), Arrow(2017, 11, 14, 22, 59)
mocker.patch('freqtrade.optimize.load_data', mocked_load_data)
mocker.patch('freqtrade.optimize.get_timeframe', get_timeframe)
mocker.patch('freqtrade.exchange.Exchange.refresh_tickers', MagicMock())
patch_exchange(mocker)
mocker.patch.multiple(
'freqtrade.optimize.backtesting.Backtesting',
backtest=MagicMock(),
_generate_text_table=MagicMock(return_value='1'),
get_timeframe=get_timeframe,
)
default_conf['exchange']['pair_whitelist'] = ['UNITTEST/BTC']
default_conf['ticker_interval'] = 1
default_conf['ticker_interval'] = "1m"
default_conf['live'] = False
default_conf['datadir'] = None
default_conf['export'] = None
@ -486,17 +470,17 @@ def test_backtesting_start(default_conf, mocker, caplog) -> None:
def test_backtesting_start_no_data(default_conf, mocker, caplog) -> None:
def get_timeframe(input1, input2):
def get_timeframe(input1):
return Arrow(2017, 11, 14, 21, 17), Arrow(2017, 11, 14, 22, 59)
mocker.patch('freqtrade.optimize.load_data', MagicMock(return_value={}))
mocker.patch('freqtrade.optimize.get_timeframe', get_timeframe)
mocker.patch('freqtrade.exchange.Exchange.refresh_tickers', MagicMock())
patch_exchange(mocker)
mocker.patch.multiple(
'freqtrade.optimize.backtesting.Backtesting',
backtest=MagicMock(),
_generate_text_table=MagicMock(return_value='1'),
get_timeframe=get_timeframe,
)
default_conf['exchange']['pair_whitelist'] = ['UNITTEST/BTC']
@ -520,7 +504,7 @@ def test_backtest(default_conf, fee, mocker) -> None:
pair = 'UNITTEST/BTC'
data = optimize.load_data(None, ticker_interval='5m', pairs=['UNITTEST/BTC'])
data = trim_dictlist(data, -200)
data_processed = backtesting.tickerdata_to_dataframe(data)
data_processed = backtesting.strategy.tickerdata_to_dataframe(data)
results = backtesting.backtest(
{
'stake_amount': default_conf['stake_amount'],
@ -571,7 +555,7 @@ def test_backtest_1min_ticker_interval(default_conf, fee, mocker) -> None:
results = backtesting.backtest(
{
'stake_amount': default_conf['stake_amount'],
'processed': backtesting.tickerdata_to_dataframe(data),
'processed': backtesting.strategy.tickerdata_to_dataframe(data),
'max_open_trades': 1,
'position_stacking': False
}
@ -585,7 +569,7 @@ def test_processed(default_conf, mocker) -> None:
backtesting = Backtesting(default_conf)
dict_of_tickerrows = load_data_test('raise')
dataframes = backtesting.tickerdata_to_dataframe(dict_of_tickerrows)
dataframes = backtesting.strategy.tickerdata_to_dataframe(dict_of_tickerrows)
dataframe = dataframes['UNITTEST/BTC']
cols = dataframe.columns
# assert the dataframe got some of the indicator columns

6
freqtrade/tests/optimize/test_hyperopt.py
View File

@ -194,7 +194,7 @@ def test_start_calls_optimizer(mocker, default_conf, caplog) -> None:
default_conf.update({'spaces': 'all'})
hyperopt = Hyperopt(default_conf)
hyperopt.tickerdata_to_dataframe = MagicMock()
hyperopt.strategy.tickerdata_to_dataframe = MagicMock()
hyperopt.start()
parallel.assert_called_once()
@ -242,7 +242,7 @@ def test_has_space(hyperopt):
def test_populate_indicators(hyperopt) -> None:
tick = load_tickerdata_file(None, 'UNITTEST/BTC', '1m')
tickerlist = {'UNITTEST/BTC': tick}
dataframes = hyperopt.tickerdata_to_dataframe(tickerlist)
dataframes = hyperopt.strategy.tickerdata_to_dataframe(tickerlist)
dataframe = hyperopt.populate_indicators(dataframes['UNITTEST/BTC'], {'pair': 'UNITTEST/BTC'})
# Check if some indicators are generated. We will not test all of them
@ -254,7 +254,7 @@ def test_populate_indicators(hyperopt) -> None:
def test_buy_strategy_generator(hyperopt) -> None:
tick = load_tickerdata_file(None, 'UNITTEST/BTC', '1m')
tickerlist = {'UNITTEST/BTC': tick}
dataframes = hyperopt.tickerdata_to_dataframe(tickerlist)
dataframes = hyperopt.strategy.tickerdata_to_dataframe(tickerlist)
dataframe = hyperopt.populate_indicators(dataframes['UNITTEST/BTC'], {'pair': 'UNITTEST/BTC'})
populate_buy_trend = hyperopt.buy_strategy_generator(

95
freqtrade/tests/optimize/test_optimize.py
View File

@ -7,7 +7,7 @@ from shutil import copyfile
import arrow
from freqtrade import optimize
from freqtrade import optimize, constants
from freqtrade.arguments import TimeRange
from freqtrade.misc import file_dump_json
from freqtrade.optimize.__init__ import (download_backtesting_testdata,
@ -15,7 +15,8 @@ from freqtrade.optimize.__init__ import (download_backtesting_testdata,
load_cached_data_for_updating,
load_tickerdata_file,
make_testdata_path, trim_tickerlist)
from freqtrade.tests.conftest import get_patched_exchange, log_has
from freqtrade.strategy.default_strategy import DefaultStrategy
from freqtrade.tests.conftest import get_patched_exchange, log_has, patch_exchange
# Change this if modifying UNITTEST/BTC testdatafile
_BTC_UNITTEST_LENGTH = 13681
@ -322,6 +323,38 @@ def test_load_tickerdata_file() -> None:
assert _BTC_UNITTEST_LENGTH == len(tickerdata)
def test_load_partial_missing(caplog) -> None:
# Make sure we start fresh - test missing data at start
start = arrow.get('2018-01-01T00:00:00')
end = arrow.get('2018-01-11T00:00:00')
tickerdata = optimize.load_data(None, '5m', ['UNITTEST/BTC'],
refresh_pairs=False,
timerange=TimeRange('date', 'date',
start.timestamp, end.timestamp))
# timedifference in 5 minutes
td = ((end - start).total_seconds() // 60 // 5) + 1
assert td != len(tickerdata['UNITTEST/BTC'])
start_real = arrow.get(tickerdata['UNITTEST/BTC'][0][0] / 1000)
assert log_has(f'Missing data at start for pair '
f'UNITTEST/BTC, data starts at {start_real.strftime("%Y-%m-%d %H:%M:%S")}',
caplog.record_tuples)
# Make sure we start fresh - test missing data at end
caplog.clear()
start = arrow.get('2018-01-10T00:00:00')
end = arrow.get('2018-02-20T00:00:00')
tickerdata = optimize.load_data(None, '5m', ['UNITTEST/BTC'],
refresh_pairs=False,
timerange=TimeRange('date', 'date',
start.timestamp, end.timestamp))
# timedifference in 5 minutes
td = ((end - start).total_seconds() // 60 // 5) + 1
assert td != len(tickerdata['UNITTEST/BTC'])
end_real = arrow.get(tickerdata['UNITTEST/BTC'][-1][0] / 1000)
assert log_has(f'Missing data at end for pair '
f'UNITTEST/BTC, data ends at {end_real.strftime("%Y-%m-%d %H:%M:%S")}',
caplog.record_tuples)
def test_init(default_conf, mocker) -> None:
exchange = get_patched_exchange(mocker, default_conf)
assert {} == optimize.load_data(
@ -433,3 +466,61 @@ def test_file_dump_json() -> None:
# Remove the file
_clean_test_file(file)
def test_get_timeframe(default_conf, mocker) -> None:
patch_exchange(mocker)
strategy = DefaultStrategy(default_conf)
data = strategy.tickerdata_to_dataframe(
optimize.load_data(
None,
ticker_interval='1m',
pairs=['UNITTEST/BTC']
)
)
min_date, max_date = optimize.get_timeframe(data)
assert min_date.isoformat() == '2017-11-04T23:02:00+00:00'
assert max_date.isoformat() == '2017-11-14T22:58:00+00:00'
def test_validate_backtest_data_warn(default_conf, mocker, caplog) -> None:
patch_exchange(mocker)
strategy = DefaultStrategy(default_conf)
data = strategy.tickerdata_to_dataframe(
optimize.load_data(
None,
ticker_interval='1m',
pairs=['UNITTEST/BTC']
)
)
min_date, max_date = optimize.get_timeframe(data)
caplog.clear()
assert optimize.validate_backtest_data(data, min_date, max_date,
constants.TICKER_INTERVAL_MINUTES["1m"])
assert len(caplog.record_tuples) == 1
assert log_has(
"UNITTEST/BTC has missing frames: expected 14396, got 13680, that's 716 missing values",
caplog.record_tuples)
def test_validate_backtest_data(default_conf, mocker, caplog) -> None:
patch_exchange(mocker)
strategy = DefaultStrategy(default_conf)
timerange = TimeRange('index', 'index', 200, 250)
data = strategy.tickerdata_to_dataframe(
optimize.load_data(
None,
ticker_interval='5m',
pairs=['UNITTEST/BTC'],
timerange=timerange
)
)
min_date, max_date = optimize.get_timeframe(data)
caplog.clear()
assert not optimize.validate_backtest_data(data, min_date, max_date,
constants.TICKER_INTERVAL_MINUTES["5m"])
assert len(caplog.record_tuples) == 0

47
freqtrade/tests/rpc/test_rpc.py
View File

@ -5,8 +5,9 @@ from datetime import datetime
from unittest.mock import MagicMock, ANY
import pytest
from numpy import isnan
from freqtrade import TemporaryError
from freqtrade import TemporaryError, DependencyException
from freqtrade.fiat_convert import CryptoToFiatConverter
from freqtrade.freqtradebot import FreqtradeBot
from freqtrade.persistence import Trade
@ -61,6 +62,27 @@ def test_rpc_trade_status(default_conf, ticker, fee, markets, mocker) -> None:
'open_order': '(limit buy rem=0.00000000)'
} == results[0]
mocker.patch('freqtrade.exchange.Exchange.get_ticker',
MagicMock(side_effect=DependencyException(f"Pair 'ETH/BTC' not available")))
# invalidate ticker cache
rpc._freqtrade.exchange._cached_ticker = {}
results = rpc._rpc_trade_status()
assert isnan(results[0]['current_profit'])
assert isnan(results[0]['current_rate'])
assert {
'trade_id': 1,
'pair': 'ETH/BTC',
'market_url': 'https://bittrex.com/Market/Index?MarketName=BTC-ETH',
'date': ANY,
'open_rate': 1.099e-05,
'close_rate': None,
'current_rate': ANY,
'amount': 90.99181074,
'close_profit': None,
'current_profit': ANY,
'open_order': '(limit buy rem=0.00000000)'
} == results[0]
def test_rpc_status_table(default_conf, ticker, fee, markets, mocker) -> None:
patch_coinmarketcap(mocker)
@ -87,6 +109,15 @@ def test_rpc_status_table(default_conf, ticker, fee, markets, mocker) -> None:
assert 'ETH/BTC' in result['Pair'].all()
assert '-0.59%' in result['Profit'].all()
mocker.patch('freqtrade.exchange.Exchange.get_ticker',
MagicMock(side_effect=DependencyException(f"Pair 'ETH/BTC' not available")))
# invalidate ticker cache
rpc._freqtrade.exchange._cached_ticker = {}
result = rpc._rpc_status_table()
assert 'just now' in result['Since'].all()
assert 'ETH/BTC' in result['Pair'].all()
assert 'nan%' in result['Profit'].all()
def test_rpc_daily_profit(default_conf, update, ticker, fee,
limit_buy_order, limit_sell_order, markets, mocker) -> None:
@ -208,6 +239,20 @@ def test_rpc_trade_statistics(default_conf, ticker, ticker_sell_up, fee,
assert stats['best_pair'] == 'ETH/BTC'
assert prec_satoshi(stats['best_rate'], 6.2)
# Test non-available pair
mocker.patch('freqtrade.exchange.Exchange.get_ticker',
MagicMock(side_effect=DependencyException(f"Pair 'ETH/BTC' not available")))
# invalidate ticker cache
rpc._freqtrade.exchange._cached_ticker = {}
stats = rpc._rpc_trade_statistics(stake_currency, fiat_display_currency)
assert stats['trade_count'] == 2
assert stats['first_trade_date'] == 'just now'
assert stats['latest_trade_date'] == 'just now'
assert stats['avg_duration'] == '0:00:00'
assert stats['best_pair'] == 'ETH/BTC'
assert prec_satoshi(stats['best_rate'], 6.2)
assert isnan(stats['profit_all_coin'])
# Test that rpc_trade_statistics can handle trades that lacks
# trade.open_rate (it is set to None)

46
freqtrade/tests/test_freqtradebot.py
View File

@ -764,6 +764,52 @@ def test_process_trade_handling(
assert result is False
def test_process_trade_no_whitelist_pair(
default_conf, ticker, limit_buy_order, markets, fee, mocker) -> None:
""" Test _process with trade not in pair list """
patch_RPCManager(mocker)
patch_exchange(mocker)
mocker.patch.multiple(
'freqtrade.exchange.Exchange',
get_ticker=ticker,
get_markets=markets,
buy=MagicMock(return_value={'id': limit_buy_order['id']}),
get_order=MagicMock(return_value=limit_buy_order),
get_fee=fee,
)
freqtrade = FreqtradeBot(default_conf)
patch_get_signal(freqtrade)
pair = 'NOCLUE/BTC'
# create open trade not in whitelist
Trade.session.add(Trade(
pair=pair,
stake_amount=0.001,
fee_open=fee.return_value,
fee_close=fee.return_value,
is_open=True,
amount=20,
open_rate=0.01,
exchange='bittrex',
))
Trade.session.add(Trade(
pair='ETH/BTC',
stake_amount=0.001,
fee_open=fee.return_value,
fee_close=fee.return_value,
is_open=True,
amount=12,
open_rate=0.001,
exchange='bittrex',
))
assert pair not in freqtrade.active_pair_whitelist
result = freqtrade._process()
assert pair in freqtrade.active_pair_whitelist
# Make sure each pair is only in the list once
assert len(freqtrade.active_pair_whitelist) == len(set(freqtrade.active_pair_whitelist))
assert result is True
def test_balance_fully_ask_side(mocker, default_conf) -> None:
default_conf['bid_strategy']['ask_last_balance'] = 0.0
freqtrade = get_patched_freqtradebot(mocker, default_conf)

6
requirements.txt
View File

@ -1,5 +1,5 @@
ccxt==1.17.411
SQLAlchemy==1.2.12
ccxt==1.17.455
SQLAlchemy==1.2.13
python-telegram-bot==11.1.0
arrow==0.12.1
cachetools==2.1.0
@ -12,7 +12,7 @@ scipy==1.1.0
jsonschema==2.6.0
numpy==1.15.3
TA-Lib==0.4.17
pytest==3.9.2
pytest==3.9.3
pytest-mock==1.10.0
pytest-asyncio==0.9.0
pytest-cov==2.6.0

381
scripts/plot_dataframe.py
View File

@ -1,381 +0,0 @@
#!/usr/bin/env python3
"""
Script to display when the bot will buy a specific pair
Mandatory Cli parameters:
-p / --pair: pair to examine
Option but recommended
-s / --strategy: strategy to use
Optional Cli parameters
-d / --datadir: path to pair backtest data
--timerange: specify what timerange of data to use.
-l / --live: Live, to download the latest ticker for the pair
-db / --db-url: Show trades stored in database
Indicators recommended
Row 1: sma, ema3, ema5, ema10, ema50
Row 3: macd, rsi, fisher_rsi, mfi, slowd, slowk, fastd, fastk
Example of usage:
> python3 scripts/plot_dataframe.py --pair BTC/EUR -d user_data/data/ --indicators1 sma,ema3
--indicators2 fastk,fastd
"""
import json
import logging
import sys
from argparse import Namespace
from pathlib import Path
from typing import Dict, List, Any
import pandas as pd
import plotly.graph_objs as go
import pytz
from plotly import tools
from plotly.offline import plot
import freqtrade.optimize as optimize
from freqtrade import persistence
from freqtrade.arguments import Arguments, TimeRange
from freqtrade.exchange import Exchange
from freqtrade.optimize.backtesting import setup_configuration
from freqtrade.persistence import Trade
from freqtrade.strategy.resolver import StrategyResolver
logger = logging.getLogger(__name__)
_CONF: Dict[str, Any] = {}
timeZone = pytz.UTC
def load_trades(args: Namespace, pair: str, timerange: TimeRange) -> pd.DataFrame:
trades: pd.DataFrame = pd.DataFrame()
if args.db_url:
persistence.init(_CONF)
columns = ["pair", "profit", "opents", "closets", "open_rate", "close_rate", "duration"]
for x in Trade.query.all():
print("date: {}".format(x.open_date))
trades = pd.DataFrame([(t.pair, t.calc_profit(),
t.open_date.replace(tzinfo=timeZone),
t.close_date.replace(tzinfo=timeZone) if t.close_date else None,
t.open_rate, t.close_rate,
t.close_date.timestamp() - t.open_date.timestamp() if t.close_date else None)
for t in Trade.query.filter(Trade.pair.is_(pair)).all()],
columns=columns)
elif args.exportfilename:
file = Path(args.exportfilename)
# must align with columns in backtest.py
columns = ["pair", "profit", "opents", "closets", "index", "duration",
"open_rate", "close_rate", "open_at_end", "sell_reason"]
with file.open() as f:
data = json.load(f)
trades = pd.DataFrame(data, columns=columns)
trades = trades.loc[trades["pair"] == pair]
if timerange:
if timerange.starttype == 'date':
trades = trades.loc[trades["opents"] >= timerange.startts]
if timerange.stoptype == 'date':
trades = trades.loc[trades["opents"] <= timerange.stopts]
trades['opents'] = pd.to_datetime(trades['opents'],
unit='s',
utc=True,
infer_datetime_format=True)
trades['closets'] = pd.to_datetime(trades['closets'],
unit='s',
utc=True,
infer_datetime_format=True)
return trades
def plot_analyzed_dataframe(args: Namespace) -> None:
"""
Calls analyze() and plots the returned dataframe
:return: None
"""
global _CONF
# Load the configuration
_CONF.update(setup_configuration(args))
print(_CONF)
# Set the pair to audit
pair = args.pair
if pair is None:
logger.critical('Parameter --pair mandatory;. E.g --pair ETH/BTC')
exit()
if '/' not in pair:
logger.critical('--pair format must be XXX/YYY')
exit()
# Set timerange to use
timerange = Arguments.parse_timerange(args.timerange)
# Load the strategy
try:
strategy = StrategyResolver(_CONF).strategy
exchange = Exchange(_CONF)
except AttributeError:
logger.critical(
'Impossible to load the strategy. Please check the file "user_data/strategies/%s.py"',
args.strategy
)
exit()
# Set the ticker to use
tick_interval = strategy.ticker_interval
# Load pair tickers
tickers = {}
if args.live:
logger.info('Downloading pair.')
exchange.refresh_tickers([pair], tick_interval)
tickers[pair] = exchange.klines[pair]
else:
tickers = optimize.load_data(
datadir=_CONF.get("datadir"),
pairs=[pair],
ticker_interval=tick_interval,
refresh_pairs=_CONF.get('refresh_pairs', False),
timerange=timerange,
exchange=Exchange(_CONF)
)
# No ticker found, or impossible to download
if tickers == {}:
exit()
# Get trades already made from the DB
trades = load_trades(args, pair, timerange)
dataframes = strategy.tickerdata_to_dataframe(tickers)
dataframe = dataframes[pair]
dataframe = strategy.advise_buy(dataframe, {'pair': pair})
dataframe = strategy.advise_sell(dataframe, {'pair': pair})
if len(dataframe.index) > args.plot_limit:
logger.warning('Ticker contained more than %s candles as defined '
'with --plot-limit, clipping.', args.plot_limit)
dataframe = dataframe.tail(args.plot_limit)
trades = trades.loc[trades['opents'] >= dataframe.iloc[0]['date']]
fig = generate_graph(
pair=pair,
trades=trades,
data=dataframe,
args=args
)
plot(fig, filename=str(Path('user_data').joinpath('freqtrade-plot.html')))
def generate_graph(pair, trades: pd.DataFrame, data: pd.DataFrame, args) -> tools.make_subplots:
"""
Generate the graph from the data generated by Backtesting or from DB
:param pair: Pair to Display on the graph
:param trades: All trades created
:param data: Dataframe
:param args: sys.argv that contrains the two params indicators1, and indicators2
:return: None
"""
# Define the graph
fig = tools.make_subplots(
rows=3,
cols=1,
shared_xaxes=True,
row_width=[1, 1, 4],
vertical_spacing=0.0001,
)
fig['layout'].update(title=pair)
fig['layout']['yaxis1'].update(title='Price')
fig['layout']['yaxis2'].update(title='Volume')
fig['layout']['yaxis3'].update(title='Other')
# Common information
candles = go.Candlestick(
x=data.date,
open=data.open,
high=data.high,
low=data.low,
close=data.close,
name='Price'
)
df_buy = data[data['buy'] == 1]
buys = go.Scattergl(
x=df_buy.date,
y=df_buy.close,
mode='markers',
name='buy',
marker=dict(
symbol='triangle-up-dot',
size=9,
line=dict(width=1),
color='green',
)
)
df_sell = data[data['sell'] == 1]
sells = go.Scattergl(
x=df_sell.date,
y=df_sell.close,
mode='markers',
name='sell',
marker=dict(
symbol='triangle-down-dot',
size=9,
line=dict(width=1),
color='red',
)
)
trade_buys = go.Scattergl(
x=trades["opents"],
y=trades["open_rate"],
mode='markers',
name='trade_buy',
marker=dict(
symbol='square-open',
size=11,
line=dict(width=2),
color='green'
)
)
trade_sells = go.Scattergl(
x=trades["closets"],
y=trades["close_rate"],
mode='markers',
name='trade_sell',
marker=dict(
symbol='square-open',
size=11,
line=dict(width=2),
color='red'
)
)
# Row 1
fig.append_trace(candles, 1, 1)
if 'bb_lowerband' in data and 'bb_upperband' in data:
bb_lower = go.Scatter(
x=data.date,
y=data.bb_lowerband,
name='BB lower',
line={'color': 'rgba(255,255,255,0)'},
)
bb_upper = go.Scatter(
x=data.date,
y=data.bb_upperband,
name='BB upper',
fill="tonexty",
fillcolor="rgba(0,176,246,0.2)",
line={'color': 'rgba(255,255,255,0)'},
)
fig.append_trace(bb_lower, 1, 1)
fig.append_trace(bb_upper, 1, 1)
fig = generate_row(fig=fig, row=1, raw_indicators=args.indicators1, data=data)
fig.append_trace(buys, 1, 1)
fig.append_trace(sells, 1, 1)
fig.append_trace(trade_buys, 1, 1)
fig.append_trace(trade_sells, 1, 1)
# Row 2
volume = go.Bar(
x=data['date'],
y=data['volume'],
name='Volume'
)
fig.append_trace(volume, 2, 1)
# Row 3
fig = generate_row(fig=fig, row=3, raw_indicators=args.indicators2, data=data)
return fig
def generate_row(fig, row, raw_indicators, data) -> tools.make_subplots:
"""
Generator all the indicator selected by the user for a specific row
"""
for indicator in raw_indicators.split(','):
if indicator in data:
scattergl = go.Scattergl(
x=data['date'],
y=data[indicator],
name=indicator
)
fig.append_trace(scattergl, row, 1)
else:
logger.info(
'Indicator "%s" ignored. Reason: This indicator is not found '
'in your strategy.',
indicator
)
return fig
def plot_parse_args(args: List[str]) -> Namespace:
"""
Parse args passed to the script
:param args: Cli arguments
:return: args: Array with all arguments
"""
arguments = Arguments(args, 'Graph dataframe')
arguments.scripts_options()
arguments.parser.add_argument(
'--indicators1',
help='Set indicators from your strategy you want in the first row of the graph. Separate '
'them with a coma. E.g: ema3,ema5 (default: %(default)s)',
type=str,
default='sma,ema3,ema5',
dest='indicators1',
)
arguments.parser.add_argument(
'--indicators2',
help='Set indicators from your strategy you want in the third row of the graph. Separate '
'them with a coma. E.g: fastd,fastk (default: %(default)s)',
type=str,
default='macd',
dest='indicators2',
)
arguments.parser.add_argument(
'--plot-limit',
help='Specify tick limit for plotting - too high values cause huge files - '
'Default: %(default)s',
dest='plot_limit',
default=750,
type=int,
)
arguments.common_args_parser()
arguments.optimizer_shared_options(arguments.parser)
arguments.backtesting_options(arguments.parser)
return arguments.parse_args()
def main(sysargv: List[str]) -> None:
"""
This function will initiate the bot and start the trading loop.
:return: None
"""
logger.info('Starting Plot Dataframe')
plot_analyzed_dataframe(
plot_parse_args(sysargv)
)
if __name__ == '__main__':
main(sys.argv[1:])