adelorenzo/stable
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Merge 93091e9b22 into 721fb3e326

Browse Source
This commit is contained in:
Matthias 2018-08-14 12:35:29 +00:00 committed by GitHub
commit f67ee6171d
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
32 changed files with 866 additions and 526 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

1
Dockerfile
View File

@ -5,6 +5,7 @@ RUN apt-get update && apt-get -y install curl build-essential && apt-get clean
RUN curl -L http://prdownloads.sourceforge.net/ta-lib/ta-lib-0.4.0-src.tar.gz | \ RUN curl -L http://prdownloads.sourceforge.net/ta-lib/ta-lib-0.4.0-src.tar.gz | \
tar xzvf - && \ tar xzvf - && \
cd ta-lib && \ cd ta-lib && \
sed -i "s|0.00000001|0.000000000000000001 |g" src/ta_func/ta_utility.h && \
./configure && make && make install && \ ./configure && make && make install && \
cd .. && rm -rf ta-lib cd .. && rm -rf ta-lib
ENV LD_LIBRARY_PATH /usr/local/lib ENV LD_LIBRARY_PATH /usr/local/lib

9
README.md
View File

@ -24,7 +24,7 @@ hesitate to read the source code and understand the mechanism of this bot.
## Exchange marketplaces supported ## Exchange marketplaces supported
- [X] [Bittrex](https://bittrex.com/) - [X] [Bittrex](https://bittrex.com/)
- [X] [Binance](https://www.binance.com/) - [X] [Binance](https://www.binance.com/) ([*Note for binance users](#a-note-on-binance))
- [ ] [113 others to tests](https://github.com/ccxt/ccxt/). _(We cannot guarantee they will work)_ - [ ] [113 others to tests](https://github.com/ccxt/ccxt/). _(We cannot guarantee they will work)_
## Features ## Features
@ -152,6 +152,13 @@ The project is currently setup in two main branches:
- `develop` - This branch has often new features, but might also cause breaking changes. - `develop` - This branch has often new features, but might also cause breaking changes.
- `master` - This branch contains the latest stable release. The bot 'should' be stable on this branch, and is generally well tested. - `master` - This branch contains the latest stable release. The bot 'should' be stable on this branch, and is generally well tested.
- `feat/*` - These are feature branches, which are beeing worked on heavily. Please don't use these unless you want to test a specific feature.
## A note on Binance
For Binance, please add `"BNB/<STAKE>"` to your blacklist to avoid issues.
Accounts having BNB accounts use this to pay for fees - if your first trade happens to be on `BNB`, further trades will consume this position and make the initial BNB order unsellable as the expected amount is not there anymore.
## Support ## Support

27
docs/backtesting.md
View File

@ -151,7 +151,7 @@ cp freqtrade/tests/testdata/pairs.json user_data/data/binance
Then run: Then run:
```bash ```bash
python scripts/download_backtest_data --exchange binance python scripts/download_backtest_data.py --exchange binance
``` ```
This will download ticker data for all the currency pairs you defined in `pairs.json`. This will download ticker data for all the currency pairs you defined in `pairs.json`.
@ -238,6 +238,31 @@ On the other hand, if you set a too high `minimal_roi` like `"0": 0.55`
profit. Hence, keep in mind that your performance is a mix of your profit. Hence, keep in mind that your performance is a mix of your
strategies, your configuration, and the crypto-currency you have set up. strategies, your configuration, and the crypto-currency you have set up.
## Backtesting multiple strategies
To backtest multiple strategies, a list of Strategies can be provided.
This is limited to 1 ticker-interval per run, however, data is only loaded once from disk so if you have multiple
strategies you'd like to compare, this should give a nice runtime boost.
All listed Strategies need to be in the same folder.
``` bash
freqtrade backtesting --timerange 20180401-20180410 --ticker-interval 5m --strategy-list Strategy001 Strategy002 --export trades
```
This will save the results to `user_data/backtest_data/backtest-result-<strategy>.json`, injecting the strategy-name into the target filename.
There will be an additional table comparing win/losses of the different strategies (identical to the "Total" row in the first table).
Detailed output for all strategies one after the other will be available, so make sure to scroll up.
```
=================================================== Strategy Summary ====================================================
| Strategy | buy count | avg profit % | cum profit % | total profit ETH | avg duration | profit | loss |
|:-----------|------------:|---------------:|---------------:|-------------------:|:----------------|---------:|-------:|
| Strategy1 | 19 | -0.76 | -14.39 | -0.01440287 | 15:48:00 | 15 | 4 |
| Strategy2 | 6 | -2.73 | -16.40 | -0.01641299 | 1 day, 14:12:00 | 3 | 3 |
```
## Next step ## Next step
Great, your strategy is profitable. What if the bot can give your the Great, your strategy is profitable. What if the bot can give your the

33
docs/bot-usage.md
View File

@ -1,13 +1,15 @@
# Bot usage # Bot usage
This page explains the difference parameters of the bot and how to run
it. This page explains the difference parameters of the bot and how to run it.
## Table of Contents ## Table of Contents
- [Bot commands](#bot-commands) - [Bot commands](#bot-commands)
- [Backtesting commands](#backtesting-commands) - [Backtesting commands](#backtesting-commands)
- [Hyperopt commands](#hyperopt-commands) - [Hyperopt commands](#hyperopt-commands)
## Bot commands ## Bot commands
``` ```
usage: freqtrade [-h] [-v] [--version] [-c PATH] [-d PATH] [-s NAME] usage: freqtrade [-h] [-v] [--version] [-c PATH] [-d PATH] [-s NAME]
[--strategy-path PATH] [--dynamic-whitelist [INT]] [--strategy-path PATH] [--dynamic-whitelist [INT]]
@ -41,6 +43,7 @@ optional arguments:
``` ```
### How to use a different config file? ### How to use a different config file?
The bot allows you to select which config file you want to use. Per The bot allows you to select which config file you want to use. Per
default, the bot will load the file `./config.json` default, the bot will load the file `./config.json`
@ -49,6 +52,7 @@ python3 ./freqtrade/main.py -c path/far/far/away/config.json
``` ```
### How to use --strategy? ### How to use --strategy?
This parameter will allow you to load your custom strategy class. This parameter will allow you to load your custom strategy class.
Per default without `--strategy` or `-s` the bot will load the Per default without `--strategy` or `-s` the bot will load the
`DefaultStrategy` included with the bot (`freqtrade/strategy/default_strategy.py`). `DefaultStrategy` included with the bot (`freqtrade/strategy/default_strategy.py`).
@ -60,6 +64,7 @@ To load a strategy, simply pass the class name (e.g.: `CustomStrategy`) in this
**Example:** **Example:**
In `user_data/strategies` you have a file `my_awesome_strategy.py` which has In `user_data/strategies` you have a file `my_awesome_strategy.py` which has
a strategy class called `AwesomeStrategy` to load it: a strategy class called `AwesomeStrategy` to load it:
```bash ```bash
python3 ./freqtrade/main.py --strategy AwesomeStrategy python3 ./freqtrade/main.py --strategy AwesomeStrategy
``` ```
@ -70,6 +75,7 @@ message the reason (File not found, or errors in your code).
Learn more about strategy file in [optimize your bot](https://github.com/freqtrade/freqtrade/blob/develop/docs/bot-optimization.md). Learn more about strategy file in [optimize your bot](https://github.com/freqtrade/freqtrade/blob/develop/docs/bot-optimization.md).
### How to use --strategy-path? ### How to use --strategy-path?
This parameter allows you to add an additional strategy lookup path, which gets This parameter allows you to add an additional strategy lookup path, which gets
checked before the default locations (The passed path must be a folder!): checked before the default locations (The passed path must be a folder!):
```bash ```bash
@ -77,21 +83,25 @@ python3 ./freqtrade/main.py --strategy AwesomeStrategy --strategy-path /some/fol
``` ```
#### How to install a strategy? #### How to install a strategy?
This is very simple. Copy paste your strategy file into the folder This is very simple. Copy paste your strategy file into the folder
`user_data/strategies` or use `--strategy-path`. And voila, the bot is ready to use it. `user_data/strategies` or use `--strategy-path`. And voila, the bot is ready to use it.
### How to use --dynamic-whitelist? ### How to use --dynamic-whitelist?
Per default `--dynamic-whitelist` will retrieve the 20 currencies based Per default `--dynamic-whitelist` will retrieve the 20 currencies based
on BaseVolume. This value can be changed when you run the script. on BaseVolume. This value can be changed when you run the script.
**By Default** **By Default**
Get the 20 currencies based on BaseVolume. Get the 20 currencies based on BaseVolume.
```bash ```bash
python3 ./freqtrade/main.py --dynamic-whitelist python3 ./freqtrade/main.py --dynamic-whitelist
``` ```
**Customize the number of currencies to retrieve** **Customize the number of currencies to retrieve**
Get the 30 currencies based on BaseVolume. Get the 30 currencies based on BaseVolume.
```bash ```bash
python3 ./freqtrade/main.py --dynamic-whitelist 30 python3 ./freqtrade/main.py --dynamic-whitelist 30
``` ```
@ -102,6 +112,7 @@ negative value (e.g -2), `--dynamic-whitelist` will use the default
value (20). value (20).
### How to use --db-url? ### How to use --db-url?
When you run the bot in Dry-run mode, per default no transactions are When you run the bot in Dry-run mode, per default no transactions are
stored in a database. If you want to store your bot actions in a DB stored in a database. If you want to store your bot actions in a DB
using `--db-url`. This can also be used to specify a custom database using `--db-url`. This can also be used to specify a custom database
@ -111,14 +122,14 @@ in production mode. Example command:
python3 ./freqtrade/main.py -c config.json --db-url sqlite:///tradesv3.dry_run.sqlite python3 ./freqtrade/main.py -c config.json --db-url sqlite:///tradesv3.dry_run.sqlite
``` ```
## Backtesting commands ## Backtesting commands
Backtesting also uses the config specified via `-c/--config`. Backtesting also uses the config specified via `-c/--config`.
``` ```
usage: main.py backtesting [-h] [-i TICKER_INTERVAL] [--eps] [--dmmp] usage: freqtrade backtesting [-h] [-i TICKER_INTERVAL] [--eps] [--dmmp]
[--timerange TIMERANGE] [-l] [-r] [--timerange TIMERANGE] [-l] [-r]
[--strategy-list STRATEGY_LIST [STRATEGY_LIST ...]]
[--export EXPORT] [--export-filename PATH] [--export EXPORT] [--export-filename PATH]
optional arguments: optional arguments:
@ -139,6 +150,13 @@ optional arguments:
refresh the pairs files in tests/testdata with the refresh the pairs files in tests/testdata with the
latest data from the exchange. Use it if you want to latest data from the exchange. Use it if you want to
run your backtesting with up-to-date data. run your backtesting with up-to-date data.
--strategy-list STRATEGY_LIST [STRATEGY_LIST ...]
Provide a commaseparated list of strategies to
backtest Please note that ticker-interval needs to be
set either in config or via command line. When using
this together with --export trades, the strategy-name
is injected into the filename (so backtest-data.json
becomes backtest-data-DefaultStrategy.json
--export EXPORT export backtest results, argument are: trades Example --export EXPORT export backtest results, argument are: trades Example
--export=trades --export=trades
--export-filename PATH --export-filename PATH
@ -151,6 +169,7 @@ optional arguments:
``` ```
### How to use --refresh-pairs-cached parameter? ### How to use --refresh-pairs-cached parameter?
The first time your run Backtesting, it will take the pairs you have The first time your run Backtesting, it will take the pairs you have
set in your config file and download data from Bittrex. set in your config file and download data from Bittrex.
@ -162,7 +181,6 @@ to come back to the previous version.**
To test your strategy with latest data, we recommend continuing using To test your strategy with latest data, we recommend continuing using
the parameter `-l` or `--live`. the parameter `-l` or `--live`.
## Hyperopt commands ## Hyperopt commands
To optimize your strategy, you can use hyperopt parameter hyperoptimization To optimize your strategy, you can use hyperopt parameter hyperoptimization
@ -194,10 +212,11 @@ optional arguments:
``` ```
## A parameter missing in the configuration? ## A parameter missing in the configuration?
All parameters for `main.py`, `backtesting`, `hyperopt` are referenced All parameters for `main.py`, `backtesting`, `hyperopt` are referenced
in [misc.py](https://github.com/freqtrade/freqtrade/blob/develop/freqtrade/misc.py#L84) in [misc.py](https://github.com/freqtrade/freqtrade/blob/develop/freqtrade/misc.py#L84)
## Next step ## Next step
The optimal strategy of the bot will change with time depending of the
market trends. The next step is to The optimal strategy of the bot will change with time depending of the market trends. The next step is to
[optimize your bot](https://github.com/freqtrade/freqtrade/blob/develop/docs/bot-optimization.md). [optimize your bot](https://github.com/freqtrade/freqtrade/blob/develop/docs/bot-optimization.md).

1
docs/installation.md
View File

@ -267,6 +267,7 @@ Official webpage: https://mrjbq7.github.io/ta-lib/install.html
wget http://prdownloads.sourceforge.net/ta-lib/ta-lib-0.4.0-src.tar.gz wget http://prdownloads.sourceforge.net/ta-lib/ta-lib-0.4.0-src.tar.gz
tar xvzf ta-lib-0.4.0-src.tar.gz tar xvzf ta-lib-0.4.0-src.tar.gz
cd ta-lib cd ta-lib
sed -i "s|0.00000001|0.000000000000000001 |g" src/ta_func/ta_utility.h
./configure --prefix=/usr ./configure --prefix=/usr
make make
make install make install

26
freqtrade/arguments.py
View File

@ -142,6 +142,16 @@ class Arguments(object):
action='store_true', action='store_true',
dest='refresh_pairs', dest='refresh_pairs',
) )
parser.add_argument(
'--strategy-list',
help='Provide a commaseparated list of strategies to backtest '
'Please note that ticker-interval needs to be set either in config '
'or via command line. When using this together with --export trades, '
'the strategy-name is injected into the filename '
'(so backtest-data.json becomes backtest-data-DefaultStrategy.json',
nargs='+',
dest='strategy_list',
)
parser.add_argument( parser.add_argument(
'--export', '--export',
help='export backtest results, argument are: trades\ help='export backtest results, argument are: trades\
@ -161,14 +171,6 @@ class Arguments(object):
dest='exportfilename', dest='exportfilename',
metavar='PATH', metavar='PATH',
) )
parser.add_argument(
'--backslap',
help="Utilize the Backslapping approach instead of the default Backtesting. This should provide more "
"accurate results, unless you are utilizing Min/Max function in your strategy.",
required=False,
dest='backslap',
action='store_true'
)
@staticmethod @staticmethod
def optimizer_shared_options(parser: argparse.ArgumentParser) -> None: def optimizer_shared_options(parser: argparse.ArgumentParser) -> None:
@ -236,7 +238,7 @@ class Arguments(object):
Builds and attaches all subcommands Builds and attaches all subcommands
:return: None :return: None
""" """
from freqtrade.optimize import backtesting, hyperopt from freqtrade.optimize import backtesting, backslapping, hyperopt
subparsers = self.parser.add_subparsers(dest='subparser') subparsers = self.parser.add_subparsers(dest='subparser')
@ -246,6 +248,12 @@ class Arguments(object):
self.optimizer_shared_options(backtesting_cmd) self.optimizer_shared_options(backtesting_cmd)
self.backtesting_options(backtesting_cmd) self.backtesting_options(backtesting_cmd)
# Add backslapping subcommand
backslapping_cmd = subparsers.add_parser('backslapping', help='backslapping module')
backslapping_cmd.set_defaults(func=backslapping.start)
self.optimizer_shared_options(backslapping_cmd)
self.backtesting_options(backslapping_cmd)
# Add hyperopt subcommand # Add hyperopt subcommand
hyperopt_cmd = subparsers.add_parser('hyperopt', help='hyperopt module') hyperopt_cmd = subparsers.add_parser('hyperopt', help='hyperopt module')
hyperopt_cmd.set_defaults(func=hyperopt.start) hyperopt_cmd.set_defaults(func=hyperopt.start)

8
freqtrade/configuration.py
View File

@ -187,6 +187,14 @@ class Configuration(object):
config.update({'refresh_pairs': True}) config.update({'refresh_pairs': True})
logger.info('Parameter -r/--refresh-pairs-cached detected ...') logger.info('Parameter -r/--refresh-pairs-cached detected ...')
if 'strategy_list' in self.args and self.args.strategy_list:
config.update({'strategy_list': self.args.strategy_list})
logger.info('Using strategy list of %s Strategies', len(self.args.strategy_list))
if 'ticker_interval' in self.args and self.args.ticker_interval:
config.update({'ticker_interval': self.args.ticker_interval})
logger.info('Overriding ticker interval with Command line argument')
# If --export is used we add it to the configuration # If --export is used we add it to the configuration
if 'export' in self.args and self.args.export: if 'export' in self.args and self.args.export:
config.update({'export': self.args.export}) config.update({'export': self.args.export})

4
freqtrade/constants.py
View File

@ -36,7 +36,7 @@ SUPPORTED_FIAT = [
"EUR", "GBP", "HKD", "HUF", "IDR", "ILS", "INR", "JPY", "EUR", "GBP", "HKD", "HUF", "IDR", "ILS", "INR", "JPY",
"KRW", "MXN", "MYR", "NOK", "NZD", "PHP", "PKR", "PLN", "KRW", "MXN", "MYR", "NOK", "NZD", "PHP", "PKR", "PLN",
"RUB", "SEK", "SGD", "THB", "TRY", "TWD", "ZAR", "USD", "RUB", "SEK", "SGD", "THB", "TRY", "TWD", "ZAR", "USD",
"BTC", "ETH", "XRP", "LTC", "BCH", "USDT" "BTC", "XBT", "ETH", "XRP", "LTC", "BCH", "USDT"
] ]
# Required json-schema for user specified config # Required json-schema for user specified config
@ -45,7 +45,7 @@ CONF_SCHEMA = {
'properties': { 'properties': {
'max_open_trades': {'type': 'integer', 'minimum': 0}, 'max_open_trades': {'type': 'integer', 'minimum': 0},
'ticker_interval': {'type': 'string', 'enum': list(TICKER_INTERVAL_MINUTES.keys())}, 'ticker_interval': {'type': 'string', 'enum': list(TICKER_INTERVAL_MINUTES.keys())},
'stake_currency': {'type': 'string', 'enum': ['BTC', 'ETH', 'USDT', 'EUR', 'USD']}, 'stake_currency': {'type': 'string', 'enum': ['BTC', 'XBT', 'ETH', 'USDT', 'EUR', 'USD']},
'stake_amount': { 'stake_amount': {
"type": ["number", "string"], "type": ["number", "string"],
"minimum": 0.0005, "minimum": 0.0005,

2
freqtrade/exchange/__init__.py
View File

@ -330,7 +330,7 @@ class Exchange(object):
return self._cached_ticker[pair] return self._cached_ticker[pair]
@retrier @retrier
def get_ticker_history(self, pair: str, tick_interval: str, def get_candle_history(self, pair: str, tick_interval: str,
since_ms: Optional[int] = None) -> List[Dict]: since_ms: Optional[int] = None) -> List[Dict]:
try: try:
# last item should be in the time interval [now - tick_interval, now] # last item should be in the time interval [now - tick_interval, now]

2
freqtrade/exchange/exchange_helpers.py
View File

@ -10,7 +10,7 @@ logger = logging.getLogger(__name__)
def parse_ticker_dataframe(ticker: list) -> DataFrame: def parse_ticker_dataframe(ticker: list) -> DataFrame:
""" """
Analyses the trend for the given ticker history Analyses the trend for the given ticker history
:param ticker: See exchange.get_ticker_history :param ticker: See exchange.get_candle_history
:return: DataFrame :return: DataFrame
""" """
cols = ['date', 'open', 'high', 'low', 'close', 'volume'] cols = ['date', 'open', 'high', 'low', 'close', 'volume']

4
freqtrade/freqtradebot.py
View File

@ -330,7 +330,7 @@ class FreqtradeBot(object):
# Pick pair based on buy signals # Pick pair based on buy signals
for _pair in whitelist: for _pair in whitelist:
thistory = self.exchange.get_ticker_history(_pair, interval) thistory = self.exchange.get_candle_history(_pair, interval)
(buy, sell) = self.strategy.get_signal(_pair, interval, thistory) (buy, sell) = self.strategy.get_signal(_pair, interval, thistory)
if buy and not sell: if buy and not sell:
@ -497,7 +497,7 @@ class FreqtradeBot(object):
(buy, sell) = (False, False) (buy, sell) = (False, False)
experimental = self.config.get('experimental', {}) experimental = self.config.get('experimental', {})
if experimental.get('use_sell_signal') or experimental.get('ignore_roi_if_buy_signal'): if experimental.get('use_sell_signal') or experimental.get('ignore_roi_if_buy_signal'):
ticker = self.exchange.get_ticker_history(trade.pair, self.strategy.ticker_interval) ticker = self.exchange.get_candle_history(trade.pair, self.strategy.ticker_interval)
(buy, sell) = self.strategy.get_signal(trade.pair, self.strategy.ticker_interval, (buy, sell) = self.strategy.get_signal(trade.pair, self.strategy.ticker_interval,
ticker) ticker)

2
freqtrade/optimize/__init__.py
View File

@ -235,7 +235,7 @@ def download_backtesting_testdata(datadir: str,
logger.debug("Current Start: %s", misc.format_ms_time(data[1][0]) if data else 'None') logger.debug("Current Start: %s", misc.format_ms_time(data[1][0]) if data else 'None')
logger.debug("Current End: %s", misc.format_ms_time(data[-1][0]) if data else 'None') logger.debug("Current End: %s", misc.format_ms_time(data[-1][0]) if data else 'None')
new_data = exchange.get_ticker_history(pair=pair, tick_interval=tick_interval, new_data = exchange.get_candle_history(pair=pair, tick_interval=tick_interval,
since_ms=since_ms) since_ms=since_ms)
data.extend(new_data) data.extend(new_data)

53
freqtrade/optimize/backslapping.py
View File

@ -1,48 +1,36 @@
import timeit import timeit
from argparse import Namespace
import logging
from typing import Dict, Any from typing import Dict, Any
from pandas import DataFrame from pandas import DataFrame
from freqtrade.exchange import Exchange from freqtrade.exchange import Exchange
from freqtrade.optimize.optimize import IOptimize, BacktestResult, OptimizeType, setup_configuration
from freqtrade.strategy import IStrategy from freqtrade.strategy import IStrategy
from freqtrade.strategy.interface import SellType from freqtrade.strategy.interface import SellType
from freqtrade.strategy.resolver import StrategyResolver from freqtrade.strategy.resolver import StrategyResolver
logger = logging.getLogger(__name__)
class Backslapping:
class Backslapping(IOptimize):
""" """
provides a quick way to evaluate strategies over a longer term of time provides a quick way to evaluate strategies over a longer term of time
""" """
def __init__(self, config: Dict[str, Any], exchange = None) -> None: def __init__(self, config: Dict[str, Any]) -> None:
""" """
constructor constructor
""" """
super().__init__(config)
self.config = config self._optimizetype = OptimizeType.BACKTEST
self.strategy: IStrategy = StrategyResolver(self.config).strategy
self.ticker_interval = self.strategy.ticker_interval
self.tickerdata_to_dataframe = self.strategy.tickerdata_to_dataframe
self.populate_buy_trend = self.strategy.populate_buy_trend
self.populate_sell_trend = self.strategy.populate_sell_trend
###
#
###
if exchange is None:
self.config['exchange']['secret'] = ''
self.config['exchange']['password'] = ''
self.config['exchange']['uid'] = ''
self.config['dry_run'] = True
self.exchange = Exchange(self.config)
else:
self.exchange = exchange
self.fee = self.exchange.get_fee() self.fee = self.exchange.get_fee()
self.stop_loss_value = self.strategy.stoploss self.stop_loss_value = self.strategy.stoploss
#### backslap config # backslap config
''' '''
Numpy arrays are used for 100x speed up Numpy arrays are used for 100x speed up
We requires setting Int values for We requires setting Int values for
@ -96,8 +84,8 @@ class Backslapping:
if self.debug_timing: # Start timer if self.debug_timing: # Start timer
fl = self.s() fl = self.s()
ticker_data = self.populate_sell_trend( ticker_data = self.advise_sell(self.advise_buy(pair_data, {'pair': pair}),
self.populate_buy_trend(pair_data))[headers].copy() {'pair': pair})[headers].copy()
if self.debug_timing: # print time taken if self.debug_timing: # print time taken
flt = self.f(fl) flt = self.f(fl)
@ -132,7 +120,7 @@ class Backslapping:
bslap_results_df = self.vector_fill_results_table(bslap_results_df, pair) bslap_results_df = self.vector_fill_results_table(bslap_results_df, pair)
else: else:
from freqtrade.optimize.backtesting import BacktestResult
bslap_results_df = [] bslap_results_df = []
bslap_results_df = DataFrame.from_records(bslap_results_df, columns=BacktestResult._fields) bslap_results_df = DataFrame.from_records(bslap_results_df, columns=BacktestResult._fields)
@ -787,3 +775,18 @@ class Backslapping:
# Send back List of trade dicts # Send back List of trade dicts
return bslap_pair_results return bslap_pair_results
def start(args: Namespace) -> None:
"""
Start Backtesting script
:param args: Cli args from Arguments()
:return: None
"""
# Initialize configuration
config = setup_configuration(args)
logger.info('Starting freqtrade in Backtesting mode')
# Initialize backtesting object
backslapping = Backslapping(config)
backslapping.start()

361
freqtrade/optimize/backtesting.py
View File

@ -4,51 +4,19 @@
This module contains the backtesting logic This module contains the backtesting logic
""" """
import logging import logging
import operator
from argparse import Namespace from argparse import Namespace
from datetime import datetime, timedelta from typing import Any, Dict, List, Optional
from typing import Any, Dict, List, NamedTuple, Optional, Tuple
import arrow from pandas import DataFrame
from pandas import DataFrame, to_datetime
from tabulate import tabulate
import freqtrade.optimize as optimize from freqtrade.optimize.optimize import IOptimize, BacktestResult, OptimizeType, setup_configuration
from freqtrade import DependencyException, constants
from freqtrade.arguments import Arguments
from freqtrade.configuration import Configuration
from freqtrade.exchange import Exchange
from freqtrade.misc import file_dump_json
from freqtrade.optimize.backslapping import Backslapping
from freqtrade.persistence import Trade from freqtrade.persistence import Trade
from freqtrade.strategy.interface import SellType from freqtrade.strategy.interface import SellType
from freqtrade.strategy.resolver import IStrategy, StrategyResolver
from collections import OrderedDict
import timeit
from time import sleep
logger = logging.getLogger(__name__) logger = logging.getLogger(__name__)
class BacktestResult(NamedTuple): class Backtesting(IOptimize):
"""
NamedTuple Defining BacktestResults inputs.
"""
pair: str
profit_percent: float
profit_abs: float
open_time: datetime
close_time: datetime
open_index: int
close_index: int
trade_duration: float
open_at_end: bool
open_rate: float
close_rate: float
sell_reason: SellType
class Backtesting(object):
""" """
Backtesting class, this class contains all the logic to run a backtest Backtesting class, this class contains all the logic to run a backtest
@ -58,139 +26,8 @@ class Backtesting(object):
""" """
def __init__(self, config: Dict[str, Any]) -> None: def __init__(self, config: Dict[str, Any]) -> None:
self.config = config super().__init__(config)
self.strategy: IStrategy = StrategyResolver(self.config).strategy self._optimizetype = OptimizeType.BACKTEST
self.ticker_interval = self.strategy.ticker_interval
self.tickerdata_to_dataframe = self.strategy.tickerdata_to_dataframe
self.advise_buy = self.strategy.advise_buy
self.advise_sell = self.strategy.advise_sell
# Reset keys for backtesting
self.config['exchange']['key'] = ''
self.config['exchange']['secret'] = ''
self.config['exchange']['password'] = ''
self.config['exchange']['uid'] = ''
self.config['dry_run'] = True
self.exchange = Exchange(self.config)
self.fee = self.exchange.get_fee()
self.stop_loss_value = self.strategy.stoploss
#### backslap config
'''
Numpy arrays are used for 100x speed up
We requires setting Int values for
buy stop triggers and stop calculated on
# buy 0 - open 1 - close 2 - sell 3 - high 4 - low 5 - stop 6
'''
self.np_buy: int = 0
self.np_open: int = 1
self.np_close: int = 2
self.np_sell: int = 3
self.np_high: int = 4
self.np_low: int = 5
self.np_stop: int = 6
self.np_bto: int = self.np_close # buys_triggered_on - should be close
self.np_bco: int = self.np_open # buys calculated on - open of the next candle.
self.np_sto: int = self.np_low # stops_triggered_on - Should be low, FT uses close
self.np_sco: int = self.np_stop # stops_calculated_on - Should be stop, FT uses close
# self.np_sto: int = self.np_close # stops_triggered_on - Should be low, FT uses close
# self.np_sco: int = self.np_close # stops_calculated_on - Should be stop, FT uses close
if 'backslap' in config:
self.use_backslap = config['backslap'] # Enable backslap - if false Orginal code is executed.
else:
self.use_backslap = False
logger.info("using backslap: {}".format(self.use_backslap))
self.debug = False # Main debug enable, very print heavy, enable 2 loops recommended
self.debug_timing = False # Stages within Backslap
self.debug_2loops = False # Limit each pair to two loops, useful when debugging
self.debug_vector = False # Debug vector calcs
self.debug_timing_main_loop = False # print overall timing per pair - works in Backtest and Backslap
self.backslap_show_trades = False # prints trades in addition to summary report
self.backslap_save_trades = True # saves trades as a pretty table to backslap.txt
self.stop_stops: int = 9999 # stop back testing any pair with this many stops, set to 999999 to not hit
self.backslap = Backslapping(config)
@staticmethod
def get_timeframe(data: Dict[str, DataFrame]) -> Tuple[arrow.Arrow, arrow.Arrow]:
"""
Get the maximum timeframe for the given backtest data
:param data: dictionary with preprocessed backtesting data
:return: tuple containing min_date, max_date
"""
timeframe = [
(arrow.get(frame['date'].min()), arrow.get(frame['date'].max()))
for frame in data.values()
]
return min(timeframe, key=operator.itemgetter(0))[0], \
max(timeframe, key=operator.itemgetter(1))[1]
def _generate_text_table(self, data: Dict[str, Dict], results: DataFrame) -> str:
"""
Generates and returns a text table for the given backtest data and the results dataframe
:return: pretty printed table with tabulate as str
"""
stake_currency = str(self.config.get('stake_currency'))
floatfmt = ('s', 'd', '.2f', '.2f', '.8f', 'd', '.1f', '.1f')
tabular_data = []
headers = ['pair', 'buy count', 'avg profit %', 'cum profit %',
'total profit ' + stake_currency, 'avg duration', 'profit', 'loss']
for pair in data:
result = results[results.pair == pair]
tabular_data.append([
pair,
len(result.index),
result.profit_percent.mean() * 100.0,
result.profit_percent.sum() * 100.0,
result.profit_abs.sum(),
str(timedelta(
minutes=round(result.trade_duration.mean()))) if not result.empty else '0:00',
len(result[result.profit_abs > 0]),
len(result[result.profit_abs < 0])
])
# Append Total
tabular_data.append([
'TOTAL',
len(results.index),
results.profit_percent.mean() * 100.0,
results.profit_percent.sum() * 100.0,
results.profit_abs.sum(),
str(timedelta(
minutes=round(results.trade_duration.mean()))) if not results.empty else '0:00',
len(results[results.profit_abs > 0]),
len(results[results.profit_abs < 0])
])
return tabulate(tabular_data, headers=headers, floatfmt=floatfmt, tablefmt="pipe")
def _generate_text_table_sell_reason(self, data: Dict[str, Dict], results: DataFrame) -> str:
"""
Generate small table outlining Backtest results
"""
tabular_data = []
headers = ['Sell Reason', 'Count']
for reason, count in results['sell_reason'].value_counts().iteritems():
tabular_data.append([reason.value, count])
return tabulate(tabular_data, headers=headers, tablefmt="pipe")
def _store_backtest_result(self, recordfilename: Optional[str], results: DataFrame) -> None:
records = [(t.pair, t.profit_percent, t.open_time.timestamp(),
t.close_time.timestamp(), t.open_index - 1, t.trade_duration,
t.open_rate, t.close_rate, t.open_at_end, t.sell_reason.value)
for index, t in results.iterrows()]
if records:
logger.info('Dumping backtest results to %s', recordfilename)
file_dump_json(recordfilename, records)
def _get_sell_trade_entry( def _get_sell_trade_entry(
self, pair: str, buy_row: DataFrame, self, pair: str, buy_row: DataFrame,
@ -217,6 +54,7 @@ class Backtesting(object):
sell = self.strategy.should_sell(trade, sell_row.open, sell_row.date, buy_signal, sell = self.strategy.should_sell(trade, sell_row.open, sell_row.date, buy_signal,
sell_row.sell) sell_row.sell)
if sell.sell_flag: if sell.sell_flag:
return BacktestResult(pair=pair, return BacktestResult(pair=pair,
profit_percent=trade.calc_profit_percent(rate=sell_row.open), profit_percent=trade.calc_profit_percent(rate=sell_row.open),
profit_abs=trade.calc_profit(rate=sell_row.open), profit_abs=trade.calc_profit(rate=sell_row.open),
@ -253,14 +91,7 @@ class Backtesting(object):
return btr return btr
return None return None
def s(self): def run(self, args: Dict) -> DataFrame:
st = timeit.default_timer()
return st
def f(self, st):
return (timeit.default_timer() - st)
def backtest(self, args: Dict) -> DataFrame:
""" """
Implements backtesting functionality Implements backtesting functionality
@ -275,26 +106,13 @@ class Backtesting(object):
position_stacking: do we allow position stacking? (default: False) position_stacking: do we allow position stacking? (default: False)
:return: DataFrame :return: DataFrame
""" """
use_backslap = self.use_backslap
debug_timing = self.debug_timing_main_loop
if use_backslap: # Use Back Slap code
return self.backslap.run(args)
else: # use Original Back test code
########################## Original BT loop
headers = ['date', 'buy', 'open', 'close', 'sell'] headers = ['date', 'buy', 'open', 'close', 'sell']
processed = args['processed'] processed = args['processed']
max_open_trades = args.get('max_open_trades', 0) max_open_trades = args.get('max_open_trades', 0)
position_stacking = args.get('position_stacking', False) position_stacking = args.get('position_stacking', False)
trades = [] trades = []
trade_count_lock: Dict = {} trade_count_lock: Dict = {}
for pair, pair_data in processed.items(): for pair, pair_data in processed.items():
if debug_timing: # Start timer
fl = self.s()
pair_data['buy'], pair_data['sell'] = 0, 0 # cleanup from previous run pair_data['buy'], pair_data['sell'] = 0, 0 # cleanup from previous run
ticker_data = self.advise_sell( ticker_data = self.advise_sell(
@ -306,11 +124,6 @@ class Backtesting(object):
ticker_data.drop(ticker_data.head(1).index, inplace=True) ticker_data.drop(ticker_data.head(1).index, inplace=True)
if debug_timing: # print time taken
flt = self.f(fl)
# print("populate_buy_trend:", pair, round(flt, 10))
st = self.s()
# Convert from Pandas to list for performance reasons # Convert from Pandas to list for performance reasons
# (Looping Pandas is slow.) # (Looping Pandas is slow.)
ticker = [x for x in ticker_data.itertuples()] ticker = [x for x in ticker_data.itertuples()]
@ -341,165 +154,7 @@ class Backtesting(object):
# This happens only if the buy-signal was with the last candle # This happens only if the buy-signal was with the last candle
lock_pair_until = ticker_data.iloc[-1].date lock_pair_until = ticker_data.iloc[-1].date
if debug_timing: # print time taken
tt = self.f(st)
print("Time to BackTest :", pair, round(tt, 10))
print("-----------------------")
return DataFrame.from_records(trades, columns=BacktestResult._fields) return DataFrame.from_records(trades, columns=BacktestResult._fields)
####################### Original BT loop end
def start(self) -> None:
"""
Run a backtesting end-to-end
:return: None
"""
data = {}
pairs = self.config['exchange']['pair_whitelist']
logger.info('Using stake_currency: %s ...', self.config['stake_currency'])
logger.info('Using stake_amount: %s ...', self.config['stake_amount'])
if self.config.get('live'):
logger.info('Downloading data for all pairs in whitelist ...')
for pair in pairs:
data[pair] = self.exchange.get_ticker_history(pair, self.ticker_interval)
else:
logger.info('Using local backtesting data (using whitelist in given config) ...')
timerange = Arguments.parse_timerange(None if self.config.get(
'timerange') is None else str(self.config.get('timerange')))
data = optimize.load_data(
self.config['datadir'],
pairs=pairs,
ticker_interval=self.ticker_interval,
refresh_pairs=self.config.get('refresh_pairs', False),
exchange=self.exchange,
timerange=timerange
)
ld_files = self.s()
if not data:
logger.critical("No data found. Terminating.")
return
# Use max_open_trades in backtesting, except --disable-max-market-positions is set
if self.config.get('use_max_market_positions', True):
max_open_trades = self.config['max_open_trades']
else:
logger.info('Ignoring max_open_trades (--disable-max-market-positions was used) ...')
max_open_trades = 0
preprocessed = self.tickerdata_to_dataframe(data)
t_t = self.f(ld_files)
print("Load from json to file to df in mem took", t_t)
# Print timeframe
min_date, max_date = self.get_timeframe(preprocessed)
logger.info(
'Measuring data from %s up to %s (%s days)..',
min_date.isoformat(),
max_date.isoformat(),
(max_date - min_date).days
)
# Execute backtest and print results
results = self.backtest(
{
'stake_amount': self.config.get('stake_amount'),
'processed': preprocessed,
'max_open_trades': max_open_trades,
'position_stacking': self.config.get('position_stacking', False),
}
)
if self.config.get('export', False):
self._store_backtest_result(self.config.get('exportfilename'), results)
if self.use_backslap:
logger.info(
'\n====================================================== '
'BackSLAP REPORT'
' =======================================================\n'
'%s',
self._generate_text_table(
data,
results
)
)
# optional print trades
if self.backslap_show_trades:
TradesFrame = results.filter(['open_time', 'pair', 'exit_type', 'profit_percent', 'profit_abs',
'buy_spend', 'sell_take', 'trade_duration', 'close_time'], axis=1)
def to_fwf(df, fname):
content = tabulate(df.values.tolist(), list(df.columns), floatfmt=".8f", tablefmt='psql')
print(content)
DataFrame.to_fwf = to_fwf(TradesFrame, "backslap.txt")
# optional save trades
if self.backslap_save_trades:
TradesFrame = results.filter(['open_time', 'pair', 'exit_type', 'profit_percent', 'profit_abs',
'buy_spend', 'sell_take', 'trade_duration', 'close_time'], axis=1)
def to_fwf(df, fname):
content = tabulate(df.values.tolist(), list(df.columns), floatfmt=".8f", tablefmt='psql')
open(fname, "w").write(content)
DataFrame.to_fwf = to_fwf(TradesFrame, "backslap.txt")
else:
logger.info(
'\n================================================= '
'BACKTEST REPORT'
' ==================================================\n'
'%s',
self._generate_text_table(
data,
results
)
)
if 'sell_reason' in results.columns:
logger.info(
'\n' +
' SELL READON STATS '.center(119, '=') +
'\n%s \n',
self._generate_text_table_sell_reason(data, results)
)
else:
logger.info("no sell reasons available!")
logger.info(
'\n' +
' LEFT OPEN TRADES REPORT '.center(119, '=') +
'\n%s',
self._generate_text_table(
data,
results.loc[results.open_at_end]
)
)
def setup_configuration(args: Namespace) -> Dict[str, Any]:
"""
Prepare the configuration for the backtesting
:param args: Cli args from Arguments()
:return: Configuration
"""
configuration = Configuration(args)
config = configuration.get_config()
# Ensure we do not use Exchange credentials
config['exchange']['key'] = ''
config['exchange']['secret'] = ''
config['backslap'] = args.backslap
if config['stake_amount'] == constants.UNLIMITED_STAKE_AMOUNT:
raise DependencyException('stake amount could not be "%s" for backtesting' %
constants.UNLIMITED_STAKE_AMOUNT)
return config
def start(args: Namespace) -> None: def start(args: Namespace) -> None:

4
freqtrade/optimize/hyperopt.py
View File

@ -24,6 +24,7 @@ import freqtrade.vendor.qtpylib.indicators as qtpylib
from freqtrade.arguments import Arguments from freqtrade.arguments import Arguments
from freqtrade.configuration import Configuration from freqtrade.configuration import Configuration
from freqtrade.optimize import load_data from freqtrade.optimize import load_data
from freqtrade.optimize.optimize import OptimizeType
from freqtrade.optimize.backtesting import Backtesting from freqtrade.optimize.backtesting import Backtesting
logger = logging.getLogger(__name__) logger = logging.getLogger(__name__)
@ -42,6 +43,7 @@ class Hyperopt(Backtesting):
""" """
def __init__(self, config: Dict[str, Any]) -> None: def __init__(self, config: Dict[str, Any]) -> None:
super().__init__(config) super().__init__(config)
self._optimizetype = OptimizeType.HYPEROPT
# set TARGET_TRADES to suit your number concurrent trades so its realistic # set TARGET_TRADES to suit your number concurrent trades so its realistic
# to the number of days # to the number of days
self.target_trades = 600 self.target_trades = 600
@ -276,7 +278,7 @@ class Hyperopt(Backtesting):
self.strategy.stoploss = params['stoploss'] self.strategy.stoploss = params['stoploss']
processed = load(TICKERDATA_PICKLE) processed = load(TICKERDATA_PICKLE)
results = self.backtest( results = self.run(
{ {
'stake_amount': self.config['stake_amount'], 'stake_amount': self.config['stake_amount'],
'processed': processed, 'processed': processed,

329
freqtrade/optimize/optimize.py Normal file
View File

@ -0,0 +1,329 @@
# pragma pylint: disable=missing-docstring, W0212, too-many-arguments
"""
This module contains the backtesting logic
"""
import logging
import operator
from abc import ABC, abstractmethod
from argparse import Namespace
from copy import deepcopy
from datetime import datetime, timedelta
from pathlib import Path
from typing import Any, Dict, List, NamedTuple, Optional, Tuple
from enum import Enum
import arrow
from pandas import DataFrame
from tabulate import tabulate
from freqtrade import DependencyException, constants
from freqtrade.arguments import Arguments
from freqtrade.configuration import Configuration
from freqtrade.exchange import Exchange
from freqtrade.misc import file_dump_json
import freqtrade.optimize as optimize
from freqtrade.strategy.interface import SellType
from freqtrade.strategy.resolver import IStrategy, StrategyResolver
logger = logging.getLogger(__name__)
class BacktestResult(NamedTuple):
"""
NamedTuple Defining BacktestResults inputs.
"""
pair: str
profit_percent: float
profit_abs: float
open_time: datetime
close_time: datetime
open_index: int
close_index: int
trade_duration: float
open_at_end: bool
open_rate: float
close_rate: float
sell_reason: SellType
class OptimizeType(Enum):
BACKTEST = "backtest"
BACKSLAP = "backslap"
HYPEROPT = "hyperopt"
class IOptimize(ABC):
"""
Backtesting Abstract class, this class contains all the logic to run a backtest
To run a backtest:
backtesting = Backtesting(config)
backtesting.start()
"""
def __init__(self, config: Dict[str, Any]) -> None:
self.config = config
# Reset keys for backtesting
self.config['exchange']['key'] = ''
self.config['exchange']['secret'] = ''
self.config['exchange']['password'] = ''
self.config['exchange']['uid'] = ''
self.config['dry_run'] = True
self.strategylist: List[IStrategy] = []
if self.config.get('strategy_list', None):
# Force one interval
self.ticker_interval = str(self.config.get('ticker_interval'))
for strat in list(self.config['strategy_list']):
stratconf = deepcopy(self.config)
stratconf['strategy'] = strat
self.strategylist.append(StrategyResolver(stratconf).strategy)
else:
# only one strategy
strat = StrategyResolver(self.config).strategy
self.strategylist.append(StrategyResolver(self.config).strategy)
# Load one strategy
self._set_strategy(self.strategylist[0])
self.exchange = Exchange(self.config)
self.fee = self.exchange.get_fee()
def _set_strategy(self, strategy):
"""
Load strategy into backtesting
"""
self.strategy = strategy
self.ticker_interval = self.config.get('ticker_interval')
self.tickerdata_to_dataframe = strategy.tickerdata_to_dataframe
self.advise_buy = strategy.advise_buy
self.advise_sell = strategy.advise_sell
def _get_timeframe(self, data: Dict[str, DataFrame]) -> Tuple[arrow.Arrow, arrow.Arrow]:
"""
Get the maximum timeframe for the given backtest data
:param data: dictionary with preprocessed backtesting data
:return: tuple containing min_date, max_date
"""
timeframe = [
(arrow.get(frame['date'].min()), arrow.get(frame['date'].max()))
for frame in data.values()
]
return min(timeframe, key=operator.itemgetter(0))[0], \
max(timeframe, key=operator.itemgetter(1))[1]
def _generate_text_table(self, data: Dict[str, Dict], results: DataFrame) -> str:
"""
Generates and returns a text table for the given backtest data and the results dataframe
:return: pretty printed table with tabulate as str
"""
stake_currency = str(self.config.get('stake_currency'))
floatfmt = ('s', 'd', '.2f', '.2f', '.8f', 'd', '.1f', '.1f')
tabular_data = []
headers = ['pair', 'buy count', 'avg profit %', 'cum profit %',
'total profit ' + stake_currency, 'avg duration', 'profit', 'loss']
for pair in data:
result = results[results.pair == pair]
tabular_data.append([
pair,
len(result.index),
result.profit_percent.mean() * 100.0,
result.profit_percent.sum() * 100.0,
result.profit_abs.sum(),
str(timedelta(
minutes=round(result.trade_duration.mean()))) if not result.empty else '0:00',
len(result[result.profit_abs > 0]),
len(result[result.profit_abs < 0])
])
# Append Total
tabular_data.append([
'TOTAL',
len(results.index),
results.profit_percent.mean() * 100.0,
results.profit_percent.sum() * 100.0,
results.profit_abs.sum(),
str(timedelta(
minutes=round(results.trade_duration.mean()))) if not results.empty else '0:00',
len(results[results.profit_abs > 0]),
len(results[results.profit_abs < 0])
])
return tabulate(tabular_data, headers=headers, floatfmt=floatfmt, tablefmt="pipe")
def _generate_text_table_sell_reason(self, data: Dict[str, Dict], results: DataFrame) -> str:
"""
Generate small table outlining Backtest results
"""
tabular_data = []
headers = ['Sell Reason', 'Count']
for reason, count in results['sell_reason'].value_counts().iteritems():
tabular_data.append([reason.value, count])
return tabulate(tabular_data, headers=headers, tablefmt="pipe")
def _generate_text_table_strategy(self, all_results: dict) -> str:
"""
Generate summary table per strategy
"""
stake_currency = str(self.config.get('stake_currency'))
floatfmt = ('s', 'd', '.2f', '.2f', '.8f', 'd', '.1f', '.1f')
tabular_data = []
headers = ['Strategy', 'buy count', 'avg profit %', 'cum profit %',
'total profit ' + stake_currency, 'avg duration', 'profit', 'loss']
for strategy, results in all_results.items():
tabular_data.append([
strategy,
len(results.index),
results.profit_percent.mean() * 100.0,
results.profit_percent.sum() * 100.0,
results.profit_abs.sum(),
str(timedelta(
minutes=round(results.trade_duration.mean()))) if not results.empty else '0:00',
len(results[results.profit_abs > 0]),
len(results[results.profit_abs < 0])
])
return tabulate(tabular_data, headers=headers, floatfmt=floatfmt, tablefmt="pipe")
def _store_backtest_result(self, recordfilename: str, results: DataFrame,
strategyname: Optional[str] = None) -> None:
records = [(t.pair, t.profit_percent, t.open_time.timestamp(),
t.close_time.timestamp(), t.open_index - 1, t.trade_duration,
t.open_rate, t.close_rate, t.open_at_end, t.sell_reason.value)
for index, t in results.iterrows()]
if records:
if strategyname:
# Inject strategyname to filename
recname = Path(recordfilename)
recordfilename = str(Path.joinpath(
recname.parent, f'{recname.stem}-{strategyname}').with_suffix(recname.suffix))
logger.info('Dumping backtest results to %s', recordfilename)
file_dump_json(recordfilename, records)
def start(self) -> None:
"""
Run a backtesting end-to-end
:return: None
"""
data = {}
pairs = self.config['exchange']['pair_whitelist']
logger.info('Using stake_currency: %s ...', self.config['stake_currency'])
logger.info('Using stake_amount: %s ...', self.config['stake_amount'])
if self.config.get('live'):
logger.info('Downloading data for all pairs in whitelist ...')
for pair in pairs:
data[pair] = self.exchange.get_candle_history(pair, self.ticker_interval)
else:
logger.info('Using local backtesting data (using whitelist in given config) ...')
timerange = Arguments.parse_timerange(None if self.config.get(
'timerange') is None else str(self.config.get('timerange')))
data = optimize.load_data(
self.config['datadir'],
pairs=pairs,
ticker_interval=self.ticker_interval,
refresh_pairs=self.config.get('refresh_pairs', False),
exchange=self.exchange,
timerange=timerange
)
if not data:
logger.critical("No data found. Terminating.")
return
# Use max_open_trades in backtesting, except --disable-max-market-positions is set
if self.config.get('use_max_market_positions', True):
max_open_trades = self.config['max_open_trades']
else:
logger.info('Ignoring max_open_trades (--disable-max-market-positions was used) ...')
max_open_trades = 0
all_results = {}
for strat in self.strategylist:
logger.info("Running backtesting for Strategy %s", strat.get_strategy_name())
self._set_strategy(strat)
# need to reprocess data every time to populate signals
preprocessed = self.tickerdata_to_dataframe(data)
# Print timeframe
min_date, max_date = self._get_timeframe(preprocessed)
logger.info(
'Measuring data from %s up to %s (%s days)..',
min_date.isoformat(),
max_date.isoformat(),
(max_date - min_date).days
)
# Execute backtest and print results
all_results[self.strategy.get_strategy_name()] = self.run(
{
'stake_amount': self.config.get('stake_amount'),
'processed': preprocessed,
'max_open_trades': max_open_trades,
'position_stacking': self.config.get('position_stacking', False),
}
)
for strategy, results in all_results.items():
if self.config.get('export', False):
self._store_backtest_result(self.config['exportfilename'], results,
strategy if len(self.strategylist) > 1 else None)
print(f"Result for strategy {strategy}")
print(f' {self._optimizetype.value.upper()} REPORT '.center(119, '='))
print(self._generate_text_table(data, results))
print(' SELL REASON STATS '.center(119, '='))
print(self._generate_text_table_sell_reason(data, results))
print(' LEFT OPEN TRADES REPORT '.center(119, '='))
print(self._generate_text_table(data, results.loc[results.open_at_end]))
print()
if len(all_results) > 1:
# Print Strategy summary table
print(' Strategy Summary '.center(119, '='))
print(self._generate_text_table_strategy(all_results))
print('\nFor more details, please look at the detail tables above')
@abstractmethod
def run(self, args: Dict) -> DataFrame:
"""
Runs backtesting functionality.
NOTE: This method is used by Hyperopt at each iteration. Please keep it optimized.
Of course try to not have ugly code. By some accessor are sometime slower than functions.
Avoid, logging on this method
:param args: a dict containing:
stake_amount: btc amount to use for each trade
processed: a processed dictionary with format {pair, data}
max_open_trades: maximum number of concurrent trades (default: 0, disabled)
position_stacking: do we allow position stacking? (default: False)
:return: DataFrame
"""
def setup_configuration(args: Namespace) -> Dict[str, Any]:
"""
Prepare the configuration for the backtesting
:param args: Cli args from Arguments()
:return: Configuration
"""
configuration = Configuration(args)
config = configuration.get_config()
# Ensure we do not use Exchange credentials
config['exchange']['key'] = ''
config['exchange']['secret'] = ''
if config['stake_amount'] == constants.UNLIMITED_STAKE_AMOUNT:
raise DependencyException('stake amount could not be "%s" for backtesting' %
constants.UNLIMITED_STAKE_AMOUNT)
return config

6
freqtrade/persistence.py
View File

@ -82,7 +82,7 @@ def check_migrate(engine) -> None:
logger.info(f'trying {table_back_name}') logger.info(f'trying {table_back_name}')
# Check for latest column # Check for latest column
if not has_column(cols, 'max_rate'): if not has_column(cols, 'ticker_interval'):
fee_open = get_column_def(cols, 'fee_open', 'fee') fee_open = get_column_def(cols, 'fee_open', 'fee')
fee_close = get_column_def(cols, 'fee_close', 'fee') fee_close = get_column_def(cols, 'fee_close', 'fee')
open_rate_requested = get_column_def(cols, 'open_rate_requested', 'null') open_rate_requested = get_column_def(cols, 'open_rate_requested', 'null')
@ -157,8 +157,8 @@ class Trade(_DECL_BASE):
id = Column(Integer, primary_key=True) id = Column(Integer, primary_key=True)
exchange = Column(String, nullable=False) exchange = Column(String, nullable=False)
pair = Column(String, nullable=False) pair = Column(String, nullable=False, index=True)
is_open = Column(Boolean, nullable=False, default=True) is_open = Column(Boolean, nullable=False, default=True, index=True)
fee_open = Column(Float, nullable=False, default=0.0) fee_open = Column(Float, nullable=False, default=0.0)
fee_close = Column(Float, nullable=False, default=0.0) fee_close = Column(Float, nullable=False, default=0.0)
open_rate = Column(Float) open_rate = Column(Float)

16
freqtrade/tests/exchange/test_exchange.py
View File

@ -524,7 +524,7 @@ def make_fetch_ohlcv_mock(data):
return fetch_ohlcv_mock return fetch_ohlcv_mock
def test_get_ticker_history(default_conf, mocker): def test_get_candle_history(default_conf, mocker):
api_mock = MagicMock() api_mock = MagicMock()
tick = [ tick = [
[ [
@ -541,7 +541,7 @@ def test_get_ticker_history(default_conf, mocker):
exchange = get_patched_exchange(mocker, default_conf, api_mock) exchange = get_patched_exchange(mocker, default_conf, api_mock)
# retrieve original ticker # retrieve original ticker
ticks = exchange.get_ticker_history('ETH/BTC', default_conf['ticker_interval']) ticks = exchange.get_candle_history('ETH/BTC', default_conf['ticker_interval'])
assert ticks[0][0] == 1511686200000 assert ticks[0][0] == 1511686200000
assert ticks[0][1] == 1 assert ticks[0][1] == 1
assert ticks[0][2] == 2 assert ticks[0][2] == 2
@ -563,7 +563,7 @@ def test_get_ticker_history(default_conf, mocker):
api_mock.fetch_ohlcv = MagicMock(side_effect=make_fetch_ohlcv_mock(new_tick)) api_mock.fetch_ohlcv = MagicMock(side_effect=make_fetch_ohlcv_mock(new_tick))
exchange = get_patched_exchange(mocker, default_conf, api_mock) exchange = get_patched_exchange(mocker, default_conf, api_mock)
ticks = exchange.get_ticker_history('ETH/BTC', default_conf['ticker_interval']) ticks = exchange.get_candle_history('ETH/BTC', default_conf['ticker_interval'])
assert ticks[0][0] == 1511686210000 assert ticks[0][0] == 1511686210000
assert ticks[0][1] == 6 assert ticks[0][1] == 6
assert ticks[0][2] == 7 assert ticks[0][2] == 7
@ -572,16 +572,16 @@ def test_get_ticker_history(default_conf, mocker):
assert ticks[0][5] == 10 assert ticks[0][5] == 10
ccxt_exceptionhandlers(mocker, default_conf, api_mock, ccxt_exceptionhandlers(mocker, default_conf, api_mock,
"get_ticker_history", "fetch_ohlcv", "get_candle_history", "fetch_ohlcv",
pair='ABCD/BTC', tick_interval=default_conf['ticker_interval']) pair='ABCD/BTC', tick_interval=default_conf['ticker_interval'])
with pytest.raises(OperationalException, match=r'Exchange .* does not support.*'): with pytest.raises(OperationalException, match=r'Exchange .* does not support.*'):
api_mock.fetch_ohlcv = MagicMock(side_effect=ccxt.NotSupported) api_mock.fetch_ohlcv = MagicMock(side_effect=ccxt.NotSupported)
exchange = get_patched_exchange(mocker, default_conf, api_mock) exchange = get_patched_exchange(mocker, default_conf, api_mock)
exchange.get_ticker_history(pair='ABCD/BTC', tick_interval=default_conf['ticker_interval']) exchange.get_candle_history(pair='ABCD/BTC', tick_interval=default_conf['ticker_interval'])
def test_get_ticker_history_sort(default_conf, mocker): def test_get_candle_history_sort(default_conf, mocker):
api_mock = MagicMock() api_mock = MagicMock()
# GDAX use-case (real data from GDAX) # GDAX use-case (real data from GDAX)
@ -604,7 +604,7 @@ def test_get_ticker_history_sort(default_conf, mocker):
exchange = get_patched_exchange(mocker, default_conf, api_mock) exchange = get_patched_exchange(mocker, default_conf, api_mock)
# Test the ticker history sort # Test the ticker history sort
ticks = exchange.get_ticker_history('ETH/BTC', default_conf['ticker_interval']) ticks = exchange.get_candle_history('ETH/BTC', default_conf['ticker_interval'])
assert ticks[0][0] == 1527830400000 assert ticks[0][0] == 1527830400000
assert ticks[0][1] == 0.07649 assert ticks[0][1] == 0.07649
assert ticks[0][2] == 0.07651 assert ticks[0][2] == 0.07651
@ -637,7 +637,7 @@ def test_get_ticker_history_sort(default_conf, mocker):
api_mock.fetch_ohlcv = MagicMock(side_effect=make_fetch_ohlcv_mock(tick)) api_mock.fetch_ohlcv = MagicMock(side_effect=make_fetch_ohlcv_mock(tick))
exchange = get_patched_exchange(mocker, default_conf, api_mock) exchange = get_patched_exchange(mocker, default_conf, api_mock)
# Test the ticker history sort # Test the ticker history sort
ticks = exchange.get_ticker_history('ETH/BTC', default_conf['ticker_interval']) ticks = exchange.get_candle_history('ETH/BTC', default_conf['ticker_interval'])
assert ticks[0][0] == 1527827700000 assert ticks[0][0] == 1527827700000
assert ticks[0][1] == 0.07659999 assert ticks[0][1] == 0.07659999
assert ticks[0][2] == 0.0766 assert ticks[0][2] == 0.0766

158
freqtrade/tests/optimize/test_backtesting.py
View File

@ -91,7 +91,7 @@ def simple_backtest(config, contour, num_results, mocker) -> None:
data = load_data_test(contour) data = load_data_test(contour)
processed = backtesting.tickerdata_to_dataframe(data) processed = backtesting.tickerdata_to_dataframe(data)
assert isinstance(processed, dict) assert isinstance(processed, dict)
results = backtesting.backtest( results = backtesting.run(
{ {
'stake_amount': config['stake_amount'], 'stake_amount': config['stake_amount'],
'processed': processed, 'processed': processed,
@ -110,7 +110,7 @@ def mocked_load_data(datadir, pairs=[], ticker_interval='0m', refresh_pairs=Fals
return pairdata return pairdata
# use for mock freqtrade.exchange.get_ticker_history' # use for mock freqtrade.exchange.get_candle_history'
def _load_pair_as_ticks(pair, tickfreq): def _load_pair_as_ticks(pair, tickfreq):
ticks = optimize.load_data(None, ticker_interval=tickfreq, pairs=[pair]) ticks = optimize.load_data(None, ticker_interval=tickfreq, pairs=[pair])
ticks = trim_dictlist(ticks, -201) ticks = trim_dictlist(ticks, -201)
@ -347,7 +347,7 @@ def test_get_timeframe(default_conf, mocker) -> None:
pairs=['UNITTEST/BTC'] pairs=['UNITTEST/BTC']
) )
) )
min_date, max_date = backtesting.get_timeframe(data) min_date, max_date = backtesting._get_timeframe(data)
assert min_date.isoformat() == '2017-11-04T23:02:00+00:00' assert min_date.isoformat() == '2017-11-04T23:02:00+00:00'
assert max_date.isoformat() == '2017-11-14T22:58:00+00:00' assert max_date.isoformat() == '2017-11-14T22:58:00+00:00'
@ -406,18 +406,62 @@ def test_generate_text_table_sell_reason(default_conf, mocker):
data={'ETH/BTC': {}}, results=results) == result_str data={'ETH/BTC': {}}, results=results) == result_str
def test_generate_text_table_strategyn(default_conf, mocker):
"""
Test Backtesting.generate_text_table_sell_reason() method
"""
patch_exchange(mocker)
backtesting = Backtesting(default_conf)
results = {}
results['ETH/BTC'] = pd.DataFrame(
{
'pair': ['ETH/BTC', 'ETH/BTC', 'ETH/BTC'],
'profit_percent': [0.1, 0.2, 0.3],
'profit_abs': [0.2, 0.4, 0.5],
'trade_duration': [10, 30, 10],
'profit': [2, 0, 0],
'loss': [0, 0, 1],
'sell_reason': [SellType.ROI, SellType.ROI, SellType.STOP_LOSS]
}
)
results['LTC/BTC'] = pd.DataFrame(
{
'pair': ['LTC/BTC', 'LTC/BTC', 'LTC/BTC'],
'profit_percent': [0.4, 0.2, 0.3],
'profit_abs': [0.4, 0.4, 0.5],
'trade_duration': [15, 30, 15],
'profit': [4, 1, 0],
'loss': [0, 0, 1],
'sell_reason': [SellType.ROI, SellType.ROI, SellType.STOP_LOSS]
}
)
result_str = (
'| Strategy | buy count | avg profit % | cum profit % '
'| total profit BTC | avg duration | profit | loss |\n'
'|:-----------|------------:|---------------:|---------------:'
'|-------------------:|:---------------|---------:|-------:|\n'
'| ETH/BTC | 3 | 20.00 | 60.00 '
'| 1.10000000 | 0:17:00 | 3 | 0 |\n'
'| LTC/BTC | 3 | 30.00 | 90.00 '
'| 1.30000000 | 0:20:00 | 3 | 0 |'
)
print(backtesting._generate_text_table_strategy(all_results=results))
assert backtesting._generate_text_table_strategy(all_results=results) == result_str
def test_backtesting_start(default_conf, mocker, caplog) -> None: def test_backtesting_start(default_conf, mocker, caplog) -> None:
def get_timeframe(input1, input2): def get_timeframe(input1, input2):
return Arrow(2017, 11, 14, 21, 17), Arrow(2017, 11, 14, 22, 59) return Arrow(2017, 11, 14, 21, 17), Arrow(2017, 11, 14, 22, 59)
mocker.patch('freqtrade.optimize.load_data', mocked_load_data) mocker.patch('freqtrade.optimize.load_data', mocked_load_data)
mocker.patch('freqtrade.exchange.Exchange.get_ticker_history') mocker.patch('freqtrade.exchange.Exchange.get_candle_history')
patch_exchange(mocker) patch_exchange(mocker)
mocker.patch.multiple( mocker.patch.multiple(
'freqtrade.optimize.backtesting.Backtesting', 'freqtrade.optimize.backtesting.Backtesting',
backtest=MagicMock(), run=MagicMock(),
_generate_text_table=MagicMock(return_value='1'), _generate_text_table=MagicMock(return_value='1'),
get_timeframe=get_timeframe, _get_timeframe=get_timeframe,
) )
default_conf['exchange']['pair_whitelist'] = ['UNITTEST/BTC'] default_conf['exchange']['pair_whitelist'] = ['UNITTEST/BTC']
@ -446,13 +490,13 @@ def test_backtesting_start_no_data(default_conf, mocker, caplog) -> None:
return Arrow(2017, 11, 14, 21, 17), Arrow(2017, 11, 14, 22, 59) return Arrow(2017, 11, 14, 21, 17), Arrow(2017, 11, 14, 22, 59)
mocker.patch('freqtrade.optimize.load_data', MagicMock(return_value={})) mocker.patch('freqtrade.optimize.load_data', MagicMock(return_value={}))
mocker.patch('freqtrade.exchange.Exchange.get_ticker_history') mocker.patch('freqtrade.exchange.Exchange.get_candle_history')
patch_exchange(mocker) patch_exchange(mocker)
mocker.patch.multiple( mocker.patch.multiple(
'freqtrade.optimize.backtesting.Backtesting', 'freqtrade.optimize.backtesting.Backtesting',
backtest=MagicMock(), run=MagicMock(),
_generate_text_table=MagicMock(return_value='1'), _generate_text_table=MagicMock(return_value='1'),
get_timeframe=get_timeframe, _get_timeframe=get_timeframe,
) )
default_conf['exchange']['pair_whitelist'] = ['UNITTEST/BTC'] default_conf['exchange']['pair_whitelist'] = ['UNITTEST/BTC']
@ -477,7 +521,7 @@ def test_backtest(default_conf, fee, mocker) -> None:
data = optimize.load_data(None, ticker_interval='5m', pairs=['UNITTEST/BTC']) data = optimize.load_data(None, ticker_interval='5m', pairs=['UNITTEST/BTC'])
data = trim_dictlist(data, -200) data = trim_dictlist(data, -200)
data_processed = backtesting.tickerdata_to_dataframe(data) data_processed = backtesting.tickerdata_to_dataframe(data)
results = backtesting.backtest( results = backtesting.run(
{ {
'stake_amount': default_conf['stake_amount'], 'stake_amount': default_conf['stake_amount'],
'processed': data_processed, 'processed': data_processed,
@ -524,7 +568,7 @@ def test_backtest_1min_ticker_interval(default_conf, fee, mocker) -> None:
# Run a backtesting for an exiting 5min ticker_interval # Run a backtesting for an exiting 5min ticker_interval
data = optimize.load_data(None, ticker_interval='1m', pairs=['UNITTEST/BTC']) data = optimize.load_data(None, ticker_interval='1m', pairs=['UNITTEST/BTC'])
data = trim_dictlist(data, -200) data = trim_dictlist(data, -200)
results = backtesting.backtest( results = backtesting.run(
{ {
'stake_amount': default_conf['stake_amount'], 'stake_amount': default_conf['stake_amount'],
'processed': backtesting.tickerdata_to_dataframe(data), 'processed': backtesting.tickerdata_to_dataframe(data),
@ -568,7 +612,7 @@ def test_backtest_ticks(default_conf, fee, mocker):
backtesting = Backtesting(default_conf) backtesting = Backtesting(default_conf)
backtesting.advise_buy = fun # Override backtesting.advise_buy = fun # Override
backtesting.advise_sell = fun # Override backtesting.advise_sell = fun # Override
results = backtesting.backtest(backtest_conf) results = backtesting.run(backtest_conf)
assert not results.empty assert not results.empty
@ -583,7 +627,7 @@ def test_backtest_clash_buy_sell(mocker, default_conf):
backtesting = Backtesting(default_conf) backtesting = Backtesting(default_conf)
backtesting.advise_buy = fun # Override backtesting.advise_buy = fun # Override
backtesting.advise_sell = fun # Override backtesting.advise_sell = fun # Override
results = backtesting.backtest(backtest_conf) results = backtesting.run(backtest_conf)
assert results.empty assert results.empty
@ -598,7 +642,7 @@ def test_backtest_only_sell(mocker, default_conf):
backtesting = Backtesting(default_conf) backtesting = Backtesting(default_conf)
backtesting.advise_buy = fun # Override backtesting.advise_buy = fun # Override
backtesting.advise_sell = fun # Override backtesting.advise_sell = fun # Override
results = backtesting.backtest(backtest_conf) results = backtesting.run(backtest_conf)
assert results.empty assert results.empty
@ -608,7 +652,7 @@ def test_backtest_alternate_buy_sell(default_conf, fee, mocker):
backtesting = Backtesting(default_conf) backtesting = Backtesting(default_conf)
backtesting.advise_buy = _trend_alternate # Override backtesting.advise_buy = _trend_alternate # Override
backtesting.advise_sell = _trend_alternate # Override backtesting.advise_sell = _trend_alternate # Override
results = backtesting.backtest(backtest_conf) results = backtesting.run(backtest_conf)
backtesting._store_backtest_result("test_.json", results) backtesting._store_backtest_result("test_.json", results)
assert len(results) == 4 assert len(results) == 4
# One trade was force-closed at the end # One trade was force-closed at the end
@ -621,7 +665,7 @@ def test_backtest_record(default_conf, fee, mocker):
patch_exchange(mocker) patch_exchange(mocker)
mocker.patch('freqtrade.exchange.Exchange.get_fee', fee) mocker.patch('freqtrade.exchange.Exchange.get_fee', fee)
mocker.patch( mocker.patch(
'freqtrade.optimize.backtesting.file_dump_json', 'freqtrade.optimize.optimize.file_dump_json',
new=lambda n, r: (names.append(n), records.append(r)) new=lambda n, r: (names.append(n), records.append(r))
) )
@ -654,6 +698,18 @@ def test_backtest_record(default_conf, fee, mocker):
records = records[0] records = records[0]
# Ensure records are of correct type # Ensure records are of correct type
assert len(records) == 4 assert len(records) == 4
# reset test to test with strategy name
names = []
records = []
backtesting._store_backtest_result("backtest-result.json", results, "DefStrat")
assert len(results) == 4
# Assert file_dump_json was only called once
assert names == ['backtest-result-DefStrat.json']
records = records[0]
# Ensure records are of correct type
assert len(records) == 4
# ('UNITTEST/BTC', 0.00331158, '1510684320', '1510691700', 0, 117) # ('UNITTEST/BTC', 0.00331158, '1510684320', '1510691700', 0, 117)
# Below follows just a typecheck of the schema/type of trade-records # Below follows just a typecheck of the schema/type of trade-records
oix = None oix = None
@ -677,24 +733,15 @@ def test_backtest_record(default_conf, fee, mocker):
def test_backtest_start_live(default_conf, mocker, caplog): def test_backtest_start_live(default_conf, mocker, caplog):
default_conf['exchange']['pair_whitelist'] = ['UNITTEST/BTC'] default_conf['exchange']['pair_whitelist'] = ['UNITTEST/BTC']
mocker.patch('freqtrade.exchange.Exchange.get_ticker_history', mocker.patch('freqtrade.exchange.Exchange.get_candle_history',
new=lambda s, n, i: _load_pair_as_ticks(n, i)) new=lambda s, n, i: _load_pair_as_ticks(n, i))
patch_exchange(mocker) patch_exchange(mocker)
mocker.patch('freqtrade.optimize.backtesting.Backtesting.backtest', MagicMock()) mocker.patch('freqtrade.optimize.backtesting.Backtesting.run', MagicMock())
mocker.patch('freqtrade.optimize.backtesting.Backtesting._generate_text_table', MagicMock()) mocker.patch('freqtrade.optimize.backtesting.Backtesting._generate_text_table', MagicMock())
mocker.patch('freqtrade.configuration.open', mocker.mock_open( mocker.patch('freqtrade.configuration.open', mocker.mock_open(
read_data=json.dumps(default_conf) read_data=json.dumps(default_conf)
)) ))
args = MagicMock()
args.ticker_interval = 1
args.level = 10
args.live = True
args.datadir = None
args.export = None
args.strategy = 'DefaultStrategy'
args.timerange = '-100' # needed due to MagicMock malleability
args = [ args = [
'--config', 'config.json', '--config', 'config.json',
'--strategy', 'DefaultStrategy', '--strategy', 'DefaultStrategy',
@ -725,3 +772,60 @@ def test_backtest_start_live(default_conf, mocker, caplog):
for line in exists: for line in exists:
assert log_has(line, caplog.record_tuples) assert log_has(line, caplog.record_tuples)
def test_backtest_start_multi_strat(default_conf, mocker, caplog):
default_conf['exchange']['pair_whitelist'] = ['UNITTEST/BTC']
mocker.patch('freqtrade.exchange.Exchange.get_candle_history',
new=lambda s, n, i: _load_pair_as_ticks(n, i))
patch_exchange(mocker)
backtestmock = MagicMock()
mocker.patch('freqtrade.optimize.backtesting.Backtesting.run', backtestmock)
gen_table_mock = MagicMock()
mocker.patch('freqtrade.optimize.backtesting.Backtesting._generate_text_table', gen_table_mock)
gen_strattable_mock = MagicMock()
mocker.patch('freqtrade.optimize.backtesting.Backtesting._generate_text_table_strategy',
gen_strattable_mock)
mocker.patch('freqtrade.configuration.open', mocker.mock_open(
read_data=json.dumps(default_conf)
))
args = [
'--config', 'config.json',
'--datadir', 'freqtrade/tests/testdata',
'backtesting',
'--ticker-interval', '1m',
'--live',
'--timerange', '-100',
'--enable-position-stacking',
'--disable-max-market-positions',
'--strategy-list',
'DefaultStrategy',
'TestStrategy',
]
args = get_args(args)
start(args)
# 2 backtests, 4 tables
assert backtestmock.call_count == 2
assert gen_table_mock.call_count == 4
assert gen_strattable_mock.call_count == 1
# check the logs, that will contain the backtest result
exists = [
'Parameter -i/--ticker-interval detected ...',
'Using ticker_interval: 1m ...',
'Parameter -l/--live detected ...',
'Ignoring max_open_trades (--disable-max-market-positions was used) ...',
'Parameter --timerange detected: -100 ...',
'Using data folder: freqtrade/tests/testdata ...',
'Using stake_currency: BTC ...',
'Using stake_amount: 0.001 ...',
'Downloading data for all pairs in whitelist ...',
'Measuring data from 2017-11-14T19:31:00+00:00 up to 2017-11-14T22:58:00+00:00 (0 days)..',
'Parameter --enable-position-stacking detected ...',
'Running backtesting for Strategy DefaultStrategy',
'Running backtesting for Strategy TestStrategy',
]
for line in exists:
assert log_has(line, caplog.record_tuples)

2
freqtrade/tests/optimize/test_hyperopt.py
View File

@ -263,7 +263,7 @@ def test_generate_optimizer(mocker, default_conf) -> None:
backtest_result = pd.DataFrame.from_records(trades, columns=labels) backtest_result = pd.DataFrame.from_records(trades, columns=labels)
mocker.patch( mocker.patch(
'freqtrade.optimize.hyperopt.Hyperopt.backtest', 'freqtrade.optimize.hyperopt.Hyperopt.run',
MagicMock(return_value=backtest_result) MagicMock(return_value=backtest_result)
) )
patch_exchange(mocker) patch_exchange(mocker)

16
freqtrade/tests/optimize/test_optimize.py
View File

@ -53,7 +53,7 @@ def _clean_test_file(file: str) -> None:
def test_load_data_30min_ticker(ticker_history, mocker, caplog, default_conf) -> None: def test_load_data_30min_ticker(ticker_history, mocker, caplog, default_conf) -> None:
mocker.patch('freqtrade.exchange.Exchange.get_ticker_history', return_value=ticker_history) mocker.patch('freqtrade.exchange.Exchange.get_candle_history', return_value=ticker_history)
file = os.path.join(os.path.dirname(__file__), '..', 'testdata', 'UNITTEST_BTC-30m.json') file = os.path.join(os.path.dirname(__file__), '..', 'testdata', 'UNITTEST_BTC-30m.json')
_backup_file(file, copy_file=True) _backup_file(file, copy_file=True)
optimize.load_data(None, pairs=['UNITTEST/BTC'], ticker_interval='30m') optimize.load_data(None, pairs=['UNITTEST/BTC'], ticker_interval='30m')
@ -63,7 +63,7 @@ def test_load_data_30min_ticker(ticker_history, mocker, caplog, default_conf) ->
def test_load_data_5min_ticker(ticker_history, mocker, caplog, default_conf) -> None: def test_load_data_5min_ticker(ticker_history, mocker, caplog, default_conf) -> None:
mocker.patch('freqtrade.exchange.Exchange.get_ticker_history', return_value=ticker_history) mocker.patch('freqtrade.exchange.Exchange.get_candle_history', return_value=ticker_history)
file = os.path.join(os.path.dirname(__file__), '..', 'testdata', 'UNITTEST_BTC-5m.json') file = os.path.join(os.path.dirname(__file__), '..', 'testdata', 'UNITTEST_BTC-5m.json')
_backup_file(file, copy_file=True) _backup_file(file, copy_file=True)
@ -74,7 +74,7 @@ def test_load_data_5min_ticker(ticker_history, mocker, caplog, default_conf) ->
def test_load_data_1min_ticker(ticker_history, mocker, caplog) -> None: def test_load_data_1min_ticker(ticker_history, mocker, caplog) -> None:
mocker.patch('freqtrade.exchange.Exchange.get_ticker_history', return_value=ticker_history) mocker.patch('freqtrade.exchange.Exchange.get_candle_history', return_value=ticker_history)
file = os.path.join(os.path.dirname(__file__), '..', 'testdata', 'UNITTEST_BTC-1m.json') file = os.path.join(os.path.dirname(__file__), '..', 'testdata', 'UNITTEST_BTC-1m.json')
_backup_file(file, copy_file=True) _backup_file(file, copy_file=True)
optimize.load_data(None, ticker_interval='1m', pairs=['UNITTEST/BTC']) optimize.load_data(None, ticker_interval='1m', pairs=['UNITTEST/BTC'])
@ -87,7 +87,7 @@ def test_load_data_with_new_pair_1min(ticker_history, mocker, caplog, default_co
""" """
Test load_data() with 1 min ticker Test load_data() with 1 min ticker
""" """
mocker.patch('freqtrade.exchange.Exchange.get_ticker_history', return_value=ticker_history) mocker.patch('freqtrade.exchange.Exchange.get_candle_history', return_value=ticker_history)
exchange = get_patched_exchange(mocker, default_conf) exchange = get_patched_exchange(mocker, default_conf)
file = os.path.join(os.path.dirname(__file__), '..', 'testdata', 'MEME_BTC-1m.json') file = os.path.join(os.path.dirname(__file__), '..', 'testdata', 'MEME_BTC-1m.json')
@ -118,7 +118,7 @@ def test_testdata_path() -> None:
def test_download_pairs(ticker_history, mocker, default_conf) -> None: def test_download_pairs(ticker_history, mocker, default_conf) -> None:
mocker.patch('freqtrade.exchange.Exchange.get_ticker_history', return_value=ticker_history) mocker.patch('freqtrade.exchange.Exchange.get_candle_history', return_value=ticker_history)
exchange = get_patched_exchange(mocker, default_conf) exchange = get_patched_exchange(mocker, default_conf)
file1_1 = os.path.join(os.path.dirname(__file__), '..', 'testdata', 'MEME_BTC-1m.json') file1_1 = os.path.join(os.path.dirname(__file__), '..', 'testdata', 'MEME_BTC-1m.json')
file1_5 = os.path.join(os.path.dirname(__file__), '..', 'testdata', 'MEME_BTC-5m.json') file1_5 = os.path.join(os.path.dirname(__file__), '..', 'testdata', 'MEME_BTC-5m.json')
@ -261,7 +261,7 @@ def test_load_cached_data_for_updating(mocker) -> None:
def test_download_pairs_exception(ticker_history, mocker, caplog, default_conf) -> None: def test_download_pairs_exception(ticker_history, mocker, caplog, default_conf) -> None:
mocker.patch('freqtrade.exchange.Exchange.get_ticker_history', return_value=ticker_history) mocker.patch('freqtrade.exchange.Exchange.get_candle_history', return_value=ticker_history)
mocker.patch('freqtrade.optimize.__init__.download_backtesting_testdata', mocker.patch('freqtrade.optimize.__init__.download_backtesting_testdata',
side_effect=BaseException('File Error')) side_effect=BaseException('File Error'))
exchange = get_patched_exchange(mocker, default_conf) exchange = get_patched_exchange(mocker, default_conf)
@ -279,7 +279,7 @@ def test_download_pairs_exception(ticker_history, mocker, caplog, default_conf)
def test_download_backtesting_testdata(ticker_history, mocker, default_conf) -> None: def test_download_backtesting_testdata(ticker_history, mocker, default_conf) -> None:
mocker.patch('freqtrade.exchange.Exchange.get_ticker_history', return_value=ticker_history) mocker.patch('freqtrade.exchange.Exchange.get_candle_history', return_value=ticker_history)
exchange = get_patched_exchange(mocker, default_conf) exchange = get_patched_exchange(mocker, default_conf)
# Download a 1 min ticker file # Download a 1 min ticker file
@ -304,7 +304,7 @@ def test_download_backtesting_testdata2(mocker, default_conf) -> None:
[1509836580000, 0.00161, 0.00161, 0.00161, 0.00161, 82.390199] [1509836580000, 0.00161, 0.00161, 0.00161, 0.00161, 82.390199]
] ]
json_dump_mock = mocker.patch('freqtrade.misc.file_dump_json', return_value=None) json_dump_mock = mocker.patch('freqtrade.misc.file_dump_json', return_value=None)
mocker.patch('freqtrade.exchange.Exchange.get_ticker_history', return_value=tick) mocker.patch('freqtrade.exchange.Exchange.get_candle_history', return_value=tick)
exchange = get_patched_exchange(mocker, default_conf) exchange = get_patched_exchange(mocker, default_conf)
download_backtesting_testdata(None, exchange, pair="UNITTEST/BTC", tick_interval='1m') download_backtesting_testdata(None, exchange, pair="UNITTEST/BTC", tick_interval='1m')
download_backtesting_testdata(None, exchange, pair="UNITTEST/BTC", tick_interval='3m') download_backtesting_testdata(None, exchange, pair="UNITTEST/BTC", tick_interval='3m')

2
freqtrade/tests/strategy/test_interface.py
View File

@ -88,7 +88,7 @@ def test_get_signal_old_dataframe(default_conf, mocker, caplog):
def test_get_signal_handles_exceptions(mocker, default_conf): def test_get_signal_handles_exceptions(mocker, default_conf):
mocker.patch('freqtrade.exchange.Exchange.get_ticker_history', return_value=MagicMock()) mocker.patch('freqtrade.exchange.Exchange.get_candle_history', return_value=MagicMock())
exchange = get_patched_exchange(mocker, default_conf) exchange = get_patched_exchange(mocker, default_conf)
mocker.patch.object( mocker.patch.object(
_STRATEGY, 'analyze_ticker', _STRATEGY, 'analyze_ticker',

8
freqtrade/tests/test_arguments.py
View File

@ -132,7 +132,11 @@ def test_parse_args_backtesting_custom() -> None:
'backtesting', 'backtesting',
'--live', '--live',
'--ticker-interval', '1m', '--ticker-interval', '1m',
'--refresh-pairs-cached'] '--refresh-pairs-cached',
'--strategy-list',
'DefaultStrategy',
'TestStrategy'
]
call_args = Arguments(args, '').get_parsed_arg() call_args = Arguments(args, '').get_parsed_arg()
assert call_args.config == 'test_conf.json' assert call_args.config == 'test_conf.json'
assert call_args.live is True assert call_args.live is True
@ -141,6 +145,8 @@ def test_parse_args_backtesting_custom() -> None:
assert call_args.func is not None assert call_args.func is not None
assert call_args.ticker_interval == '1m' assert call_args.ticker_interval == '1m'
assert call_args.refresh_pairs is True assert call_args.refresh_pairs is True
assert type(call_args.strategy_list) is list
assert len(call_args.strategy_list) == 2
def test_parse_args_hyperopt_custom() -> None: def test_parse_args_hyperopt_custom() -> None:

55
freqtrade/tests/test_configuration.py
View File

@ -292,6 +292,61 @@ def test_setup_configuration_with_arguments(mocker, default_conf, caplog) -> Non
) )
def test_setup_configuration_with_stratlist(mocker, default_conf, caplog) -> None:
"""
Test setup_configuration() function
"""
mocker.patch('freqtrade.configuration.open', mocker.mock_open(
read_data=json.dumps(default_conf)
))
arglist = [
'--config', 'config.json',
'backtesting',
'--ticker-interval', '1m',
'--export', '/bar/foo',
'--strategy-list',
'DefaultStrategy',
'TestStrategy'
]
args = Arguments(arglist, '').get_parsed_arg()
configuration = Configuration(args)
config = configuration.get_config()
assert 'max_open_trades' in config
assert 'stake_currency' in config
assert 'stake_amount' in config
assert 'exchange' in config
assert 'pair_whitelist' in config['exchange']
assert 'datadir' in config
assert log_has(
'Using data folder: {} ...'.format(config['datadir']),
caplog.record_tuples
)
assert 'ticker_interval' in config
assert log_has('Parameter -i/--ticker-interval detected ...', caplog.record_tuples)
assert log_has(
'Using ticker_interval: 1m ...',
caplog.record_tuples
)
assert 'strategy_list' in config
assert log_has('Using strategy list of 2 Strategies', caplog.record_tuples)
assert 'position_stacking' not in config
assert 'use_max_market_positions' not in config
assert 'timerange' not in config
assert 'export' in config
assert log_has(
'Parameter --export detected: {} ...'.format(config['export']),
caplog.record_tuples
)
def test_hyperopt_with_arguments(mocker, default_conf, caplog) -> None: def test_hyperopt_with_arguments(mocker, default_conf, caplog) -> None:
mocker.patch('freqtrade.configuration.open', mocker.mock_open( mocker.patch('freqtrade.configuration.open', mocker.mock_open(
read_data=json.dumps(default_conf) read_data=json.dumps(default_conf)

2
freqtrade/tests/test_dataframe.py
View File

@ -14,7 +14,7 @@ def load_dataframe_pair(pairs, strategy):
assert isinstance(pairs[0], str) assert isinstance(pairs[0], str)
dataframe = ld[pairs[0]] dataframe = ld[pairs[0]]
dataframe = strategy.analyze_ticker(dataframe, pairs[0]) dataframe = strategy.analyze_ticker(dataframe, {'pair': pairs[0]})
return dataframe return dataframe

4
freqtrade/tests/test_freqtradebot.py
View File

@ -43,7 +43,7 @@ def patch_get_signal(freqtrade: FreqtradeBot, value=(True, False)) -> None:
:return: None :return: None
""" """
freqtrade.strategy.get_signal = lambda e, s, t: value freqtrade.strategy.get_signal = lambda e, s, t: value
freqtrade.exchange.get_ticker_history = lambda p, i: None freqtrade.exchange.get_candle_history = lambda p, i: None
def patch_RPCManager(mocker) -> MagicMock: def patch_RPCManager(mocker) -> MagicMock:
@ -544,7 +544,7 @@ def test_create_trade_no_signal(default_conf, fee, mocker) -> None:
mocker.patch.multiple( mocker.patch.multiple(
'freqtrade.exchange.Exchange', 'freqtrade.exchange.Exchange',
validate_pairs=MagicMock(), validate_pairs=MagicMock(),
get_ticker_history=MagicMock(return_value=20), get_candle_history=MagicMock(return_value=20),
get_balance=MagicMock(return_value=20), get_balance=MagicMock(return_value=20),
get_fee=fee, get_fee=fee,
) )

12
freqtrade/tests/test_persistence.py
View File

@ -404,6 +404,7 @@ def test_migrate_new(mocker, default_conf, fee, caplog):
Test Database migration (starting with new pairformat) Test Database migration (starting with new pairformat)
""" """
amount = 103.223 amount = 103.223
# Always create all columns apart from the last!
create_table_old = """CREATE TABLE IF NOT EXISTS "trades" ( create_table_old = """CREATE TABLE IF NOT EXISTS "trades" (
id INTEGER NOT NULL, id INTEGER NOT NULL,
exchange VARCHAR NOT NULL, exchange VARCHAR NOT NULL,
@ -418,14 +419,21 @@ def test_migrate_new(mocker, default_conf, fee, caplog):
open_date DATETIME NOT NULL, open_date DATETIME NOT NULL,
close_date DATETIME, close_date DATETIME,
open_order_id VARCHAR, open_order_id VARCHAR,
stop_loss FLOAT,
initial_stop_loss FLOAT,
max_rate FLOAT,
sell_reason VARCHAR,
strategy VARCHAR,
PRIMARY KEY (id), PRIMARY KEY (id),
CHECK (is_open IN (0, 1)) CHECK (is_open IN (0, 1))
);""" );"""
insert_table_old = """INSERT INTO trades (exchange, pair, is_open, fee, insert_table_old = """INSERT INTO trades (exchange, pair, is_open, fee,
open_rate, stake_amount, amount, open_date) open_rate, stake_amount, amount, open_date,
stop_loss, initial_stop_loss, max_rate)
VALUES ('binance', 'ETC/BTC', 1, {fee}, VALUES ('binance', 'ETC/BTC', 1, {fee},
0.00258580, {stake}, {amount}, 0.00258580, {stake}, {amount},
'2019-11-28 12:44:24.000000') '2019-11-28 12:44:24.000000',
0.0, 0.0, 0.0)
""".format(fee=fee.return_value, """.format(fee=fee.return_value,
stake=default_conf.get("stake_amount"), stake=default_conf.get("stake_amount"),
amount=amount amount=amount

16
freqtrade/tests/test_talib.py Normal file
View File

@ -0,0 +1,16 @@
import talib.abstract as ta
import pandas as pd
def test_talib_bollingerbands_near_zero_values():
inputs = pd.DataFrame([
{'close': 0.00000010},
{'close': 0.00000011},
{'close': 0.00000012},
{'close': 0.00000013},
{'close': 0.00000014}
])
bollinger = ta.BBANDS(inputs, matype=0, timeperiod=2)
assert (bollinger['upperband'][3] != bollinger['middleband'][3])

2
install_ta-lib.sh
View File

@ -1,6 +1,6 @@
if [ ! -f "ta-lib/CHANGELOG.TXT" ]; then if [ ! -f "ta-lib/CHANGELOG.TXT" ]; then
tar zxvf ta-lib-0.4.0-src.tar.gz tar zxvf ta-lib-0.4.0-src.tar.gz
cd ta-lib && ./configure && make && sudo make install && cd .. cd ta-lib && sed -i "s|0.00000001|0.000000000000000001 |g" src/ta_func/ta_utility.h && ./configure && make && sudo make install && cd ..
else else
echo "TA-lib already installed, skipping download and build." echo "TA-lib already installed, skipping download and build."
cd ta-lib && sudo make install && cd .. cd ta-lib && sudo make install && cd ..

6
requirements.txt
View File

@ -1,4 +1,4 @@
ccxt==1.17.60 ccxt==1.17.118
SQLAlchemy==1.2.10 SQLAlchemy==1.2.10
python-telegram-bot==10.1.0 python-telegram-bot==10.1.0
arrow==0.12.1 arrow==0.12.1
@ -6,13 +6,13 @@ cachetools==2.1.0
requests==2.19.1 requests==2.19.1
urllib3==1.22 urllib3==1.22
wrapt==1.10.11 wrapt==1.10.11
pandas==0.23.3 pandas==0.23.4
scikit-learn==0.19.2 scikit-learn==0.19.2
scipy==1.1.0 scipy==1.1.0
jsonschema==2.6.0 jsonschema==2.6.0
numpy==1.15.0 numpy==1.15.0
TA-Lib==0.4.17 TA-Lib==0.4.17
pytest==3.7.0 pytest==3.7.1
pytest-mock==1.10.0 pytest-mock==1.10.0
pytest-cov==2.5.1 pytest-cov==2.5.1
tabulate==0.8.2 tabulate==0.8.2

93
scripts/get_market_pairs.py Normal file
View File

@ -0,0 +1,93 @@
import os
import sys
root = os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
sys.path.append(root + '/python')
import ccxt # noqa: E402
def style(s, style):
return style + s + '\033[0m'
def green(s):
return style(s, '\033[92m')
def blue(s):
return style(s, '\033[94m')
def yellow(s):
return style(s, '\033[93m')
def red(s):
return style(s, '\033[91m')
def pink(s):
return style(s, '\033[95m')
def bold(s):
return style(s, '\033[1m')
def underline(s):
return style(s, '\033[4m')
def dump(*args):
print(' '.join([str(arg) for arg in args]))
def print_supported_exchanges():
dump('Supported exchanges:', green(', '.join(ccxt.exchanges)))
try:
id = sys.argv[1] # get exchange id from command line arguments
# check if the exchange is supported by ccxt
exchange_found = id in ccxt.exchanges
if exchange_found:
dump('Instantiating', green(id), 'exchange')
# instantiate the exchange by id
exchange = getattr(ccxt, id)({
# 'proxy':'https://cors-anywhere.herokuapp.com/',
})
# load all markets from the exchange
markets = exchange.load_markets()
# output a list of all market symbols
dump(green(id), 'has', len(exchange.symbols), 'symbols:', exchange.symbols)
tuples = list(ccxt.Exchange.keysort(markets).items())
# debug
for (k, v) in tuples:
print(v)
# output a table of all markets
dump(pink('{:<15} {:<15} {:<15} {:<15}'.format('id', 'symbol', 'base', 'quote')))
for (k, v) in tuples:
dump('{:<15} {:<15} {:<15} {:<15}'.format(v['id'], v['symbol'], v['base'], v['quote']))
else:
dump('Exchange ' + red(id) + ' not found')
print_supported_exchanges()
except Exception as e:
dump('[' + type(e).__name__ + ']', str(e))
dump("Usage: python " + sys.argv[0], green('id'))
print_supported_exchanges()

2
scripts/plot_dataframe.py
View File

@ -138,7 +138,7 @@ def plot_analyzed_dataframe(args: Namespace) -> None:
tickers = {} tickers = {}
if args.live: if args.live:
logger.info('Downloading pair.') logger.info('Downloading pair.')
tickers[pair] = exchange.get_ticker_history(pair, tick_interval) tickers[pair] = exchange.get_candle_history(pair, tick_interval)
else: else:
tickers = optimize.load_data( tickers = optimize.load_data(
datadir=_CONF.get("datadir"), datadir=_CONF.get("datadir"),