merged upstream

This commit is contained in:
மனோஜ்குமார் பழனிச்சாமி 2022-05-13 19:36:26 +05:30
parent 203e22dc38
commit 782f3e5127
26 changed files with 1682 additions and 1493 deletions

View File

@ -16,8 +16,8 @@ repos:
- types-cachetools==5.0.1
- types-filelock==3.2.5
- types-requests==2.27.25
- types-tabulate==0.8.8
- types-python-dateutil==2.8.14
- types-tabulate==0.8.9
- types-python-dateutil==2.8.15
# stages: [push]
- repo: https://github.com/pycqa/isort

View File

@ -466,7 +466,7 @@ You can get an overview over daily / weekly or monthly results by using the `--b
To visualize daily and weekly breakdowns, you can use the following:
``` bash
freqtrade backtesting --strategy MyAwesomeStrategy --breakdown day month
freqtrade backtesting --strategy MyAwesomeStrategy --breakdown day week
```
``` output
@ -482,7 +482,7 @@ freqtrade backtesting --strategy MyAwesomeStrategy --breakdown day month
```
The output will show a table containing the realized absolute Profit (in stake currency) for the given timeperiod, as well as wins, draws and losses that materialized (closed) on this day.
The output will show a table containing the realized absolute Profit (in stake currency) for the given timeperiod, as well as wins, draws and losses that materialized (closed) on this day. Below that there will be a second table for the summarized values of weeks indicated by the date of the closing Sunday. The same would apply to a monthly breakdown indicated by the last day of the month.
### Backtest result caching

View File

@ -228,11 +228,12 @@ OKX requires a passphrase for each api key, you will therefore need to add this
```
!!! Warning
OKX only provides 300 candles per api call. Therefore, the strategy will only have a pretty low amount of data available in backtesting mode.
OKX only provides 100 candles per api call. Therefore, the strategy will only have a pretty low amount of data available in backtesting mode.
!!! Warning "Futures - position mode"
!!! Warning "Futures"
OKX Futures has the concept of "position mode" - which can be Net or long/short (hedge mode).
Freqtrade supports both modes - but changing the mode mid-trading is not supported and will lead to exceptions and failures to place trades.
OKX also only provides MARK candles for the past ~3 months. Backtesting futures prior to that date will therefore lead to slight deviations, as funding-fees cannot be calculated correctly without this data.
## Gate.io

View File

@ -44,7 +44,7 @@ It uses configuration from `exchange.pair_whitelist` and `exchange.pair_blacklis
```json
"pairlists": [
{"method": "StaticPairList"}
],
],
```
By default, only currently enabled pairs are allowed.
@ -181,7 +181,7 @@ Example to remove the first 10 pairs from the pairlist:
`VolumeFilter`.
!!! Note
An offset larger then the total length of the incoming pairlist will result in an empty pairlist.
An offset larger than the total length of the incoming pairlist will result in an empty pairlist.
#### PerformanceFilter

View File

@ -1,5 +1,5 @@
mkdocs==1.3.0
mkdocs-material==8.2.12
mkdocs-material==8.2.14
mdx_truly_sane_lists==1.2
pymdown-extensions==9.4
jinja2==3.1.2

View File

@ -119,6 +119,7 @@ This subcommand is useful for finding problems in your environment with loading
usage: freqtrade list-strategies [-h] [-v] [--logfile FILE] [-V] [-c PATH]
[-d PATH] [--userdir PATH]
[--strategy-path PATH] [-1] [--no-color]
[--recursive-strategy-search]
optional arguments:
-h, --help show this help message and exit
@ -126,6 +127,9 @@ optional arguments:
-1, --one-column Print output in one column.
--no-color Disable colorization of hyperopt results. May be
useful if you are redirecting output to a file.
--recursive-strategy-search
Recursively search for a strategy in the strategies
folder.
Common arguments:
-v, --verbose Verbose mode (-vv for more, -vvv to get all messages).
@ -134,9 +138,10 @@ Common arguments:
details.
-V, --version show program's version number and exit
-c PATH, --config PATH
Specify configuration file (default: `config.json`).
Multiple --config options may be used. Can be set to
`-` to read config from stdin.
Specify configuration file (default:
`userdir/config.json` or `config.json` whichever
exists). Multiple --config options may be used. Can be
set to `-` to read config from stdin.
-d PATH, --datadir PATH
Path to directory with historical backtesting data.
--userdir PATH, --user-data-dir PATH
@ -549,6 +554,27 @@ Show whitelist when using a [dynamic pairlist](plugins.md#pairlists).
freqtrade test-pairlist --config config.json --quote USDT BTC
```
## Convert database
`freqtrade convert-db` can be used to convert your database from one system to another (sqlite -> postgres, postgres -> other postgres), migrating all trades, orders and Pairlocks.
Please refer to the [SQL cheatsheet](sql_cheatsheet.md#use-a-different-database-system) to learn about requirements for different database systems.
```
usage: freqtrade convert-db [-h] [--db-url PATH] [--db-url-from PATH]
optional arguments:
-h, --help show this help message and exit
--db-url PATH Override trades database URL, this is useful in custom
deployments (default: `sqlite:///tradesv3.sqlite` for
Live Run mode, `sqlite:///tradesv3.dryrun.sqlite` for
Dry Run).
--db-url-from PATH Source db url to use when migrating a database.
```
!!! Warning
Please ensure to only use this on an empty target database. Freqtrade will perform a regular migration, but may fail if entries already existed.
## Webserver mode
!!! Warning "Experimental"

View File

@ -10,6 +10,7 @@ from freqtrade.commands.arguments import Arguments
from freqtrade.commands.build_config_commands import start_new_config
from freqtrade.commands.data_commands import (start_convert_data, start_convert_trades,
start_download_data, start_list_data)
from freqtrade.commands.db_commands import start_convert_db
from freqtrade.commands.deploy_commands import (start_create_userdir, start_install_ui,
start_new_strategy)
from freqtrade.commands.hyperopt_commands import start_hyperopt_list, start_hyperopt_show

View File

@ -82,7 +82,9 @@ ARGS_PLOT_DATAFRAME = ["pairs", "indicators1", "indicators2", "plot_limit",
ARGS_PLOT_PROFIT = ["pairs", "timerange", "export", "exportfilename", "db_url",
"trade_source", "timeframe", "plot_auto_open", ]
ARGS_INSTALL_UI = ["erase_ui_only", 'ui_version']
ARGS_CONVERT_DB = ["db_url", "db_url_from"]
ARGS_INSTALL_UI = ["erase_ui_only", "ui_version"]
ARGS_SHOW_TRADES = ["db_url", "trade_ids", "print_json"]
@ -181,7 +183,7 @@ class Arguments:
self._build_args(optionlist=['version'], parser=self.parser)
from freqtrade.commands import (start_backtesting, start_backtesting_show,
start_convert_data, start_convert_trades,
start_convert_data, start_convert_db, start_convert_trades,
start_create_userdir, start_download_data, start_edge,
start_hyperopt, start_hyperopt_list, start_hyperopt_show,
start_install_ui, start_list_data, start_list_exchanges,
@ -374,6 +376,14 @@ class Arguments:
test_pairlist_cmd.set_defaults(func=start_test_pairlist)
self._build_args(optionlist=ARGS_TEST_PAIRLIST, parser=test_pairlist_cmd)
# Add db-convert subcommand
convert_db = subparsers.add_parser(
"convert-db",
help="Migrate database to different system",
)
convert_db.set_defaults(func=start_convert_db)
self._build_args(optionlist=ARGS_CONVERT_DB, parser=convert_db)
# Add install-ui subcommand
install_ui_cmd = subparsers.add_parser(
'install-ui',

View File

@ -106,6 +106,11 @@ AVAILABLE_CLI_OPTIONS = {
f'`{constants.DEFAULT_DB_DRYRUN_URL}` for Dry Run).',
metavar='PATH',
),
"db_url_from": Arg(
'--db-url-from',
help='Source db url to use when migrating a database.',
metavar='PATH',
),
"sd_notify": Arg(
'--sd-notify',
help='Notify systemd service manager.',

View File

@ -0,0 +1,55 @@
import logging
from typing import Any, Dict
from sqlalchemy import func
from freqtrade.configuration.config_setup import setup_utils_configuration
from freqtrade.enums.runmode import RunMode
logger = logging.getLogger(__name__)
def start_convert_db(args: Dict[str, Any]) -> None:
from sqlalchemy.orm import make_transient
from freqtrade.persistence import Order, Trade, init_db
from freqtrade.persistence.migrations import set_sequence_ids
from freqtrade.persistence.pairlock import PairLock
config = setup_utils_configuration(args, RunMode.UTIL_NO_EXCHANGE)
init_db(config['db_url'], False)
session_target = Trade._session
init_db(config['db_url_from'], False)
logger.info("Starting db migration.")
trade_count = 0
pairlock_count = 0
for trade in Trade.get_trades():
trade_count += 1
make_transient(trade)
for o in trade.orders:
make_transient(o)
session_target.add(trade)
session_target.commit()
for pairlock in PairLock.query:
pairlock_count += 1
make_transient(pairlock)
session_target.add(pairlock)
session_target.commit()
# Update sequences
max_trade_id = session_target.query(func.max(Trade.id)).scalar()
max_order_id = session_target.query(func.max(Order.id)).scalar()
max_pairlock_id = session_target.query(func.max(PairLock.id)).scalar()
set_sequence_ids(session_target.get_bind(),
trade_id=max_trade_id,
order_id=max_order_id,
pairlock_id=max_pairlock_id)
logger.info(f"Migrated {trade_count} Trades, and {pairlock_count} Pairlocks.")

View File

@ -147,6 +147,9 @@ class Configuration:
config.update({'db_url': self.args['db_url']})
logger.info('Parameter --db-url detected ...')
self._args_to_config(config, argname='db_url_from',
logstring='Parameter --db-url-from detected ...')
if config.get('force_entry_enable', False):
logger.warning('`force_entry_enable` RPC message enabled.')

View File

@ -20,7 +20,7 @@ class Okx(Exchange):
"""
_ft_has: Dict = {
"ohlcv_candle_limit": 300,
"ohlcv_candle_limit": 100,
"mark_ohlcv_timeframe": "4h",
"funding_fee_timeframe": "8h",
}

View File

@ -1,5 +1,5 @@
# flake8: noqa: F401
from freqtrade.persistence.models import (LocalTrade, Order, Trade, clean_dry_run_db, cleanup_db,
init_db)
from freqtrade.persistence.models import clean_dry_run_db, cleanup_db, init_db
from freqtrade.persistence.pairlock_middleware import PairLocks
from freqtrade.persistence.trade_model import LocalTrade, Order, Trade

View File

@ -0,0 +1,7 @@
from typing import Any
from sqlalchemy.orm import declarative_base
_DECL_BASE: Any = declarative_base()

View File

@ -46,7 +46,7 @@ def get_last_sequence_ids(engine, trade_back_name, order_back_name):
return order_id, trade_id
def set_sequence_ids(engine, order_id, trade_id):
def set_sequence_ids(engine, order_id, trade_id, pairlock_id=None):
if engine.name == 'postgresql':
with engine.begin() as connection:
@ -54,6 +54,9 @@ def set_sequence_ids(engine, order_id, trade_id):
connection.execute(text(f"ALTER SEQUENCE orders_id_seq RESTART WITH {order_id}"))
if trade_id:
connection.execute(text(f"ALTER SEQUENCE trades_id_seq RESTART WITH {trade_id}"))
if pairlock_id:
connection.execute(
text(f"ALTER SEQUENCE pairlocks_id_seq RESTART WITH {pairlock_id}"))
def drop_index_on_table(engine, inspector, table_bak_name):
@ -99,7 +102,10 @@ def migrate_trades_and_orders_table(
liquidation_price = get_column_def(cols, 'liquidation_price',
get_column_def(cols, 'isolated_liq', 'null'))
# sqlite does not support literals for booleans
is_short = get_column_def(cols, 'is_short', '0')
if engine.name == 'postgresql':
is_short = get_column_def(cols, 'is_short', 'false')
else:
is_short = get_column_def(cols, 'is_short', '0')
# Margin Properties
interest_rate = get_column_def(cols, 'interest_rate', '0.0')

File diff suppressed because it is too large Load Diff

View File

@ -0,0 +1,70 @@
from datetime import datetime, timezone
from typing import Any, Dict, Optional
from sqlalchemy import Boolean, Column, DateTime, Integer, String, or_
from sqlalchemy.orm import Query
from freqtrade.constants import DATETIME_PRINT_FORMAT
from freqtrade.persistence.base import _DECL_BASE
class PairLock(_DECL_BASE):
"""
Pair Locks database model.
"""
__tablename__ = 'pairlocks'
id = Column(Integer, primary_key=True)
pair = Column(String(25), nullable=False, index=True)
# lock direction - long, short or * (for both)
side = Column(String(25), nullable=False, default="*")
reason = Column(String(255), nullable=True)
# Time the pair was locked (start time)
lock_time = Column(DateTime, nullable=False)
# Time until the pair is locked (end time)
lock_end_time = Column(DateTime, nullable=False, index=True)
active = Column(Boolean, nullable=False, default=True, index=True)
def __repr__(self):
lock_time = self.lock_time.strftime(DATETIME_PRINT_FORMAT)
lock_end_time = self.lock_end_time.strftime(DATETIME_PRINT_FORMAT)
return (
f'PairLock(id={self.id}, pair={self.pair}, side={self.side}, lock_time={lock_time}, '
f'lock_end_time={lock_end_time}, reason={self.reason}, active={self.active})')
@staticmethod
def query_pair_locks(pair: Optional[str], now: datetime, side: str = '*') -> Query:
"""
Get all currently active locks for this pair
:param pair: Pair to check for. Returns all current locks if pair is empty
:param now: Datetime object (generated via datetime.now(timezone.utc)).
"""
filters = [PairLock.lock_end_time > now,
# Only active locks
PairLock.active.is_(True), ]
if pair:
filters.append(PairLock.pair == pair)
if side != '*':
filters.append(or_(PairLock.side == side, PairLock.side == '*'))
else:
filters.append(PairLock.side == '*')
return PairLock.query.filter(
*filters
)
def to_json(self) -> Dict[str, Any]:
return {
'id': self.id,
'pair': self.pair,
'lock_time': self.lock_time.strftime(DATETIME_PRINT_FORMAT),
'lock_timestamp': int(self.lock_time.replace(tzinfo=timezone.utc).timestamp() * 1000),
'lock_end_time': self.lock_end_time.strftime(DATETIME_PRINT_FORMAT),
'lock_end_timestamp': int(self.lock_end_time.replace(tzinfo=timezone.utc
).timestamp() * 1000),
'reason': self.reason,
'side': self.side,
'active': self.active,
}

File diff suppressed because it is too large Load Diff

View File

@ -177,16 +177,19 @@ class RPC:
current_rate = NAN
else:
current_rate = trade.close_rate
current_profit = trade.calc_profit_ratio(current_rate)
current_profit_abs = trade.calc_profit(current_rate)
current_profit_fiat: Optional[float] = None
# Calculate fiat profit
if self._fiat_converter:
current_profit_fiat = self._fiat_converter.convert_amount(
current_profit_abs,
self._freqtrade.config['stake_currency'],
self._freqtrade.config['fiat_display_currency']
)
if len(trade.select_filled_orders(trade.entry_side)) > 0:
current_profit = trade.calc_profit_ratio(current_rate)
current_profit_abs = trade.calc_profit(current_rate)
current_profit_fiat: Optional[float] = None
# Calculate fiat profit
if self._fiat_converter:
current_profit_fiat = self._fiat_converter.convert_amount(
current_profit_abs,
self._freqtrade.config['stake_currency'],
self._freqtrade.config['fiat_display_currency']
)
else:
current_profit = current_profit_abs = current_profit_fiat = 0.0
# Calculate guaranteed profit (in case of trailing stop)
stoploss_entry_dist = trade.calc_profit(trade.stop_loss)
@ -235,8 +238,12 @@ class RPC:
trade.pair, side='exit', is_short=trade.is_short, refresh=False)
except (PricingError, ExchangeError):
current_rate = NAN
trade_profit = trade.calc_profit(current_rate)
profit_str = f'{trade.calc_profit_ratio(current_rate):.2%}'
if len(trade.select_filled_orders(trade.entry_side)) > 0:
trade_profit = trade.calc_profit(current_rate)
profit_str = f'{trade.calc_profit_ratio(current_rate):.2%}'
else:
trade_profit = 0.0
profit_str = f'{0.0:.2f}'
direction_str = ('S' if trade.is_short else 'L') if nonspot else ''
if self._fiat_converter:
fiat_profit = self._fiat_converter.convert_amount(
@ -244,7 +251,7 @@ class RPC:
stake_currency,
fiat_display_currency
)
if fiat_profit and not isnan(fiat_profit):
if not isnan(fiat_profit):
profit_str += f" ({fiat_profit:.2f})"
fiat_profit_sum = fiat_profit if isnan(fiat_profit_sum) \
else fiat_profit_sum + fiat_profit

View File

@ -15,10 +15,8 @@ from freqtrade.data.dataprovider import DataProvider
from freqtrade.enums import (CandleType, ExitCheckTuple, ExitType, SignalDirection, SignalTagType,
SignalType, TradingMode)
from freqtrade.exceptions import OperationalException, StrategyError
from freqtrade.exchange import timeframe_to_minutes, timeframe_to_seconds
from freqtrade.exchange.exchange import timeframe_to_next_date
from freqtrade.persistence import PairLocks, Trade
from freqtrade.persistence.models import LocalTrade, Order
from freqtrade.exchange import timeframe_to_minutes, timeframe_to_next_date, timeframe_to_seconds
from freqtrade.persistence import LocalTrade, Order, PairLocks, Trade
from freqtrade.strategy.hyper import HyperStrategyMixin
from freqtrade.strategy.informative_decorator import (InformativeData, PopulateIndicators,
_create_and_merge_informative_pair,

View File

@ -300,7 +300,7 @@ class Wallets:
if min_stake_amount is not None and min_stake_amount > max_stake_amount:
if self._log:
logger.warning("Minimum stake amount > available balance."
logger.warning("Minimum stake amount > available balance. "
f"{min_stake_amount} > {max_stake_amount}")
return 0
if min_stake_amount is not None and stake_amount < min_stake_amount:

View File

@ -6,9 +6,9 @@
coveralls==3.3.1
flake8==4.0.1
flake8-tidy-imports==4.6.0
flake8-tidy-imports==4.7.0
mypy==0.950
pre-commit==2.18.1
pre-commit==2.19.0
pytest==7.1.2
pytest-asyncio==0.18.3
pytest-cov==3.0.0
@ -25,5 +25,5 @@ nbconvert==6.5.0
types-cachetools==5.0.1
types-filelock==3.2.5
types-requests==2.27.25
types-tabulate==0.8.8
types-python-dateutil==2.8.14
types-tabulate==0.8.9
types-python-dateutil==2.8.15

View File

@ -2,9 +2,9 @@ numpy==1.22.3
pandas==1.4.2
pandas-ta==0.3.14b
ccxt==1.81.43
ccxt==1.81.81
# Pin cryptography for now due to rust build errors with piwheels
cryptography==37.0.1
cryptography==37.0.2
aiohttp==3.8.1
SQLAlchemy==1.4.36
python-telegram-bot==13.11
@ -12,7 +12,7 @@ arrow==1.2.2
cachetools==4.2.2
requests==2.27.1
urllib3==1.26.9
jsonschema==4.4.0
jsonschema==4.5.1
TA-Lib==0.4.24
technical==1.3.0
tabulate==0.8.9
@ -34,7 +34,7 @@ orjson==3.6.8
sdnotify==0.3.2
# API Server
fastapi==0.75.2
fastapi==0.76.0
uvicorn==0.17.6
pyjwt==2.3.0
aiofiles==0.8.0

View File

@ -1,5 +1,6 @@
import json
import re
from datetime import datetime
from io import BytesIO
from pathlib import Path
from unittest.mock import MagicMock, PropertyMock
@ -14,11 +15,14 @@ from freqtrade.commands import (start_backtesting_show, start_convert_data, star
start_list_exchanges, start_list_markets, start_list_strategies,
start_list_timeframes, start_new_strategy, start_show_trades,
start_test_pairlist, start_trading, start_webserver)
from freqtrade.commands.db_commands import start_convert_db
from freqtrade.commands.deploy_commands import (clean_ui_subdir, download_and_install_ui,
get_ui_download_url, read_ui_version)
from freqtrade.configuration import setup_utils_configuration
from freqtrade.enums import RunMode
from freqtrade.exceptions import OperationalException
from freqtrade.persistence.models import init_db
from freqtrade.persistence.pairlock_middleware import PairLocks
from tests.conftest import (CURRENT_TEST_STRATEGY, create_mock_trades, get_args, log_has,
log_has_re, patch_exchange, patched_configuration_load_config_file)
from tests.conftest_trades import MOCK_TRADE_COUNT
@ -1458,3 +1462,33 @@ def test_backtesting_show(mocker, testdatadir, capsys):
assert sbr.call_count == 1
out, err = capsys.readouterr()
assert "Pairs for Strategy" in out
def test_start_convert_db(mocker, fee, tmpdir, caplog):
db_src_file = Path(f"{tmpdir}/db.sqlite")
db_from = f"sqlite:///{db_src_file}"
db_target_file = Path(f"{tmpdir}/db_target.sqlite")
db_to = f"sqlite:///{db_target_file}"
args = [
"convert-db",
"--db-url-from",
db_from,
"--db-url",
db_to,
]
assert not db_src_file.is_file()
init_db(db_from, False)
create_mock_trades(fee)
PairLocks.timeframe = '5m'
PairLocks.lock_pair('XRP/USDT', datetime.now(), 'Random reason 125', side='long')
assert db_src_file.is_file()
assert not db_target_file.is_file()
pargs = get_args(args)
pargs['config'] = None
start_convert_db(pargs)
assert db_target_file.is_file()

View File

@ -235,9 +235,20 @@ def test_rpc_status_table(default_conf, ticker, fee, mocker) -> None:
freqtradebot.state = State.RUNNING
with pytest.raises(RPCException, match=r'.*no active trade*'):
rpc._rpc_status_table(default_conf['stake_currency'], 'USD')
mocker.patch('freqtrade.exchange.Exchange._is_dry_limit_order_filled', return_value=False)
freqtradebot.enter_positions()
result, headers, fiat_profit_sum = rpc._rpc_status_table(default_conf['stake_currency'], 'USD')
assert "Since" in headers
assert "Pair" in headers
assert 'instantly' == result[0][2]
assert 'ETH/BTC' in result[0][1]
assert '0.00' == result[0][3]
assert isnan(fiat_profit_sum)
mocker.patch('freqtrade.exchange.Exchange._is_dry_limit_order_filled', return_value=True)
freqtradebot.process()
result, headers, fiat_profit_sum = rpc._rpc_status_table(default_conf['stake_currency'], 'USD')
assert "Since" in headers
assert "Pair" in headers
@ -245,8 +256,8 @@ def test_rpc_status_table(default_conf, ticker, fee, mocker) -> None:
assert 'ETH/BTC' in result[0][1]
assert '-0.41%' == result[0][3]
assert isnan(fiat_profit_sum)
# Test with fiatconvert
# Test with fiatconvert
rpc._fiat_converter = CryptoToFiatConverter()
result, headers, fiat_profit_sum = rpc._rpc_status_table(default_conf['stake_currency'], 'USD')
assert "Since" in headers

View File

@ -1416,14 +1416,14 @@ def test_migrate_set_sequence_ids():
engine = MagicMock()
engine.begin = MagicMock()
engine.name = 'postgresql'
set_sequence_ids(engine, 22, 55)
set_sequence_ids(engine, 22, 55, 5)
assert engine.begin.call_count == 1
engine.reset_mock()
engine.begin.reset_mock()
engine.name = 'somethingelse'
set_sequence_ids(engine, 22, 55)
set_sequence_ids(engine, 22, 55, 6)
assert engine.begin.call_count == 0