diff --git a/docs/edge.md b/docs/edge.md index b208cb318..a4acffc44 100644 --- a/docs/edge.md +++ b/docs/edge.md @@ -3,159 +3,198 @@ This page explains how to use Edge Positioning module in your bot in order to enter into a trade only if the trade has a reasonable win rate and risk reward ratio, and consequently adjust your position size and stoploss. !!! Warning - Edge positioning is not compatible with dynamic whitelist. it overrides dynamic whitelist. + Edge positioning is not compatible with dynamic whitelist. If enabled, it overrides the dynamic whitelist option. !!! Note - Edge won't consider anything else than buy/sell/stoploss signals. So trailing stoploss, ROI, and everything else will be ignored in its calculation. + Edge does not consider anything else than buy/sell/stoploss signals. So trailing stoploss, ROI, and everything else are ignored in its calculation. ## Introduction -Trading is all about probability. No one can claim that he has a strategy working all the time. You have to assume that sometimes you lose.

-But it doesn't mean there is no rule, it only means rules should work "most of the time". Let's play a game: we toss a coin, heads: I give you 10$, tails: You give me 10$. Is it an interesting game ? no, it is quite boring, isn't it?

-But let's say the probability that we have heads is 80%, and the probability that we have tails is 20%. Now it is becoming interesting ... -That means 10$ x 80% versus 10$ x 20%. 8$ versus 2$. That means over time you will win 8$ risking only 2$ on each toss of coin.

-Let's complicate it more: you win 80% of the time but only 2$, I win 20% of the time but 8$. The calculation is: 80% * 2$ versus 20% * 8$. It is becoming boring again because overtime you win $1.6$ (80% x 2$) and me $1.6 (20% * 8$) too.

-The question is: How do you calculate that? how do you know if you wanna play? +Trading is all about probability. No one can claim that he has a strategy working all the time. You have to assume that sometimes you lose. + +But it doesn't mean there is no rule, it only means rules should work "most of the time". Let's play a game: we toss a coin, heads: I give you 10$, tails: you give me 10$. Is it an interesting game? No, it's quite boring, isn't it? + +But let's say the probability that we have heads is 80% (because our coin has the displaced distribution of mass or other defect), and the probability that we have tails is 20%. Now it is becoming interesting... + +That means 10$ X 80% versus 10$ X 20%. 8$ versus 2$. That means over time you will win 8$ risking only 2$ on each toss of coin. + +Let's complicate it more: you win 80% of the time but only 2$, I win 20% of the time but 8$. The calculation is: 80% X 2$ versus 20% X 8$. It is becoming boring again because overtime you win $1.6$ (80% X 2$) and me $1.6 (20% X 8$) too. + +The question is: How do you calculate that? How do you know if you wanna play? + The answer comes to two factors: - Win Rate - Risk Reward Ratio - ### Win Rate -Means over X trades what is the percentage of winning trades to total number of trades (note that we don't consider how much you gained but only If you won or not). +Win Rate (*W*) is is the mean over some amount of trades (*N*) what is the percentage of winning trades to total number of trades (note that we don't consider how much you gained but only if you won or not). + W = (Number of winning trades) / (Total number of trades) = (Number of winning trades) / N -`W = (Number of winning trades) / (Total number of trades)` +Complementary Loss Rate (*L*) is defined as + + L = (Number of losing trades) / (Total number of trades) = (Number of losing trades) / N + +or, which is the same, as + + R = 1 – W ### Risk Reward Ratio -Risk Reward Ratio is a formula used to measure the expected gains of a given investment against the risk of loss. It is basically what you potentially win divided by what you potentially lose: +Risk Reward Ratio (*R*) is a formula used to measure the expected gains of a given investment against the risk of loss. It is basically what you potentially win divided by what you potentially lose: -`R = Profit / Loss` + R = Profit / Loss Over time, on many trades, you can calculate your risk reward by dividing your average profit on winning trades by your average loss on losing trades: -`Average profit = (Sum of profits) / (Number of winning trades)` + Average profit = (Sum of profits) / (Number of winning trades) -`Average loss = (Sum of losses) / (Number of losing trades)` + Average loss = (Sum of losses) / (Number of losing trades) -`R = (Average profit) / (Average loss)` + R = (Average profit) / (Average loss) ### Expectancy +At this point we can combine *W* and *R* to create an expectancy ratio. This is a simple process of multiplying the risk reward ratio by the percentage of winning trades and subtracting the percentage of losing trades, which is calculated as follows: -At this point we can combine W and R to create an expectancy ratio. This is a simple process of multiplying the risk reward ratio by the percentage of winning trades, and subtracting the percentage of losing trades, which is calculated as follows: - -Expectancy Ratio = (Risk Reward Ratio x Win Rate) – Loss Rate + Expectancy Ratio = (Risk Reward Ratio X Win Rate) – Loss Rate = (R X W) – L So lets say your Win rate is 28% and your Risk Reward Ratio is 5: -`Expectancy = (5 * 0.28) - 0.72 = 0.68` + Expectancy = (5 X 0.28) – 0.72 = 0.68 -Superficially, this means that on average you expect this strategy’s trades to return .68 times the size of your losers. This is important for two reasons: First, it may seem obvious, but you know right away that you have a positive return. Second, you now have a number you can compare to other candidate systems to make decisions about which ones you employ. +Superficially, this means that on average you expect this strategy’s trades to return .68 times the size of your loses. This is important for two reasons: First, it may seem obvious, but you know right away that you have a positive return. Second, you now have a number you can compare to other candidate systems to make decisions about which ones you employ. It is important to remember that any system with an expectancy greater than 0 is profitable using past data. The key is finding one that will be profitable in the future. -You can also use this number to evaluate the effectiveness of modifications to this system. +You can also use this value to evaluate the effectiveness of modifications to this system. -**NOTICE:** It's important to keep in mind that Edge is testing your expectancy using historical data , there's no guarantee that you will have a similar edge in the future. It's still vital to do this testing in order to build confidence in your methodology, but be wary of "curve-fitting" your approach to the historical data as things are unlikely to play out the exact same way for future trades. +**NOTICE:** It's important to keep in mind that Edge is testing your expectancy using historical data, there's no guarantee that you will have a similar edge in the future. It's still vital to do this testing in order to build confidence in your methodology, but be wary of "curve-fitting" your approach to the historical data as things are unlikely to play out the exact same way for future trades. ## How does it work? -If enabled in config, Edge will go through historical data with a range of stoplosses in order to find buy and sell/stoploss signals. It then calculates win rate and expectancy over X trades for each stoploss. Here is an example: +If enabled in config, Edge will go through historical data with a range of stoplosses in order to find buy and sell/stoploss signals. It then calculates win rate and expectancy over *N* trades for each stoploss. Here is an example: | Pair | Stoploss | Win Rate | Risk Reward Ratio | Expectancy | |----------|:-------------:|-------------:|------------------:|-----------:| -| XZC/ETH | -0.03 | 0.52 |1.359670 | 0.228 | | XZC/ETH | -0.01 | 0.50 |1.176384 | 0.088 | | XZC/ETH | -0.02 | 0.51 |1.115941 | 0.079 | +| XZC/ETH | -0.03 | 0.52 |1.359670 | 0.228 | +| XZC/ETH | -0.04 | 0.51 |1.234539 | 0.117 | The goal here is to find the best stoploss for the strategy in order to have the maximum expectancy. In the above example stoploss at 3% leads to the maximum expectancy according to historical data. -Edge then forces stoploss to your strategy dynamically. +Edge module then forces stoploss value it evaluated to your strategy dynamically. ### Position size -Edge dictates the stake amount for each trade to the bot according to the following factors: +Edge also dictates the stake amount for each trade to the bot according to the following factors: - Allowed capital at risk - Stoploss Allowed capital at risk is calculated as follows: -**allowed capital at risk** = **capital_available_percentage** X **allowed risk per trade** + Allowed capital at risk = (Capital available_percentage) X (Allowed risk per trade) -**Stoploss** is calculated as described above against historical data. +Stoploss is calculated as described above against historical data. Your position size then will be: -**position size** = **allowed capital at risk** / **stoploss** + Position size = (Allowed capital at risk) / Stoploss -Example:
-Let's say the stake currency is ETH and you have 10 ETH on the exchange, your **capital_available_percentage** is 50% and you would allow 1% of risk for each trade. thus your available capital for trading is **10 x 0.5 = 5 ETH** and allowed capital at risk would be **5 x 0.01 = 0.05 ETH**.
-Let's assume Edge has calculated that for **XLM/ETH** market your stoploss should be at 2%. So your position size will be **0.05 / 0.02 = 2.5ETH**.
-Bot takes a position of 2.5ETH on XLM/ETH (call it trade 1). Up next, you receive another buy signal while trade 1 is still open. This time on BTC/ETH market. Edge calculated stoploss for this market at 4%. So your position size would be 0.05 / 0.04 = 1.25ETH (call it trade 2).
-Note that available capital for trading didn’t change for trade 2 even if you had already trade 1. The available capital doesn’t mean the free amount on your wallet.
-Now you have two trades open. The Bot receives yet another buy signal for another market: **ADA/ETH**. This time the stoploss is calculated at 1%. So your position size is **0.05 / 0.01 = 5ETH**. But there are already 4ETH blocked in two previous trades. So the position size for this third trade would be 1ETH.
-Available capital doesn’t change before a position is sold. Let’s assume that trade 1 receives a sell signal and it is sold with a profit of 1ETH. Your total capital on exchange would be 11 ETH and the available capital for trading becomes 5.5ETH.
-So the Bot receives another buy signal for trade 4 with a stoploss at 2% then your position size would be **0.055 / 0.02 = 2.75**. +Example: + +Let's say the stake currency is ETH and you have 10 ETH on the exchange, your capital available percentage is 50% and you would allow 1% of risk for each trade. thus your available capital for trading is **10 x 0.5 = 5 ETH** and allowed capital at risk would be **5 x 0.01 = 0.05 ETH**. + +Let's assume Edge has calculated that for **XLM/ETH** market your stoploss should be at 2%. So your position size will be **0.05 / 0.02 = 2.5 ETH**. + +Bot takes a position of 2.5 ETH on XLM/ETH (call it trade 1). Up next, you receive another buy signal while trade 1 is still open. This time on **BTC/ETH** market. Edge calculated stoploss for this market at 4%. So your position size would be 0.05 / 0.04 = 1.25 ETH (call it trade 2). + +Note that available capital for trading didn’t change for trade 2 even if you had already trade 1. The available capital doesn’t mean the free amount on your wallet. + +Now you have two trades open. The bot receives yet another buy signal for another market: **ADA/ETH**. This time the stoploss is calculated at 1%. So your position size is **0.05 / 0.01 = 5 ETH**. But there are already 3.75 ETH blocked in two previous trades. So the position size for this third trade would be **5 – 3.75 = 1.25 ETH**. + +Available capital doesn’t change before a position is sold. Let’s assume that trade 1 receives a sell signal and it is sold with a profit of 1 ETH. Your total capital on exchange would be 11 ETH and the available capital for trading becomes 5.5 ETH. + +So the Bot receives another buy signal for trade 4 with a stoploss at 2% then your position size would be **0.055 / 0.02 = 2.75 ETH**. ## Configurations -Edge has following configurations: +Edge module has following configuration options: #### enabled -If true, then Edge will run periodically.
-(default to false) +If true, then Edge will run periodically. + +(defaults to false) #### process_throttle_secs -How often should Edge run in seconds?
-(default to 3600 so one hour) +How often should Edge run in seconds? + +(defaults to 3600 so one hour) #### calculate_since_number_of_days Number of days of data against which Edge calculates Win Rate, Risk Reward and Expectancy -Note that it downloads historical data so increasing this number would lead to slowing down the bot.
-(default to 7) +Note that it downloads historical data so increasing this number would lead to slowing down the bot. + +(defaults to 7) #### capital_available_percentage -This is the percentage of the total capital on exchange in stake currency.
-As an example if you have 10 ETH available in your wallet on the exchange and this value is 0.5 (which is 50%), then the bot will use a maximum amount of 5 ETH for trading and considers it as available capital.
-(default to 0.5) +This is the percentage of the total capital on exchange in stake currency. + +As an example if you have 10 ETH available in your wallet on the exchange and this value is 0.5 (which is 50%), then the bot will use a maximum amount of 5 ETH for trading and considers it as available capital. + +(defaults to 0.5) #### allowed_risk -Percentage of allowed risk per trade.
-(default to 0.01 [1%]) +Percentage of allowed risk per trade. + +(defaults to 0.01 so 1%) #### stoploss_range_min -Minimum stoploss.
-(default to -0.01) +Minimum stoploss. + +(defaults to -0.01) #### stoploss_range_max -Maximum stoploss.
-(default to -0.10) +Maximum stoploss. + +(defaults to -0.10) #### stoploss_range_step -As an example if this is set to -0.01 then Edge will test the strategy for [-0.01, -0,02, -0,03 ..., -0.09, -0.10] ranges. -Note than having a smaller step means having a bigger range which could lead to slow calculation.
-if you set this parameter to -0.001, you then slow down the Edge calculation by a factor of 10.
-(default to -0.01) +As an example if this is set to -0.01 then Edge will test the strategy for \[-0.01, -0,02, -0,03 ..., -0.09, -0.10\] ranges. +Note than having a smaller step means having a bigger range which could lead to slow calculation. + +If you set this parameter to -0.001, you then slow down the Edge calculation by a factor of 10. + +(defaults to -0.01) #### minimum_winrate -It filters pairs which don't have at least minimum_winrate. -This comes handy if you want to be conservative and don't comprise win rate in favor of risk reward ratio.
-(default to 0.60) +It filters out pairs which don't have at least minimum_winrate. + +This comes handy if you want to be conservative and don't comprise win rate in favour of risk reward ratio. + +(defaults to 0.60) #### minimum_expectancy -It filters paris which have an expectancy lower than this number . -Having an expectancy of 0.20 means if you put 10$ on a trade you expect a 12$ return.
-(default to 0.20) +It filters out pairs which have the expectancy lower than this number. + +Having an expectancy of 0.20 means if you put 10$ on a trade you expect a 12$ return. + +(defaults to 0.20) #### min_trade_number -When calculating W and R and E (expectancy) against historical data, you always want to have a minimum number of trades. The more this number is the more Edge is reliable. Having a win rate of 100% on a single trade doesn't mean anything at all. But having a win rate of 70% over past 100 trades means clearly something.
-(default to 10, it is highly recommended not to decrease this number) +When calculating *W*, *R* and *E* (expectancy) against historical data, you always want to have a minimum number of trades. The more this number is the more Edge is reliable. + +Having a win rate of 100% on a single trade doesn't mean anything at all. But having a win rate of 70% over past 100 trades means clearly something. + +(defaults to 10, it is highly recommended not to decrease this number) #### max_trade_duration_minute -Edge will filter out trades with long duration. If a trade is profitable after 1 month, it is hard to evaluate the strategy based on it. But if most of trades are profitable and they have maximum duration of 30 minutes, then it is clearly a good sign.
-**NOTICE:** While configuring this value, you should take into consideration your ticker interval. as an example filtering out trades having duration less than one day for a strategy which has 4h interval does not make sense. default value is set assuming your strategy interval is relatively small (1m or 5m, etc).
-(default to 1 day, 1440 = 60 * 24) +Edge will filter out trades with long duration. If a trade is profitable after 1 month, it is hard to evaluate the strategy based on it. But if most of trades are profitable and they have maximum duration of 30 minutes, then it is clearly a good sign. + +**NOTICE:** While configuring this value, you should take into consideration your ticker interval. As an example filtering out trades having duration less than one day for a strategy which has 4h interval does not make sense. Default value is set assuming your strategy interval is relatively small (1m or 5m, etc.). + +(defaults to 1 day, i.e. to 60 * 24 = 1440 minutes) #### remove_pumps -Edge will remove sudden pumps in a given market while going through historical data. However, given that pumps happen very often in crypto markets, we recommend you keep this off.
-(default to false) +Edge will remove sudden pumps in a given market while going through historical data. However, given that pumps happen very often in crypto markets, we recommend you keep this off. + +(defaults to false) ## Running Edge independently @@ -199,14 +238,14 @@ python3 ./freqtrade/main.py edge --stoplosses=-0.01,-0.1,-0.001 #min,max,step python3 ./freqtrade/main.py edge --timerange=20181110-20181113 ``` -Doing --timerange=-200 will get the last 200 timeframes from your inputdata. You can also specify specific dates, or a range span indexed by start and stop. +Doing `--timerange=-200` will get the last 200 timeframes from your inputdata. You can also specify specific dates, or a range span indexed by start and stop. The full timerange specification: -* Use last 123 tickframes of data: --timerange=-123 -* Use first 123 tickframes of data: --timerange=123- -* Use tickframes from line 123 through 456: --timerange=123-456 -* Use tickframes till 2018/01/31: --timerange=-20180131 -* Use tickframes since 2018/01/31: --timerange=20180131- -* Use tickframes since 2018/01/31 till 2018/03/01 : --timerange=20180131-20180301 -* Use tickframes between POSIX timestamps 1527595200 1527618600: --timerange=1527595200-1527618600 +* Use last 123 tickframes of data: `--timerange=-123` +* Use first 123 tickframes of data: `--timerange=123-` +* Use tickframes from line 123 through 456: `--timerange=123-456` +* Use tickframes till 2018/01/31: `--timerange=-20180131` +* Use tickframes since 2018/01/31: `--timerange=20180131-` +* Use tickframes since 2018/01/31 till 2018/03/01 : `--timerange=20180131-20180301` +* Use tickframes between POSIX timestamps 1527595200 1527618600: `--timerange=1527595200-1527618600` diff --git a/docs/webhook-config.md b/docs/webhook-config.md index e5509d6c9..2b5365e32 100644 --- a/docs/webhook-config.md +++ b/docs/webhook-config.md @@ -41,7 +41,6 @@ Possible parameters are: * exchange * pair -* market_url * limit * stake_amount * stake_amount_fiat @@ -56,7 +55,6 @@ Possible parameters are: * exchange * pair * gain -* market_url * limit * amount * open_rate diff --git a/freqtrade/arguments.py b/freqtrade/arguments.py index 9b1b9a925..62f22befc 100644 --- a/freqtrade/arguments.py +++ b/freqtrade/arguments.py @@ -6,9 +6,7 @@ import argparse import os import re from typing import List, NamedTuple, Optional - import arrow - from freqtrade import __version__, constants @@ -55,6 +53,11 @@ class Arguments(object): """ parsed_arg = self.parser.parse_args(self.args) + # Workaround issue in argparse with action='append' and default value + # (see https://bugs.python.org/issue16399) + if parsed_arg.config is None: + parsed_arg.config = [constants.DEFAULT_CONFIG] + return parsed_arg def common_args_parser(self) -> None: @@ -63,7 +66,7 @@ class Arguments(object): """ self.parser.add_argument( '-v', '--verbose', - help='verbose mode (-vv for more, -vvv to get all messages)', + help='Verbose mode (-vv for more, -vvv to get all messages).', action='count', dest='loglevel', default=0, @@ -75,15 +78,16 @@ class Arguments(object): ) self.parser.add_argument( '-c', '--config', - help='specify configuration file (default: %(default)s)', + help='Specify configuration file (default: %(default)s). ' + 'Multiple --config options may be used.', dest='config', - default='config.json', + action='append', type=str, metavar='PATH', ) self.parser.add_argument( '-d', '--datadir', - help='path to backtest data', + help='Path to backtest data.', dest='datadir', default=None, type=str, @@ -91,7 +95,7 @@ class Arguments(object): ) self.parser.add_argument( '-s', '--strategy', - help='specify strategy class name (default: %(default)s)', + help='Specify strategy class name (default: %(default)s).', dest='strategy', default='DefaultStrategy', type=str, @@ -99,14 +103,14 @@ class Arguments(object): ) self.parser.add_argument( '--strategy-path', - help='specify additional strategy lookup path', + help='Specify additional strategy lookup path.', dest='strategy_path', type=str, metavar='PATH', ) self.parser.add_argument( '--customhyperopt', - help='specify hyperopt class name (default: %(default)s)', + help='Specify hyperopt class name (default: %(default)s).', dest='hyperopt', default=constants.DEFAULT_HYPEROPT, type=str, @@ -114,8 +118,8 @@ class Arguments(object): ) self.parser.add_argument( '--dynamic-whitelist', - help='dynamically generate and update whitelist' - ' based on 24h BaseVolume (default: %(const)s)' + help='Dynamically generate and update whitelist' + ' based on 24h BaseVolume (default: %(const)s).' ' DEPRECATED.', dest='dynamic_whitelist', const=constants.DYNAMIC_WHITELIST, @@ -126,7 +130,7 @@ class Arguments(object): self.parser.add_argument( '--db-url', help='Override trades database URL, this is useful if dry_run is enabled' - ' or in custom deployments (default: %(default)s)', + ' or in custom deployments (default: %(default)s).', dest='db_url', type=str, metavar='PATH', @@ -139,7 +143,7 @@ class Arguments(object): """ parser.add_argument( '--eps', '--enable-position-stacking', - help='Allow buying the same pair multiple times (position stacking)', + help='Allow buying the same pair multiple times (position stacking).', action='store_true', dest='position_stacking', default=False @@ -148,20 +152,20 @@ class Arguments(object): parser.add_argument( '--dmmp', '--disable-max-market-positions', help='Disable applying `max_open_trades` during backtest ' - '(same as setting `max_open_trades` to a very high number)', + '(same as setting `max_open_trades` to a very high number).', action='store_false', dest='use_max_market_positions', default=True ) parser.add_argument( '-l', '--live', - help='using live data', + help='Use live data.', action='store_true', dest='live', ) parser.add_argument( '-r', '--refresh-pairs-cached', - help='refresh the pairs files in tests/testdata with the latest data from the ' + help='Refresh the pairs files in tests/testdata with the latest data from the ' 'exchange. Use it if you want to run your backtesting with up-to-date data.', action='store_true', dest='refresh_pairs', @@ -178,8 +182,8 @@ class Arguments(object): ) parser.add_argument( '--export', - help='export backtest results, argument are: trades\ - Example --export=trades', + help='Export backtest results, argument are: trades. ' + 'Example --export=trades', type=str, default=None, dest='export', @@ -203,14 +207,14 @@ class Arguments(object): """ parser.add_argument( '-r', '--refresh-pairs-cached', - help='refresh the pairs files in tests/testdata with the latest data from the ' + help='Refresh the pairs files in tests/testdata with the latest data from the ' 'exchange. Use it if you want to run your edge with up-to-date data.', action='store_true', dest='refresh_pairs', ) parser.add_argument( '--stoplosses', - help='defines a range of stoploss against which edge will assess the strategy ' + help='Defines a range of stoploss against which edge will assess the strategy ' 'the format is "min,max,step" (without any space).' 'example: --stoplosses=-0.01,-0.1,-0.001', type=str, @@ -226,14 +230,14 @@ class Arguments(object): """ parser.add_argument( '-i', '--ticker-interval', - help='specify ticker interval (1m, 5m, 30m, 1h, 1d)', + help='Specify ticker interval (1m, 5m, 30m, 1h, 1d).', dest='ticker_interval', type=str, ) parser.add_argument( '--timerange', - help='specify what timerange of data to use.', + help='Specify what timerange of data to use.', default=None, type=str, dest='timerange', @@ -246,7 +250,7 @@ class Arguments(object): """ parser.add_argument( '--eps', '--enable-position-stacking', - help='Allow buying the same pair multiple times (position stacking)', + help='Allow buying the same pair multiple times (position stacking).', action='store_true', dest='position_stacking', default=False @@ -255,14 +259,14 @@ class Arguments(object): parser.add_argument( '--dmmp', '--disable-max-market-positions', help='Disable applying `max_open_trades` during backtest ' - '(same as setting `max_open_trades` to a very high number)', + '(same as setting `max_open_trades` to a very high number).', action='store_false', dest='use_max_market_positions', default=True ) parser.add_argument( '-e', '--epochs', - help='specify number of epochs (default: %(default)d)', + help='Specify number of epochs (default: %(default)d).', dest='epochs', default=constants.HYPEROPT_EPOCH, type=int, @@ -271,7 +275,7 @@ class Arguments(object): parser.add_argument( '-s', '--spaces', help='Specify which parameters to hyperopt. Space separate list. \ - Default: %(default)s', + Default: %(default)s.', choices=['all', 'buy', 'sell', 'roi', 'stoploss'], default='all', nargs='+', @@ -288,19 +292,19 @@ class Arguments(object): subparsers = self.parser.add_subparsers(dest='subparser') # Add backtesting subcommand - backtesting_cmd = subparsers.add_parser('backtesting', help='backtesting module') + backtesting_cmd = subparsers.add_parser('backtesting', help='Backtesting module.') backtesting_cmd.set_defaults(func=backtesting.start) self.optimizer_shared_options(backtesting_cmd) self.backtesting_options(backtesting_cmd) # Add edge subcommand - edge_cmd = subparsers.add_parser('edge', help='edge module') + edge_cmd = subparsers.add_parser('edge', help='Edge module.') edge_cmd.set_defaults(func=edge_cli.start) self.optimizer_shared_options(edge_cmd) self.edge_options(edge_cmd) # Add hyperopt subcommand - hyperopt_cmd = subparsers.add_parser('hyperopt', help='hyperopt module') + hyperopt_cmd = subparsers.add_parser('hyperopt', help='Hyperopt module.') hyperopt_cmd.set_defaults(func=hyperopt.start) self.optimizer_shared_options(hyperopt_cmd) self.hyperopt_options(hyperopt_cmd) @@ -364,7 +368,7 @@ class Arguments(object): """ self.parser.add_argument( '--pairs-file', - help='File containing a list of pairs to download', + help='File containing a list of pairs to download.', dest='pairs_file', default=None, metavar='PATH', @@ -372,7 +376,7 @@ class Arguments(object): self.parser.add_argument( '--export', - help='Export files to given dir', + help='Export files to given dir.', dest='export', default=None, metavar='PATH', @@ -380,16 +384,17 @@ class Arguments(object): self.parser.add_argument( '-c', '--config', - help='specify configuration file, used for additional exchange parameters', + help='Specify configuration file (default: %(default)s). ' + 'Multiple --config options may be used.', dest='config', - default=None, + action='append', type=str, metavar='PATH', ) self.parser.add_argument( '--days', - help='Download data for number of days', + help='Download data for given number of days.', dest='days', type=int, metavar='INT', @@ -398,7 +403,7 @@ class Arguments(object): self.parser.add_argument( '--exchange', - help='Exchange name (default: %(default)s). Only valid if no config is provided', + help='Exchange name (default: %(default)s). Only valid if no config is provided.', dest='exchange', type=str, default='bittrex' @@ -407,7 +412,7 @@ class Arguments(object): self.parser.add_argument( '-t', '--timeframes', help='Specify which tickers to download. Space separated list. \ - Default: %(default)s', + Default: %(default)s.', choices=['1m', '3m', '5m', '15m', '30m', '1h', '2h', '4h', '6h', '8h', '12h', '1d', '3d', '1w'], default=['1m', '5m'], @@ -417,7 +422,7 @@ class Arguments(object): self.parser.add_argument( '--erase', - help='Clean all existing data for the selected exchange/pairs/timeframes', + help='Clean all existing data for the selected exchange/pairs/timeframes.', dest='erase', action='store_true' ) diff --git a/freqtrade/configuration.py b/freqtrade/configuration.py index d972f50b8..bddf60028 100644 --- a/freqtrade/configuration.py +++ b/freqtrade/configuration.py @@ -13,6 +13,8 @@ from jsonschema.exceptions import ValidationError, best_match from freqtrade import OperationalException, constants from freqtrade.state import RunMode +from freqtrade.misc import deep_merge_dicts + logger = logging.getLogger(__name__) @@ -45,8 +47,18 @@ class Configuration(object): Extract information for sys.argv and load the bot configuration :return: Configuration dictionary """ - logger.info('Using config: %s ...', self.args.config) - config = self._load_config_file(self.args.config) + config: Dict[str, Any] = {} + # Now expecting a list of config filenames here, not a string + for path in self.args.config: + logger.info('Using config: %s ...', path) + # Merge config options, overwriting old values + config = deep_merge_dicts(self._load_config_file(path), config) + + if 'internals' not in config: + config['internals'] = {} + + logger.info('Validating configuration ...') + self._validate_config(config) # Set strategy if not specified in config and or if it's non default if self.args.strategy != constants.DEFAULT_STRATEGY or not config.get('strategy'): @@ -93,11 +105,7 @@ class Configuration(object): f'Config file "{path}" not found!' ' Please create a config file or check whether it exists.') - if 'internals' not in conf: - conf['internals'] = {} - logger.info('Validating configuration ...') - - return self._validate_config(conf) + return conf def _load_common_config(self, config: Dict[str, Any]) -> Dict[str, Any]: """ diff --git a/freqtrade/constants.py b/freqtrade/constants.py index 1b3e25249..d0fafbd26 100644 --- a/freqtrade/constants.py +++ b/freqtrade/constants.py @@ -3,6 +3,7 @@ """ bot constants """ +DEFAULT_CONFIG = 'config.json' DYNAMIC_WHITELIST = 20 # pairs PROCESS_THROTTLE_SECS = 5 # sec TICKER_INTERVAL = 5 # min diff --git a/freqtrade/edge/__init__.py b/freqtrade/edge/__init__.py index baef811de..b4dfa5624 100644 --- a/freqtrade/edge/__init__.py +++ b/freqtrade/edge/__init__.py @@ -351,91 +351,93 @@ class Edge(): return result def _detect_next_stop_or_sell_point(self, buy_column, sell_column, date_column, - ohlc_columns, stoploss, pair, start_point=0): + ohlc_columns, stoploss, pair): """ - Iterate through ohlc_columns recursively in order to find the next trade + Iterate through ohlc_columns in order to find the next trade Next trade opens from the first buy signal noticed to The sell or stoploss signal after it. - It then calls itself cutting OHLC, buy_column, sell_colum and date_column - Cut from (the exit trade index) + 1 + It then cuts OHLC, buy_column, sell_column and date_column. + Cut from (the exit trade index) + 1. + Author: https://github.com/mishaker """ result: list = [] - open_trade_index = utf1st.find_1st(buy_column, 1, utf1st.cmp_equal) + start_point = 0 - # return empty if we don't find trade entry (i.e. buy==1) or - # we find a buy but at the of array - if open_trade_index == -1 or open_trade_index == len(buy_column) - 1: - return [] - else: - open_trade_index += 1 # when a buy signal is seen, - # trade opens in reality on the next candle + while True: + open_trade_index = utf1st.find_1st(buy_column, 1, utf1st.cmp_equal) - stop_price_percentage = stoploss + 1 - open_price = ohlc_columns[open_trade_index, 0] - stop_price = (open_price * stop_price_percentage) + # Return empty if we don't find trade entry (i.e. buy==1) or + # we find a buy but at the end of array + if open_trade_index == -1 or open_trade_index == len(buy_column) - 1: + break + else: + # When a buy signal is seen, + # trade opens in reality on the next candle + open_trade_index += 1 - # Searching for the index where stoploss is hit - stop_index = utf1st.find_1st( - ohlc_columns[open_trade_index:, 2], stop_price, utf1st.cmp_smaller) + stop_price_percentage = stoploss + 1 + open_price = ohlc_columns[open_trade_index, 0] + stop_price = (open_price * stop_price_percentage) - # If we don't find it then we assume stop_index will be far in future (infinite number) - if stop_index == -1: - stop_index = float('inf') + # Searching for the index where stoploss is hit + stop_index = utf1st.find_1st( + ohlc_columns[open_trade_index:, 2], stop_price, utf1st.cmp_smaller) - # Searching for the index where sell is hit - sell_index = utf1st.find_1st(sell_column[open_trade_index:], 1, utf1st.cmp_equal) + # If we don't find it then we assume stop_index will be far in future (infinite number) + if stop_index == -1: + stop_index = float('inf') - # If we don't find it then we assume sell_index will be far in future (infinite number) - if sell_index == -1: - sell_index = float('inf') + # Searching for the index where sell is hit + sell_index = utf1st.find_1st(sell_column[open_trade_index:], 1, utf1st.cmp_equal) - # Check if we don't find any stop or sell point (in that case trade remains open) - # It is not interesting for Edge to consider it so we simply ignore the trade - # And stop iterating there is no more entry - if stop_index == sell_index == float('inf'): - return [] + # If we don't find it then we assume sell_index will be far in future (infinite number) + if sell_index == -1: + sell_index = float('inf') - if stop_index <= sell_index: - exit_index = open_trade_index + stop_index - exit_type = SellType.STOP_LOSS - exit_price = stop_price - elif stop_index > sell_index: - # if exit is SELL then we exit at the next candle - exit_index = open_trade_index + sell_index + 1 + # Check if we don't find any stop or sell point (in that case trade remains open) + # It is not interesting for Edge to consider it so we simply ignore the trade + # And stop iterating there is no more entry + if stop_index == sell_index == float('inf'): + break - # check if we have the next candle - if len(ohlc_columns) - 1 < exit_index: - return [] + if stop_index <= sell_index: + exit_index = open_trade_index + stop_index + exit_type = SellType.STOP_LOSS + exit_price = stop_price + elif stop_index > sell_index: + # If exit is SELL then we exit at the next candle + exit_index = open_trade_index + sell_index + 1 - exit_type = SellType.SELL_SIGNAL - exit_price = ohlc_columns[exit_index, 0] + # Check if we have the next candle + if len(ohlc_columns) - 1 < exit_index: + break - trade = {'pair': pair, - 'stoploss': stoploss, - 'profit_percent': '', - 'profit_abs': '', - 'open_time': date_column[open_trade_index], - 'close_time': date_column[exit_index], - 'open_index': start_point + open_trade_index, - 'close_index': start_point + exit_index, - 'trade_duration': '', - 'open_rate': round(open_price, 15), - 'close_rate': round(exit_price, 15), - 'exit_type': exit_type - } + exit_type = SellType.SELL_SIGNAL + exit_price = ohlc_columns[exit_index, 0] - result.append(trade) + trade = {'pair': pair, + 'stoploss': stoploss, + 'profit_percent': '', + 'profit_abs': '', + 'open_time': date_column[open_trade_index], + 'close_time': date_column[exit_index], + 'open_index': start_point + open_trade_index, + 'close_index': start_point + exit_index, + 'trade_duration': '', + 'open_rate': round(open_price, 15), + 'close_rate': round(exit_price, 15), + 'exit_type': exit_type + } - # Calling again the same function recursively but giving - # it a view of exit_index till the end of array - return result + self._detect_next_stop_or_sell_point( - buy_column[exit_index:], - sell_column[exit_index:], - date_column[exit_index:], - ohlc_columns[exit_index:], - stoploss, - pair, - (start_point + exit_index) - ) + result.append(trade) + + # Giving a view of exit_index till the end of array + buy_column = buy_column[exit_index:] + sell_column = sell_column[exit_index:] + date_column = date_column[exit_index:] + ohlc_columns = ohlc_columns[exit_index:] + start_point += exit_index + + return result diff --git a/freqtrade/exchange/__init__.py b/freqtrade/exchange/__init__.py index 204ed971e..f6db04da6 100644 --- a/freqtrade/exchange/__init__.py +++ b/freqtrade/exchange/__init__.py @@ -1,2 +1,3 @@ from freqtrade.exchange.exchange import Exchange # noqa: F401 from freqtrade.exchange.kraken import Kraken # noqa: F401 +from freqtrade.exchange.binance import Binance # noqa: F401 diff --git a/freqtrade/exchange/binance.py b/freqtrade/exchange/binance.py new file mode 100644 index 000000000..127f4e916 --- /dev/null +++ b/freqtrade/exchange/binance.py @@ -0,0 +1,26 @@ +""" Binance exchange subclass """ +import logging +from typing import Dict + +from freqtrade.exchange import Exchange + +logger = logging.getLogger(__name__) + + +class Binance(Exchange): + + _ft_has: Dict = { + "stoploss_on_exchange": True, + } + + def get_order_book(self, pair: str, limit: int = 100) -> dict: + """ + get order book level 2 from exchange + + 20180619: binance support limits but only on specific range + """ + limit_range = [5, 10, 20, 50, 100, 500, 1000] + # get next-higher step in the limit_range list + limit = min(list(filter(lambda x: limit <= x, limit_range))) + + return super().get_order_book(pair, limit) diff --git a/freqtrade/exchange/exchange.py b/freqtrade/exchange/exchange.py index 0ca939ec5..f5503002b 100644 --- a/freqtrade/exchange/exchange.py +++ b/freqtrade/exchange/exchange.py @@ -24,7 +24,7 @@ API_RETRY_COUNT = 4 # Urls to exchange markets, insert quote and base with .format() _EXCHANGE_URLS = { ccxt.bittrex.__name__: '/Market/Index?MarketName={quote}-{base}', - ccxt.binance.__name__: '/tradeDetail.html?symbol={base}_{quote}' + ccxt.binance.__name__: '/tradeDetail.html?symbol={base}_{quote}', } @@ -69,11 +69,17 @@ class Exchange(object): _conf: Dict = {} _params: Dict = {} + # Dict to specify which options each exchange implements + # TODO: this should be merged with attributes from subclasses + # To avoid having to copy/paste this to all subclasses. + _ft_has = { + "stoploss_on_exchange": False, + } + def __init__(self, config: dict) -> None: """ Initializes this module with the given config, - it does basic validation whether the specified - exchange and pairs are valid. + it does basic validation whether the specified exchange and pairs are valid. :return: None """ self._conf.update(config) @@ -236,11 +242,11 @@ class Exchange(object): raise OperationalException( f'Exchange {self.name} does not support market orders.') - if order_types.get('stoploss_on_exchange'): - if self.name != 'Binance': - raise OperationalException( - 'On exchange stoploss is not supported for %s.' % self.name - ) + if (order_types.get("stoploss_on_exchange") + and not self._ft_has.get("stoploss_on_exchange", False)): + raise OperationalException( + 'On exchange stoploss is not supported for %s.' % self.name + ) def validate_order_time_in_force(self, order_time_in_force: Dict) -> None: """ @@ -282,104 +288,96 @@ class Exchange(object): price = ceil(big_price) / pow(10, symbol_prec) return price - def buy(self, pair: str, ordertype: str, amount: float, - rate: float, time_in_force) -> Dict: - if self._conf['dry_run']: - order_id = f'dry_run_buy_{randint(0, 10**6)}' - self._dry_run_open_orders[order_id] = { - 'pair': pair, - 'price': rate, - 'amount': amount, - 'type': ordertype, - 'side': 'buy', - 'remaining': 0.0, - 'datetime': arrow.utcnow().isoformat(), - 'status': 'closed', - 'fee': None - } - return {'id': order_id} + def dry_run_order(self, pair: str, ordertype: str, side: str, amount: float, + rate: float, params: Dict = {}) -> Dict[str, Any]: + order_id = f'dry_run_{side}_{randint(0, 10**6)}' + dry_order = { # TODO: additional entry should be added for stoploss limit + "id": order_id, + 'pair': pair, + 'price': rate, + 'amount': amount, + "cost": amount * rate, + 'type': ordertype, + 'side': 'buy', + 'remaining': amount, + 'datetime': arrow.utcnow().isoformat(), + 'status': "open", + 'fee': None, + "info": {} + } + self._store_dry_order(dry_order) + return dry_order + def _store_dry_order(self, dry_order: Dict) -> None: + closed_order = dry_order.copy() + if closed_order["type"] in ["market", "limit"]: + closed_order.update({ + "status": "closed", + "filled": closed_order["amount"], + "remaining": 0 + }) + self._dry_run_open_orders[closed_order["id"]] = closed_order + + def create_order(self, pair: str, ordertype: str, side: str, amount: float, + rate: float, params: Dict = {}) -> Dict: try: # Set the precision for amount and price(rate) as accepted by the exchange amount = self.symbol_amount_prec(pair, amount) rate = self.symbol_price_prec(pair, rate) if ordertype != 'market' else None - params = self._params.copy() - if time_in_force != 'gtc': - params.update({'timeInForce': time_in_force}) - - return self._api.create_order(pair, ordertype, 'buy', + return self._api.create_order(pair, ordertype, side, amount, rate, params) except ccxt.InsufficientFunds as e: raise DependencyException( - f'Insufficient funds to create limit buy order on market {pair}.' - f'Tried to buy amount {amount} at rate {rate} (total {rate*amount}).' + f'Insufficient funds to create {ordertype} {side} order on market {pair}.' + f'Tried to {side} amount {amount} at rate {rate} (total {rate*amount}).' f'Message: {e}') except ccxt.InvalidOrder as e: raise DependencyException( - f'Could not create limit buy order on market {pair}.' - f'Tried to buy amount {amount} at rate {rate} (total {rate*amount}).' + f'Could not create {ordertype} {side} order on market {pair}.' + f'Tried to {side} amount {amount} at rate {rate} (total {rate*amount}).' f'Message: {e}') except (ccxt.NetworkError, ccxt.ExchangeError) as e: raise TemporaryError( - f'Could not place buy order due to {e.__class__.__name__}. Message: {e}') + f'Could not place {side} order due to {e.__class__.__name__}. Message: {e}') except ccxt.BaseError as e: raise OperationalException(e) + def buy(self, pair: str, ordertype: str, amount: float, + rate: float, time_in_force) -> Dict: + + if self._conf['dry_run']: + dry_order = self.dry_run_order(pair, ordertype, "buy", amount, rate) + return dry_order + + params = self._params.copy() + if time_in_force != 'gtc': + params.update({'timeInForce': time_in_force}) + + return self.create_order(pair, ordertype, 'buy', amount, rate, params) + def sell(self, pair: str, ordertype: str, amount: float, rate: float, time_in_force='gtc') -> Dict: + if self._conf['dry_run']: - order_id = f'dry_run_sell_{randint(0, 10**6)}' - self._dry_run_open_orders[order_id] = { - 'pair': pair, - 'price': rate, - 'amount': amount, - 'type': ordertype, - 'side': 'sell', - 'remaining': 0.0, - 'datetime': arrow.utcnow().isoformat(), - 'status': 'closed' - } - return {'id': order_id} + dry_order = self.dry_run_order(pair, ordertype, "sell", amount, rate) + return dry_order - try: - # Set the precision for amount and price(rate) as accepted by the exchange - amount = self.symbol_amount_prec(pair, amount) - rate = self.symbol_price_prec(pair, rate) if ordertype != 'market' else None + params = self._params.copy() + if time_in_force != 'gtc': + params.update({'timeInForce': time_in_force}) - params = self._params.copy() - if time_in_force != 'gtc': - params.update({'timeInForce': time_in_force}) - - return self._api.create_order(pair, ordertype, 'sell', - amount, rate, params) - - except ccxt.InsufficientFunds as e: - raise DependencyException( - f'Insufficient funds to create limit sell order on market {pair}.' - f'Tried to sell amount {amount} at rate {rate} (total {rate*amount}).' - f'Message: {e}') - except ccxt.InvalidOrder as e: - raise DependencyException( - f'Could not create limit sell order on market {pair}.' - f'Tried to sell amount {amount} at rate {rate} (total {rate*amount}).' - f'Message: {e}') - except (ccxt.NetworkError, ccxt.ExchangeError) as e: - raise TemporaryError( - f'Could not place sell order due to {e.__class__.__name__}. Message: {e}') - except ccxt.BaseError as e: - raise OperationalException(e) + return self.create_order(pair, ordertype, 'sell', amount, rate, params) def stoploss_limit(self, pair: str, amount: float, stop_price: float, rate: float) -> Dict: """ creates a stoploss limit order. NOTICE: it is not supported by all exchanges. only binance is tested for now. + TODO: implementation maybe needs to be moved to the binance subclass """ + ordertype = "stop_loss_limit" - # Set the precision for amount and price(rate) as accepted by the exchange - amount = self.symbol_amount_prec(pair, amount) - rate = self.symbol_price_prec(pair, rate) stop_price = self.symbol_price_prec(pair, stop_price) # Ensure rate is less than stop price @@ -388,50 +386,17 @@ class Exchange(object): 'In stoploss limit order, stop price should be more than limit price') if self._conf['dry_run']: - order_id = f'dry_run_buy_{randint(0, 10**6)}' - self._dry_run_open_orders[order_id] = { - 'info': {}, - 'id': order_id, - 'pair': pair, - 'price': stop_price, - 'amount': amount, - 'type': 'stop_loss_limit', - 'side': 'sell', - 'remaining': amount, - 'datetime': arrow.utcnow().isoformat(), - 'status': 'open', - 'fee': None - } - return self._dry_run_open_orders[order_id] + dry_order = self.dry_run_order( + pair, ordertype, "sell", amount, stop_price) + return dry_order - try: + params = self._params.copy() + params.update({'stopPrice': stop_price}) - params = self._params.copy() - params.update({'stopPrice': stop_price}) - - order = self._api.create_order(pair, 'stop_loss_limit', 'sell', - amount, rate, params) - logger.info('stoploss limit order added for %s. ' - 'stop price: %s. limit: %s' % (pair, stop_price, rate)) - return order - - except ccxt.InsufficientFunds as e: - raise DependencyException( - f'Insufficient funds to place stoploss limit order on market {pair}. ' - f'Tried to put a stoploss amount {amount} with ' - f'stop {stop_price} and limit {rate} (total {rate*amount}).' - f'Message: {e}') - except ccxt.InvalidOrder as e: - raise DependencyException( - f'Could not place stoploss limit order on market {pair}.' - f'Tried to place stoploss amount {amount} with ' - f'stop {stop_price} and limit {rate} (total {rate*amount}).' - f'Message: {e}') - except (ccxt.NetworkError, ccxt.ExchangeError) as e: - raise TemporaryError( - f'Could not place stoploss limit order due to {e.__class__.__name__}. Message: {e}') - except ccxt.BaseError as e: - raise OperationalException(e) + order = self.create_order(pair, ordertype, 'sell', amount, rate, params) + logger.info('stoploss limit order added for %s. ' + 'stop price: %s. limit: %s' % (pair, stop_price, rate)) + return order @retrier def get_balance(self, currency: str) -> float: @@ -578,7 +543,7 @@ class Exchange(object): interval_in_sec = constants.TICKER_INTERVAL_MINUTES[ticker_interval] * 60 return not ((self._pairs_last_refresh_time.get((pair, ticker_interval), 0) - + interval_in_sec) >= arrow.utcnow().timestamp) + + interval_in_sec) >= arrow.utcnow().timestamp) @retrier_async async def _async_get_candle_history(self, pair: str, tick_interval: str, @@ -637,9 +602,6 @@ class Exchange(object): def get_order(self, order_id: str, pair: str) -> Dict: if self._conf['dry_run']: order = self._dry_run_open_orders[order_id] - order.update({ - 'id': order_id - }) return order try: return self._api.fetch_order(order_id, pair) @@ -659,18 +621,8 @@ class Exchange(object): Notes: 20180619: bittrex doesnt support limits -.- - 20180619: binance support limits but only on specific range """ try: - if self._api.name == 'Binance': - limit_range = [5, 10, 20, 50, 100, 500, 1000] - # get next-higher step in the limit_range list - limit = min(list(filter(lambda x: limit <= x, limit_range))) - # above script works like loop below (but with slightly better performance): - # for limitx in limit_range: - # if limit <= limitx: - # limit = limitx - # break return self._api.fetch_l2_order_book(pair, limit) except ccxt.NotSupported as e: @@ -702,16 +654,6 @@ class Exchange(object): except ccxt.BaseError as e: raise OperationalException(e) - def get_pair_detail_url(self, pair: str) -> str: - try: - url_base = self._api.urls.get('www') - base, quote = pair.split('/') - - return url_base + _EXCHANGE_URLS[self._api.id].format(base=base, quote=quote) - except KeyError: - logger.warning('Could not get exchange url for %s', self.name) - return "" - @retrier def get_markets(self) -> List[dict]: try: diff --git a/freqtrade/freqtradebot.py b/freqtrade/freqtradebot.py index 28a7c9146..dce3136df 100644 --- a/freqtrade/freqtradebot.py +++ b/freqtrade/freqtradebot.py @@ -7,7 +7,7 @@ import logging import time import traceback from datetime import datetime -from typing import Any, Callable, Dict, List, Optional +from typing import Any, Callable, Dict, List, Optional, Tuple import arrow from requests.exceptions import RequestException @@ -159,24 +159,20 @@ class FreqtradeBot(object): self.pairlists.refresh_pairlist() self.active_pair_whitelist = self.pairlists.whitelist - # Calculating Edge positiong + # Calculating Edge positioning if self.edge: self.edge.calculate() self.active_pair_whitelist = self.edge.adjust(self.active_pair_whitelist) # Query trades from persistence layer - trades = Trade.query.filter(Trade.is_open.is_(True)).all() + trades = Trade.get_open_trades() # Extend active-pair whitelist with pairs from open trades - # ensures that tickers are downloaded for open trades - self.active_pair_whitelist.extend([trade.pair for trade in trades - if trade.pair not in self.active_pair_whitelist]) + # It ensures that tickers are downloaded for open trades + self._extend_whitelist_with_trades(self.active_pair_whitelist, trades) - # Create pair-whitelist tuple with (pair, ticker_interval) - pair_whitelist_tuple = [(pair, self.config['ticker_interval']) - for pair in self.active_pair_whitelist] # Refreshing candles - self.dataprovider.refresh(pair_whitelist_tuple, + self.dataprovider.refresh(self._create_pair_whitelist(self.active_pair_whitelist), self.strategy.informative_pairs()) # First process current opened trades @@ -206,6 +202,18 @@ class FreqtradeBot(object): self.state = State.STOPPED return state_changed + def _extend_whitelist_with_trades(self, whitelist: List[str], trades: List[Any]): + """ + Extend whitelist with pairs from open trades + """ + whitelist.extend([trade.pair for trade in trades if trade.pair not in whitelist]) + + def _create_pair_whitelist(self, pairs: List[str]) -> List[Tuple[str, str]]: + """ + Create pair-whitelist tuple with (pair, ticker_interval) + """ + return [(pair, self.config['ticker_interval']) for pair in pairs] + def get_target_bid(self, pair: str, tick: Dict = None) -> float: """ Calculates bid target between current ask price and last price @@ -256,7 +264,7 @@ class FreqtradeBot(object): avaliable_amount = self.wallets.get_free(self.config['stake_currency']) if stake_amount == constants.UNLIMITED_STAKE_AMOUNT: - open_trades = len(Trade.query.filter(Trade.is_open.is_(True)).all()) + open_trades = len(Trade.get_open_trades()) if open_trades >= self.config['max_open_trades']: logger.warning('Can\'t open a new trade: max number of trades is reached') return None @@ -314,7 +322,7 @@ class FreqtradeBot(object): whitelist = copy.deepcopy(self.active_pair_whitelist) # Remove currently opened and latest pairs from whitelist - for trade in Trade.query.filter(Trade.is_open.is_(True)).all(): + for trade in Trade.get_open_trades(): if trade.pair in whitelist: whitelist.remove(trade.pair) logger.debug('Ignoring %s in pair whitelist', trade.pair) @@ -371,7 +379,6 @@ class FreqtradeBot(object): :return: None """ pair_s = pair.replace('_', '/') - pair_url = self.exchange.get_pair_detail_url(pair) stake_currency = self.config['stake_currency'] fiat_currency = self.config.get('fiat_display_currency', None) time_in_force = self.strategy.order_time_in_force['buy'] @@ -436,7 +443,6 @@ class FreqtradeBot(object): 'type': RPCMessageType.BUY_NOTIFICATION, 'exchange': self.exchange.name.capitalize(), 'pair': pair_s, - 'market_url': pair_url, 'limit': buy_limit_filled_price, 'stake_amount': stake_amount, 'stake_currency': stake_currency, @@ -844,7 +850,6 @@ class FreqtradeBot(object): profit_trade = trade.calc_profit(rate=limit) current_rate = self.exchange.get_ticker(trade.pair)['bid'] profit_percent = trade.calc_profit_percent(limit) - pair_url = self.exchange.get_pair_detail_url(trade.pair) gain = "profit" if profit_percent > 0 else "loss" msg = { @@ -852,7 +857,6 @@ class FreqtradeBot(object): 'exchange': trade.exchange.capitalize(), 'pair': trade.pair, 'gain': gain, - 'market_url': pair_url, 'limit': limit, 'amount': trade.amount, 'open_rate': trade.open_rate, diff --git a/freqtrade/misc.py b/freqtrade/misc.py index d03187d77..38f758669 100644 --- a/freqtrade/misc.py +++ b/freqtrade/misc.py @@ -113,3 +113,21 @@ def format_ms_time(date: int) -> str: : epoch-string in ms """ return datetime.fromtimestamp(date/1000.0).strftime('%Y-%m-%dT%H:%M:%S') + + +def deep_merge_dicts(source, destination): + """ + >>> a = { 'first' : { 'rows' : { 'pass' : 'dog', 'number' : '1' } } } + >>> b = { 'first' : { 'rows' : { 'fail' : 'cat', 'number' : '5' } } } + >>> merge(b, a) == { 'first' : { 'rows' : { 'pass' : 'dog', 'fail' : 'cat', 'number' : '5' } } } + True + """ + for key, value in source.items(): + if isinstance(value, dict): + # get node or create one + node = destination.setdefault(key, {}) + deep_merge_dicts(value, node) + else: + destination[key] = value + + return destination diff --git a/freqtrade/persistence.py b/freqtrade/persistence.py index f9b34fc64..f603b139f 100644 --- a/freqtrade/persistence.py +++ b/freqtrade/persistence.py @@ -5,7 +5,7 @@ This module contains the class to persist trades into SQLite import logging from datetime import datetime from decimal import Decimal -from typing import Any, Dict, Optional +from typing import Any, Dict, List, Optional import arrow from sqlalchemy import (Boolean, Column, DateTime, Float, Integer, String, @@ -371,3 +371,10 @@ class Trade(_DECL_BASE): .filter(Trade.is_open.is_(True))\ .scalar() return total_open_stake_amount or 0 + + @staticmethod + def get_open_trades() -> List[Any]: + """ + Query trades from persistence layer + """ + return Trade.query.filter(Trade.is_open.is_(True)).all() diff --git a/freqtrade/rpc/rpc.py b/freqtrade/rpc/rpc.py index e83d9d41b..af64c3d67 100644 --- a/freqtrade/rpc/rpc.py +++ b/freqtrade/rpc/rpc.py @@ -83,7 +83,7 @@ class RPC(object): a remotely exposed function """ # Fetch open trade - trades = Trade.query.filter(Trade.is_open.is_(True)).all() + trades = Trade.get_open_trades() if not trades: raise RPCException('no active trade') else: @@ -103,7 +103,6 @@ class RPC(object): results.append(dict( trade_id=trade.id, pair=trade.pair, - market_url=self._freqtrade.exchange.get_pair_detail_url(trade.pair), date=arrow.get(trade.open_date), open_rate=trade.open_rate, close_rate=trade.close_rate, @@ -118,7 +117,7 @@ class RPC(object): return results def _rpc_status_table(self) -> DataFrame: - trades = Trade.query.filter(Trade.is_open.is_(True)).all() + trades = Trade.get_open_trades() if not trades: raise RPCException('no active order') else: @@ -366,7 +365,7 @@ class RPC(object): if trade_id == 'all': # Execute sell for all open orders - for trade in Trade.query.filter(Trade.is_open.is_(True)).all(): + for trade in Trade.get_open_trades(): _exec_forcesell(trade) Trade.session.flush() return @@ -442,7 +441,7 @@ class RPC(object): if self._freqtrade.state != State.RUNNING: raise RPCException('trader is not running') - return Trade.query.filter(Trade.is_open.is_(True)).all() + return Trade.get_open_trades() def _rpc_whitelist(self) -> Dict: """ Returns the currently active whitelist""" diff --git a/freqtrade/rpc/telegram.py b/freqtrade/rpc/telegram.py index 3ce7c9167..9caa7288f 100644 --- a/freqtrade/rpc/telegram.py +++ b/freqtrade/rpc/telegram.py @@ -125,7 +125,7 @@ class Telegram(RPC): else: msg['stake_amount_fiat'] = 0 - message = ("*{exchange}:* Buying [{pair}]({market_url})\n" + message = ("*{exchange}:* Buying {pair}\n" "with limit `{limit:.8f}\n" "({stake_amount:.6f} {stake_currency}").format(**msg) @@ -137,7 +137,7 @@ class Telegram(RPC): msg['amount'] = round(msg['amount'], 8) msg['profit_percent'] = round(msg['profit_percent'] * 100, 2) - message = ("*{exchange}:* Selling [{pair}]({market_url})\n" + message = ("*{exchange}:* Selling {pair}\n" "*Limit:* `{limit:.8f}`\n" "*Amount:* `{amount:.8f}`\n" "*Open Rate:* `{open_rate:.8f}`\n" @@ -193,7 +193,7 @@ class Telegram(RPC): messages = [ "*Trade ID:* `{trade_id}`\n" - "*Current Pair:* [{pair}]({market_url})\n" + "*Current Pair:* {pair}\n" "*Open Since:* `{date}`\n" "*Amount:* `{amount}`\n" "*Open Rate:* `{open_rate:.8f}`\n" diff --git a/freqtrade/tests/exchange/test_exchange.py b/freqtrade/tests/exchange/test_exchange.py index 4fadfd4b7..3b8d3ad6f 100644 --- a/freqtrade/tests/exchange/test_exchange.py +++ b/freqtrade/tests/exchange/test_exchange.py @@ -12,12 +12,16 @@ import pytest from pandas import DataFrame from freqtrade import DependencyException, OperationalException, TemporaryError -from freqtrade.exchange import Exchange, Kraken +from freqtrade.exchange import Exchange, Kraken, Binance from freqtrade.exchange.exchange import API_RETRY_COUNT from freqtrade.tests.conftest import get_patched_exchange, log_has, log_has_re from freqtrade.resolvers.exchange_resolver import ExchangeResolver +# Make sure to always keep one exchange here which is NOT subclassed!! +EXCHANGES = ['bittrex', 'binance', 'kraken', ] + + # Source: https://stackoverflow.com/questions/29881236/how-to-mock-asyncio-coroutines def get_mock_coro(return_value): async def mock_coro(*args, **kwargs): @@ -26,16 +30,17 @@ def get_mock_coro(return_value): return Mock(wraps=mock_coro) -def ccxt_exceptionhandlers(mocker, default_conf, api_mock, fun, mock_ccxt_fun, **kwargs): +def ccxt_exceptionhandlers(mocker, default_conf, api_mock, exchange_name, + fun, mock_ccxt_fun, **kwargs): with pytest.raises(TemporaryError): api_mock.__dict__[mock_ccxt_fun] = MagicMock(side_effect=ccxt.NetworkError) - exchange = get_patched_exchange(mocker, default_conf, api_mock) + exchange = get_patched_exchange(mocker, default_conf, api_mock, id=exchange_name) getattr(exchange, fun)(**kwargs) assert api_mock.__dict__[mock_ccxt_fun].call_count == API_RETRY_COUNT + 1 with pytest.raises(OperationalException): api_mock.__dict__[mock_ccxt_fun] = MagicMock(side_effect=ccxt.BaseError) - exchange = get_patched_exchange(mocker, default_conf, api_mock) + exchange = get_patched_exchange(mocker, default_conf, api_mock, id=exchange_name) getattr(exchange, fun)(**kwargs) assert api_mock.__dict__[mock_ccxt_fun].call_count == 1 @@ -113,7 +118,7 @@ def test_exchange_resolver(default_conf, mocker, caplog): mocker.patch('freqtrade.exchange.Exchange._load_async_markets', MagicMock()) mocker.patch('freqtrade.exchange.Exchange.validate_pairs', MagicMock()) mocker.patch('freqtrade.exchange.Exchange.validate_timeframes', MagicMock()) - exchange = ExchangeResolver('Binance', default_conf).exchange + exchange = ExchangeResolver('Bittrex', default_conf).exchange assert isinstance(exchange, Exchange) assert log_has_re(r"No .* specific subclass found. Using the generic class instead.", caplog.record_tuples) @@ -122,6 +127,15 @@ def test_exchange_resolver(default_conf, mocker, caplog): exchange = ExchangeResolver('Kraken', default_conf).exchange assert isinstance(exchange, Exchange) assert isinstance(exchange, Kraken) + assert not isinstance(exchange, Binance) + assert not log_has_re(r"No .* specific subclass found. Using the generic class instead.", + caplog.record_tuples) + + exchange = ExchangeResolver('Binance', default_conf).exchange + assert isinstance(exchange, Exchange) + assert isinstance(exchange, Binance) + assert not isinstance(exchange, Kraken) + assert not log_has_re(r"No .* specific subclass found. Using the generic class instead.", caplog.record_tuples) @@ -443,6 +457,58 @@ def test_exchange_has(default_conf, mocker): assert not exchange.exchange_has("deadbeef") +@pytest.mark.parametrize("side", [ + ("buy"), + ("sell") +]) +@pytest.mark.parametrize("exchange_name", EXCHANGES) +def test_dry_run_order(default_conf, mocker, side, exchange_name): + default_conf['dry_run'] = True + exchange = get_patched_exchange(mocker, default_conf, id=exchange_name) + + order = exchange.dry_run_order( + pair='ETH/BTC', ordertype='limit', side=side, amount=1, rate=200) + assert 'id' in order + assert f'dry_run_{side}_' in order["id"] + + +@pytest.mark.parametrize("side", [ + ("buy"), + ("sell") +]) +@pytest.mark.parametrize("ordertype,rate", [ + ("market", None), + ("limit", 200), + ("stop_loss_limit", 200) +]) +@pytest.mark.parametrize("exchange_name", EXCHANGES) +def test_create_order(default_conf, mocker, side, ordertype, rate, exchange_name): + api_mock = MagicMock() + order_id = 'test_prod_{}_{}'.format(side, randint(0, 10 ** 6)) + api_mock.create_order = MagicMock(return_value={ + 'id': order_id, + 'info': { + 'foo': 'bar' + } + }) + default_conf['dry_run'] = False + mocker.patch('freqtrade.exchange.Exchange.symbol_amount_prec', lambda s, x, y: y) + mocker.patch('freqtrade.exchange.Exchange.symbol_price_prec', lambda s, x, y: y) + exchange = get_patched_exchange(mocker, default_conf, api_mock, id=exchange_name) + + order = exchange.create_order( + pair='ETH/BTC', ordertype=ordertype, side=side, amount=1, rate=200) + + assert 'id' in order + assert 'info' in order + assert order['id'] == order_id + assert api_mock.create_order.call_args[0][0] == 'ETH/BTC' + assert api_mock.create_order.call_args[0][1] == ordertype + assert api_mock.create_order.call_args[0][2] == side + assert api_mock.create_order.call_args[0][3] == 1 + assert api_mock.create_order.call_args[0][4] is rate + + def test_buy_dry_run(default_conf, mocker): default_conf['dry_run'] = True exchange = get_patched_exchange(mocker, default_conf) @@ -453,7 +519,8 @@ def test_buy_dry_run(default_conf, mocker): assert 'dry_run_buy_' in order['id'] -def test_buy_prod(default_conf, mocker): +@pytest.mark.parametrize("exchange_name", EXCHANGES) +def test_buy_prod(default_conf, mocker, exchange_name): api_mock = MagicMock() order_id = 'test_prod_buy_{}'.format(randint(0, 10 ** 6)) order_type = 'market' @@ -467,7 +534,7 @@ def test_buy_prod(default_conf, mocker): default_conf['dry_run'] = False mocker.patch('freqtrade.exchange.Exchange.symbol_amount_prec', lambda s, x, y: y) mocker.patch('freqtrade.exchange.Exchange.symbol_price_prec', lambda s, x, y: y) - exchange = get_patched_exchange(mocker, default_conf, api_mock) + exchange = get_patched_exchange(mocker, default_conf, api_mock, id=exchange_name) order = exchange.buy(pair='ETH/BTC', ordertype=order_type, amount=1, rate=200, time_in_force=time_in_force) @@ -498,25 +565,25 @@ def test_buy_prod(default_conf, mocker): # test exception handling with pytest.raises(DependencyException): api_mock.create_order = MagicMock(side_effect=ccxt.InsufficientFunds) - exchange = get_patched_exchange(mocker, default_conf, api_mock) + exchange = get_patched_exchange(mocker, default_conf, api_mock, id=exchange_name) exchange.buy(pair='ETH/BTC', ordertype=order_type, amount=1, rate=200, time_in_force=time_in_force) with pytest.raises(DependencyException): api_mock.create_order = MagicMock(side_effect=ccxt.InvalidOrder) - exchange = get_patched_exchange(mocker, default_conf, api_mock) + exchange = get_patched_exchange(mocker, default_conf, api_mock, id=exchange_name) exchange.buy(pair='ETH/BTC', ordertype=order_type, amount=1, rate=200, time_in_force=time_in_force) with pytest.raises(TemporaryError): api_mock.create_order = MagicMock(side_effect=ccxt.NetworkError) - exchange = get_patched_exchange(mocker, default_conf, api_mock) + exchange = get_patched_exchange(mocker, default_conf, api_mock, id=exchange_name) exchange.buy(pair='ETH/BTC', ordertype=order_type, amount=1, rate=200, time_in_force=time_in_force) with pytest.raises(OperationalException): api_mock.create_order = MagicMock(side_effect=ccxt.BaseError) - exchange = get_patched_exchange(mocker, default_conf, api_mock) + exchange = get_patched_exchange(mocker, default_conf, api_mock, id=exchange_name) exchange.buy(pair='ETH/BTC', ordertype=order_type, amount=1, rate=200, time_in_force=time_in_force) @@ -621,7 +688,8 @@ def test_sell_dry_run(default_conf, mocker): assert 'dry_run_sell_' in order['id'] -def test_sell_prod(default_conf, mocker): +@pytest.mark.parametrize("exchange_name", EXCHANGES) +def test_sell_prod(default_conf, mocker, exchange_name): api_mock = MagicMock() order_id = 'test_prod_sell_{}'.format(randint(0, 10 ** 6)) order_type = 'market' @@ -635,7 +703,7 @@ def test_sell_prod(default_conf, mocker): mocker.patch('freqtrade.exchange.Exchange.symbol_amount_prec', lambda s, x, y: y) mocker.patch('freqtrade.exchange.Exchange.symbol_price_prec', lambda s, x, y: y) - exchange = get_patched_exchange(mocker, default_conf, api_mock) + exchange = get_patched_exchange(mocker, default_conf, api_mock, id=exchange_name) order = exchange.sell(pair='ETH/BTC', ordertype=order_type, amount=1, rate=200) @@ -660,22 +728,22 @@ def test_sell_prod(default_conf, mocker): # test exception handling with pytest.raises(DependencyException): api_mock.create_order = MagicMock(side_effect=ccxt.InsufficientFunds) - exchange = get_patched_exchange(mocker, default_conf, api_mock) + exchange = get_patched_exchange(mocker, default_conf, api_mock, id=exchange_name) exchange.sell(pair='ETH/BTC', ordertype=order_type, amount=1, rate=200) with pytest.raises(DependencyException): api_mock.create_order = MagicMock(side_effect=ccxt.InvalidOrder) - exchange = get_patched_exchange(mocker, default_conf, api_mock) + exchange = get_patched_exchange(mocker, default_conf, api_mock, id=exchange_name) exchange.sell(pair='ETH/BTC', ordertype=order_type, amount=1, rate=200) with pytest.raises(TemporaryError): api_mock.create_order = MagicMock(side_effect=ccxt.NetworkError) - exchange = get_patched_exchange(mocker, default_conf, api_mock) + exchange = get_patched_exchange(mocker, default_conf, api_mock, id=exchange_name) exchange.sell(pair='ETH/BTC', ordertype=order_type, amount=1, rate=200) with pytest.raises(OperationalException): api_mock.create_order = MagicMock(side_effect=ccxt.BaseError) - exchange = get_patched_exchange(mocker, default_conf, api_mock) + exchange = get_patched_exchange(mocker, default_conf, api_mock, id=exchange_name) exchange.sell(pair='ETH/BTC', ordertype=order_type, amount=1, rate=200) @@ -686,23 +754,24 @@ def test_get_balance_dry_run(default_conf, mocker): assert exchange.get_balance(currency='BTC') == 999.9 -def test_get_balance_prod(default_conf, mocker): +@pytest.mark.parametrize("exchange_name", EXCHANGES) +def test_get_balance_prod(default_conf, mocker, exchange_name): api_mock = MagicMock() api_mock.fetch_balance = MagicMock(return_value={'BTC': {'free': 123.4}}) default_conf['dry_run'] = False - exchange = get_patched_exchange(mocker, default_conf, api_mock) + exchange = get_patched_exchange(mocker, default_conf, api_mock, id=exchange_name) assert exchange.get_balance(currency='BTC') == 123.4 with pytest.raises(OperationalException): api_mock.fetch_balance = MagicMock(side_effect=ccxt.BaseError) - exchange = get_patched_exchange(mocker, default_conf, api_mock) + exchange = get_patched_exchange(mocker, default_conf, api_mock, id=exchange_name) exchange.get_balance(currency='BTC') with pytest.raises(TemporaryError, match=r'.*balance due to malformed exchange response:.*'): - exchange = get_patched_exchange(mocker, default_conf, api_mock) + exchange = get_patched_exchange(mocker, default_conf, api_mock, id=exchange_name) mocker.patch('freqtrade.exchange.Exchange.get_balances', MagicMock(return_value={})) exchange.get_balance(currency='BTC') @@ -713,7 +782,8 @@ def test_get_balances_dry_run(default_conf, mocker): assert exchange.get_balances() == {} -def test_get_balances_prod(default_conf, mocker): +@pytest.mark.parametrize("exchange_name", EXCHANGES) +def test_get_balances_prod(default_conf, mocker, exchange_name): balance_item = { 'free': 10.0, 'total': 10.0, @@ -727,17 +797,18 @@ def test_get_balances_prod(default_conf, mocker): '3ST': balance_item }) default_conf['dry_run'] = False - exchange = get_patched_exchange(mocker, default_conf, api_mock) + exchange = get_patched_exchange(mocker, default_conf, api_mock, id=exchange_name) assert len(exchange.get_balances()) == 3 assert exchange.get_balances()['1ST']['free'] == 10.0 assert exchange.get_balances()['1ST']['total'] == 10.0 assert exchange.get_balances()['1ST']['used'] == 0.0 - ccxt_exceptionhandlers(mocker, default_conf, api_mock, + ccxt_exceptionhandlers(mocker, default_conf, api_mock, exchange_name, "get_balances", "fetch_balance") -def test_get_tickers(default_conf, mocker): +@pytest.mark.parametrize("exchange_name", EXCHANGES) +def test_get_tickers(default_conf, mocker, exchange_name): api_mock = MagicMock() tick = {'ETH/BTC': { 'symbol': 'ETH/BTC', @@ -752,7 +823,7 @@ def test_get_tickers(default_conf, mocker): } } api_mock.fetch_tickers = MagicMock(return_value=tick) - exchange = get_patched_exchange(mocker, default_conf, api_mock) + exchange = get_patched_exchange(mocker, default_conf, api_mock, id=exchange_name) # retrieve original ticker tickers = exchange.get_tickers() @@ -763,20 +834,21 @@ def test_get_tickers(default_conf, mocker): assert tickers['BCH/BTC']['bid'] == 0.6 assert tickers['BCH/BTC']['ask'] == 0.5 - ccxt_exceptionhandlers(mocker, default_conf, api_mock, + ccxt_exceptionhandlers(mocker, default_conf, api_mock, exchange_name, "get_tickers", "fetch_tickers") with pytest.raises(OperationalException): api_mock.fetch_tickers = MagicMock(side_effect=ccxt.NotSupported) - exchange = get_patched_exchange(mocker, default_conf, api_mock) + exchange = get_patched_exchange(mocker, default_conf, api_mock, id=exchange_name) exchange.get_tickers() api_mock.fetch_tickers = MagicMock(return_value={}) - exchange = get_patched_exchange(mocker, default_conf, api_mock) + exchange = get_patched_exchange(mocker, default_conf, api_mock, id=exchange_name) exchange.get_tickers() -def test_get_ticker(default_conf, mocker): +@pytest.mark.parametrize("exchange_name", EXCHANGES) +def test_get_ticker(default_conf, mocker, exchange_name): api_mock = MagicMock() tick = { 'symbol': 'ETH/BTC', @@ -786,7 +858,7 @@ def test_get_ticker(default_conf, mocker): } api_mock.fetch_ticker = MagicMock(return_value=tick) api_mock.markets = {'ETH/BTC': {}} - exchange = get_patched_exchange(mocker, default_conf, api_mock) + exchange = get_patched_exchange(mocker, default_conf, api_mock, id=exchange_name) # retrieve original ticker ticker = exchange.get_ticker(pair='ETH/BTC') @@ -801,7 +873,7 @@ def test_get_ticker(default_conf, mocker): 'last': 42, } api_mock.fetch_ticker = MagicMock(return_value=tick) - exchange = get_patched_exchange(mocker, default_conf, api_mock) + exchange = get_patched_exchange(mocker, default_conf, api_mock, id=exchange_name) # if not caching the result we should get the same ticker # if not fetching a new result we should get the cached ticker @@ -820,20 +892,21 @@ def test_get_ticker(default_conf, mocker): exchange.get_ticker(pair='ETH/BTC', refresh=False) assert api_mock.fetch_ticker.call_count == 0 - ccxt_exceptionhandlers(mocker, default_conf, api_mock, + ccxt_exceptionhandlers(mocker, default_conf, api_mock, exchange_name, "get_ticker", "fetch_ticker", pair='ETH/BTC', refresh=True) api_mock.fetch_ticker = MagicMock(return_value={}) - exchange = get_patched_exchange(mocker, default_conf, api_mock) + exchange = get_patched_exchange(mocker, default_conf, api_mock, id=exchange_name) exchange.get_ticker(pair='ETH/BTC', refresh=True) with pytest.raises(DependencyException, match=r'Pair XRP/ETH not available'): exchange.get_ticker(pair='XRP/ETH', refresh=True) -def test_get_history(default_conf, mocker, caplog): - exchange = get_patched_exchange(mocker, default_conf) +@pytest.mark.parametrize("exchange_name", EXCHANGES) +def test_get_history(default_conf, mocker, caplog, exchange_name): + exchange = get_patched_exchange(mocker, default_conf, id=exchange_name) tick = [ [ arrow.utcnow().timestamp * 1000, # unix timestamp ms @@ -912,7 +985,8 @@ def test_refresh_latest_ohlcv(mocker, default_conf, caplog) -> None: @pytest.mark.asyncio -async def test__async_get_candle_history(default_conf, mocker, caplog): +@pytest.mark.parametrize("exchange_name", EXCHANGES) +async def test__async_get_candle_history(default_conf, mocker, caplog, exchange_name): tick = [ [ arrow.utcnow().timestamp * 1000, # unix timestamp ms @@ -925,11 +999,10 @@ async def test__async_get_candle_history(default_conf, mocker, caplog): ] caplog.set_level(logging.DEBUG) - exchange = get_patched_exchange(mocker, default_conf) + exchange = get_patched_exchange(mocker, default_conf, id=exchange_name) # Monkey-patch async function exchange._api_async.fetch_ohlcv = get_mock_coro(tick) - exchange = Exchange(default_conf) pair = 'ETH/BTC' res = await exchange._async_get_candle_history(pair, "5m") assert type(res) is tuple @@ -948,7 +1021,7 @@ async def test__async_get_candle_history(default_conf, mocker, caplog): api_mock = MagicMock() with pytest.raises(OperationalException, match=r'Could not fetch ticker data*'): api_mock.fetch_ohlcv = MagicMock(side_effect=ccxt.BaseError) - exchange = get_patched_exchange(mocker, default_conf, api_mock) + exchange = get_patched_exchange(mocker, default_conf, api_mock, id=exchange_name) await exchange._async_get_candle_history(pair, "5m", (arrow.utcnow().timestamp - 2000) * 1000) @@ -1001,12 +1074,13 @@ def test_refresh_latest_ohlcv_inv_result(default_conf, mocker, caplog): assert log_has("Async code raised an exception: TypeError", caplog.record_tuples) -def test_get_order_book(default_conf, mocker, order_book_l2): - default_conf['exchange']['name'] = 'binance' +@pytest.mark.parametrize("exchange_name", EXCHANGES) +def test_get_order_book(default_conf, mocker, order_book_l2, exchange_name): + default_conf['exchange']['name'] = exchange_name api_mock = MagicMock() api_mock.fetch_l2_order_book = order_book_l2 - exchange = get_patched_exchange(mocker, default_conf, api_mock) + exchange = get_patched_exchange(mocker, default_conf, api_mock, id=exchange_name) order_book = exchange.get_order_book(pair='ETH/BTC', limit=10) assert 'bids' in order_book assert 'asks' in order_book @@ -1014,19 +1088,20 @@ def test_get_order_book(default_conf, mocker, order_book_l2): assert len(order_book['asks']) == 10 -def test_get_order_book_exception(default_conf, mocker): +@pytest.mark.parametrize("exchange_name", EXCHANGES) +def test_get_order_book_exception(default_conf, mocker, exchange_name): api_mock = MagicMock() with pytest.raises(OperationalException): api_mock.fetch_l2_order_book = MagicMock(side_effect=ccxt.NotSupported) - exchange = get_patched_exchange(mocker, default_conf, api_mock) + exchange = get_patched_exchange(mocker, default_conf, api_mock, id=exchange_name) exchange.get_order_book(pair='ETH/BTC', limit=50) with pytest.raises(TemporaryError): api_mock.fetch_l2_order_book = MagicMock(side_effect=ccxt.NetworkError) - exchange = get_patched_exchange(mocker, default_conf, api_mock) + exchange = get_patched_exchange(mocker, default_conf, api_mock, id=exchange_name) exchange.get_order_book(pair='ETH/BTC', limit=50) with pytest.raises(OperationalException): api_mock.fetch_l2_order_book = MagicMock(side_effect=ccxt.BaseError) - exchange = get_patched_exchange(mocker, default_conf, api_mock) + exchange = get_patched_exchange(mocker, default_conf, api_mock, id=exchange_name) exchange.get_order_book(pair='ETH/BTC', limit=50) @@ -1039,8 +1114,9 @@ def make_fetch_ohlcv_mock(data): return fetch_ohlcv_mock +@pytest.mark.parametrize("exchange_name", EXCHANGES) @pytest.mark.asyncio -async def test___async_get_candle_history_sort(default_conf, mocker): +async def test___async_get_candle_history_sort(default_conf, mocker, exchange_name): def sort_data(data, key): return sorted(data, key=key) @@ -1058,7 +1134,7 @@ async def test___async_get_candle_history_sort(default_conf, mocker): [1527830700000, 0.07652, 0.07652, 0.07651, 0.07652, 10.04822687], [1527830400000, 0.07649, 0.07651, 0.07649, 0.07651, 2.5734867] ] - exchange = get_patched_exchange(mocker, default_conf) + exchange = get_patched_exchange(mocker, default_conf, id=exchange_name) exchange._api_async.fetch_ohlcv = get_mock_coro(tick) sort_mock = mocker.patch('freqtrade.exchange.exchange.sorted', MagicMock(side_effect=sort_data)) # Test the ticker history sort @@ -1120,36 +1196,39 @@ async def test___async_get_candle_history_sort(default_conf, mocker): assert ticks[9][5] == 2.31452783 -def test_cancel_order_dry_run(default_conf, mocker): +@pytest.mark.parametrize("exchange_name", EXCHANGES) +def test_cancel_order_dry_run(default_conf, mocker, exchange_name): default_conf['dry_run'] = True - exchange = get_patched_exchange(mocker, default_conf) + exchange = get_patched_exchange(mocker, default_conf, id=exchange_name) assert exchange.cancel_order(order_id='123', pair='TKN/BTC') is None # Ensure that if not dry_run, we should call API -def test_cancel_order(default_conf, mocker): +@pytest.mark.parametrize("exchange_name", EXCHANGES) +def test_cancel_order(default_conf, mocker, exchange_name): default_conf['dry_run'] = False api_mock = MagicMock() api_mock.cancel_order = MagicMock(return_value=123) - exchange = get_patched_exchange(mocker, default_conf, api_mock) + exchange = get_patched_exchange(mocker, default_conf, api_mock, id=exchange_name) assert exchange.cancel_order(order_id='_', pair='TKN/BTC') == 123 with pytest.raises(DependencyException): api_mock.cancel_order = MagicMock(side_effect=ccxt.InvalidOrder) - exchange = get_patched_exchange(mocker, default_conf, api_mock) + exchange = get_patched_exchange(mocker, default_conf, api_mock, id=exchange_name) exchange.cancel_order(order_id='_', pair='TKN/BTC') assert api_mock.cancel_order.call_count == API_RETRY_COUNT + 1 - ccxt_exceptionhandlers(mocker, default_conf, api_mock, + ccxt_exceptionhandlers(mocker, default_conf, api_mock, exchange_name, "cancel_order", "cancel_order", order_id='_', pair='TKN/BTC') -def test_get_order(default_conf, mocker): +@pytest.mark.parametrize("exchange_name", EXCHANGES) +def test_get_order(default_conf, mocker, exchange_name): default_conf['dry_run'] = True order = MagicMock() order.myid = 123 - exchange = get_patched_exchange(mocker, default_conf) + exchange = get_patched_exchange(mocker, default_conf, id=exchange_name) exchange._dry_run_open_orders['X'] = order print(exchange.get_order('X', 'TKN/BTC')) assert exchange.get_order('X', 'TKN/BTC').myid == 123 @@ -1157,67 +1236,32 @@ def test_get_order(default_conf, mocker): default_conf['dry_run'] = False api_mock = MagicMock() api_mock.fetch_order = MagicMock(return_value=456) - exchange = get_patched_exchange(mocker, default_conf, api_mock) + exchange = get_patched_exchange(mocker, default_conf, api_mock, id=exchange_name) assert exchange.get_order('X', 'TKN/BTC') == 456 with pytest.raises(DependencyException): api_mock.fetch_order = MagicMock(side_effect=ccxt.InvalidOrder) - exchange = get_patched_exchange(mocker, default_conf, api_mock) + exchange = get_patched_exchange(mocker, default_conf, api_mock, id=exchange_name) exchange.get_order(order_id='_', pair='TKN/BTC') assert api_mock.fetch_order.call_count == API_RETRY_COUNT + 1 - ccxt_exceptionhandlers(mocker, default_conf, api_mock, + ccxt_exceptionhandlers(mocker, default_conf, api_mock, exchange_name, 'get_order', 'fetch_order', order_id='_', pair='TKN/BTC') -def test_name(default_conf, mocker): +@pytest.mark.parametrize("exchange_name", EXCHANGES) +def test_name(default_conf, mocker, exchange_name): mocker.patch('freqtrade.exchange.Exchange._load_markets', MagicMock(return_value={})) - default_conf['exchange']['name'] = 'binance' + default_conf['exchange']['name'] = exchange_name exchange = Exchange(default_conf) - assert exchange.name == 'Binance' + assert exchange.name == exchange_name.title() + assert exchange.id == exchange_name -def test_id(default_conf, mocker): - mocker.patch('freqtrade.exchange.Exchange._load_markets', MagicMock(return_value={})) - default_conf['exchange']['name'] = 'binance' - exchange = Exchange(default_conf) - assert exchange.id == 'binance' - - -def test_get_pair_detail_url(default_conf, mocker, caplog): - mocker.patch('freqtrade.exchange.Exchange._load_markets', MagicMock(return_value={})) - default_conf['exchange']['name'] = 'binance' - exchange = Exchange(default_conf) - - url = exchange.get_pair_detail_url('TKN/ETH') - assert 'TKN' in url - assert 'ETH' in url - - url = exchange.get_pair_detail_url('LOOONG/BTC') - assert 'LOOONG' in url - assert 'BTC' in url - - default_conf['exchange']['name'] = 'bittrex' - exchange = Exchange(default_conf) - - url = exchange.get_pair_detail_url('TKN/ETH') - assert 'TKN' in url - assert 'ETH' in url - - url = exchange.get_pair_detail_url('LOOONG/BTC') - assert 'LOOONG' in url - assert 'BTC' in url - - default_conf['exchange']['name'] = 'poloniex' - exchange = Exchange(default_conf) - url = exchange.get_pair_detail_url('LOOONG/BTC') - assert '' == url - assert log_has('Could not get exchange url for Poloniex', caplog.record_tuples) - - -def test_get_trades_for_order(default_conf, mocker): +@pytest.mark.parametrize("exchange_name", EXCHANGES) +def test_get_trades_for_order(default_conf, mocker, exchange_name): order_id = 'ABCD-ABCD' since = datetime(2018, 5, 5) default_conf["dry_run"] = False @@ -1244,13 +1288,13 @@ def test_get_trades_for_order(default_conf, mocker): 'amount': 0.2340606, 'fee': {'cost': 0.06179, 'currency': 'BTC'} }]) - exchange = get_patched_exchange(mocker, default_conf, api_mock) + exchange = get_patched_exchange(mocker, default_conf, api_mock, id=exchange_name) orders = exchange.get_trades_for_order(order_id, 'LTC/BTC', since) assert len(orders) == 1 assert orders[0]['price'] == 165 - ccxt_exceptionhandlers(mocker, default_conf, api_mock, + ccxt_exceptionhandlers(mocker, default_conf, api_mock, exchange_name, 'get_trades_for_order', 'fetch_my_trades', order_id=order_id, pair='LTC/BTC', since=since) @@ -1258,10 +1302,11 @@ def test_get_trades_for_order(default_conf, mocker): assert exchange.get_trades_for_order(order_id, 'LTC/BTC', since) == [] -def test_get_markets(default_conf, mocker, markets): +@pytest.mark.parametrize("exchange_name", EXCHANGES) +def test_get_markets(default_conf, mocker, markets, exchange_name): api_mock = MagicMock() api_mock.fetch_markets = markets - exchange = get_patched_exchange(mocker, default_conf, api_mock) + exchange = get_patched_exchange(mocker, default_conf, api_mock, id=exchange_name) ret = exchange.get_markets() assert isinstance(ret, list) assert len(ret) == 9 @@ -1269,11 +1314,12 @@ def test_get_markets(default_conf, mocker, markets): assert ret[0]["id"] == "ethbtc" assert ret[0]["symbol"] == "ETH/BTC" - ccxt_exceptionhandlers(mocker, default_conf, api_mock, + ccxt_exceptionhandlers(mocker, default_conf, api_mock, exchange_name, 'get_markets', 'fetch_markets') -def test_get_fee(default_conf, mocker): +@pytest.mark.parametrize("exchange_name", EXCHANGES) +def test_get_fee(default_conf, mocker, exchange_name): api_mock = MagicMock() api_mock.calculate_fee = MagicMock(return_value={ 'type': 'taker', @@ -1281,11 +1327,11 @@ def test_get_fee(default_conf, mocker): 'rate': 0.025, 'cost': 0.05 }) - exchange = get_patched_exchange(mocker, default_conf, api_mock) + exchange = get_patched_exchange(mocker, default_conf, api_mock, id=exchange_name) assert exchange.get_fee() == 0.025 - ccxt_exceptionhandlers(mocker, default_conf, api_mock, + ccxt_exceptionhandlers(mocker, default_conf, api_mock, exchange_name, 'get_fee', 'calculate_fee') diff --git a/freqtrade/tests/rpc/test_rpc.py b/freqtrade/tests/rpc/test_rpc.py index bb685cad5..2de2668e8 100644 --- a/freqtrade/tests/rpc/test_rpc.py +++ b/freqtrade/tests/rpc/test_rpc.py @@ -51,7 +51,6 @@ def test_rpc_trade_status(default_conf, ticker, fee, markets, mocker) -> None: assert { 'trade_id': 1, 'pair': 'ETH/BTC', - 'market_url': 'https://bittrex.com/Market/Index?MarketName=BTC-ETH', 'date': ANY, 'open_rate': 1.099e-05, 'close_rate': None, @@ -72,7 +71,6 @@ def test_rpc_trade_status(default_conf, ticker, fee, markets, mocker) -> None: assert { 'trade_id': 1, 'pair': 'ETH/BTC', - 'market_url': 'https://bittrex.com/Market/Index?MarketName=BTC-ETH', 'date': ANY, 'open_rate': 1.099e-05, 'close_rate': None, diff --git a/freqtrade/tests/rpc/test_rpc_telegram.py b/freqtrade/tests/rpc/test_rpc_telegram.py index 686a92469..9964973e1 100644 --- a/freqtrade/tests/rpc/test_rpc_telegram.py +++ b/freqtrade/tests/rpc/test_rpc_telegram.py @@ -5,7 +5,7 @@ import re from datetime import datetime from random import randint -from unittest.mock import MagicMock, ANY +from unittest.mock import MagicMock import arrow import pytest @@ -183,7 +183,6 @@ def test_status(default_conf, update, mocker, fee, ticker, markets) -> None: mocker.patch.multiple( 'freqtrade.exchange.Exchange', get_ticker=ticker, - get_pair_detail_url=MagicMock(), get_fee=fee, get_markets=markets ) @@ -195,7 +194,6 @@ def test_status(default_conf, update, mocker, fee, ticker, markets) -> None: _rpc_trade_status=MagicMock(return_value=[{ 'trade_id': 1, 'pair': 'ETH/BTC', - 'market_url': 'https://bittrex.com/Market/Index?MarketName=BTC-ETH', 'date': arrow.utcnow(), 'open_rate': 1.099e-05, 'close_rate': None, @@ -270,7 +268,7 @@ def test_status_handle(default_conf, update, ticker, fee, markets, mocker) -> No telegram._status(bot=MagicMock(), update=update) assert msg_mock.call_count == 1 - assert '[ETH/BTC]' in msg_mock.call_args_list[0][0][0] + assert 'ETH/BTC' in msg_mock.call_args_list[0][0][0] def test_status_table_handle(default_conf, update, ticker, fee, markets, mocker) -> None: @@ -721,7 +719,6 @@ def test_forcesell_handle(default_conf, update, ticker, fee, 'exchange': 'Bittrex', 'pair': 'ETH/BTC', 'gain': 'profit', - 'market_url': 'https://bittrex.com/Market/Index?MarketName=BTC-ETH', 'limit': 1.172e-05, 'amount': 90.99181073703367, 'open_rate': 1.099e-05, @@ -776,7 +773,6 @@ def test_forcesell_down_handle(default_conf, update, ticker, fee, 'exchange': 'Bittrex', 'pair': 'ETH/BTC', 'gain': 'loss', - 'market_url': 'https://bittrex.com/Market/Index?MarketName=BTC-ETH', 'limit': 1.044e-05, 'amount': 90.99181073703367, 'open_rate': 1.099e-05, @@ -796,7 +792,6 @@ def test_forcesell_all_handle(default_conf, update, ticker, fee, markets, mocker return_value=15000.0) rpc_mock = mocker.patch('freqtrade.rpc.telegram.Telegram.send_msg', MagicMock()) mocker.patch('freqtrade.rpc.telegram.Telegram._init', MagicMock()) - mocker.patch('freqtrade.exchange.Exchange.get_pair_detail_url', MagicMock()) mocker.patch.multiple( 'freqtrade.exchange.Exchange', get_ticker=ticker, @@ -823,7 +818,6 @@ def test_forcesell_all_handle(default_conf, update, ticker, fee, markets, mocker 'exchange': 'Bittrex', 'pair': 'ETH/BTC', 'gain': 'loss', - 'market_url': ANY, 'limit': 1.098e-05, 'amount': 90.99181073703367, 'open_rate': 1.099e-05, @@ -1100,7 +1094,6 @@ def test_send_msg_buy_notification(default_conf, mocker) -> None: 'type': RPCMessageType.BUY_NOTIFICATION, 'exchange': 'Bittrex', 'pair': 'ETH/BTC', - 'market_url': 'https://bittrex.com/Market/Index?MarketName=BTC-ETH', 'limit': 1.099e-05, 'stake_amount': 0.001, 'stake_amount_fiat': 0.0, @@ -1108,7 +1101,7 @@ def test_send_msg_buy_notification(default_conf, mocker) -> None: 'fiat_currency': 'USD' }) assert msg_mock.call_args[0][0] \ - == '*Bittrex:* Buying [ETH/BTC](https://bittrex.com/Market/Index?MarketName=BTC-ETH)\n' \ + == '*Bittrex:* Buying ETH/BTC\n' \ 'with limit `0.00001099\n' \ '(0.001000 BTC,0.000 USD)`' @@ -1129,7 +1122,6 @@ def test_send_msg_sell_notification(default_conf, mocker) -> None: 'exchange': 'Binance', 'pair': 'KEY/ETH', 'gain': 'loss', - 'market_url': 'https://www.binance.com/tradeDetail.html?symbol=KEY_ETH', 'limit': 3.201e-05, 'amount': 1333.3333333333335, 'open_rate': 7.5e-05, @@ -1141,8 +1133,7 @@ def test_send_msg_sell_notification(default_conf, mocker) -> None: 'sell_reason': SellType.STOP_LOSS.value }) assert msg_mock.call_args[0][0] \ - == ('*Binance:* Selling [KEY/ETH]' - '(https://www.binance.com/tradeDetail.html?symbol=KEY_ETH)\n' + == ('*Binance:* Selling KEY/ETH\n' '*Limit:* `0.00003201`\n' '*Amount:* `1333.33333333`\n' '*Open Rate:* `0.00007500`\n' @@ -1156,7 +1147,6 @@ def test_send_msg_sell_notification(default_conf, mocker) -> None: 'exchange': 'Binance', 'pair': 'KEY/ETH', 'gain': 'loss', - 'market_url': 'https://www.binance.com/tradeDetail.html?symbol=KEY_ETH', 'limit': 3.201e-05, 'amount': 1333.3333333333335, 'open_rate': 7.5e-05, @@ -1167,8 +1157,7 @@ def test_send_msg_sell_notification(default_conf, mocker) -> None: 'sell_reason': SellType.STOP_LOSS.value }) assert msg_mock.call_args[0][0] \ - == ('*Binance:* Selling [KEY/ETH]' - '(https://www.binance.com/tradeDetail.html?symbol=KEY_ETH)\n' + == ('*Binance:* Selling KEY/ETH\n' '*Limit:* `0.00003201`\n' '*Amount:* `1333.33333333`\n' '*Open Rate:* `0.00007500`\n' @@ -1256,7 +1245,6 @@ def test_send_msg_buy_notification_no_fiat(default_conf, mocker) -> None: 'type': RPCMessageType.BUY_NOTIFICATION, 'exchange': 'Bittrex', 'pair': 'ETH/BTC', - 'market_url': 'https://bittrex.com/Market/Index?MarketName=BTC-ETH', 'limit': 1.099e-05, 'stake_amount': 0.001, 'stake_amount_fiat': 0.0, @@ -1264,7 +1252,7 @@ def test_send_msg_buy_notification_no_fiat(default_conf, mocker) -> None: 'fiat_currency': None }) assert msg_mock.call_args[0][0] \ - == '*Bittrex:* Buying [ETH/BTC](https://bittrex.com/Market/Index?MarketName=BTC-ETH)\n' \ + == '*Bittrex:* Buying ETH/BTC\n' \ 'with limit `0.00001099\n' \ '(0.001000 BTC)`' @@ -1284,7 +1272,6 @@ def test_send_msg_sell_notification_no_fiat(default_conf, mocker) -> None: 'exchange': 'Binance', 'pair': 'KEY/ETH', 'gain': 'loss', - 'market_url': 'https://www.binance.com/tradeDetail.html?symbol=KEY_ETH', 'limit': 3.201e-05, 'amount': 1333.3333333333335, 'open_rate': 7.5e-05, @@ -1296,8 +1283,7 @@ def test_send_msg_sell_notification_no_fiat(default_conf, mocker) -> None: 'sell_reason': SellType.STOP_LOSS.value }) assert msg_mock.call_args[0][0] \ - == '*Binance:* Selling [KEY/ETH]' \ - '(https://www.binance.com/tradeDetail.html?symbol=KEY_ETH)\n' \ + == '*Binance:* Selling KEY/ETH\n' \ '*Limit:* `0.00003201`\n' \ '*Amount:* `1333.33333333`\n' \ '*Open Rate:* `0.00007500`\n' \ diff --git a/freqtrade/tests/rpc/test_rpc_webhook.py b/freqtrade/tests/rpc/test_rpc_webhook.py index 002308815..da7aec0a6 100644 --- a/freqtrade/tests/rpc/test_rpc_webhook.py +++ b/freqtrade/tests/rpc/test_rpc_webhook.py @@ -48,7 +48,6 @@ def test_send_msg(default_conf, mocker): 'type': RPCMessageType.BUY_NOTIFICATION, 'exchange': 'Bittrex', 'pair': 'ETH/BTC', - 'market_url': "http://mockedurl/ETH_BTC", 'limit': 0.005, 'stake_amount': 0.8, 'stake_amount_fiat': 500, @@ -73,7 +72,6 @@ def test_send_msg(default_conf, mocker): 'exchange': 'Bittrex', 'pair': 'ETH/BTC', 'gain': "profit", - 'market_url': "http://mockedurl/ETH_BTC", 'limit': 0.005, 'amount': 0.8, 'open_rate': 0.004, @@ -127,7 +125,6 @@ def test_exception_send_msg(default_conf, mocker, caplog): 'type': RPCMessageType.BUY_NOTIFICATION, 'exchange': 'Bittrex', 'pair': 'ETH/BTC', - 'market_url': "http://mockedurl/ETH_BTC", 'limit': 0.005, 'stake_amount': 0.8, 'stake_amount_fiat': 500, diff --git a/freqtrade/tests/test_arguments.py b/freqtrade/tests/test_arguments.py index 042d43ed2..0952d1c5d 100644 --- a/freqtrade/tests/test_arguments.py +++ b/freqtrade/tests/test_arguments.py @@ -16,7 +16,7 @@ def test_parse_args_none() -> None: def test_parse_args_defaults() -> None: args = Arguments([], '').get_parsed_arg() - assert args.config == 'config.json' + assert args.config == ['config.json'] assert args.strategy_path is None assert args.datadir is None assert args.loglevel == 0 @@ -24,10 +24,15 @@ def test_parse_args_defaults() -> None: def test_parse_args_config() -> None: args = Arguments(['-c', '/dev/null'], '').get_parsed_arg() - assert args.config == '/dev/null' + assert args.config == ['/dev/null'] args = Arguments(['--config', '/dev/null'], '').get_parsed_arg() - assert args.config == '/dev/null' + assert args.config == ['/dev/null'] + + args = Arguments(['--config', '/dev/null', + '--config', '/dev/zero'], + '').get_parsed_arg() + assert args.config == ['/dev/null', '/dev/zero'] def test_parse_args_db_url() -> None: @@ -139,7 +144,7 @@ def test_parse_args_backtesting_custom() -> None: 'TestStrategy' ] call_args = Arguments(args, '').get_parsed_arg() - assert call_args.config == 'test_conf.json' + assert call_args.config == ['test_conf.json'] assert call_args.live is True assert call_args.loglevel == 0 assert call_args.subparser == 'backtesting' @@ -158,7 +163,7 @@ def test_parse_args_hyperopt_custom() -> None: '--spaces', 'buy' ] call_args = Arguments(args, '').get_parsed_arg() - assert call_args.config == 'test_conf.json' + assert call_args.config == ['test_conf.json'] assert call_args.epochs == 20 assert call_args.loglevel == 0 assert call_args.subparser == 'hyperopt' diff --git a/freqtrade/tests/test_configuration.py b/freqtrade/tests/test_configuration.py index 67445238b..51098baaa 100644 --- a/freqtrade/tests/test_configuration.py +++ b/freqtrade/tests/test_configuration.py @@ -1,15 +1,15 @@ # pragma pylint: disable=missing-docstring, protected-access, invalid-name import json -from argparse import Namespace import logging +from argparse import Namespace +from copy import deepcopy from unittest.mock import MagicMock import pytest -from jsonschema import validate, ValidationError, Draft4Validator +from jsonschema import Draft4Validator, ValidationError, validate -from freqtrade import constants -from freqtrade import OperationalException +from freqtrade import OperationalException, constants from freqtrade.arguments import Arguments from freqtrade.configuration import Configuration, set_loggers from freqtrade.constants import DEFAULT_DB_DRYRUN_URL, DEFAULT_DB_PROD_URL @@ -50,18 +50,49 @@ def test_load_config_file(default_conf, mocker, caplog) -> None: validated_conf = configuration._load_config_file('somefile') assert file_mock.call_count == 1 assert validated_conf.items() >= default_conf.items() - assert 'internals' in validated_conf - assert log_has('Validating configuration ...', caplog.record_tuples) def test_load_config_max_open_trades_zero(default_conf, mocker, caplog) -> None: default_conf['max_open_trades'] = 0 - file_mock = mocker.patch('freqtrade.configuration.open', mocker.mock_open( + mocker.patch('freqtrade.configuration.open', mocker.mock_open( read_data=json.dumps(default_conf) )) - Configuration(Namespace())._load_config_file('somefile') - assert file_mock.call_count == 1 + args = Arguments([], '').get_parsed_arg() + configuration = Configuration(args) + validated_conf = configuration.load_config() + + assert validated_conf['max_open_trades'] == 0 + assert 'internals' in validated_conf + assert log_has('Validating configuration ...', caplog.record_tuples) + + +def test_load_config_combine_dicts(default_conf, mocker, caplog) -> None: + conf1 = deepcopy(default_conf) + conf2 = deepcopy(default_conf) + del conf1['exchange']['key'] + del conf1['exchange']['secret'] + del conf2['exchange']['name'] + conf2['exchange']['pair_whitelist'] += ['NANO/BTC'] + + config_files = [conf1, conf2] + + configsmock = MagicMock(side_effect=config_files) + mocker.patch('freqtrade.configuration.Configuration._load_config_file', configsmock) + + arg_list = ['-c', 'test_conf.json', '--config', 'test2_conf.json', ] + args = Arguments(arg_list, '').get_parsed_arg() + configuration = Configuration(args) + validated_conf = configuration.load_config() + + exchange_conf = default_conf['exchange'] + assert validated_conf['exchange']['name'] == exchange_conf['name'] + assert validated_conf['exchange']['key'] == exchange_conf['key'] + assert validated_conf['exchange']['secret'] == exchange_conf['secret'] + assert validated_conf['exchange']['pair_whitelist'] != conf1['exchange']['pair_whitelist'] + assert validated_conf['exchange']['pair_whitelist'] == conf2['exchange']['pair_whitelist'] + + assert 'internals' in validated_conf assert log_has('Validating configuration ...', caplog.record_tuples) diff --git a/freqtrade/tests/test_freqtradebot.py b/freqtrade/tests/test_freqtradebot.py index a0ac6ee99..2f66a5153 100644 --- a/freqtrade/tests/test_freqtradebot.py +++ b/freqtrade/tests/test_freqtradebot.py @@ -1872,7 +1872,6 @@ def test_execute_sell_up(default_conf, ticker, fee, ticker_sell_up, markets, moc 'exchange': 'Bittrex', 'pair': 'ETH/BTC', 'gain': 'profit', - 'market_url': 'https://bittrex.com/Market/Index?MarketName=BTC-ETH', 'limit': 1.172e-05, 'amount': 90.99181073703367, 'open_rate': 1.099e-05, @@ -1919,7 +1918,6 @@ def test_execute_sell_down(default_conf, ticker, fee, ticker_sell_down, markets, 'exchange': 'Bittrex', 'pair': 'ETH/BTC', 'gain': 'loss', - 'market_url': 'https://bittrex.com/Market/Index?MarketName=BTC-ETH', 'limit': 1.044e-05, 'amount': 90.99181073703367, 'open_rate': 1.099e-05, @@ -1974,7 +1972,6 @@ def test_execute_sell_down_stoploss_on_exchange_dry_run(default_conf, ticker, fe 'exchange': 'Bittrex', 'pair': 'ETH/BTC', 'gain': 'loss', - 'market_url': 'https://bittrex.com/Market/Index?MarketName=BTC-ETH', 'limit': 1.08801e-05, 'amount': 90.99181073703367, 'open_rate': 1.099e-05, @@ -2146,7 +2143,6 @@ def test_execute_sell_without_conf_sell_up(default_conf, ticker, fee, 'exchange': 'Bittrex', 'pair': 'ETH/BTC', 'gain': 'profit', - 'market_url': 'https://bittrex.com/Market/Index?MarketName=BTC-ETH', 'limit': 1.172e-05, 'amount': 90.99181073703367, 'open_rate': 1.099e-05, @@ -2194,7 +2190,6 @@ def test_execute_sell_without_conf_sell_down(default_conf, ticker, fee, 'exchange': 'Bittrex', 'pair': 'ETH/BTC', 'gain': 'loss', - 'market_url': 'https://bittrex.com/Market/Index?MarketName=BTC-ETH', 'limit': 1.044e-05, 'amount': 90.99181073703367, 'open_rate': 1.099e-05, diff --git a/freqtrade/tests/test_main.py b/freqtrade/tests/test_main.py index 7aae98ebe..51c95a4a9 100644 --- a/freqtrade/tests/test_main.py +++ b/freqtrade/tests/test_main.py @@ -22,7 +22,7 @@ def test_parse_args_backtesting(mocker) -> None: main(['backtesting']) assert backtesting_mock.call_count == 1 call_args = backtesting_mock.call_args[0][0] - assert call_args.config == 'config.json' + assert call_args.config == ['config.json'] assert call_args.live is False assert call_args.loglevel == 0 assert call_args.subparser == 'backtesting' @@ -35,7 +35,7 @@ def test_main_start_hyperopt(mocker) -> None: main(['hyperopt']) assert hyperopt_mock.call_count == 1 call_args = hyperopt_mock.call_args[0][0] - assert call_args.config == 'config.json' + assert call_args.config == ['config.json'] assert call_args.loglevel == 0 assert call_args.subparser == 'hyperopt' assert call_args.func is not None diff --git a/freqtrade/tests/test_persistence.py b/freqtrade/tests/test_persistence.py index be6efc2ff..a9519e693 100644 --- a/freqtrade/tests/test_persistence.py +++ b/freqtrade/tests/test_persistence.py @@ -629,3 +629,48 @@ def test_adjust_stop_loss(limit_buy_order, limit_sell_order, fee): assert round(trade.stop_loss, 8) == 1.26 assert trade.max_rate == 1.4 assert trade.initial_stop_loss == 0.95 + + +def test_get_open(default_conf, fee): + init(default_conf) + + # Simulate dry_run entries + trade = Trade( + pair='ETH/BTC', + stake_amount=0.001, + amount=123.0, + fee_open=fee.return_value, + fee_close=fee.return_value, + open_rate=0.123, + exchange='bittrex', + open_order_id='dry_run_buy_12345' + ) + Trade.session.add(trade) + + trade = Trade( + pair='ETC/BTC', + stake_amount=0.001, + amount=123.0, + fee_open=fee.return_value, + fee_close=fee.return_value, + open_rate=0.123, + exchange='bittrex', + is_open=False, + open_order_id='dry_run_sell_12345' + ) + Trade.session.add(trade) + + # Simulate prod entry + trade = Trade( + pair='ETC/BTC', + stake_amount=0.001, + amount=123.0, + fee_open=fee.return_value, + fee_close=fee.return_value, + open_rate=0.123, + exchange='bittrex', + open_order_id='prod_buy_12345' + ) + Trade.session.add(trade) + + assert len(Trade.get_open_trades()) == 2 diff --git a/requirements-dev.txt b/requirements-dev.txt index 34d59d802..859d80482 100644 --- a/requirements-dev.txt +++ b/requirements-dev.txt @@ -1,7 +1,7 @@ # Include all requirements to run the bot. -r requirements.txt -flake8==3.7.6 +flake8==3.7.7 flake8-type-annotations==0.1.0 flake8-tidy-imports==2.0.0 pytest==4.3.0 diff --git a/requirements.txt b/requirements.txt index cd1d89c6d..a72198ac3 100644 --- a/requirements.txt +++ b/requirements.txt @@ -1,4 +1,4 @@ -ccxt==1.18.281 +ccxt==1.18.313 SQLAlchemy==1.2.18 python-telegram-bot==11.1.0 arrow==0.13.1 @@ -6,12 +6,12 @@ cachetools==3.1.0 requests==2.21.0 urllib3==1.24.1 wrapt==1.11.1 -numpy==1.16.1 +numpy==1.16.2 pandas==0.24.1 -scikit-learn==0.20.2 +scikit-learn==0.20.3 joblib==0.13.2 scipy==1.2.1 -jsonschema==2.6.0 +jsonschema==3.0.1 TA-Lib==0.4.17 tabulate==0.8.3 coinmarketcap==5.0.3 diff --git a/scripts/download_backtest_data.py b/scripts/download_backtest_data.py index c8fd08747..5dee41bdd 100755 --- a/scripts/download_backtest_data.py +++ b/scripts/download_backtest_data.py @@ -5,12 +5,14 @@ import json import sys from pathlib import Path import arrow +from typing import Any, Dict -from freqtrade import arguments +from freqtrade.arguments import Arguments from freqtrade.arguments import TimeRange from freqtrade.exchange import Exchange from freqtrade.data.history import download_pair_history from freqtrade.configuration import Configuration, set_loggers +from freqtrade.misc import deep_merge_dicts import logging logging.basicConfig( @@ -21,7 +23,7 @@ set_loggers(0) DEFAULT_DL_PATH = 'user_data/data' -arguments = arguments.Arguments(sys.argv[1:], 'download utility') +arguments = Arguments(sys.argv[1:], 'download utility') arguments.testdata_dl_options() args = arguments.parse_args() @@ -29,7 +31,13 @@ timeframes = args.timeframes if args.config: configuration = Configuration(args) - config = configuration._load_config_file(args.config) + + config: Dict[str, Any] = {} + # Now expecting a list of config filenames here, not a string + for path in args.config: + print('Using config: %s ...', path) + # Merge config options, overwriting old values + config = deep_merge_dicts(configuration._load_config_file(path), config) config['stake_currency'] = '' # Ensure we do not use Exchange credentials