From 1b4b10f8cdfb2e04f0acba0d6f9f674e8773ecbf Mon Sep 17 00:00:00 2001 From: Leif Segen Date: Tue, 23 Jul 2019 23:45:27 -0500 Subject: [PATCH 01/32] Update docs/installation.md Address that numpy is required before `python3 -m pip install -r requirements.txt` can run. --- docs/installation.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/installation.md b/docs/installation.md index 657273e2f..35cdcda62 100644 --- a/docs/installation.md +++ b/docs/installation.md @@ -175,6 +175,7 @@ cp config.json.example config.json ``` bash python3 -m pip install --upgrade pip +pip install numpy python3 -m pip install -r requirements.txt python3 -m pip install -e . ``` From 312533fded2c794798a860b3ad523aec6f10ecc1 Mon Sep 17 00:00:00 2001 From: Leif Segen Date: Mon, 23 Nov 2020 22:08:53 -0600 Subject: [PATCH 02/32] Match current dev file --- docs/installation.md | 249 +++++++++++++++++++------------------------ 1 file changed, 111 insertions(+), 138 deletions(-) diff --git a/docs/installation.md b/docs/installation.md index 35cdcda62..ec2d27174 100644 --- a/docs/installation.md +++ b/docs/installation.md @@ -2,6 +2,8 @@ This page explains how to prepare your environment for running the bot. +Please consider using the prebuilt [docker images](docker.md) to get started quickly while trying out freqtrade evaluating how it operates. + ## Prerequisite ### Requirements @@ -11,70 +13,78 @@ Click each one for install guide: * [Python >= 3.6.x](http://docs.python-guide.org/en/latest/starting/installation/) * [pip](https://pip.pypa.io/en/stable/installing/) * [git](https://git-scm.com/book/en/v2/Getting-Started-Installing-Git) -* [virtualenv](https://virtualenv.pypa.io/en/stable/installation/) (Recommended) +* [virtualenv](https://virtualenv.pypa.io/en/stable/installation.html) (Recommended) * [TA-Lib](https://mrjbq7.github.io/ta-lib/install.html) (install instructions below) -### API keys + We also recommend a [Telegram bot](telegram-usage.md#setup-your-telegram-bot), which is optional but recommended. -Before running your bot in production you will need to setup few -external API. In production mode, the bot will require valid Exchange API -credentials. We also recommend a [Telegram bot](telegram-usage.md#setup-your-telegram-bot) (optional but recommended). - -### Setup your exchange account - -You will need to create API Keys (Usually you get `key` and `secret`) from the Exchange website and insert this into the appropriate fields in the configuration or when asked by the installation script. +!!! Warning "Up-to-date clock" + The clock on the system running the bot must be accurate, synchronized to a NTP server frequently enough to avoid problems with communication to the exchanges. ## Quick start -Freqtrade provides a Linux/MacOS script to install all dependencies and help you to configure the bot. - -!!! Note - Python3.6 or higher and the corresponding pip are assumed to be available. The install-script will warn and stop if that's not the case. - -```bash -git clone git@github.com:freqtrade/freqtrade.git -cd freqtrade -git checkout develop -./setup.sh --install -``` +Freqtrade provides the Linux/MacOS Easy Installation script to install all dependencies and help you configure the bot. !!! Note Windows installation is explained [here](#windows). -## Easy Installation - Linux Script +The easiest way to install and run Freqtrade is to clone the bot Github repository and then run the Easy Installation script, if it's available for your platform. -If you are on Debian, Ubuntu or MacOS freqtrade provides a script to Install, Update, Configure, and Reset your bot. +!!! Note "Version considerations" + When cloning the repository the default working branch has the name `develop`. This branch contains all last features (can be considered as relatively stable, thanks to automated tests). The `stable` branch contains the code of the last release (done usually once per month on an approximately one week old snapshot of the `develop` branch to prevent packaging bugs, so potentially it's more stable). + +!!! Note + Python3.6 or higher and the corresponding `pip` are assumed to be available. The install-script will warn you and stop if that's not the case. `git` is also needed to clone the Freqtrade repository. + +This can be achieved with the following commands: + +```bash +git clone https://github.com/freqtrade/freqtrade.git +cd freqtrade +# git checkout stable # Optional, see (1) +./setup.sh --install +``` + +(1) This command switches the cloned repository to the use of the `stable` branch. It's not needed if you wish to stay on the `develop` branch. You may later switch between branches at any time with the `git checkout stable`/`git checkout develop` commands. + +## Easy Installation Script (Linux/MacOS) + +If you are on Debian, Ubuntu or MacOS Freqtrade provides the script to install, update, configure and reset the codebase of your bot. ```bash $ ./setup.sh usage: -i,--install Install freqtrade from scratch -u,--update Command git pull to update. - -r,--reset Hard reset your develop/master branch. + -r,--reset Hard reset your develop/stable branch. -c,--config Easy config generator (Will override your existing file). ``` ** --install ** -This script will install everything you need to run the bot: +With this option, the script will install the bot and most dependencies: +You will need to have git and python3.6+ installed beforehand for this to work. * Mandatory software as: `ta-lib` -* Setup your virtualenv -* Configure your `config.json` file +* Setup your virtualenv under `.env/` -This script is a combination of `install script` `--reset`, `--config` +This option is a combination of installation tasks, `--reset` and `--config`. ** --update ** -Update parameter will pull the last version of your current branch and update your virtualenv. +This option will pull the last version of your current branch and update your virtualenv. Run the script with this option periodically to update your bot. ** --reset ** -Reset parameter will hard reset your branch (only if you are on `master` or `develop`) and recreate your virtualenv. +This option will hard reset your branch (only if you are on either `stable` or `develop`) and recreate your virtualenv. ** --config ** -Config parameter is a `config.json` configurator. This script will ask you questions to setup your bot and create your `config.json`. +DEPRECATED - use `freqtrade new-config -c config.json` instead. + +### Activate your virtual environment + +Each time you open a new terminal, you must run `source .env/bin/activate`. ------ @@ -86,40 +96,50 @@ OS Specific steps are listed first, the [Common](#common) section below is neces !!! Note Python3.6 or higher and the corresponding pip are assumed to be available. -### Linux - Ubuntu 16.04 +=== "Ubuntu 16.04" + #### Install necessary dependencies -#### Install necessary dependencies + ```bash + sudo apt-get update + sudo apt-get install build-essential git + ``` -```bash -sudo apt-get update -sudo apt-get install build-essential git -``` +=== "RaspberryPi/Raspbian" + The following assumes the latest [Raspbian Buster lite image](https://www.raspberrypi.org/downloads/raspbian/) from at least September 2019. + This image comes with python3.7 preinstalled, making it easy to get freqtrade up and running. -#### Raspberry Pi / Raspbian + Tested using a Raspberry Pi 3 with the Raspbian Buster lite image, all updates applied. -Before installing FreqTrade on a Raspberry Pi running the official Raspbian Image, make sure you have at least Python 3.6 installed. The default image only provides Python 3.5. Probably the easiest way to get a recent version of python is [miniconda](https://repo.continuum.io/miniconda/). + ``` bash + sudo apt-get install python3-venv libatlas-base-dev + git clone https://github.com/freqtrade/freqtrade.git + cd freqtrade -The following assumes that miniconda3 is installed and available in your environment. Last miniconda3 installation file use python 3.4, we will update to python 3.6 on this installation. -It's recommended to use (mini)conda for this as installation/compilation of `numpy`, `scipy` and `pandas` takes a long time. + bash setup.sh -i + ``` -Additional package to install on your Raspbian, `libffi-dev` required by cryptography (from python-telegram-bot). + !!! Note "Installation duration" + Depending on your internet speed and the Raspberry Pi version, installation can take multiple hours to complete. -``` bash -conda config --add channels rpi -conda install python=3.6 -conda create -n freqtrade python=3.6 -conda activate freqtrade -conda install scipy pandas numpy - -sudo apt install libffi-dev -python3 -m pip install -r requirements-common.txt -python3 -m pip install -e . -``` + !!! Note + The above does not install hyperopt dependencies. To install these, please use `python3 -m pip install -e .[hyperopt]`. + We do not advise to run hyperopt on a Raspberry Pi, since this is a very resource-heavy operation, which should be done on powerful machine. ### Common #### 1. Install TA-Lib +Use the provided ta-lib installation script + +```bash +sudo ./build_helpers/install_ta-lib.sh +``` + +!!! Note + This will use the ta-lib tar.gz included in this repository. + +##### TA-Lib manual installation + Official webpage: https://mrjbq7.github.io/ta-lib/install.html ```bash @@ -147,126 +167,79 @@ python3 -m venv .env source .env/bin/activate ``` -#### 3. Install FreqTrade +#### 3. Install Freqtrade Clone the git repository: ```bash git clone https://github.com/freqtrade/freqtrade.git - -``` - -Optionally checkout the master branch to get the latest stable release: - -```bash -git checkout master -``` - -#### 4. Initialize the configuration - -```bash cd freqtrade -cp config.json.example config.json +git checkout stable ``` -> *To edit the config please refer to [Bot Configuration](configuration.md).* - -#### 5. Install python dependencies +#### 4. Install python dependencies ``` bash python3 -m pip install --upgrade pip -pip install numpy -python3 -m pip install -r requirements.txt python3 -m pip install -e . ``` +#### 5. Initialize the configuration + +```bash +# Initialize the user_directory +freqtrade create-userdir --userdir user_data/ + +# Create a new configuration file +freqtrade new-config --config config.json +``` + +> *To edit the config please refer to [Bot Configuration](configuration.md).* + #### 6. Run the Bot If this is the first time you run the bot, ensure you are running it in Dry-run `"dry_run": true,` otherwise it will start to buy and sell coins. ```bash -freqtrade -c config.json +freqtrade trade -c config.json ``` *Note*: If you run the bot on a server, you should consider using [Docker](docker.md) or a terminal multiplexer like `screen` or [`tmux`](https://en.wikipedia.org/wiki/Tmux) to avoid that the bot is stopped on logout. -#### 7. [Optional] Configure `freqtrade` as a `systemd` service +#### 7. (Optional) Post-installation Tasks -From the freqtrade repo... copy `freqtrade.service` to your systemd user directory (usually `~/.config/systemd/user`) and update `WorkingDirectory` and `ExecStart` to match your setup. - -After that you can start the daemon with: - -```bash -systemctl --user start freqtrade -``` - -For this to be persistent (run when user is logged out) you'll need to enable `linger` for your freqtrade user. - -```bash -sudo loginctl enable-linger "$USER" -``` - -If you run the bot as a service, you can use systemd service manager as a software watchdog monitoring freqtrade bot -state and restarting it in the case of failures. If the `internals.sd_notify` parameter is set to true in the -configuration or the `--sd-notify` command line option is used, the bot will send keep-alive ping messages to systemd -using the sd_notify (systemd notifications) protocol and will also tell systemd its current state (Running or Stopped) -when it changes. - -The `freqtrade.service.watchdog` file contains an example of the service unit configuration file which uses systemd -as the watchdog. - -!!! Note - The sd_notify communication between the bot and the systemd service manager will not work if the bot runs in a Docker container. +On Linux, as an optional post-installation task, you may wish to setup the bot to run as a `systemd` service or configure it to send the log messages to the `syslog`/`rsyslog` or `journald` daemons. See [Advanced Logging](advanced-setup.md#advanced-logging) for details. ------ -## Windows +### Anaconda -We recommend that Windows users use [Docker](docker.md) as this will work much easier and smoother (also more secure). +Freqtrade can also be installed using Anaconda (or Miniconda). -If that is not possible, try using the Windows Linux subsystem (WSL) - for which the Ubuntu instructions should work. -If that is not available on your system, feel free to try the instructions below, which led to success for some. - -### Install freqtrade manually - -#### Clone the git repository - -```bash -git clone https://github.com/freqtrade/freqtrade.git -``` - -#### Install ta-lib - -Install ta-lib according to the [ta-lib documentation](https://github.com/mrjbq7/ta-lib#windows). - -As compiling from source on windows has heavy dependencies (requires a partial visual studio installation), there is also a repository of unofficial precompiled windows Wheels [here](https://www.lfd.uci.edu/~gohlke/pythonlibs/#ta-lib), which needs to be downloaded and installed using `pip install TA_Lib‑0.4.17‑cp36‑cp36m‑win32.whl` (make sure to use the version matching your python version) - -```cmd ->cd \path\freqtrade-develop ->python -m venv .env ->cd .env\Scripts ->activate.bat ->cd \path\freqtrade-develop -REM optionally install ta-lib from wheel -REM >pip install TA_Lib‑0.4.17‑cp36‑cp36m‑win32.whl ->pip install -r requirements.txt ->pip install -e . ->python freqtrade\main.py -``` - -> Thanks [Owdr](https://github.com/Owdr) for the commands. Source: [Issue #222](https://github.com/freqtrade/freqtrade/issues/222) - -#### Error during installation under Windows +!!! Note + This requires the [ta-lib](#1-install-ta-lib) C-library to be installed first. See below. ``` bash -error: Microsoft Visual C++ 14.0 is required. Get it with "Microsoft Visual C++ Build Tools": http://landinghub.visualstudio.com/visual-cpp-build-tools +conda env create -f environment.yml ``` -Unfortunately, many packages requiring compilation don't provide a pre-build wheel. It is therefore mandatory to have a C/C++ compiler installed and available for your python environment to use. +----- +## Troubleshooting -The easiest way is to download install Microsoft Visual Studio Community [here](https://visualstudio.microsoft.com/downloads/) and make sure to install "Common Tools for Visual C++" to enable building c code on Windows. Unfortunately, this is a heavy download / dependency (~4Gb) so you might want to consider WSL or [docker](docker.md) first. +### MacOS installation error ---- +Newer versions of MacOS may have installation failed with errors like `error: command 'g++' failed with exit status 1`. + +This error will require explicit installation of the SDK Headers, which are not installed by default in this version of MacOS. +For MacOS 10.14, this can be accomplished with the below command. + +``` bash +open /Library/Developer/CommandLineTools/Packages/macOS_SDK_headers_for_macOS_10.14.pkg +``` + +If this file is inexistent, then you're probably on a different version of MacOS, so you may need to consult the internet for specific resolution details. + +----- Now you have an environment ready, the next step is -[Bot Configuration](configuration.md). +[Bot Configuration](configuration.md). \ No newline at end of file From 46ec6f498c7cf95c677955051ce17e3770cfb6ed Mon Sep 17 00:00:00 2001 From: Leif Segen Date: Fri, 27 Nov 2020 12:51:44 -0600 Subject: [PATCH 03/32] Correct link Fix prior redirection to a non-working link: https://www.freqtrade.io/en/latest/telegram-usage/configuration/#understand-forcebuy_enable --- docs/telegram-usage.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/telegram-usage.md b/docs/telegram-usage.md index 09cf21223..f4bd0a12a 100644 --- a/docs/telegram-usage.md +++ b/docs/telegram-usage.md @@ -207,7 +207,7 @@ Return a summary of your profit/loss and performance. Note that for this to work, `forcebuy_enable` needs to be set to true. -[More details](configuration.md/#understand-forcebuy_enable) +[More details](configuration.md#understand-forcebuy_enable) ### /performance From 7cbd89657f1f477451d91c962f1fad260858385c Mon Sep 17 00:00:00 2001 From: Leif Segen Date: Fri, 27 Nov 2020 21:24:40 -0600 Subject: [PATCH 04/32] Initial step towards implementing proposed code --- freqtrade/constants.py | 2 +- freqtrade/pairlist/PerformanceFilter.py | 61 +++++++++++++++++++++++++ tests/pairlist/test_pairlist.py | 22 ++++++++- 3 files changed, 83 insertions(+), 2 deletions(-) create mode 100644 freqtrade/pairlist/PerformanceFilter.py diff --git a/freqtrade/constants.py b/freqtrade/constants.py index 3271dda39..f47301fa6 100644 --- a/freqtrade/constants.py +++ b/freqtrade/constants.py @@ -25,7 +25,7 @@ HYPEROPT_LOSS_BUILTIN = ['ShortTradeDurHyperOptLoss', 'OnlyProfitHyperOptLoss', 'SortinoHyperOptLoss', 'SortinoHyperOptLossDaily'] AVAILABLE_PAIRLISTS = ['StaticPairList', 'VolumePairList', 'AgeFilter', 'PrecisionFilter', 'PriceFilter', - 'ShuffleFilter', 'SpreadFilter'] + 'ShuffleFilter', 'SpreadFilter', 'PerformanceFilter'] AVAILABLE_DATAHANDLERS = ['json', 'jsongz', 'hdf5'] DRY_RUN_WALLET = 1000 DATETIME_PRINT_FORMAT = '%Y-%m-%d %H:%M:%S' diff --git a/freqtrade/pairlist/PerformanceFilter.py b/freqtrade/pairlist/PerformanceFilter.py new file mode 100644 index 000000000..e689ba0bc --- /dev/null +++ b/freqtrade/pairlist/PerformanceFilter.py @@ -0,0 +1,61 @@ +""" +Performance pair list filter +""" +import logging +import random +from typing import Any, Dict, List + +import pandas as pd +from pandas import DataFrame, Series + +from freqtrade.pairlist.IPairList import IPairList + +from freqtrade.persistence import Trade +from datetime import timedelta, datetime, timezone + +logger = logging.getLogger(__name__) + +class PerformanceFilter(IPairList): + + def __init__(self, exchange, pairlistmanager, + config: Dict[str, Any], pairlistconfig: Dict[str, Any], + pairlist_pos: int) -> None: + super().__init__(exchange, pairlistmanager, config, pairlistconfig, pairlist_pos) + + + @property + def needstickers(self) -> bool: + """ + Boolean property defining if tickers are necessary. + If no Pairlist requries tickers, an empty List is passed + as tickers argument to filter_pairlist + """ + return False + + def short_desc(self) -> str: + """ + Short whitelist method description - used for startup-messages + """ + return f"{self.name} - Sorting pairs by performance." + + def filter_pairlist(self, pairlist: List[str], tickers: Dict) -> List[str]: + """ + Filters and sorts pairlist and returns the whitelist again. + Called on each bot iteration - please use internal caching if necessary + :param pairlist: pairlist to filter or sort + :param tickers: Tickers (from exchange.get_tickers()). May be cached. + :return: new whitelist + """ + + # Get the trading performance for pairs from database + perf = pd.DataFrame(Trade.get_overall_performance()) + # update pairlist with values from performance dataframe + # set initial value for pairs with no trades to 0 + # and sort the list using performance and count + list_df = pd.DataFrame({'pair':pairlist}) + sorted_df = list_df.join(perf.set_index('pair'), on='pair')\ + .fillna(0).sort_values(by=['profit', 'count'], ascending=False) + pairlist = sorted_df['pair'].tolist() + + + return pairlist \ No newline at end of file diff --git a/tests/pairlist/test_pairlist.py b/tests/pairlist/test_pairlist.py index 1f05bef1e..2643a0bd8 100644 --- a/tests/pairlist/test_pairlist.py +++ b/tests/pairlist/test_pairlist.py @@ -246,7 +246,7 @@ def test_VolumePairList_refresh_empty(mocker, markets_empty, whitelist_conf): {"method": "PrecisionFilter"}, {"method": "PriceFilter", "low_price_ratio": 0.03}, {"method": "SpreadFilter", "max_spread_ratio": 0.005}, - {"method": "ShuffleFilter"}], + {"method": "ShuffleFilter"}, {"method": "PerformanceFilter"}], "ETH", []), # AgeFilter and VolumePairList (require 2 days only, all should pass age test) ([{"method": "VolumePairList", "number_assets": 5, "sort_key": "quoteVolume"}, @@ -302,6 +302,18 @@ def test_VolumePairList_refresh_empty(mocker, markets_empty, whitelist_conf): ([{"method": "VolumePairList", "number_assets": 5, "sort_key": "quoteVolume"}, {"method": "ShuffleFilter"}], "USDT", 3), # whitelist_result is integer -- check only length of randomized pairlist + # PerformanceFilter + ([{"method": "VolumePairList", "number_assets": 5, "sort_key": "quoteVolume"}, + {"method": "PerformanceFilter", "seed": 77}], + "USDT", ['ADADOUBLE/USDT', 'ETH/USDT', 'NANO/USDT', 'ADAHALF/USDT']), + # PerformanceFilter, other seed + ([{"method": "VolumePairList", "number_assets": 5, "sort_key": "quoteVolume"}, + {"method": "PerformanceFilter", "seed": 42}], + "USDT", ['ADAHALF/USDT', 'NANO/USDT', 'ADADOUBLE/USDT', 'ETH/USDT']), + # PerformanceFilter, no seed + ([{"method": "VolumePairList", "number_assets": 5, "sort_key": "quoteVolume"}, + {"method": "PerformanceFilter"}], + "USDT", 3), # whitelist_result is integer -- check only length of randomized pairlist # AgeFilter only ([{"method": "AgeFilter", "min_days_listed": 2}], "BTC", 'filter_at_the_beginning'), # OperationalException expected @@ -326,6 +338,13 @@ def test_VolumePairList_refresh_empty(mocker, markets_empty, whitelist_conf): # ShuffleFilter only ([{"method": "ShuffleFilter", "seed": 42}], "BTC", 'filter_at_the_beginning'), # OperationalException expected + # PrecisionFilter after StaticPairList + ([{"method": "StaticPairList"}, + {"method": "PrecisionFilter", "seed": 42}], + "BTC", ['TKN/BTC', 'ETH/BTC', 'HOT/BTC']), + # PrecisionFilter only + ([{"method": "PrecisionFilter", "seed": 42}], + "BTC", 'filter_at_the_beginning'), # OperationalException expected # SpreadFilter after StaticPairList ([{"method": "StaticPairList"}, {"method": "SpreadFilter", "max_spread_ratio": 0.005}], @@ -379,6 +398,7 @@ def test_VolumePairList_whitelist_gen(mocker, whitelist_conf, shitcoinmarkets, t assert isinstance(whitelist, list) # Verify length of pairlist matches (used for ShuffleFilter without seed) + # TBD if this applies to PerformanceFilter if type(whitelist_result) is list: assert whitelist == whitelist_result else: From 05686998bbc497f56d7407cc96d713686c4f6d85 Mon Sep 17 00:00:00 2001 From: Leif Segen Date: Fri, 27 Nov 2020 21:26:42 -0600 Subject: [PATCH 05/32] Add starter entry in documentation --- docs/includes/pairlists.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/docs/includes/pairlists.md b/docs/includes/pairlists.md index e6a9fc1a8..f8b33b27d 100644 --- a/docs/includes/pairlists.md +++ b/docs/includes/pairlists.md @@ -15,6 +15,7 @@ Inactive markets are always removed from the resulting pairlist. Explicitly blac * [`StaticPairList`](#static-pair-list) (default, if not configured differently) * [`VolumePairList`](#volume-pair-list) * [`AgeFilter`](#agefilter) +* [`PerformanceFilter`](#performancefilter) * [`PrecisionFilter`](#precisionfilter) * [`PriceFilter`](#pricefilter) * [`ShuffleFilter`](#shufflefilter) @@ -73,6 +74,10 @@ be caught out buying before the pair has finished dropping in price. This filter allows freqtrade to ignore pairs until they have been listed for at least `min_days_listed` days. +#### PerformanceFilter + +Lorem ipsum. + #### PrecisionFilter Filters low-value coins which would not allow setting stoplosses. From c34150552f348245cac68611a27d4b26eabc5f8a Mon Sep 17 00:00:00 2001 From: Leif Segen Date: Fri, 27 Nov 2020 21:36:55 -0600 Subject: [PATCH 06/32] Revert unrelated change --- docs/telegram-usage.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/telegram-usage.md b/docs/telegram-usage.md index f4bd0a12a..09cf21223 100644 --- a/docs/telegram-usage.md +++ b/docs/telegram-usage.md @@ -207,7 +207,7 @@ Return a summary of your profit/loss and performance. Note that for this to work, `forcebuy_enable` needs to be set to true. -[More details](configuration.md#understand-forcebuy_enable) +[More details](configuration.md/#understand-forcebuy_enable) ### /performance From 335735062835636d1bce627b9117030a5595f69c Mon Sep 17 00:00:00 2001 From: Leif Segen Date: Fri, 27 Nov 2020 22:00:36 -0600 Subject: [PATCH 07/32] Revert unintended change --- docs/installation.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/installation.md b/docs/installation.md index ec2d27174..9b15c9685 100644 --- a/docs/installation.md +++ b/docs/installation.md @@ -242,4 +242,4 @@ If this file is inexistent, then you're probably on a different version of MacOS ----- Now you have an environment ready, the next step is -[Bot Configuration](configuration.md). \ No newline at end of file +[Bot Configuration](configuration.md). From 380cca225239397f04d503fee36c189ee06014aa Mon Sep 17 00:00:00 2001 From: Leif Segen Date: Fri, 27 Nov 2020 22:00:48 -0600 Subject: [PATCH 08/32] Remove unused imports --- freqtrade/pairlist/PerformanceFilter.py | 3 --- 1 file changed, 3 deletions(-) diff --git a/freqtrade/pairlist/PerformanceFilter.py b/freqtrade/pairlist/PerformanceFilter.py index e689ba0bc..a2f2eb489 100644 --- a/freqtrade/pairlist/PerformanceFilter.py +++ b/freqtrade/pairlist/PerformanceFilter.py @@ -2,16 +2,13 @@ Performance pair list filter """ import logging -import random from typing import Any, Dict, List import pandas as pd -from pandas import DataFrame, Series from freqtrade.pairlist.IPairList import IPairList from freqtrade.persistence import Trade -from datetime import timedelta, datetime, timezone logger = logging.getLogger(__name__) From afb795b6f53de9ec00ccdbe2b4b128659831ad81 Mon Sep 17 00:00:00 2001 From: Leif Segen Date: Fri, 27 Nov 2020 22:08:23 -0600 Subject: [PATCH 09/32] Remove unnecessary test PerforamnceFilter doesn't use seeds, so no need to provide different ones. --- tests/pairlist/test_pairlist.py | 6 +----- 1 file changed, 1 insertion(+), 5 deletions(-) diff --git a/tests/pairlist/test_pairlist.py b/tests/pairlist/test_pairlist.py index 2643a0bd8..64468fc05 100644 --- a/tests/pairlist/test_pairlist.py +++ b/tests/pairlist/test_pairlist.py @@ -302,14 +302,10 @@ def test_VolumePairList_refresh_empty(mocker, markets_empty, whitelist_conf): ([{"method": "VolumePairList", "number_assets": 5, "sort_key": "quoteVolume"}, {"method": "ShuffleFilter"}], "USDT", 3), # whitelist_result is integer -- check only length of randomized pairlist - # PerformanceFilter + # PerformanceFilter, unneeded seed provided ([{"method": "VolumePairList", "number_assets": 5, "sort_key": "quoteVolume"}, {"method": "PerformanceFilter", "seed": 77}], "USDT", ['ADADOUBLE/USDT', 'ETH/USDT', 'NANO/USDT', 'ADAHALF/USDT']), - # PerformanceFilter, other seed - ([{"method": "VolumePairList", "number_assets": 5, "sort_key": "quoteVolume"}, - {"method": "PerformanceFilter", "seed": 42}], - "USDT", ['ADAHALF/USDT', 'NANO/USDT', 'ADADOUBLE/USDT', 'ETH/USDT']), # PerformanceFilter, no seed ([{"method": "VolumePairList", "number_assets": 5, "sort_key": "quoteVolume"}, {"method": "PerformanceFilter"}], From 91b4c80d35611bbcf812a3a46dd5860e0ec949d2 Mon Sep 17 00:00:00 2001 From: Leif Segen Date: Fri, 27 Nov 2020 22:18:49 -0600 Subject: [PATCH 10/32] Remove unused parameters --- freqtrade/pairlist/PerformanceFilter.py | 2 -- tests/pairlist/test_pairlist.py | 8 ++++---- 2 files changed, 4 insertions(+), 6 deletions(-) diff --git a/freqtrade/pairlist/PerformanceFilter.py b/freqtrade/pairlist/PerformanceFilter.py index a2f2eb489..d4bd5936d 100644 --- a/freqtrade/pairlist/PerformanceFilter.py +++ b/freqtrade/pairlist/PerformanceFilter.py @@ -43,7 +43,6 @@ class PerformanceFilter(IPairList): :param tickers: Tickers (from exchange.get_tickers()). May be cached. :return: new whitelist """ - # Get the trading performance for pairs from database perf = pd.DataFrame(Trade.get_overall_performance()) # update pairlist with values from performance dataframe @@ -53,6 +52,5 @@ class PerformanceFilter(IPairList): sorted_df = list_df.join(perf.set_index('pair'), on='pair')\ .fillna(0).sort_values(by=['profit', 'count'], ascending=False) pairlist = sorted_df['pair'].tolist() - return pairlist \ No newline at end of file diff --git a/tests/pairlist/test_pairlist.py b/tests/pairlist/test_pairlist.py index 64468fc05..9814aea3e 100644 --- a/tests/pairlist/test_pairlist.py +++ b/tests/pairlist/test_pairlist.py @@ -425,7 +425,7 @@ def test_VolumePairList_whitelist_gen(mocker, whitelist_conf, shitcoinmarkets, t assert not log_has(logmsg, caplog) -def test_PrecisionFilter_error(mocker, whitelist_conf, tickers) -> None: +def test_PrecisionFilter_error(mocker, whitelist_conf) -> None: whitelist_conf['pairlists'] = [{"method": "StaticPairList"}, {"method": "PrecisionFilter"}] del whitelist_conf['stoploss'] @@ -498,7 +498,7 @@ def test__whitelist_for_active_markets(mocker, whitelist_conf, markets, pairlist @pytest.mark.parametrize("pairlist", AVAILABLE_PAIRLISTS) -def test__whitelist_for_active_markets_empty(mocker, whitelist_conf, markets, pairlist, tickers): +def test__whitelist_for_active_markets_empty(mocker, whitelist_conf, pairlist, tickers): whitelist_conf['pairlists'][0]['method'] = pairlist mocker.patch('freqtrade.exchange.Exchange.exchange_has', return_value=True) @@ -514,7 +514,7 @@ def test__whitelist_for_active_markets_empty(mocker, whitelist_conf, markets, pa pairlist_handler._whitelist_for_active_markets(['ETH/BTC']) -def test_volumepairlist_invalid_sortvalue(mocker, markets, whitelist_conf): +def test_volumepairlist_invalid_sortvalue(mocker, whitelist_conf): whitelist_conf['pairlists'][0].update({"sort_key": "asdf"}) mocker.patch('freqtrade.exchange.Exchange.exchange_has', MagicMock(return_value=True)) @@ -652,7 +652,7 @@ def test_pricefilter_desc(mocker, whitelist_conf, markets, pairlistconfig, freqtrade = get_patched_freqtradebot(mocker, whitelist_conf) -def test_pairlistmanager_no_pairlist(mocker, markets, whitelist_conf, caplog): +def test_pairlistmanager_no_pairlist(mocker, whitelist_conf): mocker.patch('freqtrade.exchange.Exchange.exchange_has', MagicMock(return_value=True)) whitelist_conf['pairlists'] = [] From 9538fa1d723cca40a862a265e24fc89e3f559c06 Mon Sep 17 00:00:00 2001 From: Leif Segen Date: Sat, 28 Nov 2020 00:24:48 -0600 Subject: [PATCH 11/32] Tweak main parameterized block for PerformanceFilter Remove randomized exception that was geared toward ShuffleFilter. Remove case involvoing seed, also geared toward ShuffleFilter. Mock get_overall_performance(). --- tests/pairlist/test_pairlist.py | 46 ++++++++++++++++++++++++++------- 1 file changed, 37 insertions(+), 9 deletions(-) diff --git a/tests/pairlist/test_pairlist.py b/tests/pairlist/test_pairlist.py index 9814aea3e..71d65d236 100644 --- a/tests/pairlist/test_pairlist.py +++ b/tests/pairlist/test_pairlist.py @@ -302,14 +302,10 @@ def test_VolumePairList_refresh_empty(mocker, markets_empty, whitelist_conf): ([{"method": "VolumePairList", "number_assets": 5, "sort_key": "quoteVolume"}, {"method": "ShuffleFilter"}], "USDT", 3), # whitelist_result is integer -- check only length of randomized pairlist - # PerformanceFilter, unneeded seed provided - ([{"method": "VolumePairList", "number_assets": 5, "sort_key": "quoteVolume"}, - {"method": "PerformanceFilter", "seed": 77}], - "USDT", ['ADADOUBLE/USDT', 'ETH/USDT', 'NANO/USDT', 'ADAHALF/USDT']), - # PerformanceFilter, no seed + # PerformanceFilter ([{"method": "VolumePairList", "number_assets": 5, "sort_key": "quoteVolume"}, {"method": "PerformanceFilter"}], - "USDT", 3), # whitelist_result is integer -- check only length of randomized pairlist + "USDT", ['ETH/USDT', 'NANO/USDT', 'ADAHALF/USDT', 'ADADOUBLE/USDT']), # AgeFilter only ([{"method": "AgeFilter", "min_days_listed": 2}], "BTC", 'filter_at_the_beginning'), # OperationalException expected @@ -381,6 +377,11 @@ def test_VolumePairList_whitelist_gen(mocker, whitelist_conf, shitcoinmarkets, t get_historic_ohlcv=MagicMock(return_value=ohlcv_history_list), ) + # Provide for PerformanceFilter's dependency + mocker.patch.multiple('freqtrade.persistence.Trade', + get_overall_performance=MagicMock(return_value=[{'pair':'ETH/BTC','profit':5,'count':3}]), + ) + # Set whitelist_result to None if pairlist is invalid and should produce exception if whitelist_result == 'filter_at_the_beginning': with pytest.raises(OperationalException, @@ -394,7 +395,6 @@ def test_VolumePairList_whitelist_gen(mocker, whitelist_conf, shitcoinmarkets, t assert isinstance(whitelist, list) # Verify length of pairlist matches (used for ShuffleFilter without seed) - # TBD if this applies to PerformanceFilter if type(whitelist_result) is list: assert whitelist == whitelist_result else: @@ -544,7 +544,7 @@ def test_volumepairlist_caching(mocker, markets, whitelist_conf, tickers): assert freqtrade.pairlists._pairlist_handlers[0]._last_refresh == lrf -def test_agefilter_min_days_listed_too_small(mocker, default_conf, markets, tickers, caplog): +def test_agefilter_min_days_listed_too_small(mocker, default_conf, markets, tickers): default_conf['pairlists'] = [{'method': 'VolumePairList', 'number_assets': 10}, {'method': 'AgeFilter', 'min_days_listed': -1}] @@ -559,7 +559,7 @@ def test_agefilter_min_days_listed_too_small(mocker, default_conf, markets, tick get_patched_freqtradebot(mocker, default_conf) -def test_agefilter_min_days_listed_too_large(mocker, default_conf, markets, tickers, caplog): +def test_agefilter_min_days_listed_too_large(mocker, default_conf, markets, tickers): default_conf['pairlists'] = [{'method': 'VolumePairList', 'number_assets': 10}, {'method': 'AgeFilter', 'min_days_listed': 99999}] @@ -660,3 +660,31 @@ def test_pairlistmanager_no_pairlist(mocker, whitelist_conf): with pytest.raises(OperationalException, match=r"No Pairlist Handlers defined"): get_patched_freqtradebot(mocker, whitelist_conf) + + +@pytest.mark.parametrize("pairlists,base_currency,overall_performance,expected", [ + # Happy path, descening order, all values filled + ([{"method": "StaticPairList"},{"method": "PerformanceFilter"}],'BTC',[{'pair':'ETH/BTC','profit':5,'count':3}, {'pair':'ETC/BTC','profit':4,'count':2}],['ETC/BTC']), +]) +def test_performance_filter(mocker, whitelist_conf, base_currency, pairlists, overall_performance, expected, tickers, markets, ohlcv_history_list): + whitelist_conf['pairlists'] = pairlists + whitelist_conf['stake_currency'] = base_currency + + mocker.patch('freqtrade.exchange.Exchange.exchange_has', MagicMock(return_value=True)) + + freqtrade = get_patched_freqtradebot(mocker, whitelist_conf) + mocker.patch.multiple('freqtrade.exchange.Exchange', + get_tickers=tickers, + markets=PropertyMock(return_value=markets) + ) + mocker.patch.multiple( + 'freqtrade.exchange.Exchange', + get_historic_ohlcv=MagicMock(return_value=ohlcv_history_list), + ) + + mocker.patch.multiple('freqtrade.persistence.Trade', + get_overall_performance=MagicMock(return_value=overall_performance), + ) + freqtrade.pairlists.refresh_pairlist() + whitelist = freqtrade.pairlists.whitelist + assert whitelist == expected From 4600bb807c41782420806c62f819fa56b393c6b9 Mon Sep 17 00:00:00 2001 From: Leif Segen Date: Sat, 28 Nov 2020 00:38:06 -0600 Subject: [PATCH 12/32] Existing tests pass. --- tests/pairlist/test_pairlist.py | 56 ++++++++++++++++----------------- 1 file changed, 28 insertions(+), 28 deletions(-) diff --git a/tests/pairlist/test_pairlist.py b/tests/pairlist/test_pairlist.py index 71d65d236..86e4616e0 100644 --- a/tests/pairlist/test_pairlist.py +++ b/tests/pairlist/test_pairlist.py @@ -330,12 +330,12 @@ def test_VolumePairList_refresh_empty(mocker, markets_empty, whitelist_conf): # ShuffleFilter only ([{"method": "ShuffleFilter", "seed": 42}], "BTC", 'filter_at_the_beginning'), # OperationalException expected - # PrecisionFilter after StaticPairList + # PerformanceFilter after StaticPairList ([{"method": "StaticPairList"}, - {"method": "PrecisionFilter", "seed": 42}], - "BTC", ['TKN/BTC', 'ETH/BTC', 'HOT/BTC']), - # PrecisionFilter only - ([{"method": "PrecisionFilter", "seed": 42}], + {"method": "PerformanceFilter", "seed": 42}], + "BTC", ['ETH/BTC', 'TKN/BTC', 'HOT/BTC']), # Order matches order of appearance in whitelist_conf > exchange > pair_whitelist + # PerformanceFilter only + ([{"method": "PerformanceFilter", "seed": 42}], "BTC", 'filter_at_the_beginning'), # OperationalException expected # SpreadFilter after StaticPairList ([{"method": "StaticPairList"}, @@ -662,29 +662,29 @@ def test_pairlistmanager_no_pairlist(mocker, whitelist_conf): get_patched_freqtradebot(mocker, whitelist_conf) -@pytest.mark.parametrize("pairlists,base_currency,overall_performance,expected", [ - # Happy path, descening order, all values filled - ([{"method": "StaticPairList"},{"method": "PerformanceFilter"}],'BTC',[{'pair':'ETH/BTC','profit':5,'count':3}, {'pair':'ETC/BTC','profit':4,'count':2}],['ETC/BTC']), -]) -def test_performance_filter(mocker, whitelist_conf, base_currency, pairlists, overall_performance, expected, tickers, markets, ohlcv_history_list): - whitelist_conf['pairlists'] = pairlists - whitelist_conf['stake_currency'] = base_currency +# @pytest.mark.parametrize("pairlists,base_currency,overall_performance,expected", [ +# # Happy path, descening order, all values filled +# ([{"method": "StaticPairList"},{"method": "PerformanceFilter"}],'BTC',[{'pair':'ETH/BTC','profit':5,'count':3}, {'pair':'ETC/BTC','profit':4,'count':2}],['ETC/BTC']), +# ]) +# def test_performance_filter(mocker, whitelist_conf, base_currency, pairlists, overall_performance, expected, tickers, markets, ohlcv_history_list): +# whitelist_conf['pairlists'] = pairlists +# whitelist_conf['stake_currency'] = base_currency - mocker.patch('freqtrade.exchange.Exchange.exchange_has', MagicMock(return_value=True)) +# mocker.patch('freqtrade.exchange.Exchange.exchange_has', MagicMock(return_value=True)) - freqtrade = get_patched_freqtradebot(mocker, whitelist_conf) - mocker.patch.multiple('freqtrade.exchange.Exchange', - get_tickers=tickers, - markets=PropertyMock(return_value=markets) - ) - mocker.patch.multiple( - 'freqtrade.exchange.Exchange', - get_historic_ohlcv=MagicMock(return_value=ohlcv_history_list), - ) +# freqtrade = get_patched_freqtradebot(mocker, whitelist_conf) +# mocker.patch.multiple('freqtrade.exchange.Exchange', +# get_tickers=tickers, +# markets=PropertyMock(return_value=markets) +# ) +# mocker.patch.multiple( +# 'freqtrade.exchange.Exchange', +# get_historic_ohlcv=MagicMock(return_value=ohlcv_history_list), +# ) - mocker.patch.multiple('freqtrade.persistence.Trade', - get_overall_performance=MagicMock(return_value=overall_performance), - ) - freqtrade.pairlists.refresh_pairlist() - whitelist = freqtrade.pairlists.whitelist - assert whitelist == expected +# mocker.patch.multiple('freqtrade.persistence.Trade', +# get_overall_performance=MagicMock(return_value=overall_performance), +# ) +# freqtrade.pairlists.refresh_pairlist() +# whitelist = freqtrade.pairlists.whitelist +# assert whitelist == expected From 26855800a3b0229c1ea3a3c0f9d97268e9de57f7 Mon Sep 17 00:00:00 2001 From: Leif Segen Date: Sat, 28 Nov 2020 00:39:18 -0600 Subject: [PATCH 13/32] Remove unused seed --- tests/pairlist/test_pairlist.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/tests/pairlist/test_pairlist.py b/tests/pairlist/test_pairlist.py index 86e4616e0..ae80a3975 100644 --- a/tests/pairlist/test_pairlist.py +++ b/tests/pairlist/test_pairlist.py @@ -332,10 +332,10 @@ def test_VolumePairList_refresh_empty(mocker, markets_empty, whitelist_conf): "BTC", 'filter_at_the_beginning'), # OperationalException expected # PerformanceFilter after StaticPairList ([{"method": "StaticPairList"}, - {"method": "PerformanceFilter", "seed": 42}], + {"method": "PerformanceFilter"}], "BTC", ['ETH/BTC', 'TKN/BTC', 'HOT/BTC']), # Order matches order of appearance in whitelist_conf > exchange > pair_whitelist # PerformanceFilter only - ([{"method": "PerformanceFilter", "seed": 42}], + ([{"method": "PerformanceFilter"}], "BTC", 'filter_at_the_beginning'), # OperationalException expected # SpreadFilter after StaticPairList ([{"method": "StaticPairList"}, From 662ec3207310dd0c269e1a5bc4554eefec1891a6 Mon Sep 17 00:00:00 2001 From: Leif Segen Date: Sat, 28 Nov 2020 01:15:36 -0600 Subject: [PATCH 14/32] Add test cases --- freqtrade/pairlist/PerformanceFilter.py | 4 +- tests/pairlist/test_pairlist.py | 65 ++++++++++++++++--------- 2 files changed, 44 insertions(+), 25 deletions(-) diff --git a/freqtrade/pairlist/PerformanceFilter.py b/freqtrade/pairlist/PerformanceFilter.py index d4bd5936d..b2889dc6b 100644 --- a/freqtrade/pairlist/PerformanceFilter.py +++ b/freqtrade/pairlist/PerformanceFilter.py @@ -45,10 +45,10 @@ class PerformanceFilter(IPairList): """ # Get the trading performance for pairs from database perf = pd.DataFrame(Trade.get_overall_performance()) - # update pairlist with values from performance dataframe + # get pairlist from performance dataframe values + list_df = pd.DataFrame({'pair':pairlist}) # set initial value for pairs with no trades to 0 # and sort the list using performance and count - list_df = pd.DataFrame({'pair':pairlist}) sorted_df = list_df.join(perf.set_index('pair'), on='pair')\ .fillna(0).sort_values(by=['profit', 'count'], ascending=False) pairlist = sorted_df['pair'].tolist() diff --git a/tests/pairlist/test_pairlist.py b/tests/pairlist/test_pairlist.py index ae80a3975..a99651727 100644 --- a/tests/pairlist/test_pairlist.py +++ b/tests/pairlist/test_pairlist.py @@ -662,29 +662,48 @@ def test_pairlistmanager_no_pairlist(mocker, whitelist_conf): get_patched_freqtradebot(mocker, whitelist_conf) -# @pytest.mark.parametrize("pairlists,base_currency,overall_performance,expected", [ -# # Happy path, descening order, all values filled -# ([{"method": "StaticPairList"},{"method": "PerformanceFilter"}],'BTC',[{'pair':'ETH/BTC','profit':5,'count':3}, {'pair':'ETC/BTC','profit':4,'count':2}],['ETC/BTC']), -# ]) -# def test_performance_filter(mocker, whitelist_conf, base_currency, pairlists, overall_performance, expected, tickers, markets, ohlcv_history_list): -# whitelist_conf['pairlists'] = pairlists -# whitelist_conf['stake_currency'] = base_currency +@pytest.mark.parametrize("pairlists,pair_allowlist,overall_performance,allowlist_result", [ + # Happy path, descending order, all values filled + ([{"method": "StaticPairList"},{"method": "PerformanceFilter"}], + ['ETH/BTC','TKN/BTC'], + [{'pair':'TKN/BTC','profit':5,'count':3}, {'pair':'ETH/BTC','profit':4,'count':2}], + ['TKN/BTC','ETH/BTC']), + # Performance data outside allow list ignored + ([{"method": "StaticPairList"},{"method": "PerformanceFilter"}], + ['ETH/BTC','TKN/BTC'], + [{'pair':'OTHER/BTC','profit':5,'count':3}, {'pair':'ETH/BTC','profit':4,'count':2}], + ['ETH/BTC','TKN/BTC']), + # Partial performance data missing and sorted between positive and negative profit + ([{"method": "StaticPairList"},{"method": "PerformanceFilter"}], + ['ETH/BTC','TKN/BTC','LTC/BTC'], + [{'pair':'ETH/BTC','profit':-5,'count':100}, {'pair':'TKN/BTC','profit':4,'count':2}], + ['TKN/BTC','LTC/BTC','ETH/BTC']), + # Tie in performance data broken by count + ([{"method": "StaticPairList"},{"method": "PerformanceFilter"}], + ['ETH/BTC','TKN/BTC','LTC/BTC'], + [{'pair':'LTC/BTC','profit':-5,'count':101}, {'pair':'TKN/BTC','profit':-5,'count':2}, {'pair':'ETH/BTC','profit':-5,'count':100}, ], + ['LTC/BTC','ETH/BTC','TKN/BTC']), +]) +def test_performance_filter(mocker, whitelist_conf, pairlists, pair_allowlist, overall_performance, allowlist_result, tickers, markets, ohlcv_history_list): + allowlist_conf = whitelist_conf + allowlist_conf['pairlists'] = pairlists + allowlist_conf['exchange']['pair_whitelist'] = pair_allowlist -# mocker.patch('freqtrade.exchange.Exchange.exchange_has', MagicMock(return_value=True)) + mocker.patch('freqtrade.exchange.Exchange.exchange_has', MagicMock(return_value=True)) -# freqtrade = get_patched_freqtradebot(mocker, whitelist_conf) -# mocker.patch.multiple('freqtrade.exchange.Exchange', -# get_tickers=tickers, -# markets=PropertyMock(return_value=markets) -# ) -# mocker.patch.multiple( -# 'freqtrade.exchange.Exchange', -# get_historic_ohlcv=MagicMock(return_value=ohlcv_history_list), -# ) + freqtrade = get_patched_freqtradebot(mocker, allowlist_conf) + mocker.patch.multiple('freqtrade.exchange.Exchange', + get_tickers=tickers, + markets=PropertyMock(return_value=markets) + ) + mocker.patch.multiple( + 'freqtrade.exchange.Exchange', + get_historic_ohlcv=MagicMock(return_value=ohlcv_history_list), + ) -# mocker.patch.multiple('freqtrade.persistence.Trade', -# get_overall_performance=MagicMock(return_value=overall_performance), -# ) -# freqtrade.pairlists.refresh_pairlist() -# whitelist = freqtrade.pairlists.whitelist -# assert whitelist == expected + mocker.patch.multiple('freqtrade.persistence.Trade', + get_overall_performance=MagicMock(return_value=overall_performance), + ) + freqtrade.pairlists.refresh_pairlist() + allowlist = freqtrade.pairlists.whitelist + assert allowlist == allowlist_result From dbd50fdff64e5e72a052db687082b26794790f7b Mon Sep 17 00:00:00 2001 From: Leif Segen Date: Sat, 28 Nov 2020 01:22:03 -0600 Subject: [PATCH 15/32] Document filter. --- docs/includes/pairlists.md | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/docs/includes/pairlists.md b/docs/includes/pairlists.md index f8b33b27d..50ef52653 100644 --- a/docs/includes/pairlists.md +++ b/docs/includes/pairlists.md @@ -76,7 +76,12 @@ This filter allows freqtrade to ignore pairs until they have been listed for at #### PerformanceFilter -Lorem ipsum. +Sorts pairs by performance, as follows: +1. Positive performance. +2. No closed trades yet. +3. Negative performance. + +Trade count is used as a tie breaker. #### PrecisionFilter From 966c6b308f182392c85a74568350de4ac5cd9ced Mon Sep 17 00:00:00 2001 From: Leif Segen Date: Sat, 28 Nov 2020 01:34:18 -0600 Subject: [PATCH 16/32] Satisfy linter. --- freqtrade/pairlist/PerformanceFilter.py | 6 +-- tests/pairlist/test_pairlist.py | 51 +++++++++++++------------ 2 files changed, 30 insertions(+), 27 deletions(-) diff --git a/freqtrade/pairlist/PerformanceFilter.py b/freqtrade/pairlist/PerformanceFilter.py index b2889dc6b..099b8d271 100644 --- a/freqtrade/pairlist/PerformanceFilter.py +++ b/freqtrade/pairlist/PerformanceFilter.py @@ -12,6 +12,7 @@ from freqtrade.persistence import Trade logger = logging.getLogger(__name__) + class PerformanceFilter(IPairList): def __init__(self, exchange, pairlistmanager, @@ -19,7 +20,6 @@ class PerformanceFilter(IPairList): pairlist_pos: int) -> None: super().__init__(exchange, pairlistmanager, config, pairlistconfig, pairlist_pos) - @property def needstickers(self) -> bool: """ @@ -46,11 +46,11 @@ class PerformanceFilter(IPairList): # Get the trading performance for pairs from database perf = pd.DataFrame(Trade.get_overall_performance()) # get pairlist from performance dataframe values - list_df = pd.DataFrame({'pair':pairlist}) + list_df = pd.DataFrame({'pair': pairlist}) # set initial value for pairs with no trades to 0 # and sort the list using performance and count sorted_df = list_df.join(perf.set_index('pair'), on='pair')\ .fillna(0).sort_values(by=['profit', 'count'], ascending=False) pairlist = sorted_df['pair'].tolist() - return pairlist \ No newline at end of file + return pairlist diff --git a/tests/pairlist/test_pairlist.py b/tests/pairlist/test_pairlist.py index 65a0fa835..9e2bab12c 100644 --- a/tests/pairlist/test_pairlist.py +++ b/tests/pairlist/test_pairlist.py @@ -729,27 +729,32 @@ def test_pairlistmanager_no_pairlist(mocker, whitelist_conf): @pytest.mark.parametrize("pairlists,pair_allowlist,overall_performance,allowlist_result", [ # Happy path, descending order, all values filled - ([{"method": "StaticPairList"},{"method": "PerformanceFilter"}], - ['ETH/BTC','TKN/BTC'], - [{'pair':'TKN/BTC','profit':5,'count':3}, {'pair':'ETH/BTC','profit':4,'count':2}], - ['TKN/BTC','ETH/BTC']), + ([{"method": "StaticPairList"}, {"method": "PerformanceFilter"}], + ['ETH/BTC', 'TKN/BTC'], + [{'pair': 'TKN/BTC', 'profit': 5, 'count': 3}, {'pair': 'ETH/BTC', 'profit': 4, 'count': 2}], + ['TKN/BTC', 'ETH/BTC']), # Performance data outside allow list ignored - ([{"method": "StaticPairList"},{"method": "PerformanceFilter"}], - ['ETH/BTC','TKN/BTC'], - [{'pair':'OTHER/BTC','profit':5,'count':3}, {'pair':'ETH/BTC','profit':4,'count':2}], - ['ETH/BTC','TKN/BTC']), + ([{"method": "StaticPairList"}, {"method": "PerformanceFilter"}], + ['ETH/BTC', 'TKN/BTC'], + [{'pair': 'OTHER/BTC', 'profit': 5, 'count': 3}, + {'pair': 'ETH/BTC', 'profit': 4, 'count': 2}], + ['ETH/BTC', 'TKN/BTC']), # Partial performance data missing and sorted between positive and negative profit - ([{"method": "StaticPairList"},{"method": "PerformanceFilter"}], - ['ETH/BTC','TKN/BTC','LTC/BTC'], - [{'pair':'ETH/BTC','profit':-5,'count':100}, {'pair':'TKN/BTC','profit':4,'count':2}], - ['TKN/BTC','LTC/BTC','ETH/BTC']), + ([{"method": "StaticPairList"}, {"method": "PerformanceFilter"}], + ['ETH/BTC', 'TKN/BTC', 'LTC/BTC'], + [{'pair': 'ETH/BTC', 'profit': -5, 'count': 100}, + {'pair': 'TKN/BTC', 'profit': 4, 'count': 2}], + ['TKN/BTC', 'LTC/BTC', 'ETH/BTC']), # Tie in performance data broken by count - ([{"method": "StaticPairList"},{"method": "PerformanceFilter"}], - ['ETH/BTC','TKN/BTC','LTC/BTC'], - [{'pair':'LTC/BTC','profit':-5,'count':101}, {'pair':'TKN/BTC','profit':-5,'count':2}, {'pair':'ETH/BTC','profit':-5,'count':100}, ], - ['LTC/BTC','ETH/BTC','TKN/BTC']), + ([{"method": "StaticPairList"}, {"method": "PerformanceFilter"}], + ['ETH/BTC', 'TKN/BTC', 'LTC/BTC'], + [{'pair': 'LTC/BTC', 'profit': -5, 'count': 101}, + {'pair': 'TKN/BTC', 'profit': -5, 'count': 2}, + {'pair': 'ETH/BTC', 'profit': -5, 'count': 100}], + ['LTC/BTC', 'ETH/BTC', 'TKN/BTC']), ]) -def test_performance_filter(mocker, whitelist_conf, pairlists, pair_allowlist, overall_performance, allowlist_result, tickers, markets, ohlcv_history_list): +def test_performance_filter(mocker, whitelist_conf, pairlists, pair_allowlist, overall_performance, + allowlist_result, tickers, markets, ohlcv_history_list): allowlist_conf = whitelist_conf allowlist_conf['pairlists'] = pairlists allowlist_conf['exchange']['pair_whitelist'] = pair_allowlist @@ -761,14 +766,12 @@ def test_performance_filter(mocker, whitelist_conf, pairlists, pair_allowlist, o get_tickers=tickers, markets=PropertyMock(return_value=markets) ) - mocker.patch.multiple( - 'freqtrade.exchange.Exchange', - get_historic_ohlcv=MagicMock(return_value=ohlcv_history_list), - ) - + mocker.patch.multiple('freqtrade.exchange.Exchange', + get_historic_ohlcv=MagicMock(return_value=ohlcv_history_list), + ) mocker.patch.multiple('freqtrade.persistence.Trade', - get_overall_performance=MagicMock(return_value=overall_performance), - ) + get_overall_performance=MagicMock(return_value=overall_performance), + ) freqtrade.pairlists.refresh_pairlist() allowlist = freqtrade.pairlists.whitelist assert allowlist == allowlist_result From fefa500963d7d4a8b88d740783190755605942ac Mon Sep 17 00:00:00 2001 From: Leif Segen Date: Sat, 28 Nov 2020 01:34:40 -0600 Subject: [PATCH 17/32] More lint --- tests/pairlist/test_pairlist.py | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/tests/pairlist/test_pairlist.py b/tests/pairlist/test_pairlist.py index 9e2bab12c..5e9847e3d 100644 --- a/tests/pairlist/test_pairlist.py +++ b/tests/pairlist/test_pairlist.py @@ -333,7 +333,7 @@ def test_VolumePairList_refresh_empty(mocker, markets_empty, whitelist_conf): # PerformanceFilter after StaticPairList ([{"method": "StaticPairList"}, {"method": "PerformanceFilter"}], - "BTC", ['ETH/BTC', 'TKN/BTC', 'HOT/BTC']), # Order matches order of appearance in whitelist_conf > exchange > pair_whitelist + "BTC", ['ETH/BTC', 'TKN/BTC', 'HOT/BTC']), # PerformanceFilter only ([{"method": "PerformanceFilter"}], "BTC", 'filter_at_the_beginning'), # OperationalException expected @@ -383,7 +383,8 @@ def test_VolumePairList_whitelist_gen(mocker, whitelist_conf, shitcoinmarkets, t # Provide for PerformanceFilter's dependency mocker.patch.multiple('freqtrade.persistence.Trade', - get_overall_performance=MagicMock(return_value=[{'pair':'ETH/BTC','profit':5,'count':3}]), + get_overall_performance=MagicMock( + return_value=[{'pair': 'ETH/BTC', 'profit': 5, 'count' :3}]), ) # Set whitelist_result to None if pairlist is invalid and should produce exception From ecce5265f5e5fa153261095c563e505c243fc0a7 Mon Sep 17 00:00:00 2001 From: Leif Segen Date: Sat, 28 Nov 2020 01:43:19 -0600 Subject: [PATCH 18/32] Linting --- tests/pairlist/test_pairlist.py | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/tests/pairlist/test_pairlist.py b/tests/pairlist/test_pairlist.py index 5e9847e3d..a4df031c9 100644 --- a/tests/pairlist/test_pairlist.py +++ b/tests/pairlist/test_pairlist.py @@ -383,8 +383,8 @@ def test_VolumePairList_whitelist_gen(mocker, whitelist_conf, shitcoinmarkets, t # Provide for PerformanceFilter's dependency mocker.patch.multiple('freqtrade.persistence.Trade', - get_overall_performance=MagicMock( - return_value=[{'pair': 'ETH/BTC', 'profit': 5, 'count' :3}]), + get_overall_performance=MagicMock(return_value=\ + [{'pair': 'ETH/BTC', 'profit': 5, 'count': 3}]), ) # Set whitelist_result to None if pairlist is invalid and should produce exception @@ -737,7 +737,7 @@ def test_pairlistmanager_no_pairlist(mocker, whitelist_conf): # Performance data outside allow list ignored ([{"method": "StaticPairList"}, {"method": "PerformanceFilter"}], ['ETH/BTC', 'TKN/BTC'], - [{'pair': 'OTHER/BTC', 'profit': 5, 'count': 3}, + [{'pair': 'OTHER/BTC', 'profit': 5, 'count': 3}, {'pair': 'ETH/BTC', 'profit': 4, 'count': 2}], ['ETH/BTC', 'TKN/BTC']), # Partial performance data missing and sorted between positive and negative profit @@ -769,7 +769,7 @@ def test_performance_filter(mocker, whitelist_conf, pairlists, pair_allowlist, o ) mocker.patch.multiple('freqtrade.exchange.Exchange', get_historic_ohlcv=MagicMock(return_value=ohlcv_history_list), - ) + ) mocker.patch.multiple('freqtrade.persistence.Trade', get_overall_performance=MagicMock(return_value=overall_performance), ) From f448564073b2d7c487ccb2d75e749bfe949bd547 Mon Sep 17 00:00:00 2001 From: Leif Segen Date: Sat, 28 Nov 2020 01:49:46 -0600 Subject: [PATCH 19/32] Lint --- tests/pairlist/test_pairlist.py | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/tests/pairlist/test_pairlist.py b/tests/pairlist/test_pairlist.py index a4df031c9..d40cece41 100644 --- a/tests/pairlist/test_pairlist.py +++ b/tests/pairlist/test_pairlist.py @@ -383,9 +383,9 @@ def test_VolumePairList_whitelist_gen(mocker, whitelist_conf, shitcoinmarkets, t # Provide for PerformanceFilter's dependency mocker.patch.multiple('freqtrade.persistence.Trade', - get_overall_performance=MagicMock(return_value=\ - [{'pair': 'ETH/BTC', 'profit': 5, 'count': 3}]), - ) + get_overall_performance=MagicMock( + return_value=[{'pair': 'ETH/BTC', 'profit': 5, 'count': 3}]), + ) # Set whitelist_result to None if pairlist is invalid and should produce exception if whitelist_result == 'filter_at_the_beginning': From 37d2e476df19bfafa86c6405404d0bc578f270d9 Mon Sep 17 00:00:00 2001 From: Leif Segen Date: Sat, 28 Nov 2020 01:59:30 -0600 Subject: [PATCH 20/32] isort imports --- freqtrade/pairlist/PerformanceFilter.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/freqtrade/pairlist/PerformanceFilter.py b/freqtrade/pairlist/PerformanceFilter.py index 099b8d271..bd56a4607 100644 --- a/freqtrade/pairlist/PerformanceFilter.py +++ b/freqtrade/pairlist/PerformanceFilter.py @@ -7,9 +7,9 @@ from typing import Any, Dict, List import pandas as pd from freqtrade.pairlist.IPairList import IPairList - from freqtrade.persistence import Trade + logger = logging.getLogger(__name__) From e1d42ba78ce670a662420b7bb19337c57cc335ca Mon Sep 17 00:00:00 2001 From: Leif Segen Date: Sat, 28 Nov 2020 09:44:01 -0600 Subject: [PATCH 21/32] Alphabetize --- freqtrade/constants.py | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/freqtrade/constants.py b/freqtrade/constants.py index 20cc70d2e..9d0078d21 100644 --- a/freqtrade/constants.py +++ b/freqtrade/constants.py @@ -24,9 +24,9 @@ HYPEROPT_LOSS_BUILTIN = ['ShortTradeDurHyperOptLoss', 'OnlyProfitHyperOptLoss', 'SharpeHyperOptLoss', 'SharpeHyperOptLossDaily', 'SortinoHyperOptLoss', 'SortinoHyperOptLossDaily'] AVAILABLE_PAIRLISTS = ['StaticPairList', 'VolumePairList', - 'AgeFilter', 'PrecisionFilter', 'PriceFilter', - 'RangeStabilityFilter', 'ShuffleFilter', 'SpreadFilter', - 'PerformanceFilter'] + 'AgeFilter', 'PerformanceFilter', 'PrecisionFilter', + 'PriceFilter', 'RangeStabilityFilter', 'ShuffleFilter', + 'SpreadFilter'] AVAILABLE_DATAHANDLERS = ['json', 'jsongz', 'hdf5'] DRY_RUN_WALLET = 1000 DATETIME_PRINT_FORMAT = '%Y-%m-%d %H:%M:%S' From 03c5714399d57af12cb18ffa7f4b6937ed3cd6b4 Mon Sep 17 00:00:00 2001 From: Leif Segen Date: Sat, 28 Nov 2020 09:45:17 -0600 Subject: [PATCH 22/32] Use explicit merge without depending on library detail. Add no trades case. --- freqtrade/pairlist/PerformanceFilter.py | 9 +++++++-- tests/pairlist/test_pairlist.py | 8 +++++--- 2 files changed, 12 insertions(+), 5 deletions(-) diff --git a/freqtrade/pairlist/PerformanceFilter.py b/freqtrade/pairlist/PerformanceFilter.py index bd56a4607..2d360a346 100644 --- a/freqtrade/pairlist/PerformanceFilter.py +++ b/freqtrade/pairlist/PerformanceFilter.py @@ -44,12 +44,17 @@ class PerformanceFilter(IPairList): :return: new whitelist """ # Get the trading performance for pairs from database - perf = pd.DataFrame(Trade.get_overall_performance()) + performance = pd.DataFrame(Trade.get_overall_performance()) + + # Skip performance-based sorting if no performance data is available + if len(performance) == 0: + return pairlist + # get pairlist from performance dataframe values list_df = pd.DataFrame({'pair': pairlist}) # set initial value for pairs with no trades to 0 # and sort the list using performance and count - sorted_df = list_df.join(perf.set_index('pair'), on='pair')\ + sorted_df = list_df.merge(performance, on='pair', how='left')\ .fillna(0).sort_values(by=['profit', 'count'], ascending=False) pairlist = sorted_df['pair'].tolist() diff --git a/tests/pairlist/test_pairlist.py b/tests/pairlist/test_pairlist.py index d40cece41..c62ec81f3 100644 --- a/tests/pairlist/test_pairlist.py +++ b/tests/pairlist/test_pairlist.py @@ -383,8 +383,7 @@ def test_VolumePairList_whitelist_gen(mocker, whitelist_conf, shitcoinmarkets, t # Provide for PerformanceFilter's dependency mocker.patch.multiple('freqtrade.persistence.Trade', - get_overall_performance=MagicMock( - return_value=[{'pair': 'ETH/BTC', 'profit': 5, 'count': 3}]), + get_overall_performance=MagicMock(return_value=[]) ) # Set whitelist_result to None if pairlist is invalid and should produce exception @@ -729,7 +728,10 @@ def test_pairlistmanager_no_pairlist(mocker, whitelist_conf): @pytest.mark.parametrize("pairlists,pair_allowlist,overall_performance,allowlist_result", [ - # Happy path, descending order, all values filled + # No trades yet + ([{"method": "StaticPairList"}, {"method": "PerformanceFilter"}], + ['ETH/BTC', 'TKN/BTC', 'LTC/BTC'], [], ['ETH/BTC', 'TKN/BTC', 'LTC/BTC']), + # Happy path: Descending order, all values filled ([{"method": "StaticPairList"}, {"method": "PerformanceFilter"}], ['ETH/BTC', 'TKN/BTC'], [{'pair': 'TKN/BTC', 'profit': 5, 'count': 3}, {'pair': 'ETH/BTC', 'profit': 4, 'count': 2}], From 6a74c57c3d5c4ab422ced69643459c70269981ab Mon Sep 17 00:00:00 2001 From: Leif Segen Date: Sat, 28 Nov 2020 11:33:25 -0600 Subject: [PATCH 23/32] Pair name-based sorting. Attempt at more rational string sorting. Change test to show not working as expected. --- freqtrade/pairlist/PerformanceFilter.py | 2 +- tests/pairlist/test_pairlist.py | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/freqtrade/pairlist/PerformanceFilter.py b/freqtrade/pairlist/PerformanceFilter.py index 2d360a346..5e1ec3c66 100644 --- a/freqtrade/pairlist/PerformanceFilter.py +++ b/freqtrade/pairlist/PerformanceFilter.py @@ -55,7 +55,7 @@ class PerformanceFilter(IPairList): # set initial value for pairs with no trades to 0 # and sort the list using performance and count sorted_df = list_df.merge(performance, on='pair', how='left')\ - .fillna(0).sort_values(by=['profit', 'count'], ascending=False) + .fillna(0).sort_values(by=['profit', 'count', 'pair'], ascending=False) pairlist = sorted_df['pair'].tolist() return pairlist diff --git a/tests/pairlist/test_pairlist.py b/tests/pairlist/test_pairlist.py index c62ec81f3..475691327 100644 --- a/tests/pairlist/test_pairlist.py +++ b/tests/pairlist/test_pairlist.py @@ -305,7 +305,7 @@ def test_VolumePairList_refresh_empty(mocker, markets_empty, whitelist_conf): # PerformanceFilter ([{"method": "VolumePairList", "number_assets": 5, "sort_key": "quoteVolume"}, {"method": "PerformanceFilter"}], - "USDT", ['ETH/USDT', 'NANO/USDT', 'ADAHALF/USDT', 'ADADOUBLE/USDT']), + "USDT", ['ETH/USDT', 'NANO/USDT', 'ADADOUBLE/USDT', 'ADAHALF/USDT']), # AgeFilter only ([{"method": "AgeFilter", "min_days_listed": 2}], "BTC", 'filter_at_the_beginning'), # OperationalException expected From 323c0657f8a80b0af14d0ff18920f431f66af7a0 Mon Sep 17 00:00:00 2001 From: Leif Segen Date: Sat, 28 Nov 2020 12:17:03 -0600 Subject: [PATCH 24/32] Sort by profit after sort by count/pair --- freqtrade/pairlist/PerformanceFilter.py | 21 +++++++++++++-------- tests/pairlist/test_pairlist.py | 17 ++++++++++++----- 2 files changed, 25 insertions(+), 13 deletions(-) diff --git a/freqtrade/pairlist/PerformanceFilter.py b/freqtrade/pairlist/PerformanceFilter.py index 5e1ec3c66..cdc3c78ad 100644 --- a/freqtrade/pairlist/PerformanceFilter.py +++ b/freqtrade/pairlist/PerformanceFilter.py @@ -31,17 +31,17 @@ class PerformanceFilter(IPairList): def short_desc(self) -> str: """ - Short whitelist method description - used for startup-messages + Short allowlist method description - used for startup-messages """ return f"{self.name} - Sorting pairs by performance." def filter_pairlist(self, pairlist: List[str], tickers: Dict) -> List[str]: """ - Filters and sorts pairlist and returns the whitelist again. + Filters and sorts pairlist and returns the allowlist again. Called on each bot iteration - please use internal caching if necessary :param pairlist: pairlist to filter or sort :param tickers: Tickers (from exchange.get_tickers()). May be cached. - :return: new whitelist + :return: new allowlist """ # Get the trading performance for pairs from database performance = pd.DataFrame(Trade.get_overall_performance()) @@ -49,13 +49,18 @@ class PerformanceFilter(IPairList): # Skip performance-based sorting if no performance data is available if len(performance) == 0: return pairlist - - # get pairlist from performance dataframe values + + # Get pairlist from performance dataframe values list_df = pd.DataFrame({'pair': pairlist}) - # set initial value for pairs with no trades to 0 - # and sort the list using performance and count + + # Set initial value for pairs with no trades to 0 + # Sort the list using: + # - primarily performance (high to low) + # - then count (low to high, so as to favor same performance with fewer trades) + # - then pair name alphametically sorted_df = list_df.merge(performance, on='pair', how='left')\ - .fillna(0).sort_values(by=['profit', 'count', 'pair'], ascending=False) + .fillna(0).sort_values(by=['count', 'pair'], ascending=True)\ + .sort_values(by=['profit'], ascending=False) pairlist = sorted_df['pair'].tolist() return pairlist diff --git a/tests/pairlist/test_pairlist.py b/tests/pairlist/test_pairlist.py index 475691327..244f92d8b 100644 --- a/tests/pairlist/test_pairlist.py +++ b/tests/pairlist/test_pairlist.py @@ -748,13 +748,20 @@ def test_pairlistmanager_no_pairlist(mocker, whitelist_conf): [{'pair': 'ETH/BTC', 'profit': -5, 'count': 100}, {'pair': 'TKN/BTC', 'profit': 4, 'count': 2}], ['TKN/BTC', 'LTC/BTC', 'ETH/BTC']), - # Tie in performance data broken by count + # Tie in performance data broken by count (ascending) ([{"method": "StaticPairList"}, {"method": "PerformanceFilter"}], ['ETH/BTC', 'TKN/BTC', 'LTC/BTC'], - [{'pair': 'LTC/BTC', 'profit': -5, 'count': 101}, - {'pair': 'TKN/BTC', 'profit': -5, 'count': 2}, - {'pair': 'ETH/BTC', 'profit': -5, 'count': 100}], - ['LTC/BTC', 'ETH/BTC', 'TKN/BTC']), + [{'pair': 'LTC/BTC', 'profit': -5.01, 'count': 101}, + {'pair': 'TKN/BTC', 'profit': -5.01, 'count': 2}, + {'pair': 'ETH/BTC', 'profit': -5.01, 'count': 100}], + ['TKN/BTC', 'ETH/BTC', 'LTC/BTC']), + # Tie in performance and count, broken by alphabetical sort + ([{"method": "StaticPairList"}, {"method": "PerformanceFilter"}], + ['ETH/BTC', 'TKN/BTC', 'LTC/BTC'], + [{'pair': 'LTC/BTC', 'profit': -5.01, 'count': 1}, + {'pair': 'TKN/BTC', 'profit': -5.01, 'count': 1}, + {'pair': 'ETH/BTC', 'profit': -5.01, 'count': 1}], + ['ETH/BTC', 'LTC/BTC', 'TKN/BTC']), ]) def test_performance_filter(mocker, whitelist_conf, pairlists, pair_allowlist, overall_performance, allowlist_result, tickers, markets, ohlcv_history_list): From d6c93919246a7e1e250f2934b3da7417a292118b Mon Sep 17 00:00:00 2001 From: Leif Segen Date: Sat, 28 Nov 2020 12:18:23 -0600 Subject: [PATCH 25/32] Restoring expectation --- tests/pairlist/test_pairlist.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/pairlist/test_pairlist.py b/tests/pairlist/test_pairlist.py index 244f92d8b..4b4f51b37 100644 --- a/tests/pairlist/test_pairlist.py +++ b/tests/pairlist/test_pairlist.py @@ -305,7 +305,7 @@ def test_VolumePairList_refresh_empty(mocker, markets_empty, whitelist_conf): # PerformanceFilter ([{"method": "VolumePairList", "number_assets": 5, "sort_key": "quoteVolume"}, {"method": "PerformanceFilter"}], - "USDT", ['ETH/USDT', 'NANO/USDT', 'ADADOUBLE/USDT', 'ADAHALF/USDT']), + "USDT", ['ETH/USDT', 'NANO/USDT', 'ADAHALF/USDT', 'ADADOUBLE/USDT']), # AgeFilter only ([{"method": "AgeFilter", "min_days_listed": 2}], "BTC", 'filter_at_the_beginning'), # OperationalException expected From e7a035eefe3bdcaeca5688051b778f7b48788505 Mon Sep 17 00:00:00 2001 From: Leif Segen Date: Sat, 28 Nov 2020 12:29:31 -0600 Subject: [PATCH 26/32] Lint --- freqtrade/pairlist/PerformanceFilter.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/freqtrade/pairlist/PerformanceFilter.py b/freqtrade/pairlist/PerformanceFilter.py index cdc3c78ad..92a97099e 100644 --- a/freqtrade/pairlist/PerformanceFilter.py +++ b/freqtrade/pairlist/PerformanceFilter.py @@ -49,7 +49,7 @@ class PerformanceFilter(IPairList): # Skip performance-based sorting if no performance data is available if len(performance) == 0: return pairlist - + # Get pairlist from performance dataframe values list_df = pd.DataFrame({'pair': pairlist}) @@ -60,7 +60,7 @@ class PerformanceFilter(IPairList): # - then pair name alphametically sorted_df = list_df.merge(performance, on='pair', how='left')\ .fillna(0).sort_values(by=['count', 'pair'], ascending=True)\ - .sort_values(by=['profit'], ascending=False) + .sort_values(by=['profit'], ascending=False) pairlist = sorted_df['pair'].tolist() return pairlist From 4b6f5b92b59e21896035c2f47475a3fcdf07c377 Mon Sep 17 00:00:00 2001 From: Leif Segen Date: Sat, 28 Nov 2020 12:47:36 -0600 Subject: [PATCH 27/32] Remove non-pertinent test case --- tests/pairlist/test_pairlist.py | 4 ---- 1 file changed, 4 deletions(-) diff --git a/tests/pairlist/test_pairlist.py b/tests/pairlist/test_pairlist.py index 4b4f51b37..1d2f16b45 100644 --- a/tests/pairlist/test_pairlist.py +++ b/tests/pairlist/test_pairlist.py @@ -302,10 +302,6 @@ def test_VolumePairList_refresh_empty(mocker, markets_empty, whitelist_conf): ([{"method": "VolumePairList", "number_assets": 5, "sort_key": "quoteVolume"}, {"method": "ShuffleFilter"}], "USDT", 3), # whitelist_result is integer -- check only length of randomized pairlist - # PerformanceFilter - ([{"method": "VolumePairList", "number_assets": 5, "sort_key": "quoteVolume"}, - {"method": "PerformanceFilter"}], - "USDT", ['ETH/USDT', 'NANO/USDT', 'ADAHALF/USDT', 'ADADOUBLE/USDT']), # AgeFilter only ([{"method": "AgeFilter", "min_days_listed": 2}], "BTC", 'filter_at_the_beginning'), # OperationalException expected From 1791495475e1e0bbab0b820a065b627b92f28f7d Mon Sep 17 00:00:00 2001 From: Leif Segen Date: Sat, 28 Nov 2020 16:50:44 -0600 Subject: [PATCH 28/32] Trigger another run of tests --- tests/pairlist/test_pairlist.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/pairlist/test_pairlist.py b/tests/pairlist/test_pairlist.py index 1d2f16b45..884be3c24 100644 --- a/tests/pairlist/test_pairlist.py +++ b/tests/pairlist/test_pairlist.py @@ -332,7 +332,7 @@ def test_VolumePairList_refresh_empty(mocker, markets_empty, whitelist_conf): "BTC", ['ETH/BTC', 'TKN/BTC', 'HOT/BTC']), # PerformanceFilter only ([{"method": "PerformanceFilter"}], - "BTC", 'filter_at_the_beginning'), # OperationalException expected + "BTC", 'filter_at_the_beginning'), # OperationalException expected # SpreadFilter after StaticPairList ([{"method": "StaticPairList"}, {"method": "SpreadFilter", "max_spread_ratio": 0.005}], From 90070f0dc54cd2ea92268ce75240ff3972523666 Mon Sep 17 00:00:00 2001 From: Leif Segen Date: Sat, 28 Nov 2020 17:17:40 -0600 Subject: [PATCH 29/32] Force test rerun --- tests/pairlist/test_pairlist.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/pairlist/test_pairlist.py b/tests/pairlist/test_pairlist.py index 884be3c24..1d2f16b45 100644 --- a/tests/pairlist/test_pairlist.py +++ b/tests/pairlist/test_pairlist.py @@ -332,7 +332,7 @@ def test_VolumePairList_refresh_empty(mocker, markets_empty, whitelist_conf): "BTC", ['ETH/BTC', 'TKN/BTC', 'HOT/BTC']), # PerformanceFilter only ([{"method": "PerformanceFilter"}], - "BTC", 'filter_at_the_beginning'), # OperationalException expected + "BTC", 'filter_at_the_beginning'), # OperationalException expected # SpreadFilter after StaticPairList ([{"method": "StaticPairList"}, {"method": "SpreadFilter", "max_spread_ratio": 0.005}], From 5f8e67d2b25c40454343414653e02b48fd4518d8 Mon Sep 17 00:00:00 2001 From: Leif Segen Date: Sun, 29 Nov 2020 05:05:54 -0600 Subject: [PATCH 30/32] Update docs/includes/pairlists.md Co-authored-by: Matthias --- docs/includes/pairlists.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/includes/pairlists.md b/docs/includes/pairlists.md index a1bbebbf7..844f1d70a 100644 --- a/docs/includes/pairlists.md +++ b/docs/includes/pairlists.md @@ -77,7 +77,7 @@ This filter allows freqtrade to ignore pairs until they have been listed for at #### PerformanceFilter -Sorts pairs by performance, as follows: +Sorts pairs by past trade performance, as follows: 1. Positive performance. 2. No closed trades yet. 3. Negative performance. From 99abe52043ab7b7f54e79fd75c1c80831eafebf5 Mon Sep 17 00:00:00 2001 From: Leif Segen Date: Sun, 29 Nov 2020 10:30:02 -0600 Subject: [PATCH 31/32] Trigger CI --- tests/pairlist/test_pairlist.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/pairlist/test_pairlist.py b/tests/pairlist/test_pairlist.py index 1d2f16b45..1f434ae34 100644 --- a/tests/pairlist/test_pairlist.py +++ b/tests/pairlist/test_pairlist.py @@ -326,7 +326,7 @@ def test_VolumePairList_refresh_empty(mocker, markets_empty, whitelist_conf): # ShuffleFilter only ([{"method": "ShuffleFilter", "seed": 42}], "BTC", 'filter_at_the_beginning'), # OperationalException expected - # PerformanceFilter after StaticPairList + # PerformanceFilter after StaticPairList ([{"method": "StaticPairList"}, {"method": "PerformanceFilter"}], "BTC", ['ETH/BTC', 'TKN/BTC', 'HOT/BTC']), From b7de18608d8f977578e56d3c83b88fbbbb459f3e Mon Sep 17 00:00:00 2001 From: Leif Segen Date: Sun, 29 Nov 2020 10:30:43 -0600 Subject: [PATCH 32/32] Trigger CI --- tests/pairlist/test_pairlist.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/pairlist/test_pairlist.py b/tests/pairlist/test_pairlist.py index 1f434ae34..1d2f16b45 100644 --- a/tests/pairlist/test_pairlist.py +++ b/tests/pairlist/test_pairlist.py @@ -326,7 +326,7 @@ def test_VolumePairList_refresh_empty(mocker, markets_empty, whitelist_conf): # ShuffleFilter only ([{"method": "ShuffleFilter", "seed": 42}], "BTC", 'filter_at_the_beginning'), # OperationalException expected - # PerformanceFilter after StaticPairList + # PerformanceFilter after StaticPairList ([{"method": "StaticPairList"}, {"method": "PerformanceFilter"}], "BTC", ['ETH/BTC', 'TKN/BTC', 'HOT/BTC']),