diff --git a/.devcontainer/Dockerfile b/.devcontainer/Dockerfile new file mode 100644 index 000000000..b333dc19d --- /dev/null +++ b/.devcontainer/Dockerfile @@ -0,0 +1,18 @@ +FROM freqtradeorg/freqtrade:develop + +# Install dependencies +COPY requirements-dev.txt /freqtrade/ +RUN apt-get update \ + && apt-get -y install git sudo vim \ + && apt-get clean \ + && pip install autopep8 -r docs/requirements-docs.txt -r requirements-dev.txt --no-cache-dir \ + && useradd -u 1000 -U -m ftuser \ + && mkdir -p /home/ftuser/.vscode-server /home/ftuser/.vscode-server-insiders /home/ftuser/commandhistory \ + && echo "export PROMPT_COMMAND='history -a'" >> /home/ftuser/.bashrc \ + && echo "export HISTFILE=~/commandhistory/.bash_history" >> /home/ftuser/.bashrc \ + && chown ftuser: -R /home/ftuser/ + +USER ftuser + +# Empty the ENTRYPOINT to allow all commands +ENTRYPOINT [] diff --git a/.devcontainer/devcontainer.json b/.devcontainer/devcontainer.json new file mode 100644 index 000000000..1882e3bdf --- /dev/null +++ b/.devcontainer/devcontainer.json @@ -0,0 +1,44 @@ +{ + "name": "freqtrade Develop", + + "dockerComposeFile": [ + "docker-compose.yml" + ], + + "service": "ft_vscode", + + "workspaceFolder": "/freqtrade/", + + "settings": { + "terminal.integrated.shell.linux": "/bin/bash", + "editor.insertSpaces": true, + "files.trimTrailingWhitespace": true, + "[markdown]": { + "files.trimTrailingWhitespace": false, + }, + "python.pythonPath": "/usr/local/bin/python", + }, + + // Add the IDs of extensions you want installed when the container is created. + "extensions": [ + "ms-python.python", + "ms-python.vscode-pylance", + "davidanson.vscode-markdownlint", + "ms-azuretools.vscode-docker", + ], + + // Use 'forwardPorts' to make a list of ports inside the container available locally. + // "forwardPorts": [], + + // Uncomment the next line if you want start specific services in your Docker Compose config. + // "runServices": [], + + // Uncomment the next line if you want to keep your containers running after VS Code shuts down. + // "shutdownAction": "none", + + // Uncomment the next line to run commands after the container is created - for example installing curl. + // "postCreateCommand": "sudo apt-get update && apt-get install -y git", + + // Uncomment to connect as a non-root user if you've added one. See https://aka.ms/vscode-remote/containers/non-root. + "remoteUser": "ftuser" +} diff --git a/.devcontainer/docker-compose.yml b/.devcontainer/docker-compose.yml new file mode 100644 index 000000000..7b5e64609 --- /dev/null +++ b/.devcontainer/docker-compose.yml @@ -0,0 +1,24 @@ +--- +version: '3' +services: + ft_vscode: + build: + context: .. + dockerfile: ".devcontainer/Dockerfile" + volumes: + # Allow git usage within container + - "/home/${USER}/.ssh:/home/ftuser/.ssh:ro" + - "/home/${USER}/.gitconfig:/home/ftuser/.gitconfig:ro" + - ..:/freqtrade:cached + # Persist bash-history + - freqtrade-vscode-server:/home/ftuser/.vscode-server + - freqtrade-bashhistory:/home/ftuser/commandhistory + # Expose API port + ports: + - "127.0.0.1:8080:8080" + command: /bin/sh -c "while sleep 1000; do :; done" + + +volumes: + freqtrade-vscode-server: + freqtrade-bashhistory: diff --git a/.dockerignore b/.dockerignore index 223b3b110..09f4c9f0c 100644 --- a/.dockerignore +++ b/.dockerignore @@ -13,3 +13,4 @@ CONTRIBUTING.md MANIFEST.in README.md freqtrade.service +user_data diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 16bf1959d..f259129d4 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -4,8 +4,8 @@ on: push: branches: - master + - stable - develop - - github_actions_tests tags: release: types: [published] @@ -19,14 +19,14 @@ jobs: runs-on: ${{ matrix.os }} strategy: matrix: - os: [ ubuntu-18.04, macos-latest ] + os: [ ubuntu-18.04, ubuntu-20.04, macos-latest ] python-version: [3.7, 3.8] steps: - uses: actions/checkout@v2 - name: Set up Python - uses: actions/setup-python@v1 + uses: actions/setup-python@v2 with: python-version: ${{ matrix.python-version }} @@ -70,7 +70,7 @@ jobs: pytest --random-order --cov=freqtrade --cov-config=.coveragerc - name: Coveralls - if: (startsWith(matrix.os, 'ubuntu') && matrix.python-version == '3.8') + if: (startsWith(matrix.os, 'ubuntu-20') && matrix.python-version == '3.8') env: # Coveralls token. Not used as secret due to github not providing secrets to forked repositories COVERALLS_REPO_TOKEN: 6D1m0xupS3FgutfuGao8keFf9Hc0FpIXu @@ -88,12 +88,16 @@ jobs: run: | cp config.json.example config.json freqtrade create-userdir --userdir user_data - freqtrade hyperopt --datadir tests/testdata -e 5 --strategy SampleStrategy --hyperopt SampleHyperOpt --print-all + freqtrade hyperopt --datadir tests/testdata -e 5 --strategy SampleStrategy --hyperopt SampleHyperOpt --hyperopt-loss SharpeHyperOptLossDaily --print-all - name: Flake8 run: | flake8 + - name: Sort imports (isort) + run: | + isort --check . + - name: Mypy run: | mypy freqtrade scripts @@ -121,7 +125,7 @@ jobs: - uses: actions/checkout@v2 - name: Set up Python - uses: actions/setup-python@v1 + uses: actions/setup-python@v2 with: python-version: ${{ matrix.python-version }} @@ -150,7 +154,7 @@ jobs: run: | cp config.json.example config.json freqtrade create-userdir --userdir user_data - freqtrade hyperopt --datadir tests/testdata -e 5 --strategy SampleStrategy --hyperopt SampleHyperOpt --print-all + freqtrade hyperopt --datadir tests/testdata -e 5 --strategy SampleStrategy --hyperopt SampleHyperOpt --hyperopt-loss SharpeHyperOptLossDaily --print-all - name: Flake8 run: | @@ -172,7 +176,7 @@ jobs: url: ${{ secrets.SLACK_WEBHOOK }} docs_check: - runs-on: ubuntu-latest + runs-on: ubuntu-20.04 steps: - uses: actions/checkout@v2 @@ -180,6 +184,17 @@ jobs: run: | ./tests/test_docs.sh + - name: Set up Python + uses: actions/setup-python@v2 + with: + python-version: 3.8 + + - name: Documentation build + run: | + pip install -r docs/requirements-docs.txt + pip install mkdocs + mkdocs build + - name: Slack Notification uses: homoluctus/slatify@v1.8.0 if: failure() && ( github.event_name != 'pull_request' || github.event.pull_request.head.repo.fork == false) @@ -190,18 +205,18 @@ jobs: url: ${{ secrets.SLACK_WEBHOOK }} cleanup-prior-runs: - runs-on: ubuntu-latest + runs-on: ubuntu-20.04 steps: - name: Cleanup previous runs on this branch uses: rokroskar/workflow-run-cleanup-action@v0.2.2 - if: "!startsWith(github.ref, 'refs/tags/') && github.ref != 'refs/heads/master' && github.repository == 'freqtrade/freqtrade'" + if: "!startsWith(github.ref, 'refs/tags/') && github.ref != 'refs/heads/stable' && github.repository == 'freqtrade/freqtrade'" env: GITHUB_TOKEN: "${{ secrets.GITHUB_TOKEN }}" # Notify on slack only once - when CI completes (and after deploy) in case it's successfull notify-complete: needs: [ build, build_windows, docs_check ] - runs-on: ubuntu-latest + runs-on: ubuntu-20.04 steps: - name: Slack Notification uses: homoluctus/slatify@v1.8.0 @@ -214,13 +229,13 @@ jobs: deploy: needs: [ build, build_windows, docs_check ] - runs-on: ubuntu-18.04 + runs-on: ubuntu-20.04 if: (github.event_name == 'push' || github.event_name == 'schedule' || github.event_name == 'release') && github.repository == 'freqtrade/freqtrade' steps: - uses: actions/checkout@v2 - name: Set up Python - uses: actions/setup-python@v1 + uses: actions/setup-python@v2 with: python-version: 3.8 @@ -236,7 +251,7 @@ jobs: - name: Publish to PyPI (Test) uses: pypa/gh-action-pypi-publish@master - if: (steps.extract_branch.outputs.branch == 'master' || github.event_name == 'release') + if: (github.event_name == 'release') with: user: __token__ password: ${{ secrets.pypi_test_password }} @@ -244,7 +259,7 @@ jobs: - name: Publish to PyPI uses: pypa/gh-action-pypi-publish@master - if: (steps.extract_branch.outputs.branch == 'master' || github.event_name == 'release') + if: (github.event_name == 'release') with: user: __token__ password: ${{ secrets.pypi_password }} diff --git a/.github/workflows/docker_update_readme.yml b/.github/workflows/docker_update_readme.yml index 57a7e591e..95e69be2a 100644 --- a/.github/workflows/docker_update_readme.yml +++ b/.github/workflows/docker_update_readme.yml @@ -2,7 +2,7 @@ name: Update Docker Hub Description on: push: branches: - - master + - stable jobs: dockerHubDescription: diff --git a/.travis.yml b/.travis.yml index 0cb76b78b..9b8448db5 100644 --- a/.travis.yml +++ b/.travis.yml @@ -33,7 +33,7 @@ jobs: - script: - cp config.json.example config.json - freqtrade create-userdir --userdir user_data - - freqtrade hyperopt --datadir tests/testdata -e 5 --strategy SampleStrategy --hyperopt SampleHyperOpt + - freqtrade hyperopt --datadir tests/testdata -e 5 --strategy SampleStrategy --hyperopt SampleHyperOpt --hyperopt-loss SharpeHyperOptLossDaily name: hyperopt - script: flake8 name: flake8 diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 90594866a..399588f88 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -8,8 +8,9 @@ Issues labeled [good first issue](https://github.com/freqtrade/freqtrade/labels/ Few pointers for contributions: -- Create your PR against the `develop` branch, not `master`. -- New features need to contain unit tests and must be PEP8 conformant (max-line-length = 100). +- Create your PR against the `develop` branch, not `stable`. +- New features need to contain unit tests, must conform to PEP8 (max-line-length = 100) and should be documented with the introduction PR. +- PR's can be declared as `[WIP]` - which signify Work in Progress Pull Requests (which are not finished). If you are unsure, discuss the feature on our [Slack](https://join.slack.com/t/highfrequencybot/shared_invite/enQtNjU5ODcwNjI1MDU3LTU1MTgxMjkzNmYxNWE1MDEzYzQ3YmU4N2MwZjUyNjJjODRkMDVkNjg4YTAyZGYzYzlhOTZiMTE4ZjQ4YzM0OGE) or in a [issue](https://github.com/freqtrade/freqtrade/issues) before a PR. @@ -18,7 +19,7 @@ or in a [issue](https://github.com/freqtrade/freqtrade/issues) before a PR. Best start by reading the [documentation](https://www.freqtrade.io/) to get a feel for what is possible with the bot, or head straight to the [Developer-documentation](https://www.freqtrade.io/en/latest/developer/) (WIP) which should help you getting started. -## Before sending the PR: +## Before sending the PR ### 1. Run unit tests @@ -64,6 +65,14 @@ Guide for installing them is [here](http://flake8.pycqa.org/en/latest/user/using mypy freqtrade ``` +### 4. Ensure all imports are correct + +#### Run isort + +``` bash +isort . +``` + ## (Core)-Committer Guide ### Process: Pull Requests @@ -114,6 +123,6 @@ Contributors may be given commit privileges. Preference will be given to those w 1. Access to resources for cross-platform development and testing. 1. Time to devote to the project regularly. -Being a Committer does not grant write permission on `develop` or `master` for security reasons (Users trust Freqtrade with their Exchange API keys). +Being a Committer does not grant write permission on `develop` or `stable` for security reasons (Users trust Freqtrade with their Exchange API keys). After being Committer for some time, a Committer may be named Core Committer and given full repository access. diff --git a/Dockerfile b/Dockerfile index 22b0c43a7..2be65274e 100644 --- a/Dockerfile +++ b/Dockerfile @@ -1,4 +1,4 @@ -FROM python:3.8.5-slim-buster +FROM python:3.8.6-slim-buster RUN apt-get update \ && apt-get -y install curl build-essential libssl-dev sqlite3 \ @@ -22,7 +22,8 @@ RUN pip install numpy --no-cache-dir \ # Install and execute COPY . /freqtrade/ -RUN pip install -e . --no-cache-dir +RUN pip install -e . --no-cache-dir \ + && mkdir /freqtrade/user_data/ ENTRYPOINT ["freqtrade"] # Default to trade mode CMD [ "trade" ] diff --git a/README.md b/README.md index 90f303c6d..c9f4d0a52 100644 --- a/README.md +++ b/README.md @@ -55,9 +55,8 @@ Please find the complete documentation on our [website](https://www.freqtrade.io Freqtrade provides a Linux/macOS script to install all dependencies and help you to configure the bot. ```bash -git clone git@github.com:freqtrade/freqtrade.git +git clone -b develop https://github.com/freqtrade/freqtrade.git cd freqtrade -git checkout develop ./setup.sh --install ``` @@ -111,35 +110,38 @@ optional arguments: Telegram is not mandatory. However, this is a great way to control your bot. More details and the full command list on our [documentation](https://www.freqtrade.io/en/latest/telegram-usage/) -- `/start`: Starts the trader -- `/stop`: Stops the trader -- `/status [table]`: Lists all open trades -- `/count`: Displays number of open trades +- `/start`: Starts the trader. +- `/stop`: Stops the trader. +- `/stopbuy`: Stop entering new trades. +- `/status [table]`: Lists all open trades. - `/profit`: Lists cumulative profit from all finished trades - `/forcesell |all`: Instantly sells the given trade (Ignoring `minimum_roi`). - `/performance`: Show performance of each finished trade grouped by pair -- `/balance`: Show account balance per currency -- `/daily `: Shows profit or loss per day, over the last n days -- `/help`: Show help message -- `/version`: Show version +- `/balance`: Show account balance per currency. +- `/daily `: Shows profit or loss per day, over the last n days. +- `/help`: Show help message. +- `/version`: Show version. ## Development branches The project is currently setup in two main branches: -- `develop` - This branch has often new features, but might also cause breaking changes. -- `master` - This branch contains the latest stable release. The bot 'should' be stable on this branch, and is generally well tested. +- `develop` - This branch has often new features, but might also contain breaking changes. We try hard to keep this branch as stable as possible. +- `stable` - This branch contains the latest stable release. This branch is generally well tested. - `feat/*` - These are feature branches, which are being worked on heavily. Please don't use these unless you want to test a specific feature. ## Support -### Help / Slack +### Help / Slack / Discord -For any questions not covered by the documentation or for further -information about the bot, we encourage you to join our slack channel. +For any questions not covered by the documentation or for further information about the bot, we encourage you to join our slack channel. - [Click here to join Slack channel](https://join.slack.com/t/highfrequencybot/shared_invite/enQtNjU5ODcwNjI1MDU3LTU1MTgxMjkzNmYxNWE1MDEzYzQ3YmU4N2MwZjUyNjJjODRkMDVkNjg4YTAyZGYzYzlhOTZiMTE4ZjQ4YzM0OGE). +Alternatively, check out the newly created [discord server](https://discord.gg/MA9v74M). + +*Note*: Since the discord server is relatively new, answers to questions might be slightly delayed as currently the user base quite small. + ### [Bugs / Issues](https://github.com/freqtrade/freqtrade/issues?q=is%3Aissue) If you discover a bug in the bot, please @@ -166,18 +168,18 @@ Please read our [Contributing document](https://github.com/freqtrade/freqtrade/blob/develop/CONTRIBUTING.md) to understand the requirements before sending your pull-requests. -Coding is not a neccessity to contribute - maybe start with improving our documentation? +Coding is not a necessity to contribute - maybe start with improving our documentation? Issues labeled [good first issue](https://github.com/freqtrade/freqtrade/labels/good%20first%20issue) can be good first contributions, and will help get you familiar with the codebase. **Note** before starting any major new feature work, *please open an issue describing what you are planning to do* or talk to us on [Slack](https://join.slack.com/t/highfrequencybot/shared_invite/enQtNjU5ODcwNjI1MDU3LTU1MTgxMjkzNmYxNWE1MDEzYzQ3YmU4N2MwZjUyNjJjODRkMDVkNjg4YTAyZGYzYzlhOTZiMTE4ZjQ4YzM0OGE). This will ensure that interested parties can give valuable feedback on the feature, and let others know that you are working on it. -**Important:** Always create your PR against the `develop` branch, not `master`. +**Important:** Always create your PR against the `develop` branch, not `stable`. ## Requirements -### Uptodate clock +### Up-to-date clock -The clock must be accurate, syncronized to a NTP server very frequently to avoid problems with communication to the exchanges. +The clock must be accurate, synchronized to a NTP server very frequently to avoid problems with communication to the exchanges. ### Min hardware required diff --git a/build_helpers/TA_Lib-0.4.18-cp37-cp37m-win_amd64.whl b/build_helpers/TA_Lib-0.4.18-cp37-cp37m-win_amd64.whl deleted file mode 100644 index bd61e812b..000000000 Binary files a/build_helpers/TA_Lib-0.4.18-cp37-cp37m-win_amd64.whl and /dev/null differ diff --git a/build_helpers/TA_Lib-0.4.18-cp38-cp38-win_amd64.whl b/build_helpers/TA_Lib-0.4.18-cp38-cp38-win_amd64.whl deleted file mode 100644 index f81addb44..000000000 Binary files a/build_helpers/TA_Lib-0.4.18-cp38-cp38-win_amd64.whl and /dev/null differ diff --git a/build_helpers/TA_Lib-0.4.19-cp37-cp37m-win_amd64.whl b/build_helpers/TA_Lib-0.4.19-cp37-cp37m-win_amd64.whl new file mode 100644 index 000000000..5adbda8a7 Binary files /dev/null and b/build_helpers/TA_Lib-0.4.19-cp37-cp37m-win_amd64.whl differ diff --git a/build_helpers/TA_Lib-0.4.19-cp38-cp38-win_amd64.whl b/build_helpers/TA_Lib-0.4.19-cp38-cp38-win_amd64.whl new file mode 100644 index 000000000..b652c7ee0 Binary files /dev/null and b/build_helpers/TA_Lib-0.4.19-cp38-cp38-win_amd64.whl differ diff --git a/build_helpers/install_windows.ps1 b/build_helpers/install_windows.ps1 index 0a55b6ddd..5747db335 100644 --- a/build_helpers/install_windows.ps1 +++ b/build_helpers/install_windows.ps1 @@ -7,10 +7,10 @@ python -m pip install --upgrade pip $pyv = python -c "import sys; print(f'{sys.version_info.major}.{sys.version_info.minor}')" if ($pyv -eq '3.7') { - pip install build_helpers\TA_Lib-0.4.18-cp37-cp37m-win_amd64.whl + pip install build_helpers\TA_Lib-0.4.19-cp37-cp37m-win_amd64.whl } if ($pyv -eq '3.8') { - pip install build_helpers\TA_Lib-0.4.18-cp38-cp38-win_amd64.whl + pip install build_helpers\TA_Lib-0.4.19-cp38-cp38-win_amd64.whl } pip install -r requirements-dev.txt diff --git a/build_helpers/publish_docker.sh b/build_helpers/publish_docker.sh index 03a95161b..ac0cd2461 100755 --- a/build_helpers/publish_docker.sh +++ b/build_helpers/publish_docker.sh @@ -2,6 +2,7 @@ # Replace / with _ to create a valid tag TAG=$(echo "${BRANCH_NAME}" | sed -e "s/\//_/g") +TAG_PLOT=${TAG}_plot echo "Running for ${TAG}" # Add commit and commit_message to docker container @@ -16,6 +17,12 @@ else docker pull ${IMAGE_NAME}:${TAG} docker build --cache-from ${IMAGE_NAME}:${TAG} -t freqtrade:${TAG} . fi +# Tag image for upload and next build step +docker tag freqtrade:$TAG ${IMAGE_NAME}:$TAG + +docker build --cache-from freqtrade:${TAG} --build-arg sourceimage=${TAG} -t freqtrade:${TAG_PLOT} -f docker/Dockerfile.plot . + +docker tag freqtrade:$TAG_PLOT ${IMAGE_NAME}:$TAG_PLOT if [ $? -ne 0 ]; then echo "failed building image" @@ -30,8 +37,6 @@ if [ $? -ne 0 ]; then return 1 fi -# Tag image for upload -docker tag freqtrade:$TAG ${IMAGE_NAME}:$TAG if [ $? -ne 0 ]; then echo "failed tagging image" return 1 diff --git a/config.json.example b/config.json.example index ab517b77c..af45dac74 100644 --- a/config.json.example +++ b/config.json.example @@ -5,15 +5,15 @@ "tradable_balance_ratio": 0.99, "fiat_display_currency": "USD", "timeframe": "5m", - "dry_run": false, + "dry_run": true, "cancel_open_orders_on_exit": false, "unfilledtimeout": { "buy": 10, "sell": 30 }, "bid_strategy": { - "ask_last_balance": 0.0, "use_order_book": false, + "ask_last_balance": 0.0, "order_book_top": 1, "check_depth_of_market": { "enabled": false, diff --git a/config_full.json.example b/config_full.json.example index d5bfd3fe1..45c5c695c 100644 --- a/config_full.json.example +++ b/config_full.json.example @@ -7,7 +7,7 @@ "amount_reserve_percent": 0.05, "amend_last_stake_amount": false, "last_stake_amount_min_ratio": 0.5, - "dry_run": false, + "dry_run": true, "cancel_open_orders_on_exit": false, "timeframe": "5m", "trailing_stop": false, @@ -116,7 +116,16 @@ "telegram": { "enabled": true, "token": "your_telegram_token", - "chat_id": "your_telegram_chat_id" + "chat_id": "your_telegram_chat_id", + "notification_settings": { + "status": "on", + "warning": "on", + "startup": "on", + "buy": "on", + "sell": "on", + "buy_cancel": "on", + "sell_cancel": "on" + } }, "api_server": { "enabled": false, diff --git a/config_kraken.json.example b/config_kraken.json.example index fd0b2b95d..5f3b57854 100644 --- a/config_kraken.json.example +++ b/config_kraken.json.example @@ -27,12 +27,11 @@ "use_sell_signal": true, "sell_profit_only": false, "ignore_roi_if_buy_signal": false - }, "exchange": { "name": "kraken", - "key": "", - "secret": "", + "key": "your_exchange_key", + "secret": "your_exchange_key", "ccxt_config": {"enableRateLimit": true}, "ccxt_async_config": { "enableRateLimit": true, diff --git a/docker-compose.develop.yml b/docker-compose.develop.yml deleted file mode 100644 index 562b5960a..000000000 --- a/docker-compose.develop.yml +++ /dev/null @@ -1,20 +0,0 @@ ---- -version: '3' -services: - freqtrade_develop: - build: - context: . - dockerfile: "./Dockerfile.develop" - volumes: - - ".:/freqtrade" - entrypoint: - - "freqtrade" - - freqtrade_bash: - build: - context: . - dockerfile: "./Dockerfile.develop" - volumes: - - ".:/freqtrade" - entrypoint: - - "/bin/bash" diff --git a/docker-compose.yml b/docker-compose.yml index 49d83aa5e..a99aac3c7 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -2,8 +2,10 @@ version: '3' services: freqtrade: - image: freqtradeorg/freqtrade:master + image: freqtradeorg/freqtrade:stable # image: freqtradeorg/freqtrade:develop + # Use plotting image + # image: freqtradeorg/freqtrade:develop_plot # Build step - only needed when additional dependencies are needed # build: # context: . diff --git a/Dockerfile.develop b/docker/Dockerfile.develop similarity index 99% rename from Dockerfile.develop rename to docker/Dockerfile.develop index 8f6871c55..cb49984e2 100644 --- a/Dockerfile.develop +++ b/docker/Dockerfile.develop @@ -2,6 +2,7 @@ FROM freqtradeorg/freqtrade:develop # Install dependencies COPY requirements-dev.txt /freqtrade/ + RUN pip install numpy --no-cache-dir \ && pip install -r requirements-dev.txt --no-cache-dir diff --git a/docker/Dockerfile.jupyter b/docker/Dockerfile.jupyter new file mode 100644 index 000000000..b7499eeef --- /dev/null +++ b/docker/Dockerfile.jupyter @@ -0,0 +1,7 @@ +FROM freqtradeorg/freqtrade:develop_plot + + +RUN pip install jupyterlab --no-cache-dir + +# Empty the ENTRYPOINT to allow all commands +ENTRYPOINT [] diff --git a/docker/Dockerfile.plot b/docker/Dockerfile.plot new file mode 100644 index 000000000..40bc72bc5 --- /dev/null +++ b/docker/Dockerfile.plot @@ -0,0 +1,7 @@ +ARG sourceimage=develop +FROM freqtradeorg/freqtrade:${sourceimage} + +# Install dependencies +COPY requirements-plot.txt /freqtrade/ + +RUN pip install -r requirements-plot.txt --no-cache-dir diff --git a/Dockerfile.technical b/docker/Dockerfile.technical similarity index 100% rename from Dockerfile.technical rename to docker/Dockerfile.technical diff --git a/docker/docker-compose-jupyter.yml b/docker/docker-compose-jupyter.yml new file mode 100644 index 000000000..11a01705c --- /dev/null +++ b/docker/docker-compose-jupyter.yml @@ -0,0 +1,16 @@ +--- +version: '3' +services: + ft_jupyterlab: + build: + context: .. + dockerfile: docker/Dockerfile.jupyter + restart: unless-stopped + container_name: freqtrade + ports: + - "127.0.0.1:8888:8888" + volumes: + - "./user_data:/freqtrade/user_data" + # Default command used when running `docker compose up` + command: > + jupyter lab --port=8888 --ip 0.0.0.0 --allow-root diff --git a/docs/advanced-hyperopt.md b/docs/advanced-hyperopt.md index 5fc674b03..59ebc16b5 100644 --- a/docs/advanced-hyperopt.md +++ b/docs/advanced-hyperopt.md @@ -27,9 +27,9 @@ class MyAwesomeHyperOpt2(MyAwesomeHyperOpt): and then quickly switch between hyperopt classes, running optimization process with hyperopt class you need in each particular case: ``` -$ freqtrade hyperopt --hyperopt MyAwesomeHyperOpt ... +$ freqtrade hyperopt --hyperopt MyAwesomeHyperOpt --hyperopt-loss SharpeHyperOptLossDaily --strategy MyAwesomeStrategy ... or -$ freqtrade hyperopt --hyperopt MyAwesomeHyperOpt2 ... +$ freqtrade hyperopt --hyperopt MyAwesomeHyperOpt2 --hyperopt-loss SharpeHyperOptLossDaily --strategy MyAwesomeStrategy ... ``` ## Creating and using a custom loss function diff --git a/docs/bot-usage.md b/docs/bot-usage.md index 4a4496bbc..4d07435c7 100644 --- a/docs/bot-usage.md +++ b/docs/bot-usage.md @@ -303,7 +303,7 @@ usage: freqtrade hyperopt [-h] [-v] [--logfile FILE] [-V] [-c PATH] [-d PATH] [--spaces {all,buy,sell,roi,stoploss,trailing,default} [{all,buy,sell,roi,stoploss,trailing,default} ...]] [--dmmp] [--print-all] [--no-color] [--print-json] [-j JOBS] [--random-state INT] [--min-trades INT] - [--continue] [--hyperopt-loss NAME] + [--hyperopt-loss NAME] optional arguments: -h, --help show this help message and exit @@ -349,18 +349,14 @@ optional arguments: reproducible hyperopt results. --min-trades INT Set minimal desired number of trades for evaluations in the hyperopt optimization path (default: 1). - --continue Continue hyperopt from previous runs. By default, - temporary files will be removed and hyperopt will - start from scratch. --hyperopt-loss NAME Specify the class name of the hyperopt loss function class (IHyperOptLoss). Different functions can generate completely different results, since the target for optimization is different. Built-in - Hyperopt-loss-functions are: DefaultHyperOptLoss, + Hyperopt-loss-functions are: ShortTradeDurHyperOptLoss, OnlyProfitHyperOptLoss, SharpeHyperOptLoss, SharpeHyperOptLossDaily, SortinoHyperOptLoss, - SortinoHyperOptLossDaily.(default: - `DefaultHyperOptLoss`). + SortinoHyperOptLossDaily. Common arguments: -v, --verbose Verbose mode (-vv for more, -vvv to get all messages). diff --git a/docs/configuration.md b/docs/configuration.md index bf141f8e8..f8e8aabcd 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -375,7 +375,7 @@ Freqtrade is based on [CCXT library](https://github.com/ccxt/ccxt) that supports exchange markets and trading APIs. The complete up-to-date list can be found in the [CCXT repo homepage](https://github.com/ccxt/ccxt/tree/master/python). However, the bot was tested by the development team with only Bittrex, Binance and Kraken, - so the these are the only officially supported exhanges: + so the these are the only officially supported exchanges: - [Bittrex](https://bittrex.com/): "bittrex" - [Binance](https://www.binance.com/): "binance" @@ -574,144 +574,7 @@ Assuming both buy and sell are using market orders, a configuration similar to t ``` Obviously, if only one side is using limit orders, different pricing combinations can be used. - -## Pairlists and Pairlist Handlers - -Pairlist Handlers define the list of pairs (pairlist) that the bot should trade. They are configured in the `pairlists` section of the configuration settings. - -In your configuration, you can use Static Pairlist (defined by the [`StaticPairList`](#static-pair-list) Pairlist Handler) and Dynamic Pairlist (defined by the [`VolumePairList`](#volume-pair-list) Pairlist Handler). - -Additionaly, [`AgeFilter`](#agefilter), [`PrecisionFilter`](#precisionfilter), [`PriceFilter`](#pricefilter), [`ShuffleFilter`](#shufflefilter) and [`SpreadFilter`](#spreadfilter) act as Pairlist Filters, removing certain pairs and/or moving their positions in the pairlist. - -If multiple Pairlist Handlers are used, they are chained and a combination of all Pairlist Handlers forms the resulting pairlist the bot uses for trading and backtesting. Pairlist Handlers are executed in the sequence they are configured. You should always configure either `StaticPairList` or `VolumePairList` as the starting Pairlist Handler. - -Inactive markets are always removed from the resulting pairlist. Explicitly blacklisted pairs (those in the `pair_blacklist` configuration setting) are also always removed from the resulting pairlist. - -### Available Pairlist Handlers - -* [`StaticPairList`](#static-pair-list) (default, if not configured differently) -* [`VolumePairList`](#volume-pair-list) -* [`AgeFilter`](#agefilter) -* [`PrecisionFilter`](#precisionfilter) -* [`PriceFilter`](#pricefilter) -* [`ShuffleFilter`](#shufflefilter) -* [`SpreadFilter`](#spreadfilter) - -!!! Tip "Testing pairlists" - Pairlist configurations can be quite tricky to get right. Best use the [`test-pairlist`](utils.md#test-pairlist) utility subcommand to test your configuration quickly. - -#### Static Pair List - -By default, the `StaticPairList` method is used, which uses a statically defined pair whitelist from the configuration. - -It uses configuration from `exchange.pair_whitelist` and `exchange.pair_blacklist`. - -```json -"pairlists": [ - {"method": "StaticPairList"} - ], -``` - -#### Volume Pair List - -`VolumePairList` employs sorting/filtering of pairs by their trading volume. It selects `number_assets` top pairs with sorting based on the `sort_key` (which can only be `quoteVolume`). - -When used in the chain of Pairlist Handlers in a non-leading position (after StaticPairList and other Pairlist Filters), `VolumePairList` considers outputs of previous Pairlist Handlers, adding its sorting/selection of the pairs by the trading volume. - -When used on the leading position of the chain of Pairlist Handlers, it does not consider `pair_whitelist` configuration setting, but selects the top assets from all available markets (with matching stake-currency) on the exchange. - -The `refresh_period` setting allows to define the period (in seconds), at which the pairlist will be refreshed. Defaults to 1800s (30 minutes). - -`VolumePairList` is based on the ticker data from exchange, as reported by the ccxt library: - -* The `quoteVolume` is the amount of quote (stake) currency traded (bought or sold) in last 24 hours. - -```json -"pairlists": [{ - "method": "VolumePairList", - "number_assets": 20, - "sort_key": "quoteVolume", - "refresh_period": 1800, -}], -``` - -#### AgeFilter - -Removes pairs that have been listed on the exchange for less than `min_days_listed` days (defaults to `10`). - -When pairs are first listed on an exchange they can suffer huge price drops and volatility -in the first few days while the pair goes through its price-discovery period. Bots can often -be caught out buying before the pair has finished dropping in price. - -This filter allows freqtrade to ignore pairs until they have been listed for at least `min_days_listed` days. - -#### PrecisionFilter - -Filters low-value coins which would not allow setting stoplosses. - -#### PriceFilter - -The `PriceFilter` allows filtering of pairs by price. Currently the following price filters are supported: - -* `min_price` -* `max_price` -* `low_price_ratio` - -The `min_price` setting removes pairs where the price is below the specified price. This is useful if you wish to avoid trading very low-priced pairs. -This option is disabled by default, and will only apply if set to > 0. - -The `max_price` setting removes pairs where the price is above the specified price. This is useful if you wish to trade only low-priced pairs. -This option is disabled by default, and will only apply if set to > 0. - -The `low_price_ratio` setting removes pairs where a raise of 1 price unit (pip) is above the `low_price_ratio` ratio. -This option is disabled by default, and will only apply if set to > 0. - -For `PriceFiler` at least one of its `min_price`, `max_price` or `low_price_ratio` settings must be applied. - -Calculation example: - -Min price precision for SHITCOIN/BTC is 8 decimals. If its price is 0.00000011 - one price step above would be 0.00000012, which is ~9% higher than the previous price value. You may filter out this pair by using PriceFilter with `low_price_ratio` set to 0.09 (9%) or with `min_price` set to 0.00000011, correspondingly. - -!!! Warning "Low priced pairs" - Low priced pairs with high "1 pip movements" are dangerous since they are often illiquid and it may also be impossible to place the desired stoploss, which can often result in high losses since price needs to be rounded to the next tradable price - so instead of having a stoploss of -5%, you could end up with a stoploss of -9% simply due to price rounding. - -#### ShuffleFilter - -Shuffles (randomizes) pairs in the pairlist. It can be used for preventing the bot from trading some of the pairs more frequently then others when you want all pairs be treated with the same priority. - -!!! Tip - You may set the `seed` value for this Pairlist to obtain reproducible results, which can be useful for repeated backtesting sessions. If `seed` is not set, the pairs are shuffled in the non-repeatable random order. - -#### SpreadFilter - -Removes pairs that have a difference between asks and bids above the specified ratio, `max_spread_ratio` (defaults to `0.005`). - -Example: - -If `DOGE/BTC` maximum bid is 0.00000026 and minimum ask is 0.00000027, the ratio is calculated as: `1 - bid/ask ~= 0.037` which is `> 0.005` and this pair will be filtered out. - -### Full example of Pairlist Handlers - -The below example blacklists `BNB/BTC`, uses `VolumePairList` with `20` assets, sorting pairs by `quoteVolume` and applies both [`PrecisionFilter`](#precisionfilter) and [`PriceFilter`](#price-filter), filtering all assets where 1 priceunit is > 1%. Then the `SpreadFilter` is applied and pairs are finally shuffled with the random seed set to some predefined value. - -```json -"exchange": { - "pair_whitelist": [], - "pair_blacklist": ["BNB/BTC"] -}, -"pairlists": [ - { - "method": "VolumePairList", - "number_assets": 20, - "sort_key": "quoteVolume", - }, - {"method": "AgeFilter", "min_days_listed": 10}, - {"method": "PrecisionFilter"}, - {"method": "PriceFilter", "low_price_ratio": 0.01}, - {"method": "SpreadFilter", "max_spread_ratio": 0.005}, - {"method": "ShuffleFilter", "seed": 42} - ], -``` +--8<-- "includes/pairlists.md" ## Switch to Dry-run mode diff --git a/docs/data-analysis.md b/docs/data-analysis.md index fc4693b17..17da98935 100644 --- a/docs/data-analysis.md +++ b/docs/data-analysis.md @@ -1,12 +1,22 @@ # Analyzing bot data with Jupyter notebooks -You can analyze the results of backtests and trading history easily using Jupyter notebooks. Sample notebooks are located at `user_data/notebooks/`. +You can analyze the results of backtests and trading history easily using Jupyter notebooks. Sample notebooks are located at `user_data/notebooks/` after initializing the user directory with `freqtrade create-userdir --userdir user_data`. -## Pro tips +## Quick start with docker + +Freqtrade provides a docker-compose file which starts up a jupyter lab server. +You can run this server using the following command: `docker-compose -f docker/docker-compose-jupyter.yml up` + +This will create a dockercontainer running jupyter lab, which will be accessible using `https://127.0.0.1:8888/lab`. +Please use the link that's printed in the console after startup for simplified login. + +For more information, Please visit the [Data analysis with Docker](docker_quickstart.md#data-analayis-using-docker-compose) section. + +### Pro tips * See [jupyter.org](https://jupyter.org/documentation) for usage instructions. * Don't forget to start a Jupyter notebook server from within your conda or venv environment or use [nb_conda_kernels](https://github.com/Anaconda-Platform/nb_conda_kernels)* -* Copy the example notebook before use so your changes don't get clobbered with the next freqtrade update. +* Copy the example notebook before use so your changes don't get overwritten with the next freqtrade update. ### Using virtual environment with system-wide Jupyter installation @@ -28,10 +38,8 @@ ipython kernel install --user --name=freqtrade !!! Note This section is provided for completeness, the Freqtrade Team won't provide full support for problems with this setup and will recommend to install Jupyter in the virtual environment directly, as that is the easiest way to get jupyter notebooks up and running. For help with this setup please refer to the [Project Jupyter](https://jupyter.org/) [documentation](https://jupyter.org/documentation) or [help channels](https://jupyter.org/community). - -## Fine print - -Some tasks don't work especially well in notebooks. For example, anything using asynchronous execution is a problem for Jupyter. Also, freqtrade's primary entry point is the shell cli, so using pure python in a notebook bypasses arguments that provide required objects and parameters to helper functions. You may need to set those values or create expected objects manually. +!!! Warning + Some tasks don't work especially well in notebooks. For example, anything using asynchronous execution is a problem for Jupyter. Also, freqtrade's primary entry point is the shell cli, so using pure python in a notebook bypasses arguments that provide required objects and parameters to helper functions. You may need to set those values or create expected objects manually. ## Recommended workflow diff --git a/docs/data-download.md b/docs/data-download.md index 0b22ec9ce..e9c5c1865 100644 --- a/docs/data-download.md +++ b/docs/data-download.md @@ -8,9 +8,11 @@ If no additional parameter is specified, freqtrade will download data for `"1m"` Exchange and pairs will come from `config.json` (if specified using `-c/--config`). Otherwise `--exchange` becomes mandatory. +You can use a relative timerange (`--days 20`) or an absolute starting point (`--timerange 20200101`). For incremental downloads, the relative approach should be used. + !!! Tip "Tip: Updating existing data" If you already have backtesting data available in your data-directory and would like to refresh this data up to today, use `--days xx` with a number slightly higher than the missing number of days. Freqtrade will keep the available data and only download the missing data. - Be carefull though: If the number is too small (which would result in a few missing days), the whole dataset will be removed and only xx days will be downloaded. + Be careful though: If the number is too small (which would result in a few missing days), the whole dataset will be removed and only xx days will be downloaded. ### Usage @@ -18,9 +20,9 @@ Otherwise `--exchange` becomes mandatory. usage: freqtrade download-data [-h] [-v] [--logfile FILE] [-V] [-c PATH] [-d PATH] [--userdir PATH] [-p PAIRS [PAIRS ...]] [--pairs-file FILE] - [--days INT] [--dl-trades] - [--exchange EXCHANGE] - [-t {1m,3m,5m,15m,30m,1h,2h,4h,6h,8h,12h,1d,3d,1w} [{1m,3m,5m,15m,30m,1h,2h,4h,6h,8h,12h,1d,3d,1w} ...]] + [--days INT] [--timerange TIMERANGE] + [--dl-trades] [--exchange EXCHANGE] + [-t {1m,3m,5m,15m,30m,1h,2h,4h,6h,8h,12h,1d,3d,1w,2w,1M,1y} [{1m,3m,5m,15m,30m,1h,2h,4h,6h,8h,12h,1d,3d,1w,2w,1M,1y} ...]] [--erase] [--data-format-ohlcv {json,jsongz,hdf5}] [--data-format-trades {json,jsongz,hdf5}] @@ -32,12 +34,14 @@ optional arguments: separated. --pairs-file FILE File containing a list of pairs to download. --days INT Download data for given number of days. + --timerange TIMERANGE + Specify what timerange of data to use. --dl-trades Download trades instead of OHLCV data. The bot will resample trades to the desired timeframe as specified as --timeframes/-t. --exchange EXCHANGE Exchange name (default: `bittrex`). Only valid if no config is provided. - -t {1m,3m,5m,15m,30m,1h,2h,4h,6h,8h,12h,1d,3d,1w} [{1m,3m,5m,15m,30m,1h,2h,4h,6h,8h,12h,1d,3d,1w} ...], --timeframes {1m,3m,5m,15m,30m,1h,2h,4h,6h,8h,12h,1d,3d,1w} [{1m,3m,5m,15m,30m,1h,2h,4h,6h,8h,12h,1d,3d,1w} ...] + -t {1m,3m,5m,15m,30m,1h,2h,4h,6h,8h,12h,1d,3d,1w,2w,1M,1y} [{1m,3m,5m,15m,30m,1h,2h,4h,6h,8h,12h,1d,3d,1w,2w,1M,1y} ...], --timeframes {1m,3m,5m,15m,30m,1h,2h,4h,6h,8h,12h,1d,3d,1w,2w,1M,1y} [{1m,3m,5m,15m,30m,1h,2h,4h,6h,8h,12h,1d,3d,1w,2w,1M,1y} ...] Specify which tickers to download. Space-separated list. Default: `1m 5m`. --erase Clean all existing data for the selected @@ -67,6 +71,11 @@ Common arguments: ``` +!!! Note "Startup period" + `download-data` is a strategy-independent command. The idea is to download a big chunk of data once, and then iteratively increase the amount of data stored. + + For that reason, `download-data` does not care about the "startup-period" defined in a strategy. It's up to the user to download additional days if the backtest should start at a specific point in time (while respecting startup period). + ### Data format Freqtrade currently supports 3 data-formats for both OHLCV and trades data: @@ -100,7 +109,7 @@ usage: freqtrade convert-data [-h] [-v] [--logfile FILE] [-V] [-c PATH] [-p PAIRS [PAIRS ...]] --format-from {json,jsongz,hdf5} --format-to {json,jsongz,hdf5} [--erase] - [-t {1m,3m,5m,15m,30m,1h,2h,4h,6h,8h,12h,1d,3d,1w} [{1m,3m,5m,15m,30m,1h,2h,4h,6h,8h,12h,1d,3d,1w} ...]] + [-t {1m,3m,5m,15m,30m,1h,2h,4h,6h,8h,12h,1d,3d,1w,2w,1M,1y} [{1m,3m,5m,15m,30m,1h,2h,4h,6h,8h,12h,1d,3d,1w,2w,1M,1y} ...]] optional arguments: -h, --help show this help message and exit @@ -113,7 +122,7 @@ optional arguments: Destination format for data conversion. --erase Clean all existing data for the selected exchange/pairs/timeframes. - -t {1m,3m,5m,15m,30m,1h,2h,4h,6h,8h,12h,1d,3d,1w} [{1m,3m,5m,15m,30m,1h,2h,4h,6h,8h,12h,1d,3d,1w} ...], --timeframes {1m,3m,5m,15m,30m,1h,2h,4h,6h,8h,12h,1d,3d,1w} [{1m,3m,5m,15m,30m,1h,2h,4h,6h,8h,12h,1d,3d,1w} ...] + -t {1m,3m,5m,15m,30m,1h,2h,4h,6h,8h,12h,1d,3d,1w,2w,1M,1y} [{1m,3m,5m,15m,30m,1h,2h,4h,6h,8h,12h,1d,3d,1w,2w,1M,1y} ...], --timeframes {1m,3m,5m,15m,30m,1h,2h,4h,6h,8h,12h,1d,3d,1w,2w,1M,1y} [{1m,3m,5m,15m,30m,1h,2h,4h,6h,8h,12h,1d,3d,1w,2w,1M,1y} ...] Specify which tickers to download. Space-separated list. Default: `1m 5m`. @@ -286,6 +295,7 @@ This will download historical candle (OHLCV) data for all the currency pairs you - To change the exchange used to download the historical data from, please use a different configuration file (you'll probably need to adjust rate limits etc.) - To use `pairs.json` from some other directory, use `--pairs-file some_other_dir/pairs.json`. - To download historical candle (OHLCV) data for only 10 days, use `--days 10` (defaults to 30 days). +- To download historical candle (OHLCV) data from a fixed starting point, use `--timerange 20200101-` - which will download all data from January 1st, 2020. Eventually set end dates are ignored. - Use `--timeframes` to specify what timeframe download the historical candle (OHLCV) data for. Default is `--timeframes 1m 5m` which will download 1-minute and 5-minute data. - To use exchange, timeframe and list of pairs as defined in your configuration file, use the `-c/--config` option. With this, the script uses the whitelist defined in the config as the list of currency pairs to download data for and does not require the pairs.json file. You can combine `-c/--config` with most other options. diff --git a/docs/developer.md b/docs/developer.md index 111c7a96f..8ef816d5d 100644 --- a/docs/developer.md +++ b/docs/developer.md @@ -21,11 +21,24 @@ This will spin up a local server (usually on port 8000) so you can see if everyt ## Developer setup -To configure a development environment, best use the `setup.sh` script and answer "y" when asked "Do you want to install dependencies for dev [y/N]? ". -Alternatively (if your system is not supported by the setup.sh script), follow the manual installation process and run `pip3 install -e .[all]`. +To configure a development environment, you can either use the provided [DevContainer](#devcontainer-setup), or use the `setup.sh` script and answer "y" when asked "Do you want to install dependencies for dev [y/N]? ". +Alternatively (e.g. if your system is not supported by the setup.sh script), follow the manual installation process and run `pip3 install -e .[all]`. This will install all required tools for development, including `pytest`, `flake8`, `mypy`, and `coveralls`. +### Devcontainer setup + +The fastest and easiest way to get started is to use [VSCode](https://code.visualstudio.com/) with the Remote container extension. +This gives developers the ability to start the bot with all required dependencies *without* needing to install any freqtrade specific dependencies on your local machine. + +#### Devcontainer dependencies + +* [VSCode](https://code.visualstudio.com/) +* [docker](https://docs.docker.com/install/) +* [Remote container extension documentation](https://code.visualstudio.com/docs/remote) + +For more information about the [Remote container extension](https://code.visualstudio.com/docs/remote), best consult the documentation. + ### Tests New code should be covered by basic unittests. Depending on the complexity of the feature, Reviewers may request more in-depth unittests. @@ -50,51 +63,6 @@ def test_method_to_test(caplog): ``` -### Local docker usage - -The fastest and easiest way to start up is to use docker-compose.develop which gives developers the ability to start the bot up with all the required dependencies, *without* needing to install any freqtrade specific dependencies on your local machine. - -#### Install - -* [git](https://git-scm.com/book/en/v2/Getting-Started-Installing-Git) -* [docker](https://docs.docker.com/install/) -* [docker-compose](https://docs.docker.com/compose/install/) - -#### Starting the bot - -##### Use the develop dockerfile - -``` bash -rm docker-compose.yml && mv docker-compose.develop.yml docker-compose.yml -``` - -#### Docker Compose - -##### Starting - -``` bash -docker-compose up -``` - -![Docker compose up](https://user-images.githubusercontent.com/419355/65456322-47f63a80-de06-11e9-90c6-3c74d1bad0b8.png) - -##### Rebuilding - -``` bash -docker-compose build -``` - -##### Executing (effectively SSH into the container) - -The `exec` command requires that the container already be running, if you want to start it -that can be effected by `docker-compose up` or `docker-compose run freqtrade_develop` - -``` bash -docker-compose exec freqtrade_develop /bin/bash -``` - -![image](https://user-images.githubusercontent.com/419355/65456522-ba671a80-de06-11e9-9598-df9ca0d8dcac.png) - ## ErrorHandling Freqtrade Exceptions all inherit from `FreqtradeException`. @@ -120,13 +88,15 @@ Below is an outline of exception inheritance hierarchy: | +---+ InvalidOrderException | | | +---+ RetryableOrderError +| | +| +---+ InsufficientFundsError | +---+ StrategyError ``` ## Modules -### Dynamic Pairlist +### Pairlists You have a great idea for a new pair selection algorithm you would like to try out? Great. Hopefully you also want to contribute this back upstream. @@ -261,13 +231,14 @@ jupyter nbconvert --ClearOutputPreprocessor.enabled=True --to markdown freqtrade This documents some decisions taken for the CI Pipeline. * CI runs on all OS variants, Linux (ubuntu), macOS and Windows. -* Docker images are build for the branches `master` and `develop`. -* Raspberry PI Docker images are postfixed with `_pi` - so tags will be `:master_pi` and `develop_pi`. +* Docker images are build for the branches `stable` and `develop`. +* Docker images containing Plot dependencies are also available as `stable_plot` and `develop_plot`. +* Raspberry PI Docker images are postfixed with `_pi` - so tags will be `:stable_pi` and `develop_pi`. * Docker images contain a file, `/freqtrade/freqtrade_commit` containing the commit this image is based of. * Full docker image rebuilds are run once a week via schedule. * Deployments run on ubuntu. * ta-lib binaries are contained in the build_helpers directory to avoid fails related to external unavailability. -* All tests must pass for a PR to be merged to `master` or `develop`. +* All tests must pass for a PR to be merged to `stable` or `develop`. ## Creating a release @@ -284,19 +255,19 @@ git checkout -b new_release Determine if crucial bugfixes have been made between this commit and the current state, and eventually cherry-pick these. -* Merge the release branch (master) into this branch. +* Merge the release branch (stable) into this branch. * Edit `freqtrade/__init__.py` and add the version matching the current date (for example `2019.7` for July 2019). Minor versions can be `2019.7.1` should we need to do a second release that month. Version numbers must follow allowed versions from PEP0440 to avoid failures pushing to pypi. * Commit this part -* push that branch to the remote and create a PR against the master branch +* push that branch to the remote and create a PR against the stable branch ### Create changelog from git commits !!! Note - Make sure that the master branch is up-to-date! + Make sure that the `stable` branch is up-to-date! ``` bash # Needs to be done before merging / pulling that branch. -git log --oneline --no-decorate --no-merges master..new_release +git log --oneline --no-decorate --no-merges stable..new_release ``` To keep the release-log short, best wrap the full git changelog into a collapsible details section. @@ -312,11 +283,11 @@ To keep the release-log short, best wrap the full git changelog into a collapsib ### Create github release / tag -Once the PR against master is merged (best right after merging): +Once the PR against stable is merged (best right after merging): * Use the button "Draft a new release" in the Github UI (subsection releases). * Use the version-number specified as tag. -* Use "master" as reference (this step comes after the above PR is merged). +* Use "stable" as reference (this step comes after the above PR is merged). * Use the above changelog as release comment (as codeblock) ## Releases diff --git a/docs/docker.md b/docs/docker.md index 3fe335cf0..f4699cf4c 100644 --- a/docs/docker.md +++ b/docs/docker.md @@ -10,9 +10,9 @@ Pull the image from docker hub. Branches / tags available can be checked out on [Dockerhub tags page](https://hub.docker.com/r/freqtradeorg/freqtrade/tags/). ```bash -docker pull freqtradeorg/freqtrade:master +docker pull freqtradeorg/freqtrade:stable # Optionally tag the repository so the run-commands remain shorter -docker tag freqtradeorg/freqtrade:master freqtrade +docker tag freqtradeorg/freqtrade:stable freqtrade ``` To update the image, simply run the above commands again and restart your running container. @@ -20,7 +20,7 @@ To update the image, simply run the above commands again and restart your runnin Should you require additional libraries, please [build the image yourself](#build-your-own-docker-image). !!! Note "Docker image update frequency" - The official docker images with tags `master`, `develop` and `latest` are automatically rebuild once a week to keep the base image uptodate. + The official docker images with tags `stable`, `develop` and `latest` are automatically rebuild once a week to keep the base image up-to-date. In addition to that, every merge to `develop` will trigger a rebuild for `develop` and `latest`. ### Prepare the configuration files @@ -70,16 +70,16 @@ cp -n config.json.example config.json Best start by pulling the official docker image from dockerhub as explained [here](#download-the-official-docker-image) to speed up building. -To add additional libraries to your docker image, best check out [Dockerfile.technical](https://github.com/freqtrade/freqtrade/blob/develop/Dockerfile.technical) which adds the [technical](https://github.com/freqtrade/technical) module to the image. +To add additional libraries to your docker image, best check out [Dockerfile.technical](https://github.com/freqtrade/freqtrade/blob/develop/docker/Dockerfile.technical) which adds the [technical](https://github.com/freqtrade/technical) module to the image. ```bash -docker build -t freqtrade -f Dockerfile.technical . +docker build -t freqtrade -f docker/Dockerfile.technical . ``` -If you are developing using Docker, use `Dockerfile.develop` to build a dev Docker image, which will also set up develop dependencies: +If you are developing using Docker, use `docker/Dockerfile.develop` to build a dev Docker image, which will also set up develop dependencies: ```bash -docker build -f Dockerfile.develop -t freqtrade-dev . +docker build -f docker/Dockerfile.develop -t freqtrade-dev . ``` !!! Warning "Include your config file manually" diff --git a/docs/docker_quickstart.md b/docs/docker_quickstart.md index c033e827b..48ee34954 100644 --- a/docs/docker_quickstart.md +++ b/docs/docker_quickstart.md @@ -29,7 +29,7 @@ Create a new directory and place the [docker-compose file](https://github.com/fr mkdir ft_userdata cd ft_userdata/ # Download the docker-compose file from the repository - curl https://raw.githubusercontent.com/freqtrade/freqtrade/master/docker-compose.yml -o docker-compose.yml + curl https://raw.githubusercontent.com/freqtrade/freqtrade/stable/docker-compose.yml -o docker-compose.yml # Pull the freqtrade image docker-compose pull @@ -46,7 +46,7 @@ Create a new directory and place the [docker-compose file](https://github.com/fr mkdir ft_userdata cd ft_userdata/ # Download the docker-compose file from the repository - curl https://raw.githubusercontent.com/freqtrade/freqtrade/master/docker-compose.yml -o docker-compose.yml + curl https://raw.githubusercontent.com/freqtrade/freqtrade/stable/docker-compose.yml -o docker-compose.yml # Pull the freqtrade image docker-compose pull @@ -61,7 +61,7 @@ Create a new directory and place the [docker-compose file](https://github.com/fr !!! Note "Change your docker Image" You have to change the docker image in the docker-compose file for your Raspberry build to work properly. ``` yml - image: freqtradeorg/freqtrade:master_pi + image: freqtradeorg/freqtrade:stable_pi # image: freqtradeorg/freqtrade:develop_pi ``` @@ -148,7 +148,7 @@ Head over to the [Backtesting Documentation](backtesting.md) to learn more. ### Additional dependencies with docker-compose If your strategy requires dependencies not included in the default image (like [technical](https://github.com/freqtrade/technical)) - it will be necessary to build the image on your host. -For this, please create a Dockerfile containing installation steps for the additional dependencies (have a look at [Dockerfile.technical](https://github.com/freqtrade/freqtrade/blob/develop/Dockerfile.technical) for an example). +For this, please create a Dockerfile containing installation steps for the additional dependencies (have a look at [docker/Dockerfile.technical](https://github.com/freqtrade/freqtrade/blob/develop/docker/Dockerfile.technical) for an example). You'll then also need to modify the `docker-compose.yml` file and uncomment the build step, as well as rename the image to avoid naming collisions. @@ -160,3 +160,32 @@ You'll then also need to modify the `docker-compose.yml` file and uncomment the ``` You can then run `docker-compose build` to build the docker image, and run it using the commands described above. + +## Plotting with docker-compose + +Commands `freqtrade plot-profit` and `freqtrade plot-dataframe` ([Documentation](plotting.md)) are available by changing the image to `*_plot` in your docker-compose.yml file. +You can then use these commands as follows: + +``` bash +docker-compose run --rm freqtrade plot-dataframe --strategy AwesomeStrategy -p BTC/ETH --timerange=20180801-20180805 +``` + +The output will be stored in the `user_data/plot` directory, and can be opened with any modern browser. + +## Data analayis using docker compose + +Freqtrade provides a docker-compose file which starts up a jupyter lab server. +You can run this server using the following command: + +``` bash +docker-compose --rm -f docker/docker-compose-jupyter.yml up +``` + +This will create a dockercontainer running jupyter lab, which will be accessible using `https://127.0.0.1:8888/lab`. +Please use the link that's printed in the console after startup for simplified login. + +Since part of this image is built on your machine, it is recommended to rebuild the image from time to time to keep freqtrade (and dependencies) uptodate. + +``` bash +docker-compose -f docker/docker-compose-jupyter.yml build --no-cache +``` diff --git a/docs/edge.md b/docs/edge.md index 500c3c833..7442f1927 100644 --- a/docs/edge.md +++ b/docs/edge.md @@ -82,20 +82,34 @@ Risk Reward Ratio ($R$) is a formula used to measure the expected gains of a giv $$ R = \frac{\text{potential_profit}}{\text{potential_loss}} $$ ???+ Example "Worked example of $R$ calculation" - Let's say that you think that the price of *stonecoin* today is $10.0. You believe that, because they will start mining stonecoin, it will go up to $15.0 tomorrow. There is the risk that the stone is too hard, and the GPUs can't mine it, so the price might go to $0 tomorrow. You are planning to invest $100.
- Your potential profit is calculated as:
+ Let's say that you think that the price of *stonecoin* today is $10.0. You believe that, because they will start mining stonecoin, it will go up to $15.0 tomorrow. There is the risk that the stone is too hard, and the GPUs can't mine it, so the price might go to $0 tomorrow. You are planning to invest $100, which will give you 10 shares (100 / 10). + + Your potential profit is calculated as: + $\begin{aligned} - \text{potential_profit} &= (\text{potential_price} - \text{cost_per_unit}) * \frac{\text{investment}}{\text{cost_per_unit}} \\ - &= (15 - 10) * \frac{100}{15}\\ - &= 33.33 - \end{aligned}$
- Since the price might go to $0, the $100 dolars invested could turn into 0. We can compute the Risk Reward Ratio as follows:
+ \text{potential_profit} &= (\text{potential_price} - \text{entry_price}) * \frac{\text{investment}}{\text{entry_price}} \\ + &= (15 - 10) * (100 / 10) \\ + &= 50 + \end{aligned}$ + + Since the price might go to $0, the $100 dollars invested could turn into 0. + + We do however use a stoploss of 15% - so in the worst case, we'll sell 15% below entry price (or at 8.5$). + + $\begin{aligned} + \text{potential_loss} &= (\text{entry_price} - \text{stoploss}) * \frac{\text{investment}}{\text{entry_price}} \\ + &= (10 - 8.5) * (100 / 10)\\ + &= 15 + \end{aligned}$ + + We can compute the Risk Reward Ratio as follows: + $\begin{aligned} R &= \frac{\text{potential_profit}}{\text{potential_loss}}\\ - &= \frac{33.33}{100}\\ - &= 0.333... + &= \frac{50}{15}\\ + &= 3.33 \end{aligned}$
- What it effectivelly means is that the strategy have the potential to make $0.33 for each $1 invested. + What it effectively means is that the strategy have the potential to make 3.33$ for each $1 invested. On a long horizon, that is, on many trades, we can calculate the risk reward by dividing the strategy' average profit on winning trades by the strategy' average loss on losing trades. We can calculate the average profit, $\mu_{win}$, as follows: diff --git a/docs/faq.md b/docs/faq.md index 48f52a566..a775060de 100644 --- a/docs/faq.md +++ b/docs/faq.md @@ -89,7 +89,7 @@ Same fix should be done in the configuration file, if order types are defined in ### How do I search the bot logs for something? -By default, the bot writes its log into stderr stream. This is implemented this way so that you can easily separate the bot's diagnostics messages from Backtesting, Edge and Hyperopt results, output from other various Freqtrade utility subcommands, as well as from the output of your custom `print()`'s you may have inserted into your strategy. So if you need to search the log messages with the grep utility, you need to redirect stderr to stdout and disregard stdout. +By default, the bot writes its log into stderr stream. This is implemented this way so that you can easily separate the bot's diagnostics messages from Backtesting, Edge and Hyperopt results, output from other various Freqtrade utility sub-commands, as well as from the output of your custom `print()`'s you may have inserted into your strategy. So if you need to search the log messages with the grep utility, you need to redirect stderr to stdout and disregard stdout. * In unix shells, this normally can be done as simple as: ```shell @@ -114,7 +114,7 @@ and then grep it as: ```shell $ cat /path/to/mylogfile.log | grep 'something' ``` -or even on the fly, as the bot works and the logfile grows: +or even on the fly, as the bot works and the log file grows: ```shell $ tail -f /path/to/mylogfile.log | grep 'something' ``` @@ -137,23 +137,17 @@ compute. Since hyperopt uses Bayesian search, running for too many epochs may not produce greater results. -It's therefore recommended to run between 500-1000 epochs over and over until you hit at least 10.000 epocs in total (or are satisfied with the result). You can best judge by looking at the results - if the bot keeps discovering better strategies, it's best to keep on going. +It's therefore recommended to run between 500-1000 epochs over and over until you hit at least 10.000 epochs in total (or are satisfied with the result). You can best judge by looking at the results - if the bot keeps discovering better strategies, it's best to keep on going. ```bash -freqtrade hyperopt -e 1000 -``` - -or if you want intermediate result to see - -```bash -for i in {1..100}; do freqtrade hyperopt -e 1000; done +freqtrade hyperopt --hyperop SampleHyperopt --hyperopt-loss SharpeHyperOptLossDaily --strategy SampleStrategy -e 1000 ``` ### Why does it take a long time to run hyperopt? * Discovering a great strategy with Hyperopt takes time. Study www.freqtrade.io, the Freqtrade Documentation page, join the Freqtrade [Slack community](https://join.slack.com/t/highfrequencybot/shared_invite/enQtNjU5ODcwNjI1MDU3LTU1MTgxMjkzNmYxNWE1MDEzYzQ3YmU4N2MwZjUyNjJjODRkMDVkNjg4YTAyZGYzYzlhOTZiMTE4ZjQ4YzM0OGE) - or the Freqtrade [discord community](https://discord.gg/X89cVG). While you patiently wait for the most advanced, free crypto bot in the world, to hand you a possible golden strategy specially designed just for you. -* If you wonder why it can take from 20 minutes to days to do 1000 epocs here are some answers: +* If you wonder why it can take from 20 minutes to days to do 1000 epochs here are some answers: This answer was written during the release 0.15.1, when we had: @@ -167,7 +161,7 @@ already 8\*10^9\*10 evaluations. A roughly total of 80 billion evals. Did you run 100 000 evals? Congrats, you've done roughly 1 / 100 000 th of the search space, assuming that the bot never tests the same parameters more than once. -* The time it takes to run 1000 hyperopt epocs depends on things like: The available cpu, harddisk, ram, timeframe, timerange, indicator settings, indicator count, amount of coins that hyperopt test strategies on and the resulting trade count - which can be 650 trades in a year or 10.0000 trades depending if the strategy aims for big profits by trading rarely or for many low profit trades. +* The time it takes to run 1000 hyperopt epochs depends on things like: The available cpu, hard-disk, ram, timeframe, timerange, indicator settings, indicator count, amount of coins that hyperopt test strategies on and the resulting trade count - which can be 650 trades in a year or 10.0000 trades depending if the strategy aims for big profits by trading rarely or for many low profit trades. Example: 4% profit 650 times vs 0,3% profit a trade 10.000 times in a year. If we assume you set the --timerange to 365 days. @@ -180,7 +174,7 @@ Example: The Edge module is mostly a result of brainstorming of [@mishaker](https://github.com/mishaker) and [@creslinux](https://github.com/creslinux) freqtrade team members. -You can find further info on expectancy, winrate, risk management and position size in the following sources: +You can find further info on expectancy, win rate, risk management and position size in the following sources: - https://www.tradeciety.com/ultimate-math-guide-for-traders/ - http://www.vantharp.com/tharp-concepts/expectancy.asp diff --git a/docs/hyperopt.md b/docs/hyperopt.md index 530faf700..fc7a0dd93 100644 --- a/docs/hyperopt.md +++ b/docs/hyperopt.md @@ -37,12 +37,20 @@ pip install -r requirements-hyperopt.txt Before we start digging into Hyperopt, we recommend you to take a look at the sample hyperopt file located in [user_data/hyperopts/](https://github.com/freqtrade/freqtrade/blob/develop/freqtrade/templates/sample_hyperopt.py). -Configuring hyperopt is similar to writing your own strategy, and many tasks will be similar and a lot of code can be copied across from the strategy. +Configuring hyperopt is similar to writing your own strategy, and many tasks will be similar. -The simplest way to get started is to use `freqtrade new-hyperopt --hyperopt AwesomeHyperopt`. -This will create a new hyperopt file from a template, which will be located under `user_data/hyperopts/AwesomeHyperopt.py`. +!!! Tip "About this page" + For this page, we will be using a fictional strategy called `AwesomeStrategy` - which will be optimized using the `AwesomeHyperopt` class. -### Checklist on all tasks / possibilities in hyperopt +The simplest way to get started is to use the following, command, which will create a new hyperopt file from a template, which will be located under `user_data/hyperopts/AwesomeHyperopt.py`. + +``` bash +freqtrade new-hyperopt --hyperopt AwesomeHyperopt +``` + +### Hyperopt checklist + +Checklist on all tasks / possibilities in hyperopt Depending on the space you want to optimize, only some of the below are required: @@ -54,17 +62,15 @@ Depending on the space you want to optimize, only some of the below are required !!! Note `populate_indicators` needs to create all indicators any of thee spaces may use, otherwise hyperopt will not work. -Optional - can also be loaded from a strategy: +Optional in hyperopt - can also be loaded from a strategy (recommended): * copy `populate_indicators` from your strategy - otherwise default-strategy will be used * copy `populate_buy_trend` from your strategy - otherwise default-strategy will be used * copy `populate_sell_trend` from your strategy - otherwise default-strategy will be used -!!! Note - Assuming the optional methods are not in your hyperopt file, please use `--strategy AweSomeStrategy` which contains these methods so hyperopt can use these methods instead. - !!! Note You always have to provide a strategy to Hyperopt, even if your custom Hyperopt class contains all methods. + Assuming the optional methods are not in your hyperopt file, please use `--strategy AweSomeStrategy` which contains these methods so hyperopt can use these methods instead. Rarely you may also need to override: @@ -80,17 +86,20 @@ Rarely you may also need to override: # Have a working strategy at hand. freqtrade new-hyperopt --hyperopt EmptyHyperopt - freqtrade hyperopt --hyperopt EmptyHyperopt --spaces roi stoploss trailing --strategy MyWorkingStrategy --config config.json -e 100 + freqtrade hyperopt --hyperopt EmptyHyperopt --hyperopt-loss SharpeHyperOptLossDaily --spaces roi stoploss trailing --strategy MyWorkingStrategy --config config.json -e 100 ``` -### 1. Install a Custom Hyperopt File +### Create a Custom Hyperopt File -Put your hyperopt file into the directory `user_data/hyperopts`. +Let assume you want a hyperopt file `AwesomeHyperopt.py`: -Let assume you want a hyperopt file `awesome_hyperopt.py`: -Copy the file `user_data/hyperopts/sample_hyperopt.py` into `user_data/hyperopts/awesome_hyperopt.py` +``` bash +freqtrade new-hyperopt --hyperopt AwesomeHyperopt +``` -### 2. Configure your Guards and Triggers +This command will create a new hyperopt file from a template, allowing you to get started quickly. + +### Configure your Guards and Triggers There are two places you need to change in your hyperopt file to add a new buy hyperopt for testing: @@ -102,14 +111,16 @@ There you have two different types of indicators: 1. `guards` and 2. `triggers`. 1. Guards are conditions like "never buy if ADX < 10", or never buy if current price is over EMA10. 2. Triggers are ones that actually trigger buy in specific moment, like "buy when EMA5 crosses over EMA10" or "buy when close price touches lower Bollinger band". -Hyperoptimization will, for each eval round, pick one trigger and possibly -multiple guards. The constructed strategy will be something like -"*buy exactly when close price touches lower Bollinger band, BUT only if +!!! Hint "Guards and Triggers" + Technically, there is no difference between Guards and Triggers. + However, this guide will make this distinction to make it clear that signals should not be "sticking". + Sticking signals are signals that are active for multiple candles. This can lead into buying a signal late (right before the signal disappears - which means that the chance of success is a lot lower than right at the beginning). + +Hyper-optimization will, for each epoch round, pick one trigger and possibly +multiple guards. The constructed strategy will be something like "*buy exactly when close price touches lower Bollinger band, BUT only if ADX > 10*". -If you have updated the buy strategy, i.e. changed the contents of -`populate_buy_trend()` method, you have to update the `guards` and -`triggers` your hyperopt must use correspondingly. +If you have updated the buy strategy, i.e. changed the contents of `populate_buy_trend()` method, you have to update the `guards` and `triggers` your hyperopt must use correspondingly. #### Sell optimization @@ -126,7 +137,7 @@ To avoid naming collisions in the search-space, please prefix all sell-spaces wi The Strategy class exposes the timeframe value as the `self.timeframe` attribute. The same value is available as class-attribute `HyperoptName.timeframe`. -In the case of the linked sample-value this would be `SampleHyperOpt.timeframe`. +In the case of the linked sample-value this would be `AwesomeHyperopt.timeframe`. ## Solving a Mystery @@ -154,7 +165,7 @@ We will start by defining a search space: Above definition says: I have five parameters I want you to randomly combine to find the best combination. Two of them are integer values (`adx-value` -and `rsi-value`) and I want you test in the range of values 20 to 40. +and `rsi-value`) and I want you test in the range of values 20 to 40. Then we have three category variables. First two are either `True` or `False`. We use these to either enable or disable the ADX and RSI guards. The last one we call `trigger` and use it to decide which buy trigger we want to use. @@ -192,27 +203,25 @@ So let's write the buy strategy using these values: return populate_buy_trend ``` -Hyperopting will now call this `populate_buy_trend` as many times you ask it (`epochs`) -with different value combinations. It will then use the given historical data and make -buys based on the buy signals generated with the above function and based on the results -it will end with telling you which parameter combination produced the best profits. +Hyperopt will now call `populate_buy_trend()` many times (`epochs`) with different value combinations. +It will use the given historical data and make buys based on the buy signals generated with the above function. +Based on the results, hyperopt will tell you which parameter combination produced the best results (based on the configured [loss function](#loss-functions)). -The above setup expects to find ADX, RSI and Bollinger Bands in the populated indicators. -When you want to test an indicator that isn't used by the bot currently, remember to -add it to the `populate_indicators()` method in your custom hyperopt file. +!!! Note + The above setup expects to find ADX, RSI and Bollinger Bands in the populated indicators. + When you want to test an indicator that isn't used by the bot currently, remember to + add it to the `populate_indicators()` method in your strategy or hyperopt file. ## Loss-functions Each hyperparameter tuning requires a target. This is usually defined as a loss function (sometimes also called objective function), which should decrease for more desirable results, and increase for bad results. -By default, Freqtrade uses a loss function, which has been with freqtrade since the beginning and optimizes mostly for short trade duration and avoiding losses. - -A different loss function can be specified by using the `--hyperopt-loss ` argument. +A loss function must be specified via the `--hyperopt-loss ` argument (or optionally via the configuration under the `"hyperopt_loss"` key). This class should be in its own file within the `user_data/hyperopts/` directory. Currently, the following loss functions are builtin: -* `DefaultHyperOptLoss` (default legacy Freqtrade hyperoptimization loss function) +* `ShortTradeDurHyperOptLoss` (default legacy Freqtrade hyperoptimization loss function) - Mostly for short trade duration and avoiding losses. * `OnlyProfitHyperOptLoss` (which takes only amount of profit into consideration) * `SharpeHyperOptLoss` (optimizes Sharpe Ratio calculated on trade returns relative to standard deviation) * `SharpeHyperOptLossDaily` (optimizes Sharpe Ratio calculated on **daily** trade returns relative to standard deviation) @@ -229,21 +238,20 @@ Because hyperopt tries a lot of combinations to find the best parameters it will We strongly recommend to use `screen` or `tmux` to prevent any connection loss. ```bash -freqtrade hyperopt --config config.json --hyperopt -e 5000 --spaces all +freqtrade hyperopt --config config.json --hyperopt --hyperopt-loss --strategy -e 500 --spaces all ``` Use `` as the name of the custom hyperopt used. -The `-e` option will set how many evaluations hyperopt will do. We recommend -running at least several thousand evaluations. +The `-e` option will set how many evaluations hyperopt will do. Since hyperopt uses Bayesian search, running too many epochs at once may not produce greater results. Experience has shown that best results are usually not improving much after 500-1000 epochs. +Doing multiple runs (executions) with a few 1000 epochs and different random state will most likely produce different results. The `--spaces all` option determines that all possible parameters should be optimized. Possibilities are listed below. !!! Note - By default, hyperopt will erase previous results and start from scratch. Continuation can be archived by using `--continue`. - -!!! Warning - When switching parameters or changing configuration options, make sure to not use the argument `--continue` so temporary results can be removed. + Hyperopt will store hyperopt results with the timestamp of the hyperopt start time. + Reading commands (`hyperopt-list`, `hyperopt-show`) can use `--hyperopt-filename ` to read and display older hyperopt results. + You can find a list of filenames with `ls -l user_data/hyperopt_results/`. ### Execute Hyperopt with different historical data source @@ -251,13 +259,13 @@ If you would like to hyperopt parameters using an alternate historical data set you have on-disk, use the `--datadir PATH` option. By default, hyperopt uses data from directory `user_data/data`. -### Running Hyperopt with Smaller Testset +### Running Hyperopt with a smaller test-set -Use the `--timerange` argument to change how much of the testset you want to use. +Use the `--timerange` argument to change how much of the test-set you want to use. For example, to use one month of data, pass the following parameter to the hyperopt call: ```bash -freqtrade hyperopt --timerange 20180401-20180501 +freqtrade hyperopt --hyperopt --strategy --timerange 20180401-20180501 ``` ### Running Hyperopt using methods from a strategy @@ -265,16 +273,15 @@ freqtrade hyperopt --timerange 20180401-20180501 Hyperopt can reuse `populate_indicators`, `populate_buy_trend`, `populate_sell_trend` from your strategy, assuming these methods are **not** in your custom hyperopt file, and a strategy is provided. ```bash -freqtrade hyperopt --strategy SampleStrategy --hyperopt SampleHyperopt +freqtrade hyperopt --hyperopt AwesomeHyperopt --hyperopt-loss SharpeHyperOptLossDaily --strategy AwesomeStrategy ``` ### Running Hyperopt with Smaller Search Space Use the `--spaces` option to limit the search space used by hyperopt. -Letting Hyperopt optimize everything is a huuuuge search space. Often it -might make more sense to start by just searching for initial buy algorithm. -Or maybe you just want to optimize your stoploss or roi table for that awesome -new buy strategy you have. +Letting Hyperopt optimize everything is a huuuuge search space. +Often it might make more sense to start by just searching for initial buy algorithm. +Or maybe you just want to optimize your stoploss or roi table for that awesome new buy strategy you have. Legal values are: @@ -318,7 +325,7 @@ The initial state for generation of these random values (random state) is contro If you have not set this value explicitly in the command line options, Hyperopt seeds the random state with some random value for you. The random state value for each Hyperopt run is shown in the log, so you can copy and paste it into the `--random-state` command line option to repeat the set of the initial random epochs used. -If you have not changed anything in the command line options, configuration, timerange, Strategy and Hyperopt classes, historical data and the Loss Function -- you should obtain same hyperoptimization results with same random state value used. +If you have not changed anything in the command line options, configuration, timerange, Strategy and Hyperopt classes, historical data and the Loss Function -- you should obtain same hyper-optimization results with same random state value used. ## Understand the Hyperopt Result @@ -371,7 +378,7 @@ By default, hyperopt prints colorized results -- epochs with positive profit are You can use the `--print-all` command line option if you would like to see all results in the hyperopt output, not only the best ones. When `--print-all` is used, current best results are also colorized by default -- they are printed in bold (bright) style. This can also be switched off with the `--no-color` command line option. !!! Note "Windows and color output" - Windows does not support color-output nativly, therefore it is automatically disabled. To have color-output for hyperopt running under windows, please consider using WSL. + Windows does not support color-output natively, therefore it is automatically disabled. To have color-output for hyperopt running under windows, please consider using WSL. ### Understand Hyperopt ROI results @@ -419,7 +426,9 @@ These ranges should be sufficient in most cases. The minutes in the steps (ROI d If you have the `generate_roi_table()` and `roi_space()` methods in your custom hyperopt file, remove them in order to utilize these adaptive ROI tables and the ROI hyperoptimization space generated by Freqtrade by default. -Override the `roi_space()` method if you need components of the ROI tables to vary in other ranges. Override the `generate_roi_table()` and `roi_space()` methods and implement your own custom approach for generation of the ROI tables during hyperoptimization if you need a different structure of the ROI tables or other amount of rows (steps). A sample for these methods can be found in [user_data/hyperopts/sample_hyperopt_advanced.py](https://github.com/freqtrade/freqtrade/blob/develop/freqtrade/templates/sample_hyperopt_advanced.py). +Override the `roi_space()` method if you need components of the ROI tables to vary in other ranges. Override the `generate_roi_table()` and `roi_space()` methods and implement your own custom approach for generation of the ROI tables during hyperoptimization if you need a different structure of the ROI tables or other amount of rows (steps). + +A sample for these methods can be found in [sample_hyperopt_advanced.py](https://github.com/freqtrade/freqtrade/blob/develop/freqtrade/templates/sample_hyperopt_advanced.py). ### Understand Hyperopt Stoploss results @@ -441,7 +450,7 @@ Stoploss: -0.27996 In order to use this best stoploss value found by Hyperopt in backtesting and for live trades/dry-run, copy-paste it as the value of the `stoploss` attribute of your custom strategy: -``` +``` python # Optimal stoploss designed for the strategy # This attribute will be overridden if the config file contains "stoploss" stoploss = -0.27996 @@ -475,7 +484,7 @@ Trailing stop: In order to use these best trailing stop parameters found by Hyperopt in backtesting and for live trades/dry-run, copy-paste them as the values of the corresponding attributes of your custom strategy: -``` +``` python # Trailing stop # These attributes will be overridden if the config file contains corresponding values. trailing_stop = True @@ -494,10 +503,14 @@ Override the `trailing_space()` method and define the desired range in it if you ## Show details of Hyperopt results -After you run Hyperopt for the desired amount of epochs, you can later list all results for analysis, select only best or profitable once, and show the details for any of the epochs previously evaluated. This can be done with the `hyperopt-list` and `hyperopt-show` subcommands. The usage of these subcommands is described in the [Utils](utils.md#list-hyperopt-results) chapter. +After you run Hyperopt for the desired amount of epochs, you can later list all results for analysis, select only best or profitable once, and show the details for any of the epochs previously evaluated. This can be done with the `hyperopt-list` and `hyperopt-show` sub-commands. The usage of these sub-commands is described in the [Utils](utils.md#list-hyperopt-results) chapter. ## Validate backtesting results Once the optimized strategy has been implemented into your strategy, you should backtest this strategy to make sure everything is working as expected. -To achieve same results (number of trades, their durations, profit, etc.) than during Hyperopt, please use same set of arguments `--dmmp`/`--disable-max-market-positions` and `--eps`/`--enable-position-stacking` for Backtesting. +To achieve same results (number of trades, their durations, profit, etc.) than during Hyperopt, please use same configuration and parameters (timerange, timeframe, ...) used for hyperopt `--dmmp`/`--disable-max-market-positions` and `--eps`/`--enable-position-stacking` for Backtesting. + +Should results don't match, please double-check to make sure you transferred all conditions correctly. +Pay special care to the stoploss (and trailing stoploss) parameters, as these are often set in configuration files, which override changes to the strategy. +You should also carefully review the log of your backtest to ensure that there were no parameters inadvertently set by the configuration (like `stoploss` or `trailing_stop`). diff --git a/docs/includes/pairlists.md b/docs/includes/pairlists.md new file mode 100644 index 000000000..ae4ec818d --- /dev/null +++ b/docs/includes/pairlists.md @@ -0,0 +1,137 @@ +## Pairlists and Pairlist Handlers + +Pairlist Handlers define the list of pairs (pairlist) that the bot should trade. They are configured in the `pairlists` section of the configuration settings. + +In your configuration, you can use Static Pairlist (defined by the [`StaticPairList`](#static-pair-list) Pairlist Handler) and Dynamic Pairlist (defined by the [`VolumePairList`](#volume-pair-list) Pairlist Handler). + +Additionally, [`AgeFilter`](#agefilter), [`PrecisionFilter`](#precisionfilter), [`PriceFilter`](#pricefilter), [`ShuffleFilter`](#shufflefilter) and [`SpreadFilter`](#spreadfilter) act as Pairlist Filters, removing certain pairs and/or moving their positions in the pairlist. + +If multiple Pairlist Handlers are used, they are chained and a combination of all Pairlist Handlers forms the resulting pairlist the bot uses for trading and backtesting. Pairlist Handlers are executed in the sequence they are configured. You should always configure either `StaticPairList` or `VolumePairList` as the starting Pairlist Handler. + +Inactive markets are always removed from the resulting pairlist. Explicitly blacklisted pairs (those in the `pair_blacklist` configuration setting) are also always removed from the resulting pairlist. + +### Available Pairlist Handlers + +* [`StaticPairList`](#static-pair-list) (default, if not configured differently) +* [`VolumePairList`](#volume-pair-list) +* [`AgeFilter`](#agefilter) +* [`PrecisionFilter`](#precisionfilter) +* [`PriceFilter`](#pricefilter) +* [`ShuffleFilter`](#shufflefilter) +* [`SpreadFilter`](#spreadfilter) + +!!! Tip "Testing pairlists" + Pairlist configurations can be quite tricky to get right. Best use the [`test-pairlist`](utils.md#test-pairlist) utility sub-command to test your configuration quickly. + +#### Static Pair List + +By default, the `StaticPairList` method is used, which uses a statically defined pair whitelist from the configuration. + +It uses configuration from `exchange.pair_whitelist` and `exchange.pair_blacklist`. + +```json +"pairlists": [ + {"method": "StaticPairList"} + ], +``` + +#### Volume Pair List + +`VolumePairList` employs sorting/filtering of pairs by their trading volume. It selects `number_assets` top pairs with sorting based on the `sort_key` (which can only be `quoteVolume`). + +When used in the chain of Pairlist Handlers in a non-leading position (after StaticPairList and other Pairlist Filters), `VolumePairList` considers outputs of previous Pairlist Handlers, adding its sorting/selection of the pairs by the trading volume. + +When used on the leading position of the chain of Pairlist Handlers, it does not consider `pair_whitelist` configuration setting, but selects the top assets from all available markets (with matching stake-currency) on the exchange. + +The `refresh_period` setting allows to define the period (in seconds), at which the pairlist will be refreshed. Defaults to 1800s (30 minutes). + +`VolumePairList` is based on the ticker data from exchange, as reported by the ccxt library: + +* The `quoteVolume` is the amount of quote (stake) currency traded (bought or sold) in last 24 hours. + +```json +"pairlists": [{ + "method": "VolumePairList", + "number_assets": 20, + "sort_key": "quoteVolume", + "refresh_period": 1800, +}], +``` + +#### AgeFilter + +Removes pairs that have been listed on the exchange for less than `min_days_listed` days (defaults to `10`). + +When pairs are first listed on an exchange they can suffer huge price drops and volatility +in the first few days while the pair goes through its price-discovery period. Bots can often +be caught out buying before the pair has finished dropping in price. + +This filter allows freqtrade to ignore pairs until they have been listed for at least `min_days_listed` days. + +#### PrecisionFilter + +Filters low-value coins which would not allow setting stoplosses. + +#### PriceFilter + +The `PriceFilter` allows filtering of pairs by price. Currently the following price filters are supported: + +* `min_price` +* `max_price` +* `low_price_ratio` + +The `min_price` setting removes pairs where the price is below the specified price. This is useful if you wish to avoid trading very low-priced pairs. +This option is disabled by default, and will only apply if set to > 0. + +The `max_price` setting removes pairs where the price is above the specified price. This is useful if you wish to trade only low-priced pairs. +This option is disabled by default, and will only apply if set to > 0. + +The `low_price_ratio` setting removes pairs where a raise of 1 price unit (pip) is above the `low_price_ratio` ratio. +This option is disabled by default, and will only apply if set to > 0. + +For `PriceFiler` at least one of its `min_price`, `max_price` or `low_price_ratio` settings must be applied. + +Calculation example: + +Min price precision for SHITCOIN/BTC is 8 decimals. If its price is 0.00000011 - one price step above would be 0.00000012, which is ~9% higher than the previous price value. You may filter out this pair by using PriceFilter with `low_price_ratio` set to 0.09 (9%) or with `min_price` set to 0.00000011, correspondingly. + +!!! Warning "Low priced pairs" + Low priced pairs with high "1 pip movements" are dangerous since they are often illiquid and it may also be impossible to place the desired stoploss, which can often result in high losses since price needs to be rounded to the next tradable price - so instead of having a stoploss of -5%, you could end up with a stoploss of -9% simply due to price rounding. + +#### ShuffleFilter + +Shuffles (randomizes) pairs in the pairlist. It can be used for preventing the bot from trading some of the pairs more frequently then others when you want all pairs be treated with the same priority. + +!!! Tip + You may set the `seed` value for this Pairlist to obtain reproducible results, which can be useful for repeated backtesting sessions. If `seed` is not set, the pairs are shuffled in the non-repeatable random order. + +#### SpreadFilter + +Removes pairs that have a difference between asks and bids above the specified ratio, `max_spread_ratio` (defaults to `0.005`). + +Example: + +If `DOGE/BTC` maximum bid is 0.00000026 and minimum ask is 0.00000027, the ratio is calculated as: `1 - bid/ask ~= 0.037` which is `> 0.005` and this pair will be filtered out. + +### Full example of Pairlist Handlers + +The below example blacklists `BNB/BTC`, uses `VolumePairList` with `20` assets, sorting pairs by `quoteVolume` and applies both [`PrecisionFilter`](#precisionfilter) and [`PriceFilter`](#price-filter), filtering all assets where 1 price unit is > 1%. Then the `SpreadFilter` is applied and pairs are finally shuffled with the random seed set to some predefined value. + +```json +"exchange": { + "pair_whitelist": [], + "pair_blacklist": ["BNB/BTC"] +}, +"pairlists": [ + { + "method": "VolumePairList", + "number_assets": 20, + "sort_key": "quoteVolume", + }, + {"method": "AgeFilter", "min_days_listed": 10}, + {"method": "PrecisionFilter"}, + {"method": "PriceFilter", "low_price_ratio": 0.01}, + {"method": "SpreadFilter", "max_spread_ratio": 0.005}, + {"method": "ShuffleFilter", "seed": 42} + ], +``` diff --git a/docs/index.md b/docs/index.md index 397c549aa..5608587db 100644 --- a/docs/index.md +++ b/docs/index.md @@ -8,7 +8,7 @@ Fork -Download +Download Follow @freqtrade @@ -59,11 +59,17 @@ Alternatively ## Support -### Help / Slack +### Help / Slack / Discord + For any questions not covered by the documentation or for further information about the bot, we encourage you to join our passionate Slack community. Click [here](https://join.slack.com/t/highfrequencybot/shared_invite/enQtNjU5ODcwNjI1MDU3LTU1MTgxMjkzNmYxNWE1MDEzYzQ3YmU4N2MwZjUyNjJjODRkMDVkNjg4YTAyZGYzYzlhOTZiMTE4ZjQ4YzM0OGE) to join the Freqtrade Slack channel. +Alternatively, check out the newly created [discord server](https://discord.gg/MA9v74M). + +!!! Note + Since the discord server is relatively new, answers to questions might be slightly delayed as currently the user base quite small. + ## Ready to try? -Begin by reading our installation guide [for docker](docker.md), or for [installation without docker](installation.md). +Begin by reading our installation guide [for docker](docker.md) (recommended), or for [installation without docker](installation.md). diff --git a/docs/installation.md b/docs/installation.md index 5f4807b99..9b15c9685 100644 --- a/docs/installation.md +++ b/docs/installation.md @@ -31,7 +31,7 @@ Freqtrade provides the Linux/MacOS Easy Installation script to install all depen The easiest way to install and run Freqtrade is to clone the bot Github repository and then run the Easy Installation script, if it's available for your platform. !!! Note "Version considerations" - When cloning the repository the default working branch has the name `develop`. This branch contains all last features (can be considered as relatively stable, thanks to automated tests). The `master` branch contains the code of the last release (done usually once per month on an approximately one week old snapshot of the `develop` branch to prevent packaging bugs, so potentially it's more stable). + When cloning the repository the default working branch has the name `develop`. This branch contains all last features (can be considered as relatively stable, thanks to automated tests). The `stable` branch contains the code of the last release (done usually once per month on an approximately one week old snapshot of the `develop` branch to prevent packaging bugs, so potentially it's more stable). !!! Note Python3.6 or higher and the corresponding `pip` are assumed to be available. The install-script will warn you and stop if that's not the case. `git` is also needed to clone the Freqtrade repository. @@ -41,11 +41,11 @@ This can be achieved with the following commands: ```bash git clone https://github.com/freqtrade/freqtrade.git cd freqtrade -# git checkout master # Optional, see (1) +# git checkout stable # Optional, see (1) ./setup.sh --install ``` -(1) This command switches the cloned repository to the use of the `master` branch. It's not needed if you wish to stay on the `develop` branch. You may later switch between branches at any time with the `git checkout master`/`git checkout develop` commands. +(1) This command switches the cloned repository to the use of the `stable` branch. It's not needed if you wish to stay on the `develop` branch. You may later switch between branches at any time with the `git checkout stable`/`git checkout develop` commands. ## Easy Installation Script (Linux/MacOS) @@ -56,7 +56,7 @@ $ ./setup.sh usage: -i,--install Install freqtrade from scratch -u,--update Command git pull to update. - -r,--reset Hard reset your develop/master branch. + -r,--reset Hard reset your develop/stable branch. -c,--config Easy config generator (Will override your existing file). ``` @@ -76,7 +76,7 @@ This option will pull the last version of your current branch and update your vi ** --reset ** -This option will hard reset your branch (only if you are on either `master` or `develop`) and recreate your virtualenv. +This option will hard reset your branch (only if you are on either `stable` or `develop`) and recreate your virtualenv. ** --config ** @@ -174,7 +174,7 @@ Clone the git repository: ```bash git clone https://github.com/freqtrade/freqtrade.git cd freqtrade -git checkout master +git checkout stable ``` #### 4. Install python dependencies diff --git a/docs/requirements-docs.txt b/docs/requirements-docs.txt index 6408616a0..f30710a1f 100644 --- a/docs/requirements-docs.txt +++ b/docs/requirements-docs.txt @@ -1,2 +1,3 @@ -mkdocs-material==5.5.12 +mkdocs-material==6.1.0 mdx_truly_sane_lists==1.2 +pymdown-extensions==8.0.1 diff --git a/docs/rest-api.md b/docs/rest-api.md index 075bd7e64..7726ab875 100644 --- a/docs/rest-api.md +++ b/docs/rest-api.md @@ -104,32 +104,42 @@ By default, the script assumes `127.0.0.1` (localhost) and port `8080` to be use python3 scripts/rest_client.py --config rest_config.json [optional parameters] ``` -## Available commands +## Available endpoints | Command | Description | |----------|-------------| | `ping` | Simple command testing the API Readiness - requires no authentication. -| `start` | Starts the trader -| `stop` | Stops the trader +| `start` | Starts the trader. +| `stop` | Stops the trader. | `stopbuy` | Stops the trader from opening new trades. Gracefully closes open trades according to their rules. -| `reload_config` | Reloads the configuration file +| `reload_config` | Reloads the configuration file. | `trades` | List last trades. | `delete_trade ` | Remove trade from the database. Tries to close open orders. Requires manual handling of this trade on the exchange. -| `show_config` | Shows part of the current configuration with relevant settings to operation -| `logs` | Shows last log messages -| `status` | Lists all open trades -| `count` | Displays number of trades used and available -| `profit` | Display a summary of your profit/loss from close trades and some stats about your performance +| `show_config` | Shows part of the current configuration with relevant settings to operation. +| `logs` | Shows last log messages. +| `status` | Lists all open trades. +| `count` | Displays number of trades used and available. +| `locks` | Displays currently locked pairs. +| `profit` | Display a summary of your profit/loss from close trades and some stats about your performance. | `forcesell ` | Instantly sells the given trade (Ignoring `minimum_roi`). | `forcesell all` | Instantly sells all open trades (Ignoring `minimum_roi`). | `forcebuy [rate]` | Instantly buys the given pair. Rate is optional. (`forcebuy_enable` must be set to True) -| `performance` | Show performance of each finished trade grouped by pair -| `balance` | Show account balance per currency -| `daily ` | Shows profit or loss per day, over the last n days (n defaults to 7) -| `whitelist` | Show the current whitelist +| `performance` | Show performance of each finished trade grouped by pair. +| `balance` | Show account balance per currency. +| `daily ` | Shows profit or loss per day, over the last n days (n defaults to 7). +| `whitelist` | Show the current whitelist. | `blacklist [pair]` | Show the current blacklist, or adds a pair to the blacklist. | `edge` | Show validated pairs by Edge if it is enabled. -| `version` | Show version +| `pair_candles` | Returns dataframe for a pair / timeframe combination while the bot is running. **Alpha** +| `pair_history` | Returns an analyzed dataframe for a given timerange, analyzed by a given strategy. **Alpha** +| `plot_config` | Get plot config from the strategy (or nothing if not configured). **Alpha** +| `strategies` | List strategies in strategy directory. **Alpha** +| `strategy ` | Get specific Strategy content. **Alpha** +| `available_pairs` | List available backtest data. **Alpha** +| `version` | Show version. + +!!! Warning "Alpha status" + Endpoints labeled with *Alpha status* above may change at any time without notice. Possible commands can be listed from the rest-client script using the `help` command. @@ -140,6 +150,12 @@ python3 scripts/rest_client.py help ``` output Possible commands: +available_pairs + Return available pair (backtest data) based on timeframe / stake_currency selection + + :param timeframe: Only pairs with this timeframe available. + :param stake_currency: Only pairs that include this timeframe + balance Get the account balance. @@ -179,9 +195,27 @@ logs :param limit: Limits log messages to the last logs. No limit to get all the trades. +pair_candles + Return live dataframe for . + + :param pair: Pair to get data for + :param timeframe: Only pairs with this timeframe available. + :param limit: Limit result to the last n candles. + +pair_history + Return historic, analyzed dataframe + + :param pair: Pair to get data for + :param timeframe: Only pairs with this timeframe available. + :param strategy: Strategy to analyze and get values for + :param timerange: Timerange to get data for (same format than --timerange endpoints) + performance Return the performance of the different coins. +plot_config + Return plot configuration if the strategy defines one. + profit Return the profit summary. @@ -204,6 +238,14 @@ stop stopbuy Stop buying (but handle sells gracefully). Use `reload_config` to reset. +strategies + Lists available strategies + +strategy + Get strategy details + + :param strategy: Strategy class name + trades Return trades history. @@ -215,7 +257,6 @@ version whitelist Show the current whitelist. - ``` ## Advanced API usage using JWT tokens diff --git a/docs/sql_cheatsheet.md b/docs/sql_cheatsheet.md index 168d416ab..569af33ff 100644 --- a/docs/sql_cheatsheet.md +++ b/docs/sql_cheatsheet.md @@ -43,52 +43,6 @@ sqlite3 .schema ``` -### Trade table structure - -```sql -CREATE TABLE trades( - id INTEGER NOT NULL, - exchange VARCHAR NOT NULL, - pair VARCHAR NOT NULL, - is_open BOOLEAN NOT NULL, - fee_open FLOAT NOT NULL, - fee_open_cost FLOAT, - fee_open_currency VARCHAR, - fee_close FLOAT NOT NULL, - fee_close_cost FLOAT, - fee_close_currency VARCHAR, - open_rate FLOAT, - open_rate_requested FLOAT, - open_trade_price FLOAT, - close_rate FLOAT, - close_rate_requested FLOAT, - close_profit FLOAT, - close_profit_abs FLOAT, - stake_amount FLOAT NOT NULL, - amount FLOAT, - open_date DATETIME NOT NULL, - close_date DATETIME, - open_order_id VARCHAR, - stop_loss FLOAT, - stop_loss_pct FLOAT, - initial_stop_loss FLOAT, - initial_stop_loss_pct FLOAT, - stoploss_order_id VARCHAR, - stoploss_last_update DATETIME, - max_rate FLOAT, - min_rate FLOAT, - sell_reason VARCHAR, - strategy VARCHAR, - timeframe INTEGER, - PRIMARY KEY (id), - CHECK (is_open IN (0, 1)) -); -CREATE INDEX ix_trades_stoploss_order_id ON trades (stoploss_order_id); -CREATE INDEX ix_trades_pair ON trades (pair); -CREATE INDEX ix_trades_is_open ON trades (is_open); - -``` - ## Get all trades in the table ```sql @@ -98,11 +52,11 @@ SELECT * FROM trades; ## Fix trade still open after a manual sell on the exchange !!! Warning - Manually selling a pair on the exchange will not be detected by the bot and it will try to sell anyway. Whenever possible, forcesell should be used to accomplish the same thing. - It is strongly advised to backup your database file before making any manual changes. + Manually selling a pair on the exchange will not be detected by the bot and it will try to sell anyway. Whenever possible, forcesell should be used to accomplish the same thing. + It is strongly advised to backup your database file before making any manual changes. !!! Note - This should not be necessary after /forcesell, as forcesell orders are closed automatically by the bot on the next iteration. + This should not be necessary after /forcesell, as forcesell orders are closed automatically by the bot on the next iteration. ```sql UPDATE trades @@ -128,23 +82,12 @@ SET is_open=0, WHERE id=31; ``` -## Manually insert a new trade - -```sql -INSERT INTO trades (exchange, pair, is_open, fee_open, fee_close, open_rate, stake_amount, amount, open_date) -VALUES ('binance', 'ETH/BTC', 1, 0.0025, 0.0025, , , , '') -``` - -### Insert trade example - -```sql -INSERT INTO trades (exchange, pair, is_open, fee_open, fee_close, open_rate, stake_amount, amount, open_date) -VALUES ('binance', 'ETH/BTC', 1, 0.0025, 0.0025, 0.00258580, 0.002, 0.7715262081, '2020-06-28 12:44:24.000000') -``` - ## Remove trade from the database -Maybe you'd like to remove a trade from the database, because something went wrong. +!!! Tip "Use RPC Methods to delete trades" + Consider using `/delete ` via telegram or rest API. That's the recommended way to deleting trades. + +If you'd still like to remove a trade from the database directly, you can use the below query. ```sql DELETE FROM trades WHERE id = ; diff --git a/docs/strategy-customization.md b/docs/strategy-customization.md index 14d5fcd84..c0506203f 100644 --- a/docs/strategy-customization.md +++ b/docs/strategy-customization.md @@ -312,12 +312,17 @@ The name of the variable can be chosen at will, but should be prefixed with `cus class Awesomestrategy(IStrategy): # Create custom dictionary cust_info = {} + def populate_indicators(self, dataframe: DataFrame, metadata: dict) -> DataFrame: # Check if the entry already exists + if not metadata["pair"] in self._cust_info: + # Create empty entry for this pair + self._cust_info[metadata["pair"]] = {} + if "crosstime" in self.cust_info[metadata["pair"]: - self.cust_info[metadata["pair"]["crosstime"] += 1 + self.cust_info[metadata["pair"]]["crosstime"] += 1 else: - self.cust_info[metadata["pair"]["crosstime"] = 1 + self.cust_info[metadata["pair"]]["crosstime"] = 1 ``` !!! Warning @@ -688,15 +693,15 @@ Locked pairs will show the message `Pair is currently locked.`. Sometimes it may be desired to lock a pair after certain events happen (e.g. multiple losing trades in a row). -Freqtrade has an easy method to do this from within the strategy, by calling `self.lock_pair(pair, until)`. -`until` must be a datetime object in the future, after which trading will be reenabled for that pair. +Freqtrade has an easy method to do this from within the strategy, by calling `self.lock_pair(pair, until, [reason])`. +`until` must be a datetime object in the future, after which trading will be re-enabled for that pair, while `reason` is an optional string detailing why the pair was locked. Locks can also be lifted manually, by calling `self.unlock_pair(pair)`. To verify if a pair is currently locked, use `self.is_pair_locked(pair)`. !!! Note - Locked pairs are not persisted, so a restart of the bot, or calling `/reload_config` will reset locked pairs. + Locked pairs will always be rounded up to the next candle. So assuming a `5m` timeframe, a lock with `until` set to 10:18 will lock the pair until the candle from 10:15-10:20 will be finished. !!! Warning Locking pairs is not functioning during backtesting. diff --git a/docs/telegram-usage.md b/docs/telegram-usage.md index 5f804386d..ce2d715a0 100644 --- a/docs/telegram-usage.md +++ b/docs/telegram-usage.md @@ -41,6 +41,34 @@ Talk to the [userinfobot](https://telegram.me/userinfobot) Get your "Id", you will use it for the config parameter `chat_id`. +## Control telegram noise + +Freqtrade provides means to control the verbosity of your telegram bot. +Each setting has the following possible values: + +* `on` - Messages will be sent, and user will be notified. +* `silent` - Message will be sent, Notification will be without sound / vibration. +* `off` - Skip sending a message-type all together. + +Example configuration showing the different settings: + +``` json +"telegram": { + "enabled": true, + "token": "your_telegram_token", + "chat_id": "your_telegram_chat_id", + "notification_settings": { + "status": "silent", + "warning": "on", + "startup": "off", + "buy": "silent", + "sell": "on", + "buy_cancel": "silent", + "sell_cancel": "on" + } + }, +``` + ## Telegram commands Per default, the Telegram bot shows predefined commands. Some commands diff --git a/docs/utils.md b/docs/utils.md index 8c7e381ff..409bcc134 100644 --- a/docs/utils.md +++ b/docs/utils.md @@ -423,7 +423,7 @@ freqtrade test-pairlist --config config.json --quote USDT BTC ## List Hyperopt results -You can list the hyperoptimization epochs the Hyperopt module evaluated previously with the `hyperopt-list` subcommand. +You can list the hyperoptimization epochs the Hyperopt module evaluated previously with the `hyperopt-list` sub-command. ``` usage: freqtrade hyperopt-list [-h] [-v] [--logfile FILE] [-V] [-c PATH] @@ -432,10 +432,11 @@ usage: freqtrade hyperopt-list [-h] [-v] [--logfile FILE] [-V] [-c PATH] [--max-trades INT] [--min-avg-time FLOAT] [--max-avg-time FLOAT] [--min-avg-profit FLOAT] [--max-avg-profit FLOAT] - [--min-total-profit FLOAT] [--max-total-profit FLOAT] + [--min-total-profit FLOAT] + [--max-total-profit FLOAT] [--min-objective FLOAT] [--max-objective FLOAT] [--no-color] [--print-json] [--no-details] - [--export-csv FILE] + [--hyperopt-filename PATH] [--export-csv FILE] optional arguments: -h, --help show this help message and exit @@ -443,24 +444,27 @@ optional arguments: --profitable Select only profitable epochs. --min-trades INT Select epochs with more than INT trades. --max-trades INT Select epochs with less than INT trades. - --min-avg-time FLOAT Select epochs on above average time. - --max-avg-time FLOAT Select epochs on under average time. + --min-avg-time FLOAT Select epochs above average time. + --max-avg-time FLOAT Select epochs below average time. --min-avg-profit FLOAT - Select epochs on above average profit. + Select epochs above average profit. --max-avg-profit FLOAT - Select epochs on below average profit. + Select epochs below average profit. --min-total-profit FLOAT - Select epochs on above total profit. + Select epochs above total profit. --max-total-profit FLOAT - Select epochs on below total profit. + Select epochs below total profit. --min-objective FLOAT - Select epochs on above objective (- is added by default). + Select epochs above objective. --max-objective FLOAT - Select epochs on below objective (- is added by default). + Select epochs below objective. --no-color Disable colorization of hyperopt results. May be useful if you are redirecting output to a file. - --print-json Print best result detailization in JSON format. + --print-json Print output in JSON format. --no-details Do not print best epoch details. + --hyperopt-filename FILENAME + Hyperopt result filename.Example: `--hyperopt- + filename=hyperopt_results_2020-09-27_16-20-48.pickle` --export-csv FILE Export to CSV-File. This will disable table print. Example: --export-csv hyperopt.csv @@ -480,7 +484,11 @@ Common arguments: --userdir PATH, --user-data-dir PATH Path to userdata directory. ``` - + +!!! Note + `hyperopt-list` will automatically use the latest available hyperopt results file. + You can override this using the `--hyperopt-filename` argument, and specify another, available filename (without path!). + ### Examples List all results, print details of the best result at the end: @@ -501,17 +509,41 @@ You can show the details of any hyperoptimization epoch previously evaluated by usage: freqtrade hyperopt-show [-h] [-v] [--logfile FILE] [-V] [-c PATH] [-d PATH] [--userdir PATH] [--best] [--profitable] [-n INT] [--print-json] - [--no-header] + [--hyperopt-filename PATH] [--no-header] optional arguments: -h, --help show this help message and exit --best Select only best epochs. --profitable Select only profitable epochs. -n INT, --index INT Specify the index of the epoch to print details for. - --print-json Print best result detailization in JSON format. + --print-json Print output in JSON format. + --hyperopt-filename FILENAME + Hyperopt result filename.Example: `--hyperopt- + filename=hyperopt_results_2020-09-27_16-20-48.pickle` --no-header Do not print epoch details header. + +Common arguments: + -v, --verbose Verbose mode (-vv for more, -vvv to get all messages). + --logfile FILE Log to the file specified. Special values are: + 'syslog', 'journald'. See the documentation for more + details. + -V, --version show program's version number and exit + -c PATH, --config PATH + Specify configuration file (default: + `userdir/config.json` or `config.json` whichever + exists). Multiple --config options may be used. Can be + set to `-` to read config from stdin. + -d PATH, --datadir PATH + Path to directory with historical backtesting data. + --userdir PATH, --user-data-dir PATH + Path to userdata directory. + ``` +!!! Note + `hyperopt-show` will automatically use the latest available hyperopt results file. + You can override this using the `--hyperopt-filename` argument, and specify another, available filename (without path!). + ### Examples Print details for the epoch 168 (the number of the epoch is shown by the `hyperopt-list` subcommand or by Hyperopt itself during hyperoptimization run): diff --git a/docs/windows_installation.md b/docs/windows_installation.md index f7900d85a..0ef0f131f 100644 --- a/docs/windows_installation.md +++ b/docs/windows_installation.md @@ -21,7 +21,7 @@ git clone https://github.com/freqtrade/freqtrade.git Install ta-lib according to the [ta-lib documentation](https://github.com/mrjbq7/ta-lib#windows). -As compiling from source on windows has heavy dependencies (requires a partial visual studio installation), there is also a repository of unofficial precompiled windows Wheels [here](https://www.lfd.uci.edu/~gohlke/pythonlibs/#ta-lib), which needs to be downloaded and installed using `pip install TA_Lib‑0.4.18‑cp38‑cp38‑win_amd64.whl` (make sure to use the version matching your python version) +As compiling from source on windows has heavy dependencies (requires a partial visual studio installation), there is also a repository of unofficial precompiled windows Wheels [here](https://www.lfd.uci.edu/~gohlke/pythonlibs/#ta-lib), which needs to be downloaded and installed using `pip install TA_Lib‑0.4.19‑cp38‑cp38‑win_amd64.whl` (make sure to use the version matching your python version) Freqtrade provides these dependencies for the latest 2 Python versions (3.7 and 3.8) and for 64bit Windows. Other versions must be downloaded from the above link. @@ -32,7 +32,7 @@ python -m venv .env .env\Scripts\activate.ps1 # optionally install ta-lib from wheel # Eventually adjust the below filename to match the downloaded wheel -pip install build_helpes/TA_Lib‑0.4.18‑cp38‑cp38‑win_amd64.whl +pip install build_helpes/TA_Lib‑0.4.19‑cp38‑cp38‑win_amd64.whl pip install -r requirements.txt pip install -e . freqtrade diff --git a/freqtrade/__main__.py b/freqtrade/__main__.py index 97ed9ae67..881a2f562 100644 --- a/freqtrade/__main__.py +++ b/freqtrade/__main__.py @@ -8,5 +8,6 @@ To launch Freqtrade as a module from freqtrade import main + if __name__ == '__main__': main.main() diff --git a/freqtrade/commands/__init__.py b/freqtrade/commands/__init__.py index 4ce3eb421..21c5d6812 100644 --- a/freqtrade/commands/__init__.py +++ b/freqtrade/commands/__init__.py @@ -8,23 +8,15 @@ Note: Be careful with file-scoped imports in these subfiles. """ from freqtrade.commands.arguments import Arguments from freqtrade.commands.build_config_commands import start_new_config -from freqtrade.commands.data_commands import (start_convert_data, - start_download_data, +from freqtrade.commands.data_commands import (start_convert_data, start_download_data, start_list_data) -from freqtrade.commands.deploy_commands import (start_create_userdir, - start_new_hyperopt, +from freqtrade.commands.deploy_commands import (start_create_userdir, start_new_hyperopt, start_new_strategy) -from freqtrade.commands.hyperopt_commands import (start_hyperopt_list, - start_hyperopt_show) -from freqtrade.commands.list_commands import (start_list_exchanges, - start_list_hyperopts, - start_list_markets, - start_list_strategies, - start_list_timeframes, - start_show_trades) -from freqtrade.commands.optimize_commands import (start_backtesting, - start_edge, start_hyperopt) +from freqtrade.commands.hyperopt_commands import start_hyperopt_list, start_hyperopt_show +from freqtrade.commands.list_commands import (start_list_exchanges, start_list_hyperopts, + start_list_markets, start_list_strategies, + start_list_timeframes, start_show_trades) +from freqtrade.commands.optimize_commands import start_backtesting, start_edge, start_hyperopt from freqtrade.commands.pairlist_commands import start_test_pairlist -from freqtrade.commands.plot_commands import (start_plot_dataframe, - start_plot_profit) +from freqtrade.commands.plot_commands import start_plot_dataframe, start_plot_profit from freqtrade.commands.trade_commands import start_trading diff --git a/freqtrade/commands/arguments.py b/freqtrade/commands/arguments.py index 7268c3c8f..aa58ff585 100644 --- a/freqtrade/commands/arguments.py +++ b/freqtrade/commands/arguments.py @@ -9,6 +9,7 @@ from typing import Any, Dict, List, Optional from freqtrade.commands.cli_options import AVAILABLE_CLI_OPTIONS from freqtrade.constants import DEFAULT_CONFIG + ARGS_COMMON = ["verbosity", "logfile", "version", "config", "datadir", "user_data_dir"] ARGS_STRATEGY = ["strategy", "strategy_path"] @@ -26,7 +27,7 @@ ARGS_HYPEROPT = ARGS_COMMON_OPTIMIZE + ["hyperopt", "hyperopt_path", "use_max_market_positions", "print_all", "print_colorized", "print_json", "hyperopt_jobs", "hyperopt_random_state", "hyperopt_min_trades", - "hyperopt_continue", "hyperopt_loss"] + "hyperopt_loss"] ARGS_EDGE = ARGS_COMMON_OPTIMIZE + ["stoploss_range"] @@ -56,7 +57,7 @@ ARGS_CONVERT_DATA_OHLCV = ARGS_CONVERT_DATA + ["timeframes"] ARGS_LIST_DATA = ["exchange", "dataformat_ohlcv", "pairs"] -ARGS_DOWNLOAD_DATA = ["pairs", "pairs_file", "days", "download_trades", "exchange", +ARGS_DOWNLOAD_DATA = ["pairs", "pairs_file", "days", "timerange", "download_trades", "exchange", "timeframes", "erase", "dataformat_ohlcv", "dataformat_trades"] ARGS_PLOT_DATAFRAME = ["pairs", "indicators1", "indicators2", "plot_limit", @@ -75,10 +76,10 @@ ARGS_HYPEROPT_LIST = ["hyperopt_list_best", "hyperopt_list_profitable", "hyperopt_list_min_total_profit", "hyperopt_list_max_total_profit", "hyperopt_list_min_objective", "hyperopt_list_max_objective", "print_colorized", "print_json", "hyperopt_list_no_details", - "export_csv"] + "hyperoptexportfilename", "export_csv"] ARGS_HYPEROPT_SHOW = ["hyperopt_list_best", "hyperopt_list_profitable", "hyperopt_show_index", - "print_json", "hyperopt_show_no_header"] + "print_json", "hyperoptexportfilename", "hyperopt_show_no_header"] NO_CONF_REQURIED = ["convert-data", "convert-trade-data", "download-data", "list-timeframes", "list-markets", "list-pairs", "list-strategies", "list-data", @@ -161,16 +162,14 @@ class Arguments: self.parser = argparse.ArgumentParser(description='Free, open source crypto trading bot') self._build_args(optionlist=['version'], parser=self.parser) - from freqtrade.commands import (start_create_userdir, start_convert_data, - start_download_data, start_list_data, - start_hyperopt_list, start_hyperopt_show, + from freqtrade.commands import (start_backtesting, start_convert_data, start_create_userdir, + start_download_data, start_edge, start_hyperopt, + start_hyperopt_list, start_hyperopt_show, start_list_data, start_list_exchanges, start_list_hyperopts, start_list_markets, start_list_strategies, - start_list_timeframes, start_new_config, - start_new_hyperopt, start_new_strategy, - start_plot_dataframe, start_plot_profit, start_show_trades, - start_backtesting, start_hyperopt, start_edge, - start_test_pairlist, start_trading) + start_list_timeframes, start_new_config, start_new_hyperopt, + start_new_strategy, start_plot_dataframe, start_plot_profit, + start_show_trades, start_test_pairlist, start_trading) subparsers = self.parser.add_subparsers(dest='command', # Use custom message when no subhandler is added diff --git a/freqtrade/commands/build_config_commands.py b/freqtrade/commands/build_config_commands.py index 0c98b2e55..7bdbcc057 100644 --- a/freqtrade/commands/build_config_commands.py +++ b/freqtrade/commands/build_config_commands.py @@ -1,13 +1,15 @@ import logging from pathlib import Path -from typing import Any, Dict +from typing import Any, Dict, List from questionary import Separator, prompt from freqtrade.constants import UNLIMITED_STAKE_AMOUNT -from freqtrade.exchange import available_exchanges, MAP_EXCHANGE_CHILDCLASS -from freqtrade.misc import render_template from freqtrade.exceptions import OperationalException +from freqtrade.exchange import MAP_EXCHANGE_CHILDCLASS, available_exchanges +from freqtrade.misc import render_template + + logger = logging.getLogger(__name__) @@ -46,7 +48,7 @@ def ask_user_config() -> Dict[str, Any]: Interactive questions built using https://github.com/tmbo/questionary :returns: Dict with keys to put into template """ - questions = [ + questions: List[Dict[str, Any]] = [ { "type": "confirm", "name": "dry_run", diff --git a/freqtrade/commands/cli_options.py b/freqtrade/commands/cli_options.py index 8eb5c3ce8..8ea945ae7 100644 --- a/freqtrade/commands/cli_options.py +++ b/freqtrade/commands/cli_options.py @@ -252,23 +252,20 @@ AVAILABLE_CLI_OPTIONS = { metavar='INT', default=1, ), - "hyperopt_continue": Arg( - "--continue", - help="Continue hyperopt from previous runs. " - "By default, temporary files will be removed and hyperopt will start from scratch.", - default=False, - action='store_true', - ), "hyperopt_loss": Arg( '--hyperopt-loss', help='Specify the class name of the hyperopt loss function class (IHyperOptLoss). ' 'Different functions can generate completely different results, ' 'since the target for optimization is different. Built-in Hyperopt-loss-functions are: ' - 'DefaultHyperOptLoss, OnlyProfitHyperOptLoss, SharpeHyperOptLoss, SharpeHyperOptLossDaily, ' - 'SortinoHyperOptLoss, SortinoHyperOptLossDaily.' - '(default: `%(default)s`).', + 'ShortTradeDurHyperOptLoss, OnlyProfitHyperOptLoss, SharpeHyperOptLoss, ' + 'SharpeHyperOptLossDaily, SortinoHyperOptLoss, SortinoHyperOptLossDaily.', metavar='NAME', - default=constants.DEFAULT_HYPEROPT_LOSS, + ), + "hyperoptexportfilename": Arg( + '--hyperopt-filename', + help='Hyperopt result filename.' + 'Example: `--hyperopt-filename=hyperopt_results_2020-09-27_16-20-48.pickle`', + metavar='FILENAME', ), # List exchanges "print_one_column": Arg( @@ -375,7 +372,7 @@ AVAILABLE_CLI_OPTIONS = { help='Specify which tickers to download. Space-separated list. ' 'Default: `1m 5m`.', choices=['1m', '3m', '5m', '15m', '30m', '1h', '2h', '4h', - '6h', '8h', '12h', '1d', '3d', '1w'], + '6h', '8h', '12h', '1d', '3d', '1w', '2w', '1M', '1y'], default=['1m', '5m'], nargs='+', ), diff --git a/freqtrade/commands/data_commands.py b/freqtrade/commands/data_commands.py index da1eb0cf5..7102eee38 100644 --- a/freqtrade/commands/data_commands.py +++ b/freqtrade/commands/data_commands.py @@ -6,16 +6,15 @@ from typing import Any, Dict, List import arrow from freqtrade.configuration import TimeRange, setup_utils_configuration -from freqtrade.data.converter import (convert_ohlcv_format, - convert_trades_format) -from freqtrade.data.history import (convert_trades_to_ohlcv, - refresh_backtest_ohlcv_data, +from freqtrade.data.converter import convert_ohlcv_format, convert_trades_format +from freqtrade.data.history import (convert_trades_to_ohlcv, refresh_backtest_ohlcv_data, refresh_backtest_trades_data) from freqtrade.exceptions import OperationalException from freqtrade.exchange import timeframe_to_minutes from freqtrade.resolvers import ExchangeResolver from freqtrade.state import RunMode + logger = logging.getLogger(__name__) @@ -25,11 +24,17 @@ def start_download_data(args: Dict[str, Any]) -> None: """ config = setup_utils_configuration(args, RunMode.UTIL_EXCHANGE) + if 'days' in config and 'timerange' in config: + raise OperationalException("--days and --timerange are mutually exclusive. " + "You can only specify one or the other.") timerange = TimeRange() if 'days' in config: time_since = arrow.utcnow().shift(days=-config['days']).strftime("%Y%m%d") timerange = TimeRange.parse_timerange(f'{time_since}-') + if 'timerange' in config: + timerange = timerange.parse_timerange(config['timerange']) + if 'pairs' not in config: raise OperationalException( "Downloading data requires a list of pairs. " @@ -99,8 +104,9 @@ def start_list_data(args: Dict[str, Any]) -> None: config = setup_utils_configuration(args, RunMode.UTIL_NO_EXCHANGE) - from freqtrade.data.history.idatahandler import get_datahandler from tabulate import tabulate + + from freqtrade.data.history.idatahandler import get_datahandler dhc = get_datahandler(config['datadir'], config['dataformat_ohlcv']) paircombs = dhc.ohlcv_get_available_data(config['datadir']) diff --git a/freqtrade/commands/deploy_commands.py b/freqtrade/commands/deploy_commands.py index bfd68cb9b..0a49c55de 100644 --- a/freqtrade/commands/deploy_commands.py +++ b/freqtrade/commands/deploy_commands.py @@ -4,13 +4,13 @@ from pathlib import Path from typing import Any, Dict from freqtrade.configuration import setup_utils_configuration -from freqtrade.configuration.directory_operations import (copy_sample_files, - create_userdata_dir) +from freqtrade.configuration.directory_operations import copy_sample_files, create_userdata_dir from freqtrade.constants import USERPATH_HYPEROPTS, USERPATH_STRATEGIES from freqtrade.exceptions import OperationalException from freqtrade.misc import render_template, render_template_with_fallback from freqtrade.state import RunMode + logger = logging.getLogger(__name__) diff --git a/freqtrade/commands/hyperopt_commands.py b/freqtrade/commands/hyperopt_commands.py index 4fae51e28..fd8f737f0 100755 --- a/freqtrade/commands/hyperopt_commands.py +++ b/freqtrade/commands/hyperopt_commands.py @@ -5,9 +5,11 @@ from typing import Any, Dict, List from colorama import init as colorama_init from freqtrade.configuration import setup_utils_configuration +from freqtrade.data.btanalysis import get_latest_hyperopt_file from freqtrade.exceptions import OperationalException from freqtrade.state import RunMode + logger = logging.getLogger(__name__) @@ -40,8 +42,9 @@ def start_hyperopt_list(args: Dict[str, Any]) -> None: 'filter_max_objective': config.get('hyperopt_list_max_objective', None), } - results_file = (config['user_data_dir'] / - 'hyperopt_results' / 'hyperopt_results.pickle') + results_file = get_latest_hyperopt_file( + config['user_data_dir'] / 'hyperopt_results', + config.get('hyperoptexportfilename')) # Previous evaluations epochs = Hyperopt.load_previous_results(results_file) @@ -80,8 +83,10 @@ def start_hyperopt_show(args: Dict[str, Any]) -> None: print_json = config.get('print_json', False) no_header = config.get('hyperopt_show_no_header', False) - results_file = (config['user_data_dir'] / - 'hyperopt_results' / 'hyperopt_results.pickle') + results_file = get_latest_hyperopt_file( + config['user_data_dir'] / 'hyperopt_results', + config.get('hyperoptexportfilename')) + n = config.get('hyperopt_show_index', -1) filteroptions = { diff --git a/freqtrade/commands/list_commands.py b/freqtrade/commands/list_commands.py index c8c820c61..9e6076dfb 100644 --- a/freqtrade/commands/list_commands.py +++ b/freqtrade/commands/list_commands.py @@ -5,20 +5,20 @@ from collections import OrderedDict from pathlib import Path from typing import Any, Dict, List -from colorama import init as colorama_init -from colorama import Fore, Style import rapidjson +from colorama import Fore, Style +from colorama import init as colorama_init from tabulate import tabulate from freqtrade.configuration import setup_utils_configuration from freqtrade.constants import USERPATH_HYPEROPTS, USERPATH_STRATEGIES from freqtrade.exceptions import OperationalException -from freqtrade.exchange import (available_exchanges, ccxt_exchanges, - market_is_active) +from freqtrade.exchange import available_exchanges, ccxt_exchanges, market_is_active from freqtrade.misc import plural from freqtrade.resolvers import ExchangeResolver, StrategyResolver from freqtrade.state import RunMode + logger = logging.getLogger(__name__) @@ -203,15 +203,16 @@ def start_show_trades(args: Dict[str, Any]) -> None: """ Show trades """ - from freqtrade.persistence import init, Trade import json + + from freqtrade.persistence import Trade, init_db config = setup_utils_configuration(args, RunMode.UTIL_NO_EXCHANGE) if 'db_url' not in config: raise OperationalException("--db-url is required for this command.") logger.info(f'Using DB: "{config["db_url"]}"') - init(config['db_url'], clean_open_orders=False) + init_db(config['db_url'], clean_open_orders=False) tfilter = [] if config.get('trade_ids'): diff --git a/freqtrade/commands/optimize_commands.py b/freqtrade/commands/optimize_commands.py index 2fc605926..7411ca9c6 100644 --- a/freqtrade/commands/optimize_commands.py +++ b/freqtrade/commands/optimize_commands.py @@ -6,6 +6,7 @@ from freqtrade.configuration import setup_utils_configuration from freqtrade.exceptions import DependencyException, OperationalException from freqtrade.state import RunMode + logger = logging.getLogger(__name__) @@ -58,6 +59,7 @@ def start_hyperopt(args: Dict[str, Any]) -> None: # Import here to avoid loading hyperopt module when it's not used try: from filelock import FileLock, Timeout + from freqtrade.optimize.hyperopt import Hyperopt except ImportError as e: raise OperationalException( @@ -98,6 +100,7 @@ def start_edge(args: Dict[str, Any]) -> None: :return: None """ from freqtrade.optimize.edge_cli import EdgeCli + # Initialize configuration config = setup_optimize_configuration(args, RunMode.EDGE) logger.info('Starting freqtrade in Edge mode') diff --git a/freqtrade/commands/pairlist_commands.py b/freqtrade/commands/pairlist_commands.py index 77bcb04b4..e4ee80ca5 100644 --- a/freqtrade/commands/pairlist_commands.py +++ b/freqtrade/commands/pairlist_commands.py @@ -7,6 +7,7 @@ from freqtrade.configuration import setup_utils_configuration from freqtrade.resolvers import ExchangeResolver from freqtrade.state import RunMode + logger = logging.getLogger(__name__) diff --git a/freqtrade/commands/trade_commands.py b/freqtrade/commands/trade_commands.py index c058e4f9d..535844844 100644 --- a/freqtrade/commands/trade_commands.py +++ b/freqtrade/commands/trade_commands.py @@ -1,5 +1,4 @@ import logging - from typing import Any, Dict diff --git a/freqtrade/configuration/__init__.py b/freqtrade/configuration/__init__.py index d41ac97ec..607f9cdef 100644 --- a/freqtrade/configuration/__init__.py +++ b/freqtrade/configuration/__init__.py @@ -1,7 +1,7 @@ # flake8: noqa: F401 -from freqtrade.configuration.config_setup import setup_utils_configuration from freqtrade.configuration.check_exchange import check_exchange, remove_credentials -from freqtrade.configuration.timerange import TimeRange -from freqtrade.configuration.configuration import Configuration +from freqtrade.configuration.config_setup import setup_utils_configuration from freqtrade.configuration.config_validation import validate_config_consistency +from freqtrade.configuration.configuration import Configuration +from freqtrade.configuration.timerange import TimeRange diff --git a/freqtrade/configuration/check_exchange.py b/freqtrade/configuration/check_exchange.py index 92daaf251..aa36de3ff 100644 --- a/freqtrade/configuration/check_exchange.py +++ b/freqtrade/configuration/check_exchange.py @@ -2,11 +2,11 @@ import logging from typing import Any, Dict from freqtrade.exceptions import OperationalException -from freqtrade.exchange import (available_exchanges, get_exchange_bad_reason, - is_exchange_bad, is_exchange_known_ccxt, - is_exchange_officially_supported) +from freqtrade.exchange import (available_exchanges, get_exchange_bad_reason, is_exchange_bad, + is_exchange_known_ccxt, is_exchange_officially_supported) from freqtrade.state import RunMode + logger = logging.getLogger(__name__) diff --git a/freqtrade/configuration/config_setup.py b/freqtrade/configuration/config_setup.py index 64f283e42..3b0f778f4 100644 --- a/freqtrade/configuration/config_setup.py +++ b/freqtrade/configuration/config_setup.py @@ -1,10 +1,12 @@ import logging from typing import Any, Dict +from freqtrade.state import RunMode + +from .check_exchange import remove_credentials from .config_validation import validate_config_consistency from .configuration import Configuration -from .check_exchange import remove_credentials -from freqtrade.state import RunMode + logger = logging.getLogger(__name__) diff --git a/freqtrade/configuration/config_validation.py b/freqtrade/configuration/config_validation.py index 5ba7ff294..d4612d8e0 100644 --- a/freqtrade/configuration/config_validation.py +++ b/freqtrade/configuration/config_validation.py @@ -9,6 +9,7 @@ from freqtrade import constants from freqtrade.exceptions import OperationalException from freqtrade.state import RunMode + logger = logging.getLogger(__name__) diff --git a/freqtrade/configuration/configuration.py b/freqtrade/configuration/configuration.py index 930917fae..1ca3187fb 100644 --- a/freqtrade/configuration/configuration.py +++ b/freqtrade/configuration/configuration.py @@ -10,14 +10,14 @@ from typing import Any, Callable, Dict, List, Optional from freqtrade import constants from freqtrade.configuration.check_exchange import check_exchange from freqtrade.configuration.deprecated_settings import process_temporary_deprecated_settings -from freqtrade.configuration.directory_operations import (create_datadir, - create_userdata_dir) +from freqtrade.configuration.directory_operations import create_datadir, create_userdata_dir from freqtrade.configuration.load_config import load_config_file from freqtrade.exceptions import OperationalException from freqtrade.loggers import setup_logging from freqtrade.misc import deep_merge_dicts, json_load from freqtrade.state import NON_UTIL_MODES, TRADING_MODES, RunMode + logger = logging.getLogger(__name__) @@ -263,6 +263,9 @@ class Configuration: self._args_to_config(config, argname='hyperopt_path', logstring='Using additional Hyperopt lookup path: {}') + self._args_to_config(config, argname='hyperoptexportfilename', + logstring='Using hyperopt file: {}') + self._args_to_config(config, argname='epochs', logstring='Parameter --epochs detected ... ' 'Will run Hyperopt with for {} epochs ...' @@ -295,9 +298,6 @@ class Configuration: self._args_to_config(config, argname='hyperopt_min_trades', logstring='Parameter --min-trades detected: {}') - self._args_to_config(config, argname='hyperopt_continue', - logstring='Hyperopt continue: {}') - self._args_to_config(config, argname='hyperopt_loss', logstring='Using Hyperopt loss class name: {}') diff --git a/freqtrade/configuration/directory_operations.py b/freqtrade/configuration/directory_operations.py index 6b8c8cb5a..51310f013 100644 --- a/freqtrade/configuration/directory_operations.py +++ b/freqtrade/configuration/directory_operations.py @@ -3,8 +3,9 @@ import shutil from pathlib import Path from typing import Any, Dict, Optional -from freqtrade.exceptions import OperationalException from freqtrade.constants import USER_DATA_FILES +from freqtrade.exceptions import OperationalException + logger = logging.getLogger(__name__) diff --git a/freqtrade/configuration/load_config.py b/freqtrade/configuration/load_config.py index a24ee3d0a..726126034 100644 --- a/freqtrade/configuration/load_config.py +++ b/freqtrade/configuration/load_config.py @@ -11,6 +11,7 @@ import rapidjson from freqtrade.exceptions import OperationalException + logger = logging.getLogger(__name__) diff --git a/freqtrade/constants.py b/freqtrade/constants.py index c71b94bcb..8e92d3ed8 100644 --- a/freqtrade/constants.py +++ b/freqtrade/constants.py @@ -11,7 +11,6 @@ DEFAULT_EXCHANGE = 'bittrex' PROCESS_THROTTLE_SECS = 5 # sec HYPEROPT_EPOCH = 100 # epochs RETRY_TIMEOUT = 30 # sec -DEFAULT_HYPEROPT_LOSS = 'DefaultHyperOptLoss' DEFAULT_DB_PROD_URL = 'sqlite:///tradesv3.sqlite' DEFAULT_DB_DRYRUN_URL = 'sqlite:///tradesv3.dryrun.sqlite' UNLIMITED_STAKE_AMOUNT = 'unlimited' @@ -39,6 +38,8 @@ USERPATH_HYPEROPTS = 'hyperopts' USERPATH_STRATEGIES = 'strategies' USERPATH_NOTEBOOKS = 'notebooks' +TELEGRAM_SETTING_OPTIONS = ['on', 'off', 'silent'] + # Soure files with destination directories within user-directory USER_DATA_FILES = { 'sample_strategy.py': USERPATH_STRATEGIES, @@ -201,6 +202,18 @@ CONF_SCHEMA = { 'enabled': {'type': 'boolean'}, 'token': {'type': 'string'}, 'chat_id': {'type': 'string'}, + 'notification_settings': { + 'type': 'object', + 'properties': { + 'status': {'type': 'string', 'enum': TELEGRAM_SETTING_OPTIONS}, + 'warning': {'type': 'string', 'enum': TELEGRAM_SETTING_OPTIONS}, + 'startup': {'type': 'string', 'enum': TELEGRAM_SETTING_OPTIONS}, + 'buy': {'type': 'string', 'enum': TELEGRAM_SETTING_OPTIONS}, + 'sell': {'type': 'string', 'enum': TELEGRAM_SETTING_OPTIONS}, + 'buy_cancel': {'type': 'string', 'enum': TELEGRAM_SETTING_OPTIONS}, + 'sell_cancel': {'type': 'string', 'enum': TELEGRAM_SETTING_OPTIONS} + } + } }, 'required': ['enabled', 'token', 'chat_id'] }, diff --git a/freqtrade/data/btanalysis.py b/freqtrade/data/btanalysis.py index 2d45a7222..513fba9e7 100644 --- a/freqtrade/data/btanalysis.py +++ b/freqtrade/data/btanalysis.py @@ -2,17 +2,17 @@ Helpers when analyzing backtest data """ import logging +from datetime import timezone from pathlib import Path -from typing import Dict, Union, Tuple, Any, Optional +from typing import Any, Dict, Optional, Tuple, Union import numpy as np import pandas as pd -from datetime import timezone -from freqtrade import persistence from freqtrade.constants import LAST_BT_RESULT_FN from freqtrade.misc import json_load -from freqtrade.persistence import Trade +from freqtrade.persistence import Trade, init_db + logger = logging.getLogger(__name__) @@ -21,10 +21,11 @@ BT_DATA_COLUMNS = ["pair", "profit_percent", "open_date", "close_date", "index", "open_rate", "close_rate", "open_at_end", "sell_reason"] -def get_latest_backtest_filename(directory: Union[Path, str]) -> str: +def get_latest_optimize_filename(directory: Union[Path, str], variant: str) -> str: """ Get latest backtest export based on '.last_result.json'. :param directory: Directory to search for last result + :param variant: 'backtest' or 'hyperopt' - the method to return :return: string containing the filename of the latest backtest result :raises: ValueError in the following cases: * Directory does not exist @@ -44,10 +45,57 @@ def get_latest_backtest_filename(directory: Union[Path, str]) -> str: with filename.open() as file: data = json_load(file) - if 'latest_backtest' not in data: + if f'latest_{variant}' not in data: raise ValueError(f"Invalid '{LAST_BT_RESULT_FN}' format.") - return data['latest_backtest'] + return data[f'latest_{variant}'] + + +def get_latest_backtest_filename(directory: Union[Path, str]) -> str: + """ + Get latest backtest export based on '.last_result.json'. + :param directory: Directory to search for last result + :return: string containing the filename of the latest backtest result + :raises: ValueError in the following cases: + * Directory does not exist + * `directory/.last_result.json` does not exist + * `directory/.last_result.json` has the wrong content + """ + return get_latest_optimize_filename(directory, 'backtest') + + +def get_latest_hyperopt_filename(directory: Union[Path, str]) -> str: + """ + Get latest hyperopt export based on '.last_result.json'. + :param directory: Directory to search for last result + :return: string containing the filename of the latest hyperopt result + :raises: ValueError in the following cases: + * Directory does not exist + * `directory/.last_result.json` does not exist + * `directory/.last_result.json` has the wrong content + """ + try: + return get_latest_optimize_filename(directory, 'hyperopt') + except ValueError: + # Return default (legacy) pickle filename + return 'hyperopt_results.pickle' + + +def get_latest_hyperopt_file(directory: Union[Path, str], predef_filename: str = None) -> Path: + """ + Get latest hyperopt export based on '.last_result.json'. + :param directory: Directory to search for last result + :return: string containing the filename of the latest hyperopt result + :raises: ValueError in the following cases: + * Directory does not exist + * `directory/.last_result.json` does not exist + * `directory/.last_result.json` has the wrong content + """ + if isinstance(directory, str): + directory = Path(directory) + if predef_filename: + return directory / predef_filename + return directory / get_latest_hyperopt_filename(directory) def load_backtest_stats(filename: Union[Path, str]) -> Dict[str, Any]: @@ -169,7 +217,7 @@ def load_trades_from_db(db_url: str, strategy: Optional[str] = None) -> pd.DataF Can also serve as protection to load the correct result. :return: Dataframe containing Trades """ - persistence.init(db_url, clean_open_orders=False) + init_db(db_url, clean_open_orders=False) columns = ["pair", "open_date", "close_date", "profit", "profit_percent", "open_rate", "close_rate", "amount", "trade_duration", "sell_reason", diff --git a/freqtrade/data/converter.py b/freqtrade/data/converter.py index 100a578a2..38fa670e9 100644 --- a/freqtrade/data/converter.py +++ b/freqtrade/data/converter.py @@ -10,8 +10,8 @@ from typing import Any, Dict, List import pandas as pd from pandas import DataFrame, to_datetime -from freqtrade.constants import (DEFAULT_DATAFRAME_COLUMNS, - DEFAULT_TRADES_COLUMNS) +from freqtrade.constants import DEFAULT_DATAFRAME_COLUMNS, DEFAULT_TRADES_COLUMNS + logger = logging.getLogger(__name__) diff --git a/freqtrade/data/dataprovider.py b/freqtrade/data/dataprovider.py index ccb6cbf56..07dd94fc1 100644 --- a/freqtrade/data/dataprovider.py +++ b/freqtrade/data/dataprovider.py @@ -17,6 +17,7 @@ from freqtrade.exceptions import ExchangeError, OperationalException from freqtrade.exchange import Exchange from freqtrade.state import RunMode + logger = logging.getLogger(__name__) diff --git a/freqtrade/data/history/__init__.py b/freqtrade/data/history/__init__.py index 23f635a98..107f9c401 100644 --- a/freqtrade/data/history/__init__.py +++ b/freqtrade/data/history/__init__.py @@ -5,10 +5,8 @@ Includes: * load data for a pair (or a list of pairs) from disk * download data from exchange and store to disk """ - -from .history_utils import (convert_trades_to_ohlcv, # noqa: F401 - get_timerange, load_data, load_pair_history, - refresh_backtest_ohlcv_data, - refresh_backtest_trades_data, refresh_data, +# flake8: noqa: F401 +from .history_utils import (convert_trades_to_ohlcv, get_timerange, load_data, load_pair_history, + refresh_backtest_ohlcv_data, refresh_backtest_trades_data, refresh_data, validate_backtest_data) -from .idatahandler import get_datahandler # noqa: F401 +from .idatahandler import get_datahandler diff --git a/freqtrade/data/history/hdf5datahandler.py b/freqtrade/data/history/hdf5datahandler.py index 594a1598a..f6cf9e0d9 100644 --- a/freqtrade/data/history/hdf5datahandler.py +++ b/freqtrade/data/history/hdf5datahandler.py @@ -7,12 +7,12 @@ import pandas as pd from freqtrade import misc from freqtrade.configuration import TimeRange -from freqtrade.constants import (DEFAULT_DATAFRAME_COLUMNS, - DEFAULT_TRADES_COLUMNS, +from freqtrade.constants import (DEFAULT_DATAFRAME_COLUMNS, DEFAULT_TRADES_COLUMNS, ListPairsWithTimeframes) from .idatahandler import IDataHandler, TradeList + logger = logging.getLogger(__name__) diff --git a/freqtrade/data/history/history_utils.py b/freqtrade/data/history/history_utils.py index dd09c4c05..a420b9dcc 100644 --- a/freqtrade/data/history/history_utils.py +++ b/freqtrade/data/history/history_utils.py @@ -9,15 +9,14 @@ from pandas import DataFrame from freqtrade.configuration import TimeRange from freqtrade.constants import DEFAULT_DATAFRAME_COLUMNS -from freqtrade.data.converter import (clean_ohlcv_dataframe, - ohlcv_to_dataframe, - trades_remove_duplicates, - trades_to_ohlcv) +from freqtrade.data.converter import (clean_ohlcv_dataframe, ohlcv_to_dataframe, + trades_remove_duplicates, trades_to_ohlcv) from freqtrade.data.history.idatahandler import IDataHandler, get_datahandler from freqtrade.exceptions import OperationalException from freqtrade.exchange import Exchange from freqtrade.misc import format_ms_time + logger = logging.getLogger(__name__) @@ -136,7 +135,6 @@ def _load_cached_data_for_updating(pair: str, timeframe: str, timerange: Optiona start = None if timerange: if timerange.starttype == 'date': - # TODO: convert to date for conversion start = datetime.fromtimestamp(timerange.startts, tz=timezone.utc) # Intentionally don't pass timerange in - since we need to load the full dataset. diff --git a/freqtrade/data/history/idatahandler.py b/freqtrade/data/history/idatahandler.py index 01b14f501..a170a9dc5 100644 --- a/freqtrade/data/history/idatahandler.py +++ b/freqtrade/data/history/idatahandler.py @@ -14,10 +14,10 @@ from pandas import DataFrame from freqtrade.configuration import TimeRange from freqtrade.constants import ListPairsWithTimeframes -from freqtrade.data.converter import (clean_ohlcv_dataframe, - trades_remove_duplicates, trim_dataframe) +from freqtrade.data.converter import clean_ohlcv_dataframe, trades_remove_duplicates, trim_dataframe from freqtrade.exchange import timeframe_to_seconds + logger = logging.getLogger(__name__) # Type for trades list diff --git a/freqtrade/data/history/jsondatahandler.py b/freqtrade/data/history/jsondatahandler.py index 2e7c0f773..6436aa13d 100644 --- a/freqtrade/data/history/jsondatahandler.py +++ b/freqtrade/data/history/jsondatahandler.py @@ -8,12 +8,12 @@ from pandas import DataFrame, read_json, to_datetime from freqtrade import misc from freqtrade.configuration import TimeRange -from freqtrade.constants import (DEFAULT_DATAFRAME_COLUMNS, - ListPairsWithTimeframes) +from freqtrade.constants import DEFAULT_DATAFRAME_COLUMNS, ListPairsWithTimeframes from freqtrade.data.converter import trades_dict_to_list from .idatahandler import IDataHandler, TradeList + logger = logging.getLogger(__name__) diff --git a/freqtrade/edge/edge_positioning.py b/freqtrade/edge/edge_positioning.py index 3213c1ef8..a40b63d67 100644 --- a/freqtrade/edge/edge_positioning.py +++ b/freqtrade/edge/edge_positioning.py @@ -9,11 +9,12 @@ import utils_find_1st as utf1st from pandas import DataFrame from freqtrade.configuration import TimeRange -from freqtrade.constants import UNLIMITED_STAKE_AMOUNT, DATETIME_PRINT_FORMAT -from freqtrade.exceptions import OperationalException +from freqtrade.constants import DATETIME_PRINT_FORMAT, UNLIMITED_STAKE_AMOUNT from freqtrade.data.history import get_timerange, load_data, refresh_data +from freqtrade.exceptions import OperationalException from freqtrade.strategy.interface import SellType + logger = logging.getLogger(__name__) @@ -309,8 +310,10 @@ class Edge: # Calculating number of losing trades, average win and average loss df['nb_loss_trades'] = df['nb_trades'] - df['nb_win_trades'] - df['average_win'] = df['profit_sum'] / df['nb_win_trades'] - df['average_loss'] = df['loss_sum'] / df['nb_loss_trades'] + df['average_win'] = np.where(df['nb_win_trades'] == 0, 0.0, + df['profit_sum'] / df['nb_win_trades']) + df['average_loss'] = np.where(df['nb_loss_trades'] == 0, 0.0, + df['loss_sum'] / df['nb_loss_trades']) # Win rate = number of profitable trades / number of trades df['winrate'] = df['nb_win_trades'] / df['nb_trades'] diff --git a/freqtrade/exceptions.py b/freqtrade/exceptions.py index e2bc969a9..caf970606 100644 --- a/freqtrade/exceptions.py +++ b/freqtrade/exceptions.py @@ -51,6 +51,13 @@ class RetryableOrderError(InvalidOrderException): """ +class InsufficientFundsError(InvalidOrderException): + """ + This error is used when there are not enough funds available on the exchange + to create an order. + """ + + class TemporaryError(ExchangeError): """ Temporary network or exchange related error. diff --git a/freqtrade/exchange/__init__.py b/freqtrade/exchange/__init__.py index bdf1f91ec..5b58d7a95 100644 --- a/freqtrade/exchange/__init__.py +++ b/freqtrade/exchange/__init__.py @@ -1,19 +1,16 @@ # flake8: noqa: F401 +# isort: off from freqtrade.exchange.common import MAP_EXCHANGE_CHILDCLASS from freqtrade.exchange.exchange import Exchange -from freqtrade.exchange.exchange import (get_exchange_bad_reason, - is_exchange_bad, - is_exchange_known_ccxt, - is_exchange_officially_supported, - ccxt_exchanges, - available_exchanges) -from freqtrade.exchange.exchange import (timeframe_to_seconds, - timeframe_to_minutes, - timeframe_to_msecs, - timeframe_to_next_date, - timeframe_to_prev_date) -from freqtrade.exchange.exchange import (market_is_active) -from freqtrade.exchange.kraken import Kraken -from freqtrade.exchange.binance import Binance +# isort: on from freqtrade.exchange.bibox import Bibox +from freqtrade.exchange.binance import Binance +from freqtrade.exchange.bittrex import Bittrex +from freqtrade.exchange.exchange import (available_exchanges, ccxt_exchanges, + get_exchange_bad_reason, is_exchange_bad, + is_exchange_known_ccxt, is_exchange_officially_supported, + market_is_active, timeframe_to_minutes, timeframe_to_msecs, + timeframe_to_next_date, timeframe_to_prev_date, + timeframe_to_seconds) from freqtrade.exchange.ftx import Ftx +from freqtrade.exchange.kraken import Kraken diff --git a/freqtrade/exchange/bibox.py b/freqtrade/exchange/bibox.py index 229abe766..f0c2dd00b 100644 --- a/freqtrade/exchange/bibox.py +++ b/freqtrade/exchange/bibox.py @@ -4,6 +4,7 @@ from typing import Dict from freqtrade.exchange import Exchange + logger = logging.getLogger(__name__) diff --git a/freqtrade/exchange/binance.py b/freqtrade/exchange/binance.py index f2fe1d6ad..099f282a2 100644 --- a/freqtrade/exchange/binance.py +++ b/freqtrade/exchange/binance.py @@ -4,12 +4,12 @@ from typing import Dict import ccxt -from freqtrade.exceptions import (DDosProtection, ExchangeError, - InvalidOrderException, OperationalException, - TemporaryError) +from freqtrade.exceptions import (DDosProtection, InsufficientFundsError, InvalidOrderException, + OperationalException, TemporaryError) from freqtrade.exchange import Exchange from freqtrade.exchange.common import retrier + logger = logging.getLogger(__name__) @@ -20,20 +20,9 @@ class Binance(Exchange): "order_time_in_force": ['gtc', 'fok', 'ioc'], "trades_pagination": "id", "trades_pagination_arg": "fromId", + "l2_limit_range": [5, 10, 20, 50, 100, 500, 1000], } - def fetch_l2_order_book(self, pair: str, limit: int = 100) -> dict: - """ - get order book level 2 from exchange - - 20180619: binance support limits but only on specific range - """ - limit_range = [5, 10, 20, 50, 100, 500, 1000] - # get next-higher step in the limit_range list - limit = min(list(filter(lambda x: limit <= x, limit_range))) - - return super().fetch_l2_order_book(pair, limit) - def stoploss_adjust(self, stop_loss: float, order: Dict) -> bool: """ Verify stop_loss against stoploss-order value (limit or price) @@ -80,7 +69,7 @@ class Binance(Exchange): 'stop price: %s. limit: %s', pair, stop_price, rate) return order except ccxt.InsufficientFunds as e: - raise ExchangeError( + raise InsufficientFundsError( f'Insufficient funds to create {ordertype} sell order on market {pair}. ' f'Tried to sell amount {amount} at rate {rate}. ' f'Message: {e}') from e diff --git a/freqtrade/exchange/bittrex.py b/freqtrade/exchange/bittrex.py new file mode 100644 index 000000000..4318f9cf0 --- /dev/null +++ b/freqtrade/exchange/bittrex.py @@ -0,0 +1,23 @@ +""" Bittrex exchange subclass """ +import logging +from typing import Dict + +from freqtrade.exchange import Exchange + + +logger = logging.getLogger(__name__) + + +class Bittrex(Exchange): + """ + Bittrex exchange class. Contains adjustments needed for Freqtrade to work + with this exchange. + + Please note that this exchange is not included in the list of exchanges + officially supported by the Freqtrade development team. So some features + may still not work as expected. + """ + + _ft_has: Dict = { + "l2_limit_range": [1, 25, 500], + } diff --git a/freqtrade/exchange/common.py b/freqtrade/exchange/common.py index 7f6dfe0eb..ce0fde9e4 100644 --- a/freqtrade/exchange/common.py +++ b/freqtrade/exchange/common.py @@ -3,13 +3,17 @@ import logging import time from functools import wraps -from freqtrade.exceptions import (DDosProtection, RetryableOrderError, - TemporaryError) +from freqtrade.exceptions import DDosProtection, RetryableOrderError, TemporaryError + logger = logging.getLogger(__name__) +# Maximum default retry count. +# Functions are always called RETRY_COUNT + 1 times (for the original call) API_RETRY_COUNT = 4 +API_FETCH_ORDER_RETRY_COUNT = 5 + BAD_EXCHANGES = { "bitmex": "Various reasons.", "bitstamp": "Does not provide history. " diff --git a/freqtrade/exchange/exchange.py b/freqtrade/exchange/exchange.py index b89da14eb..c0d737f26 100644 --- a/freqtrade/exchange/exchange.py +++ b/freqtrade/exchange/exchange.py @@ -8,24 +8,25 @@ import logging from copy import deepcopy from datetime import datetime, timezone from math import ceil -from random import randint from typing import Any, Dict, List, Optional, Tuple import arrow import ccxt import ccxt.async_support as ccxt_async -from ccxt.base.decimal_to_precision import (ROUND_DOWN, ROUND_UP, TICK_SIZE, - TRUNCATE, decimal_to_precision) +from ccxt.base.decimal_to_precision import (ROUND_DOWN, ROUND_UP, TICK_SIZE, TRUNCATE, + decimal_to_precision) from pandas import DataFrame from freqtrade.constants import ListPairsWithTimeframes from freqtrade.data.converter import ohlcv_to_dataframe, trades_dict_to_list -from freqtrade.exceptions import (DDosProtection, ExchangeError, - InvalidOrderException, OperationalException, - RetryableOrderError, TemporaryError) -from freqtrade.exchange.common import BAD_EXCHANGES, retrier, retrier_async +from freqtrade.exceptions import (DDosProtection, ExchangeError, InsufficientFundsError, + InvalidOrderException, OperationalException, RetryableOrderError, + TemporaryError) +from freqtrade.exchange.common import (API_FETCH_ORDER_RETRY_COUNT, BAD_EXCHANGES, retrier, + retrier_async) from freqtrade.misc import deep_merge_dicts, safe_value_fallback2 + CcxtModuleType = Any @@ -52,7 +53,7 @@ class Exchange: "ohlcv_partial_candle": True, "trades_pagination": "time", # Possible are "time" or "id" "trades_pagination_arg": "since", - + "l2_limit_range": None, } _ft_has: Dict = {} @@ -487,11 +488,11 @@ class Exchange: def dry_run_order(self, pair: str, ordertype: str, side: str, amount: float, rate: float, params: Dict = {}) -> Dict[str, Any]: - order_id = f'dry_run_{side}_{randint(0, 10**6)}' + order_id = f'dry_run_{side}_{datetime.now().timestamp()}' _amount = self.amount_to_precision(pair, amount) dry_order = { - "id": order_id, - 'pair': pair, + 'id': order_id, + 'symbol': pair, 'price': rate, 'average': rate, 'amount': _amount, @@ -500,6 +501,7 @@ class Exchange: 'side': side, 'remaining': _amount, 'datetime': arrow.utcnow().isoformat(), + 'timestamp': int(arrow.utcnow().timestamp * 1000), 'status': "closed" if ordertype == "market" else "open", 'fee': None, 'info': {} @@ -538,7 +540,7 @@ class Exchange: amount, rate_for_order, params) except ccxt.InsufficientFunds as e: - raise ExchangeError( + raise InsufficientFundsError( f'Insufficient funds to create {ordertype} {side} order on market {pair}. ' f'Tried to {side} amount {amount} at rate {rate}.' f'Message: {e}') from e @@ -1027,7 +1029,7 @@ class Exchange: return order - @retrier(retries=5) + @retrier(retries=API_FETCH_ORDER_RETRY_COUNT) def fetch_order(self, order_id: str, pair: str) -> Dict: if self._config['dry_run']: try: @@ -1056,6 +1058,27 @@ class Exchange: # Assign method to fetch_stoploss_order to allow easy overriding in other classes fetch_stoploss_order = fetch_order + def fetch_order_or_stoploss_order(self, order_id: str, pair: str, + stoploss_order: bool = False) -> Dict: + """ + Simple wrapper calling either fetch_order or fetch_stoploss_order depending on + the stoploss_order parameter + :param stoploss_order: If true, uses fetch_stoploss_order, otherwise fetch_order. + """ + if stoploss_order: + return self.fetch_stoploss_order(order_id, pair) + return self.fetch_order(order_id, pair) + + @staticmethod + def get_next_limit_in_list(limit: int, limit_range: Optional[List[int]]): + """ + Get next greater value in the list. + Used by fetch_l2_order_book if the api only supports a limited range + """ + if not limit_range: + return limit + return min([x for x in limit_range if limit <= x] + [max(limit_range)]) + @retrier def fetch_l2_order_book(self, pair: str, limit: int = 100) -> dict: """ @@ -1064,9 +1087,10 @@ class Exchange: Returns a dict in the format {'asks': [price, volume], 'bids': [price, volume]} """ + limit1 = self.get_next_limit_in_list(limit, self._ft_has['l2_limit_range']) try: - return self._api.fetch_l2_order_book(pair, limit) + return self._api.fetch_l2_order_book(pair, limit1) except ccxt.NotSupported as e: raise OperationalException( f'Exchange {self._api.name} does not support fetching order book.' diff --git a/freqtrade/exchange/ftx.py b/freqtrade/exchange/ftx.py index 441d97215..f05490cbb 100644 --- a/freqtrade/exchange/ftx.py +++ b/freqtrade/exchange/ftx.py @@ -4,11 +4,11 @@ from typing import Any, Dict import ccxt -from freqtrade.exceptions import (DDosProtection, ExchangeError, - InvalidOrderException, OperationalException, - TemporaryError) +from freqtrade.exceptions import (DDosProtection, InsufficientFundsError, InvalidOrderException, + OperationalException, TemporaryError) from freqtrade.exchange import Exchange -from freqtrade.exchange.common import retrier +from freqtrade.exchange.common import API_FETCH_ORDER_RETRY_COUNT, retrier + logger = logging.getLogger(__name__) @@ -71,7 +71,7 @@ class Ftx(Exchange): 'stop price: %s.', pair, stop_price) return order except ccxt.InsufficientFunds as e: - raise ExchangeError( + raise InsufficientFundsError( f'Insufficient funds to create {ordertype} sell order on market {pair}. ' f'Tried to create stoploss with amount {amount} at stoploss {stop_price}. ' f'Message: {e}') from e @@ -88,7 +88,7 @@ class Ftx(Exchange): except ccxt.BaseError as e: raise OperationalException(e) from e - @retrier(retries=5) + @retrier(retries=API_FETCH_ORDER_RETRY_COUNT) def fetch_stoploss_order(self, order_id: str, pair: str) -> Dict: if self._config['dry_run']: try: diff --git a/freqtrade/exchange/kraken.py b/freqtrade/exchange/kraken.py index 52b992dcc..5b7aa5c5b 100644 --- a/freqtrade/exchange/kraken.py +++ b/freqtrade/exchange/kraken.py @@ -4,12 +4,12 @@ from typing import Any, Dict import ccxt -from freqtrade.exceptions import (DDosProtection, ExchangeError, - InvalidOrderException, OperationalException, - TemporaryError) +from freqtrade.exceptions import (DDosProtection, InsufficientFundsError, InvalidOrderException, + OperationalException, TemporaryError) from freqtrade.exchange import Exchange from freqtrade.exchange.common import retrier + logger = logging.getLogger(__name__) @@ -98,7 +98,7 @@ class Kraken(Exchange): 'stop price: %s.', pair, stop_price) return order except ccxt.InsufficientFunds as e: - raise ExchangeError( + raise InsufficientFundsError( f'Insufficient funds to create {ordertype} sell order on market {pair}. ' f'Tried to create stoploss with amount {amount} at stoploss {stop_price}. ' f'Message: {e}') from e diff --git a/freqtrade/freqtradebot.py b/freqtrade/freqtradebot.py index 3c6e6b726..6112a599e 100644 --- a/freqtrade/freqtradebot.py +++ b/freqtrade/freqtradebot.py @@ -12,17 +12,17 @@ from typing import Any, Dict, List, Optional import arrow from cachetools import TTLCache -from freqtrade import __version__, constants, persistence +from freqtrade import __version__, constants from freqtrade.configuration import validate_config_consistency from freqtrade.data.converter import order_book_to_dataframe from freqtrade.data.dataprovider import DataProvider from freqtrade.edge import Edge -from freqtrade.exceptions import (DependencyException, ExchangeError, +from freqtrade.exceptions import (DependencyException, ExchangeError, InsufficientFundsError, InvalidOrderException, PricingError) from freqtrade.exchange import timeframe_to_minutes, timeframe_to_next_date from freqtrade.misc import safe_value_fallback, safe_value_fallback2 from freqtrade.pairlist.pairlistmanager import PairListManager -from freqtrade.persistence import Trade +from freqtrade.persistence import Order, Trade, cleanup_db, init_db from freqtrade.resolvers import ExchangeResolver, StrategyResolver from freqtrade.rpc import RPCManager, RPCMessageType from freqtrade.state import State @@ -30,6 +30,7 @@ from freqtrade.strategy.interface import IStrategy, SellType from freqtrade.strategy.strategy_wrapper import strategy_safe_wrapper from freqtrade.wallets import Wallets + logger = logging.getLogger(__name__) @@ -57,8 +58,8 @@ class FreqtradeBot: # Cache values for 1800 to avoid frequent polling of the exchange for prices # Caching only applies to RPC methods, so prices for open trades are still # refreshed once every iteration. - self._sell_rate_cache = TTLCache(maxsize=100, ttl=1800) - self._buy_rate_cache = TTLCache(maxsize=100, ttl=1800) + self._sell_rate_cache: TTLCache = TTLCache(maxsize=100, ttl=1800) + self._buy_rate_cache: TTLCache = TTLCache(maxsize=100, ttl=1800) self.strategy: IStrategy = StrategyResolver.load_strategy(self.config) @@ -67,7 +68,7 @@ class FreqtradeBot: self.exchange = ExchangeResolver.load_exchange(self.config['exchange']['name'], self.config) - persistence.init(self.config.get('db_url', None), clean_open_orders=self.config['dry_run']) + init_db(self.config.get('db_url', None), clean_open_orders=self.config['dry_run']) self.wallets = Wallets(self.config, self.exchange) @@ -122,7 +123,7 @@ class FreqtradeBot: self.check_for_open_trades() self.rpc.cleanup() - persistence.cleanup() + cleanup_db() def startup(self) -> None: """ @@ -134,6 +135,10 @@ class FreqtradeBot: # Adjust stoploss if it was changed Trade.stoploss_reinitialization(self.strategy.stoploss) + # Only update open orders on startup + # This will update the database after the initial migration + self.update_open_orders() + def process(self) -> None: """ Queries the persistence layer for open trades and handles them, @@ -144,6 +149,8 @@ class FreqtradeBot: # Check whether markets have to be reloaded and reload them when it's needed self.exchange.reload_markets() + self.update_closed_trades_without_assigned_fees() + # Query trades from persistence layer trades = Trade.get_open_trades() @@ -227,6 +234,104 @@ class FreqtradeBot: open_trades = len(Trade.get_open_trades()) return max(0, self.config['max_open_trades'] - open_trades) + def update_open_orders(self): + """ + Updates open orders based on order list kept in the database. + Mainly updates the state of orders - but may also close trades + """ + orders = Order.get_open_orders() + logger.info(f"Updating {len(orders)} open orders.") + for order in orders: + try: + fo = self.exchange.fetch_order_or_stoploss_order(order.order_id, order.ft_pair, + order.ft_order_side == 'stoploss') + + self.update_trade_state(order.trade, order.order_id, fo) + + except ExchangeError as e: + logger.warning(f"Error updating Order {order.order_id} due to {e}") + + def update_closed_trades_without_assigned_fees(self): + """ + Update closed trades without close fees assigned. + Only acts when Orders are in the database, otherwise the last orderid is unknown. + """ + trades: List[Trade] = Trade.get_sold_trades_without_assigned_fees() + for trade in trades: + + if not trade.is_open and not trade.fee_updated('sell'): + # Get sell fee + order = trade.select_order('sell', False) + if order: + logger.info(f"Updating sell-fee on trade {trade} for order {order.order_id}.") + self.update_trade_state(trade, order.order_id, + stoploss_order=order.ft_order_side == 'stoploss') + + trades: List[Trade] = Trade.get_open_trades_without_assigned_fees() + for trade in trades: + if trade.is_open and not trade.fee_updated('buy'): + order = trade.select_order('buy', False) + if order: + logger.info(f"Updating buy-fee on trade {trade} for order {order.order_id}.") + self.update_trade_state(trade, order.order_id) + + def handle_insufficient_funds(self, trade: Trade): + """ + Determine if we ever opened a sell order for this trade. + If not, try update buy fees - otherwise "refind" the open order we obviously lost. + """ + sell_order = trade.select_order('sell', None) + if sell_order: + self.refind_lost_order(trade) + else: + self.reupdate_buy_order_fees(trade) + + def reupdate_buy_order_fees(self, trade: Trade): + """ + Get buy order from database, and try to reupdate. + Handles trades where the initial fee-update did not work. + """ + logger.info(f"Trying to reupdate buy fees for {trade}") + order = trade.select_order('buy', False) + if order: + logger.info(f"Updating buy-fee on trade {trade} for order {order.order_id}.") + self.update_trade_state(trade, order.order_id) + + def refind_lost_order(self, trade): + """ + Try refinding a lost trade. + Only used when InsufficientFunds appears on sell orders (stoploss or sell). + Tries to walk the stored orders and sell them off eventually. + """ + logger.info(f"Trying to refind lost order for {trade}") + for order in trade.orders: + logger.info(f"Trying to refind {order}") + fo = None + if not order.ft_is_open: + logger.debug(f"Order {order} is no longer open.") + continue + if order.ft_order_side == 'buy': + # Skip buy side - this is handled by reupdate_buy_order_fees + continue + try: + fo = self.exchange.fetch_order_or_stoploss_order(order.order_id, order.ft_pair, + order.ft_order_side == 'stoploss') + if order.ft_order_side == 'stoploss': + if fo and fo['status'] == 'open': + # Assume this as the open stoploss order + trade.stoploss_order_id = order.order_id + elif order.ft_order_side == 'sell': + if fo and fo['status'] == 'open': + # Assume this as the open order + trade.open_order_id = order.order_id + if fo: + logger.info(f"Found {order} for trade {trade}.jj") + self.update_trade_state(trade, order.order_id, fo, + stoploss_order=order.ft_order_side == 'stoploss') + + except ExchangeError: + logger.warning(f"Error updating {order.order_id}.") + # # BUY / enter positions / open trades logic and methods # @@ -528,6 +633,7 @@ class FreqtradeBot: order = self.exchange.buy(pair=pair, ordertype=order_type, amount=amount, rate=buy_limit_requested, time_in_force=time_in_force) + order_obj = Order.parse_from_ccxt_object(order, pair, 'buy') order_id = order['id'] order_status = order.get('status', None) @@ -556,7 +662,6 @@ class FreqtradeBot: stake_amount = order['cost'] amount = safe_value_fallback(order, 'filled', 'amount') buy_limit_filled_price = safe_value_fallback(order, 'average', 'price') - order_id = None # in case of FOK the order may be filled immediately and fully elif order_status == 'closed': @@ -581,10 +686,11 @@ class FreqtradeBot: strategy=self.strategy.get_strategy_name(), timeframe=timeframe_to_minutes(self.config['timeframe']) ) + trade.orders.append(order_obj) # Update fees if order is closed if order_status == 'closed': - self.update_trade_state(trade, order) + self.update_trade_state(trade, order_id, order) Trade.session.add(trade) Trade.session.flush() @@ -783,8 +889,16 @@ class FreqtradeBot: stoploss_order = self.exchange.stoploss(pair=trade.pair, amount=trade.amount, stop_price=stop_price, order_types=self.strategy.order_types) + + order_obj = Order.parse_from_ccxt_object(stoploss_order, trade.pair, 'stoploss') + trade.orders.append(order_obj) trade.stoploss_order_id = str(stoploss_order['id']) return True + except InsufficientFundsError as e: + logger.warning(f"Unable to place stoploss order {e}.") + # Try to figure out what went wrong + self.handle_insufficient_funds(trade) + except InvalidOrderException as e: trade.stoploss_order_id = None logger.error(f'Unable to place a stoploss order on exchange. {e}') @@ -814,13 +928,17 @@ class FreqtradeBot: except InvalidOrderException as exception: logger.warning('Unable to fetch stoploss order: %s', exception) + if stoploss_order: + trade.update_order(stoploss_order) + # We check if stoploss order is fulfilled if stoploss_order and stoploss_order['status'] in ('closed', 'triggered'): trade.sell_reason = SellType.STOPLOSS_ON_EXCHANGE.value - self.update_trade_state(trade, stoploss_order, sl_order=True) + self.update_trade_state(trade, trade.stoploss_order_id, stoploss_order, + stoploss_order=True) # Lock pair for one candle to prevent immediate rebuys - self.strategy.lock_pair(trade.pair, - timeframe_to_next_date(self.config['timeframe'])) + self.strategy.lock_pair(trade.pair, timeframe_to_next_date(self.config['timeframe']), + reason='Auto lock') self._notify_sell(trade, "stoploss") return True @@ -869,10 +987,11 @@ class FreqtradeBot: update_beat = self.strategy.order_types.get('stoploss_on_exchange_interval', 60) if (datetime.utcnow() - trade.stoploss_last_update).total_seconds() >= update_beat: # cancelling the current stoploss on exchange first - logger.info('Trailing stoploss: cancelling current stoploss on exchange (id:{%s}) ' - 'in order to add another one ...', order['id']) + logger.info(f"Cancelling current stoploss on exchange for pair {trade.pair} " + f"(orderid:{order['id']}) in order to add another one ...") try: - self.exchange.cancel_stoploss_order(order['id'], trade.pair) + co = self.exchange.cancel_stoploss_order(order['id'], trade.pair) + trade.update_order(co) except InvalidOrderException: logger.exception(f"Could not cancel stoploss order {order['id']} " f"for pair {trade.pair}") @@ -927,7 +1046,7 @@ class FreqtradeBot: logger.info('Cannot query order for %s due to %s', trade, traceback.format_exc()) continue - fully_cancelled = self.update_trade_state(trade, order) + fully_cancelled = self.update_trade_state(trade, trade.open_order_id, order) if (order['side'] == 'buy' and (order['status'] == 'open' or fully_cancelled) and ( fully_cancelled @@ -995,8 +1114,7 @@ class FreqtradeBot: if isclose(filled_amount, 0.0, abs_tol=constants.MATH_CLOSE_PREC): logger.info('Buy order fully cancelled. Removing %s from database.', trade) # if trade is not partially completed, just delete the trade - Trade.session.delete(trade) - Trade.session.flush() + trade.delete() was_trade_fully_canceled = True reason += f", {constants.CANCEL_REASON['FULLY_CANCELLED']}" else: @@ -1007,7 +1125,7 @@ class FreqtradeBot: # we need to fall back to the values from order if corder does not contain these keys. trade.amount = filled_amount trade.stake_amount = trade.amount * trade.open_rate - self.update_trade_state(trade, corder, trade.amount) + self.update_trade_state(trade, trade.open_order_id, corder) trade.open_order_id = None logger.info('Partial buy order timeout for %s.', trade) @@ -1121,23 +1239,33 @@ class FreqtradeBot: logger.info(f"User requested abortion of selling {trade.pair}") return False - # Execute sell and update trade record - order = self.exchange.sell(pair=str(trade.pair), - ordertype=order_type, - amount=amount, rate=limit, - time_in_force=time_in_force - ) + try: + # Execute sell and update trade record + order = self.exchange.sell(pair=trade.pair, + ordertype=order_type, + amount=amount, rate=limit, + time_in_force=time_in_force + ) + except InsufficientFundsError as e: + logger.warning(f"Unable to place order {e}.") + # Try to figure out what went wrong + self.handle_insufficient_funds(trade) + return False + + order_obj = Order.parse_from_ccxt_object(order, trade.pair, 'sell') + trade.orders.append(order_obj) trade.open_order_id = order['id'] trade.close_rate_requested = limit trade.sell_reason = sell_reason.value # In case of market sell orders the order can be closed immediately if order.get('status', 'unknown') == 'closed': - self.update_trade_state(trade, order) + self.update_trade_state(trade, trade.open_order_id, order) Trade.session.flush() # Lock pair for one candle to prevent immediate rebuys - self.strategy.lock_pair(trade.pair, timeframe_to_next_date(self.config['timeframe'])) + self.strategy.lock_pair(trade.pair, timeframe_to_next_date(self.config['timeframe']), + reason='Auto lock') self._notify_sell(trade, order_type) @@ -1230,30 +1358,35 @@ class FreqtradeBot: # Common update trade state methods # - def update_trade_state(self, trade: Trade, action_order: dict = None, - order_amount: float = None, sl_order: bool = False) -> bool: + def update_trade_state(self, trade: Trade, order_id: str, action_order: Dict[str, Any] = None, + stoploss_order: bool = False) -> bool: """ Checks trades with open orders and updates the amount if necessary Handles closing both buy and sell orders. + :param trade: Trade object of the trade we're analyzing + :param order_id: Order-id of the order we're analyzing + :param action_order: Already aquired order object :return: True if order has been cancelled without being filled partially, False otherwise """ - # Get order details for actual price per unit - if trade.open_order_id: - order_id = trade.open_order_id - elif trade.stoploss_order_id and sl_order: - order_id = trade.stoploss_order_id - else: + if not order_id: + logger.warning(f'Orderid for trade {trade} is empty.') return False + # Update trade with order values logger.info('Found open order for %s', trade) try: - order = action_order or self.exchange.fetch_order(order_id, trade.pair) + order = action_order or self.exchange.fetch_order_or_stoploss_order(order_id, + trade.pair, + stoploss_order) except InvalidOrderException as exception: logger.warning('Unable to fetch order %s: %s', order_id, exception) return False + + trade.update_order(order) + # Try update amount (binance-fix) try: - new_amount = self.get_real_amount(trade, order, order_amount) + new_amount = self.get_real_amount(trade, order) if not isclose(safe_value_fallback(order, 'filled', 'amount'), new_amount, abs_tol=constants.MATH_CLOSE_PREC): order['amount'] = new_amount @@ -1291,7 +1424,7 @@ class FreqtradeBot: return real_amount return amount - def get_real_amount(self, trade: Trade, order: Dict, order_amount: float = None) -> float: + def get_real_amount(self, trade: Trade, order: Dict) -> float: """ Detect and update trade fee. Calls trade.update_fee() uppon correct detection. @@ -1300,8 +1433,7 @@ class FreqtradeBot: :return: identical (or new) amount for the trade """ # Init variables - if order_amount is None: - order_amount = safe_value_fallback(order, 'filled', 'amount') + order_amount = safe_value_fallback(order, 'filled', 'amount') # Only run for closed orders if trade.fee_updated(order.get('side', '')) or order['status'] == 'open': return order_amount @@ -1325,7 +1457,7 @@ class FreqtradeBot: """ fee-detection fallback to Trades. Parses result of fetch_my_trades to get correct fee. """ - trades = self.exchange.get_trades_for_order(trade.open_order_id, trade.pair, + trades = self.exchange.get_trades_for_order(order['id'], trade.pair, trade.open_date) if len(trades) == 0: diff --git a/freqtrade/loggers.py b/freqtrade/loggers.py index 8f5da9bee..169cd2610 100644 --- a/freqtrade/loggers.py +++ b/freqtrade/loggers.py @@ -1,12 +1,12 @@ import logging import sys from logging import Formatter -from logging.handlers import (BufferingHandler, RotatingFileHandler, - SysLogHandler) +from logging.handlers import BufferingHandler, RotatingFileHandler, SysLogHandler from typing import Any, Dict from freqtrade.exceptions import OperationalException + logger = logging.getLogger(__name__) LOGFORMAT = '%(asctime)s - %(name)s - %(levelname)s - %(message)s' diff --git a/freqtrade/main.py b/freqtrade/main.py index dc26c2a46..5f8d5d19d 100755 --- a/freqtrade/main.py +++ b/freqtrade/main.py @@ -7,6 +7,7 @@ import logging import sys from typing import Any, List + # check min. python version if sys.version_info < (3, 6): sys.exit("Freqtrade requires Python version >= 3.6") diff --git a/freqtrade/misc.py b/freqtrade/misc.py index 623f6cb8f..359d0d0e4 100644 --- a/freqtrade/misc.py +++ b/freqtrade/misc.py @@ -12,6 +12,7 @@ from typing.io import IO import numpy as np import rapidjson + logger = logging.getLogger(__name__) @@ -41,7 +42,7 @@ def datesarray_to_datetimearray(dates: np.ndarray) -> np.ndarray: return dates.dt.to_pydatetime() -def file_dump_json(filename: Path, data: Any, is_zip: bool = False) -> None: +def file_dump_json(filename: Path, data: Any, is_zip: bool = False, log: bool = True) -> None: """ Dump JSON data into a file :param filename: file to create @@ -52,12 +53,14 @@ def file_dump_json(filename: Path, data: Any, is_zip: bool = False) -> None: if is_zip: if filename.suffix != '.gz': filename = filename.with_suffix('.gz') - logger.info(f'dumping json to "{filename}"') + if log: + logger.info(f'dumping json to "{filename}"') - with gzip.open(filename, 'w') as fp: - rapidjson.dump(data, fp, default=str, number_mode=rapidjson.NM_NATIVE) + with gzip.open(filename, 'w') as fpz: + rapidjson.dump(data, fpz, default=str, number_mode=rapidjson.NM_NATIVE) else: - logger.info(f'dumping json to "{filename}"') + if log: + logger.info(f'dumping json to "{filename}"') with open(filename, 'w') as fp: rapidjson.dump(data, fp, default=str, number_mode=rapidjson.NM_NATIVE) diff --git a/freqtrade/optimize/backtesting.py b/freqtrade/optimize/backtesting.py index 005ec9fb8..47bb9edd9 100644 --- a/freqtrade/optimize/backtesting.py +++ b/freqtrade/optimize/backtesting.py @@ -4,31 +4,39 @@ This module contains the backtesting logic """ import logging +from collections import defaultdict from copy import deepcopy from datetime import datetime, timedelta from typing import Any, Dict, List, NamedTuple, Optional, Tuple -import arrow from pandas import DataFrame -from freqtrade.configuration import (TimeRange, remove_credentials, - validate_config_consistency) +from freqtrade.configuration import TimeRange, remove_credentials, validate_config_consistency from freqtrade.constants import DATETIME_PRINT_FORMAT from freqtrade.data import history from freqtrade.data.converter import trim_dataframe from freqtrade.data.dataprovider import DataProvider from freqtrade.exceptions import OperationalException from freqtrade.exchange import timeframe_to_minutes, timeframe_to_seconds -from freqtrade.optimize.optimize_reports import (generate_backtest_stats, - show_backtest_results, +from freqtrade.optimize.optimize_reports import (generate_backtest_stats, show_backtest_results, store_backtest_stats) from freqtrade.pairlist.pairlistmanager import PairListManager from freqtrade.persistence import Trade from freqtrade.resolvers import ExchangeResolver, StrategyResolver from freqtrade.strategy.interface import IStrategy, SellCheckTuple, SellType + logger = logging.getLogger(__name__) +# Indexes for backtest tuples +DATE_IDX = 0 +BUY_IDX = 1 +OPEN_IDX = 2 +CLOSE_IDX = 3 +SELL_IDX = 4 +LOW_IDX = 5 +HIGH_IDX = 6 + class BacktestResult(NamedTuple): """ @@ -116,7 +124,7 @@ class Backtesting: """ Load strategy into backtesting """ - self.strategy = strategy + self.strategy: IStrategy = strategy # Set stoploss_on_exchange to false for backtesting, # since a "perfect" stoploss-sell is assumed anyway # And the regular "stoploss" function would not apply to that case @@ -148,12 +156,14 @@ class Backtesting: return data, timerange - def _get_ohlcv_as_lists(self, processed: Dict) -> Dict[str, DataFrame]: + def _get_ohlcv_as_lists(self, processed: Dict[str, DataFrame]) -> Dict[str, Tuple]: """ Helper function to convert a processed dataframes into lists for performance reasons. Used by backtest() - so keep this optimized for performance. """ + # Every change to this headers list must evaluate further usages of the resulting tuple + # and eventually change the constants for indexes at the top headers = ['date', 'buy', 'open', 'close', 'sell', 'low', 'high'] data: Dict = {} # Create dict with data @@ -173,10 +183,10 @@ class Backtesting: # Convert from Pandas to list for performance reasons # (Looping Pandas is slow.) - data[pair] = [x for x in df_analyzed.itertuples()] + data[pair] = [x for x in df_analyzed.itertuples(index=False, name=None)] return data - def _get_close_rate(self, sell_row, trade: Trade, sell: SellCheckTuple, + def _get_close_rate(self, sell_row: Tuple, trade: Trade, sell: SellCheckTuple, trade_dur: int) -> float: """ Get close rate for backtesting result @@ -187,12 +197,12 @@ class Backtesting: return trade.stop_loss elif sell.sell_type == (SellType.ROI): roi_entry, roi = self.strategy.min_roi_reached_entry(trade_dur) - if roi is not None: + if roi is not None and roi_entry is not None: if roi == -1 and roi_entry % self.timeframe_min == 0: # When forceselling with ROI=-1, the roi time will always be equal to trade_dur. # If that entry is a multiple of the timeframe (so on candle open) # - we'll use open instead of close - return sell_row.open + return sell_row[OPEN_IDX] # - (Expected abs profit + open_rate + open_fee) / (fee_close -1) close_rate = - (trade.open_rate * roi + trade.open_rate * @@ -200,91 +210,79 @@ class Backtesting: if (trade_dur > 0 and trade_dur == roi_entry and roi_entry % self.timeframe_min == 0 - and sell_row.open > close_rate): + and sell_row[OPEN_IDX] > close_rate): # new ROI entry came into effect. # use Open rate if open_rate > calculated sell rate - return sell_row.open + return sell_row[OPEN_IDX] # Use the maximum between close_rate and low as we # cannot sell outside of a candle. # Applies when a new ROI setting comes in place and the whole candle is above that. - return max(close_rate, sell_row.low) + return max(close_rate, sell_row[LOW_IDX]) else: # This should not be reached... - return sell_row.open + return sell_row[OPEN_IDX] else: - return sell_row.open + return sell_row[OPEN_IDX] - def _get_sell_trade_entry( - self, pair: str, buy_row: DataFrame, - partial_ohlcv: List, trade_count_lock: Dict, - stake_amount: float, max_open_trades: int) -> Optional[BacktestResult]: + def _get_sell_trade_entry(self, trade: Trade, sell_row: Tuple) -> Optional[BacktestResult]: - trade = Trade( - pair=pair, - open_rate=buy_row.open, - open_date=buy_row.date, - stake_amount=stake_amount, - amount=round(stake_amount / buy_row.open, 8), - fee_open=self.fee, - fee_close=self.fee, - is_open=True, - ) - logger.debug(f"{pair} - Backtesting emulates creation of new trade: {trade}.") - # calculate win/lose forwards from buy point - for sell_row in partial_ohlcv: - if max_open_trades > 0: - # Increase trade_count_lock for every iteration - trade_count_lock[sell_row.date] = trade_count_lock.get(sell_row.date, 0) + 1 + sell = self.strategy.should_sell(trade, sell_row[OPEN_IDX], sell_row[DATE_IDX], + sell_row[BUY_IDX], sell_row[SELL_IDX], + low=sell_row[LOW_IDX], high=sell_row[HIGH_IDX]) + if sell.sell_flag: + trade_dur = int((sell_row[DATE_IDX] - trade.open_date).total_seconds() // 60) + closerate = self._get_close_rate(sell_row, trade, sell, trade_dur) - sell = self.strategy.should_sell(trade, sell_row.open, sell_row.date, sell_row.buy, - sell_row.sell, low=sell_row.low, high=sell_row.high) - if sell.sell_flag: - trade_dur = int((sell_row.date - buy_row.date).total_seconds() // 60) - closerate = self._get_close_rate(sell_row, trade, sell, trade_dur) - - return BacktestResult(pair=pair, - profit_percent=trade.calc_profit_ratio(rate=closerate), - profit_abs=trade.calc_profit(rate=closerate), - open_date=buy_row.date, - open_rate=buy_row.open, - open_fee=self.fee, - close_date=sell_row.date, - close_rate=closerate, - close_fee=self.fee, - amount=trade.amount, - trade_duration=trade_dur, - open_at_end=False, - sell_reason=sell.sell_type - ) - if partial_ohlcv: - # no sell condition found - trade stil open at end of backtest period - sell_row = partial_ohlcv[-1] - bt_res = BacktestResult(pair=pair, - profit_percent=trade.calc_profit_ratio(rate=sell_row.open), - profit_abs=trade.calc_profit(rate=sell_row.open), - open_date=buy_row.date, - open_rate=buy_row.open, - open_fee=self.fee, - close_date=sell_row.date, - close_rate=sell_row.open, - close_fee=self.fee, - amount=trade.amount, - trade_duration=int(( - sell_row.date - buy_row.date).total_seconds() // 60), - open_at_end=True, - sell_reason=SellType.FORCE_SELL - ) - logger.debug(f"{pair} - Force selling still open trade, " - f"profit percent: {bt_res.profit_percent}, " - f"profit abs: {bt_res.profit_abs}") - - return bt_res + return BacktestResult(pair=trade.pair, + profit_percent=trade.calc_profit_ratio(rate=closerate), + profit_abs=trade.calc_profit(rate=closerate), + open_date=trade.open_date, + open_rate=trade.open_rate, + open_fee=self.fee, + close_date=sell_row[DATE_IDX], + close_rate=closerate, + close_fee=self.fee, + amount=trade.amount, + trade_duration=trade_dur, + open_at_end=False, + sell_reason=sell.sell_type + ) return None + def handle_left_open(self, open_trades: Dict[str, List[Trade]], + data: Dict[str, List[Tuple]]) -> List[BacktestResult]: + """ + Handling of left open trades at the end of backtesting + """ + trades = [] + for pair in open_trades.keys(): + if len(open_trades[pair]) > 0: + for trade in open_trades[pair]: + sell_row = data[pair][-1] + trade_entry = BacktestResult(pair=trade.pair, + profit_percent=trade.calc_profit_ratio( + rate=sell_row[OPEN_IDX]), + profit_abs=trade.calc_profit(sell_row[OPEN_IDX]), + open_date=trade.open_date, + open_rate=trade.open_rate, + open_fee=self.fee, + close_date=sell_row[DATE_IDX], + close_rate=sell_row[OPEN_IDX], + close_fee=self.fee, + amount=trade.amount, + trade_duration=int(( + sell_row[DATE_IDX] - trade.open_date + ).total_seconds() // 60), + open_at_end=True, + sell_reason=SellType.FORCE_SELL + ) + trades.append(trade_entry) + return trades + def backtest(self, processed: Dict, stake_amount: float, - start_date: arrow.Arrow, end_date: arrow.Arrow, + start_date: datetime, end_date: datetime, max_open_trades: int = 0, position_stacking: bool = False) -> DataFrame: """ Implement backtesting functionality @@ -306,19 +304,21 @@ class Backtesting: f"max_open_trades: {max_open_trades}, position_stacking: {position_stacking}" ) trades = [] - trade_count_lock: Dict = {} # Use dict of lists with data for performance # (looping lists is a lot faster than pandas DataFrames) data: Dict = self._get_ohlcv_as_lists(processed) - lock_pair_until: Dict = {} # Indexes per pair, so some pairs are allowed to have a missing start. indexes: Dict = {} tmp = start_date + timedelta(minutes=self.timeframe_min) + open_trades: Dict[str, List] = defaultdict(list) + open_trade_count = 0 + # Loop timerange and get candle for each pair at that point in time - while tmp < end_date: + while tmp <= end_date: + open_trade_count_start = open_trade_count for i, pair in enumerate(data): if pair not in indexes: @@ -332,42 +332,52 @@ class Backtesting: continue # Waits until the time-counter reaches the start of the data for this pair. - if row.date > tmp.datetime: + if row[DATE_IDX] > tmp: continue - indexes[pair] += 1 - if row.buy == 0 or row.sell == 1: - continue # skip rows where no buy signal or that would immediately sell off + # without positionstacking, we can only have one open trade per pair. + # max_open_trades must be respected + # don't open on the last row + if ((position_stacking or len(open_trades[pair]) == 0) + and max_open_trades > 0 and open_trade_count_start < max_open_trades + and tmp != end_date + and row[BUY_IDX] == 1 and row[SELL_IDX] != 1): + # Enter trade + trade = Trade( + pair=pair, + open_rate=row[OPEN_IDX], + open_date=row[DATE_IDX], + stake_amount=stake_amount, + amount=round(stake_amount / row[OPEN_IDX], 8), + fee_open=self.fee, + fee_close=self.fee, + is_open=True, + ) + # TODO: hacky workaround to avoid opening > max_open_trades + # This emulates previous behaviour - not sure if this is correct + # Prevents buying if the trade-slot was freed in this candle + open_trade_count_start += 1 + open_trade_count += 1 + # logger.debug(f"{pair} - Backtesting emulates creation of new trade: {trade}.") + open_trades[pair].append(trade) - if (not position_stacking and pair in lock_pair_until - and row.date <= lock_pair_until[pair]): - # without positionstacking, we can only have one open trade per pair. - continue - - if max_open_trades > 0: - # Check if max_open_trades has already been reached for the given date - if not trade_count_lock.get(row.date, 0) < max_open_trades: - continue - trade_count_lock[row.date] = trade_count_lock.get(row.date, 0) + 1 - - # since indexes has been incremented before, we need to go one step back to - # also check the buying candle for sell conditions. - trade_entry = self._get_sell_trade_entry(pair, row, data[pair][indexes[pair]-1:], - trade_count_lock, stake_amount, - max_open_trades) - - if trade_entry: - logger.debug(f"{pair} - Locking pair till " - f"close_date={trade_entry.close_date}") - lock_pair_until[pair] = trade_entry.close_date - trades.append(trade_entry) - else: - # Set lock_pair_until to end of testing period if trade could not be closed - lock_pair_until[pair] = end_date.datetime + for trade in open_trades[pair]: + # since indexes has been incremented before, we need to go one step back to + # also check the buying candle for sell conditions. + trade_entry = self._get_sell_trade_entry(trade, row) + # Sell occured + if trade_entry: + # logger.debug(f"{pair} - Backtesting sell {trade}") + open_trade_count -= 1 + open_trades[pair].remove(trade) + trades.append(trade_entry) # Move time one configured time_interval ahead. tmp += timedelta(minutes=self.timeframe_min) + + trades += self.handle_left_open(open_trades, data=data) + return DataFrame.from_records(trades, columns=BacktestResult._fields) def start(self) -> None: @@ -380,12 +390,6 @@ class Backtesting: logger.info('Using stake_currency: %s ...', self.config['stake_currency']) logger.info('Using stake_amount: %s ...', self.config['stake_amount']) - # Use max_open_trades in backtesting, except --disable-max-market-positions is set - if self.config.get('use_max_market_positions', True): - max_open_trades = self.config['max_open_trades'] - else: - logger.info('Ignoring max_open_trades (--disable-max-market-positions was used) ...') - max_open_trades = 0 position_stacking = self.config.get('position_stacking', False) data, timerange = self.load_bt_data() @@ -395,6 +399,15 @@ class Backtesting: logger.info("Running backtesting for Strategy %s", strat.get_strategy_name()) self._set_strategy(strat) + # Use max_open_trades in backtesting, except --disable-max-market-positions is set + if self.config.get('use_max_market_positions', True): + # Must come from strategy config, as the strategy may modify this setting. + max_open_trades = self.strategy.config['max_open_trades'] + else: + logger.info( + 'Ignoring max_open_trades (--disable-max-market-positions was used) ...') + max_open_trades = 0 + # need to reprocess data every time to populate signals preprocessed = self.strategy.ohlcvdata_to_dataframe(data) @@ -407,17 +420,21 @@ class Backtesting: f'up to {max_date.strftime(DATETIME_PRINT_FORMAT)} ' f'({(max_date - min_date).days} days)..') # Execute backtest and print results - all_results[self.strategy.get_strategy_name()] = self.backtest( + results = self.backtest( processed=preprocessed, stake_amount=self.config['stake_amount'], - start_date=min_date, - end_date=max_date, + start_date=min_date.datetime, + end_date=max_date.datetime, max_open_trades=max_open_trades, position_stacking=position_stacking, ) + all_results[self.strategy.get_strategy_name()] = { + 'results': results, + 'config': self.strategy.config, + } + + stats = generate_backtest_stats(data, all_results, min_date=min_date, max_date=max_date) - stats = generate_backtest_stats(self.config, data, all_results, - min_date=min_date, max_date=max_date) if self.config.get('export', False): store_backtest_stats(self.config['exportfilename'], stats) diff --git a/freqtrade/optimize/default_hyperopt_loss.py b/freqtrade/optimize/default_hyperopt_loss.py index 9e780d0ea..9dbdc4403 100644 --- a/freqtrade/optimize/default_hyperopt_loss.py +++ b/freqtrade/optimize/default_hyperopt_loss.py @@ -1,5 +1,5 @@ """ -DefaultHyperOptLoss +ShortTradeDurHyperOptLoss This module defines the default HyperoptLoss class which is being used for Hyperoptimization. """ @@ -26,7 +26,7 @@ EXPECTED_MAX_PROFIT = 3.0 MAX_ACCEPTED_TRADE_DURATION = 300 -class DefaultHyperOptLoss(IHyperOptLoss): +class ShortTradeDurHyperOptLoss(IHyperOptLoss): """ Defines the default loss function for hyperopt """ @@ -50,3 +50,7 @@ class DefaultHyperOptLoss(IHyperOptLoss): duration_loss = 0.4 * min(trade_duration / MAX_ACCEPTED_TRADE_DURATION, 1) result = trade_loss + profit_loss + duration_loss return result + + +# Create an alias for This to allow the legacy Method to work as well. +DefaultHyperOptLoss = ShortTradeDurHyperOptLoss diff --git a/freqtrade/optimize/edge_cli.py b/freqtrade/optimize/edge_cli.py index be19688d8..a5f505bee 100644 --- a/freqtrade/optimize/edge_cli.py +++ b/freqtrade/optimize/edge_cli.py @@ -7,12 +7,12 @@ import logging from typing import Any, Dict from freqtrade import constants -from freqtrade.configuration import (TimeRange, remove_credentials, - validate_config_consistency) +from freqtrade.configuration import TimeRange, remove_credentials, validate_config_consistency from freqtrade.edge import Edge from freqtrade.optimize.optimize_reports import generate_edge_table from freqtrade.resolvers import ExchangeResolver, StrategyResolver + logger = logging.getLogger(__name__) diff --git a/freqtrade/optimize/hyperopt.py b/freqtrade/optimize/hyperopt.py index b9db3c09a..7870ba1cf 100644 --- a/freqtrade/optimize/hyperopt.py +++ b/freqtrade/optimize/hyperopt.py @@ -10,6 +10,7 @@ import logging import random import warnings from collections import OrderedDict +from datetime import datetime from math import ceil from operator import itemgetter from pathlib import Path @@ -21,24 +22,22 @@ import rapidjson import tabulate from colorama import Fore, Style from colorama import init as colorama_init -from joblib import (Parallel, cpu_count, delayed, dump, load, - wrap_non_picklable_objects) +from joblib import Parallel, cpu_count, delayed, dump, load, wrap_non_picklable_objects from pandas import DataFrame, isna, json_normalize -from freqtrade.constants import DATETIME_PRINT_FORMAT +from freqtrade.constants import DATETIME_PRINT_FORMAT, LAST_BT_RESULT_FN from freqtrade.data.converter import trim_dataframe from freqtrade.data.history import get_timerange from freqtrade.exceptions import OperationalException -from freqtrade.misc import plural, round_dict +from freqtrade.misc import file_dump_json, plural, round_dict from freqtrade.optimize.backtesting import Backtesting # Import IHyperOpt and IHyperOptLoss to allow unpickling classes from these modules from freqtrade.optimize.hyperopt_interface import IHyperOpt # noqa: F401 -from freqtrade.optimize.hyperopt_loss_interface import \ - IHyperOptLoss # noqa: F401 -from freqtrade.resolvers.hyperopt_resolver import (HyperOptLossResolver, - HyperOptResolver) +from freqtrade.optimize.hyperopt_loss_interface import IHyperOptLoss # noqa: F401 +from freqtrade.resolvers.hyperopt_resolver import HyperOptLossResolver, HyperOptResolver from freqtrade.strategy import IStrategy + # Suppress scikit-learn FutureWarnings from skopt with warnings.catch_warnings(): warnings.filterwarnings("ignore", category=FutureWarning) @@ -77,19 +76,16 @@ class Hyperopt: self.custom_hyperoptloss = HyperOptLossResolver.load_hyperoptloss(self.config) self.calculate_loss = self.custom_hyperoptloss.hyperopt_loss_function - + time_now = datetime.now().strftime("%Y-%m-%d_%H-%M-%S") self.results_file = (self.config['user_data_dir'] / - 'hyperopt_results' / 'hyperopt_results.pickle') + 'hyperopt_results' / f'hyperopt_results_{time_now}.pickle') self.data_pickle_file = (self.config['user_data_dir'] / 'hyperopt_results' / 'hyperopt_tickerdata.pkl') self.total_epochs = config.get('epochs', 0) self.current_best_loss = 100 - if not self.config.get('hyperopt_continue'): - self.clean_hyperopt() - else: - logger.info("Continuing on previous hyperopt results.") + self.clean_hyperopt() self.num_epochs_saved = 0 @@ -98,14 +94,14 @@ class Hyperopt: # Populate functions here (hasattr is slow so should not be run during "regular" operations) if hasattr(self.custom_hyperopt, 'populate_indicators'): - self.backtesting.strategy.advise_indicators = \ - self.custom_hyperopt.populate_indicators # type: ignore + self.backtesting.strategy.advise_indicators = ( # type: ignore + self.custom_hyperopt.populate_indicators) # type: ignore if hasattr(self.custom_hyperopt, 'populate_buy_trend'): - self.backtesting.strategy.advise_buy = \ - self.custom_hyperopt.populate_buy_trend # type: ignore + self.backtesting.strategy.advise_buy = ( # type: ignore + self.custom_hyperopt.populate_buy_trend) # type: ignore if hasattr(self.custom_hyperopt, 'populate_sell_trend'): - self.backtesting.strategy.advise_sell = \ - self.custom_hyperopt.populate_sell_trend # type: ignore + self.backtesting.strategy.advise_sell = ( # type: ignore + self.custom_hyperopt.populate_sell_trend) # type: ignore # Use max_open_trades for hyperopt as well, except --disable-max-market-positions is set if self.config.get('use_max_market_positions', True): @@ -165,6 +161,10 @@ class Hyperopt: self.num_epochs_saved = num_epochs logger.debug(f"{self.num_epochs_saved} {plural(self.num_epochs_saved, 'epoch')} " f"saved to '{self.results_file}'.") + # Store hyperopt filename + latest_filename = Path.joinpath(self.results_file.parent, LAST_BT_RESULT_FN) + file_dump_json(latest_filename, {'latest_hyperopt': str(self.results_file.name)}, + log=False) @staticmethod def _read_results(results_file: Path) -> List: @@ -262,6 +262,11 @@ class Hyperopt: ), default=str, indent=4, number_mode=rapidjson.NM_NATIVE) params_result += f"minimal_roi = {minimal_roi_result}" + elif space == 'trailing': + + for k, v in space_params.items(): + params_result += f'{k} = {v}\n' + else: params_result += f"{space}_params = {pformat(space_params, indent=4)}" params_result = params_result.replace("}", "\n}").replace("{", "{\n ") @@ -324,8 +329,9 @@ class Hyperopt: 'results_metrics.avg_profit', 'results_metrics.total_profit', 'results_metrics.profit', 'results_metrics.duration', 'loss', 'is_initial_point', 'is_best']] - trials.columns = ['Best', 'Epoch', 'Trades', 'W/D/L', 'Avg profit', 'Total profit', - 'Profit', 'Avg duration', 'Objective', 'is_initial_point', 'is_best'] + trials.columns = ['Best', 'Epoch', 'Trades', ' Win Draw Loss', 'Avg profit', + 'Total profit', 'Profit', 'Avg duration', 'Objective', + 'is_initial_point', 'is_best'] trials['is_profit'] = False trials.loc[trials['is_initial_point'], 'Best'] = '* ' trials.loc[trials['is_best'], 'Best'] = 'Best' @@ -502,16 +508,16 @@ class Hyperopt: params_details = self._get_params_details(params_dict) if self.has_space('roi'): - self.backtesting.strategy.minimal_roi = \ - self.custom_hyperopt.generate_roi_table(params_dict) + self.backtesting.strategy.minimal_roi = ( # type: ignore + self.custom_hyperopt.generate_roi_table(params_dict)) if self.has_space('buy'): - self.backtesting.strategy.advise_buy = \ - self.custom_hyperopt.buy_strategy_generator(params_dict) + self.backtesting.strategy.advise_buy = ( # type: ignore + self.custom_hyperopt.buy_strategy_generator(params_dict)) if self.has_space('sell'): - self.backtesting.strategy.advise_sell = \ - self.custom_hyperopt.sell_strategy_generator(params_dict) + self.backtesting.strategy.advise_sell = ( # type: ignore + self.custom_hyperopt.sell_strategy_generator(params_dict)) if self.has_space('stoploss'): self.backtesting.strategy.stoploss = params_dict['stoploss'] @@ -532,8 +538,8 @@ class Hyperopt: backtesting_results = self.backtesting.backtest( processed=processed, stake_amount=self.config['stake_amount'], - start_date=min_date, - end_date=max_date, + start_date=min_date.datetime, + end_date=max_date.datetime, max_open_trades=self.max_open_trades, position_stacking=self.position_stacking, ) @@ -574,7 +580,7 @@ class Hyperopt: 'wins': wins, 'draws': draws, 'losses': losses, - 'winsdrawslosses': f"{wins}/{draws}/{losses}", + 'winsdrawslosses': f"{wins:>4} {draws:>4} {losses:>4}", 'avg_profit': backtesting_results.profit_percent.mean() * 100.0, 'median_profit': backtesting_results.profit_percent.median() * 100.0, 'total_profit': backtesting_results.profit_abs.sum(), @@ -656,8 +662,6 @@ class Hyperopt: self.backtesting.strategy.dp = None # type: ignore IStrategy.dp = None # type: ignore - self.epochs = self.load_previous_results(self.results_file) - cpus = cpu_count() logger.info(f"Found {cpus} CPU cores. Let's make them scream!") config_jobs = self.config.get('hyperopt_jobs', -1) diff --git a/freqtrade/optimize/hyperopt_interface.py b/freqtrade/optimize/hyperopt_interface.py index 65069b984..b8c44ed59 100644 --- a/freqtrade/optimize/hyperopt_interface.py +++ b/freqtrade/optimize/hyperopt_interface.py @@ -13,6 +13,7 @@ from freqtrade.exceptions import OperationalException from freqtrade.exchange import timeframe_to_minutes from freqtrade.misc import round_dict + logger = logging.getLogger(__name__) diff --git a/freqtrade/optimize/hyperopt_loss_sharpe.py b/freqtrade/optimize/hyperopt_loss_sharpe.py index 29377bdd5..232fb33b6 100644 --- a/freqtrade/optimize/hyperopt_loss_sharpe.py +++ b/freqtrade/optimize/hyperopt_loss_sharpe.py @@ -6,8 +6,8 @@ Hyperoptimization. """ from datetime import datetime -from pandas import DataFrame import numpy as np +from pandas import DataFrame from freqtrade.optimize.hyperopt import IHyperOptLoss diff --git a/freqtrade/optimize/hyperopt_loss_sortino.py b/freqtrade/optimize/hyperopt_loss_sortino.py index d470a9977..c0ff0773a 100644 --- a/freqtrade/optimize/hyperopt_loss_sortino.py +++ b/freqtrade/optimize/hyperopt_loss_sortino.py @@ -6,8 +6,8 @@ Hyperoptimization. """ from datetime import datetime -from pandas import DataFrame import numpy as np +from pandas import DataFrame from freqtrade.optimize.hyperopt import IHyperOptLoss diff --git a/freqtrade/optimize/optimize_reports.py b/freqtrade/optimize/optimize_reports.py index b5e5da4af..3db9a312a 100644 --- a/freqtrade/optimize/optimize_reports.py +++ b/freqtrade/optimize/optimize_reports.py @@ -1,17 +1,18 @@ import logging from datetime import datetime, timedelta, timezone from pathlib import Path -from typing import Any, Dict, List +from typing import Any, Dict, List, Union from arrow import Arrow -from pandas import DataFrame from numpy import int64 +from pandas import DataFrame from tabulate import tabulate from freqtrade.constants import DATETIME_PRINT_FORMAT, LAST_BT_RESULT_FN -from freqtrade.data.btanalysis import calculate_max_drawdown, calculate_market_change +from freqtrade.data.btanalysis import calculate_market_change, calculate_max_drawdown from freqtrade.misc import file_dump_json + logger = logging.getLogger(__name__) @@ -122,7 +123,7 @@ def generate_sell_reason_stats(max_open_trades: int, results: DataFrame) -> List profit_mean = result['profit_percent'].mean() profit_sum = result["profit_percent"].sum() - profit_percent_tot = round(result['profit_percent'].sum() * 100.0 / max_open_trades, 2) + profit_percent_tot = result['profit_percent'].sum() / max_open_trades tabular_data.append( { @@ -136,25 +137,25 @@ def generate_sell_reason_stats(max_open_trades: int, results: DataFrame) -> List 'profit_sum': profit_sum, 'profit_sum_pct': round(profit_sum * 100, 2), 'profit_total_abs': result['profit_abs'].sum(), - 'profit_total_pct': profit_percent_tot, + 'profit_total': profit_percent_tot, + 'profit_total_pct': round(profit_percent_tot * 100, 2), } ) return tabular_data -def generate_strategy_metrics(stake_currency: str, max_open_trades: int, - all_results: Dict) -> List[Dict]: +def generate_strategy_metrics(all_results: Dict) -> List[Dict]: """ Generate summary per strategy - :param stake_currency: stake-currency - used to correctly name headers - :param max_open_trades: Maximum allowed open trades used for backtest :param all_results: Dict of containing results for all strategies :return: List of Dicts containing the metrics per Strategy """ tabular_data = [] for strategy, results in all_results.items(): - tabular_data.append(_generate_result_line(results, max_open_trades, strategy)) + tabular_data.append(_generate_result_line( + results['results'], results['config']['max_open_trades'], strategy) + ) return tabular_data @@ -218,25 +219,29 @@ def generate_daily_stats(results: DataFrame) -> Dict[str, Any]: } -def generate_backtest_stats(config: Dict, btdata: Dict[str, DataFrame], - all_results: Dict[str, DataFrame], +def generate_backtest_stats(btdata: Dict[str, DataFrame], + all_results: Dict[str, Dict[str, Union[DataFrame, Dict]]], min_date: Arrow, max_date: Arrow ) -> Dict[str, Any]: """ - :param config: Configuration object used for backtest :param btdata: Backtest data - :param all_results: backtest result - dictionary with { Strategy: results}. + :param all_results: backtest result - dictionary in the form: + { Strategy: {'results: results, 'config: config}}. :param min_date: Backtest start date :param max_date: Backtest end date :return: Dictionary containing results per strategy and a stratgy summary. """ - stake_currency = config['stake_currency'] - max_open_trades = config['max_open_trades'] result: Dict[str, Any] = {'strategy': {}} market_change = calculate_market_change(btdata, 'close') - for strategy, results in all_results.items(): + for strategy, content in all_results.items(): + results: Dict[str, DataFrame] = content['results'] + if not isinstance(results, DataFrame): + continue + config = content['config'] + max_open_trades = config['max_open_trades'] + stake_currency = config['stake_currency'] pair_results = generate_pair_metrics(btdata, stake_currency=stake_currency, max_open_trades=max_open_trades, @@ -276,6 +281,16 @@ def generate_backtest_stats(config: Dict, btdata: Dict[str, DataFrame], 'max_open_trades': (config['max_open_trades'] if config['max_open_trades'] != float('inf') else -1), 'timeframe': config['timeframe'], + # Parameters relevant for backtesting + 'stoploss': config['stoploss'], + 'trailing_stop': config.get('trailing_stop', False), + 'trailing_stop_positive': config.get('trailing_stop_positive'), + 'trailing_stop_positive_offset': config.get('trailing_stop_positive_offset', 0.0), + 'trailing_only_offset_is_reached': config.get('trailing_only_offset_is_reached', False), + 'minimal_roi': config['minimal_roi'], + 'use_sell_signal': config['ask_strategy']['use_sell_signal'], + 'sell_profit_only': config['ask_strategy']['sell_profit_only'], + 'ignore_roi_if_buy_signal': config['ask_strategy']['ignore_roi_if_buy_signal'], **daily_stats, } result['strategy'][strategy] = strat_stats @@ -299,9 +314,7 @@ def generate_backtest_stats(config: Dict, btdata: Dict[str, DataFrame], 'drawdown_end_ts': 0, }) - strategy_results = generate_strategy_metrics(stake_currency=stake_currency, - max_open_trades=max_open_trades, - all_results=all_results) + strategy_results = generate_strategy_metrics(all_results=all_results) result['strategy_comparison'] = strategy_results diff --git a/freqtrade/pairlist/AgeFilter.py b/freqtrade/pairlist/AgeFilter.py index 64f01cb61..19cf1c090 100644 --- a/freqtrade/pairlist/AgeFilter.py +++ b/freqtrade/pairlist/AgeFilter.py @@ -2,9 +2,10 @@ Minimum age (days listed) pair list filter """ import logging -import arrow from typing import Any, Dict +import arrow + from freqtrade.exceptions import OperationalException from freqtrade.misc import plural from freqtrade.pairlist.IPairList import IPairList diff --git a/freqtrade/pairlist/IPairList.py b/freqtrade/pairlist/IPairList.py index 67a96cc60..6b5bd11e7 100644 --- a/freqtrade/pairlist/IPairList.py +++ b/freqtrade/pairlist/IPairList.py @@ -36,7 +36,7 @@ class IPairList(ABC): self._pairlist_pos = pairlist_pos self.refresh_period = self._pairlistconfig.get('refresh_period', 1800) self._last_refresh = 0 - self._log_cache = TTLCache(maxsize=1024, ttl=self.refresh_period) + self._log_cache: TTLCache = TTLCache(maxsize=1024, ttl=self.refresh_period) @property def name(self) -> str: diff --git a/freqtrade/pairlist/PrecisionFilter.py b/freqtrade/pairlist/PrecisionFilter.py index 3061d3d01..cf853397b 100644 --- a/freqtrade/pairlist/PrecisionFilter.py +++ b/freqtrade/pairlist/PrecisionFilter.py @@ -4,8 +4,9 @@ Precision pair list filter import logging from typing import Any, Dict -from freqtrade.pairlist.IPairList import IPairList from freqtrade.exceptions import OperationalException +from freqtrade.pairlist.IPairList import IPairList + logger = logging.getLogger(__name__) diff --git a/freqtrade/pairlist/pairlistmanager.py b/freqtrade/pairlist/pairlistmanager.py index 81e52768e..89bab99be 100644 --- a/freqtrade/pairlist/pairlistmanager.py +++ b/freqtrade/pairlist/pairlistmanager.py @@ -7,10 +7,10 @@ from typing import Dict, List from cachetools import TTLCache, cached +from freqtrade.constants import ListPairsWithTimeframes from freqtrade.exceptions import OperationalException from freqtrade.pairlist.IPairList import IPairList from freqtrade.resolvers import PairListResolver -from freqtrade.constants import ListPairsWithTimeframes logger = logging.getLogger(__name__) diff --git a/freqtrade/persistence/__init__.py b/freqtrade/persistence/__init__.py new file mode 100644 index 000000000..e184e7d9a --- /dev/null +++ b/freqtrade/persistence/__init__.py @@ -0,0 +1,4 @@ +# flake8: noqa: F401 + +from freqtrade.persistence.models import (Order, PairLock, Trade, clean_dry_run_db, cleanup_db, + init_db) diff --git a/freqtrade/persistence/migrations.py b/freqtrade/persistence/migrations.py new file mode 100644 index 000000000..84f3ed7e6 --- /dev/null +++ b/freqtrade/persistence/migrations.py @@ -0,0 +1,150 @@ +import logging +from typing import List + +from sqlalchemy import inspect + + +logger = logging.getLogger(__name__) + + +def get_table_names_for_table(inspector, tabletype): + return [t for t in inspector.get_table_names() if t.startswith(tabletype)] + + +def has_column(columns: List, searchname: str) -> bool: + return len(list(filter(lambda x: x["name"] == searchname, columns))) == 1 + + +def get_column_def(columns: List, column: str, default: str) -> str: + return default if not has_column(columns, column) else column + + +def get_backup_name(tabs, backup_prefix: str): + table_back_name = backup_prefix + for i, table_back_name in enumerate(tabs): + table_back_name = f'{backup_prefix}{i}' + logger.debug(f'trying {table_back_name}') + + return table_back_name + + +def migrate_trades_table(decl_base, inspector, engine, table_back_name: str, cols: List): + fee_open = get_column_def(cols, 'fee_open', 'fee') + fee_open_cost = get_column_def(cols, 'fee_open_cost', 'null') + fee_open_currency = get_column_def(cols, 'fee_open_currency', 'null') + fee_close = get_column_def(cols, 'fee_close', 'fee') + fee_close_cost = get_column_def(cols, 'fee_close_cost', 'null') + fee_close_currency = get_column_def(cols, 'fee_close_currency', 'null') + open_rate_requested = get_column_def(cols, 'open_rate_requested', 'null') + close_rate_requested = get_column_def(cols, 'close_rate_requested', 'null') + stop_loss = get_column_def(cols, 'stop_loss', '0.0') + stop_loss_pct = get_column_def(cols, 'stop_loss_pct', 'null') + initial_stop_loss = get_column_def(cols, 'initial_stop_loss', '0.0') + initial_stop_loss_pct = get_column_def(cols, 'initial_stop_loss_pct', 'null') + stoploss_order_id = get_column_def(cols, 'stoploss_order_id', 'null') + stoploss_last_update = get_column_def(cols, 'stoploss_last_update', 'null') + max_rate = get_column_def(cols, 'max_rate', '0.0') + min_rate = get_column_def(cols, 'min_rate', 'null') + sell_reason = get_column_def(cols, 'sell_reason', 'null') + strategy = get_column_def(cols, 'strategy', 'null') + # If ticker-interval existed use that, else null. + if has_column(cols, 'ticker_interval'): + timeframe = get_column_def(cols, 'timeframe', 'ticker_interval') + else: + timeframe = get_column_def(cols, 'timeframe', 'null') + + open_trade_price = get_column_def(cols, 'open_trade_price', + f'amount * open_rate * (1 + {fee_open})') + close_profit_abs = get_column_def( + cols, 'close_profit_abs', + f"(amount * close_rate * (1 - {fee_close})) - {open_trade_price}") + sell_order_status = get_column_def(cols, 'sell_order_status', 'null') + amount_requested = get_column_def(cols, 'amount_requested', 'amount') + + # Schema migration necessary + engine.execute(f"alter table trades rename to {table_back_name}") + # drop indexes on backup table + for index in inspector.get_indexes(table_back_name): + engine.execute(f"drop index {index['name']}") + # let SQLAlchemy create the schema as required + decl_base.metadata.create_all(engine) + + # Copy data back - following the correct schema + engine.execute(f"""insert into trades + (id, exchange, pair, is_open, + fee_open, fee_open_cost, fee_open_currency, + fee_close, fee_close_cost, fee_open_currency, open_rate, + open_rate_requested, close_rate, close_rate_requested, close_profit, + stake_amount, amount, amount_requested, open_date, close_date, open_order_id, + stop_loss, stop_loss_pct, initial_stop_loss, initial_stop_loss_pct, + stoploss_order_id, stoploss_last_update, + max_rate, min_rate, sell_reason, sell_order_status, strategy, + timeframe, open_trade_price, close_profit_abs + ) + select id, lower(exchange), + case + when instr(pair, '_') != 0 then + substr(pair, instr(pair, '_') + 1) || '/' || + substr(pair, 1, instr(pair, '_') - 1) + else pair + end + pair, + is_open, {fee_open} fee_open, {fee_open_cost} fee_open_cost, + {fee_open_currency} fee_open_currency, {fee_close} fee_close, + {fee_close_cost} fee_close_cost, {fee_close_currency} fee_close_currency, + open_rate, {open_rate_requested} open_rate_requested, close_rate, + {close_rate_requested} close_rate_requested, close_profit, + stake_amount, amount, {amount_requested}, open_date, close_date, open_order_id, + {stop_loss} stop_loss, {stop_loss_pct} stop_loss_pct, + {initial_stop_loss} initial_stop_loss, + {initial_stop_loss_pct} initial_stop_loss_pct, + {stoploss_order_id} stoploss_order_id, {stoploss_last_update} stoploss_last_update, + {max_rate} max_rate, {min_rate} min_rate, {sell_reason} sell_reason, + {sell_order_status} sell_order_status, + {strategy} strategy, {timeframe} timeframe, + {open_trade_price} open_trade_price, {close_profit_abs} close_profit_abs + from {table_back_name} + """) + + +def migrate_open_orders_to_trades(engine): + engine.execute(""" + insert into orders (ft_trade_id, ft_pair, order_id, ft_order_side, ft_is_open) + select id ft_trade_id, pair ft_pair, open_order_id, + case when close_rate_requested is null then 'buy' + else 'sell' end ft_order_side, 1 ft_is_open + from trades + where open_order_id is not null + union all + select id ft_trade_id, pair ft_pair, stoploss_order_id order_id, + 'stoploss' ft_order_side, 1 ft_is_open + from trades + where stoploss_order_id is not null + """) + + +def check_migrate(engine, decl_base, previous_tables) -> None: + """ + Checks if migration is necessary and migrates if necessary + """ + inspector = inspect(engine) + + cols = inspector.get_columns('trades') + tabs = get_table_names_for_table(inspector, 'trades') + table_back_name = get_backup_name(tabs, 'trades_bak') + + # Check for latest column + if not has_column(cols, 'amount_requested'): + logger.info(f'Running database migration for trades - backup: {table_back_name}') + migrate_trades_table(decl_base, inspector, engine, table_back_name, cols) + # Reread columns - the above recreated the table! + inspector = inspect(engine) + cols = inspector.get_columns('trades') + + if 'orders' not in previous_tables: + logger.info('Moving open orders to Orders table.') + migrate_open_orders_to_trades(engine) + else: + pass + # Empty for now - as there is only one iteration of the orders table so far. + # table_back_name = get_backup_name(tabs, 'orders_bak') diff --git a/freqtrade/persistence.py b/freqtrade/persistence/models.py similarity index 64% rename from freqtrade/persistence.py rename to freqtrade/persistence/models.py index 9eebadd8d..477a94bad 100644 --- a/freqtrade/persistence.py +++ b/freqtrade/persistence/models.py @@ -7,17 +7,21 @@ from decimal import Decimal from typing import Any, Dict, List, Optional import arrow -from sqlalchemy import (Boolean, Column, DateTime, Float, Integer, String, +from sqlalchemy import (Boolean, Column, DateTime, Float, ForeignKey, Integer, String, create_engine, desc, func, inspect) from sqlalchemy.exc import NoSuchModuleError from sqlalchemy.ext.declarative import declarative_base -from sqlalchemy.orm import Query +from sqlalchemy.orm import Query, relationship from sqlalchemy.orm.scoping import scoped_session from sqlalchemy.orm.session import sessionmaker from sqlalchemy.pool import StaticPool +from sqlalchemy.sql.schema import UniqueConstraint -from freqtrade.exceptions import OperationalException +from freqtrade.constants import DATETIME_PRINT_FORMAT +from freqtrade.exceptions import DependencyException, OperationalException from freqtrade.misc import safe_value_fallback +from freqtrade.persistence.migrations import check_migrate + logger = logging.getLogger(__name__) @@ -26,7 +30,7 @@ _DECL_BASE: Any = declarative_base() _SQL_DOCS_URL = 'http://docs.sqlalchemy.org/en/latest/core/engines.html#database-urls' -def init(db_url: str, clean_open_orders: bool = False) -> None: +def init_db(db_url: str, clean_open_orders: bool = False) -> None: """ Initializes this module with the given config, registers all known command handlers @@ -57,122 +61,22 @@ def init(db_url: str, clean_open_orders: bool = False) -> None: # We should use the scoped_session object - not a seperately initialized version Trade.session = scoped_session(sessionmaker(bind=engine, autoflush=True, autocommit=True)) Trade.query = Trade.session.query_property() + # Copy session attributes to order object too + Order.session = Trade.session + Order.query = Order.session.query_property() + PairLock.session = Trade.session + PairLock.query = PairLock.session.query_property() + + previous_tables = inspect(engine).get_table_names() _DECL_BASE.metadata.create_all(engine) - check_migrate(engine) + check_migrate(engine, decl_base=_DECL_BASE, previous_tables=previous_tables) # Clean dry_run DB if the db is not in-memory if clean_open_orders and db_url != 'sqlite://': clean_dry_run_db() -def has_column(columns: List, searchname: str) -> bool: - return len(list(filter(lambda x: x["name"] == searchname, columns))) == 1 - - -def get_column_def(columns: List, column: str, default: str) -> str: - return default if not has_column(columns, column) else column - - -def check_migrate(engine) -> None: - """ - Checks if migration is necessary and migrates if necessary - """ - inspector = inspect(engine) - - cols = inspector.get_columns('trades') - tabs = inspector.get_table_names() - table_back_name = 'trades_bak' - for i, table_back_name in enumerate(tabs): - table_back_name = f'trades_bak{i}' - logger.debug(f'trying {table_back_name}') - - # Check for latest column - if not has_column(cols, 'amount_requested'): - logger.info(f'Running database migration - backup available as {table_back_name}') - - fee_open = get_column_def(cols, 'fee_open', 'fee') - fee_open_cost = get_column_def(cols, 'fee_open_cost', 'null') - fee_open_currency = get_column_def(cols, 'fee_open_currency', 'null') - fee_close = get_column_def(cols, 'fee_close', 'fee') - fee_close_cost = get_column_def(cols, 'fee_close_cost', 'null') - fee_close_currency = get_column_def(cols, 'fee_close_currency', 'null') - open_rate_requested = get_column_def(cols, 'open_rate_requested', 'null') - close_rate_requested = get_column_def(cols, 'close_rate_requested', 'null') - stop_loss = get_column_def(cols, 'stop_loss', '0.0') - stop_loss_pct = get_column_def(cols, 'stop_loss_pct', 'null') - initial_stop_loss = get_column_def(cols, 'initial_stop_loss', '0.0') - initial_stop_loss_pct = get_column_def(cols, 'initial_stop_loss_pct', 'null') - stoploss_order_id = get_column_def(cols, 'stoploss_order_id', 'null') - stoploss_last_update = get_column_def(cols, 'stoploss_last_update', 'null') - max_rate = get_column_def(cols, 'max_rate', '0.0') - min_rate = get_column_def(cols, 'min_rate', 'null') - sell_reason = get_column_def(cols, 'sell_reason', 'null') - strategy = get_column_def(cols, 'strategy', 'null') - # If ticker-interval existed use that, else null. - if has_column(cols, 'ticker_interval'): - timeframe = get_column_def(cols, 'timeframe', 'ticker_interval') - else: - timeframe = get_column_def(cols, 'timeframe', 'null') - - open_trade_price = get_column_def(cols, 'open_trade_price', - f'amount * open_rate * (1 + {fee_open})') - close_profit_abs = get_column_def( - cols, 'close_profit_abs', - f"(amount * close_rate * (1 - {fee_close})) - {open_trade_price}") - sell_order_status = get_column_def(cols, 'sell_order_status', 'null') - amount_requested = get_column_def(cols, 'amount_requested', 'amount') - - # Schema migration necessary - engine.execute(f"alter table trades rename to {table_back_name}") - # drop indexes on backup table - for index in inspector.get_indexes(table_back_name): - engine.execute(f"drop index {index['name']}") - # let SQLAlchemy create the schema as required - _DECL_BASE.metadata.create_all(engine) - - # Copy data back - following the correct schema - engine.execute(f"""insert into trades - (id, exchange, pair, is_open, - fee_open, fee_open_cost, fee_open_currency, - fee_close, fee_close_cost, fee_open_currency, open_rate, - open_rate_requested, close_rate, close_rate_requested, close_profit, - stake_amount, amount, amount_requested, open_date, close_date, open_order_id, - stop_loss, stop_loss_pct, initial_stop_loss, initial_stop_loss_pct, - stoploss_order_id, stoploss_last_update, - max_rate, min_rate, sell_reason, sell_order_status, strategy, - timeframe, open_trade_price, close_profit_abs - ) - select id, lower(exchange), - case - when instr(pair, '_') != 0 then - substr(pair, instr(pair, '_') + 1) || '/' || - substr(pair, 1, instr(pair, '_') - 1) - else pair - end - pair, - is_open, {fee_open} fee_open, {fee_open_cost} fee_open_cost, - {fee_open_currency} fee_open_currency, {fee_close} fee_close, - {fee_close_cost} fee_close_cost, {fee_close_currency} fee_close_currency, - open_rate, {open_rate_requested} open_rate_requested, close_rate, - {close_rate_requested} close_rate_requested, close_profit, - stake_amount, amount, {amount_requested}, open_date, close_date, open_order_id, - {stop_loss} stop_loss, {stop_loss_pct} stop_loss_pct, - {initial_stop_loss} initial_stop_loss, - {initial_stop_loss_pct} initial_stop_loss_pct, - {stoploss_order_id} stoploss_order_id, {stoploss_last_update} stoploss_last_update, - {max_rate} max_rate, {min_rate} min_rate, {sell_reason} sell_reason, - {sell_order_status} sell_order_status, - {strategy} strategy, {timeframe} timeframe, - {open_trade_price} open_trade_price, {close_profit_abs} close_profit_abs - from {table_back_name} - """) - - # Reread columns - the above recreated the table! - inspector = inspect(engine) - cols = inspector.get_columns('trades') - - -def cleanup() -> None: +def cleanup_db() -> None: """ Flushes all pending operations to disk. :return: None @@ -191,13 +95,117 @@ def clean_dry_run_db() -> None: trade.open_order_id = None +class Order(_DECL_BASE): + """ + Order database model + Keeps a record of all orders placed on the exchange + + One to many relationship with Trades: + - One trade can have many orders + - One Order can only be associated with one Trade + + Mirrors CCXT Order structure + """ + __tablename__ = 'orders' + # Uniqueness should be ensured over pair, order_id + # its likely that order_id is unique per Pair on some exchanges. + __table_args__ = (UniqueConstraint('ft_pair', 'order_id', name="_order_pair_order_id"),) + + id = Column(Integer, primary_key=True) + ft_trade_id = Column(Integer, ForeignKey('trades.id'), index=True) + + trade = relationship("Trade", back_populates="orders") + + ft_order_side = Column(String, nullable=False) + ft_pair = Column(String, nullable=False) + ft_is_open = Column(Boolean, nullable=False, default=True, index=True) + + order_id = Column(String, nullable=False, index=True) + status = Column(String, nullable=True) + symbol = Column(String, nullable=True) + order_type = Column(String, nullable=True) + side = Column(String, nullable=True) + price = Column(Float, nullable=True) + amount = Column(Float, nullable=True) + filled = Column(Float, nullable=True) + remaining = Column(Float, nullable=True) + cost = Column(Float, nullable=True) + order_date = Column(DateTime, nullable=True, default=datetime.utcnow) + order_filled_date = Column(DateTime, nullable=True) + order_update_date = Column(DateTime, nullable=True) + + def __repr__(self): + + return (f'Order(id={self.id}, order_id={self.order_id}, trade_id={self.ft_trade_id}, ' + f'side={self.side}, order_type={self.order_type}, status={self.status})') + + def update_from_ccxt_object(self, order): + """ + Update Order from ccxt response + Only updates if fields are available from ccxt - + """ + if self.order_id != str(order['id']): + raise DependencyException("Order-id's don't match") + + self.status = order.get('status', self.status) + self.symbol = order.get('symbol', self.symbol) + self.order_type = order.get('type', self.order_type) + self.side = order.get('side', self.side) + self.price = order.get('price', self.price) + self.amount = order.get('amount', self.amount) + self.filled = order.get('filled', self.filled) + self.remaining = order.get('remaining', self.remaining) + self.cost = order.get('cost', self.cost) + if 'timestamp' in order and order['timestamp'] is not None: + self.order_date = datetime.fromtimestamp(order['timestamp'] / 1000, tz=timezone.utc) + + self.ft_is_open = True + if self.status in ('closed', 'canceled', 'cancelled'): + self.ft_is_open = False + if order.get('filled', 0) > 0: + self.order_filled_date = arrow.utcnow().datetime + self.order_update_date = arrow.utcnow().datetime + + @staticmethod + def update_orders(orders: List['Order'], order: Dict[str, Any]): + """ + Get all non-closed orders - useful when trying to batch-update orders + """ + filtered_orders = [o for o in orders if o.order_id == order.get('id')] + if filtered_orders: + oobj = filtered_orders[0] + oobj.update_from_ccxt_object(order) + else: + logger.warning(f"Did not find order for {order}.") + + @staticmethod + def parse_from_ccxt_object(order: Dict[str, Any], pair: str, side: str) -> 'Order': + """ + Parse an order from a ccxt object and return a new order Object. + """ + o = Order(order_id=str(order['id']), ft_order_side=side, ft_pair=pair) + + o.update_from_ccxt_object(order) + return o + + @staticmethod + def get_open_orders() -> List['Order']: + """ + """ + return Order.query.filter(Order.ft_is_open.is_(True)).all() + + class Trade(_DECL_BASE): """ - Class used to define a trade structure + Trade database model. + Also handles updating and querying trades """ __tablename__ = 'trades' id = Column(Integer, primary_key=True) + + orders = relationship("Order", order_by="Order.id", cascade="all, delete-orphan") + exchange = Column(String, nullable=False) pair = Column(String, nullable=False, index=True) is_open = Column(Boolean, nullable=False, default=True, index=True) @@ -247,7 +255,7 @@ class Trade(_DECL_BASE): self.recalc_open_trade_price() def __repr__(self): - open_since = self.open_date.strftime('%Y-%m-%d %H:%M:%S') if self.is_open else 'closed' + open_since = self.open_date.strftime(DATETIME_PRINT_FORMAT) if self.is_open else 'closed' return (f'Trade(id={self.id}, pair={self.pair}, amount={self.amount:.8f}, ' f'open_rate={self.open_rate:.8f}, open_since={open_since})') @@ -273,7 +281,7 @@ class Trade(_DECL_BASE): 'fee_close_currency': self.fee_close_currency, 'open_date_hum': arrow.get(self.open_date).humanize(), - 'open_date': self.open_date.strftime("%Y-%m-%d %H:%M:%S"), + 'open_date': self.open_date.strftime(DATETIME_PRINT_FORMAT), 'open_timestamp': int(self.open_date.replace(tzinfo=timezone.utc).timestamp() * 1000), 'open_rate': self.open_rate, 'open_rate_requested': self.open_rate_requested, @@ -281,7 +289,7 @@ class Trade(_DECL_BASE): 'close_date_hum': (arrow.get(self.close_date).humanize() if self.close_date else None), - 'close_date': (self.close_date.strftime("%Y-%m-%d %H:%M:%S") + 'close_date': (self.close_date.strftime(DATETIME_PRINT_FORMAT) if self.close_date else None), 'close_timestamp': int(self.close_date.replace( tzinfo=timezone.utc).timestamp() * 1000) if self.close_date else None, @@ -297,7 +305,7 @@ class Trade(_DECL_BASE): 'stop_loss_ratio': self.stop_loss_pct if self.stop_loss_pct else None, 'stop_loss_pct': (self.stop_loss_pct * 100) if self.stop_loss_pct else None, 'stoploss_order_id': self.stoploss_order_id, - 'stoploss_last_update': (self.stoploss_last_update.strftime("%Y-%m-%d %H:%M:%S") + 'stoploss_last_update': (self.stoploss_last_update.strftime(DATETIME_PRINT_FORMAT) if self.stoploss_last_update else None), 'stoploss_last_update_timestamp': int(self.stoploss_last_update.replace( tzinfo=timezone.utc).timestamp() * 1000) if self.stoploss_last_update else None, @@ -380,19 +388,22 @@ class Trade(_DECL_BASE): self.open_rate = Decimal(safe_value_fallback(order, 'average', 'price')) self.amount = Decimal(safe_value_fallback(order, 'filled', 'amount')) self.recalc_open_trade_price() - logger.info('%s_BUY has been fulfilled for %s.', order_type.upper(), self) + if self.is_open: + logger.info(f'{order_type.upper()}_BUY has been fulfilled for {self}.') self.open_order_id = None elif order_type in ('market', 'limit') and order['side'] == 'sell': + if self.is_open: + logger.info(f'{order_type.upper()}_SELL has been fulfilled for {self}.') self.close(safe_value_fallback(order, 'average', 'price')) - logger.info('%s_SELL has been fulfilled for %s.', order_type.upper(), self) elif order_type in ('stop_loss_limit', 'stop-loss', 'stop'): self.stoploss_order_id = None self.close_rate_requested = self.stop_loss - logger.info('%s is hit for %s.', order_type.upper(), self) + if self.is_open: + logger.info(f'{order_type.upper()} is hit for {self}.') self.close(order['average']) else: raise ValueError(f'Unknown order type: {order_type}') - cleanup() + cleanup_db() def close(self, rate: float) -> None: """ @@ -402,7 +413,7 @@ class Trade(_DECL_BASE): self.close_rate = Decimal(rate) self.close_profit = self.calc_profit_ratio() self.close_profit_abs = self.calc_profit() - self.close_date = datetime.utcnow() + self.close_date = self.close_date or datetime.utcnow() self.is_open = False self.sell_order_status = 'closed' self.open_order_id = None @@ -440,6 +451,17 @@ class Trade(_DECL_BASE): else: return False + def update_order(self, order: Dict) -> None: + Order.update_orders(self.orders, order) + + def delete(self) -> None: + + for order in self.orders: + Order.session.delete(order) + + Trade.session.delete(self) + Trade.session.flush() + def _calc_open_trade_price(self) -> float: """ Calculate the open_rate including open_fee. @@ -506,6 +528,21 @@ class Trade(_DECL_BASE): profit_ratio = (close_trade_price / self.open_trade_price) - 1 return float(f"{profit_ratio:.8f}") + def select_order(self, order_side: str, is_open: Optional[bool]) -> Optional[Order]: + """ + Finds latest order for this orderside and status + :param order_side: Side of the order (either 'buy' or 'sell') + :param is_open: Only search for open orders? + :return: latest Order object if it exists, else None + """ + orders = [o for o in self.orders if o.side == order_side] + if is_open is not None: + orders = [o for o in orders if o.ft_is_open == is_open] + if len(orders) > 0: + return orders[-1] + else: + return None + @staticmethod def get_trades(trade_filter=None) -> Query: """ @@ -537,6 +574,26 @@ class Trade(_DECL_BASE): """ return Trade.get_trades(Trade.open_order_id.isnot(None)).all() + @staticmethod + def get_open_trades_without_assigned_fees(): + """ + Returns all open trades which don't have open fees set correctly + """ + return Trade.get_trades([Trade.fee_open_currency.is_(None), + Trade.orders.any(), + Trade.is_open.is_(True), + ]).all() + + @staticmethod + def get_sold_trades_without_assigned_fees(): + """ + Returns all closed trades which don't have fees set correctly + """ + return Trade.get_trades([Trade.fee_close_currency.is_(None), + Trade.orders.any(), + Trade.is_open.is_(False), + ]).all() + @staticmethod def total_open_trades_stakes() -> float: """ @@ -601,3 +658,105 @@ class Trade(_DECL_BASE): trade.stop_loss = None trade.adjust_stop_loss(trade.open_rate, desired_stoploss) logger.info(f"New stoploss: {trade.stop_loss}.") + + +class PairLock(_DECL_BASE): + """ + Pair Locks database model. + """ + __tablename__ = 'pairlocks' + + id = Column(Integer, primary_key=True) + + pair = Column(String, nullable=False, index=True) + reason = Column(String, nullable=True) + # Time the pair was locked (start time) + lock_time = Column(DateTime, nullable=False) + # Time until the pair is locked (end time) + lock_end_time = Column(DateTime, nullable=False, index=True) + + active = Column(Boolean, nullable=False, default=True, index=True) + + def __repr__(self): + lock_time = self.lock_time.strftime(DATETIME_PRINT_FORMAT) + lock_end_time = self.lock_end_time.strftime(DATETIME_PRINT_FORMAT) + return (f'PairLock(id={self.id}, pair={self.pair}, lock_time={lock_time}, ' + f'lock_end_time={lock_end_time})') + + @staticmethod + def lock_pair(pair: str, until: datetime, reason: str = None) -> None: + lock = PairLock( + pair=pair, + lock_time=datetime.now(timezone.utc), + lock_end_time=until, + reason=reason, + active=True + ) + PairLock.session.add(lock) + PairLock.session.flush() + + @staticmethod + def get_pair_locks(pair: Optional[str], now: Optional[datetime] = None) -> List['PairLock']: + """ + Get all locks for this pair + :param pair: Pair to check for. Returns all current locks if pair is empty + :param now: Datetime object (generated via datetime.now(timezone.utc)). + defaults to datetime.utcnow() + """ + if not now: + now = datetime.now(timezone.utc) + + filters = [func.datetime(PairLock.lock_end_time) >= now, + # Only active locks + PairLock.active.is_(True), ] + if pair: + filters.append(PairLock.pair == pair) + return PairLock.query.filter( + *filters + ).all() + + @staticmethod + def unlock_pair(pair: str, now: Optional[datetime] = None) -> None: + """ + Release all locks for this pair. + :param pair: Pair to unlock + :param now: Datetime object (generated via datetime.now(timezone.utc)). + defaults to datetime.utcnow() + """ + if not now: + now = datetime.now(timezone.utc) + + logger.info(f"Releasing all locks for {pair}.") + locks = PairLock.get_pair_locks(pair, now) + for lock in locks: + lock.active = False + PairLock.session.flush() + + @staticmethod + def is_pair_locked(pair: str, now: Optional[datetime] = None) -> bool: + """ + :param pair: Pair to check for + :param now: Datetime object (generated via datetime.now(timezone.utc)). + defaults to datetime.utcnow() + """ + if not now: + now = datetime.now(timezone.utc) + + return PairLock.query.filter( + PairLock.pair == pair, + func.datetime(PairLock.lock_end_time) >= now, + # Only active locks + PairLock.active.is_(True), + ).scalar() is not None + + def to_json(self) -> Dict[str, Any]: + return { + 'pair': self.pair, + 'lock_time': self.lock_time.strftime(DATETIME_PRINT_FORMAT), + 'lock_timestamp': int(self.lock_time.replace(tzinfo=timezone.utc).timestamp() * 1000), + 'lock_end_time': self.lock_end_time.strftime(DATETIME_PRINT_FORMAT), + 'lock_end_timestamp': int(self.lock_end_time.replace(tzinfo=timezone.utc + ).timestamp() * 1000), + 'reason': self.reason, + 'active': self.active, + } diff --git a/freqtrade/plot/plotting.py b/freqtrade/plot/plotting.py index 270fe615b..a89732df5 100644 --- a/freqtrade/plot/plotting.py +++ b/freqtrade/plot/plotting.py @@ -5,11 +5,8 @@ from typing import Any, Dict, List import pandas as pd from freqtrade.configuration import TimeRange -from freqtrade.data.btanalysis import (calculate_max_drawdown, - combine_dataframes_with_mean, - create_cum_profit, - extract_trades_of_period, - load_trades) +from freqtrade.data.btanalysis import (calculate_max_drawdown, combine_dataframes_with_mean, + create_cum_profit, extract_trades_of_period, load_trades) from freqtrade.data.converter import trim_dataframe from freqtrade.data.dataprovider import DataProvider from freqtrade.data.history import load_data @@ -19,13 +16,14 @@ from freqtrade.misc import pair_to_filename from freqtrade.resolvers import ExchangeResolver, StrategyResolver from freqtrade.strategy import IStrategy + logger = logging.getLogger(__name__) try: - from plotly.subplots import make_subplots - from plotly.offline import plot import plotly.graph_objects as go + from plotly.offline import plot + from plotly.subplots import make_subplots except ImportError: logger.exception("Module plotly not found \n Please install using `pip3 install plotly`") exit(1) diff --git a/freqtrade/resolvers/__init__.py b/freqtrade/resolvers/__init__.py index 8f79349fe..b42ec4931 100644 --- a/freqtrade/resolvers/__init__.py +++ b/freqtrade/resolvers/__init__.py @@ -1,6 +1,12 @@ -from freqtrade.resolvers.iresolver import IResolver # noqa: F401 -from freqtrade.resolvers.exchange_resolver import ExchangeResolver # noqa: F401 +# flake8: noqa: F401 +# isort: off +from freqtrade.resolvers.iresolver import IResolver +from freqtrade.resolvers.exchange_resolver import ExchangeResolver +# isort: on # Don't import HyperoptResolver to avoid loading the whole Optimize tree -# from freqtrade.resolvers.hyperopt_resolver import HyperOptResolver # noqa: F401 -from freqtrade.resolvers.pairlist_resolver import PairListResolver # noqa: F401 -from freqtrade.resolvers.strategy_resolver import StrategyResolver # noqa: F401 +# from freqtrade.resolvers.hyperopt_resolver import HyperOptResolver +from freqtrade.resolvers.pairlist_resolver import PairListResolver +from freqtrade.resolvers.strategy_resolver import StrategyResolver + + + diff --git a/freqtrade/resolvers/exchange_resolver.py b/freqtrade/resolvers/exchange_resolver.py index 2b6a731a9..ed6715d15 100644 --- a/freqtrade/resolvers/exchange_resolver.py +++ b/freqtrade/resolvers/exchange_resolver.py @@ -3,10 +3,11 @@ This module loads custom exchanges """ import logging -from freqtrade.exchange import Exchange, MAP_EXCHANGE_CHILDCLASS import freqtrade.exchange as exchanges +from freqtrade.exchange import MAP_EXCHANGE_CHILDCLASS, Exchange from freqtrade.resolvers import IResolver + logger = logging.getLogger(__name__) diff --git a/freqtrade/resolvers/hyperopt_resolver.py b/freqtrade/resolvers/hyperopt_resolver.py index 5dcf73d67..328dc488b 100644 --- a/freqtrade/resolvers/hyperopt_resolver.py +++ b/freqtrade/resolvers/hyperopt_resolver.py @@ -7,12 +7,13 @@ import logging from pathlib import Path from typing import Dict -from freqtrade.constants import DEFAULT_HYPEROPT_LOSS, USERPATH_HYPEROPTS +from freqtrade.constants import USERPATH_HYPEROPTS from freqtrade.exceptions import OperationalException from freqtrade.optimize.hyperopt_interface import IHyperOpt from freqtrade.optimize.hyperopt_loss_interface import IHyperOptLoss from freqtrade.resolvers import IResolver + logger = logging.getLogger(__name__) @@ -69,10 +70,10 @@ class HyperOptLossResolver(IResolver): :param config: configuration dictionary """ - # Verify the hyperopt_loss is in the configuration, otherwise fallback to the - # default hyperopt loss - hyperoptloss_name = config.get('hyperopt_loss') or DEFAULT_HYPEROPT_LOSS - + hyperoptloss_name = config.get('hyperopt_loss') + if not hyperoptloss_name: + raise OperationalException("No Hyperopt loss set. Please use `--hyperopt-loss` to " + "specify the Hyperopt-Loss class to use.") hyperoptloss = HyperOptLossResolver.load_object(hyperoptloss_name, config, kwargs={}, extra_dir=config.get('hyperopt_path')) @@ -81,8 +82,4 @@ class HyperOptLossResolver(IResolver): hyperoptloss.__class__.ticker_interval = str(config['timeframe']) hyperoptloss.__class__.timeframe = str(config['timeframe']) - if not hasattr(hyperoptloss, 'hyperopt_loss_function'): - raise OperationalException( - f"Found HyperoptLoss class {hyperoptloss_name} does not " - "implement `hyperopt_loss_function`.") return hyperoptloss diff --git a/freqtrade/resolvers/iresolver.py b/freqtrade/resolvers/iresolver.py index b7d25ef2c..37cfd70e6 100644 --- a/freqtrade/resolvers/iresolver.py +++ b/freqtrade/resolvers/iresolver.py @@ -11,6 +11,7 @@ from typing import Any, Dict, Iterator, List, Optional, Tuple, Type, Union from freqtrade.exceptions import OperationalException + logger = logging.getLogger(__name__) @@ -50,7 +51,8 @@ class IResolver: :param object_name: Class name of the object :param enum_failed: If True, will return None for modules which fail. Otherwise, failing modules are skipped. - :return: generator containing matching objects + :return: generator containing tuple of matching objects + Tuple format: [Object, source] """ # Generate spec based on absolute path @@ -66,14 +68,16 @@ class IResolver: return iter([None]) valid_objects_gen = ( - obj for name, obj in inspect.getmembers(module, inspect.isclass) - if ((object_name is None or object_name == name) and - issubclass(obj, cls.object_type) and obj is not cls.object_type) + (obj, inspect.getsource(module)) for + name, obj in inspect.getmembers( + module, inspect.isclass) if ((object_name is None or object_name == name) + and issubclass(obj, cls.object_type) + and obj is not cls.object_type) ) return valid_objects_gen @classmethod - def _search_object(cls, directory: Path, object_name: str + def _search_object(cls, directory: Path, *, object_name: str, add_source: bool = False ) -> Union[Tuple[Any, Path], Tuple[None, None]]: """ Search for the objectname in the given directory @@ -92,11 +96,14 @@ class IResolver: obj = next(cls._get_valid_object(module_path, object_name), None) if obj: - return (obj, module_path) + obj[0].__file__ = str(entry) + if add_source: + obj[0].__source__ = obj[1] + return (obj[0], module_path) return (None, None) @classmethod - def _load_object(cls, paths: List[Path], object_name: str, + def _load_object(cls, paths: List[Path], *, object_name: str, add_source: bool = False, kwargs: dict = {}) -> Optional[Any]: """ Try to load object from path list. @@ -105,7 +112,8 @@ class IResolver: for _path in paths: try: (module, module_path) = cls._search_object(directory=_path, - object_name=object_name) + object_name=object_name, + add_source=add_source) if module: logger.info( f"Using resolved {cls.object_type.__name__.lower()[1:]} {object_name} " @@ -117,7 +125,7 @@ class IResolver: return None @classmethod - def load_object(cls, object_name: str, config: dict, kwargs: dict, + def load_object(cls, object_name: str, config: dict, *, kwargs: dict, extra_dir: Optional[str] = None) -> Any: """ Search and loads the specified object as configured in hte child class. @@ -132,10 +140,10 @@ class IResolver: user_subdir=cls.user_subdir, extra_dir=extra_dir) - pairlist = cls._load_object(paths=abs_paths, object_name=object_name, - kwargs=kwargs) - if pairlist: - return pairlist + found_object = cls._load_object(paths=abs_paths, object_name=object_name, + kwargs=kwargs) + if found_object: + return found_object raise OperationalException( f"Impossible to load {cls.object_type_str} '{object_name}'. This class does not exist " "or contains Python code errors." @@ -163,8 +171,8 @@ class IResolver: for obj in cls._get_valid_object(module_path, object_name=None, enum_failed=enum_failed): objects.append( - {'name': obj.__name__ if obj is not None else '', - 'class': obj, + {'name': obj[0].__name__ if obj is not None else '', + 'class': obj[0] if obj is not None else None, 'location': entry, }) return objects diff --git a/freqtrade/resolvers/pairlist_resolver.py b/freqtrade/resolvers/pairlist_resolver.py index 77db74084..4df5da37c 100644 --- a/freqtrade/resolvers/pairlist_resolver.py +++ b/freqtrade/resolvers/pairlist_resolver.py @@ -9,6 +9,7 @@ from pathlib import Path from freqtrade.pairlist.IPairList import IPairList from freqtrade.resolvers import IResolver + logger = logging.getLogger(__name__) diff --git a/freqtrade/resolvers/strategy_resolver.py b/freqtrade/resolvers/strategy_resolver.py index 121a04877..63a3f784e 100644 --- a/freqtrade/resolvers/strategy_resolver.py +++ b/freqtrade/resolvers/strategy_resolver.py @@ -11,12 +11,12 @@ from inspect import getfullargspec from pathlib import Path from typing import Any, Dict, Optional -from freqtrade.constants import (REQUIRED_ORDERTIF, REQUIRED_ORDERTYPES, - USERPATH_STRATEGIES) +from freqtrade.constants import REQUIRED_ORDERTIF, REQUIRED_ORDERTYPES, USERPATH_STRATEGIES from freqtrade.exceptions import OperationalException from freqtrade.resolvers import IResolver from freqtrade.strategy.interface import IStrategy + logger = logging.getLogger(__name__) @@ -174,7 +174,9 @@ class StrategyResolver(IResolver): strategy = StrategyResolver._load_object(paths=abs_paths, object_name=strategy_name, - kwargs={'config': config}) + add_source=True, + kwargs={'config': config}, + ) if strategy: strategy._populate_fun_len = len(getfullargspec(strategy.populate_indicators).args) strategy._buy_fun_len = len(getfullargspec(strategy.populate_buy_trend).args) diff --git a/freqtrade/rpc/__init__.py b/freqtrade/rpc/__init__.py index 31c854f82..88978519b 100644 --- a/freqtrade/rpc/__init__.py +++ b/freqtrade/rpc/__init__.py @@ -1,2 +1,3 @@ -from .rpc import RPC, RPCMessageType, RPCException # noqa -from .rpc_manager import RPCManager # noqa +# flake8: noqa: F401 +from .rpc import RPC, RPCException, RPCMessageType +from .rpc_manager import RPCManager diff --git a/freqtrade/rpc/api_server.py b/freqtrade/rpc/api_server.py index 0ae0698cd..be21179ad 100644 --- a/freqtrade/rpc/api_server.py +++ b/freqtrade/rpc/api_server.py @@ -1,40 +1,43 @@ import logging import threading +from copy import deepcopy from datetime import date, datetime from ipaddress import IPv4Address +from pathlib import Path from typing import Any, Callable, Dict from arrow import Arrow from flask import Flask, jsonify, request from flask.json import JSONEncoder from flask_cors import CORS -from flask_jwt_extended import (JWTManager, create_access_token, - create_refresh_token, get_jwt_identity, - jwt_refresh_token_required, +from flask_jwt_extended import (JWTManager, create_access_token, create_refresh_token, + get_jwt_identity, jwt_refresh_token_required, verify_jwt_in_request_optional) from werkzeug.security import safe_str_cmp from werkzeug.serving import make_server from freqtrade.__init__ import __version__ -from freqtrade.constants import DATETIME_PRINT_FORMAT +from freqtrade.constants import DATETIME_PRINT_FORMAT, USERPATH_STRATEGIES +from freqtrade.exceptions import OperationalException from freqtrade.persistence import Trade from freqtrade.rpc.fiat_convert import CryptoToFiatConverter from freqtrade.rpc.rpc import RPC, RPCException + logger = logging.getLogger(__name__) BASE_URI = "/api/v1" -class ArrowJSONEncoder(JSONEncoder): +class FTJSONEncoder(JSONEncoder): def default(self, obj): try: if isinstance(obj, Arrow): return obj.for_json() - elif isinstance(obj, date): - return obj.strftime("%Y-%m-%d") elif isinstance(obj, datetime): return obj.strftime(DATETIME_PRINT_FORMAT) + elif isinstance(obj, date): + return obj.strftime("%Y-%m-%d") iterable = iter(obj) except TypeError: pass @@ -108,7 +111,7 @@ class ApiServer(RPC): 'jwt_secret_key', 'super-secret') self.jwt = JWTManager(self.app) - self.app.json_encoder = ArrowJSONEncoder + self.app.json_encoder = FTJSONEncoder self.app.teardown_appcontext(shutdown_session) @@ -160,16 +163,12 @@ class ApiServer(RPC): """ pass - def rest_dump(self, return_value): - """ Helper function to jsonify object for a webserver """ - return jsonify(return_value) - - def rest_error(self, error_msg): - return jsonify({"error": error_msg}), 502 + def rest_error(self, error_msg, error_code=502): + return jsonify({"error": error_msg}), error_code def register_rest_rpc_urls(self): """ - Registers flask app URLs that are calls to functonality in rpc.rpc. + Registers flask app URLs that are calls to functionality in rpc.rpc. First two arguments passed are /URL and 'Label' Label can be used as a shortcut when refactoring @@ -193,6 +192,7 @@ class ApiServer(RPC): self.app.add_url_rule(f'{BASE_URI}/balance', 'balance', view_func=self._balance, methods=['GET']) self.app.add_url_rule(f'{BASE_URI}/count', 'count', view_func=self._count, methods=['GET']) + self.app.add_url_rule(f'{BASE_URI}/locks', 'locks', view_func=self._locks, methods=['GET']) self.app.add_url_rule(f'{BASE_URI}/daily', 'daily', view_func=self._daily, methods=['GET']) self.app.add_url_rule(f'{BASE_URI}/edge', 'edge', view_func=self._edge, methods=['GET']) self.app.add_url_rule(f'{BASE_URI}/logs', 'log', view_func=self._get_logs, methods=['GET']) @@ -212,6 +212,20 @@ class ApiServer(RPC): view_func=self._trades, methods=['GET']) self.app.add_url_rule(f'{BASE_URI}/trades/', 'trades_delete', view_func=self._trades_delete, methods=['DELETE']) + + self.app.add_url_rule(f'{BASE_URI}/pair_candles', 'pair_candles', + view_func=self._analysed_candles, methods=['GET']) + self.app.add_url_rule(f'{BASE_URI}/pair_history', 'pair_history', + view_func=self._analysed_history, methods=['GET']) + self.app.add_url_rule(f'{BASE_URI}/plot_config', 'plot_config', + view_func=self._plot_config, methods=['GET']) + self.app.add_url_rule(f'{BASE_URI}/strategies', 'strategies', + view_func=self._list_strategies, methods=['GET']) + self.app.add_url_rule(f'{BASE_URI}/strategy/', 'strategy', + view_func=self._get_strategy, methods=['GET']) + self.app.add_url_rule(f'{BASE_URI}/available_pairs', 'pairs', + view_func=self._list_available_pairs, methods=['GET']) + # Combined actions and infos self.app.add_url_rule(f'{BASE_URI}/blacklist', 'blacklist', view_func=self._blacklist, methods=['GET', 'POST']) @@ -222,15 +236,12 @@ class ApiServer(RPC): self.app.add_url_rule(f'{BASE_URI}/forcesell', 'forcesell', view_func=self._forcesell, methods=['POST']) - # TODO: Implement the following - # help (?) - @require_login def page_not_found(self, error): """ Return "404 not found", 404. """ - return self.rest_dump({ + return jsonify({ 'status': 'error', 'reason': f"There's no API call for {request.base_url}.", 'code': 404 @@ -250,7 +261,7 @@ class ApiServer(RPC): 'access_token': create_access_token(identity=keystuff), 'refresh_token': create_refresh_token(identity=keystuff), } - return self.rest_dump(ret) + return jsonify(ret) return jsonify({"error": "Unauthorized"}), 401 @@ -265,7 +276,7 @@ class ApiServer(RPC): new_token = create_access_token(identity=current_user, fresh=False) ret = {'access_token': new_token} - return self.rest_dump(ret) + return jsonify(ret) @require_login @rpc_catch_errors @@ -275,7 +286,7 @@ class ApiServer(RPC): Starts TradeThread in bot if stopped. """ msg = self._rpc_start() - return self.rest_dump(msg) + return jsonify(msg) @require_login @rpc_catch_errors @@ -285,7 +296,7 @@ class ApiServer(RPC): Stops TradeThread in bot if running """ msg = self._rpc_stop() - return self.rest_dump(msg) + return jsonify(msg) @require_login @rpc_catch_errors @@ -295,14 +306,14 @@ class ApiServer(RPC): Sets max_open_trades to 0 and gracefully sells all open trades """ msg = self._rpc_stopbuy() - return self.rest_dump(msg) + return jsonify(msg) @rpc_catch_errors def _ping(self): """ - simple poing version + simple ping version """ - return self.rest_dump({"status": "pong"}) + return jsonify({"status": "pong"}) @require_login @rpc_catch_errors @@ -310,7 +321,7 @@ class ApiServer(RPC): """ Prints the bot's version """ - return self.rest_dump({"version": __version__}) + return jsonify({"version": __version__}) @require_login @rpc_catch_errors @@ -318,7 +329,7 @@ class ApiServer(RPC): """ Prints the bot's version """ - return self.rest_dump(self._rpc_show_config()) + return jsonify(self._rpc_show_config(self._config)) @require_login @rpc_catch_errors @@ -328,7 +339,7 @@ class ApiServer(RPC): Triggers a config file reload """ msg = self._rpc_reload_config() - return self.rest_dump(msg) + return jsonify(msg) @require_login @rpc_catch_errors @@ -338,7 +349,16 @@ class ApiServer(RPC): Returns the number of trades running """ msg = self._rpc_count() - return self.rest_dump(msg) + return jsonify(msg) + + @require_login + @rpc_catch_errors + def _locks(self): + """ + Handler for /locks. + Returns the currently active locks. + """ + return jsonify(self._rpc_locks()) @require_login @rpc_catch_errors @@ -356,7 +376,7 @@ class ApiServer(RPC): self._config.get('fiat_display_currency', '') ) - return self.rest_dump(stats) + return jsonify(stats) @require_login @rpc_catch_errors @@ -368,7 +388,7 @@ class ApiServer(RPC): limit: Only get a certain number of records """ limit = int(request.args.get('limit', 0)) or None - return self.rest_dump(self._rpc_get_logs(limit)) + return jsonify(self._rpc_get_logs(limit)) @require_login @rpc_catch_errors @@ -379,7 +399,7 @@ class ApiServer(RPC): """ stats = self._rpc_edge() - return self.rest_dump(stats) + return jsonify(stats) @require_login @rpc_catch_errors @@ -395,7 +415,7 @@ class ApiServer(RPC): self._config.get('fiat_display_currency') ) - return self.rest_dump(stats) + return jsonify(stats) @require_login @rpc_catch_errors @@ -408,7 +428,7 @@ class ApiServer(RPC): """ stats = self._rpc_performance() - return self.rest_dump(stats) + return jsonify(stats) @require_login @rpc_catch_errors @@ -420,9 +440,9 @@ class ApiServer(RPC): """ try: results = self._rpc_trade_status() - return self.rest_dump(results) + return jsonify(results) except RPCException: - return self.rest_dump([]) + return jsonify([]) @require_login @rpc_catch_errors @@ -434,7 +454,7 @@ class ApiServer(RPC): """ results = self._rpc_balance(self._config['stake_currency'], self._config.get('fiat_display_currency', '')) - return self.rest_dump(results) + return jsonify(results) @require_login @rpc_catch_errors @@ -446,7 +466,7 @@ class ApiServer(RPC): """ limit = int(request.args.get('limit', 0)) results = self._rpc_trade_history(limit) - return self.rest_dump(results) + return jsonify(results) @require_login @rpc_catch_errors @@ -459,7 +479,7 @@ class ApiServer(RPC): tradeid: Numeric trade-id assigned to the trade. """ result = self._rpc_delete(tradeid) - return self.rest_dump(result) + return jsonify(result) @require_login @rpc_catch_errors @@ -468,7 +488,7 @@ class ApiServer(RPC): Handler for /whitelist. """ results = self._rpc_whitelist() - return self.rest_dump(results) + return jsonify(results) @require_login @rpc_catch_errors @@ -478,7 +498,7 @@ class ApiServer(RPC): """ add = request.json.get("blacklist", None) if request.method == 'POST' else None results = self._rpc_blacklist(add) - return self.rest_dump(results) + return jsonify(results) @require_login @rpc_catch_errors @@ -490,9 +510,9 @@ class ApiServer(RPC): price = request.json.get("price", None) trade = self._rpc_forcebuy(asset, price) if trade: - return self.rest_dump(trade.to_json()) + return jsonify(trade.to_json()) else: - return self.rest_dump({"status": f"Error buying pair {asset}."}) + return jsonify({"status": f"Error buying pair {asset}."}) @require_login @rpc_catch_errors @@ -502,4 +522,132 @@ class ApiServer(RPC): """ tradeid = request.json.get("tradeid") results = self._rpc_forcesell(tradeid) - return self.rest_dump(results) + return jsonify(results) + + @require_login + @rpc_catch_errors + def _analysed_candles(self): + """ + Handler for /pair_candles. + Returns the dataframe the bot is using during live/dry operations. + Takes the following get arguments: + get: + parameters: + - pair: Pair + - timeframe: Timeframe to get data for (should be aligned to strategy.timeframe) + - limit: Limit return length to the latest X candles + """ + pair = request.args.get("pair") + timeframe = request.args.get("timeframe") + limit = request.args.get("limit", type=int) + if not pair or not timeframe: + return self.rest_error("Mandatory parameter missing.", 400) + + results = self._rpc_analysed_dataframe(pair, timeframe, limit) + return jsonify(results) + + @require_login + @rpc_catch_errors + def _analysed_history(self): + """ + Handler for /pair_history. + Returns the dataframe of a given timerange + Takes the following get arguments: + get: + parameters: + - pair: Pair + - timeframe: Timeframe to get data for (should be aligned to strategy.timeframe) + - strategy: Strategy to use - Must exist in configured strategy-path! + - timerange: timerange in the format YYYYMMDD-YYYYMMDD (YYYYMMDD- or (-YYYYMMDD)) + are als possible. If omitted uses all available data. + """ + pair = request.args.get("pair") + timeframe = request.args.get("timeframe") + timerange = request.args.get("timerange") + strategy = request.args.get("strategy") + + if not pair or not timeframe or not timerange or not strategy: + return self.rest_error("Mandatory parameter missing.", 400) + + config = deepcopy(self._config) + config.update({ + 'strategy': strategy, + }) + results = RPC._rpc_analysed_history_full(config, pair, timeframe, timerange) + return jsonify(results) + + @require_login + @rpc_catch_errors + def _plot_config(self): + """ + Handler for /plot_config. + """ + return jsonify(self._rpc_plot_config()) + + @require_login + @rpc_catch_errors + def _list_strategies(self): + directory = Path(self._config.get( + 'strategy_path', self._config['user_data_dir'] / USERPATH_STRATEGIES)) + from freqtrade.resolvers.strategy_resolver import StrategyResolver + strategy_objs = StrategyResolver.search_all_objects(directory, False) + strategy_objs = sorted(strategy_objs, key=lambda x: x['name']) + + return jsonify({'strategies': [x['name'] for x in strategy_objs]}) + + @require_login + @rpc_catch_errors + def _get_strategy(self, strategy: str): + """ + Get a single strategy + get: + parameters: + - strategy: Only get this strategy + """ + config = deepcopy(self._config) + from freqtrade.resolvers.strategy_resolver import StrategyResolver + try: + strategy_obj = StrategyResolver._load_strategy(strategy, config, + extra_dir=config.get('strategy_path')) + except OperationalException: + return self.rest_error("Strategy not found.", 404) + + return jsonify({ + 'strategy': strategy_obj.get_strategy_name(), + 'code': strategy_obj.__source__, + }) + + @require_login + @rpc_catch_errors + def _list_available_pairs(self): + """ + Handler for /available_pairs. + Returns an object, with pairs, available pair length and pair_interval combinations + Takes the following get arguments: + get: + parameters: + - stake_currency: Filter on this stake currency + - timeframe: Timeframe to get data for Filter elements to this timeframe + """ + timeframe = request.args.get("timeframe") + stake_currency = request.args.get("stake_currency") + + from freqtrade.data.history import get_datahandler + dh = get_datahandler(self._config['datadir'], self._config.get('dataformat_ohlcv', None)) + + pair_interval = dh.ohlcv_get_available_data(self._config['datadir']) + + if timeframe: + pair_interval = [pair for pair in pair_interval if pair[1] == timeframe] + if stake_currency: + pair_interval = [pair for pair in pair_interval if pair[0].endswith(stake_currency)] + pair_interval = sorted(pair_interval, key=lambda x: x[0]) + + pairs = list({x[0] for x in pair_interval}) + + result = { + 'length': len(pairs), + 'pairs': pairs, + 'pair_interval': pair_interval, + } + return jsonify(result) diff --git a/freqtrade/rpc/rpc.py b/freqtrade/rpc/rpc.py index 0b9196f2e..de8bcaefb 100644 --- a/freqtrade/rpc/rpc.py +++ b/freqtrade/rpc/rpc.py @@ -9,25 +9,29 @@ from math import isnan from typing import Any, Dict, List, Optional, Tuple, Union import arrow -from numpy import NAN, mean +from numpy import NAN, int64, mean +from pandas import DataFrame -from freqtrade.constants import CANCEL_REASON +from freqtrade.configuration.timerange import TimeRange +from freqtrade.constants import CANCEL_REASON, DATETIME_PRINT_FORMAT +from freqtrade.data.history import load_data from freqtrade.exceptions import ExchangeError, PricingError from freqtrade.exchange import timeframe_to_minutes, timeframe_to_msecs from freqtrade.loggers import bufferHandler from freqtrade.misc import shorten_date -from freqtrade.persistence import Trade +from freqtrade.persistence import PairLock, Trade from freqtrade.rpc.fiat_convert import CryptoToFiatConverter from freqtrade.state import State from freqtrade.strategy.interface import SellType + logger = logging.getLogger(__name__) class RPCMessageType(Enum): STATUS_NOTIFICATION = 'status' WARNING_NOTIFICATION = 'warning' - CUSTOM_NOTIFICATION = 'custom' + STARTUP_NOTIFICATION = 'startup' BUY_NOTIFICATION = 'buy' BUY_CANCEL_NOTIFICATION = 'buy_cancel' SELL_NOTIFICATION = 'sell' @@ -36,6 +40,9 @@ class RPCMessageType(Enum): def __repr__(self): return self.value + def __str__(self): + return self.value + class RPCException(Exception): """ @@ -86,13 +93,12 @@ class RPC: def send_msg(self, msg: Dict[str, str]) -> None: """ Sends a message to all registered rpc modules """ - def _rpc_show_config(self) -> Dict[str, Any]: + def _rpc_show_config(self, config) -> Dict[str, Any]: """ Return a dict of config options. Explicitly does NOT return the full config to avoid leakage of sensitive information via rpc. """ - config = self._freqtrade.config val = { 'dry_run': config['dry_run'], 'stake_currency': config['stake_currency'], @@ -113,7 +119,7 @@ class RPC: 'forcebuy_enabled': config.get('forcebuy_enable', False), 'ask_strategy': config.get('ask_strategy', {}), 'bid_strategy': config.get('bid_strategy', {}), - 'state': str(self._freqtrade.state) + 'state': str(self._freqtrade.state) if self._freqtrade else '', } return val @@ -562,8 +568,7 @@ class RPC: except (ExchangeError): pass - Trade.session.delete(trade) - Trade.session.flush() + trade.delete() self._freqtrade.wallets.update() return { 'result': 'success', @@ -594,6 +599,17 @@ class RPC: 'total_stake': sum((trade.open_rate * trade.amount) for trade in trades) } + def _rpc_locks(self) -> Dict[str, Any]: + """ Returns the current locks""" + if self._freqtrade.state != State.RUNNING: + raise RPCException('trader is not running') + + locks = PairLock.get_pair_locks(None) + return { + 'lock_count': len(locks), + 'locks': [lock.to_json() for lock in locks] + } + def _rpc_whitelist(self) -> Dict: """ Returns the currently active whitelist""" res = {'method': self._freqtrade.pairlists.name_list, @@ -633,7 +649,7 @@ class RPC: buffer = bufferHandler.buffer[-limit:] else: buffer = bufferHandler.buffer - records = [[datetime.fromtimestamp(r.created).strftime("%Y-%m-%d %H:%M:%S"), + records = [[datetime.fromtimestamp(r.created).strftime(DATETIME_PRINT_FORMAT), r.created * 1000, r.name, r.levelname, r.message + ('\n' + r.exc_text if r.exc_text else '')] for r in buffer] @@ -650,3 +666,82 @@ class RPC: if not self._freqtrade.edge: raise RPCException('Edge is not enabled.') return self._freqtrade.edge.accepted_pairs() + + @staticmethod + def _convert_dataframe_to_dict(strategy: str, pair: str, timeframe: str, dataframe: DataFrame, + last_analyzed: datetime) -> Dict[str, Any]: + has_content = len(dataframe) != 0 + buy_signals = 0 + sell_signals = 0 + if has_content: + + dataframe.loc[:, '__date_ts'] = dataframe.loc[:, 'date'].astype(int64) // 1000 // 1000 + # Move open to seperate column when signal for easy plotting + if 'buy' in dataframe.columns: + buy_mask = (dataframe['buy'] == 1) + buy_signals = int(buy_mask.sum()) + dataframe.loc[buy_mask, '_buy_signal_open'] = dataframe.loc[buy_mask, 'open'] + if 'sell' in dataframe.columns: + sell_mask = (dataframe['sell'] == 1) + sell_signals = int(sell_mask.sum()) + dataframe.loc[sell_mask, '_sell_signal_open'] = dataframe.loc[sell_mask, 'open'] + dataframe = dataframe.replace({NAN: None}) + + res = { + 'pair': pair, + 'timeframe': timeframe, + 'timeframe_ms': timeframe_to_msecs(timeframe), + 'strategy': strategy, + 'columns': list(dataframe.columns), + 'data': dataframe.values.tolist(), + 'length': len(dataframe), + 'buy_signals': buy_signals, + 'sell_signals': sell_signals, + 'last_analyzed': last_analyzed, + 'last_analyzed_ts': int(last_analyzed.timestamp()), + 'data_start': '', + 'data_start_ts': 0, + 'data_stop': '', + 'data_stop_ts': 0, + } + if has_content: + res.update({ + 'data_start': str(dataframe.iloc[0]['date']), + 'data_start_ts': int(dataframe.iloc[0]['__date_ts']), + 'data_stop': str(dataframe.iloc[-1]['date']), + 'data_stop_ts': int(dataframe.iloc[-1]['__date_ts']), + }) + return res + + def _rpc_analysed_dataframe(self, pair: str, timeframe: str, limit: int) -> Dict[str, Any]: + + _data, last_analyzed = self._freqtrade.dataprovider.get_analyzed_dataframe( + pair, timeframe) + _data = _data.copy() + if limit: + _data = _data.iloc[-limit:] + return self._convert_dataframe_to_dict(self._freqtrade.config['strategy'], + pair, timeframe, _data, last_analyzed) + + @staticmethod + def _rpc_analysed_history_full(config, pair: str, timeframe: str, + timerange: str) -> Dict[str, Any]: + timerange_parsed = TimeRange.parse_timerange(timerange) + + _data = load_data( + datadir=config.get("datadir"), + pairs=[pair], + timeframe=timeframe, + timerange=timerange_parsed, + data_format=config.get('dataformat_ohlcv', 'json'), + ) + from freqtrade.resolvers.strategy_resolver import StrategyResolver + strategy = StrategyResolver.load_strategy(config) + df_analyzed = strategy.analyze_ticker(_data[pair], {'pair': pair}) + + return RPC._convert_dataframe_to_dict(strategy.get_strategy_name(), pair, timeframe, + df_analyzed, arrow.Arrow.utcnow().datetime) + + def _rpc_plot_config(self) -> Dict[str, Any]: + + return self._freqtrade.strategy.plot_config diff --git a/freqtrade/rpc/rpc_manager.py b/freqtrade/rpc/rpc_manager.py index 2cb44fec8..b97a5357b 100644 --- a/freqtrade/rpc/rpc_manager.py +++ b/freqtrade/rpc/rpc_manager.py @@ -6,6 +6,7 @@ from typing import Any, Dict, List from freqtrade.rpc import RPC, RPCMessageType + logger = logging.getLogger(__name__) @@ -59,7 +60,7 @@ class RPCManager: try: mod.send_msg(msg) except NotImplementedError: - logger.error(f"Message type {msg['type']} not implemented by handler {mod.name}.") + logger.error(f"Message type '{msg['type']}' not implemented by handler {mod.name}.") def startup_messages(self, config: Dict[str, Any], pairlist) -> None: if config['dry_run']: @@ -76,7 +77,7 @@ class RPCManager: exchange_name = config['exchange']['name'] strategy_name = config.get('strategy', '') self.send_msg({ - 'type': RPCMessageType.CUSTOM_NOTIFICATION, + 'type': RPCMessageType.STARTUP_NOTIFICATION, 'status': f'*Exchange:* `{exchange_name}`\n' f'*Stake per trade:* `{stake_amount} {stake_currency}`\n' f'*Minimum ROI:* `{minimal_roi}`\n' @@ -85,7 +86,7 @@ class RPCManager: f'*Strategy:* `{strategy_name}`' }) self.send_msg({ - 'type': RPCMessageType.STATUS_NOTIFICATION, + 'type': RPCMessageType.STARTUP_NOTIFICATION, 'status': f'Searching for {stake_currency} pairs to buy and sell ' f'based on {pairlist.short_desc()}' }) diff --git a/freqtrade/rpc/telegram.py b/freqtrade/rpc/telegram.py index 8404625f1..4a6736942 100644 --- a/freqtrade/rpc/telegram.py +++ b/freqtrade/rpc/telegram.py @@ -5,9 +5,9 @@ This module manage Telegram communication """ import json import logging -import arrow from typing import Any, Callable, Dict, List +import arrow from tabulate import tabulate from telegram import ParseMode, ReplyKeyboardMarkup, Update from telegram.error import NetworkError, TelegramError @@ -18,6 +18,7 @@ from freqtrade.__init__ import __version__ from freqtrade.rpc import RPC, RPCException, RPCMessageType from freqtrade.rpc.fiat_convert import CryptoToFiatConverter + logger = logging.getLogger(__name__) logger.debug('Included module rpc.telegram ...') @@ -99,6 +100,7 @@ class Telegram(RPC): CommandHandler('performance', self._performance), CommandHandler('daily', self._daily), CommandHandler('count', self._count), + CommandHandler('locks', self._locks), CommandHandler(['reload_config', 'reload_conf'], self._reload_config), CommandHandler(['show_config', 'show_conf'], self._show_config), CommandHandler('stopbuy', self._stopbuy), @@ -133,6 +135,13 @@ class Telegram(RPC): def send_msg(self, msg: Dict[str, Any]) -> None: """ Send a message to telegram channel """ + noti = self._config['telegram'].get('notification_settings', {} + ).get(str(msg['type']), 'on') + if noti == 'off': + logger.info(f"Notification '{msg['type']}' not sent.") + # Notification disabled + return + if msg['type'] == RPCMessageType.BUY_NOTIFICATION: if self._fiat_converter: msg['stake_amount_fiat'] = self._fiat_converter.convert_amount( @@ -191,13 +200,13 @@ class Telegram(RPC): elif msg['type'] == RPCMessageType.WARNING_NOTIFICATION: message = '\N{WARNING SIGN} *Warning:* `{status}`'.format(**msg) - elif msg['type'] == RPCMessageType.CUSTOM_NOTIFICATION: + elif msg['type'] == RPCMessageType.STARTUP_NOTIFICATION: message = '{status}'.format(**msg) else: raise NotImplementedError('Unknown message type: {}'.format(msg['type'])) - self._send_msg(message) + self._send_msg(message, disable_notification=(noti == 'silent')) def _get_sell_emoji(self, msg): """ @@ -601,6 +610,26 @@ class Telegram(RPC): except RPCException as e: self._send_msg(str(e)) + @authorized_only + def _locks(self, update: Update, context: CallbackContext) -> None: + """ + Handler for /locks. + Returns the currently active locks + """ + try: + locks = self._rpc_locks() + message = tabulate([[ + lock['pair'], + lock['lock_end_time'], + lock['reason']] for lock in locks['locks']], + headers=['Pair', 'Until', 'Reason'], + tablefmt='simple') + message = "
{}
".format(message) + logger.debug(message) + self._send_msg(message, parse_mode=ParseMode.HTML) + except RPCException as e: + self._send_msg(str(e)) + @authorized_only def _whitelist(self, update: Update, context: CallbackContext) -> None: """ @@ -712,8 +741,8 @@ class Telegram(RPC): "*/delete :* `Instantly delete the given trade in the database`\n" "*/performance:* `Show performance of each finished trade grouped by pair`\n" "*/daily :* `Shows profit or loss per day, over the last n days`\n" - "*/count:* `Show number of trades running compared to allowed number of trades`" - "\n" + "*/count:* `Show number of active trades compared to allowed number of trades`\n" + "*/locks:* `Show currently locked pairs`\n" "*/balance:* `Show account balance per currency`\n" "*/stopbuy:* `Stops buying, but handles open trades gracefully` \n" "*/reload_config:* `Reload configuration file` \n" @@ -804,7 +833,7 @@ class Telegram(RPC): :param update: message update :return: None """ - val = self._rpc_show_config() + val = self._rpc_show_config(self._freqtrade.config) if val['trailing_stop']: sl_info = ( f"*Initial Stoploss:* `{val['stoploss']}`\n" @@ -830,7 +859,8 @@ class Telegram(RPC): f"*Current state:* `{val['state']}`" ) - def _send_msg(self, msg: str, parse_mode: ParseMode = ParseMode.MARKDOWN) -> None: + def _send_msg(self, msg: str, parse_mode: ParseMode = ParseMode.MARKDOWN, + disable_notification: bool = False) -> None: """ Send given markdown message :param msg: message @@ -851,7 +881,8 @@ class Telegram(RPC): self._config['telegram']['chat_id'], text=msg, parse_mode=parse_mode, - reply_markup=reply_markup + reply_markup=reply_markup, + disable_notification=disable_notification, ) except NetworkError as network_err: # Sometimes the telegram server resets the current connection, @@ -864,7 +895,8 @@ class Telegram(RPC): self._config['telegram']['chat_id'], text=msg, parse_mode=parse_mode, - reply_markup=reply_markup + reply_markup=reply_markup, + disable_notification=disable_notification, ) except TelegramError as telegram_err: logger.warning( diff --git a/freqtrade/rpc/webhook.py b/freqtrade/rpc/webhook.py index 322d990ee..21413f165 100644 --- a/freqtrade/rpc/webhook.py +++ b/freqtrade/rpc/webhook.py @@ -2,9 +2,9 @@ This module manages webhook communication """ import logging -from typing import Any, Dict +from typing import Any, Dict -from requests import post, RequestException +from requests import RequestException, post from freqtrade.rpc import RPC, RPCMessageType @@ -48,13 +48,13 @@ class Webhook(RPC): elif msg['type'] == RPCMessageType.SELL_CANCEL_NOTIFICATION: valuedict = self._config['webhook'].get('webhooksellcancel', None) elif msg['type'] in (RPCMessageType.STATUS_NOTIFICATION, - RPCMessageType.CUSTOM_NOTIFICATION, + RPCMessageType.STARTUP_NOTIFICATION, RPCMessageType.WARNING_NOTIFICATION): valuedict = self._config['webhook'].get('webhookstatus', None) else: raise NotImplementedError('Unknown message type: {}'.format(msg['type'])) if not valuedict: - logger.info("Message type %s not configured for webhooks", msg['type']) + logger.info("Message type '%s' not configured for webhooks", msg['type']) return payload = {key: value.format(**msg) for (key, value) in valuedict.items()} diff --git a/freqtrade/strategy/__init__.py b/freqtrade/strategy/__init__.py index d1510489e..662156ae9 100644 --- a/freqtrade/strategy/__init__.py +++ b/freqtrade/strategy/__init__.py @@ -1,5 +1,5 @@ # flake8: noqa: F401 -from freqtrade.exchange import (timeframe_to_minutes, timeframe_to_prev_date, - timeframe_to_seconds, timeframe_to_next_date, timeframe_to_msecs) +from freqtrade.exchange import (timeframe_to_minutes, timeframe_to_msecs, timeframe_to_next_date, + timeframe_to_prev_date, timeframe_to_seconds) from freqtrade.strategy.interface import IStrategy from freqtrade.strategy.strategy_helper import merge_informative_pair diff --git a/freqtrade/strategy/interface.py b/freqtrade/strategy/interface.py index 92d9f6c48..e6256cafb 100644 --- a/freqtrade/strategy/interface.py +++ b/freqtrade/strategy/interface.py @@ -17,10 +17,11 @@ from freqtrade.data.dataprovider import DataProvider from freqtrade.exceptions import OperationalException, StrategyError from freqtrade.exchange import timeframe_to_minutes from freqtrade.exchange.exchange import timeframe_to_next_date -from freqtrade.persistence import Trade +from freqtrade.persistence import PairLock, Trade from freqtrade.strategy.strategy_wrapper import strategy_safe_wrapper from freqtrade.wallets import Wallets + logger = logging.getLogger(__name__) @@ -122,6 +123,8 @@ class IStrategy(ABC): # and wallets - access to the current balance. dp: Optional[DataProvider] = None wallets: Optional[Wallets] = None + # container variable for strategy source code + __source__: str = '' # Definition of plot_config. See plotting documentation for more details. plot_config: Dict = {} @@ -130,7 +133,6 @@ class IStrategy(ABC): self.config = config # Dict to determine if analysis is necessary self._last_candle_seen_per_pair: Dict[str, datetime] = {} - self._pair_locked_until: Dict[str, datetime] = {} @abstractmethod def populate_indicators(self, dataframe: DataFrame, metadata: dict) -> DataFrame: @@ -275,7 +277,7 @@ class IStrategy(ABC): """ return self.__class__.__name__ - def lock_pair(self, pair: str, until: datetime) -> None: + def lock_pair(self, pair: str, until: datetime, reason: str = None) -> None: """ Locks pair until a given timestamp happens. Locked pairs are not analyzed, and are prevented from opening new trades. @@ -284,9 +286,9 @@ class IStrategy(ABC): :param pair: Pair to lock :param until: datetime in UTC until the pair should be blocked from opening new trades. Needs to be timezone aware `datetime.now(timezone.utc)` + :param reason: Optional string explaining why the pair was locked. """ - if pair not in self._pair_locked_until or self._pair_locked_until[pair] < until: - self._pair_locked_until[pair] = until + PairLock.lock_pair(pair, until, reason) def unlock_pair(self, pair: str) -> None: """ @@ -295,8 +297,7 @@ class IStrategy(ABC): manually from within the strategy, to allow an easy way to unlock pairs. :param pair: Unlock pair to allow trading again """ - if pair in self._pair_locked_until: - del self._pair_locked_until[pair] + PairLock.unlock_pair(pair, datetime.now(timezone.utc)) def is_pair_locked(self, pair: str, candle_date: datetime = None) -> bool: """ @@ -308,15 +309,13 @@ class IStrategy(ABC): :param candle_date: Date of the last candle. Optional, defaults to current date :returns: locking state of the pair in question. """ - if pair not in self._pair_locked_until: - return False + if not candle_date: - return self._pair_locked_until[pair] >= datetime.now(timezone.utc) + # Simple call ... + return PairLock.is_pair_locked(pair, candle_date) else: - # Locking should happen until a new candle arrives lock_time = timeframe_to_next_date(self.timeframe, candle_date) - # lock_time = candle_date + timedelta(minutes=timeframe_to_minutes(self.timeframe)) - return self._pair_locked_until[pair] > lock_time + return PairLock.is_pair_locked(pair, lock_time) def analyze_ticker(self, dataframe: DataFrame, metadata: dict) -> DataFrame: """ diff --git a/freqtrade/strategy/strategy_helper.py b/freqtrade/strategy/strategy_helper.py index 1a5b2d0f8..ea0e234ec 100644 --- a/freqtrade/strategy/strategy_helper.py +++ b/freqtrade/strategy/strategy_helper.py @@ -1,4 +1,5 @@ import pandas as pd + from freqtrade.exchange import timeframe_to_minutes diff --git a/freqtrade/strategy/strategy_wrapper.py b/freqtrade/strategy/strategy_wrapper.py index 8fc548074..121189b68 100644 --- a/freqtrade/strategy/strategy_wrapper.py +++ b/freqtrade/strategy/strategy_wrapper.py @@ -2,6 +2,7 @@ import logging from freqtrade.exceptions import StrategyError + logger = logging.getLogger(__name__) diff --git a/freqtrade/templates/sample_hyperopt.py b/freqtrade/templates/sample_hyperopt.py index 0b6d030db..10743e911 100644 --- a/freqtrade/templates/sample_hyperopt.py +++ b/freqtrade/templates/sample_hyperopt.py @@ -1,4 +1,5 @@ # pragma pylint: disable=missing-docstring, invalid-name, pointless-string-statement +# isort: skip_file # --- Do not remove these libs --- from functools import reduce diff --git a/freqtrade/templates/sample_hyperopt_advanced.py b/freqtrade/templates/sample_hyperopt_advanced.py index 7f05c4430..52e397466 100644 --- a/freqtrade/templates/sample_hyperopt_advanced.py +++ b/freqtrade/templates/sample_hyperopt_advanced.py @@ -1,5 +1,5 @@ # pragma pylint: disable=missing-docstring, invalid-name, pointless-string-statement - +# isort: skip_file # --- Do not remove these libs --- from functools import reduce from typing import Any, Callable, Dict, List diff --git a/freqtrade/templates/sample_hyperopt_loss.py b/freqtrade/templates/sample_hyperopt_loss.py index 4173d97f5..59e6d814a 100644 --- a/freqtrade/templates/sample_hyperopt_loss.py +++ b/freqtrade/templates/sample_hyperopt_loss.py @@ -1,10 +1,11 @@ -from math import exp from datetime import datetime +from math import exp from pandas import DataFrame from freqtrade.optimize.hyperopt import IHyperOptLoss + # Define some constants: # set TARGET_TRADES to suit your number concurrent trades so its realistic diff --git a/freqtrade/templates/sample_strategy.py b/freqtrade/templates/sample_strategy.py index e269848d2..103f68a43 100644 --- a/freqtrade/templates/sample_strategy.py +++ b/freqtrade/templates/sample_strategy.py @@ -1,5 +1,5 @@ # pragma pylint: disable=missing-docstring, invalid-name, pointless-string-statement - +# isort: skip_file # --- Do not remove these libs --- import numpy as np # noqa import pandas as pd # noqa diff --git a/freqtrade/vendor/qtpylib/indicators.py b/freqtrade/vendor/qtpylib/indicators.py index e5a404862..4c0fb5b5c 100644 --- a/freqtrade/vendor/qtpylib/indicators.py +++ b/freqtrade/vendor/qtpylib/indicators.py @@ -19,14 +19,15 @@ # limitations under the License. # -import warnings import sys +import warnings from datetime import datetime, timedelta import numpy as np import pandas as pd from pandas.core.base import PandasObject + # ============================================= # check min, python version if sys.version_info < (3, 4): diff --git a/freqtrade/wallets.py b/freqtrade/wallets.py index b913155bc..21a9466e1 100644 --- a/freqtrade/wallets.py +++ b/freqtrade/wallets.py @@ -2,6 +2,7 @@ """ Wallet """ import logging +from copy import deepcopy from typing import Any, Dict, NamedTuple import arrow @@ -9,6 +10,7 @@ import arrow from freqtrade.exchange import Exchange from freqtrade.persistence import Trade + logger = logging.getLogger(__name__) @@ -93,6 +95,10 @@ class Wallets: balances[currency].get('used', None), balances[currency].get('total', None) ) + # Remove currencies no longer in get_balances output + for currency in deepcopy(self._wallets): + if currency not in balances: + del self._wallets[currency] def update(self, require_update: bool = True) -> None: """ diff --git a/freqtrade/worker.py b/freqtrade/worker.py index 2fc206bd5..ec9331eef 100755 --- a/freqtrade/worker.py +++ b/freqtrade/worker.py @@ -15,6 +15,7 @@ from freqtrade.exceptions import OperationalException, TemporaryError from freqtrade.freqtradebot import FreqtradeBot from freqtrade.state import State + logger = logging.getLogger(__name__) diff --git a/mkdocs.yml b/mkdocs.yml index 26494ae45..8d1ce1cfe 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -55,16 +55,16 @@ markdown_extensions: permalink: true - pymdownx.arithmatex: generic: true - - pymdownx.caret - - pymdownx.critic - pymdownx.details - pymdownx.inlinehilite - pymdownx.magiclink - - pymdownx.mark + - pymdownx.pathconverter - pymdownx.smartsymbols + - pymdownx.snippets: + base_path: docs + check_paths: true - pymdownx.tabbed - pymdownx.superfences - pymdownx.tasklist: custom_checkbox: true - - pymdownx.tilde - mdx_truly_sane_lists diff --git a/requirements-dev.txt b/requirements-dev.txt index c14a146fa..916bb2ec2 100644 --- a/requirements-dev.txt +++ b/requirements-dev.txt @@ -4,15 +4,16 @@ -r requirements-hyperopt.txt coveralls==2.1.2 -flake8==3.8.3 +flake8==3.8.4 flake8-type-annotations==0.1.0 flake8-tidy-imports==4.1.0 -mypy==0.782 -pytest==6.0.2 +mypy==0.790 +pytest==6.1.1 pytest-asyncio==0.14.0 pytest-cov==2.10.1 pytest-mock==3.3.1 pytest-random-order==1.0.4 +isort==5.6.4 # Convert jupyter notebooks to markdown documents -nbconvert==6.0.2 +nbconvert==6.0.7 diff --git a/requirements-hyperopt.txt b/requirements-hyperopt.txt index b47331aa3..5b68c1ea1 100644 --- a/requirements-hyperopt.txt +++ b/requirements-hyperopt.txt @@ -2,9 +2,9 @@ -r requirements.txt # Required for hyperopt -scipy==1.5.2 +scipy==1.5.3 scikit-learn==0.23.2 scikit-optimize==0.8.1 filelock==3.0.12 -joblib==0.16.0 +joblib==0.17.0 progressbar2==3.53.1 diff --git a/requirements-plot.txt b/requirements-plot.txt index a91b3bd38..7c3e04723 100644 --- a/requirements-plot.txt +++ b/requirements-plot.txt @@ -1,5 +1,5 @@ # Include all requirements to run the bot. -r requirements.txt -plotly==4.10.0 +plotly==4.11.0 diff --git a/requirements.txt b/requirements.txt index 5da544a3c..76e92eb3f 100644 --- a/requirements.txt +++ b/requirements.txt @@ -1,18 +1,20 @@ numpy==1.19.2 -pandas==1.1.2 +pandas==1.1.3 -ccxt==1.34.25 -SQLAlchemy==1.3.19 -python-telegram-bot==12.8 -arrow==0.16.0 +ccxt==1.36.66 +multidict==4.7.6 +aiohttp==3.6.3 +SQLAlchemy==1.3.20 +python-telegram-bot==13.0 +arrow==0.17.0 cachetools==4.1.1 requests==2.24.0 urllib3==1.25.10 wrapt==1.12.1 jsonschema==3.2.0 -TA-Lib==0.4.18 +TA-Lib==0.4.19 tabulate==0.8.7 -pycoingecko==1.3.0 +pycoingecko==1.4.0 jinja2==2.11.2 tables==3.6.1 blosc==1.9.2 @@ -32,7 +34,7 @@ flask-jwt-extended==3.24.1 flask-cors==3.0.9 # Support for colorized terminal output -colorama==0.4.3 +colorama==0.4.4 # Building config files interactively -questionary==1.5.2 -prompt-toolkit==3.0.7 +questionary==1.7.0 +prompt-toolkit==3.0.8 diff --git a/scripts/rest_client.py b/scripts/rest_client.py index 95685fd64..268e81397 100755 --- a/scripts/rest_client.py +++ b/scripts/rest_client.py @@ -10,8 +10,8 @@ so it can be used as a standalone script. import argparse import inspect import json -import re import logging +import re import sys from pathlib import Path from urllib.parse import urlencode, urlparse, urlunparse @@ -20,6 +20,7 @@ import rapidjson import requests from requests.exceptions import ConnectionError + logging.basicConfig( level=logging.INFO, format='%(asctime)s - %(name)s - %(levelname)s - %(message)s', @@ -110,6 +111,13 @@ class FtRestClient(): """ return self._get("count") + def locks(self): + """Return current locks + + :return: json object + """ + return self._get("locks") + def daily(self, days=None): """Return the amount of open trades. @@ -223,6 +231,70 @@ class FtRestClient(): return self._post("forcesell", data={"tradeid": tradeid}) + def strategies(self): + """Lists available strategies + + :return: json object + """ + return self._get("strategies") + + def strategy(self, strategy): + """Get strategy details + + :param strategy: Strategy class name + :return: json object + """ + return self._get(f"strategy/{strategy}") + + def plot_config(self): + """Return plot configuration if the strategy defines one. + + :return: json object + """ + return self._get("plot_config") + + def available_pairs(self, timeframe=None, stake_currency=None): + """Return available pair (backtest data) based on timeframe / stake_currency selection + + :param timeframe: Only pairs with this timeframe available. + :param stake_currency: Only pairs that include this timeframe + :return: json object + """ + return self._get("available_pairs", params={ + "stake_currency": stake_currency if timeframe else '', + "timeframe": timeframe if timeframe else '', + }) + + def pair_candles(self, pair, timeframe, limit=None): + """Return live dataframe for . + + :param pair: Pair to get data for + :param timeframe: Only pairs with this timeframe available. + :param limit: Limit result to the last n candles. + :return: json object + """ + return self._get("available_pairs", params={ + "pair": pair, + "timeframe": timeframe, + "limit": limit, + }) + + def pair_history(self, pair, timeframe, strategy, timerange=None): + """Return historic, analyzed dataframe + + :param pair: Pair to get data for + :param timeframe: Only pairs with this timeframe available. + :param strategy: Strategy to analyze and get values for + :param timerange: Timerange to get data for (same format than --timerange endpoints) + :return: json object + """ + return self._get("pair_history", params={ + "pair": pair, + "timeframe": timeframe, + "strategy": strategy, + "timerange": timerange if timerange else '', + }) + def add_arguments(): parser = argparse.ArgumentParser() diff --git a/setup.cfg b/setup.cfg index 34f25482b..be2cd450c 100644 --- a/setup.cfg +++ b/setup.cfg @@ -8,6 +8,11 @@ exclude = .eggs, user_data, +[isort] +line_length=100 +multi_line_output=0 +lines_after_imports=2 + [mypy] ignore_missing_imports = True diff --git a/setup.py b/setup.py index 88d754668..9b57e8d2c 100644 --- a/setup.py +++ b/setup.py @@ -1,12 +1,15 @@ from sys import version_info + from setuptools import setup + if version_info.major == 3 and version_info.minor < 6 or \ version_info.major < 3: print('Your Python interpreter must be 3.6 or greater!') exit(1) from pathlib import Path # noqa: E402 + from freqtrade import __version__ # noqa: E402 @@ -108,6 +111,7 @@ setup(name='freqtrade', 'License :: OSI Approved :: GNU General Public License v3 (GPLv3)', 'Programming Language :: Python :: 3.6', 'Programming Language :: Python :: 3.7', + 'Programming Language :: Python :: 3.8', 'Operating System :: MacOS', 'Operating System :: Unix', 'Topic :: Office/Business :: Financial :: Investment', diff --git a/setup.sh b/setup.sh index 918c41e6b..049a6a77e 100755 --- a/setup.sh +++ b/setup.sh @@ -120,13 +120,13 @@ function update() { updateenv } -# Reset Develop or Master branch +# Reset Develop or Stable branch function reset() { echo "----------------------------" echo "Reseting branch and virtual env" echo "----------------------------" - if [ "1" == $(git branch -vv |grep -cE "\* develop|\* master") ] + if [ "1" == $(git branch -vv |grep -cE "\* develop|\* stable") ] then read -p "Reset git branch? (This will remove all changes you made!) [y/N]? " @@ -138,14 +138,14 @@ function reset() { then echo "- Hard resetting of 'develop' branch." git reset --hard origin/develop - elif [ "1" == $(git branch -vv |grep -c "* master") ] + elif [ "1" == $(git branch -vv |grep -c "* stable") ] then - echo "- Hard resetting of 'master' branch." - git reset --hard origin/master + echo "- Hard resetting of 'stable' branch." + git reset --hard origin/stable fi fi else - echo "Reset ignored because you are not on 'master' or 'develop'." + echo "Reset ignored because you are not on 'stable' or 'develop'." fi if [ -d ".env" ]; then @@ -270,7 +270,7 @@ function help() { echo "usage:" echo " -i,--install Install freqtrade from scratch" echo " -u,--update Command git pull to update." - echo " -r,--reset Hard reset your develop/master branch." + echo " -r,--reset Hard reset your develop/stable branch." echo " -c,--config Easy config generator (Will override your existing file)." echo " -p,--plot Install dependencies for Plotting scripts." } diff --git a/tests/commands/test_build_config.py b/tests/commands/test_build_config.py index 69b277e3b..291720f4b 100644 --- a/tests/commands/test_build_config.py +++ b/tests/commands/test_build_config.py @@ -4,10 +4,8 @@ from unittest.mock import MagicMock import pytest import rapidjson -from freqtrade.commands.build_config_commands import (ask_user_config, - ask_user_overwrite, - start_new_config, - validate_is_float, +from freqtrade.commands.build_config_commands import (ask_user_config, ask_user_overwrite, + start_new_config, validate_is_float, validate_is_int) from freqtrade.exceptions import OperationalException from tests.conftest import get_args, log_has_re diff --git a/tests/commands/test_commands.py b/tests/commands/test_commands.py index 26875fb7f..713386a8e 100644 --- a/tests/commands/test_commands.py +++ b/tests/commands/test_commands.py @@ -2,22 +2,21 @@ import re from pathlib import Path from unittest.mock import MagicMock, PropertyMock +import arrow import pytest -from freqtrade.commands import (start_convert_data, start_create_userdir, - start_download_data, start_hyperopt_list, - start_hyperopt_show, start_list_data, - start_list_exchanges, start_list_hyperopts, - start_list_markets, start_list_strategies, - start_list_timeframes, start_new_hyperopt, - start_new_strategy, start_show_trades, - start_test_pairlist, start_trading) +from freqtrade.commands import (start_convert_data, start_create_userdir, start_download_data, + start_hyperopt_list, start_hyperopt_show, start_list_data, + start_list_exchanges, start_list_hyperopts, start_list_markets, + start_list_strategies, start_list_timeframes, start_new_hyperopt, + start_new_strategy, start_show_trades, start_test_pairlist, + start_trading) from freqtrade.configuration import setup_utils_configuration from freqtrade.exceptions import OperationalException from freqtrade.state import RunMode -from tests.conftest import (create_mock_trades, get_args, log_has, log_has_re, - patch_exchange, +from tests.conftest import (create_mock_trades, get_args, log_has, log_has_re, patch_exchange, patched_configuration_load_config_file) +from tests.conftest_trades import MOCK_TRADE_COUNT def test_setup_utils_configuration(): @@ -552,6 +551,50 @@ def test_download_data_keyboardInterrupt(mocker, caplog, markets): assert dl_mock.call_count == 1 +def test_download_data_timerange(mocker, caplog, markets): + dl_mock = mocker.patch('freqtrade.commands.data_commands.refresh_backtest_ohlcv_data', + MagicMock(return_value=["ETH/BTC", "XRP/BTC"])) + patch_exchange(mocker) + mocker.patch( + 'freqtrade.exchange.Exchange.markets', PropertyMock(return_value=markets) + ) + args = [ + "download-data", + "--exchange", "binance", + "--pairs", "ETH/BTC", "XRP/BTC", + "--days", "20", + "--timerange", "20200101-" + ] + with pytest.raises(OperationalException, + match=r"--days and --timerange are mutually.*"): + start_download_data(get_args(args)) + assert dl_mock.call_count == 0 + + args = [ + "download-data", + "--exchange", "binance", + "--pairs", "ETH/BTC", "XRP/BTC", + "--days", "20", + ] + start_download_data(get_args(args)) + assert dl_mock.call_count == 1 + # 20days ago + days_ago = arrow.get(arrow.utcnow().shift(days=-20).date()).timestamp + assert dl_mock.call_args_list[0][1]['timerange'].startts == days_ago + + dl_mock.reset_mock() + args = [ + "download-data", + "--exchange", "binance", + "--pairs", "ETH/BTC", "XRP/BTC", + "--timerange", "20200101-" + ] + start_download_data(get_args(args)) + assert dl_mock.call_count == 1 + + assert dl_mock.call_args_list[0][1]['timerange'].startts == arrow.Arrow(2020, 1, 1).timestamp + + def test_download_data_no_markets(mocker, caplog): dl_mock = mocker.patch('freqtrade.commands.data_commands.refresh_backtest_ohlcv_data', MagicMock(return_value=["ETH/BTC", "XRP/BTC"])) @@ -1106,7 +1149,7 @@ def test_start_list_data(testdatadir, capsys): @pytest.mark.usefixtures("init_persistence") def test_show_trades(mocker, fee, capsys, caplog): - mocker.patch("freqtrade.persistence.init") + mocker.patch("freqtrade.persistence.init_db") create_mock_trades(fee) args = [ "show-trades", @@ -1116,7 +1159,7 @@ def test_show_trades(mocker, fee, capsys, caplog): pargs = get_args(args) pargs['config'] = None start_show_trades(pargs) - assert log_has("Printing 4 Trades: ", caplog) + assert log_has(f"Printing {MOCK_TRADE_COUNT} Trades: ", caplog) captured = capsys.readouterr() assert "Trade(id=1" in captured.out assert "Trade(id=2" in captured.out diff --git a/tests/conftest.py b/tests/conftest.py index dbed08ec5..520b53b31 100644 --- a/tests/conftest.py +++ b/tests/conftest.py @@ -13,15 +13,18 @@ import numpy as np import pytest from telegram import Chat, Message, Update -from freqtrade import constants, persistence +from freqtrade import constants from freqtrade.commands import Arguments from freqtrade.data.converter import ohlcv_to_dataframe from freqtrade.edge import Edge, PairInfo from freqtrade.exchange import Exchange from freqtrade.freqtradebot import FreqtradeBot -from freqtrade.persistence import Trade +from freqtrade.persistence import Trade, init_db from freqtrade.resolvers import ExchangeResolver from freqtrade.worker import Worker +from tests.conftest_trades import (mock_trade_1, mock_trade_2, mock_trade_3, mock_trade_4, + mock_trade_5, mock_trade_6) + logging.getLogger('').setLevel(logging.INFO) @@ -128,7 +131,7 @@ def patch_freqtradebot(mocker, config) -> None: :return: None """ mocker.patch('freqtrade.freqtradebot.RPCManager', MagicMock()) - persistence.init(config['db_url']) + init_db(config['db_url']) patch_exchange(mocker) mocker.patch('freqtrade.freqtradebot.RPCManager._init', MagicMock()) mocker.patch('freqtrade.freqtradebot.RPCManager.send_msg', MagicMock()) @@ -143,6 +146,7 @@ def get_patched_freqtradebot(mocker, config) -> FreqtradeBot: :return: FreqtradeBot """ patch_freqtradebot(mocker, config) + config['datadir'] = Path(config['datadir']) return FreqtradeBot(config) @@ -172,64 +176,22 @@ def create_mock_trades(fee): Create some fake trades ... """ # Simulate dry_run entries - trade = Trade( - pair='ETH/BTC', - stake_amount=0.001, - amount=123.0, - amount_requested=123.0, - fee_open=fee.return_value, - fee_close=fee.return_value, - open_rate=0.123, - exchange='bittrex', - open_order_id='dry_run_buy_12345', - strategy='DefaultStrategy', - ) + trade = mock_trade_1(fee) Trade.session.add(trade) - trade = Trade( - pair='ETC/BTC', - stake_amount=0.001, - amount=123.0, - amount_requested=123.0, - fee_open=fee.return_value, - fee_close=fee.return_value, - open_rate=0.123, - close_rate=0.128, - close_profit=0.005, - exchange='bittrex', - is_open=False, - open_order_id='dry_run_sell_12345', - strategy='DefaultStrategy', - ) + trade = mock_trade_2(fee) Trade.session.add(trade) - trade = Trade( - pair='XRP/BTC', - stake_amount=0.001, - amount=123.0, - fee_open=fee.return_value, - fee_close=fee.return_value, - open_rate=0.05, - close_rate=0.06, - close_profit=0.01, - exchange='bittrex', - is_open=False, - ) + trade = mock_trade_3(fee) Trade.session.add(trade) - # Simulate prod entry - trade = Trade( - pair='ETC/BTC', - stake_amount=0.001, - amount=123.0, - amount_requested=124.0, - fee_open=fee.return_value, - fee_close=fee.return_value, - open_rate=0.123, - exchange='bittrex', - open_order_id='prod_buy_12345', - strategy='DefaultStrategy', - ) + trade = mock_trade_4(fee) + Trade.session.add(trade) + + trade = mock_trade_5(fee) + Trade.session.add(trade) + + trade = mock_trade_6(fee) Trade.session.add(trade) @@ -257,7 +219,7 @@ def patch_coingekko(mocker) -> None: @pytest.fixture(scope='function') def init_persistence(default_conf): - persistence.init(default_conf['db_url'], default_conf['dry_run']) + init_db(default_conf['db_url'], default_conf['dry_run']) @pytest.fixture(scope="function") @@ -335,7 +297,7 @@ def default_conf(testdatadir): @pytest.fixture def update(): _update = Update(0) - _update.message = Message(0, 0, datetime.utcnow(), Chat(0, 0)) + _update.message = Message(0, datetime.utcnow(), Chat(0, 0)) return _update @@ -823,22 +785,32 @@ def markets_empty(): @pytest.fixture(scope='function') -def limit_buy_order(): +def limit_buy_order_open(): return { 'id': 'mocked_limit_buy', 'type': 'limit', 'side': 'buy', 'symbol': 'mocked', 'datetime': arrow.utcnow().isoformat(), + 'timestamp': arrow.utcnow().timestamp, 'price': 0.00001099, 'amount': 90.99181073, - 'filled': 90.99181073, + 'filled': 0.0, 'cost': 0.0009999, - 'remaining': 0.0, - 'status': 'closed' + 'remaining': 90.99181073, + 'status': 'open' } +@pytest.fixture(scope='function') +def limit_buy_order(limit_buy_order_open): + order = deepcopy(limit_buy_order_open) + order['status'] = 'closed' + order['filled'] = order['amount'] + order['remaining'] = 0.0 + return order + + @pytest.fixture(scope='function') def market_buy_order(): return { @@ -1021,21 +993,31 @@ def limit_buy_order_canceled_empty(request): @pytest.fixture -def limit_sell_order(): +def limit_sell_order_open(): return { 'id': 'mocked_limit_sell', 'type': 'limit', 'side': 'sell', 'pair': 'mocked', 'datetime': arrow.utcnow().isoformat(), + 'timestamp': arrow.utcnow().timestamp, 'price': 0.00001173, 'amount': 90.99181073, - 'filled': 90.99181073, - 'remaining': 0.0, - 'status': 'closed' + 'filled': 0.0, + 'remaining': 90.99181073, + 'status': 'open' } +@pytest.fixture +def limit_sell_order(limit_sell_order_open): + order = deepcopy(limit_sell_order_open) + order['remaining'] = 0.0 + order['filled'] = order['amount'] + order['status'] = 'closed' + return order + + @pytest.fixture def order_book_l2(): return MagicMock(return_value={ diff --git a/tests/conftest_trades.py b/tests/conftest_trades.py new file mode 100644 index 000000000..78388f022 --- /dev/null +++ b/tests/conftest_trades.py @@ -0,0 +1,279 @@ +from freqtrade.persistence.models import Order, Trade + + +MOCK_TRADE_COUNT = 6 + + +def mock_order_1(): + return { + 'id': '1234', + 'symbol': 'ETH/BTC', + 'status': 'closed', + 'side': 'buy', + 'type': 'limit', + 'price': 0.123, + 'amount': 123.0, + 'filled': 123.0, + 'remaining': 0.0, + } + + +def mock_trade_1(fee): + trade = Trade( + pair='ETH/BTC', + stake_amount=0.001, + amount=123.0, + amount_requested=123.0, + fee_open=fee.return_value, + fee_close=fee.return_value, + open_rate=0.123, + exchange='bittrex', + open_order_id='dry_run_buy_12345', + strategy='DefaultStrategy', + ) + o = Order.parse_from_ccxt_object(mock_order_1(), 'ETH/BTC', 'buy') + trade.orders.append(o) + return trade + + +def mock_order_2(): + return { + 'id': '1235', + 'symbol': 'ETC/BTC', + 'status': 'closed', + 'side': 'buy', + 'type': 'limit', + 'price': 0.123, + 'amount': 123.0, + 'filled': 123.0, + 'remaining': 0.0, + } + + +def mock_order_2_sell(): + return { + 'id': '12366', + 'symbol': 'ETC/BTC', + 'status': 'closed', + 'side': 'sell', + 'type': 'limit', + 'price': 0.128, + 'amount': 123.0, + 'filled': 123.0, + 'remaining': 0.0, + } + + +def mock_trade_2(fee): + """ + Closed trade... + """ + trade = Trade( + pair='ETC/BTC', + stake_amount=0.001, + amount=123.0, + amount_requested=123.0, + fee_open=fee.return_value, + fee_close=fee.return_value, + open_rate=0.123, + close_rate=0.128, + close_profit=0.005, + exchange='bittrex', + is_open=False, + open_order_id='dry_run_sell_12345', + strategy='DefaultStrategy', + ) + o = Order.parse_from_ccxt_object(mock_order_2(), 'ETC/BTC', 'buy') + trade.orders.append(o) + o = Order.parse_from_ccxt_object(mock_order_2_sell(), 'ETC/BTC', 'sell') + trade.orders.append(o) + return trade + + +def mock_order_3(): + return { + 'id': '41231a12a', + 'symbol': 'XRP/BTC', + 'status': 'closed', + 'side': 'buy', + 'type': 'limit', + 'price': 0.05, + 'amount': 123.0, + 'filled': 123.0, + 'remaining': 0.0, + } + + +def mock_order_3_sell(): + return { + 'id': '41231a666a', + 'symbol': 'XRP/BTC', + 'status': 'closed', + 'side': 'sell', + 'type': 'stop_loss_limit', + 'price': 0.06, + 'average': 0.06, + 'amount': 123.0, + 'filled': 123.0, + 'remaining': 0.0, + } + + +def mock_trade_3(fee): + """ + Closed trade + """ + trade = Trade( + pair='XRP/BTC', + stake_amount=0.001, + amount=123.0, + fee_open=fee.return_value, + fee_close=fee.return_value, + open_rate=0.05, + close_rate=0.06, + close_profit=0.01, + exchange='bittrex', + is_open=False, + ) + o = Order.parse_from_ccxt_object(mock_order_3(), 'XRP/BTC', 'buy') + trade.orders.append(o) + o = Order.parse_from_ccxt_object(mock_order_3_sell(), 'XRP/BTC', 'sell') + trade.orders.append(o) + return trade + + +def mock_order_4(): + return { + 'id': 'prod_buy_12345', + 'symbol': 'ETC/BTC', + 'status': 'open', + 'side': 'buy', + 'type': 'limit', + 'price': 0.123, + 'amount': 123.0, + 'filled': 0.0, + 'remaining': 123.0, + } + + +def mock_trade_4(fee): + """ + Simulate prod entry + """ + trade = Trade( + pair='ETC/BTC', + stake_amount=0.001, + amount=123.0, + amount_requested=124.0, + fee_open=fee.return_value, + fee_close=fee.return_value, + open_rate=0.123, + exchange='bittrex', + open_order_id='prod_buy_12345', + strategy='DefaultStrategy', + ) + o = Order.parse_from_ccxt_object(mock_order_4(), 'ETC/BTC', 'buy') + trade.orders.append(o) + return trade + + +def mock_order_5(): + return { + 'id': 'prod_buy_3455', + 'symbol': 'XRP/BTC', + 'status': 'closed', + 'side': 'buy', + 'type': 'limit', + 'price': 0.123, + 'amount': 123.0, + 'filled': 123.0, + 'remaining': 0.0, + } + + +def mock_order_5_stoploss(): + return { + 'id': 'prod_stoploss_3455', + 'symbol': 'XRP/BTC', + 'status': 'open', + 'side': 'sell', + 'type': 'stop_loss_limit', + 'price': 0.123, + 'amount': 123.0, + 'filled': 0.0, + 'remaining': 123.0, + } + + +def mock_trade_5(fee): + """ + Simulate prod entry with stoploss + """ + trade = Trade( + pair='XRP/BTC', + stake_amount=0.001, + amount=123.0, + amount_requested=124.0, + fee_open=fee.return_value, + fee_close=fee.return_value, + open_rate=0.123, + exchange='bittrex', + strategy='SampleStrategy', + stoploss_order_id='prod_stoploss_3455' + ) + o = Order.parse_from_ccxt_object(mock_order_5(), 'XRP/BTC', 'buy') + trade.orders.append(o) + o = Order.parse_from_ccxt_object(mock_order_5_stoploss(), 'XRP/BTC', 'stoploss') + trade.orders.append(o) + return trade + + +def mock_order_6(): + return { + 'id': 'prod_buy_6', + 'symbol': 'LTC/BTC', + 'status': 'closed', + 'side': 'buy', + 'type': 'limit', + 'price': 0.15, + 'amount': 2.0, + 'filled': 2.0, + 'remaining': 0.0, + } + + +def mock_order_6_sell(): + return { + 'id': 'prod_sell_6', + 'symbol': 'LTC/BTC', + 'status': 'open', + 'side': 'sell', + 'type': 'limit', + 'price': 0.20, + 'amount': 2.0, + 'filled': 0.0, + 'remaining': 2.0, + } + + +def mock_trade_6(fee): + """ + Simulate prod entry with open sell order + """ + trade = Trade( + pair='LTC/BTC', + stake_amount=0.001, + amount=2.0, + amount_requested=2.0, + fee_open=fee.return_value, + fee_close=fee.return_value, + open_rate=0.15, + exchange='bittrex', + strategy='SampleStrategy', + open_order_id="prod_sell_6", + ) + o = Order.parse_from_ccxt_object(mock_order_6(), 'LTC/BTC', 'buy') + trade.orders.append(o) + o = Order.parse_from_ccxt_object(mock_order_6_sell(), 'LTC/BTC', 'sell') + trade.orders.append(o) + return trade diff --git a/tests/data/test_btanalysis.py b/tests/data/test_btanalysis.py index e2ca66bd8..1592fac10 100644 --- a/tests/data/test_btanalysis.py +++ b/tests/data/test_btanalysis.py @@ -7,19 +7,16 @@ from pandas import DataFrame, DateOffset, Timestamp, to_datetime from freqtrade.configuration import TimeRange from freqtrade.constants import LAST_BT_RESULT_FN -from freqtrade.data.btanalysis import (BT_DATA_COLUMNS, - analyze_trade_parallelism, - calculate_market_change, - calculate_max_drawdown, - combine_dataframes_with_mean, - create_cum_profit, - extract_trades_of_period, - get_latest_backtest_filename, - load_backtest_data, load_trades, +from freqtrade.data.btanalysis import (BT_DATA_COLUMNS, analyze_trade_parallelism, + calculate_market_change, calculate_max_drawdown, + combine_dataframes_with_mean, create_cum_profit, + extract_trades_of_period, get_latest_backtest_filename, + get_latest_hyperopt_file, load_backtest_data, load_trades, load_trades_from_db) from freqtrade.data.history import load_data, load_pair_history from freqtrade.optimize.backtesting import BacktestResult from tests.conftest import create_mock_trades +from tests.conftest_trades import MOCK_TRADE_COUNT def test_get_latest_backtest_filename(testdatadir, mocker): @@ -42,6 +39,17 @@ def test_get_latest_backtest_filename(testdatadir, mocker): get_latest_backtest_filename(testdatadir) +def test_get_latest_hyperopt_file(testdatadir, mocker): + res = get_latest_hyperopt_file(testdatadir / 'does_not_exist', 'testfile.pickle') + assert res == testdatadir / 'does_not_exist/testfile.pickle' + + res = get_latest_hyperopt_file(testdatadir.parent) + assert res == testdatadir.parent / "hyperopt_results.pickle" + + res = get_latest_hyperopt_file(str(testdatadir.parent)) + assert res == testdatadir.parent / "hyperopt_results.pickle" + + def test_load_backtest_data_old_format(testdatadir): filename = testdatadir / "backtest-result_test.json" @@ -106,11 +114,11 @@ def test_load_trades_from_db(default_conf, fee, mocker): create_mock_trades(fee) # remove init so it does not init again - init_mock = mocker.patch('freqtrade.persistence.init', MagicMock()) + init_mock = mocker.patch('freqtrade.data.btanalysis.init_db', MagicMock()) trades = load_trades_from_db(db_url=default_conf['db_url']) assert init_mock.call_count == 1 - assert len(trades) == 4 + assert len(trades) == MOCK_TRADE_COUNT assert isinstance(trades, DataFrame) assert "pair" in trades.columns assert "open_date" in trades.columns diff --git a/tests/data/test_converter.py b/tests/data/test_converter.py index 4a580366f..fdba7900f 100644 --- a/tests/data/test_converter.py +++ b/tests/data/test_converter.py @@ -2,13 +2,11 @@ import logging from freqtrade.configuration.timerange import TimeRange -from freqtrade.data.converter import (convert_ohlcv_format, - convert_trades_format, - ohlcv_fill_up_missing_data, - ohlcv_to_dataframe, trades_dict_to_list, - trades_remove_duplicates, trim_dataframe) -from freqtrade.data.history import (get_timerange, load_data, - load_pair_history, validate_backtest_data) +from freqtrade.data.converter import (convert_ohlcv_format, convert_trades_format, + ohlcv_fill_up_missing_data, ohlcv_to_dataframe, + trades_dict_to_list, trades_remove_duplicates, trim_dataframe) +from freqtrade.data.history import (get_timerange, load_data, load_pair_history, + validate_backtest_data) from tests.conftest import log_has from tests.data.test_history import _backup_file, _clean_test_file diff --git a/tests/data/test_dataprovider.py b/tests/data/test_dataprovider.py index c2ecf4b80..a64dce908 100644 --- a/tests/data/test_dataprovider.py +++ b/tests/data/test_dataprovider.py @@ -132,7 +132,7 @@ def test_orderbook(mocker, default_conf, order_book_l2): res = dp.orderbook('ETH/BTC', 5) assert order_book_l2.call_count == 1 assert order_book_l2.call_args_list[0][0][0] == 'ETH/BTC' - assert order_book_l2.call_args_list[0][0][1] == 5 + assert order_book_l2.call_args_list[0][0][1] >= 5 assert type(res) is dict assert 'bids' in res diff --git a/tests/data/test_history.py b/tests/data/test_history.py index 787f62a75..c8324cf0b 100644 --- a/tests/data/test_history.py +++ b/tests/data/test_history.py @@ -15,20 +15,19 @@ from freqtrade.configuration import TimeRange from freqtrade.constants import AVAILABLE_DATAHANDLERS from freqtrade.data.converter import ohlcv_to_dataframe from freqtrade.data.history.hdf5datahandler import HDF5DataHandler -from freqtrade.data.history.history_utils import ( - _download_pair_history, _download_trades_history, - _load_cached_data_for_updating, convert_trades_to_ohlcv, get_timerange, - load_data, load_pair_history, refresh_backtest_ohlcv_data, - refresh_backtest_trades_data, refresh_data, validate_backtest_data) -from freqtrade.data.history.idatahandler import (IDataHandler, get_datahandler, - get_datahandlerclass) -from freqtrade.data.history.jsondatahandler import (JsonDataHandler, - JsonGzDataHandler) +from freqtrade.data.history.history_utils import (_download_pair_history, _download_trades_history, + _load_cached_data_for_updating, + convert_trades_to_ohlcv, get_timerange, load_data, + load_pair_history, refresh_backtest_ohlcv_data, + refresh_backtest_trades_data, refresh_data, + validate_backtest_data) +from freqtrade.data.history.idatahandler import IDataHandler, get_datahandler, get_datahandlerclass +from freqtrade.data.history.jsondatahandler import JsonDataHandler, JsonGzDataHandler from freqtrade.exchange import timeframe_to_minutes from freqtrade.misc import file_dump_json from freqtrade.resolvers import StrategyResolver -from tests.conftest import (get_patched_exchange, log_has, log_has_re, - patch_exchange) +from tests.conftest import get_patched_exchange, log_has, log_has_re, patch_exchange + # Change this if modifying UNITTEST/BTC testdatafile _BTC_UNITTEST_LENGTH = 13681 diff --git a/tests/edge/test_edge.py b/tests/edge/test_edge.py index d35f7fcf6..a4bfa1085 100644 --- a/tests/edge/test_edge.py +++ b/tests/edge/test_edge.py @@ -10,14 +10,15 @@ import numpy as np import pytest from pandas import DataFrame, to_datetime -from freqtrade.exceptions import OperationalException from freqtrade.data.converter import ohlcv_to_dataframe from freqtrade.edge import Edge, PairInfo +from freqtrade.exceptions import OperationalException from freqtrade.strategy.interface import SellType from tests.conftest import get_patched_freqtradebot, log_has from tests.optimize import (BTContainer, BTrade, _build_backtest_dataframe, _get_frame_time_from_offset) + # Cases to be tested: # 1) Open trade should be removed from the end # 2) Two complete trades within dataframe (with sell hit for all) @@ -498,3 +499,61 @@ def test_process_expectancy_remove_pumps(mocker, edge_conf, fee,): assert final['TEST/BTC'].stoploss == -0.9 assert final['TEST/BTC'].nb_trades == len(trades_df) - 1 assert round(final['TEST/BTC'].winrate, 10) == 0.0 + + +def test_process_expectancy_only_wins(mocker, edge_conf, fee,): + edge_conf['edge']['min_trade_number'] = 2 + freqtrade = get_patched_freqtradebot(mocker, edge_conf) + + freqtrade.exchange.get_fee = fee + edge = Edge(edge_conf, freqtrade.exchange, freqtrade.strategy) + + trades = [ + {'pair': 'TEST/BTC', + 'stoploss': -0.9, + 'profit_percent': '', + 'profit_abs': '', + 'open_date': np.datetime64('2018-10-03T00:05:00.000000000'), + 'close_date': np.datetime64('2018-10-03T00:10:00.000000000'), + 'open_index': 1, + 'close_index': 1, + 'trade_duration': '', + 'open_rate': 15, + 'close_rate': 17, + 'exit_type': 'sell_signal'}, + {'pair': 'TEST/BTC', + 'stoploss': -0.9, + 'profit_percent': '', + 'profit_abs': '', + 'open_date': np.datetime64('2018-10-03T00:20:00.000000000'), + 'close_date': np.datetime64('2018-10-03T00:25:00.000000000'), + 'open_index': 4, + 'close_index': 4, + 'trade_duration': '', + 'open_rate': 10, + 'close_rate': 20, + 'exit_type': 'sell_signal'}, + {'pair': 'TEST/BTC', + 'stoploss': -0.9, + 'profit_percent': '', + 'profit_abs': '', + 'open_date': np.datetime64('2018-10-03T00:30:00.000000000'), + 'close_date': np.datetime64('2018-10-03T00:40:00.000000000'), + 'open_index': 6, + 'close_index': 7, + 'trade_duration': '', + 'open_rate': 26, + 'close_rate': 134, + 'exit_type': 'sell_signal'} + ] + + trades_df = DataFrame(trades) + trades_df = edge._fill_calculable_fields(trades_df) + final = edge._process_expectancy(trades_df) + + assert 'TEST/BTC' in final + assert final['TEST/BTC'].stoploss == -0.9 + assert final['TEST/BTC'].nb_trades == len(trades_df) + assert round(final['TEST/BTC'].winrate, 10) == 1.0 + assert round(final['TEST/BTC'].risk_reward_ratio, 10) == float('inf') + assert round(final['TEST/BTC'].expectancy, 10) == float('inf') diff --git a/tests/exchange/test_binance.py b/tests/exchange/test_binance.py index 72da708b4..f2b508761 100644 --- a/tests/exchange/test_binance.py +++ b/tests/exchange/test_binance.py @@ -4,8 +4,7 @@ from unittest.mock import MagicMock import ccxt import pytest -from freqtrade.exceptions import (DependencyException, InvalidOrderException, - OperationalException) +from freqtrade.exceptions import DependencyException, InvalidOrderException, OperationalException from tests.conftest import get_patched_exchange from tests.exchange.test_exchange import ccxt_exceptionhandlers diff --git a/tests/exchange/test_exchange.py b/tests/exchange/test_exchange.py index c254d6a09..19f2c7239 100644 --- a/tests/exchange/test_exchange.py +++ b/tests/exchange/test_exchange.py @@ -1,5 +1,3 @@ -# pragma pylint: disable=missing-docstring, C0103, bad-continuation, global-statement -# pragma pylint: disable=protected-access import copy import logging from datetime import datetime, timezone @@ -11,20 +9,18 @@ import ccxt import pytest from pandas import DataFrame -from freqtrade.exceptions import (DDosProtection, DependencyException, - InvalidOrderException, OperationalException, - TemporaryError) -from freqtrade.exchange import Binance, Exchange, Kraken -from freqtrade.exchange.common import API_RETRY_COUNT, calculate_backoff -from freqtrade.exchange.exchange import (market_is_active, - timeframe_to_minutes, - timeframe_to_msecs, - timeframe_to_next_date, - timeframe_to_prev_date, +from freqtrade.exceptions import (DDosProtection, DependencyException, InvalidOrderException, + OperationalException, TemporaryError) +from freqtrade.exchange import Binance, Bittrex, Exchange, Kraken +from freqtrade.exchange.common import (API_FETCH_ORDER_RETRY_COUNT, API_RETRY_COUNT, + calculate_backoff) +from freqtrade.exchange.exchange import (market_is_active, timeframe_to_minutes, timeframe_to_msecs, + timeframe_to_next_date, timeframe_to_prev_date, timeframe_to_seconds) from freqtrade.resolvers.exchange_resolver import ExchangeResolver from tests.conftest import get_patched_exchange, log_has, log_has_re + # Make sure to always keep one exchange here which is NOT subclassed!! EXCHANGES = ['bittrex', 'binance', 'kraken', 'ftx'] @@ -152,11 +148,19 @@ def test_exchange_resolver(default_conf, mocker, caplog): mocker.patch('freqtrade.exchange.Exchange.validate_pairs') mocker.patch('freqtrade.exchange.Exchange.validate_timeframes') mocker.patch('freqtrade.exchange.Exchange.validate_stakecurrency') - exchange = ExchangeResolver.load_exchange('Bittrex', default_conf) + + exchange = ExchangeResolver.load_exchange('huobi', default_conf) assert isinstance(exchange, Exchange) assert log_has_re(r"No .* specific subclass found. Using the generic class instead.", caplog) caplog.clear() + exchange = ExchangeResolver.load_exchange('Bittrex', default_conf) + assert isinstance(exchange, Exchange) + assert isinstance(exchange, Bittrex) + assert not log_has_re(r"No .* specific subclass found. Using the generic class instead.", + caplog) + caplog.clear() + exchange = ExchangeResolver.load_exchange('kraken', default_conf) assert isinstance(exchange, Exchange) assert isinstance(exchange, Kraken) @@ -808,7 +812,7 @@ def test_dry_run_order(default_conf, mocker, side, exchange_name): assert f'dry_run_{side}_' in order["id"] assert order["side"] == side assert order["type"] == "limit" - assert order["pair"] == "ETH/BTC" + assert order["symbol"] == "ETH/BTC" @pytest.mark.parametrize("side", [ @@ -1442,6 +1446,27 @@ def test_refresh_latest_ohlcv_inv_result(default_conf, mocker, caplog): assert log_has("Async code raised an exception: TypeError", caplog) +def test_get_next_limit_in_list(): + limit_range = [5, 10, 20, 50, 100, 500, 1000] + assert Exchange.get_next_limit_in_list(1, limit_range) == 5 + assert Exchange.get_next_limit_in_list(5, limit_range) == 5 + assert Exchange.get_next_limit_in_list(6, limit_range) == 10 + assert Exchange.get_next_limit_in_list(9, limit_range) == 10 + assert Exchange.get_next_limit_in_list(10, limit_range) == 10 + assert Exchange.get_next_limit_in_list(11, limit_range) == 20 + assert Exchange.get_next_limit_in_list(19, limit_range) == 20 + assert Exchange.get_next_limit_in_list(21, limit_range) == 50 + assert Exchange.get_next_limit_in_list(51, limit_range) == 100 + assert Exchange.get_next_limit_in_list(1000, limit_range) == 1000 + # Going over the limit ... + assert Exchange.get_next_limit_in_list(1001, limit_range) == 1000 + assert Exchange.get_next_limit_in_list(2000, limit_range) == 1000 + + assert Exchange.get_next_limit_in_list(21, None) == 21 + assert Exchange.get_next_limit_in_list(100, None) == 100 + assert Exchange.get_next_limit_in_list(1000, None) == 1000 + + @pytest.mark.parametrize("exchange_name", EXCHANGES) def test_fetch_l2_order_book(default_conf, mocker, order_book_l2, exchange_name): default_conf['exchange']['name'] = exchange_name @@ -1454,6 +1479,19 @@ def test_fetch_l2_order_book(default_conf, mocker, order_book_l2, exchange_name) assert 'asks' in order_book assert len(order_book['bids']) == 10 assert len(order_book['asks']) == 10 + assert api_mock.fetch_l2_order_book.call_args_list[0][0][0] == 'ETH/BTC' + + for val in [1, 5, 10, 12, 20, 50, 100]: + api_mock.fetch_l2_order_book.reset_mock() + + order_book = exchange.fetch_l2_order_book(pair='ETH/BTC', limit=val) + assert api_mock.fetch_l2_order_book.call_args_list[0][0][0] == 'ETH/BTC' + # Not all exchanges support all limits for orderbook + if not exchange._ft_has['l2_limit_range'] or val in exchange._ft_has['l2_limit_range']: + assert api_mock.fetch_l2_order_book.call_args_list[0][0][1] == val + else: + next_limit = exchange.get_next_limit_in_list(val, exchange._ft_has['l2_limit_range']) + assert api_mock.fetch_l2_order_book.call_args_list[0][0][1] == next_limit @pytest.mark.parametrize("exchange_name", EXCHANGES) @@ -1766,7 +1804,7 @@ def test_cancel_order_dry_run(default_conf, mocker, exchange_name): cancel_order = exchange.cancel_order(order_id=order['id'], pair='ETH/BTC') assert order['id'] == cancel_order['id'] assert order['amount'] == cancel_order['amount'] - assert order['pair'] == cancel_order['pair'] + assert order['symbol'] == cancel_order['symbol'] assert cancel_order['status'] == 'canceled' @@ -1903,12 +1941,14 @@ def test_fetch_order(default_conf, mocker, exchange_name): # Ensure backoff is called assert tm.call_args_list[0][0][0] == 1 assert tm.call_args_list[1][0][0] == 2 - assert tm.call_args_list[2][0][0] == 5 - assert tm.call_args_list[3][0][0] == 10 - assert api_mock.fetch_order.call_count == 6 + if API_FETCH_ORDER_RETRY_COUNT > 2: + assert tm.call_args_list[2][0][0] == 5 + if API_FETCH_ORDER_RETRY_COUNT > 3: + assert tm.call_args_list[3][0][0] == 10 + assert api_mock.fetch_order.call_count == API_FETCH_ORDER_RETRY_COUNT + 1 ccxt_exceptionhandlers(mocker, default_conf, api_mock, exchange_name, - 'fetch_order', 'fetch_order', retries=6, + 'fetch_order', 'fetch_order', retries=API_FETCH_ORDER_RETRY_COUNT + 1, order_id='_', pair='TKN/BTC') @@ -1941,10 +1981,35 @@ def test_fetch_stoploss_order(default_conf, mocker, exchange_name): ccxt_exceptionhandlers(mocker, default_conf, api_mock, exchange_name, 'fetch_stoploss_order', 'fetch_order', - retries=6, + retries=API_FETCH_ORDER_RETRY_COUNT + 1, order_id='_', pair='TKN/BTC') +def test_fetch_order_or_stoploss_order(default_conf, mocker): + exchange = get_patched_exchange(mocker, default_conf, id='binance') + fetch_order_mock = MagicMock() + fetch_stoploss_order_mock = MagicMock() + mocker.patch.multiple('freqtrade.exchange.Exchange', + fetch_order=fetch_order_mock, + fetch_stoploss_order=fetch_stoploss_order_mock, + ) + + exchange.fetch_order_or_stoploss_order('1234', 'ETH/BTC', False) + assert fetch_order_mock.call_count == 1 + assert fetch_order_mock.call_args_list[0][0][0] == '1234' + assert fetch_order_mock.call_args_list[0][0][1] == 'ETH/BTC' + assert fetch_stoploss_order_mock.call_count == 0 + + fetch_order_mock.reset_mock() + fetch_stoploss_order_mock.reset_mock() + + exchange.fetch_order_or_stoploss_order('1234', 'ETH/BTC', True) + assert fetch_order_mock.call_count == 0 + assert fetch_stoploss_order_mock.call_count == 1 + assert fetch_stoploss_order_mock.call_args_list[0][0][0] == '1234' + assert fetch_stoploss_order_mock.call_args_list[0][0][1] == 'ETH/BTC' + + @pytest.mark.parametrize("exchange_name", EXCHANGES) def test_name(default_conf, mocker, exchange_name): exchange = get_patched_exchange(mocker, default_conf, id=exchange_name) diff --git a/tests/exchange/test_ftx.py b/tests/exchange/test_ftx.py index bed92d276..17cfb26fa 100644 --- a/tests/exchange/test_ftx.py +++ b/tests/exchange/test_ftx.py @@ -1,5 +1,3 @@ -# pragma pylint: disable=missing-docstring, C0103, bad-continuation, global-statement -# pragma pylint: disable=protected-access from random import randint from unittest.mock import MagicMock @@ -7,10 +5,12 @@ import ccxt import pytest from freqtrade.exceptions import DependencyException, InvalidOrderException +from freqtrade.exchange.common import API_FETCH_ORDER_RETRY_COUNT from tests.conftest import get_patched_exchange from .test_exchange import ccxt_exceptionhandlers + STOPLOSS_ORDERTYPE = 'stop' @@ -154,5 +154,5 @@ def test_fetch_stoploss_order(default_conf, mocker): ccxt_exceptionhandlers(mocker, default_conf, api_mock, 'ftx', 'fetch_stoploss_order', 'fetch_orders', - retries=6, + retries=API_FETCH_ORDER_RETRY_COUNT + 1, order_id='_', pair='TKN/BTC') diff --git a/tests/exchange/test_kraken.py b/tests/exchange/test_kraken.py index 9451c0b9e..31b79a202 100644 --- a/tests/exchange/test_kraken.py +++ b/tests/exchange/test_kraken.py @@ -1,5 +1,3 @@ -# pragma pylint: disable=missing-docstring, C0103, bad-continuation, global-statement -# pragma pylint: disable=protected-access from random import randint from unittest.mock import MagicMock @@ -10,6 +8,7 @@ from freqtrade.exceptions import DependencyException, InvalidOrderException from tests.conftest import get_patched_exchange from tests.exchange.test_exchange import ccxt_exceptionhandlers + STOPLOSS_ORDERTYPE = 'stop-loss' diff --git a/tests/optimize/__init__.py b/tests/optimize/__init__.py index 8bc66f02c..306850ff6 100644 --- a/tests/optimize/__init__.py +++ b/tests/optimize/__init__.py @@ -6,6 +6,7 @@ from pandas import DataFrame from freqtrade.exchange import timeframe_to_minutes from freqtrade.strategy.interface import SellType + tests_start_time = arrow.get(2018, 10, 3) tests_timeframe = '1h' diff --git a/tests/optimize/test_backtest_detail.py b/tests/optimize/test_backtest_detail.py index f6ac95aeb..a5de64fe4 100644 --- a/tests/optimize/test_backtest_detail.py +++ b/tests/optimize/test_backtest_detail.py @@ -11,6 +11,7 @@ from tests.conftest import patch_exchange from tests.optimize import (BTContainer, BTrade, _build_backtest_dataframe, _get_frame_time_from_offset, tests_timeframe) + # Test 0: Sell with signal sell in candle 3 # Test with Stop-loss at 1% tc0 = BTContainer(data=[ diff --git a/tests/optimize/test_backtesting.py b/tests/optimize/test_backtesting.py index f5c313520..45cbea68e 100644 --- a/tests/optimize/test_backtesting.py +++ b/tests/optimize/test_backtesting.py @@ -10,11 +10,10 @@ import pytest from arrow import Arrow from freqtrade import constants -from freqtrade.commands.optimize_commands import (setup_optimize_configuration, - start_backtesting) +from freqtrade.commands.optimize_commands import setup_optimize_configuration, start_backtesting from freqtrade.configuration import TimeRange from freqtrade.data import history -from freqtrade.data.btanalysis import evaluate_result_multi +from freqtrade.data.btanalysis import BT_DATA_COLUMNS, evaluate_result_multi from freqtrade.data.converter import clean_ohlcv_dataframe from freqtrade.data.dataprovider import DataProvider from freqtrade.data.history import get_timerange @@ -26,6 +25,7 @@ from freqtrade.strategy.interface import SellType from tests.conftest import (get_args, log_has, log_has_re, patch_exchange, patched_configuration_load_config_file) + ORDER_TYPES = [ { 'buy': 'limit', @@ -694,7 +694,7 @@ def test_backtest_start_timerange(default_conf, mocker, caplog, testdatadir): def test_backtest_start_multi_strat(default_conf, mocker, caplog, testdatadir): patch_exchange(mocker) - backtestmock = MagicMock() + backtestmock = MagicMock(return_value=pd.DataFrame(columns=BT_DATA_COLUMNS + ['profit_abs'])) mocker.patch('freqtrade.pairlist.pairlistmanager.PairListManager.whitelist', PropertyMock(return_value=['UNITTEST/BTC'])) mocker.patch('freqtrade.optimize.backtesting.Backtesting.backtest', backtestmock) diff --git a/tests/optimize/test_hyperopt.py b/tests/optimize/test_hyperopt.py index bb6f043e7..41ad6f5de 100644 --- a/tests/optimize/test_hyperopt.py +++ b/tests/optimize/test_hyperopt.py @@ -1,11 +1,12 @@ # pragma pylint: disable=missing-docstring,W0212,C0103 import locale import logging +import re +from copy import deepcopy from datetime import datetime from pathlib import Path -from copy import deepcopy from typing import Dict, List -from unittest.mock import MagicMock, PropertyMock +from unittest.mock import MagicMock import pandas as pd import pytest @@ -13,14 +14,12 @@ from arrow import Arrow from filelock import Timeout from freqtrade import constants -from freqtrade.commands.optimize_commands import (setup_optimize_configuration, - start_hyperopt) +from freqtrade.commands.optimize_commands import setup_optimize_configuration, start_hyperopt from freqtrade.data.history import load_data from freqtrade.exceptions import DependencyException, OperationalException -from freqtrade.optimize.default_hyperopt_loss import DefaultHyperOptLoss +from freqtrade.optimize.default_hyperopt_loss import ShortTradeDurHyperOptLoss from freqtrade.optimize.hyperopt import Hyperopt -from freqtrade.resolvers.hyperopt_resolver import (HyperOptLossResolver, - HyperOptResolver) +from freqtrade.resolvers.hyperopt_resolver import HyperOptLossResolver, HyperOptResolver from freqtrade.state import RunMode from freqtrade.strategy.interface import SellType from tests.conftest import (get_args, log_has, log_has_re, patch_exchange, @@ -34,6 +33,7 @@ def hyperopt_conf(default_conf): hyperconf = deepcopy(default_conf) hyperconf.update({ 'hyperopt': 'DefaultHyperOpt', + 'hyperopt_loss': 'ShortTradeDurHyperOptLoss', 'hyperopt_path': str(Path(__file__).parent / 'hyperopts'), 'epochs': 1, 'timerange': None, @@ -81,13 +81,14 @@ def create_results(mocker, hyperopt, testdatadir) -> List[Dict]: mocker.patch.object(Path, "is_file", MagicMock(return_value=False)) stat_mock = MagicMock() - stat_mock.st_size = PropertyMock(return_value=1) - mocker.patch.object(Path, "stat", MagicMock(return_value=False)) + stat_mock.st_size = 1 + mocker.patch.object(Path, "stat", MagicMock(return_value=stat_mock)) mocker.patch.object(Path, "unlink", MagicMock(return_value=True)) mocker.patch('freqtrade.optimize.hyperopt.dump', return_value=None) + mocker.patch('freqtrade.optimize.hyperopt.file_dump_json') - return [{'loss': 1, 'result': 'foo', 'params': {}}] + return [{'loss': 1, 'result': 'foo', 'params': {}, 'is_best': True}] def test_setup_hyperopt_configuration_without_arguments(mocker, default_conf, caplog) -> None: @@ -229,13 +230,21 @@ def test_hyperoptresolver_noname(default_conf): HyperOptResolver.load_hyperopt(default_conf) +def test_hyperoptlossresolver_noname(default_conf): + with pytest.raises(OperationalException, + match="No Hyperopt loss set. Please use `--hyperopt-loss` to specify " + "the Hyperopt-Loss class to use."): + HyperOptLossResolver.load_hyperoptloss(default_conf) + + def test_hyperoptlossresolver(mocker, default_conf) -> None: - hl = DefaultHyperOptLoss + hl = ShortTradeDurHyperOptLoss mocker.patch( 'freqtrade.resolvers.hyperopt_resolver.HyperOptLossResolver.load_object', MagicMock(return_value=hl) ) + default_conf.update({'hyperopt_loss': 'SharpeHyperOptLossDaily'}) x = HyperOptLossResolver.load_hyperoptloss(default_conf) assert hasattr(x, "hyperopt_loss_function") @@ -278,6 +287,7 @@ def test_start(mocker, hyperopt_conf, caplog) -> None: 'hyperopt', '--config', 'config.json', '--hyperopt', 'DefaultHyperOpt', + '--hyperopt-loss', 'SharpeHyperOptLossDaily', '--epochs', '5' ] pargs = get_args(args) @@ -301,6 +311,7 @@ def test_start_no_data(mocker, hyperopt_conf) -> None: 'hyperopt', '--config', 'config.json', '--hyperopt', 'DefaultHyperOpt', + '--hyperopt-loss', 'SharpeHyperOptLossDaily', '--epochs', '5' ] pargs = get_args(args) @@ -318,6 +329,7 @@ def test_start_filelock(mocker, hyperopt_conf, caplog) -> None: 'hyperopt', '--config', 'config.json', '--hyperopt', 'DefaultHyperOpt', + '--hyperopt-loss', 'SharpeHyperOptLossDaily', '--epochs', '5' ] pargs = get_args(args) @@ -325,8 +337,8 @@ def test_start_filelock(mocker, hyperopt_conf, caplog) -> None: assert log_has("Another running instance of freqtrade Hyperopt detected.", caplog) -def test_loss_calculation_prefer_correct_trade_count(default_conf, hyperopt_results) -> None: - hl = HyperOptLossResolver.load_hyperoptloss(default_conf) +def test_loss_calculation_prefer_correct_trade_count(hyperopt_conf, hyperopt_results) -> None: + hl = HyperOptLossResolver.load_hyperoptloss(hyperopt_conf) correct = hl.hyperopt_loss_function(hyperopt_results, 600, datetime(2019, 1, 1), datetime(2019, 5, 1)) over = hl.hyperopt_loss_function(hyperopt_results, 600 + 100, @@ -337,11 +349,11 @@ def test_loss_calculation_prefer_correct_trade_count(default_conf, hyperopt_resu assert under > correct -def test_loss_calculation_prefer_shorter_trades(default_conf, hyperopt_results) -> None: +def test_loss_calculation_prefer_shorter_trades(hyperopt_conf, hyperopt_results) -> None: resultsb = hyperopt_results.copy() resultsb.loc[1, 'trade_duration'] = 20 - hl = HyperOptLossResolver.load_hyperoptloss(default_conf) + hl = HyperOptLossResolver.load_hyperoptloss(hyperopt_conf) longer = hl.hyperopt_loss_function(hyperopt_results, 100, datetime(2019, 1, 1), datetime(2019, 5, 1)) shorter = hl.hyperopt_loss_function(resultsb, 100, @@ -349,13 +361,13 @@ def test_loss_calculation_prefer_shorter_trades(default_conf, hyperopt_results) assert shorter < longer -def test_loss_calculation_has_limited_profit(default_conf, hyperopt_results) -> None: +def test_loss_calculation_has_limited_profit(hyperopt_conf, hyperopt_results) -> None: results_over = hyperopt_results.copy() results_over['profit_percent'] = hyperopt_results['profit_percent'] * 2 results_under = hyperopt_results.copy() results_under['profit_percent'] = hyperopt_results['profit_percent'] / 2 - hl = HyperOptLossResolver.load_hyperoptloss(default_conf) + hl = HyperOptLossResolver.load_hyperoptloss(hyperopt_conf) correct = hl.hyperopt_loss_function(hyperopt_results, 600, datetime(2019, 1, 1), datetime(2019, 5, 1)) over = hl.hyperopt_loss_function(results_over, 600, @@ -372,7 +384,7 @@ def test_sharpe_loss_prefers_higher_profits(default_conf, hyperopt_results) -> N results_under = hyperopt_results.copy() results_under['profit_percent'] = hyperopt_results['profit_percent'] / 2 - default_conf.update({'hyperopt_loss': 'SharpeHyperOptLoss'}) + default_conf.update({'hyperopt_loss': 'SharpeHyperOptLossDaily'}) hl = HyperOptLossResolver.load_hyperoptloss(default_conf) correct = hl.hyperopt_loss_function(hyperopt_results, len(hyperopt_results), datetime(2019, 1, 1), datetime(2019, 5, 1)) @@ -497,6 +509,7 @@ def test_no_log_if_loss_does_not_improve(hyperopt, caplog) -> None: def test_save_results_saves_epochs(mocker, hyperopt, testdatadir, caplog) -> None: epochs = create_results(mocker, hyperopt, testdatadir) mock_dump = mocker.patch('freqtrade.optimize.hyperopt.dump', return_value=None) + mock_dump_json = mocker.patch('freqtrade.optimize.hyperopt.file_dump_json', return_value=None) results_file = testdatadir / 'optimize' / 'ut_results.pickle' caplog.set_level(logging.DEBUG) @@ -505,6 +518,7 @@ def test_save_results_saves_epochs(mocker, hyperopt, testdatadir, caplog) -> Non hyperopt._save_results() assert log_has(f"1 epoch saved to '{results_file}'.", caplog) mock_dump.assert_called_once() + mock_dump_json.assert_called_once() hyperopt.epochs = epochs + epochs hyperopt._save_results() @@ -521,6 +535,28 @@ def test_read_results_returns_epochs(mocker, hyperopt, testdatadir, caplog) -> N mock_load.assert_called_once() +def test_load_previous_results(mocker, hyperopt, testdatadir, caplog) -> None: + epochs = create_results(mocker, hyperopt, testdatadir) + mock_load = mocker.patch('freqtrade.optimize.hyperopt.load', return_value=epochs) + mocker.patch.object(Path, 'is_file', MagicMock(return_value=True)) + statmock = MagicMock() + statmock.st_size = 5 + # mocker.patch.object(Path, 'stat', MagicMock(return_value=statmock)) + + results_file = testdatadir / 'optimize' / 'ut_results.pickle' + + hyperopt_epochs = hyperopt.load_previous_results(results_file) + + assert hyperopt_epochs == epochs + mock_load.assert_called_once() + + del epochs[0]['is_best'] + mock_load = mocker.patch('freqtrade.optimize.hyperopt.load', return_value=epochs) + + with pytest.raises(OperationalException): + hyperopt.load_previous_results(results_file) + + def test_roi_table_generation(hyperopt) -> None: params = { 'roi_t1': 5, @@ -536,6 +572,8 @@ def test_roi_table_generation(hyperopt) -> None: def test_start_calls_optimizer(mocker, hyperopt_conf, capsys) -> None: dumper = mocker.patch('freqtrade.optimize.hyperopt.dump', MagicMock()) + mocker.patch('freqtrade.optimize.hyperopt.file_dump_json') + mocker.patch('freqtrade.optimize.backtesting.Backtesting.load_bt_data', MagicMock(return_value=(MagicMock(), None))) mocker.patch( @@ -813,7 +851,7 @@ def test_generate_optimizer(mocker, hyperopt_conf) -> None: 'draws': 0, 'duration': 100.0, 'losses': 0, - 'winsdrawslosses': '1/0/0', + 'winsdrawslosses': ' 1 0 0', 'median_profit': 2.3117, 'profit': 2.3117, 'total_profit': 0.000233, @@ -839,19 +877,10 @@ def test_clean_hyperopt(mocker, hyperopt_conf, caplog): assert log_has(f"Removing `{h.data_pickle_file}`.", caplog) -def test_continue_hyperopt(mocker, hyperopt_conf, caplog): - patch_exchange(mocker) - hyperopt_conf.update({'hyperopt_continue': True}) - mocker.patch("freqtrade.optimize.hyperopt.Path.is_file", MagicMock(return_value=True)) - unlinkmock = mocker.patch("freqtrade.optimize.hyperopt.Path.unlink", MagicMock()) - Hyperopt(hyperopt_conf) - - assert unlinkmock.call_count == 0 - assert log_has("Continuing on previous hyperopt results.", caplog) - - def test_print_json_spaces_all(mocker, hyperopt_conf, capsys) -> None: dumper = mocker.patch('freqtrade.optimize.hyperopt.dump', MagicMock()) + mocker.patch('freqtrade.optimize.hyperopt.file_dump_json') + mocker.patch('freqtrade.optimize.backtesting.Backtesting.load_bt_data', MagicMock(return_value=(MagicMock(), None))) mocker.patch( @@ -907,6 +936,7 @@ def test_print_json_spaces_all(mocker, hyperopt_conf, capsys) -> None: def test_print_json_spaces_default(mocker, hyperopt_conf, capsys) -> None: dumper = mocker.patch('freqtrade.optimize.hyperopt.dump', MagicMock()) + mocker.patch('freqtrade.optimize.hyperopt.file_dump_json') mocker.patch('freqtrade.optimize.backtesting.Backtesting.load_bt_data', MagicMock(return_value=(MagicMock(), None))) mocker.patch( @@ -954,6 +984,7 @@ def test_print_json_spaces_default(mocker, hyperopt_conf, capsys) -> None: def test_print_json_spaces_roi_stoploss(mocker, hyperopt_conf, capsys) -> None: dumper = mocker.patch('freqtrade.optimize.hyperopt.dump', MagicMock()) + mocker.patch('freqtrade.optimize.hyperopt.file_dump_json') mocker.patch('freqtrade.optimize.backtesting.Backtesting.load_bt_data', MagicMock(return_value=(MagicMock(), None))) mocker.patch( @@ -1000,6 +1031,7 @@ def test_print_json_spaces_roi_stoploss(mocker, hyperopt_conf, capsys) -> None: def test_simplified_interface_roi_stoploss(mocker, hyperopt_conf, capsys) -> None: dumper = mocker.patch('freqtrade.optimize.hyperopt.dump', MagicMock()) + mocker.patch('freqtrade.optimize.hyperopt.file_dump_json') mocker.patch('freqtrade.optimize.backtesting.Backtesting.load_bt_data', MagicMock(return_value=(MagicMock(), None))) mocker.patch( @@ -1052,6 +1084,7 @@ def test_simplified_interface_roi_stoploss(mocker, hyperopt_conf, capsys) -> Non def test_simplified_interface_all_failed(mocker, hyperopt_conf) -> None: mocker.patch('freqtrade.optimize.hyperopt.dump', MagicMock()) + mocker.patch('freqtrade.optimize.hyperopt.file_dump_json') mocker.patch('freqtrade.optimize.backtesting.Backtesting.load_bt_data', MagicMock(return_value=(MagicMock(), None))) mocker.patch( @@ -1078,6 +1111,7 @@ def test_simplified_interface_all_failed(mocker, hyperopt_conf) -> None: def test_simplified_interface_buy(mocker, hyperopt_conf, capsys) -> None: dumper = mocker.patch('freqtrade.optimize.hyperopt.dump', MagicMock()) + mocker.patch('freqtrade.optimize.hyperopt.file_dump_json') mocker.patch('freqtrade.optimize.backtesting.Backtesting.load_bt_data', MagicMock(return_value=(MagicMock(), None))) mocker.patch( @@ -1130,6 +1164,7 @@ def test_simplified_interface_buy(mocker, hyperopt_conf, capsys) -> None: def test_simplified_interface_sell(mocker, hyperopt_conf, capsys) -> None: dumper = mocker.patch('freqtrade.optimize.hyperopt.dump', MagicMock()) + mocker.patch('freqtrade.optimize.hyperopt.file_dump_json') mocker.patch('freqtrade.optimize.backtesting.Backtesting.load_bt_data', MagicMock(return_value=(MagicMock(), None))) mocker.patch( @@ -1188,6 +1223,7 @@ def test_simplified_interface_sell(mocker, hyperopt_conf, capsys) -> None: ]) def test_simplified_interface_failed(mocker, hyperopt_conf, method, space) -> None: mocker.patch('freqtrade.optimize.hyperopt.dump', MagicMock()) + mocker.patch('freqtrade.optimize.hyperopt.file_dump_json') mocker.patch('freqtrade.optimize.backtesting.Backtesting.load_bt_data', MagicMock(return_value=(MagicMock(), None))) mocker.patch( @@ -1207,3 +1243,39 @@ def test_simplified_interface_failed(mocker, hyperopt_conf, method, space) -> No with pytest.raises(OperationalException, match=f"The '{space}' space is included into *"): hyperopt.start() + + +def test_print_epoch_details(capsys): + test_result = { + 'params_details': { + 'trailing': { + 'trailing_stop': True, + 'trailing_stop_positive': 0.02, + 'trailing_stop_positive_offset': 0.04, + 'trailing_only_offset_is_reached': True + }, + 'roi': { + 0: 0.18, + 90: 0.14, + 225: 0.05, + 430: 0}, + }, + 'results_explanation': 'foo result', + 'is_initial_point': False, + 'total_profit': 0, + 'current_epoch': 2, # This starts from 1 (in a human-friendly manner) + 'is_best': True + } + + Hyperopt.print_epoch_details(test_result, 5, False, no_header=True) + captured = capsys.readouterr() + assert '# Trailing stop:' in captured.out + # re.match(r"Pairs for .*", captured.out) + assert re.search(r'^\s+trailing_stop = True$', captured.out, re.MULTILINE) + assert re.search(r'^\s+trailing_stop_positive = 0.02$', captured.out, re.MULTILINE) + assert re.search(r'^\s+trailing_stop_positive_offset = 0.04$', captured.out, re.MULTILINE) + assert re.search(r'^\s+trailing_only_offset_is_reached = True$', captured.out, re.MULTILINE) + + assert '# ROI table:' in captured.out + assert re.search(r'^\s+minimal_roi = \{$', captured.out, re.MULTILINE) + assert re.search(r'^\s+\"90\"\:\s0.14,\s*$', captured.out, re.MULTILINE) diff --git a/tests/optimize/test_optimize_reports.py b/tests/optimize/test_optimize_reports.py index 4f62e2e23..d04929164 100644 --- a/tests/optimize/test_optimize_reports.py +++ b/tests/optimize/test_optimize_reports.py @@ -9,24 +9,20 @@ from arrow import Arrow from freqtrade.configuration import TimeRange from freqtrade.constants import LAST_BT_RESULT_FN from freqtrade.data import history -from freqtrade.data.btanalysis import (get_latest_backtest_filename, - load_backtest_data) +from freqtrade.data.btanalysis import get_latest_backtest_filename, load_backtest_data from freqtrade.edge import PairInfo -from freqtrade.optimize.optimize_reports import (generate_backtest_stats, - generate_daily_stats, - generate_edge_table, - generate_pair_metrics, +from freqtrade.optimize.optimize_reports import (generate_backtest_stats, generate_daily_stats, + generate_edge_table, generate_pair_metrics, generate_sell_reason_stats, - generate_strategy_metrics, - store_backtest_stats, - text_table_bt_results, - text_table_sell_reason, + generate_strategy_metrics, store_backtest_stats, + text_table_bt_results, text_table_sell_reason, text_table_strategy) +from freqtrade.resolvers.strategy_resolver import StrategyResolver from freqtrade.strategy.interface import SellType from tests.data.test_history import _backup_file, _clean_test_file -def test_text_table_bt_results(default_conf, mocker): +def test_text_table_bt_results(): results = pd.DataFrame( { @@ -57,32 +53,38 @@ def test_text_table_bt_results(default_conf, mocker): def test_generate_backtest_stats(default_conf, testdatadir): - results = {'DefStrat': pd.DataFrame({"pair": ["UNITTEST/BTC", "UNITTEST/BTC", - "UNITTEST/BTC", "UNITTEST/BTC"], - "profit_percent": [0.003312, 0.010801, 0.013803, 0.002780], - "profit_abs": [0.000003, 0.000011, 0.000014, 0.000003], - "open_date": [Arrow(2017, 11, 14, 19, 32, 00).datetime, - Arrow(2017, 11, 14, 21, 36, 00).datetime, - Arrow(2017, 11, 14, 22, 12, 00).datetime, - Arrow(2017, 11, 14, 22, 44, 00).datetime], - "close_date": [Arrow(2017, 11, 14, 21, 35, 00).datetime, - Arrow(2017, 11, 14, 22, 10, 00).datetime, - Arrow(2017, 11, 14, 22, 43, 00).datetime, - Arrow(2017, 11, 14, 22, 58, 00).datetime], - "open_rate": [0.002543, 0.003003, 0.003089, 0.003214], - "close_rate": [0.002546, 0.003014, 0.003103, 0.003217], - "trade_duration": [123, 34, 31, 14], - "open_at_end": [False, False, False, True], - "sell_reason": [SellType.ROI, SellType.STOP_LOSS, - SellType.ROI, SellType.FORCE_SELL] - })} + default_conf.update({'strategy': 'DefaultStrategy'}) + StrategyResolver.load_strategy(default_conf) + + results = {'DefStrat': { + 'results': pd.DataFrame({"pair": ["UNITTEST/BTC", "UNITTEST/BTC", + "UNITTEST/BTC", "UNITTEST/BTC"], + "profit_percent": [0.003312, 0.010801, 0.013803, 0.002780], + "profit_abs": [0.000003, 0.000011, 0.000014, 0.000003], + "open_date": [Arrow(2017, 11, 14, 19, 32, 00).datetime, + Arrow(2017, 11, 14, 21, 36, 00).datetime, + Arrow(2017, 11, 14, 22, 12, 00).datetime, + Arrow(2017, 11, 14, 22, 44, 00).datetime], + "close_date": [Arrow(2017, 11, 14, 21, 35, 00).datetime, + Arrow(2017, 11, 14, 22, 10, 00).datetime, + Arrow(2017, 11, 14, 22, 43, 00).datetime, + Arrow(2017, 11, 14, 22, 58, 00).datetime], + "open_rate": [0.002543, 0.003003, 0.003089, 0.003214], + "close_rate": [0.002546, 0.003014, 0.003103, 0.003217], + "trade_duration": [123, 34, 31, 14], + "open_at_end": [False, False, False, True], + "sell_reason": [SellType.ROI, SellType.STOP_LOSS, + SellType.ROI, SellType.FORCE_SELL] + }), + 'config': default_conf} + } timerange = TimeRange.parse_timerange('1510688220-1510700340') min_date = Arrow.fromtimestamp(1510688220) max_date = Arrow.fromtimestamp(1510700340) btdata = history.load_data(testdatadir, '1m', ['UNITTEST/BTC'], timerange=timerange, fill_up_missing=True) - stats = generate_backtest_stats(default_conf, btdata, results, min_date, max_date) + stats = generate_backtest_stats(btdata, results, min_date, max_date) assert isinstance(stats, dict) assert 'strategy' in stats assert 'DefStrat' in stats['strategy'] @@ -90,29 +92,32 @@ def test_generate_backtest_stats(default_conf, testdatadir): strat_stats = stats['strategy']['DefStrat'] assert strat_stats['backtest_start'] == min_date.datetime assert strat_stats['backtest_end'] == max_date.datetime - assert strat_stats['total_trades'] == len(results['DefStrat']) + assert strat_stats['total_trades'] == len(results['DefStrat']['results']) # Above sample had no loosing trade assert strat_stats['max_drawdown'] == 0.0 - results = {'DefStrat': pd.DataFrame( - {"pair": ["UNITTEST/BTC", "UNITTEST/BTC", "UNITTEST/BTC", "UNITTEST/BTC"], - "profit_percent": [0.003312, 0.010801, -0.013803, 0.002780], - "profit_abs": [0.000003, 0.000011, -0.000014, 0.000003], - "open_date": [Arrow(2017, 11, 14, 19, 32, 00).datetime, - Arrow(2017, 11, 14, 21, 36, 00).datetime, - Arrow(2017, 11, 14, 22, 12, 00).datetime, - Arrow(2017, 11, 14, 22, 44, 00).datetime], - "close_date": [Arrow(2017, 11, 14, 21, 35, 00).datetime, - Arrow(2017, 11, 14, 22, 10, 00).datetime, - Arrow(2017, 11, 14, 22, 43, 00).datetime, - Arrow(2017, 11, 14, 22, 58, 00).datetime], - "open_rate": [0.002543, 0.003003, 0.003089, 0.003214], - "close_rate": [0.002546, 0.003014, 0.0032903, 0.003217], - "trade_duration": [123, 34, 31, 14], - "open_at_end": [False, False, False, True], - "sell_reason": [SellType.ROI, SellType.STOP_LOSS, - SellType.ROI, SellType.FORCE_SELL] - })} + results = {'DefStrat': { + 'results': pd.DataFrame( + {"pair": ["UNITTEST/BTC", "UNITTEST/BTC", "UNITTEST/BTC", "UNITTEST/BTC"], + "profit_percent": [0.003312, 0.010801, -0.013803, 0.002780], + "profit_abs": [0.000003, 0.000011, -0.000014, 0.000003], + "open_date": [Arrow(2017, 11, 14, 19, 32, 00).datetime, + Arrow(2017, 11, 14, 21, 36, 00).datetime, + Arrow(2017, 11, 14, 22, 12, 00).datetime, + Arrow(2017, 11, 14, 22, 44, 00).datetime], + "close_date": [Arrow(2017, 11, 14, 21, 35, 00).datetime, + Arrow(2017, 11, 14, 22, 10, 00).datetime, + Arrow(2017, 11, 14, 22, 43, 00).datetime, + Arrow(2017, 11, 14, 22, 58, 00).datetime], + "open_rate": [0.002543, 0.003003, 0.003089, 0.003214], + "close_rate": [0.002546, 0.003014, 0.0032903, 0.003217], + "trade_duration": [123, 34, 31, 14], + "open_at_end": [False, False, False, True], + "sell_reason": [SellType.ROI, SellType.STOP_LOSS, + SellType.ROI, SellType.FORCE_SELL] + }), + 'config': default_conf} + } assert strat_stats['max_drawdown'] == 0.0 assert strat_stats['drawdown_start'] == Arrow.fromtimestamp(0).datetime @@ -165,7 +170,7 @@ def test_store_backtest_stats(testdatadir, mocker): assert str(dump_mock.call_args_list[0][0][0]).startswith(str(testdatadir / 'testresult')) -def test_generate_pair_metrics(default_conf, mocker): +def test_generate_pair_metrics(): results = pd.DataFrame( { @@ -213,7 +218,7 @@ def test_generate_daily_stats(testdatadir): assert res['losing_days'] == 0 -def test_text_table_sell_reason(default_conf): +def test_text_table_sell_reason(): results = pd.DataFrame( { @@ -245,7 +250,7 @@ def test_text_table_sell_reason(default_conf): stake_currency='BTC') == result_str -def test_generate_sell_reason_stats(default_conf): +def test_generate_sell_reason_stats(): results = pd.DataFrame( { @@ -280,9 +285,10 @@ def test_generate_sell_reason_stats(default_conf): assert stop_result['profit_mean_pct'] == round(stop_result['profit_mean'] * 100, 2) -def test_text_table_strategy(default_conf, mocker): +def test_text_table_strategy(default_conf): + default_conf['max_open_trades'] = 2 results = {} - results['TestStrategy1'] = pd.DataFrame( + results['TestStrategy1'] = {'results': pd.DataFrame( { 'pair': ['ETH/BTC', 'ETH/BTC', 'ETH/BTC'], 'profit_percent': [0.1, 0.2, 0.3], @@ -293,8 +299,8 @@ def test_text_table_strategy(default_conf, mocker): 'losses': [0, 0, 1], 'sell_reason': [SellType.ROI, SellType.ROI, SellType.STOP_LOSS] } - ) - results['TestStrategy2'] = pd.DataFrame( + ), 'config': default_conf} + results['TestStrategy2'] = {'results': pd.DataFrame( { 'pair': ['LTC/BTC', 'LTC/BTC', 'LTC/BTC'], 'profit_percent': [0.4, 0.2, 0.3], @@ -305,7 +311,7 @@ def test_text_table_strategy(default_conf, mocker): 'losses': [0, 0, 1], 'sell_reason': [SellType.ROI, SellType.ROI, SellType.STOP_LOSS] } - ) + ), 'config': default_conf} result_str = ( '| Strategy | Buys | Avg Profit % | Cum Profit % | Tot' @@ -318,14 +324,12 @@ def test_text_table_strategy(default_conf, mocker): ' 45.00 | 0:20:00 | 3 | 0 | 0 |' ) - strategy_results = generate_strategy_metrics(stake_currency='BTC', - max_open_trades=2, - all_results=results) + strategy_results = generate_strategy_metrics(all_results=results) assert text_table_strategy(strategy_results, 'BTC') == result_str -def test_generate_edge_table(edge_conf, mocker): +def test_generate_edge_table(): results = {} results['ETH/BTC'] = PairInfo(-0.01, 0.60, 2, 1, 3, 10, 60) diff --git a/tests/rpc/test_rpc.py b/tests/rpc/test_rpc.py index 102ed12fe..977dfbc20 100644 --- a/tests/rpc/test_rpc.py +++ b/tests/rpc/test_rpc.py @@ -13,8 +13,7 @@ from freqtrade.persistence import Trade from freqtrade.rpc import RPC, RPCException from freqtrade.rpc.fiat_convert import CryptoToFiatConverter from freqtrade.state import State -from tests.conftest import (create_mock_trades, get_patched_freqtradebot, - patch_get_signal) +from tests.conftest import create_mock_trades, get_patched_freqtradebot, patch_get_signal # Functions for recurrent object patching @@ -313,7 +312,6 @@ def test_rpc_delete_trade(mocker, default_conf, fee, markets, caplog): with pytest.raises(RPCException, match='invalid argument'): rpc._rpc_delete('200') - create_mock_trades(fee) trades = Trade.query.all() trades[1].stoploss_order_id = '1234' trades[2].stoploss_order_id = '1234' @@ -717,11 +715,13 @@ def test_rpc_forcesell(default_conf, ticker, fee, mocker) -> None: mocker.patch( 'freqtrade.exchange.Exchange.fetch_order', side_effect=[{ + 'id': '1234', 'status': 'open', 'type': 'limit', 'side': 'buy', 'filled': filled_amount }, { + 'id': '1234', 'status': 'closed', 'type': 'limit', 'side': 'buy', @@ -837,10 +837,10 @@ def test_rpc_count(mocker, default_conf, ticker, fee) -> None: assert counts["current"] == 1 -def test_rpcforcebuy(mocker, default_conf, ticker, fee, limit_buy_order) -> None: +def test_rpcforcebuy(mocker, default_conf, ticker, fee, limit_buy_order_open) -> None: default_conf['forcebuy_enable'] = True mocker.patch('freqtrade.rpc.telegram.Telegram', MagicMock()) - buy_mm = MagicMock(return_value={'id': limit_buy_order['id']}) + buy_mm = MagicMock(return_value=limit_buy_order_open) mocker.patch.multiple( 'freqtrade.exchange.Exchange', get_balances=MagicMock(return_value=ticker), diff --git a/tests/rpc/test_rpc_apiserver.py b/tests/rpc/test_rpc_apiserver.py index 626586a4a..34e959875 100644 --- a/tests/rpc/test_rpc_apiserver.py +++ b/tests/rpc/test_rpc_apiserver.py @@ -2,7 +2,8 @@ Unit test file for rpc/api_server.py """ -from datetime import datetime +from datetime import datetime, timedelta +from pathlib import Path from unittest.mock import ANY, MagicMock, PropertyMock import pytest @@ -11,11 +12,11 @@ from requests.auth import _basic_auth_str from freqtrade.__init__ import __version__ from freqtrade.loggers import setup_logging, setup_logging_pre -from freqtrade.persistence import Trade +from freqtrade.persistence import PairLock, Trade from freqtrade.rpc.api_server import BASE_URI, ApiServer from freqtrade.state import State -from tests.conftest import (create_mock_trades, get_patched_freqtradebot, - log_has, patch_get_signal) +from tests.conftest import create_mock_trades, get_patched_freqtradebot, log_has, patch_get_signal + _TEST_USER = "FreqTrader" _TEST_PASS = "SuperSecurePassword1!" @@ -327,6 +328,30 @@ def test_api_count(botclient, mocker, ticker, fee, markets): assert rc.json["max"] == 1.0 +def test_api_locks(botclient): + ftbot, client = botclient + + rc = client_get(client, f"{BASE_URI}/locks") + assert_response(rc) + + assert 'locks' in rc.json + + assert rc.json['lock_count'] == 0 + assert rc.json['lock_count'] == len(rc.json['locks']) + + PairLock.lock_pair('ETH/BTC', datetime.utcnow() + timedelta(minutes=4), 'randreason') + PairLock.lock_pair('XRP/BTC', datetime.utcnow() + timedelta(minutes=20), 'deadbeef') + + rc = client_get(client, f"{BASE_URI}/locks") + assert_response(rc) + + assert rc.json['lock_count'] == 2 + assert rc.json['lock_count'] == len(rc.json['locks']) + assert 'ETH/BTC' in (rc.json['locks'][0]['pair'], rc.json['locks'][1]['pair']) + assert 'randreason' in (rc.json['locks'][0]['reason'], rc.json['locks'][1]['reason']) + assert 'deadbeef' in (rc.json['locks'][0]['reason'], rc.json['locks'][1]['reason']) + + def test_api_show_config(botclient, mocker): ftbot, client = botclient patch_get_signal(ftbot, (True, False)) @@ -811,3 +836,176 @@ def test_api_forcesell(botclient, mocker, ticker, fee, markets): data='{"tradeid": "1"}') assert_response(rc) assert rc.json == {'result': 'Created sell order for trade 1.'} + + +def test_api_pair_candles(botclient, ohlcv_history): + ftbot, client = botclient + timeframe = '5m' + amount = 2 + + # No pair + rc = client_get(client, + f"{BASE_URI}/pair_candles?limit={amount}&timeframe={timeframe}") + assert_response(rc, 400) + + # No timeframe + rc = client_get(client, + f"{BASE_URI}/pair_candles?pair=XRP%2FBTC") + assert_response(rc, 400) + + rc = client_get(client, + f"{BASE_URI}/pair_candles?limit={amount}&pair=XRP%2FBTC&timeframe={timeframe}") + assert_response(rc) + assert 'columns' in rc.json + assert 'data_start_ts' in rc.json + assert 'data_start' in rc.json + assert 'data_stop' in rc.json + assert 'data_stop_ts' in rc.json + assert len(rc.json['data']) == 0 + ohlcv_history['sma'] = ohlcv_history['close'].rolling(2).mean() + ohlcv_history['buy'] = 0 + ohlcv_history.loc[1, 'buy'] = 1 + ohlcv_history['sell'] = 0 + + ftbot.dataprovider._set_cached_df("XRP/BTC", timeframe, ohlcv_history) + + rc = client_get(client, + f"{BASE_URI}/pair_candles?limit={amount}&pair=XRP%2FBTC&timeframe={timeframe}") + assert_response(rc) + assert 'strategy' in rc.json + assert rc.json['strategy'] == 'DefaultStrategy' + assert 'columns' in rc.json + assert 'data_start_ts' in rc.json + assert 'data_start' in rc.json + assert 'data_stop' in rc.json + assert 'data_stop_ts' in rc.json + assert rc.json['data_start'] == '2017-11-26 08:50:00+00:00' + assert rc.json['data_start_ts'] == 1511686200000 + assert rc.json['data_stop'] == '2017-11-26 08:55:00+00:00' + assert rc.json['data_stop_ts'] == 1511686500000 + assert isinstance(rc.json['columns'], list) + assert rc.json['columns'] == ['date', 'open', 'high', + 'low', 'close', 'volume', 'sma', 'buy', 'sell', + '__date_ts', '_buy_signal_open', '_sell_signal_open'] + assert 'pair' in rc.json + assert rc.json['pair'] == 'XRP/BTC' + + assert 'data' in rc.json + assert len(rc.json['data']) == amount + + assert (rc.json['data'] == + [['2017-11-26 08:50:00', 8.794e-05, 8.948e-05, 8.794e-05, 8.88e-05, 0.0877869, + None, 0, 0, 1511686200000, None, None], + ['2017-11-26 08:55:00', 8.88e-05, 8.942e-05, 8.88e-05, + 8.893e-05, 0.05874751, 8.886500000000001e-05, 1, 0, 1511686500000, 8.88e-05, None] + ]) + + +def test_api_pair_history(botclient, ohlcv_history): + ftbot, client = botclient + timeframe = '5m' + + # No pair + rc = client_get(client, + f"{BASE_URI}/pair_history?timeframe={timeframe}" + "&timerange=20180111-20180112&strategy=DefaultStrategy") + assert_response(rc, 400) + + # No Timeframe + rc = client_get(client, + f"{BASE_URI}/pair_history?pair=UNITTEST%2FBTC" + "&timerange=20180111-20180112&strategy=DefaultStrategy") + assert_response(rc, 400) + + # No timerange + rc = client_get(client, + f"{BASE_URI}/pair_history?pair=UNITTEST%2FBTC&timeframe={timeframe}" + "&strategy=DefaultStrategy") + assert_response(rc, 400) + + # No strategy + rc = client_get(client, + f"{BASE_URI}/pair_history?pair=UNITTEST%2FBTC&timeframe={timeframe}" + "&timerange=20180111-20180112") + assert_response(rc, 400) + + # Working + rc = client_get(client, + f"{BASE_URI}/pair_history?pair=UNITTEST%2FBTC&timeframe={timeframe}" + "&timerange=20180111-20180112&strategy=DefaultStrategy") + assert_response(rc, 200) + assert rc.json['length'] == 289 + assert len(rc.json['data']) == rc.json['length'] + assert 'columns' in rc.json + assert 'data' in rc.json + assert rc.json['pair'] == 'UNITTEST/BTC' + assert rc.json['strategy'] == 'DefaultStrategy' + assert rc.json['data_start'] == '2018-01-11 00:00:00+00:00' + assert rc.json['data_start_ts'] == 1515628800000 + assert rc.json['data_stop'] == '2018-01-12 00:00:00+00:00' + assert rc.json['data_stop_ts'] == 1515715200000 + + +def test_api_plot_config(botclient): + ftbot, client = botclient + + rc = client_get(client, f"{BASE_URI}/plot_config") + assert_response(rc) + assert rc.json == {} + + ftbot.strategy.plot_config = {'main_plot': {'sma': {}}, + 'subplots': {'RSI': {'rsi': {'color': 'red'}}}} + rc = client_get(client, f"{BASE_URI}/plot_config") + assert_response(rc) + assert rc.json == ftbot.strategy.plot_config + assert isinstance(rc.json['main_plot'], dict) + + +def test_api_strategies(botclient): + ftbot, client = botclient + + rc = client_get(client, f"{BASE_URI}/strategies") + + assert_response(rc) + assert rc.json == {'strategies': ['DefaultStrategy', 'TestStrategyLegacy']} + + +def test_api_strategy(botclient): + ftbot, client = botclient + + rc = client_get(client, f"{BASE_URI}/strategy/DefaultStrategy") + + assert_response(rc) + assert rc.json['strategy'] == 'DefaultStrategy' + + data = (Path(__file__).parents[1] / "strategy/strats/default_strategy.py").read_text() + assert rc.json['code'] == data + + rc = client_get(client, f"{BASE_URI}/strategy/NoStrat") + assert_response(rc, 404) + + +def test_list_available_pairs(botclient): + ftbot, client = botclient + + rc = client_get(client, f"{BASE_URI}/available_pairs") + + assert_response(rc) + assert rc.json['length'] == 12 + assert isinstance(rc.json['pairs'], list) + + rc = client_get(client, f"{BASE_URI}/available_pairs?timeframe=5m") + assert_response(rc) + assert rc.json['length'] == 12 + + rc = client_get(client, f"{BASE_URI}/available_pairs?stake_currency=ETH") + assert_response(rc) + assert rc.json['length'] == 1 + assert rc.json['pairs'] == ['XRP/ETH'] + assert len(rc.json['pair_interval']) == 2 + + rc = client_get(client, f"{BASE_URI}/available_pairs?stake_currency=ETH&timeframe=5m") + assert_response(rc) + assert rc.json['length'] == 1 + assert rc.json['pairs'] == ['XRP/ETH'] + assert len(rc.json['pair_interval']) == 1 diff --git a/tests/rpc/test_rpc_manager.py b/tests/rpc/test_rpc_manager.py index edf6bae4d..4b715fc37 100644 --- a/tests/rpc/test_rpc_manager.py +++ b/tests/rpc/test_rpc_manager.py @@ -1,10 +1,10 @@ # pragma pylint: disable=missing-docstring, C0103 -import time import logging +import time from unittest.mock import MagicMock -from freqtrade.rpc import RPCMessageType, RPCManager -from tests.conftest import log_has, get_patched_freqtradebot +from freqtrade.rpc import RPCManager, RPCMessageType +from tests.conftest import get_patched_freqtradebot, log_has def test__init__(mocker, default_conf) -> None: @@ -124,10 +124,10 @@ def test_send_msg_webhook_CustomMessagetype(mocker, default_conf, caplog) -> Non rpc_manager = RPCManager(get_patched_freqtradebot(mocker, default_conf)) assert 'webhook' in [mod.name for mod in rpc_manager.registered_modules] - rpc_manager.send_msg({'type': RPCMessageType.CUSTOM_NOTIFICATION, + rpc_manager.send_msg({'type': RPCMessageType.STARTUP_NOTIFICATION, 'status': 'TestMessage'}) assert log_has( - "Message type RPCMessageType.CUSTOM_NOTIFICATION not implemented by handler webhook.", + "Message type 'startup' not implemented by handler webhook.", caplog) diff --git a/tests/rpc/test_rpc_telegram.py b/tests/rpc/test_rpc_telegram.py index bcb9abc85..86d9656b5 100644 --- a/tests/rpc/test_rpc_telegram.py +++ b/tests/rpc/test_rpc_telegram.py @@ -18,14 +18,13 @@ from freqtrade.constants import CANCEL_REASON from freqtrade.edge import PairInfo from freqtrade.freqtradebot import FreqtradeBot from freqtrade.loggers import setup_logging -from freqtrade.persistence import Trade +from freqtrade.persistence import PairLock, Trade from freqtrade.rpc import RPCMessageType from freqtrade.rpc.telegram import Telegram, authorized_only from freqtrade.state import State from freqtrade.strategy.interface import SellType -from tests.conftest import (create_mock_trades, get_patched_freqtradebot, - log_has, patch_exchange, patch_get_signal, - patch_whitelist) +from tests.conftest import (create_mock_trades, get_patched_freqtradebot, log_has, patch_exchange, + patch_get_signal, patch_whitelist) class DummyCls(Telegram): @@ -76,15 +75,15 @@ def test_telegram_init(default_conf, mocker, caplog) -> None: message_str = ("rpc.telegram is listening for following commands: [['status'], ['profit'], " "['balance'], ['start'], ['stop'], ['forcesell'], ['forcebuy'], ['trades'], " - "['delete'], ['performance'], ['daily'], ['count'], ['reload_config', " - "'reload_conf'], ['show_config', 'show_conf'], ['stopbuy'], " + "['delete'], ['performance'], ['daily'], ['count'], ['locks'], " + "['reload_config', 'reload_conf'], ['show_config', 'show_conf'], ['stopbuy'], " "['whitelist'], ['blacklist'], ['logs'], ['edge'], ['help'], ['version'], " "['stats']]") assert log_has(message_str, caplog) -def test_cleanup(default_conf, mocker) -> None: +def test_cleanup(default_conf, mocker, ) -> None: updater_mock = MagicMock() updater_mock.stop = MagicMock() mocker.patch('freqtrade.rpc.telegram.Updater', updater_mock) @@ -94,13 +93,9 @@ def test_cleanup(default_conf, mocker) -> None: assert telegram._updater.stop.call_count == 1 -def test_authorized_only(default_conf, mocker, caplog) -> None: +def test_authorized_only(default_conf, mocker, caplog, update) -> None: patch_exchange(mocker) - chat = Chat(0, 0) - update = Update(randint(1, 100)) - update.message = Message(randint(1, 100), 0, datetime.utcnow(), chat) - default_conf['telegram']['enabled'] = False bot = FreqtradeBot(default_conf) patch_get_signal(bot, (True, False)) @@ -116,7 +111,7 @@ def test_authorized_only_unauthorized(default_conf, mocker, caplog) -> None: patch_exchange(mocker) chat = Chat(0xdeadbeef, 0) update = Update(randint(1, 100)) - update.message = Message(randint(1, 100), 0, datetime.utcnow(), chat) + update.message = Message(randint(1, 100), datetime.utcnow(), chat) default_conf['telegram']['enabled'] = False bot = FreqtradeBot(default_conf) @@ -129,12 +124,9 @@ def test_authorized_only_unauthorized(default_conf, mocker, caplog) -> None: assert not log_has('Exception occurred within Telegram module', caplog) -def test_authorized_only_exception(default_conf, mocker, caplog) -> None: +def test_authorized_only_exception(default_conf, mocker, caplog, update) -> None: patch_exchange(mocker) - update = Update(randint(1, 100)) - update.message = Message(randint(1, 100), 0, datetime.utcnow(), Chat(0, 0)) - default_conf['telegram']['enabled'] = False bot = FreqtradeBot(default_conf) @@ -148,7 +140,7 @@ def test_authorized_only_exception(default_conf, mocker, caplog) -> None: assert log_has('Exception occurred within Telegram module', caplog) -def test_telegram_status(default_conf, update, mocker, fee, ticker,) -> None: +def test_telegram_status(default_conf, update, mocker) -> None: update.message.chat.id = "123" default_conf['telegram']['enabled'] = False default_conf['telegram']['chat_id'] = "123" @@ -254,7 +246,6 @@ def test_status_table_handle(default_conf, update, ticker, fee, mocker) -> None: mocker.patch.multiple( 'freqtrade.exchange.Exchange', fetch_ticker=ticker, - buy=MagicMock(return_value={'id': 'mocked_order_id'}), get_fee=fee, ) msg_mock = MagicMock() @@ -1008,7 +999,6 @@ def test_count_handle(default_conf, update, ticker, fee, mocker) -> None: mocker.patch.multiple( 'freqtrade.exchange.Exchange', fetch_ticker=ticker, - buy=MagicMock(return_value={'id': 'mocked_order_id'}), get_fee=fee, ) freqtradebot = get_patched_freqtradebot(mocker, default_conf) @@ -1035,6 +1025,43 @@ def test_count_handle(default_conf, update, ticker, fee, mocker) -> None: assert msg in msg_mock.call_args_list[0][0][0] +def test_telegram_lock_handle(default_conf, update, ticker, fee, mocker) -> None: + msg_mock = MagicMock() + mocker.patch.multiple( + 'freqtrade.rpc.telegram.Telegram', + _init=MagicMock(), + _send_msg=msg_mock + ) + mocker.patch.multiple( + 'freqtrade.exchange.Exchange', + fetch_ticker=ticker, + get_fee=fee, + ) + freqtradebot = get_patched_freqtradebot(mocker, default_conf) + patch_get_signal(freqtradebot, (True, False)) + telegram = Telegram(freqtradebot) + + freqtradebot.state = State.STOPPED + telegram._locks(update=update, context=MagicMock()) + assert msg_mock.call_count == 1 + assert 'not running' in msg_mock.call_args_list[0][0][0] + msg_mock.reset_mock() + freqtradebot.state = State.RUNNING + + PairLock.lock_pair('ETH/BTC', arrow.utcnow().shift(minutes=4).datetime, 'randreason') + PairLock.lock_pair('XRP/BTC', arrow.utcnow().shift(minutes=20).datetime, 'deadbeef') + + telegram._locks(update=update, context=MagicMock()) + + assert 'Pair' in msg_mock.call_args_list[0][0][0] + assert 'Until' in msg_mock.call_args_list[0][0][0] + assert 'Reason\n' in msg_mock.call_args_list[0][0][0] + assert 'ETH/BTC' in msg_mock.call_args_list[0][0][0] + assert 'XRP/BTC' in msg_mock.call_args_list[0][0][0] + assert 'deadbeef' in msg_mock.call_args_list[0][0][0] + assert 'randreason' in msg_mock.call_args_list[0][0][0] + + def test_whitelist_static(default_conf, update, mocker) -> None: msg_mock = MagicMock() mocker.patch.multiple( @@ -1142,9 +1169,9 @@ def test_telegram_logs(default_conf, update, mocker) -> None: context = MagicMock() context.args = [] telegram._logs(update=update, context=context) - # Called at least 3 times. Exact times will change with unrelated changes to setup messages + # Called at least 2 times. Exact times will change with unrelated changes to setup messages # Therefore we don't test for this explicitly. - assert msg_mock.call_count > 3 + assert msg_mock.call_count >= 2 def test_edge_disabled(default_conf, update, mocker) -> None: @@ -1302,16 +1329,14 @@ def test_show_config_handle(default_conf, update, mocker) -> None: assert '*Initial Stoploss:* `-0.1`' in msg_mock.call_args_list[0][0][0] -def test_send_msg_buy_notification(default_conf, mocker) -> None: +def test_send_msg_buy_notification(default_conf, mocker, caplog) -> None: msg_mock = MagicMock() mocker.patch.multiple( 'freqtrade.rpc.telegram.Telegram', _init=MagicMock(), _send_msg=msg_mock ) - freqtradebot = get_patched_freqtradebot(mocker, default_conf) - telegram = Telegram(freqtradebot) - telegram.send_msg({ + msg = { 'type': RPCMessageType.BUY_NOTIFICATION, 'exchange': 'Bittrex', 'pair': 'ETH/BTC', @@ -1324,7 +1349,10 @@ def test_send_msg_buy_notification(default_conf, mocker) -> None: 'current_rate': 1.099e-05, 'amount': 1333.3333333333335, 'open_date': arrow.utcnow().shift(hours=-1) - }) + } + freqtradebot = get_patched_freqtradebot(mocker, default_conf) + telegram = Telegram(freqtradebot) + telegram.send_msg(msg) assert msg_mock.call_args[0][0] \ == '\N{LARGE BLUE CIRCLE} *Bittrex:* Buying ETH/BTC\n' \ '*Amount:* `1333.33333333`\n' \ @@ -1332,6 +1360,21 @@ def test_send_msg_buy_notification(default_conf, mocker) -> None: '*Current Rate:* `0.00001099`\n' \ '*Total:* `(0.001000 BTC, 12.345 USD)`' + freqtradebot.config['telegram']['notification_settings'] = {'buy': 'off'} + caplog.clear() + msg_mock.reset_mock() + telegram.send_msg(msg) + msg_mock.call_count == 0 + log_has("Notification 'buy' not sent.", caplog) + + freqtradebot.config['telegram']['notification_settings'] = {'buy': 'silent'} + caplog.clear() + msg_mock.reset_mock() + + telegram.send_msg(msg) + msg_mock.call_count == 1 + msg_mock.call_args_list[0][1]['disable_notification'] is True + def test_send_msg_buy_cancel_notification(default_conf, mocker) -> None: msg_mock = MagicMock() @@ -1488,7 +1531,7 @@ def test_warning_notification(default_conf, mocker) -> None: assert msg_mock.call_args[0][0] == '\N{WARNING SIGN} *Warning:* `message`' -def test_custom_notification(default_conf, mocker) -> None: +def test_startup_notification(default_conf, mocker) -> None: msg_mock = MagicMock() mocker.patch.multiple( 'freqtrade.rpc.telegram.Telegram', @@ -1498,7 +1541,7 @@ def test_custom_notification(default_conf, mocker) -> None: freqtradebot = get_patched_freqtradebot(mocker, default_conf) telegram = Telegram(freqtradebot) telegram.send_msg({ - 'type': RPCMessageType.CUSTOM_NOTIFICATION, + 'type': RPCMessageType.STARTUP_NOTIFICATION, 'status': '*Custom:* `Hello World`' }) assert msg_mock.call_args[0][0] == '*Custom:* `Hello World`' diff --git a/tests/rpc/test_rpc_webhook.py b/tests/rpc/test_rpc_webhook.py index 1ced62746..9256a5316 100644 --- a/tests/rpc/test_rpc_webhook.py +++ b/tests/rpc/test_rpc_webhook.py @@ -150,7 +150,7 @@ def test_send_msg(default_conf, mocker): default_conf["webhook"]["webhooksellcancel"]["value3"].format(**msg)) for msgtype in [RPCMessageType.STATUS_NOTIFICATION, RPCMessageType.WARNING_NOTIFICATION, - RPCMessageType.CUSTOM_NOTIFICATION]: + RPCMessageType.STARTUP_NOTIFICATION]: # Test notification msg = { 'type': msgtype, @@ -174,7 +174,7 @@ def test_exception_send_msg(default_conf, mocker, caplog): webhook = Webhook(get_patched_freqtradebot(mocker, default_conf)) webhook.send_msg({'type': RPCMessageType.BUY_NOTIFICATION}) - assert log_has(f"Message type {RPCMessageType.BUY_NOTIFICATION} not configured for webhooks", + assert log_has(f"Message type '{RPCMessageType.BUY_NOTIFICATION}' not configured for webhooks", caplog) default_conf["webhook"] = get_webhook_dict() diff --git a/tests/strategy/strats/legacy_strategy.py b/tests/strategy/strats/legacy_strategy.py index 9cbce0ad5..1e7bb5e1e 100644 --- a/tests/strategy/strats/legacy_strategy.py +++ b/tests/strategy/strats/legacy_strategy.py @@ -1,12 +1,13 @@ # --- Do not remove these libs --- -from freqtrade.strategy.interface import IStrategy -from pandas import DataFrame -# -------------------------------- - # Add your lib to import here import talib.abstract as ta +from pandas import DataFrame +from freqtrade.strategy.interface import IStrategy + + +# -------------------------------- # This class is a sample. Feel free to customize it. class TestStrategyLegacy(IStrategy): diff --git a/tests/strategy/test_interface.py b/tests/strategy/test_interface.py index f1b5d0244..dc5cd47e7 100644 --- a/tests/strategy/test_interface.py +++ b/tests/strategy/test_interface.py @@ -1,5 +1,4 @@ # pragma pylint: disable=missing-docstring, C0103 - import logging from datetime import datetime, timedelta, timezone from unittest.mock import MagicMock @@ -12,13 +11,14 @@ from freqtrade.configuration import TimeRange from freqtrade.data.dataprovider import DataProvider from freqtrade.data.history import load_data from freqtrade.exceptions import StrategyError -from freqtrade.persistence import Trade +from freqtrade.persistence import PairLock, Trade from freqtrade.resolvers import StrategyResolver from freqtrade.strategy.strategy_wrapper import strategy_safe_wrapper from tests.conftest import log_has, log_has_re from .strats.default_strategy import DefaultStrategy + # Avoid to reinit the same object again and again _STRATEGY = DefaultStrategy(config={}) _STRATEGY.dp = DataProvider({}, None, None) @@ -359,11 +359,12 @@ def test__analyze_ticker_internal_skip_analyze(ohlcv_history, mocker, caplog) -> assert log_has('Skipping TA Analysis for already analyzed candle', caplog) +@pytest.mark.usefixtures("init_persistence") def test_is_pair_locked(default_conf): default_conf.update({'strategy': 'DefaultStrategy'}) strategy = StrategyResolver.load_strategy(default_conf) - # dict should be empty - assert not strategy._pair_locked_until + # No lock should be present + assert len(PairLock.query.all()) == 0 pair = 'ETH/BTC' assert not strategy.is_pair_locked(pair) @@ -371,11 +372,6 @@ def test_is_pair_locked(default_conf): # ETH/BTC locked for 4 minutes assert strategy.is_pair_locked(pair) - # Test lock does not change - lock = strategy._pair_locked_until[pair] - strategy.lock_pair(pair, arrow.utcnow().shift(minutes=2).datetime) - assert lock == strategy._pair_locked_until[pair] - # XRP/BTC should not be locked now pair = 'XRP/BTC' assert not strategy.is_pair_locked(pair) @@ -392,7 +388,7 @@ def test_is_pair_locked(default_conf): # Lock until 14:30 lock_time = datetime(2020, 5, 1, 14, 30, 0, tzinfo=timezone.utc) strategy.lock_pair(pair, lock_time) - # Lock is in the past ... + assert not strategy.is_pair_locked(pair) # latest candle is from 14:20, lock goes to 14:30 assert strategy.is_pair_locked(pair, lock_time + timedelta(minutes=-10)) diff --git a/tests/strategy/test_strategy.py b/tests/strategy/test_strategy.py index 240f3d8ec..1c692d2da 100644 --- a/tests/strategy/test_strategy.py +++ b/tests/strategy/test_strategy.py @@ -18,13 +18,15 @@ def test_search_strategy(): s, _ = StrategyResolver._search_object( directory=default_location, - object_name='DefaultStrategy' + object_name='DefaultStrategy', + add_source=True, ) assert issubclass(s, IStrategy) s, _ = StrategyResolver._search_object( directory=default_location, - object_name='NotFoundStrategy' + object_name='NotFoundStrategy', + add_source=True, ) assert s is None @@ -53,6 +55,9 @@ def test_load_strategy(default_conf, result): 'strategy_path': str(Path(__file__).parents[2] / 'freqtrade/templates') }) strategy = StrategyResolver.load_strategy(default_conf) + assert isinstance(strategy.__source__, str) + assert 'class SampleStrategy' in strategy.__source__ + assert isinstance(strategy.__file__, str) assert 'rsi' in strategy.advise_indicators(result, {'pair': 'ETH/BTC'}) diff --git a/tests/strategy/test_strategy_helpers.py b/tests/strategy/test_strategy_helpers.py index 4b29bf304..1d3e80d24 100644 --- a/tests/strategy/test_strategy_helpers.py +++ b/tests/strategy/test_strategy_helpers.py @@ -1,5 +1,5 @@ -import pandas as pd import numpy as np +import pandas as pd from freqtrade.strategy import merge_informative_pair, timeframe_to_minutes diff --git a/tests/test_configuration.py b/tests/test_configuration.py index 4428fe240..7d6c81f74 100644 --- a/tests/test_configuration.py +++ b/tests/test_configuration.py @@ -11,20 +11,18 @@ import pytest from jsonschema import ValidationError from freqtrade.commands import Arguments -from freqtrade.configuration import (Configuration, check_exchange, - remove_credentials, +from freqtrade.configuration import (Configuration, check_exchange, remove_credentials, validate_config_consistency) from freqtrade.configuration.config_validation import validate_config_schema -from freqtrade.configuration.deprecated_settings import ( - check_conflicting_settings, process_deprecated_setting, - process_temporary_deprecated_settings) +from freqtrade.configuration.deprecated_settings import (check_conflicting_settings, + process_deprecated_setting, + process_temporary_deprecated_settings) from freqtrade.configuration.load_config import load_config_file, log_config_error_range from freqtrade.constants import DEFAULT_DB_DRYRUN_URL, DEFAULT_DB_PROD_URL from freqtrade.exceptions import OperationalException from freqtrade.loggers import _set_loggers, setup_logging, setup_logging_pre from freqtrade.state import RunMode -from tests.conftest import (log_has, log_has_re, - patched_configuration_load_config_file) +from tests.conftest import log_has, log_has_re, patched_configuration_load_config_file @pytest.fixture(scope="function") diff --git a/tests/test_directory_operations.py b/tests/test_directory_operations.py index 71c91549f..a8058c514 100644 --- a/tests/test_directory_operations.py +++ b/tests/test_directory_operations.py @@ -4,8 +4,7 @@ from unittest.mock import MagicMock import pytest -from freqtrade.configuration.directory_operations import (copy_sample_files, - create_datadir, +from freqtrade.configuration.directory_operations import (copy_sample_files, create_datadir, create_userdata_dir) from freqtrade.exceptions import OperationalException from tests.conftest import log_has, log_has_re diff --git a/tests/test_freqtradebot.py b/tests/test_freqtradebot.py index ac6d3791a..2a1b0c3cc 100644 --- a/tests/test_freqtradebot.py +++ b/tests/test_freqtradebot.py @@ -10,21 +10,22 @@ from unittest.mock import ANY, MagicMock, PropertyMock import arrow import pytest -from freqtrade.constants import (CANCEL_REASON, MATH_CLOSE_PREC, - UNLIMITED_STAKE_AMOUNT) -from freqtrade.exceptions import (DependencyException, ExchangeError, - InvalidOrderException, OperationalException, - PricingError, TemporaryError) +from freqtrade.constants import CANCEL_REASON, MATH_CLOSE_PREC, UNLIMITED_STAKE_AMOUNT +from freqtrade.exceptions import (DependencyException, ExchangeError, InsufficientFundsError, + InvalidOrderException, OperationalException, PricingError, + TemporaryError) from freqtrade.freqtradebot import FreqtradeBot -from freqtrade.persistence import Trade +from freqtrade.persistence import Order, PairLock, Trade from freqtrade.rpc import RPCMessageType from freqtrade.state import RunMode, State from freqtrade.strategy.interface import SellCheckTuple, SellType from freqtrade.worker import Worker -from tests.conftest import (create_mock_trades, get_patched_freqtradebot, - get_patched_worker, log_has, log_has_re, - patch_edge, patch_exchange, patch_get_signal, +from tests.conftest import (create_mock_trades, get_patched_freqtradebot, get_patched_worker, + log_has, log_has_re, patch_edge, patch_exchange, patch_get_signal, patch_wallet, patch_whitelist) +from tests.conftest_trades import (MOCK_TRADE_COUNT, mock_order_1, mock_order_2, mock_order_2_sell, + mock_order_3, mock_order_3_sell, mock_order_4, + mock_order_5_stoploss, mock_order_6_sell) def patch_RPCManager(mocker) -> MagicMock: @@ -64,7 +65,7 @@ def test_process_stopped(mocker, default_conf) -> None: def test_bot_cleanup(mocker, default_conf, caplog) -> None: - mock_cleanup = mocker.patch('freqtrade.persistence.cleanup') + mock_cleanup = mocker.patch('freqtrade.freqtradebot.cleanup_db') coo_mock = mocker.patch('freqtrade.freqtradebot.FreqtradeBot.cancel_all_open_orders') freqtrade = get_patched_freqtradebot(mocker, default_conf) freqtrade.cleanup() @@ -170,7 +171,7 @@ def test_get_trade_stake_amount(default_conf, ticker, mocker) -> None: (True, 0.0027, 3, 0.5, [0.001, 0.001, 0.000673]), (True, 0.0022, 3, 1, [0.001, 0.001, 0.0]), ]) -def test_check_available_stake_amount(default_conf, ticker, mocker, fee, limit_buy_order, +def test_check_available_stake_amount(default_conf, ticker, mocker, fee, limit_buy_order_open, amend_last, wallet, max_open, lsamr, expected) -> None: patch_RPCManager(mocker) patch_exchange(mocker) @@ -178,7 +179,7 @@ def test_check_available_stake_amount(default_conf, ticker, mocker, fee, limit_b 'freqtrade.exchange.Exchange', fetch_ticker=ticker, get_balance=MagicMock(return_value=default_conf['stake_amount'] * 2), - buy=MagicMock(return_value={'id': limit_buy_order['id']}), + buy=MagicMock(return_value=limit_buy_order_open), get_fee=fee ) default_conf['dry_run_wallet'] = wallet @@ -191,6 +192,7 @@ def test_check_available_stake_amount(default_conf, ticker, mocker, fee, limit_b for i in range(0, max_open): if expected[i] is not None: + limit_buy_order_open['id'] = str(i) result = freqtrade.get_trade_stake_amount('ETH/BTC') assert pytest.approx(result) == expected[i] freqtrade.execute_buy('ETH/BTC', result) @@ -216,13 +218,13 @@ def test_get_trade_stake_amount_no_stake_amount(default_conf, mocker) -> None: (0.50, 0.0025), ]) def test_get_trade_stake_amount_unlimited_amount(default_conf, ticker, balance_ratio, result1, - limit_buy_order, fee, mocker) -> None: + limit_buy_order_open, fee, mocker) -> None: patch_RPCManager(mocker) patch_exchange(mocker) mocker.patch.multiple( 'freqtrade.exchange.Exchange', fetch_ticker=ticker, - buy=MagicMock(return_value={'id': limit_buy_order['id']}), + buy=MagicMock(return_value=limit_buy_order_open), get_fee=fee ) @@ -303,7 +305,6 @@ def test_edge_overrides_stoploss(limit_buy_order, fee, caplog, mocker, edge_conf 'ask': buy_price * 0.79, 'last': buy_price * 0.79 }), - buy=MagicMock(return_value={'id': limit_buy_order['id']}), get_fee=fee, ) ############################################# @@ -343,7 +344,6 @@ def test_edge_should_ignore_strategy_stoploss(limit_buy_order, fee, 'ask': buy_price * 0.85, 'last': buy_price * 0.85 }), - buy=MagicMock(return_value={'id': limit_buy_order['id']}), get_fee=fee, ) ############################################# @@ -362,8 +362,7 @@ def test_edge_should_ignore_strategy_stoploss(limit_buy_order, fee, assert freqtrade.handle_trade(trade) is False -def test_total_open_trades_stakes(mocker, default_conf, ticker, - limit_buy_order, fee) -> None: +def test_total_open_trades_stakes(mocker, default_conf, ticker, fee) -> None: patch_RPCManager(mocker) patch_exchange(mocker) default_conf['stake_amount'] = 0.00098751 @@ -371,7 +370,6 @@ def test_total_open_trades_stakes(mocker, default_conf, ticker, mocker.patch.multiple( 'freqtrade.exchange.Exchange', fetch_ticker=ticker, - buy=MagicMock(return_value={'id': limit_buy_order['id']}), get_fee=fee, ) freqtrade = FreqtradeBot(default_conf) @@ -534,7 +532,6 @@ def test_create_trade(default_conf, ticker, limit_buy_order, fee, mocker) -> Non mocker.patch.multiple( 'freqtrade.exchange.Exchange', fetch_ticker=ticker, - buy=MagicMock(return_value={'id': limit_buy_order['id']}), get_fee=fee, ) @@ -568,7 +565,6 @@ def test_create_trade_no_stake_amount(default_conf, ticker, limit_buy_order, mocker.patch.multiple( 'freqtrade.exchange.Exchange', fetch_ticker=ticker, - buy=MagicMock(return_value={'id': limit_buy_order['id']}), get_fee=fee, ) freqtrade = FreqtradeBot(default_conf) @@ -578,11 +574,11 @@ def test_create_trade_no_stake_amount(default_conf, ticker, limit_buy_order, freqtrade.create_trade('ETH/BTC') -def test_create_trade_minimal_amount(default_conf, ticker, limit_buy_order, +def test_create_trade_minimal_amount(default_conf, ticker, limit_buy_order_open, fee, mocker) -> None: patch_RPCManager(mocker) patch_exchange(mocker) - buy_mock = MagicMock(return_value={'id': limit_buy_order['id']}) + buy_mock = MagicMock(return_value=limit_buy_order_open) mocker.patch.multiple( 'freqtrade.exchange.Exchange', fetch_ticker=ticker, @@ -598,11 +594,11 @@ def test_create_trade_minimal_amount(default_conf, ticker, limit_buy_order, assert rate * amount <= default_conf['stake_amount'] -def test_create_trade_too_small_stake_amount(default_conf, ticker, limit_buy_order, +def test_create_trade_too_small_stake_amount(default_conf, ticker, limit_buy_order_open, fee, mocker) -> None: patch_RPCManager(mocker) patch_exchange(mocker) - buy_mock = MagicMock(return_value={'id': limit_buy_order['id']}) + buy_mock = MagicMock(return_value=limit_buy_order_open) mocker.patch.multiple( 'freqtrade.exchange.Exchange', fetch_ticker=ticker, @@ -618,14 +614,14 @@ def test_create_trade_too_small_stake_amount(default_conf, ticker, limit_buy_ord assert not freqtrade.create_trade('ETH/BTC') -def test_create_trade_limit_reached(default_conf, ticker, limit_buy_order, - fee, markets, mocker) -> None: +def test_create_trade_limit_reached(default_conf, ticker, limit_buy_order_open, + fee, mocker) -> None: patch_RPCManager(mocker) patch_exchange(mocker) mocker.patch.multiple( 'freqtrade.exchange.Exchange', fetch_ticker=ticker, - buy=MagicMock(return_value={'id': limit_buy_order['id']}), + buy=MagicMock(return_value=limit_buy_order_open), get_balance=MagicMock(return_value=default_conf['stake_amount']), get_fee=fee, ) @@ -639,14 +635,14 @@ def test_create_trade_limit_reached(default_conf, ticker, limit_buy_order, assert freqtrade.get_trade_stake_amount('ETH/BTC') == 0 -def test_enter_positions_no_pairs_left(default_conf, ticker, limit_buy_order, fee, +def test_enter_positions_no_pairs_left(default_conf, ticker, limit_buy_order_open, fee, mocker, caplog) -> None: patch_RPCManager(mocker) patch_exchange(mocker) mocker.patch.multiple( 'freqtrade.exchange.Exchange', fetch_ticker=ticker, - buy=MagicMock(return_value={'id': limit_buy_order['id']}), + buy=MagicMock(return_value=limit_buy_order_open), get_fee=fee, ) @@ -702,7 +698,7 @@ def test_create_trade_no_signal(default_conf, fee, mocker) -> None: @pytest.mark.parametrize("max_open", range(0, 5)) @pytest.mark.parametrize("tradable_balance_ratio,modifier", [(1.0, 1), (0.99, 0.8), (0.5, 0.5)]) -def test_create_trades_multiple_trades(default_conf, ticker, fee, mocker, +def test_create_trades_multiple_trades(default_conf, ticker, fee, mocker, limit_buy_order_open, max_open, tradable_balance_ratio, modifier) -> None: patch_RPCManager(mocker) patch_exchange(mocker) @@ -713,7 +709,7 @@ def test_create_trades_multiple_trades(default_conf, ticker, fee, mocker, mocker.patch.multiple( 'freqtrade.exchange.Exchange', fetch_ticker=ticker, - buy=MagicMock(return_value={'id': "12355555"}), + buy=MagicMock(return_value=limit_buy_order_open), get_fee=fee, ) freqtrade = FreqtradeBot(default_conf) @@ -727,14 +723,14 @@ def test_create_trades_multiple_trades(default_conf, ticker, fee, mocker, assert len(trades) == max(int(max_open * modifier), 0) -def test_create_trades_preopen(default_conf, ticker, fee, mocker) -> None: +def test_create_trades_preopen(default_conf, ticker, fee, mocker, limit_buy_order_open) -> None: patch_RPCManager(mocker) patch_exchange(mocker) default_conf['max_open_trades'] = 4 mocker.patch.multiple( 'freqtrade.exchange.Exchange', fetch_ticker=ticker, - buy=MagicMock(return_value={'id': "12355555"}), + buy=MagicMock(return_value=limit_buy_order_open), get_fee=fee, ) freqtrade = FreqtradeBot(default_conf) @@ -745,6 +741,8 @@ def test_create_trades_preopen(default_conf, ticker, fee, mocker) -> None: freqtrade.execute_buy('NEO/BTC', default_conf['stake_amount']) assert len(Trade.get_open_trades()) == 2 + # Change order_id for new orders + limit_buy_order_open['id'] = '123444' # Create 2 new trades using create_trades assert freqtrade.create_trade('ETH/BTC') @@ -754,14 +752,14 @@ def test_create_trades_preopen(default_conf, ticker, fee, mocker) -> None: assert len(trades) == 4 -def test_process_trade_creation(default_conf, ticker, limit_buy_order, +def test_process_trade_creation(default_conf, ticker, limit_buy_order, limit_buy_order_open, fee, mocker, caplog) -> None: patch_RPCManager(mocker) patch_exchange(mocker) mocker.patch.multiple( 'freqtrade.exchange.Exchange', fetch_ticker=ticker, - buy=MagicMock(return_value={'id': limit_buy_order['id']}), + buy=MagicMock(return_value=limit_buy_order_open), fetch_order=MagicMock(return_value=limit_buy_order), get_fee=fee, ) @@ -824,14 +822,14 @@ def test_process_operational_exception(default_conf, ticker, mocker) -> None: assert 'OperationalException' in msg_mock.call_args_list[-1][0][0]['status'] -def test_process_trade_handling(default_conf, ticker, limit_buy_order, fee, mocker) -> None: +def test_process_trade_handling(default_conf, ticker, limit_buy_order_open, fee, mocker) -> None: patch_RPCManager(mocker) patch_exchange(mocker) mocker.patch.multiple( 'freqtrade.exchange.Exchange', fetch_ticker=ticker, - buy=MagicMock(return_value={'id': limit_buy_order['id']}), - fetch_order=MagicMock(return_value=limit_buy_order), + buy=MagicMock(return_value=limit_buy_order_open), + fetch_order=MagicMock(return_value=limit_buy_order_open), get_fee=fee, ) freqtrade = FreqtradeBot(default_conf) @@ -971,7 +969,7 @@ def test_get_buy_rate(mocker, default_conf, caplog, side, ask, bid, assert not log_has("Using cached buy rate for ETH/BTC.", caplog) -def test_execute_buy(mocker, default_conf, fee, limit_buy_order) -> None: +def test_execute_buy(mocker, default_conf, fee, limit_buy_order, limit_buy_order_open) -> None: patch_RPCManager(mocker) patch_exchange(mocker) freqtrade = FreqtradeBot(default_conf) @@ -984,7 +982,7 @@ def test_execute_buy(mocker, default_conf, fee, limit_buy_order) -> None: get_buy_rate=buy_rate_mock, _get_min_pair_stake_amount=MagicMock(return_value=1) ) - buy_mm = MagicMock(return_value={'id': limit_buy_order['id']}) + buy_mm = MagicMock(return_value=limit_buy_order_open) mocker.patch.multiple( 'freqtrade.exchange.Exchange', fetch_ticker=MagicMock(return_value={ @@ -1003,6 +1001,7 @@ def test_execute_buy(mocker, default_conf, fee, limit_buy_order) -> None: assert freqtrade.strategy.confirm_trade_entry.call_count == 1 buy_rate_mock.reset_mock() + limit_buy_order_open['id'] = '22' freqtrade.strategy.confirm_trade_entry = MagicMock(return_value=True) assert freqtrade.execute_buy(pair, stake_amount) assert buy_rate_mock.call_count == 1 @@ -1018,9 +1017,10 @@ def test_execute_buy(mocker, default_conf, fee, limit_buy_order) -> None: trade = Trade.query.first() assert trade assert trade.is_open is True - assert trade.open_order_id == limit_buy_order['id'] + assert trade.open_order_id == '22' # Test calling with price + limit_buy_order_open['id'] = '33' fix_price = 0.06 assert freqtrade.execute_buy(pair, stake_amount, fix_price) # Make sure get_buy_rate wasn't called again @@ -1036,6 +1036,8 @@ def test_execute_buy(mocker, default_conf, fee, limit_buy_order) -> None: limit_buy_order['status'] = 'closed' limit_buy_order['price'] = 10 limit_buy_order['cost'] = 100 + limit_buy_order['id'] = '444' + mocker.patch('freqtrade.exchange.Exchange.buy', MagicMock(return_value=limit_buy_order)) assert freqtrade.execute_buy(pair, stake_amount) trade = Trade.query.all()[2] @@ -1051,11 +1053,12 @@ def test_execute_buy(mocker, default_conf, fee, limit_buy_order) -> None: limit_buy_order['remaining'] = 10.00 limit_buy_order['price'] = 0.5 limit_buy_order['cost'] = 40.495905365 + limit_buy_order['id'] = '555' mocker.patch('freqtrade.exchange.Exchange.buy', MagicMock(return_value=limit_buy_order)) assert freqtrade.execute_buy(pair, stake_amount) trade = Trade.query.all()[3] assert trade - assert trade.open_order_id is None + assert trade.open_order_id == '555' assert trade.open_rate == 0.5 assert trade.stake_amount == 40.495905365 @@ -1066,6 +1069,7 @@ def test_execute_buy(mocker, default_conf, fee, limit_buy_order) -> None: limit_buy_order['remaining'] = 90.99181073 limit_buy_order['price'] = 0.5 limit_buy_order['cost'] = 0.0 + limit_buy_order['id'] = '66' mocker.patch('freqtrade.exchange.Exchange.buy', MagicMock(return_value=limit_buy_order)) assert not freqtrade.execute_buy(pair, stake_amount) @@ -1093,9 +1097,11 @@ def test_execute_buy_confirm_error(mocker, default_conf, fee, limit_buy_order) - freqtrade.strategy.confirm_trade_entry = MagicMock(side_effect=ValueError) assert freqtrade.execute_buy(pair, stake_amount) + limit_buy_order['id'] = '222' freqtrade.strategy.confirm_trade_entry = MagicMock(side_effect=Exception) assert freqtrade.execute_buy(pair, stake_amount) + limit_buy_order['id'] = '2223' freqtrade.strategy.confirm_trade_entry = MagicMock(return_value=True) assert freqtrade.execute_buy(pair, stake_amount) @@ -1201,6 +1207,7 @@ def test_handle_stoploss_on_exchange(mocker, default_conf, fee, caplog, assert trade stoploss_order_hit = MagicMock(return_value={ + 'id': 100, 'status': 'closed', 'type': 'stop_loss_limit', 'price': 3, @@ -1209,7 +1216,7 @@ def test_handle_stoploss_on_exchange(mocker, default_conf, fee, caplog, }) mocker.patch('freqtrade.exchange.Exchange.fetch_stoploss_order', stoploss_order_hit) assert freqtrade.handle_stoploss_on_exchange(trade) is True - assert log_has('STOP_LOSS_LIMIT is hit for {}.'.format(trade), caplog) + assert log_has_re(r'STOP_LOSS_LIMIT is hit for Trade\(id=1, .*\)\.', caplog) assert trade.stoploss_order_id is None assert trade.is_open is False @@ -1258,7 +1265,7 @@ def test_handle_sle_cancel_cant_recreate(mocker, default_conf, fee, caplog, buy=MagicMock(return_value={'id': limit_buy_order['id']}), sell=MagicMock(return_value={'id': limit_sell_order['id']}), get_fee=fee, - fetch_stoploss_order=MagicMock(return_value={'status': 'canceled'}), + fetch_stoploss_order=MagicMock(return_value={'status': 'canceled', 'id': 100}), stoploss=MagicMock(side_effect=ExchangeError()), ) freqtrade = FreqtradeBot(default_conf) @@ -1278,7 +1285,7 @@ def test_handle_sle_cancel_cant_recreate(mocker, default_conf, fee, caplog, def test_create_stoploss_order_invalid_order(mocker, default_conf, caplog, fee, - limit_buy_order, limit_sell_order): + limit_buy_order_open, limit_sell_order): rpc_mock = patch_RPCManager(mocker) patch_exchange(mocker) sell_mock = MagicMock(return_value={'id': limit_sell_order['id']}) @@ -1289,7 +1296,7 @@ def test_create_stoploss_order_invalid_order(mocker, default_conf, caplog, fee, 'ask': 0.00001173, 'last': 0.00001172 }), - buy=MagicMock(return_value={'id': limit_buy_order['id']}), + buy=MagicMock(return_value=limit_buy_order_open), sell=sell_mock, get_fee=fee, fetch_order=MagicMock(return_value={'status': 'canceled'}), @@ -1320,7 +1327,46 @@ def test_create_stoploss_order_invalid_order(mocker, default_conf, caplog, fee, assert rpc_mock.call_args_list[1][0][0]['order_type'] == 'market' -def test_handle_stoploss_on_exchange_trailing(mocker, default_conf, fee, caplog, +def test_create_stoploss_order_insufficient_funds(mocker, default_conf, caplog, fee, + limit_buy_order_open, limit_sell_order): + sell_mock = MagicMock(return_value={'id': limit_sell_order['id']}) + freqtrade = get_patched_freqtradebot(mocker, default_conf) + + mock_insuf = mocker.patch('freqtrade.freqtradebot.FreqtradeBot.handle_insufficient_funds') + mocker.patch.multiple( + 'freqtrade.exchange.Exchange', + fetch_ticker=MagicMock(return_value={ + 'bid': 0.00001172, + 'ask': 0.00001173, + 'last': 0.00001172 + }), + buy=MagicMock(return_value=limit_buy_order_open), + sell=sell_mock, + get_fee=fee, + fetch_order=MagicMock(return_value={'status': 'canceled'}), + stoploss=MagicMock(side_effect=InsufficientFundsError()), + ) + patch_get_signal(freqtrade) + freqtrade.strategy.order_types['stoploss_on_exchange'] = True + + freqtrade.enter_positions() + trade = Trade.query.first() + caplog.clear() + freqtrade.create_stoploss_order(trade, 200) + # stoploss_orderid was empty before + assert trade.stoploss_order_id is None + assert mock_insuf.call_count == 1 + mock_insuf.reset_mock() + + trade.stoploss_order_id = 'stoploss_orderid' + freqtrade.create_stoploss_order(trade, 200) + # No change to stoploss-orderid + assert trade.stoploss_order_id == 'stoploss_orderid' + assert mock_insuf.call_count == 1 + + +@pytest.mark.usefixtures("init_persistence") +def test_handle_stoploss_on_exchange_trailing(mocker, default_conf, fee, limit_buy_order, limit_sell_order) -> None: # When trailing stoploss is set stoploss = MagicMock(return_value={'id': 13434334}) @@ -1389,7 +1435,7 @@ def test_handle_stoploss_on_exchange_trailing(mocker, default_conf, fee, caplog, })) cancel_order_mock = MagicMock() - stoploss_order_mock = MagicMock() + stoploss_order_mock = MagicMock(return_value={'id': 13434334}) mocker.patch('freqtrade.exchange.Exchange.cancel_stoploss_order', cancel_order_mock) mocker.patch('freqtrade.exchange.Exchange.stoploss', stoploss_order_mock) @@ -1691,8 +1737,10 @@ def test_update_trade_state(mocker, default_conf, limit_buy_order, caplog) -> No open_date=arrow.utcnow().datetime, amount=11, ) + assert not freqtrade.update_trade_state(trade, None) + assert log_has_re(r'Orderid for trade .* is empty.', caplog) # Add datetime explicitly since sqlalchemy defaults apply only once written to database - freqtrade.update_trade_state(trade) + freqtrade.update_trade_state(trade, '123') # Test amount not modified by fee-logic assert not log_has_re(r'Applying fee to .*', caplog) assert trade.open_order_id is None @@ -1702,14 +1750,14 @@ def test_update_trade_state(mocker, default_conf, limit_buy_order, caplog) -> No mocker.patch('freqtrade.freqtradebot.FreqtradeBot.get_real_amount', return_value=90.81) assert trade.amount != 90.81 # test amount modified by fee-logic - freqtrade.update_trade_state(trade) + freqtrade.update_trade_state(trade, '123') assert trade.amount == 90.81 assert trade.open_order_id is None trade.is_open = True trade.open_order_id = None # Assert we call handle_trade() if trade is feasible for execution - freqtrade.update_trade_state(trade) + freqtrade.update_trade_state(trade, '123') assert log_has_re('Found open order for.*', caplog) @@ -1734,7 +1782,7 @@ def test_update_trade_state_withorderdict(default_conf, trades_for_order, limit_ open_order_id="123456", is_open=True, ) - freqtrade.update_trade_state(trade, limit_buy_order) + freqtrade.update_trade_state(trade, '123456', limit_buy_order) assert trade.amount != amount assert trade.amount == limit_buy_order['amount'] @@ -1756,11 +1804,11 @@ def test_update_trade_state_withorderdict_rounding_fee(default_conf, trades_for_ open_rate=0.245441, fee_open=fee.return_value, fee_close=fee.return_value, - open_order_id="123456", + open_order_id='123456', is_open=True, open_date=arrow.utcnow().datetime, ) - freqtrade.update_trade_state(trade, limit_buy_order) + freqtrade.update_trade_state(trade, '123456', limit_buy_order) assert trade.amount != amount assert trade.amount == limit_buy_order['amount'] assert log_has_re(r'Applying fee on amount for .*', caplog) @@ -1780,7 +1828,7 @@ def test_update_trade_state_exception(mocker, default_conf, 'freqtrade.freqtradebot.FreqtradeBot.get_real_amount', side_effect=DependencyException() ) - freqtrade.update_trade_state(trade) + freqtrade.update_trade_state(trade, trade.open_order_id) assert log_has('Could not update trade amount: ', caplog) @@ -1795,12 +1843,13 @@ def test_update_trade_state_orderexception(mocker, default_conf, caplog) -> None # Test raise of OperationalException exception grm_mock = mocker.patch("freqtrade.freqtradebot.FreqtradeBot.get_real_amount", MagicMock()) - freqtrade.update_trade_state(trade) + freqtrade.update_trade_state(trade, trade.open_order_id) assert grm_mock.call_count == 0 assert log_has(f'Unable to fetch order {trade.open_order_id}: ', caplog) -def test_update_trade_state_sell(default_conf, trades_for_order, limit_sell_order, mocker): +def test_update_trade_state_sell(default_conf, trades_for_order, limit_sell_order_open, + limit_sell_order, mocker): mocker.patch('freqtrade.exchange.Exchange.get_trades_for_order', return_value=trades_for_order) # fetch_order should not be called!! mocker.patch('freqtrade.exchange.Exchange.fetch_order', MagicMock(side_effect=ValueError)) @@ -1823,14 +1872,20 @@ def test_update_trade_state_sell(default_conf, trades_for_order, limit_sell_orde open_order_id="123456", is_open=True, ) - freqtrade.update_trade_state(trade, limit_sell_order) + order = Order.parse_from_ccxt_object(limit_sell_order_open, 'LTC/ETH', 'sell') + trade.orders.append(order) + assert order.status == 'open' + freqtrade.update_trade_state(trade, trade.open_order_id, limit_sell_order) assert trade.amount == limit_sell_order['amount'] # Wallet needs to be updated after closing a limit-sell order to reenable buying assert wallet_mock.call_count == 1 assert not trade.is_open + # Order is updated by update_trade_state + assert order.status == 'closed' -def test_handle_trade(default_conf, limit_buy_order, limit_sell_order, fee, mocker) -> None: +def test_handle_trade(default_conf, limit_buy_order, limit_sell_order_open, limit_sell_order, + fee, mocker) -> None: patch_RPCManager(mocker) patch_exchange(mocker) mocker.patch.multiple( @@ -1840,8 +1895,8 @@ def test_handle_trade(default_conf, limit_buy_order, limit_sell_order, fee, mock 'ask': 0.00001173, 'last': 0.00001172 }), - buy=MagicMock(return_value={'id': limit_buy_order['id']}), - sell=MagicMock(return_value={'id': limit_sell_order['id']}), + buy=MagicMock(return_value=limit_buy_order), + sell=MagicMock(return_value=limit_sell_order_open), get_fee=fee, ) freqtrade = FreqtradeBot(default_conf) @@ -1870,13 +1925,14 @@ def test_handle_trade(default_conf, limit_buy_order, limit_sell_order, fee, mock assert trade.close_date is not None -def test_handle_overlapping_signals(default_conf, ticker, limit_buy_order, fee, mocker) -> None: +def test_handle_overlapping_signals(default_conf, ticker, limit_buy_order_open, + fee, mocker) -> None: patch_RPCManager(mocker) patch_exchange(mocker) mocker.patch.multiple( 'freqtrade.exchange.Exchange', fetch_ticker=ticker, - buy=MagicMock(return_value={'id': limit_buy_order['id']}), + buy=MagicMock(return_value=limit_buy_order_open), get_fee=fee, ) @@ -1921,7 +1977,7 @@ def test_handle_overlapping_signals(default_conf, ticker, limit_buy_order, fee, assert freqtrade.handle_trade(trades[0]) is True -def test_handle_trade_roi(default_conf, ticker, limit_buy_order, +def test_handle_trade_roi(default_conf, ticker, limit_buy_order_open, fee, mocker, caplog) -> None: caplog.set_level(logging.DEBUG) @@ -1929,7 +1985,7 @@ def test_handle_trade_roi(default_conf, ticker, limit_buy_order, mocker.patch.multiple( 'freqtrade.exchange.Exchange', fetch_ticker=ticker, - buy=MagicMock(return_value={'id': limit_buy_order['id']}), + buy=MagicMock(return_value=limit_buy_order_open), get_fee=fee, ) @@ -1954,14 +2010,14 @@ def test_handle_trade_roi(default_conf, ticker, limit_buy_order, def test_handle_trade_use_sell_signal( - default_conf, ticker, limit_buy_order, fee, mocker, caplog) -> None: + default_conf, ticker, limit_buy_order_open, fee, mocker, caplog) -> None: # use_sell_signal is True buy default caplog.set_level(logging.DEBUG) patch_RPCManager(mocker) mocker.patch.multiple( 'freqtrade.exchange.Exchange', fetch_ticker=ticker, - buy=MagicMock(return_value={'id': limit_buy_order['id']}), + buy=MagicMock(return_value=limit_buy_order_open), get_fee=fee, ) @@ -1982,14 +2038,14 @@ def test_handle_trade_use_sell_signal( caplog) -def test_close_trade(default_conf, ticker, limit_buy_order, limit_sell_order, +def test_close_trade(default_conf, ticker, limit_buy_order, limit_buy_order_open, limit_sell_order, fee, mocker) -> None: patch_RPCManager(mocker) patch_exchange(mocker) mocker.patch.multiple( 'freqtrade.exchange.Exchange', fetch_ticker=ticker, - buy=MagicMock(return_value={'id': limit_buy_order['id']}), + buy=MagicMock(return_value=limit_buy_order_open), get_fee=fee, ) freqtrade = FreqtradeBot(default_conf) @@ -2743,6 +2799,7 @@ def test_execute_sell_sloe_cancel_exception(mocker, default_conf, ticker, fee, c trade = Trade.query.first() Trade.session = MagicMock() + PairLock.session = MagicMock() freqtrade.config['dry_run'] = False trade.stoploss_order_id = "abcd" @@ -2807,7 +2864,7 @@ def test_execute_sell_with_stoploss_on_exchange(default_conf, ticker, fee, ticke def test_may_execute_sell_after_stoploss_on_exchange_hit(default_conf, ticker, fee, - limit_buy_order, mocker) -> None: + mocker) -> None: default_conf['exchange']['name'] = 'binance' rpc_mock = patch_RPCManager(mocker) patch_exchange(mocker) @@ -2929,7 +2986,36 @@ def test_execute_sell_market_order(default_conf, ticker, fee, } == last_msg -def test_sell_profit_only_enable_profit(default_conf, limit_buy_order, +def test_execute_sell_insufficient_funds_error(default_conf, ticker, fee, + ticker_sell_up, mocker) -> None: + freqtrade = get_patched_freqtradebot(mocker, default_conf) + mock_insuf = mocker.patch('freqtrade.freqtradebot.FreqtradeBot.handle_insufficient_funds') + mocker.patch.multiple( + 'freqtrade.exchange.Exchange', + fetch_ticker=ticker, + get_fee=fee, + sell=MagicMock(side_effect=InsufficientFundsError()) + ) + patch_get_signal(freqtrade) + + # Create some test data + freqtrade.enter_positions() + + trade = Trade.query.first() + assert trade + + # Increase the price and sell it + mocker.patch.multiple( + 'freqtrade.exchange.Exchange', + fetch_ticker=ticker_sell_up + ) + + assert not freqtrade.execute_sell(trade=trade, limit=ticker_sell_up()['bid'], + sell_reason=SellType.ROI) + assert mock_insuf.call_count == 1 + + +def test_sell_profit_only_enable_profit(default_conf, limit_buy_order, limit_buy_order_open, fee, mocker) -> None: patch_RPCManager(mocker) patch_exchange(mocker) @@ -2940,7 +3026,7 @@ def test_sell_profit_only_enable_profit(default_conf, limit_buy_order, 'ask': 0.00002173, 'last': 0.00002172 }), - buy=MagicMock(return_value={'id': limit_buy_order['id']}), + buy=MagicMock(return_value=limit_buy_order_open), get_fee=fee, ) default_conf['ask_strategy'] = { @@ -2961,7 +3047,7 @@ def test_sell_profit_only_enable_profit(default_conf, limit_buy_order, assert trade.sell_reason == SellType.SELL_SIGNAL.value -def test_sell_profit_only_disable_profit(default_conf, limit_buy_order, +def test_sell_profit_only_disable_profit(default_conf, limit_buy_order, limit_buy_order_open, fee, mocker) -> None: patch_RPCManager(mocker) patch_exchange(mocker) @@ -2972,7 +3058,7 @@ def test_sell_profit_only_disable_profit(default_conf, limit_buy_order, 'ask': 0.00002173, 'last': 0.00002172 }), - buy=MagicMock(return_value={'id': limit_buy_order['id']}), + buy=MagicMock(return_value=limit_buy_order_open), get_fee=fee, ) default_conf['ask_strategy'] = { @@ -2992,7 +3078,8 @@ def test_sell_profit_only_disable_profit(default_conf, limit_buy_order, assert trade.sell_reason == SellType.SELL_SIGNAL.value -def test_sell_profit_only_enable_loss(default_conf, limit_buy_order, fee, mocker) -> None: +def test_sell_profit_only_enable_loss(default_conf, limit_buy_order, limit_buy_order_open, + fee, mocker) -> None: patch_RPCManager(mocker) patch_exchange(mocker) mocker.patch.multiple( @@ -3002,7 +3089,7 @@ def test_sell_profit_only_enable_loss(default_conf, limit_buy_order, fee, mocker 'ask': 0.00000173, 'last': 0.00000172 }), - buy=MagicMock(return_value={'id': limit_buy_order['id']}), + buy=MagicMock(return_value=limit_buy_order_open), get_fee=fee, ) default_conf['ask_strategy'] = { @@ -3021,7 +3108,8 @@ def test_sell_profit_only_enable_loss(default_conf, limit_buy_order, fee, mocker assert freqtrade.handle_trade(trade) is False -def test_sell_profit_only_disable_loss(default_conf, limit_buy_order, fee, mocker) -> None: +def test_sell_profit_only_disable_loss(default_conf, limit_buy_order, limit_buy_order_open, + fee, mocker) -> None: patch_RPCManager(mocker) patch_exchange(mocker) mocker.patch.multiple( @@ -3031,7 +3119,7 @@ def test_sell_profit_only_disable_loss(default_conf, limit_buy_order, fee, mocke 'ask': 0.0000173, 'last': 0.0000172 }), - buy=MagicMock(return_value={'id': limit_buy_order['id']}), + buy=MagicMock(return_value=limit_buy_order_open), get_fee=fee, ) default_conf['ask_strategy'] = { @@ -3053,7 +3141,7 @@ def test_sell_profit_only_disable_loss(default_conf, limit_buy_order, fee, mocke assert trade.sell_reason == SellType.SELL_SIGNAL.value -def test_sell_not_enough_balance(default_conf, limit_buy_order, +def test_sell_not_enough_balance(default_conf, limit_buy_order, limit_buy_order_open, fee, mocker, caplog) -> None: patch_RPCManager(mocker) patch_exchange(mocker) @@ -3064,7 +3152,7 @@ def test_sell_not_enough_balance(default_conf, limit_buy_order, 'ask': 0.00002173, 'last': 0.00002172 }), - buy=MagicMock(return_value={'id': limit_buy_order['id']}), + buy=MagicMock(return_value=limit_buy_order_open), get_fee=fee, ) @@ -3162,7 +3250,6 @@ def test_locked_pairs(default_conf, ticker, fee, ticker_sell_down, mocker, caplo freqtrade.execute_sell(trade=trade, limit=ticker_sell_down()['bid'], sell_reason=SellType.STOP_LOSS) trade.close(ticker_sell_down()['bid']) - assert trade.pair in freqtrade.strategy._pair_locked_until assert freqtrade.strategy.is_pair_locked(trade.pair) # reinit - should buy other pair. @@ -3172,7 +3259,8 @@ def test_locked_pairs(default_conf, ticker, fee, ticker_sell_down, mocker, caplo assert log_has(f"Pair {trade.pair} is currently locked.", caplog) -def test_ignore_roi_if_buy_signal(default_conf, limit_buy_order, fee, mocker) -> None: +def test_ignore_roi_if_buy_signal(default_conf, limit_buy_order, limit_buy_order_open, + fee, mocker) -> None: patch_RPCManager(mocker) patch_exchange(mocker) mocker.patch.multiple( @@ -3182,7 +3270,7 @@ def test_ignore_roi_if_buy_signal(default_conf, limit_buy_order, fee, mocker) -> 'ask': 0.0000173, 'last': 0.0000172 }), - buy=MagicMock(return_value={'id': limit_buy_order['id']}), + buy=MagicMock(return_value=limit_buy_order_open), get_fee=fee, ) default_conf['ask_strategy'] = { @@ -3206,7 +3294,8 @@ def test_ignore_roi_if_buy_signal(default_conf, limit_buy_order, fee, mocker) -> assert trade.sell_reason == SellType.ROI.value -def test_trailing_stop_loss(default_conf, limit_buy_order, fee, caplog, mocker) -> None: +def test_trailing_stop_loss(default_conf, limit_buy_order_open, limit_buy_order, + fee, caplog, mocker) -> None: patch_RPCManager(mocker) patch_exchange(mocker) mocker.patch.multiple( @@ -3216,7 +3305,7 @@ def test_trailing_stop_loss(default_conf, limit_buy_order, fee, caplog, mocker) 'ask': 0.00001099, 'last': 0.00001099 }), - buy=MagicMock(return_value={'id': limit_buy_order['id']}), + buy=MagicMock(return_value=limit_buy_order_open), get_fee=fee, ) default_conf['trailing_stop'] = True @@ -3256,7 +3345,7 @@ def test_trailing_stop_loss(default_conf, limit_buy_order, fee, caplog, mocker) assert trade.sell_reason == SellType.TRAILING_STOP_LOSS.value -def test_trailing_stop_loss_positive(default_conf, limit_buy_order, fee, +def test_trailing_stop_loss_positive(default_conf, limit_buy_order, limit_buy_order_open, fee, caplog, mocker) -> None: buy_price = limit_buy_order['price'] patch_RPCManager(mocker) @@ -3268,7 +3357,7 @@ def test_trailing_stop_loss_positive(default_conf, limit_buy_order, fee, 'ask': buy_price - 0.000001, 'last': buy_price - 0.000001 }), - buy=MagicMock(return_value={'id': limit_buy_order['id']}), + buy=MagicMock(return_value=limit_buy_order_open), get_fee=fee, ) default_conf['trailing_stop'] = True @@ -3313,7 +3402,7 @@ def test_trailing_stop_loss_positive(default_conf, limit_buy_order, fee, f"initial stoploss was at 0.000010, trade opened at 0.000011", caplog) -def test_trailing_stop_loss_offset(default_conf, limit_buy_order, fee, +def test_trailing_stop_loss_offset(default_conf, limit_buy_order, limit_buy_order_open, fee, caplog, mocker) -> None: buy_price = limit_buy_order['price'] patch_RPCManager(mocker) @@ -3325,7 +3414,7 @@ def test_trailing_stop_loss_offset(default_conf, limit_buy_order, fee, 'ask': buy_price - 0.000001, 'last': buy_price - 0.000001 }), - buy=MagicMock(return_value={'id': limit_buy_order['id']}), + buy=MagicMock(return_value=limit_buy_order_open), get_fee=fee, ) patch_whitelist(mocker, default_conf) @@ -3371,7 +3460,7 @@ def test_trailing_stop_loss_offset(default_conf, limit_buy_order, fee, assert trade.sell_reason == SellType.TRAILING_STOP_LOSS.value -def test_tsl_only_offset_reached(default_conf, limit_buy_order, fee, +def test_tsl_only_offset_reached(default_conf, limit_buy_order, limit_buy_order_open, fee, caplog, mocker) -> None: buy_price = limit_buy_order['price'] # buy_price: 0.00001099 @@ -3385,7 +3474,7 @@ def test_tsl_only_offset_reached(default_conf, limit_buy_order, fee, 'ask': buy_price, 'last': buy_price }), - buy=MagicMock(return_value={'id': limit_buy_order['id']}), + buy=MagicMock(return_value=limit_buy_order_open), get_fee=fee, ) patch_whitelist(mocker, default_conf) @@ -3434,7 +3523,7 @@ def test_tsl_only_offset_reached(default_conf, limit_buy_order, fee, assert trade.stop_loss == 0.0000117705 -def test_disable_ignore_roi_if_buy_signal(default_conf, limit_buy_order, +def test_disable_ignore_roi_if_buy_signal(default_conf, limit_buy_order, limit_buy_order_open, fee, mocker) -> None: patch_RPCManager(mocker) patch_exchange(mocker) @@ -3445,7 +3534,7 @@ def test_disable_ignore_roi_if_buy_signal(default_conf, limit_buy_order, 'ask': 0.00000173, 'last': 0.00000172 }), - buy=MagicMock(return_value={'id': limit_buy_order['id']}), + buy=MagicMock(return_value=limit_buy_order_open), get_fee=fee, ) default_conf['ask_strategy'] = { @@ -3787,8 +3876,8 @@ def test_apply_fee_conditional(default_conf, fee, caplog, mocker, assert walletmock.call_count == 1 -def test_order_book_depth_of_market(default_conf, ticker, limit_buy_order, fee, mocker, - order_book_l2): +def test_order_book_depth_of_market(default_conf, ticker, limit_buy_order_open, limit_buy_order, + fee, mocker, order_book_l2): default_conf['bid_strategy']['check_depth_of_market']['enabled'] = True default_conf['bid_strategy']['check_depth_of_market']['bids_to_ask_delta'] = 0.1 patch_RPCManager(mocker) @@ -3797,7 +3886,7 @@ def test_order_book_depth_of_market(default_conf, ticker, limit_buy_order, fee, mocker.patch.multiple( 'freqtrade.exchange.Exchange', fetch_ticker=ticker, - buy=MagicMock(return_value={'id': limit_buy_order['id']}), + buy=MagicMock(return_value=limit_buy_order_open), get_fee=fee, ) @@ -3912,8 +4001,8 @@ def test_check_depth_of_market_buy(default_conf, mocker, order_book_l2) -> None: assert freqtrade._check_depth_of_market_buy('ETH/BTC', conf) is False -def test_order_book_ask_strategy(default_conf, limit_buy_order, limit_sell_order, - fee, mocker, order_book_l2, caplog) -> None: +def test_order_book_ask_strategy(default_conf, limit_buy_order_open, limit_buy_order, fee, + limit_sell_order_open, mocker, order_book_l2, caplog) -> None: """ test order book ask strategy """ @@ -3932,8 +4021,8 @@ def test_order_book_ask_strategy(default_conf, limit_buy_order, limit_sell_order 'ask': 0.00001173, 'last': 0.00001172 }), - buy=MagicMock(return_value={'id': limit_buy_order['id']}), - sell=MagicMock(return_value={'id': limit_sell_order['id']}), + buy=MagicMock(return_value=limit_buy_order_open), + sell=MagicMock(return_value=limit_sell_order_open), get_fee=fee, ) freqtrade = FreqtradeBot(default_conf) @@ -4078,7 +4167,7 @@ def test_startup_trade_reinit(default_conf, edge_conf, mocker): @pytest.mark.usefixtures("init_persistence") -def test_sync_wallet_dry_run(mocker, default_conf, ticker, fee, limit_buy_order, caplog): +def test_sync_wallet_dry_run(mocker, default_conf, ticker, fee, limit_buy_order_open, caplog): default_conf['dry_run'] = True # Initialize to 2 times stake amount default_conf['dry_run_wallet'] = 0.002 @@ -4088,7 +4177,7 @@ def test_sync_wallet_dry_run(mocker, default_conf, ticker, fee, limit_buy_order, mocker.patch.multiple( 'freqtrade.exchange.Exchange', fetch_ticker=ticker, - buy=MagicMock(return_value={'id': limit_buy_order['id']}), + buy=MagicMock(return_value=limit_buy_order_open), get_fee=fee, ) @@ -4113,21 +4202,22 @@ def test_sync_wallet_dry_run(mocker, default_conf, ticker, fee, limit_buy_order, def test_cancel_all_open_orders(mocker, default_conf, fee, limit_buy_order, limit_sell_order): default_conf['cancel_open_orders_on_exit'] = True mocker.patch('freqtrade.exchange.Exchange.fetch_order', - side_effect=[ExchangeError(), limit_sell_order, limit_buy_order]) + side_effect=[ + ExchangeError(), limit_sell_order, limit_buy_order, limit_sell_order]) buy_mock = mocker.patch('freqtrade.freqtradebot.FreqtradeBot.handle_cancel_buy') sell_mock = mocker.patch('freqtrade.freqtradebot.FreqtradeBot.handle_cancel_sell') freqtrade = get_patched_freqtradebot(mocker, default_conf) create_mock_trades(fee) trades = Trade.query.all() - assert len(trades) == 4 + assert len(trades) == MOCK_TRADE_COUNT freqtrade.cancel_all_open_orders() assert buy_mock.call_count == 1 - assert sell_mock.call_count == 1 + assert sell_mock.call_count == 2 @pytest.mark.usefixtures("init_persistence") -def test_check_for_open_trades(mocker, default_conf, fee, limit_buy_order, limit_sell_order): +def test_check_for_open_trades(mocker, default_conf, fee): freqtrade = get_patched_freqtradebot(mocker, default_conf) freqtrade.check_for_open_trades() @@ -4140,3 +4230,246 @@ def test_check_for_open_trades(mocker, default_conf, fee, limit_buy_order, limit freqtrade.check_for_open_trades() assert freqtrade.rpc.send_msg.call_count == 1 assert 'Handle these trades manually' in freqtrade.rpc.send_msg.call_args[0][0]['status'] + + +@pytest.mark.usefixtures("init_persistence") +def test_update_open_orders(mocker, default_conf, fee, caplog): + freqtrade = get_patched_freqtradebot(mocker, default_conf) + create_mock_trades(fee) + + freqtrade.update_open_orders() + assert log_has_re(r"Error updating Order .*", caplog) + caplog.clear() + + assert len(Order.get_open_orders()) == 3 + matching_buy_order = mock_order_4() + matching_buy_order.update({ + 'status': 'closed', + }) + mocker.patch('freqtrade.exchange.Exchange.fetch_order', return_value=matching_buy_order) + freqtrade.update_open_orders() + # Only stoploss and sell orders are kept open + assert len(Order.get_open_orders()) == 2 + + +@pytest.mark.usefixtures("init_persistence") +def test_update_closed_trades_without_assigned_fees(mocker, default_conf, fee): + freqtrade = get_patched_freqtradebot(mocker, default_conf) + + def patch_with_fee(order): + order.update({'fee': {'cost': 0.1, 'rate': 0.2, + 'currency': order['symbol'].split('/')[0]}}) + return order + + mocker.patch('freqtrade.exchange.Exchange.fetch_order_or_stoploss_order', + side_effect=[ + patch_with_fee(mock_order_2_sell()), + patch_with_fee(mock_order_3_sell()), + patch_with_fee(mock_order_1()), + patch_with_fee(mock_order_2()), + patch_with_fee(mock_order_3()), + patch_with_fee(mock_order_4()), + ] + ) + + create_mock_trades(fee) + trades = Trade.get_trades().all() + assert len(trades) == MOCK_TRADE_COUNT + for trade in trades: + assert trade.fee_open_cost is None + assert trade.fee_open_currency is None + assert trade.fee_close_cost is None + assert trade.fee_close_currency is None + + freqtrade.update_closed_trades_without_assigned_fees() + + trades = Trade.get_trades().all() + assert len(trades) == MOCK_TRADE_COUNT + + for trade in trades: + if trade.is_open: + # Exclude Trade 4 - as the order is still open. + if trade.select_order('buy', False): + assert trade.fee_open_cost is not None + assert trade.fee_open_currency is not None + else: + assert trade.fee_open_cost is None + assert trade.fee_open_currency is None + + else: + assert trade.fee_close_cost is not None + assert trade.fee_close_currency is not None + + +@pytest.mark.usefixtures("init_persistence") +def test_reupdate_buy_order_fees(mocker, default_conf, fee, caplog): + freqtrade = get_patched_freqtradebot(mocker, default_conf) + mock_uts = mocker.patch('freqtrade.freqtradebot.FreqtradeBot.update_trade_state') + + create_mock_trades(fee) + trades = Trade.get_trades().all() + + freqtrade.reupdate_buy_order_fees(trades[0]) + assert log_has_re(r"Trying to reupdate buy fees for .*", caplog) + assert mock_uts.call_count == 1 + assert mock_uts.call_args_list[0][0][0] == trades[0] + assert mock_uts.call_args_list[0][0][1] == mock_order_1()['id'] + assert log_has_re(r"Updating buy-fee on trade .* for order .*\.", caplog) + mock_uts.reset_mock() + caplog.clear() + + # Test with trade without orders + trade = Trade( + pair='XRP/ETH', + stake_amount=0.001, + fee_open=fee.return_value, + fee_close=fee.return_value, + open_date=arrow.utcnow().datetime, + is_open=True, + amount=20, + open_rate=0.01, + exchange='bittrex', + ) + Trade.session.add(trade) + + freqtrade.reupdate_buy_order_fees(trade) + assert log_has_re(r"Trying to reupdate buy fees for .*", caplog) + assert mock_uts.call_count == 0 + assert not log_has_re(r"Updating buy-fee on trade .* for order .*\.", caplog) + + +@pytest.mark.usefixtures("init_persistence") +def test_handle_insufficient_funds(mocker, default_conf, fee): + freqtrade = get_patched_freqtradebot(mocker, default_conf) + mock_rlo = mocker.patch('freqtrade.freqtradebot.FreqtradeBot.refind_lost_order') + mock_bof = mocker.patch('freqtrade.freqtradebot.FreqtradeBot.reupdate_buy_order_fees') + create_mock_trades(fee) + trades = Trade.get_trades().all() + + # Trade 0 has only a open buy order, no closed order + freqtrade.handle_insufficient_funds(trades[0]) + assert mock_rlo.call_count == 0 + assert mock_bof.call_count == 1 + + mock_rlo.reset_mock() + mock_bof.reset_mock() + + # Trade 1 has closed buy and sell orders + freqtrade.handle_insufficient_funds(trades[1]) + assert mock_rlo.call_count == 1 + assert mock_bof.call_count == 0 + + mock_rlo.reset_mock() + mock_bof.reset_mock() + + # Trade 2 has closed buy and sell orders + freqtrade.handle_insufficient_funds(trades[2]) + assert mock_rlo.call_count == 1 + assert mock_bof.call_count == 0 + + mock_rlo.reset_mock() + mock_bof.reset_mock() + + # Trade 3 has an opne buy order + freqtrade.handle_insufficient_funds(trades[3]) + assert mock_rlo.call_count == 0 + assert mock_bof.call_count == 1 + + +@pytest.mark.usefixtures("init_persistence") +def test_refind_lost_order(mocker, default_conf, fee, caplog): + caplog.set_level(logging.DEBUG) + freqtrade = get_patched_freqtradebot(mocker, default_conf) + mock_uts = mocker.patch('freqtrade.freqtradebot.FreqtradeBot.update_trade_state') + + mock_fo = mocker.patch('freqtrade.exchange.Exchange.fetch_order_or_stoploss_order', + return_value={'status': 'open'}) + + def reset_open_orders(trade): + trade.open_order_id = None + trade.stoploss_order_id = None + + create_mock_trades(fee) + trades = Trade.get_trades().all() + + caplog.clear() + + # No open order + trade = trades[0] + reset_open_orders(trade) + assert trade.open_order_id is None + assert trade.stoploss_order_id is None + + freqtrade.refind_lost_order(trade) + order = mock_order_1() + assert log_has_re(r"Order Order(.*order_id=" + order['id'] + ".*) is no longer open.", caplog) + assert mock_fo.call_count == 0 + assert mock_uts.call_count == 0 + # No change to orderid - as update_trade_state is mocked + assert trade.open_order_id is None + assert trade.stoploss_order_id is None + + caplog.clear() + mock_fo.reset_mock() + + # Open buy order + trade = trades[3] + reset_open_orders(trade) + assert trade.open_order_id is None + assert trade.stoploss_order_id is None + + freqtrade.refind_lost_order(trade) + order = mock_order_4() + assert log_has_re(r"Trying to refind Order\(.*", caplog) + assert mock_fo.call_count == 0 + assert mock_uts.call_count == 0 + # No change to orderid - as update_trade_state is mocked + assert trade.open_order_id is None + assert trade.stoploss_order_id is None + + caplog.clear() + mock_fo.reset_mock() + + # Open stoploss order + trade = trades[4] + reset_open_orders(trade) + assert trade.open_order_id is None + assert trade.stoploss_order_id is None + + freqtrade.refind_lost_order(trade) + order = mock_order_5_stoploss() + assert log_has_re(r"Trying to refind Order\(.*", caplog) + assert mock_fo.call_count == 1 + assert mock_uts.call_count == 1 + # stoploss_order_id is "refound" and added to the trade + assert trade.open_order_id is None + assert trade.stoploss_order_id is not None + + caplog.clear() + mock_fo.reset_mock() + mock_uts.reset_mock() + + # Open sell order + trade = trades[5] + reset_open_orders(trade) + assert trade.open_order_id is None + assert trade.stoploss_order_id is None + + freqtrade.refind_lost_order(trade) + order = mock_order_6_sell() + assert log_has_re(r"Trying to refind Order\(.*", caplog) + assert mock_fo.call_count == 1 + assert mock_uts.call_count == 1 + # sell-orderid is "refound" and added to the trade + assert trade.open_order_id == order['id'] + assert trade.stoploss_order_id is None + + caplog.clear() + + # Test error case + mock_fo = mocker.patch('freqtrade.exchange.Exchange.fetch_order_or_stoploss_order', + side_effect=ExchangeError()) + order = mock_order_5_stoploss() + + freqtrade.refind_lost_order(trades[4]) + assert log_has(f"Error updating {order['id']}.", caplog) diff --git a/tests/test_indicators.py b/tests/test_indicators.py index 2f9bdc0f9..8d02330a1 100644 --- a/tests/test_indicators.py +++ b/tests/test_indicators.py @@ -1,7 +1,8 @@ -import freqtrade.vendor.qtpylib.indicators as qtpylib import numpy as np import pandas as pd +import freqtrade.vendor.qtpylib.indicators as qtpylib + def test_crossed_numpy_types(): """ diff --git a/tests/test_main.py b/tests/test_main.py index dd0c877e8..f55aea336 100644 --- a/tests/test_main.py +++ b/tests/test_main.py @@ -1,11 +1,11 @@ # pragma pylint: disable=missing-docstring from copy import deepcopy +from pathlib import Path from unittest.mock import MagicMock, PropertyMock import pytest -from pathlib import Path from freqtrade.commands import Arguments from freqtrade.exceptions import FreqtradeException, OperationalException from freqtrade.freqtradebot import FreqtradeBot @@ -65,7 +65,7 @@ def test_main_fatal_exception(mocker, default_conf, caplog) -> None: mocker.patch('freqtrade.worker.Worker._worker', MagicMock(side_effect=Exception)) patched_configuration_load_config_file(mocker, default_conf) mocker.patch('freqtrade.freqtradebot.RPCManager', MagicMock()) - mocker.patch('freqtrade.freqtradebot.persistence.init', MagicMock()) + mocker.patch('freqtrade.freqtradebot.init_db', MagicMock()) args = ['trade', '-c', 'config.json.example'] @@ -83,7 +83,7 @@ def test_main_keyboard_interrupt(mocker, default_conf, caplog) -> None: patched_configuration_load_config_file(mocker, default_conf) mocker.patch('freqtrade.freqtradebot.RPCManager', MagicMock()) mocker.patch('freqtrade.wallets.Wallets.update', MagicMock()) - mocker.patch('freqtrade.freqtradebot.persistence.init', MagicMock()) + mocker.patch('freqtrade.freqtradebot.init_db', MagicMock()) args = ['trade', '-c', 'config.json.example'] @@ -104,7 +104,7 @@ def test_main_operational_exception(mocker, default_conf, caplog) -> None: patched_configuration_load_config_file(mocker, default_conf) mocker.patch('freqtrade.wallets.Wallets.update', MagicMock()) mocker.patch('freqtrade.freqtradebot.RPCManager', MagicMock()) - mocker.patch('freqtrade.freqtradebot.persistence.init', MagicMock()) + mocker.patch('freqtrade.freqtradebot.init_db', MagicMock()) args = ['trade', '-c', 'config.json.example'] @@ -155,7 +155,7 @@ def test_main_reload_config(mocker, default_conf, caplog) -> None: reconfigure_mock = mocker.patch('freqtrade.worker.Worker._reconfigure', MagicMock()) mocker.patch('freqtrade.freqtradebot.RPCManager', MagicMock()) - mocker.patch('freqtrade.freqtradebot.persistence.init', MagicMock()) + mocker.patch('freqtrade.freqtradebot.init_db', MagicMock()) args = Arguments(['trade', '-c', 'config.json.example']).get_parsed_arg() worker = Worker(args=args, config=default_conf) @@ -178,7 +178,7 @@ def test_reconfigure(mocker, default_conf) -> None: mocker.patch('freqtrade.wallets.Wallets.update', MagicMock()) patched_configuration_load_config_file(mocker, default_conf) mocker.patch('freqtrade.freqtradebot.RPCManager', MagicMock()) - mocker.patch('freqtrade.freqtradebot.persistence.init', MagicMock()) + mocker.patch('freqtrade.freqtradebot.init_db', MagicMock()) args = Arguments(['trade', '-c', 'config.json.example']).get_parsed_arg() worker = Worker(args=args, config=default_conf) diff --git a/tests/test_misc.py b/tests/test_misc.py index a185cbba4..6dcd9fbe5 100644 --- a/tests/test_misc.py +++ b/tests/test_misc.py @@ -7,9 +7,8 @@ from unittest.mock import MagicMock import pytest from freqtrade.data.converter import ohlcv_to_dataframe -from freqtrade.misc import (datesarray_to_datetimearray, file_dump_json, - file_load_json, format_ms_time, pair_to_filename, - plural, render_template, +from freqtrade.misc import (datesarray_to_datetimearray, file_dump_json, file_load_json, + format_ms_time, pair_to_filename, plural, render_template, render_template_with_fallback, safe_value_fallback, safe_value_fallback2, shorten_date) diff --git a/tests/test_persistence.py b/tests/test_persistence.py index 65c83e05b..243da3396 100644 --- a/tests/test_persistence.py +++ b/tests/test_persistence.py @@ -1,5 +1,6 @@ # pragma pylint: disable=missing-docstring, C0103 import logging +from datetime import datetime, timedelta, timezone from unittest.mock import MagicMock import arrow @@ -7,14 +8,14 @@ import pytest from sqlalchemy import create_engine from freqtrade import constants -from freqtrade.exceptions import OperationalException -from freqtrade.persistence import Trade, clean_dry_run_db, init -from tests.conftest import log_has, create_mock_trades +from freqtrade.exceptions import DependencyException, OperationalException +from freqtrade.persistence import Order, PairLock, Trade, clean_dry_run_db, init_db +from tests.conftest import create_mock_trades, log_has, log_has_re def test_init_create_session(default_conf): # Check if init create a session - init(default_conf['db_url'], default_conf['dry_run']) + init_db(default_conf['db_url'], default_conf['dry_run']) assert hasattr(Trade, 'session') assert 'scoped_session' in type(Trade.session).__name__ @@ -22,9 +23,9 @@ def test_init_create_session(default_conf): def test_init_custom_db_url(default_conf, mocker): # Update path to a value other than default, but still in-memory default_conf.update({'db_url': 'sqlite:///tmp/freqtrade2_test.sqlite'}) - create_engine_mock = mocker.patch('freqtrade.persistence.create_engine', MagicMock()) + create_engine_mock = mocker.patch('freqtrade.persistence.models.create_engine', MagicMock()) - init(default_conf['db_url'], default_conf['dry_run']) + init_db(default_conf['db_url'], default_conf['dry_run']) assert create_engine_mock.call_count == 1 assert create_engine_mock.mock_calls[0][1][0] == 'sqlite:///tmp/freqtrade2_test.sqlite' @@ -33,16 +34,16 @@ def test_init_invalid_db_url(default_conf): # Update path to a value other than default, but still in-memory default_conf.update({'db_url': 'unknown:///some.url'}) with pytest.raises(OperationalException, match=r'.*no valid database URL*'): - init(default_conf['db_url'], default_conf['dry_run']) + init_db(default_conf['db_url'], default_conf['dry_run']) def test_init_prod_db(default_conf, mocker): default_conf.update({'dry_run': False}) default_conf.update({'db_url': constants.DEFAULT_DB_PROD_URL}) - create_engine_mock = mocker.patch('freqtrade.persistence.create_engine', MagicMock()) + create_engine_mock = mocker.patch('freqtrade.persistence.models.create_engine', MagicMock()) - init(default_conf['db_url'], default_conf['dry_run']) + init_db(default_conf['db_url'], default_conf['dry_run']) assert create_engine_mock.call_count == 1 assert create_engine_mock.mock_calls[0][1][0] == 'sqlite:///tradesv3.sqlite' @@ -51,9 +52,9 @@ def test_init_dryrun_db(default_conf, mocker): default_conf.update({'dry_run': True}) default_conf.update({'db_url': constants.DEFAULT_DB_DRYRUN_URL}) - create_engine_mock = mocker.patch('freqtrade.persistence.create_engine', MagicMock()) + create_engine_mock = mocker.patch('freqtrade.persistence.models.create_engine', MagicMock()) - init(default_conf['db_url'], default_conf['dry_run']) + init_db(default_conf['db_url'], default_conf['dry_run']) assert create_engine_mock.call_count == 1 assert create_engine_mock.mock_calls[0][1][0] == 'sqlite:///tradesv3.dryrun.sqlite' @@ -93,6 +94,8 @@ def test_update_with_bittrex(limit_buy_order, limit_sell_order, fee, caplog): stake_amount=0.001, open_rate=0.01, amount=5, + is_open=True, + open_date=arrow.utcnow().datetime, fee_open=fee.return_value, fee_close=fee.return_value, exchange='bittrex', @@ -107,9 +110,9 @@ def test_update_with_bittrex(limit_buy_order, limit_sell_order, fee, caplog): assert trade.open_rate == 0.00001099 assert trade.close_profit is None assert trade.close_date is None - assert log_has("LIMIT_BUY has been fulfilled for Trade(id=2, " - "pair=ETH/BTC, amount=90.99181073, open_rate=0.00001099, open_since=closed).", - caplog) + assert log_has_re(r"LIMIT_BUY has been fulfilled for Trade\(id=2, " + r"pair=ETH/BTC, amount=90.99181073, open_rate=0.00001099, open_since=.*\).", + caplog) caplog.clear() trade.open_order_id = 'something' @@ -118,9 +121,9 @@ def test_update_with_bittrex(limit_buy_order, limit_sell_order, fee, caplog): assert trade.close_rate == 0.00001173 assert trade.close_profit == 0.06201058 assert trade.close_date is not None - assert log_has("LIMIT_SELL has been fulfilled for Trade(id=2, " - "pair=ETH/BTC, amount=90.99181073, open_rate=0.00001099, open_since=closed).", - caplog) + assert log_has_re(r"LIMIT_SELL has been fulfilled for Trade\(id=2, " + r"pair=ETH/BTC, amount=90.99181073, open_rate=0.00001099, open_since=.*\).", + caplog) @pytest.mark.usefixtures("init_persistence") @@ -131,8 +134,10 @@ def test_update_market_order(market_buy_order, market_sell_order, fee, caplog): stake_amount=0.001, amount=5, open_rate=0.01, + is_open=True, fee_open=fee.return_value, fee_close=fee.return_value, + open_date=arrow.utcnow().datetime, exchange='bittrex', ) @@ -142,20 +147,21 @@ def test_update_market_order(market_buy_order, market_sell_order, fee, caplog): assert trade.open_rate == 0.00004099 assert trade.close_profit is None assert trade.close_date is None - assert log_has("MARKET_BUY has been fulfilled for Trade(id=1, " - "pair=ETH/BTC, amount=91.99181073, open_rate=0.00004099, open_since=closed).", - caplog) + assert log_has_re(r"MARKET_BUY has been fulfilled for Trade\(id=1, " + r"pair=ETH/BTC, amount=91.99181073, open_rate=0.00004099, open_since=.*\).", + caplog) caplog.clear() + trade.is_open = True trade.open_order_id = 'something' trade.update(market_sell_order) assert trade.open_order_id is None assert trade.close_rate == 0.00004173 assert trade.close_profit == 0.01297561 assert trade.close_date is not None - assert log_has("MARKET_SELL has been fulfilled for Trade(id=1, " - "pair=ETH/BTC, amount=91.99181073, open_rate=0.00004099, open_since=closed).", - caplog) + assert log_has_re(r"MARKET_SELL has been fulfilled for Trade\(id=1, " + r"pair=ETH/BTC, amount=91.99181073, open_rate=0.00004099, open_since=.*\).", + caplog) @pytest.mark.usefixtures("init_persistence") @@ -184,6 +190,36 @@ def test_calc_open_close_trade_price(limit_buy_order, limit_sell_order, fee): assert trade.calc_profit_ratio() == 0.06201058 +@pytest.mark.usefixtures("init_persistence") +def test_trade_close(limit_buy_order, limit_sell_order, fee): + trade = Trade( + pair='ETH/BTC', + stake_amount=0.001, + open_rate=0.01, + amount=5, + is_open=True, + fee_open=fee.return_value, + fee_close=fee.return_value, + open_date=arrow.Arrow(2020, 2, 1, 15, 5, 1).datetime, + exchange='bittrex', + ) + assert trade.close_profit is None + assert trade.close_date is None + assert trade.is_open is True + trade.close(0.02) + assert trade.is_open is False + assert trade.close_profit == 0.99002494 + assert trade.close_date is not None + + new_date = arrow.Arrow(2020, 2, 2, 15, 6, 1).datetime, + assert trade.close_date != new_date + # Close should NOT update close_date if the trade has been closed already + assert trade.is_open is False + trade.close_date = new_date + trade.close(0.02) + assert trade.close_date == new_date + + @pytest.mark.usefixtures("init_persistence") def test_calc_close_trade_price_exception(limit_buy_order, fee): trade = Trade( @@ -421,9 +457,9 @@ def test_migrate_old(mocker, default_conf, fee): PRIMARY KEY (id), CHECK (is_open IN (0, 1)) );""" - insert_table_old = """INSERT INTO trades (exchange, pair, is_open, fee, + insert_table_old = """INSERT INTO trades (exchange, pair, is_open, open_order_id, fee, open_rate, stake_amount, amount, open_date) - VALUES ('BITTREX', 'BTC_ETC', 1, {fee}, + VALUES ('BITTREX', 'BTC_ETC', 1, '123123', {fee}, 0.00258580, {stake}, {amount}, '2017-11-28 12:44:24.000000') """.format(fee=fee.return_value, @@ -440,14 +476,14 @@ def test_migrate_old(mocker, default_conf, fee): amount=amount ) engine = create_engine('sqlite://') - mocker.patch('freqtrade.persistence.create_engine', lambda *args, **kwargs: engine) + mocker.patch('freqtrade.persistence.models.create_engine', lambda *args, **kwargs: engine) # Create table using the old format engine.execute(create_table_old) engine.execute(insert_table_old) engine.execute(insert_table_old2) # Run init to test migration - init(default_conf['db_url'], default_conf['dry_run']) + init_db(default_conf['db_url'], default_conf['dry_run']) assert len(Trade.query.filter(Trade.id == 1).all()) == 1 trade = Trade.query.filter(Trade.id == 1).first() @@ -481,6 +517,12 @@ def test_migrate_old(mocker, default_conf, fee): assert pytest.approx(trade.close_profit_abs) == trade.calc_profit() assert trade.sell_order_status is None + # Should've created one order + assert len(Order.query.all()) == 1 + order = Order.query.first() + assert order.order_id == '123123' + assert order.ft_order_side == 'buy' + def test_migrate_new(mocker, default_conf, fee, caplog): """ @@ -509,22 +551,25 @@ def test_migrate_new(mocker, default_conf, fee, caplog): sell_reason VARCHAR, strategy VARCHAR, ticker_interval INTEGER, + stoploss_order_id VARCHAR, PRIMARY KEY (id), CHECK (is_open IN (0, 1)) );""" insert_table_old = """INSERT INTO trades (exchange, pair, is_open, fee, open_rate, stake_amount, amount, open_date, - stop_loss, initial_stop_loss, max_rate, ticker_interval) + stop_loss, initial_stop_loss, max_rate, ticker_interval, + open_order_id, stoploss_order_id) VALUES ('binance', 'ETC/BTC', 1, {fee}, 0.00258580, {stake}, {amount}, '2019-11-28 12:44:24.000000', - 0.0, 0.0, 0.0, '5m') + 0.0, 0.0, 0.0, '5m', + 'buy_order', 'stop_order_id222') """.format(fee=fee.return_value, stake=default_conf.get("stake_amount"), amount=amount ) engine = create_engine('sqlite://') - mocker.patch('freqtrade.persistence.create_engine', lambda *args, **kwargs: engine) + mocker.patch('freqtrade.persistence.models.create_engine', lambda *args, **kwargs: engine) # Create table using the old format engine.execute(create_table_old) @@ -537,7 +582,7 @@ def test_migrate_new(mocker, default_conf, fee, caplog): engine.execute("create table trades_bak1 as select * from trades") # Run init to test migration - init(default_conf['db_url'], default_conf['dry_run']) + init_db(default_conf['db_url'], default_conf['dry_run']) assert len(Trade.query.filter(Trade.id == 1).all()) == 1 trade = Trade.query.filter(Trade.id == 1).first() @@ -558,14 +603,23 @@ def test_migrate_new(mocker, default_conf, fee, caplog): assert trade.sell_reason is None assert trade.strategy is None assert trade.timeframe == '5m' - assert trade.stoploss_order_id is None + assert trade.stoploss_order_id == 'stop_order_id222' assert trade.stoploss_last_update is None assert log_has("trying trades_bak1", caplog) assert log_has("trying trades_bak2", caplog) - assert log_has("Running database migration - backup available as trades_bak2", caplog) + assert log_has("Running database migration for trades - backup: trades_bak2", caplog) assert trade.open_trade_price == trade._calc_open_trade_price() assert trade.close_profit_abs is None + assert log_has("Moving open orders to Orders table.", caplog) + orders = Order.query.all() + assert len(orders) == 2 + assert orders[0].order_id == 'buy_order' + assert orders[0].ft_order_side == 'buy' + + assert orders[1].order_id == 'stop_order_id222' + assert orders[1].ft_order_side == 'stoploss' + def test_migrate_mid_state(mocker, default_conf, fee, caplog): """ @@ -601,14 +655,14 @@ def test_migrate_mid_state(mocker, default_conf, fee, caplog): amount=amount ) engine = create_engine('sqlite://') - mocker.patch('freqtrade.persistence.create_engine', lambda *args, **kwargs: engine) + mocker.patch('freqtrade.persistence.models.create_engine', lambda *args, **kwargs: engine) # Create table using the old format engine.execute(create_table_old) engine.execute(insert_table_old) # Run init to test migration - init(default_conf['db_url'], default_conf['dry_run']) + init_db(default_conf['db_url'], default_conf['dry_run']) assert len(Trade.query.filter(Trade.id == 1).all()) == 1 trade = Trade.query.filter(Trade.id == 1).first() @@ -626,7 +680,7 @@ def test_migrate_mid_state(mocker, default_conf, fee, caplog): assert trade.initial_stop_loss == 0.0 assert trade.open_trade_price == trade._calc_open_trade_price() assert log_has("trying trades_bak0", caplog) - assert log_has("Running database migration - backup available as trades_bak0", caplog) + assert log_has("Running database migration for trades - backup: trades_bak0", caplog) def test_adjust_stop_loss(fee): @@ -713,10 +767,10 @@ def test_adjust_min_max_rates(fee): @pytest.mark.usefixtures("init_persistence") -def test_get_open(default_conf, fee): +def test_get_open(fee): create_mock_trades(fee) - assert len(Trade.get_open_trades()) == 2 + assert len(Trade.get_open_trades()) == 4 @pytest.mark.usefixtures("init_persistence") @@ -851,7 +905,7 @@ def test_to_json(default_conf, fee): def test_stoploss_reinitialization(default_conf, fee): - init(default_conf['db_url']) + init_db(default_conf['db_url']) trade = Trade( pair='ETH/BTC', stake_amount=0.001, @@ -986,7 +1040,7 @@ def test_total_open_trades_stakes(fee): assert res == 0 create_mock_trades(fee) res = Trade.total_open_trades_stakes() - assert res == 0.002 + assert res == 0.004 @pytest.mark.usefixtures("init_persistence") @@ -1012,3 +1066,142 @@ def test_get_best_pair(fee): assert len(res) == 2 assert res[0] == 'XRP/BTC' assert res[1] == 0.01 + + +@pytest.mark.usefixtures("init_persistence") +def test_update_order_from_ccxt(): + # Most basic order return (only has orderid) + o = Order.parse_from_ccxt_object({'id': '1234'}, 'ETH/BTC', 'buy') + assert isinstance(o, Order) + assert o.ft_pair == 'ETH/BTC' + assert o.ft_order_side == 'buy' + assert o.order_id == '1234' + assert o.ft_is_open + ccxt_order = { + 'id': '1234', + 'side': 'buy', + 'symbol': 'ETH/BTC', + 'type': 'limit', + 'price': 1234.5, + 'amount': 20.0, + 'filled': 9, + 'remaining': 11, + 'status': 'open', + 'timestamp': 1599394315123 + } + o = Order.parse_from_ccxt_object(ccxt_order, 'ETH/BTC', 'buy') + assert isinstance(o, Order) + assert o.ft_pair == 'ETH/BTC' + assert o.ft_order_side == 'buy' + assert o.order_id == '1234' + assert o.order_type == 'limit' + assert o.price == 1234.5 + assert o.filled == 9 + assert o.remaining == 11 + assert o.order_date is not None + assert o.ft_is_open + assert o.order_filled_date is None + + # Order has been closed + ccxt_order.update({'filled': 20.0, 'remaining': 0.0, 'status': 'closed'}) + o.update_from_ccxt_object(ccxt_order) + + assert o.filled == 20.0 + assert o.remaining == 0.0 + assert not o.ft_is_open + assert o.order_filled_date is not None + + ccxt_order.update({'id': 'somethingelse'}) + with pytest.raises(DependencyException, match=r"Order-id's don't match"): + o.update_from_ccxt_object(ccxt_order) + + +@pytest.mark.usefixtures("init_persistence") +def test_select_order(fee): + create_mock_trades(fee) + + trades = Trade.get_trades().all() + + # Open buy order, no sell order + order = trades[0].select_order('buy', True) + assert order is None + order = trades[0].select_order('buy', False) + assert order is not None + order = trades[0].select_order('sell', None) + assert order is None + + # closed buy order, and open sell order + order = trades[1].select_order('buy', True) + assert order is None + order = trades[1].select_order('buy', False) + assert order is not None + order = trades[1].select_order('buy', None) + assert order is not None + order = trades[1].select_order('sell', True) + assert order is None + order = trades[1].select_order('sell', False) + assert order is not None + + # Has open buy order + order = trades[3].select_order('buy', True) + assert order is not None + order = trades[3].select_order('buy', False) + assert order is None + + # Open sell order + order = trades[4].select_order('buy', True) + assert order is None + order = trades[4].select_order('buy', False) + assert order is not None + + order = trades[4].select_order('sell', True) + assert order is not None + assert order.ft_order_side == 'stoploss' + order = trades[4].select_order('sell', False) + assert order is None + + +@pytest.mark.usefixtures("init_persistence") +def test_PairLock(default_conf): + # No lock should be present + assert len(PairLock.query.all()) == 0 + + pair = 'ETH/BTC' + assert not PairLock.is_pair_locked(pair) + PairLock.lock_pair(pair, arrow.utcnow().shift(minutes=4).datetime) + # ETH/BTC locked for 4 minutes + assert PairLock.is_pair_locked(pair) + + # XRP/BTC should not be locked now + pair = 'XRP/BTC' + assert not PairLock.is_pair_locked(pair) + # Unlocking a pair that's not locked should not raise an error + PairLock.unlock_pair(pair) + + PairLock.lock_pair(pair, arrow.utcnow().shift(minutes=4).datetime) + assert PairLock.is_pair_locked(pair) + + # Get both locks from above + locks = PairLock.get_pair_locks(None) + assert len(locks) == 2 + + # Unlock original pair + pair = 'ETH/BTC' + PairLock.unlock_pair(pair) + assert not PairLock.is_pair_locked(pair) + + pair = 'BTC/USDT' + # Lock until 14:30 + lock_time = datetime(2020, 5, 1, 14, 30, 0, tzinfo=timezone.utc) + PairLock.lock_pair(pair, lock_time) + + assert not PairLock.is_pair_locked(pair) + assert PairLock.is_pair_locked(pair, lock_time + timedelta(minutes=-10)) + assert PairLock.is_pair_locked(pair, lock_time + timedelta(minutes=-50)) + + # Should not be locked after time expired + assert not PairLock.is_pair_locked(pair, lock_time + timedelta(minutes=10)) + + locks = PairLock.get_pair_locks(pair, lock_time + timedelta(minutes=-2)) + assert len(locks) == 1 + assert 'PairLock' in str(locks[0]) diff --git a/tests/test_plotting.py b/tests/test_plotting.py index bcababbf1..401f66b60 100644 --- a/tests/test_plotting.py +++ b/tests/test_plotting.py @@ -13,13 +13,10 @@ from freqtrade.configuration import TimeRange from freqtrade.data import history from freqtrade.data.btanalysis import create_cum_profit, load_backtest_data from freqtrade.exceptions import OperationalException -from freqtrade.plot.plotting import (add_indicators, add_profit, - create_plotconfig, - generate_candlestick_graph, - generate_plot_filename, - generate_profit_graph, init_plotscript, - load_and_plot_trades, plot_profit, - plot_trades, store_plot_file) +from freqtrade.plot.plotting import (add_indicators, add_profit, create_plotconfig, + generate_candlestick_graph, generate_plot_filename, + generate_profit_graph, init_plotscript, load_and_plot_trades, + plot_profit, plot_trades, store_plot_file) from freqtrade.resolvers import StrategyResolver from tests.conftest import get_args, log_has, log_has_re, patch_exchange diff --git a/tests/test_talib.py b/tests/test_talib.py index 4effc129b..f526fdd4d 100644 --- a/tests/test_talib.py +++ b/tests/test_talib.py @@ -1,5 +1,5 @@ -import talib.abstract as ta import pandas as pd +import talib.abstract as ta def test_talib_bollingerbands_near_zero_values(): diff --git a/tests/test_wallets.py b/tests/test_wallets.py index 884470014..450dabc4d 100644 --- a/tests/test_wallets.py +++ b/tests/test_wallets.py @@ -19,12 +19,17 @@ def test_sync_wallet_at_boot(mocker, default_conf): "used": 0.0, "total": 0.260739 }, + "USDT": { + "free": 20, + "used": 20, + "total": 40 + }, }) ) freqtrade = get_patched_freqtradebot(mocker, default_conf) - assert len(freqtrade.wallets._wallets) == 2 + assert len(freqtrade.wallets._wallets) == 3 assert freqtrade.wallets._wallets['BNT'].free == 1.0 assert freqtrade.wallets._wallets['BNT'].used == 2.0 assert freqtrade.wallets._wallets['BNT'].total == 3.0 @@ -32,6 +37,7 @@ def test_sync_wallet_at_boot(mocker, default_conf): assert freqtrade.wallets._wallets['GAS'].used == 0.0 assert freqtrade.wallets._wallets['GAS'].total == 0.260739 assert freqtrade.wallets.get_free('BNT') == 1.0 + assert 'USDT' in freqtrade.wallets._wallets assert freqtrade.wallets._last_wallet_refresh > 0 mocker.patch.multiple( 'freqtrade.exchange.Exchange', @@ -51,6 +57,7 @@ def test_sync_wallet_at_boot(mocker, default_conf): freqtrade.wallets.update() + # USDT is missing from the 2nd result - so should not be in this either. assert len(freqtrade.wallets._wallets) == 2 assert freqtrade.wallets._wallets['BNT'].free == 1.2 assert freqtrade.wallets._wallets['BNT'].used == 1.9