stable/docs/data-analysis.md

128 lines
5.2 KiB
Markdown
Raw Normal View History

# Analyzing bot data with Jupyter notebooks
2019-06-22 14:18:22 +00:00
You can analyze the results of backtests and trading history easily using Jupyter notebooks. Sample notebooks are located at `user_data/notebooks/` after initializing the user directory with `freqtrade create-userdir --userdir user_data`.
2019-06-22 14:18:22 +00:00
2020-09-20 13:12:30 +00:00
## Quick start with docker
Freqtrade provides a docker-compose file which starts up a jupyter lab server.
You can run this server using the following command: `docker-compose -f docker/docker-compose-jupyter.yml up`
This will create a dockercontainer running jupyter lab, which will be accessible using `https://127.0.0.1:8888/lab`.
Please use the link that's printed in the console after startup for simplified login.
For more information, Please visit the [Data analysis with Docker](docker_quickstart.md#data-analayis-using-docker-compose) section.
### Pro tips
2019-06-22 14:18:22 +00:00
2019-08-11 16:52:37 +00:00
* See [jupyter.org](https://jupyter.org/documentation) for usage instructions.
2019-08-20 04:47:10 +00:00
* Don't forget to start a Jupyter notebook server from within your conda or venv environment or use [nb_conda_kernels](https://github.com/Anaconda-Platform/nb_conda_kernels)*
2020-09-20 13:12:30 +00:00
* Copy the example notebook before use so your changes don't get overwritten with the next freqtrade update.
2019-08-11 16:52:37 +00:00
2019-12-29 18:35:42 +00:00
### Using virtual environment with system-wide Jupyter installation
Sometimes it can be desired to use a system-wide installation of Jupyter notebook, and use a jupyter kernel from the virtual environment.
This prevents you from installing the full jupyter suite multiple times per system, and provides an easy way to switch between tasks (freqtrade / other analytics tasks).
For this to work, first activate your virtual environment and run the following commands:
``` bash
# Activate virtual environment
source .env/bin/activate
pip install ipykernel
ipython kernel install --user --name=freqtrade
# Restart jupyter (lab / notebook)
# select kernel "freqtrade" in the notebook
```
!!! Note
2019-12-29 18:35:42 +00:00
This section is provided for completeness, the Freqtrade Team won't provide full support for problems with this setup and will recommend to install Jupyter in the virtual environment directly, as that is the easiest way to get jupyter notebooks up and running. For help with this setup please refer to the [Project Jupyter](https://jupyter.org/) [documentation](https://jupyter.org/documentation) or [help channels](https://jupyter.org/community).
2020-09-20 13:12:30 +00:00
!!! Warning
Some tasks don't work especially well in notebooks. For example, anything using asynchronous execution is a problem for Jupyter. Also, freqtrade's primary entry point is the shell cli, so using pure python in a notebook bypasses arguments that provide required objects and parameters to helper functions. You may need to set those values or create expected objects manually.
2019-08-11 16:52:37 +00:00
## Recommended workflow
2019-08-11 16:52:37 +00:00
| Task | Tool |
--- | ---
Bot operations | CLI
2019-08-20 04:47:10 +00:00
Repetitive tasks | Shell scripts
Data analysis & visualization | Notebook
2019-08-11 16:52:37 +00:00
1. Use the CLI to
2019-08-11 16:52:37 +00:00
* download historical data
* run a backtest
* run with real-time data
* export results
2019-08-11 16:52:37 +00:00
1. Collect these actions in shell scripts
2019-08-11 16:52:37 +00:00
* save complicated commands with arguments
* execute multi-step operations
2019-08-20 04:47:10 +00:00
* automate testing strategies and preparing data for analysis
2019-08-11 16:52:37 +00:00
1. Use a notebook to
2019-08-20 04:47:10 +00:00
* visualize data
* mangle and plot to generate insights
2019-08-11 16:52:37 +00:00
## Example utility snippets
2019-08-11 16:52:37 +00:00
### Change directory to root
2019-08-11 16:52:37 +00:00
2019-08-20 04:47:10 +00:00
Jupyter notebooks execute from the notebook directory. The following snippet searches for the project root, so relative paths remain consistent.
2019-08-11 16:52:37 +00:00
```python
import os
from pathlib import Path
2019-08-20 04:47:10 +00:00
# Change directory
# Modify this cell to insure that the output shows the correct path.
# Define all paths relative to the project root shown in the cell output
2019-08-11 16:52:37 +00:00
project_root = "somedir/freqtrade"
i=0
try:
os.chdir(project_root)
2019-08-11 16:52:37 +00:00
assert Path('LICENSE').is_file()
except:
while i<4 and (not Path('LICENSE').is_file()):
os.chdir(Path(Path.cwd(), '../'))
i+=1
project_root = Path.cwd()
print(Path.cwd())
```
2019-08-10 17:55:33 +00:00
### Load multiple configuration files
2019-09-13 05:08:42 +00:00
This option can be useful to inspect the results of passing in multiple configs.
This will also run through the whole Configuration initialization, so the configuration is completely initialized to be passed to other methods.
2019-08-10 17:55:33 +00:00
``` python
2019-08-20 04:47:10 +00:00
import json
2019-08-10 17:55:33 +00:00
from freqtrade.configuration import Configuration
2019-08-20 04:47:10 +00:00
# Load config from multiple files
2019-08-10 17:55:33 +00:00
config = Configuration.from_files(["config1.json", "config2.json"])
2019-08-11 16:52:37 +00:00
# Show the config in memory
print(json.dumps(config['original_config'], indent=2))
2019-09-13 05:08:42 +00:00
```
For Interactive environments, have an additional configuration specifying `user_data_dir` and pass this in last, so you don't have to change directories while running the bot.
2019-09-14 07:54:40 +00:00
Best avoid relative paths, since this starts at the storage location of the jupyter notebook, unless the directory is changed.
2019-09-13 05:08:42 +00:00
``` json
{
"user_data_dir": "~/.freqtrade/"
}
2019-08-11 16:52:37 +00:00
```
### Further Data analysis documentation
2019-08-11 16:52:37 +00:00
* [Strategy debugging](strategy_analysis_example.md) - also available as Jupyter notebook (`user_data/notebooks/strategy_analysis_example.ipynb`)
* [Plotting](plotting.md)
* [Tag Analysis](advanced-backtesting.md)
2019-06-22 14:18:22 +00:00
2019-06-24 15:20:41 +00:00
Feel free to submit an issue or Pull Request enhancing this document if you would like to share ideas on how to best analyze the data.