adelorenzo/stable
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Reorganise multipage doc

Browse Source
This commit is contained in:
th0rntwig
2022-09-17 17:43:39 +02:00
parent ede282392f
commit c210d6614c
10 changed files with 359 additions and 392 deletions
Download Patch File Download Diff File Expand all files Collapse all files

60
docs/freqai.md
View File

@@ -1,27 +1,29 @@
![freqai-logo](assets/freqai_doc_logo.svg)
# FreqAI
# FreqAI
##Introduction
FreqAI is a module designed to automate a variety of tasks associated with training a predictive machine learning model to generate market forecasts given a set of input features.
Features include:
* **Self-adaptive retraining**: retrain models during [live deployments](freqai-running.md#running-the-model-live) to self-adapt to the market in an unsupervised manner.
* **Rapid feature engineering**: create large rich [feature sets](freqai-feature-engineering.md#feature-engineering) (10k+ features) based on simple user-created strategies.
* **High performance**: adaptive retraining occurs on a separate thread (or on GPU if available) from inferencing and bot trade operations. Newest models and data are kept in memory for rapid inferencing.
* **Realistic backtesting**: emulate self-adaptive retraining with a [backtesting module](freqai-running.md#backtesting) that automates past retraining.
* **Extensibility**: use the generalized and robust architecture for incorporating any [machine learning library/method](freqai-configuration.md#building-a-custom-prediction-model) available in Python. Eight examples are currently available, including classifiers, regressors, and a convolutional neural network.
* **Smart outlier removal**: remove outliers from training and prediction data sets using a variety of [outlier detection techniques](freqai-feature-engineering.md#outlier-removal).
* **Crash resilience**: store model to disk to make reloading from a crash fast and easy, and [purge obsolete files](freqai-data-handling.md#purging-old-model-data) for sustained dry/live runs.
* **Automatic data normalization**: [normalize the data](freqai-feature-engineering.md#feature-normalization) in a smart and statistically safe way.
* **Automatic data download**: compute the data download timerange and update historic data (in live deployments).
* **Cleaning of incoming data**: handle NaNs safely before training and prediction.
* **Dimensionality reduction**: reduce the size of the training data via [Principal Component Analysis](freqai-feature-engineering.md#reducing-data-dimensionality-with-principal-component-analysis).
* **Deploying bot fleets**: set one bot to train models while a fleet of [follower bots](freqai-running.md#setting-up-a-follower) inference the models and handle trades.
* **Self-adaptive retraining** - Retrain models during [live deployments](freqai-running.md#running-the-model-live) to self-adapt to the market in a supervised manner
* **Rapid feature engineering** - Create large rich [feature sets](freqai-feature-engineering.md#feature-engineering) (10k+ features) based on simple user-created strategies
* **High performance** - Threading allows for adaptive model retraining on a separate thread (or on GPU if available) from model inferencing (prediction) and bot trade operations. Newest models and data are kept in RAM for rapid inferencing
* **Realistic backtesting** - Emulate self-adaptive training on historic data with a [backtesting module](freqai-running.md#backtesting) that automates retraining
* **Extensibility** - The generalized and robust architecture allows for incorporating any [machine learning library/method](freqai-configuration.md#using-different-prediction-models) available in Python. Eight examples are currently available, including classifiers, regressors, and a convolutional neural network
* **Smart outlier removal** - Remove outliers from training and prediction data sets using a variety of [outlier detection techniques](freqai-outlier-detection.md)
* **Crash resilience** - Store trained models to disk to make reloading from a crash fast and easy, and [purge obsolete files](freqai-data-handling.md#purging-old-model-data) for sustained dry/live runs
* **Automatic data normalization** - [Normalize the data](freqai-feature-engineering.md#feature-normalization) in a smart and statistically safe way
* **Automatic data download** - Compute timeranges for data downloads and update historic data (in live deployments)
* **Cleaning of incoming data** - Handle NaNs safely before training and model inferencing
* **Dimensionality reduction** - Reduce the size of the training data via [Principal Component Analysis](freqai-feature-engineering.md#data-dimensionality-reduction-with-principal-component-analysis)
* **Deploying bot fleets** - Set one bot to train models while a fleet of [follower bots](freqai-running.md#setting-up-a-follower) inference the models and handle trades
## Quick start
The easiest way to quickly test FreqAI is to run it in dry mode with the following command
The easiest way to quickly test FreqAI is to run it in dry mode with the following command:
```bash
freqtrade trade --config config_examples/config_freqai.example.json --strategy FreqaiExampleStrategy --freqaimodel LightGBMRegressor --strategy-path freqtrade/templates
@@ -29,29 +31,31 @@ freqtrade trade --config config_examples/config_freqai.example.json --strategy F
The user will see the boot-up process of automatic data downloading, followed by simultaneous training and trading.
The example strategy, example prediction model, and example config can be found in
An example strategy, prediction model, and config to use as a starting points can be found in
`freqtrade/templates/FreqaiExampleStrategy.py`, `freqtrade/freqai/prediction_models/LightGBMRegressor.py`, and
`config_examples/config_freqai.example.json`, respectively.
## General approach
The user provides FreqAI with a set of custom *base* indicators (the same way as in a typical Freqtrade strategy) as well as target values (*labels*). FreqAI trains a model to predict the target values based on the input of custom indicators, for each pair in the whitelist. These models are consistently retrained to adapt to market conditions. FreqAI offers the ability to both backtest strategies (emulating reality with periodic retraining) and deploy dry/live runs. In dry/live conditions, FreqAI can be set to constant retraining in a background thread in an effort to keep models as up to date as possible.
The user provides FreqAI with a set of custom *base indicators* (the same way as in a [typical Freqtrade strategy](strategy-customization.md)) as well as target values (*labels*). For each pair in the whitelist, FreqAI trains a model to predict the target values based on the input of custom indicators. The models are then consistently retrained, with a frequency set by the user, to adapt to market conditions. FreqAI offers the ability to both backtest strategies (emulating reality with periodic retraining on historic data) and deploy dry/live runs. In dry/live conditions, FreqAI can be set to constant retraining in a background thread to keep models as up to date as possible.
An overview of the algorithm is shown below, explaining the data processing pipeline and the model usage.
An overview of the algorithm, explaining the data processing pipeline and model usage, is shown below.
![freqai-algo](assets/freqai_algo.jpg)
### Important machine learning vocabulary
**Features** - the quantities with which a model is trained. All features for a single candle is stored as a vector. In FreqAI, the user builds the feature sets from anything they can construct in the strategy.
**Features** - the parameters, based on historic data, on which a model is trained. All features for a single candle is stored as a vector. In FreqAI, the user builds a feature data sets from anything they can construct in the strategy.
**Labels** - the target values that a model is trained toward. Each set of features is associated with a single label that is defined by the user within the strategy. These labels intentionally look into the future, and are not available to the model during dry/live/backtesting.
**Labels** - the target values that a model is trained toward. Each feature vector is associated with a single label that is defined by the user within the strategy. These labels intentionally look into the future, and are not available to the model during dry/live/backtesting.
**Training** - the process of feeding individual feature sets, composed of historic data, with associated labels into the model with the goal of matching input feature sets to associated labels.
**Training** - the process of "teaching" the model to match the feature sets to the associated labels. Different types of models "learn" in different ways. More information about the different models can be found [here](freqai-configuration.md#using-different-prediction-models).
**Train data** - a subset of the historic data that is fed to the model during training. This data directly influences weight connections in the model.
**Train data** - a subset of the feature data set that is fed to the model during training. This data directly influences weight connections in the model.
**Test data** - a subset of the historic data that is used to evaluate the performance of the model after training. This data does not influence nodal weights within the model.
**Test data** - a subset of the feature data set that is used to evaluate the performance of the model after training. This data does not influence nodal weights within the model.
**Inferencing** - the process of feeding a trained model new data on which it will make a prediction.
## Install prerequisites
@@ -62,26 +66,26 @@ pip install -r requirements-freqai.txt
```
!!! Note
Catboost will not be installed on arm devices (raspberry, Mac M1, ARM based VPS, ...), since Catboost does not provide wheels for this platform.
Catboost will not be installed on arm devices (raspberry, Mac M1, ARM based VPS, ...), since it does not provide wheels for this platform.
### Usage with docker
For docker users, a dedicated tag with freqAI dependencies is available as `:freqai`. As such - you can replace the image line in your docker-compose file with `image: freqtradeorg/freqtrade:develop_freqai`. This image contains the regular freqAI dependencies. Similar to native installs, Catboost will not be available on ARM based devices.
For docker users, a dedicated tag with FreqAI dependencies is available as `:freqai`. As such - you can replace the image line in your docker-compose file with `image: freqtradeorg/freqtrade:develop_freqai`. This image contains the regular FreqAI dependencies. Similar to native installs, Catboost will not be available on ARM based devices.
## Credits
FreqAI was developed by a group of individuals who all contributed specific skillsets to the project.
FreqAI is developed by a group of individuals who all contribute specific skillsets to the project.
Conception and software development:
Robert Caulk @robcaulk
Theoretical brainstorming, data analysis:
Theoretical brainstorming and data analysis:
Elin Törnquist @th0rntwig
Code review, software architecture brainstorming:
Code review and software architecture brainstorming:
@xmatthias
Development:
Software development:
Wagner Costa @wagnercosta
Beta testing and bug reporting: