Commit Graph

267 Commits

Author SHA1 Message Date
initrv
b707a6da35 Add ability to plot feature importance 2022-09-16 19:17:41 +03:00
robcaulk
3b97b3d5c8 fix mypy error for strategy 2022-09-15 00:56:51 +02:00
robcaulk
8aac644009 add tests. add guardrails. 2022-09-15 00:46:35 +02:00
robcaulk
81417cb795 Merge branch 'develop' into dev-merge-rl 2022-09-14 22:49:11 +02:00
Matthias
e4caccc353
Merge pull request #7367 from freqtrade/add-continual-learning
add continual learning to catboost and friends
2022-09-10 20:17:28 +02:00
Robert Caulk
2077f84f9b
Merge pull request #7363 from wagnercosta/fixHyperoptFreqai
Fix hyperopt - freqai
2022-09-10 17:02:01 +02:00
Matthias
5705b8759b Merge branch 'develop' into add-continual-learning 2022-09-10 08:24:04 +02:00
Matthias
97be3318f4 Use Datetime_format from constants 2022-09-09 20:31:02 +02:00
robcaulk
a826c0eb83 ensure signatures match, reduce verbosity 2022-09-09 19:30:53 +02:00
robcaulk
e51d352777 ensure pca is handling same DF as inlier 2022-09-07 19:11:54 +02:00
robcaulk
4c9ac6b7c0 add kwargs, reduce duplicated code 2022-09-07 18:58:55 +02:00
robcaulk
ea7bdac9ed ensure inlier metric can be combined with other cleaning methods 2022-09-07 18:45:16 +02:00
wagnercosta
6b7644029c
Merge branch 'freqtrade:develop' into fixHyperoptFreqai 2022-09-07 11:07:51 -03:00
Wagner Costa Santos
8d16dd804d hyperopt - freqai - docs and refactoring 2022-09-06 15:42:47 -03:00
Robert Caulk
d7585161b2
Merge branch 'develop' into add-inlier-metric 2022-09-06 20:40:21 +02:00
robcaulk
97077ba18a add continual learning to catboost and friends 2022-09-06 20:30:46 +02:00
robcaulk
4b28d0495f fix timestamping, move imports, add words to doc 2022-09-06 19:46:58 +02:00
Wagner Costa Santos
5b826150df fix hyperopt - freqai 2022-09-05 17:43:28 -03:00
robcaulk
78d01810ed reduce code redundancy, ensure live always gets the latest data 2022-09-05 00:12:00 +02:00
Wagner Costa Santos
d43ed186fc fix issue with freqai backtesting at slice dataframe 2022-09-05 00:12:00 +02:00
robcaulk
69b3fcfd32 Merge branch 'develop' into dev-merge-rl 2022-09-04 11:23:25 +02:00
Timothy Pogue
dae3b3d86a support shutting down freqai 2022-09-03 13:24:14 -06:00
Robert Caulk
63514b0443
Merge pull request #7325 from wagnercosta/develop
Improve Freqai backtesting performance
2022-09-03 19:28:04 +02:00
robcaulk
c9be66b5b6 increase test coverage for dk, improve function naming, extra cleaning 2022-09-03 15:52:29 +02:00
Matthias
80b5f035ab Remove typo in log message 2022-09-03 15:01:28 +02:00
robcaulk
599c1c79fb reorganized backtest utilities, test new functionality, improve/update doc 2022-09-03 14:00:01 +02:00
Wagner Costa Santos
af5460cebf Add option to keep models only in memory for backtest 2022-09-02 22:01:53 -03:00
Wagner Costa Santos
d6e115178a refactoring freqai backtesting - remove duplicate code 2022-09-01 07:09:23 -03:00
Wagner Costa Santos
44d3a9140d Merge branch 'develop' of https://github.com/wagnercosta/freqtrade into develop 2022-08-31 15:37:06 -03:00
Wagner Costa Santos
7bed0450d2 pr review - refactoring backtesting freqai 2022-08-31 15:36:29 -03:00
wagnercosta
3d4497467c
Merge branch 'freqtrade:develop' into develop 2022-08-31 14:47:06 -03:00
Wagner Costa Santos
df51da22ee refactoring freqai backtesting 2022-08-31 11:23:48 -03:00
Matthias
a88ffd2c9d Merge branch 'develop' into move_datadownload 2022-08-31 10:23:45 +00:00
robcaulk
7f52908e87 ensure the lost points are prepended for FreqUI 2022-08-30 18:55:58 +02:00
robcaulk
a58dd0bbf9 add noise feature, improve docstrings 2022-08-30 18:26:24 +02:00
robcaulk
b11742a4c5 integrate inlier metric function 2022-08-30 18:26:24 +02:00
robcaulk
7766350c15 refactor environment inheritence tree to accommodate flexible action types/counts. fix bug in train profit handling 2022-08-28 19:21:57 +02:00
robcaulk
f9a49744e6 add strategy to the freqai object 2022-08-24 13:00:55 +02:00
robcaulk
05ed1b544f Working base for reinforcement learning model 2022-08-24 13:00:40 +02:00
robcaulk
96d8882f1e Plug mem leak, add training timer 2022-08-22 13:30:30 +02:00
robcaulk
5155afb4e7 clean up code remnants 2022-08-17 15:22:48 +02:00
robcaulk
0c34104e45 extract download-data from freqai to prepare for future async changes 2022-08-17 15:18:44 +02:00
Matthias
c7e1719215 Fix interface import sorting 2022-08-15 06:53:02 +02:00
robcaulk
8961b8d560 merge in inference timer and historic predictions handling improvements. 2022-08-14 20:31:15 +02:00
robcaulk
ad846cdb76 fix lock bug, update docstring 2022-08-14 20:24:29 +02:00
Matthias
464d99808f Update doc table formatting 2022-08-14 18:22:01 +02:00
Matthias
a29402ddde Rename and move analysis_lock to data_kitchen 2022-08-14 17:23:14 +02:00
robcaulk
31be707cc8 clean up code, add docstrings 2022-08-13 19:40:24 +02:00
robcaulk
3e38c1b0bd take dynamic sized tail off historic_predictions as return dataframe to strategy. 2022-08-13 19:40:24 +02:00
robcaulk
7d448fd4ac allow fit_live_predictions access to current pair 2022-08-13 19:40:24 +02:00
robcaulk
1f192be43b avoid denormalizing labels twice 2022-08-13 19:40:24 +02:00
robcaulk
b1b76a2dbe debug classifier with predict proba 2022-08-13 19:40:24 +02:00
robcaulk
2cae3c42e6 remove trade database analyzer, clean up a bit 2022-08-10 17:43:06 +02:00
Matthias
77b3b8a134 Use main exchange instead of creating a separate instance. 2022-08-08 18:34:11 +00:00
robcaulk
eb8bde37c1 Add lightgbm classifier, add classifier check test, fix classifier bug. 2022-08-06 17:51:21 +02:00
Matthias
47a30047eb Fix typo 2022-08-06 15:03:10 +02:00
Matthias
b16f57cb0d Minor stylistic fixes 2022-08-06 14:55:46 +02:00
Robert Caulk
c172ce1011 improve flexibility of user defined prediction dataframe 2022-08-06 13:51:19 +02:00
Robert Caulk
07763d0d4f add classifier, improve model naming scheme 2022-08-06 08:33:55 +02:00
robcaulk
a42a060ab5 fix DB once and for all. Make DBSCAN more efficient and robust. 2022-08-05 21:29:03 +02:00
robcaulk
db1d367941 fix bug associated to fit_live_predictions_candles 2022-08-05 13:46:20 +02:00
Robert Caulk
51a6b4289f improve DBSCAN performance for subsequent trainings 2022-08-04 17:41:58 +02:00
robcaulk
29225e4baf add DBSCAN outlier detection feature, add supporting documentation 2022-08-04 12:15:16 +02:00
robcaulk
eae82d0222 fix bug with database url during backtesting. comment out example trade db analysis. 2022-08-03 16:17:57 +02:00
robcaulk
95d3009a95 give user ability to analyze live trade dataframe inside custom prediction model. Add documentation to explain new functionality 2022-08-02 20:14:02 +02:00
robcaulk
f22b140782 fix backtesting bug, undo move of label stat calc, fix example strat exit logic 2022-07-29 17:27:35 +02:00
robcaulk
c84d54b35e Fix typing issue, avoid using .get() when unnecessary, convert to fstrings 2022-07-29 08:12:50 +02:00
Matthias
efbd83c56d Small type and typo fixes in freqai_interface 2022-07-28 07:24:30 +02:00
Matthias
a2a0d35a24 Update missing typehints 2022-07-28 07:07:40 +02:00
robcaulk
324e54c015 fix possible memory leak associated with Catboost Pool object 2022-07-26 17:29:29 +02:00
robcaulk
ad25a4cb56 reduce number of pair_dict lookups, remove coin_first 2022-07-26 15:58:40 +02:00
robcaulk
e213d0ad55 isolate data_drawer functions from data_kitchen, accommodate tests, add new test 2022-07-26 10:24:14 +02:00
Robert Caulk
897f18a8c8 ensure proper integer type casting for timestamps. Add check test for backtesting subdaily time periods 2022-07-25 15:07:09 +02:00
Robert Caulk
7b105532d1 fix mypy error and add test for principal component analysis 2022-07-25 11:46:59 +02:00
Robert Caulk
c9d46a5237 finish bringing follow_mode up to date 2022-07-25 09:24:40 +02:00
Matthias
1885deb632 More docstring changes 2022-07-24 16:54:39 +02:00
Matthias
70b7a254af Update some areas to use default docstring formatting 2022-07-24 16:51:48 +02:00
robcaulk
f3d46613ee move prediction denormalization into datakitchen. remove duplicate associated code. avoid normalization/denormalization for string dtypes. 2022-07-23 17:14:33 +02:00
robcaulk
c2d6a0e891 add record of contribution to doc and source 2022-07-23 13:04:06 +02:00
robcaulk
5559e605b8 small PR conversation resolutions 2022-07-22 17:46:14 +02:00
robcaulk
afcb0bec00 clean up obsolete comments, move remove_features_from_df to datakitchen 2022-07-22 12:29:20 +02:00
robcaulk
8033e0bf23 add counter to backtesting log so users know how many more pairs and how many more models will need to be trained 2022-07-21 13:22:12 +02:00
robcaulk
e694ea1cfd make sure backtesting gets the populated indicators with slimmed down user strat 2022-07-21 12:48:09 +02:00
robcaulk
8f86b0deaa *breaking change* simplify user strat by consolidating feature loops into backend 2022-07-21 12:24:22 +02:00
robcaulk
e7337728bf add separator in folder name just incase an asset ends in an integer 2022-07-21 11:25:28 +02:00
robcaulk
921a7ef216 add requirements-freqai.txt to builds 2022-07-20 15:51:25 +02:00
robcaulk
d43c146676 add more tests for datakitchen functionalities, add regression tests for freqai_interface train/backtest 2022-07-20 12:56:46 +02:00
robcaulk
ef409dd345 Add ground work for TensorFlow models, add protections from common mistakes 2022-07-12 18:09:17 +02:00
Robert Caulk
8ce6b18318 start collecting indefinite history of predictions. Allow user to generate statistics on these predictions. Direct FreqAI to save these to disk and reload them if available. 2022-07-11 22:01:48 +02:00
Robert Caulk
607455919e Change config parameter names to improve clarity and consistency throughout the code (!!breaking change, please check discord support channel for migration instructions or review templates/FreqaiExampleStrategy.py config_examples/config_freqai_futures.example.json file changes!!) 2022-07-10 12:35:44 +02:00
robcaulk
d9acdc9767 remove excess, increase no model warning clarity 2022-07-06 18:20:21 +02:00
robcaulk
4cac67fd66 Catch infrequent issue associated with grabbing first candle 2022-07-05 12:43:33 +02:00
robcaulk
bd3a6ba2fe update backtesting to handle new output framework 2022-07-03 17:34:44 +02:00
robcaulk
ffb39a5029 black formatting on freqai files 2022-07-03 10:59:38 +02:00
robcaulk
106131ff0f Rehaul organization of return values 2022-07-02 18:09:38 +02:00
robcaulk
93e1410ed9 first step toward cleaning output and enabling multimodel training per pair 2022-07-01 14:00:30 +02:00
robcaulk
6c7d02cb18 expose nu in the SVM outlier detection via svm_nu in config 2022-06-28 15:12:25 +02:00
robcaulk
7dfbd432d1 fix config saving bug, assign priorities to pairs in scanning, sleep the scanning loop to reduce CPU usage 2022-06-27 11:35:33 +02:00
robcaulk
68bafa9517 archive config to the model folder, filter out features before returning dataframe to strategy (to alleviate frequi issues)` 2022-06-26 23:03:48 +02:00
robcaulk
b35c64b6c0 fix bug in backtest, typo in example strat 2022-06-19 16:41:09 +02:00
robcaulk
3599d18ff6 fix bug in follow_mode, thanks @blood4rc 2022-06-18 12:05:28 +02:00
robcaulk
6da7a98857 add docstrings to new functions, remove superceded code 2022-06-17 16:16:23 +02:00
robcaulk
f631ae911b add model expiration feature, fix bug in DI return values 2022-06-17 14:55:40 +02:00
robcaulk
0b0688a91e ensure scanning purges models 2022-06-16 16:12:38 +02:00
robcaulk
c5de0c49e4 first functional scanning commit 2022-06-16 00:24:18 +02:00
robcaulk
4d472a0ea1 merging datarehaul into scanning branch 2022-06-16 00:22:49 +02:00
robcaulk
66800c7a45 ensure newest candles are always appended 2022-06-07 20:24:23 +02:00
robcaulk
15d049cffe detect if upper tf candles are new or not, append if so. Correct the epoch for candle update check 2022-06-07 19:49:20 +02:00
robcaulk
4b26b6aaec add lock to any historic data access 2022-06-07 00:54:18 +02:00
robcaulk
d6b8801f41 fix follower bug 2022-06-05 04:40:58 +02:00
robcaulk
16b4a5b71f rehaul of backend data management - increasing performance by holding history in memory, reducing load on the ratelimit by only pinging exchange once per candle. Improve code readability. 2022-06-03 15:19:46 +02:00
robcaulk
ace951bf7e another queue bug fix for fresh starts 2022-06-02 13:28:06 +02:00
robcaulk
833d25bda0 Save data after queue reorg 2022-06-01 08:05:38 +02:00
robcaulk
0b0dd8dd80 Force high priority pair to be trained before anything else. 2022-06-01 07:55:05 +02:00
robcaulk
7523ed825e automatically detect maximum required data based on user fed indicators (to avoid NaNs in dataset for rolling indicators), add new config parameter for backtesting to let users increase their startup_candles to accommodate high timeframe indicators, add docs to explain all. Add new feature for automatic indicator duplication according to user defined intervals (exhibited in example strat and configs now). 2022-05-31 18:42:27 +02:00
robcaulk
9b3b08a2bb let follower purge old model files 2022-05-31 15:37:38 +02:00
robcaulk
bac4ced382 Ensure follower predictions are persistent and uniquely stored 2022-05-31 14:35:04 +02:00
robcaulk
70adf55643 Automatically detect and change follower data_path to accommodate remote systems 2022-05-31 12:35:09 +02:00
robcaulk
0306f5ca13 Add autopurge feature so that FreqAI cleans up after itself when it no longer needs old models on disk 2022-05-31 11:58:21 +02:00
robcaulk
45f4f0f603 ensure follower sends back null arrays in case leader hasnt created a model file yet 2022-05-31 01:48:48 +02:00
robcaulk
606f18e5c1 Add follow_mode feature so that secondary bots can be launched with the same identifier and load models trained by the leader 2022-05-30 21:35:48 +02:00
robcaulk
5b4c649d43 detect variable sized dataframes coming from strat, adjust our stored/returned data accordingly 2022-05-30 13:55:46 +02:00
robcaulk
e229902381 fix bug in previous commit 2022-05-30 12:48:22 +02:00
robcaulk
a20651efd8 Increase performance by only predicting on most recent candle instead of full strat provided dataframe. Collect predictions and store them so that we can feed true predictions back to strategy (so that frequi isnt updating historic predictions based on newly trained models). 2022-05-30 11:37:05 +02:00
robcaulk
d59eac3321 revert a79032b 2022-05-29 21:33:38 +02:00
robcaulk
a79032bf75 fixing bug in training queue 2022-05-29 20:19:32 +02:00
robcaulk
3f72263278 allow pairs deeper in the queue to get trained if the higher priority pairs dont need training 2022-05-29 20:02:43 +02:00
robcaulk
cc6cae47ec allow pairs deeper in the queue to get trained if the higher priority pairs dont need training 2022-05-29 19:49:43 +02:00
robcaulk
4eb4753e20 allow subdaily retraining for backtesting 2022-05-29 17:44:35 +02:00
robcaulk
ce365eb9e3 improve example strat so that it has dynamic buy and sell logic according to original prediction 2022-05-29 14:45:46 +02:00
robcaulk
4eb29c8810 Dont reset pair priority if it doesnt successfully train 2022-05-28 18:34:26 +02:00
robcaulk
83dd453723 catch errors occuring on background thread, and make sure to keep the ball rolling. Improve pair retraining queue. 2022-05-28 18:26:19 +02:00
robcaulk
e54614fa2f remove remnants of single threaded version, ensure pair queue priority is checked before retraining 2022-05-28 14:55:07 +02:00
robcaulk
2a4d1e2d64 fix bug in setting new timerange for retraining 2022-05-28 12:23:26 +02:00
robcaulk
7870a86e9a fix live retraining bug 2022-05-28 11:38:57 +02:00
robcaulk
0bf915054d handle key check correctly 2022-05-28 11:22:32 +02:00
robcaulk
c5a16e91fb throw user error if user tries to load models but feeds the wrong features (while using PCA) 2022-05-28 11:11:41 +02:00
robcaulk
6193205012 fix bug for target_mean/std array merging in backtesting 2022-05-26 21:07:50 +02:00
robcaulk
b79d4e8876 Allow user to go live and start from pretrained models (after a completed backtest) by simply reusing the identifier config parameter while dry/live. 2022-05-25 14:40:32 +02:00
robcaulk
7486d9d9e2 proper validation of freqai config parameters 2022-05-25 12:37:25 +02:00
robcaulk
7ff3258607 remove assertions, log error if user has not assigned freqai in config, fix stratify bug 2022-05-25 11:43:45 +02:00
robcaulk
35bed842cb cleanup, add clarity to comments and docstrings 2022-05-25 11:31:03 +02:00
robcaulk
58b5abbaa6 improve multithreaded training queue system 2022-05-24 15:28:38 +02:00
robcaulk
255d35976e add priority metadata to pairs to avoid a sync of train time + train period 2022-05-24 12:58:53 +02:00
robcaulk
059c285425 paying closer attention to managing live retraining on separate thread without affecting prediction of other coins on master thread 2022-05-24 12:01:01 +02:00
robcaulk
b0d2d13eb1 improve data persistence/mapping for live/dry. This accommodates quick reloads after crash and handles multi-pair cleanly 2022-05-23 21:05:05 +02:00
robcaulk
e1c068ca66 add config asserts, use .get method with default values for optional functionality, move data_cleaning_* to freqai_interface (away from user custom pred model) since it is controlled by config params. 2022-05-23 12:07:09 +02:00
robcaulk
ee3cdd0ffe more cleanup 2022-05-23 09:55:58 +02:00
robcaulk
af0cc21af9 Enable hourly/minute retraining in live/dry. Suppress catboost folder output. Update config + constants + docs to reflect updates. 2022-05-23 00:06:26 +02:00
robcaulk
42d95af829 Aggregated commit. Adding support vector machine for outlier detection, improve user interface to dry/live, better standardization, fix various other bugs 2022-05-22 17:51:49 +02:00