Commit Graph

127 Commits

Author SHA1 Message Date
robcaulk
d9acdc9767 remove excess, increase no model warning clarity 2022-07-06 18:20:21 +02:00
robcaulk
4cac67fd66 Catch infrequent issue associated with grabbing first candle 2022-07-05 12:43:33 +02:00
robcaulk
bd3a6ba2fe update backtesting to handle new output framework 2022-07-03 17:34:44 +02:00
robcaulk
ffb39a5029 black formatting on freqai files 2022-07-03 10:59:38 +02:00
robcaulk
106131ff0f Rehaul organization of return values 2022-07-02 18:09:38 +02:00
robcaulk
93e1410ed9 first step toward cleaning output and enabling multimodel training per pair 2022-07-01 14:00:30 +02:00
robcaulk
6c7d02cb18 expose nu in the SVM outlier detection via svm_nu in config 2022-06-28 15:12:25 +02:00
robcaulk
7dfbd432d1 fix config saving bug, assign priorities to pairs in scanning, sleep the scanning loop to reduce CPU usage 2022-06-27 11:35:33 +02:00
robcaulk
68bafa9517 archive config to the model folder, filter out features before returning dataframe to strategy (to alleviate frequi issues)` 2022-06-26 23:03:48 +02:00
robcaulk
b35c64b6c0 fix bug in backtest, typo in example strat 2022-06-19 16:41:09 +02:00
robcaulk
3599d18ff6 fix bug in follow_mode, thanks @blood4rc 2022-06-18 12:05:28 +02:00
robcaulk
6da7a98857 add docstrings to new functions, remove superceded code 2022-06-17 16:16:23 +02:00
robcaulk
f631ae911b add model expiration feature, fix bug in DI return values 2022-06-17 14:55:40 +02:00
robcaulk
0b0688a91e ensure scanning purges models 2022-06-16 16:12:38 +02:00
robcaulk
c5de0c49e4 first functional scanning commit 2022-06-16 00:24:18 +02:00
robcaulk
4d472a0ea1 merging datarehaul into scanning branch 2022-06-16 00:22:49 +02:00
robcaulk
66800c7a45 ensure newest candles are always appended 2022-06-07 20:24:23 +02:00
robcaulk
15d049cffe detect if upper tf candles are new or not, append if so. Correct the epoch for candle update check 2022-06-07 19:49:20 +02:00
robcaulk
4b26b6aaec add lock to any historic data access 2022-06-07 00:54:18 +02:00
robcaulk
d6b8801f41 fix follower bug 2022-06-05 04:40:58 +02:00
robcaulk
16b4a5b71f rehaul of backend data management - increasing performance by holding history in memory, reducing load on the ratelimit by only pinging exchange once per candle. Improve code readability. 2022-06-03 15:19:46 +02:00
robcaulk
ace951bf7e another queue bug fix for fresh starts 2022-06-02 13:28:06 +02:00
robcaulk
833d25bda0 Save data after queue reorg 2022-06-01 08:05:38 +02:00
robcaulk
0b0dd8dd80 Force high priority pair to be trained before anything else. 2022-06-01 07:55:05 +02:00
robcaulk
7523ed825e automatically detect maximum required data based on user fed indicators (to avoid NaNs in dataset for rolling indicators), add new config parameter for backtesting to let users increase their startup_candles to accommodate high timeframe indicators, add docs to explain all. Add new feature for automatic indicator duplication according to user defined intervals (exhibited in example strat and configs now). 2022-05-31 18:42:27 +02:00
robcaulk
9b3b08a2bb let follower purge old model files 2022-05-31 15:37:38 +02:00
robcaulk
bac4ced382 Ensure follower predictions are persistent and uniquely stored 2022-05-31 14:35:04 +02:00
robcaulk
70adf55643 Automatically detect and change follower data_path to accommodate remote systems 2022-05-31 12:35:09 +02:00
robcaulk
0306f5ca13 Add autopurge feature so that FreqAI cleans up after itself when it no longer needs old models on disk 2022-05-31 11:58:21 +02:00
robcaulk
45f4f0f603 ensure follower sends back null arrays in case leader hasnt created a model file yet 2022-05-31 01:48:48 +02:00
robcaulk
606f18e5c1 Add follow_mode feature so that secondary bots can be launched with the same identifier and load models trained by the leader 2022-05-30 21:35:48 +02:00
robcaulk
5b4c649d43 detect variable sized dataframes coming from strat, adjust our stored/returned data accordingly 2022-05-30 13:55:46 +02:00
robcaulk
e229902381 fix bug in previous commit 2022-05-30 12:48:22 +02:00
robcaulk
a20651efd8 Increase performance by only predicting on most recent candle instead of full strat provided dataframe. Collect predictions and store them so that we can feed true predictions back to strategy (so that frequi isnt updating historic predictions based on newly trained models). 2022-05-30 11:37:05 +02:00
robcaulk
d59eac3321 revert a79032b 2022-05-29 21:33:38 +02:00
robcaulk
a79032bf75 fixing bug in training queue 2022-05-29 20:19:32 +02:00
robcaulk
3f72263278 allow pairs deeper in the queue to get trained if the higher priority pairs dont need training 2022-05-29 20:02:43 +02:00
robcaulk
cc6cae47ec allow pairs deeper in the queue to get trained if the higher priority pairs dont need training 2022-05-29 19:49:43 +02:00
robcaulk
4eb4753e20 allow subdaily retraining for backtesting 2022-05-29 17:44:35 +02:00
robcaulk
ce365eb9e3 improve example strat so that it has dynamic buy and sell logic according to original prediction 2022-05-29 14:45:46 +02:00
robcaulk
4eb29c8810 Dont reset pair priority if it doesnt successfully train 2022-05-28 18:34:26 +02:00
robcaulk
83dd453723 catch errors occuring on background thread, and make sure to keep the ball rolling. Improve pair retraining queue. 2022-05-28 18:26:19 +02:00
robcaulk
e54614fa2f remove remnants of single threaded version, ensure pair queue priority is checked before retraining 2022-05-28 14:55:07 +02:00
robcaulk
2a4d1e2d64 fix bug in setting new timerange for retraining 2022-05-28 12:23:26 +02:00
robcaulk
7870a86e9a fix live retraining bug 2022-05-28 11:38:57 +02:00
robcaulk
0bf915054d handle key check correctly 2022-05-28 11:22:32 +02:00
robcaulk
c5a16e91fb throw user error if user tries to load models but feeds the wrong features (while using PCA) 2022-05-28 11:11:41 +02:00
robcaulk
6193205012 fix bug for target_mean/std array merging in backtesting 2022-05-26 21:07:50 +02:00
robcaulk
b79d4e8876 Allow user to go live and start from pretrained models (after a completed backtest) by simply reusing the identifier config parameter while dry/live. 2022-05-25 14:40:32 +02:00
robcaulk
7486d9d9e2 proper validation of freqai config parameters 2022-05-25 12:37:25 +02:00
robcaulk
7ff3258607 remove assertions, log error if user has not assigned freqai in config, fix stratify bug 2022-05-25 11:43:45 +02:00
robcaulk
35bed842cb cleanup, add clarity to comments and docstrings 2022-05-25 11:31:03 +02:00
robcaulk
58b5abbaa6 improve multithreaded training queue system 2022-05-24 15:28:38 +02:00
robcaulk
255d35976e add priority metadata to pairs to avoid a sync of train time + train period 2022-05-24 12:58:53 +02:00
robcaulk
059c285425 paying closer attention to managing live retraining on separate thread without affecting prediction of other coins on master thread 2022-05-24 12:01:01 +02:00
robcaulk
b0d2d13eb1 improve data persistence/mapping for live/dry. This accommodates quick reloads after crash and handles multi-pair cleanly 2022-05-23 21:05:05 +02:00
robcaulk
e1c068ca66 add config asserts, use .get method with default values for optional functionality, move data_cleaning_* to freqai_interface (away from user custom pred model) since it is controlled by config params. 2022-05-23 12:07:09 +02:00
robcaulk
ee3cdd0ffe more cleanup 2022-05-23 09:55:58 +02:00
robcaulk
af0cc21af9 Enable hourly/minute retraining in live/dry. Suppress catboost folder output. Update config + constants + docs to reflect updates. 2022-05-23 00:06:26 +02:00
robcaulk
42d95af829 Aggregated commit. Adding support vector machine for outlier detection, improve user interface to dry/live, better standardization, fix various other bugs 2022-05-22 17:51:49 +02:00
robcaulk
c5ecf94177 move live retraining to separate thread. 2022-05-19 21:15:58 +02:00
robcaulk
89eacf2f47 Retrain model if FreqAI found a pretrained model but user strategy is not passing the expected features (user has changed the features in the strategy but has passed a the same config[freqai][identifier]). Logger warning output to user. 2022-05-19 17:15:50 +02:00
Matthias
c81b960791 Fix some typos 2022-05-17 19:58:36 +02:00
robcaulk
8664e8f9a3 create a prediction_models folder where basic prediction models can live (similar to optimize/hyperopt-loss. Update resolver/docs/and gitignore to accommodate change 2022-05-17 17:13:38 +02:00
robcaulk
9b3e5faebe create more flexible whitelist, avoid duplicating whitelist features into corr_pairlist, update docs 2022-05-15 17:42:15 +02:00
robcaulk
22bd5556ed add self-retraining functionality for live/dry 2022-05-15 17:42:15 +02:00
robcaulk
178c2014b0 appease mypy 2022-05-15 17:42:15 +02:00
robcaulk
f653ace24b another attempt at fixing datalength bug 2022-05-15 17:42:15 +02:00
robcaulk
b08c0888bb add USERPATH_FREQAIMODELS, remove return values from @abstract methods 2022-05-15 17:42:15 +02:00
robcaulk
00ff0c9b91 ensure user defined timerange truncates final backtest so that we arent mismatching data lengths upon return to strategy. Rename DataHandler class to FreqaiDataKitchen 2022-05-15 17:41:34 +02:00
robcaulk
def71a0afe auto build full_timerange and self manage training_timerange 2022-05-15 17:41:34 +02:00
robcaulk
764f9449b4 fix logger, debug some flake8 appeasements 2022-05-15 17:41:34 +02:00
robcaulk
29c2d1d189 use logger in favor of print 2022-05-15 17:38:58 +02:00
robcaulk
99f7e44c30 flake8 passing, use pathlib in lieu of os.path to accommodate windows/mac OS 2022-05-15 17:38:58 +02:00
robcaulk
630d201546 remove trained_stake 2022-05-15 17:38:58 +02:00
robcaulk
b40f8f88ac cleaning and bug fixing 2022-05-15 17:38:58 +02:00
robcaulk
fc837c4daa add freqao backend machinery, user interface, documentation 2022-05-15 17:38:58 +02:00