Merge pull request #5350 from freqtrade/environment_vars

Add environment variable support
This commit is contained in:
Matthias 2021-08-02 20:48:23 +02:00 committed by GitHub
commit adb33e763b
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
5 changed files with 131 additions and 15 deletions

View File

@ -11,6 +11,37 @@ Per default, the bot loads the configuration from the `config.json` file, locate
You can specify a different configuration file used by the bot with the `-c/--config` command-line option. You can specify a different configuration file used by the bot with the `-c/--config` command-line option.
If you used the [Quick start](installation.md/#quick-start) method for installing
the bot, the installation script should have already created the default configuration file (`config.json`) for you.
If the default configuration file is not created we recommend to use `freqtrade new-config --config config.json` to generate a basic configuration file.
The Freqtrade configuration file is to be written in JSON format.
Additionally to the standard JSON syntax, you may use one-line `// ...` and multi-line `/* ... */` comments in your configuration files and trailing commas in the lists of parameters.
Do not worry if you are not familiar with JSON format -- simply open the configuration file with an editor of your choice, make some changes to the parameters you need, save your changes and, finally, restart the bot or, if it was previously stopped, run it again with the changes you made to the configuration. The bot validates the syntax of the configuration file at startup and will warn you if you made any errors editing it, pointing out problematic lines.
### Environment variables
Set options in the Freqtrade configuration via environment variables.
This takes priority over the corresponding value in configuration or strategy.
Environment variables must be prefixed with `FREQTRADE__` to be loaded to the freqtrade configuration.
`__` serves as level separator, so the format used should correspond to `FREQTRADE__{section}__{key}`.
As such - an environment variable defined as `export FREQTRADE__STAKE_AMOUNT=200` would result in `{stake_amount: 200}`.
A more complex example might be `export FREQTRADE__EXCHANGE__KEY=<yourExchangeKey>` to keep your exchange key secret. This will move the value to the `exchange.key` section of the configuration.
Using this scheme, all configuration settings will also be available as environment variables.
Please note that Environment variables will overwrite corresponding settings in your configuration, but command line Arguments will always win.
!!! Note
Environment variables detected are logged at startup - so if you can't find why a value is not what you think it should be based on the configuration, make sure it's not loaded from an environment variable.
### Multiple configuration files
Multiple configuration files can be specified and used by the bot or the bot can read its configuration parameters from the process standard input stream. Multiple configuration files can be specified and used by the bot or the bot can read its configuration parameters from the process standard input stream.
!!! Tip "Use multiple configuration files to keep secrets secret" !!! Tip "Use multiple configuration files to keep secrets secret"
@ -22,17 +53,6 @@ Multiple configuration files can be specified and used by the bot or the bot can
The 2nd file should only specify what you intend to override. The 2nd file should only specify what you intend to override.
If a key is in more than one of the configurations, then the "last specified configuration" wins (in the above example, `config-private.json`). If a key is in more than one of the configurations, then the "last specified configuration" wins (in the above example, `config-private.json`).
If you used the [Quick start](installation.md/#quick-start) method for installing
the bot, the installation script should have already created the default configuration file (`config.json`) for you.
If the default configuration file is not created we recommend you to use `freqtrade new-config --config config.json` to generate a basic configuration file.
The Freqtrade configuration file is to be written in JSON format.
Additionally to the standard JSON syntax, you may use one-line `// ...` and multi-line `/* ... */` comments in your configuration files and trailing commas in the lists of parameters.
Do not worry if you are not familiar with JSON format -- simply open the configuration file with an editor of your choice, make some changes to the parameters you need, save your changes and, finally, restart the bot or, if it was previously stopped, run it again with the changes you made to the configuration. The bot validates the syntax of the configuration file at startup and will warn you if you made any errors editing it, pointing out problematic lines.
## Configuration parameters ## Configuration parameters
The table below will list all configuration parameters available. The table below will list all configuration parameters available.
@ -41,6 +61,7 @@ Freqtrade can also load many options via command line (CLI) arguments (check out
The prevalence for all Options is as follows: The prevalence for all Options is as follows:
- CLI arguments override any other option - CLI arguments override any other option
- [Environment Variables](#environment-variables)
- Configuration files are used in sequence (the last file wins) and override Strategy configurations. - Configuration files are used in sequence (the last file wins) and override Strategy configurations.
- Strategy configurations are only used if they are not set via configuration or command-line arguments. These options are marked with [Strategy Override](#parameters-in-the-strategy) in the below table. - Strategy configurations are only used if they are not set via configuration or command-line arguments. These options are marked with [Strategy Override](#parameters-in-the-strategy) in the below table.
@ -526,9 +547,10 @@ Once you will be happy with your bot performance running in the Dry-run mode, yo
## Switch to production mode ## Switch to production mode
In production mode, the bot will engage your money. Be careful, since a wrong In production mode, the bot will engage your money. Be careful, since a wrong strategy can lose all your money.
strategy can lose all your money. Be aware of what you are doing when Be aware of what you are doing when you run it in production mode.
you run it in production mode.
When switching to Production mode, please make sure to use a different / fresh database to avoid dry-run trades messing with your exchange money and eventually tainting your statistics.
### Setup your exchange account ### Setup your exchange account

View File

@ -11,6 +11,7 @@ from freqtrade import constants
from freqtrade.configuration.check_exchange import check_exchange from freqtrade.configuration.check_exchange import check_exchange
from freqtrade.configuration.deprecated_settings import process_temporary_deprecated_settings from freqtrade.configuration.deprecated_settings import process_temporary_deprecated_settings
from freqtrade.configuration.directory_operations import create_datadir, create_userdata_dir from freqtrade.configuration.directory_operations import create_datadir, create_userdata_dir
from freqtrade.configuration.environment_vars import enironment_vars_to_dict
from freqtrade.configuration.load_config import load_config_file, load_file from freqtrade.configuration.load_config import load_config_file, load_file
from freqtrade.enums import NON_UTIL_MODES, TRADING_MODES, RunMode from freqtrade.enums import NON_UTIL_MODES, TRADING_MODES, RunMode
from freqtrade.exceptions import OperationalException from freqtrade.exceptions import OperationalException
@ -71,6 +72,11 @@ class Configuration:
# Merge config options, overwriting old values # Merge config options, overwriting old values
config = deep_merge_dicts(load_config_file(path), config) config = deep_merge_dicts(load_config_file(path), config)
# Load environment variables
env_data = enironment_vars_to_dict()
config = deep_merge_dicts(env_data, config)
config['config_files'] = files config['config_files'] = files
# Normalize config # Normalize config
if 'internals' not in config: if 'internals' not in config:

View File

@ -0,0 +1,54 @@
import logging
import os
from typing import Any, Dict
from freqtrade.constants import ENV_VAR_PREFIX
from freqtrade.misc import deep_merge_dicts
logger = logging.getLogger(__name__)
def get_var_typed(val):
try:
return int(val)
except ValueError:
try:
return float(val)
except ValueError:
if val.lower() in ('t', 'true'):
return True
elif val.lower() in ('f', 'false'):
return False
# keep as string
return val
def flat_vars_to_nested_dict(env_dict: Dict[str, Any], prefix: str) -> Dict[str, Any]:
"""
Environment variables must be prefixed with FREQTRADE.
FREQTRADE__{section}__{key}
:param env_dict: Dictionary to validate - usually os.environ
:param prefix: Prefix to consider (usually FREQTRADE__)
:return: Nested dict based on available and relevant variables.
"""
relevant_vars: Dict[str, Any] = {}
for env_var, val in sorted(env_dict.items()):
if env_var.startswith(prefix):
logger.info(f"Loading variable '{env_var}'")
key = env_var.replace(prefix, '')
for k in reversed(key.split('__')):
val = {k.lower(): get_var_typed(val) if type(val) != dict else val}
relevant_vars = deep_merge_dicts(val, relevant_vars)
return relevant_vars
def enironment_vars_to_dict() -> Dict[str, Any]:
"""
Read environment variables and return a nested dict for relevant variables
Relevant variables must follow the FREQTRADE__{section}__{key} pattern
:return: Nested dict based on available and relevant variables.
"""
return flat_vars_to_nested_dict(os.environ.copy(), ENV_VAR_PREFIX)

View File

@ -47,6 +47,7 @@ USERPATH_STRATEGIES = 'strategies'
USERPATH_NOTEBOOKS = 'notebooks' USERPATH_NOTEBOOKS = 'notebooks'
TELEGRAM_SETTING_OPTIONS = ['on', 'off', 'silent'] TELEGRAM_SETTING_OPTIONS = ['on', 'off', 'silent']
ENV_VAR_PREFIX = 'FREQTRADE__'
# Define decimals per coin for outputs # Define decimals per coin for outputs

View File

@ -18,8 +18,9 @@ from freqtrade.configuration.deprecated_settings import (check_conflicting_setti
process_deprecated_setting, process_deprecated_setting,
process_removed_setting, process_removed_setting,
process_temporary_deprecated_settings) process_temporary_deprecated_settings)
from freqtrade.configuration.environment_vars import flat_vars_to_nested_dict
from freqtrade.configuration.load_config import load_config_file, load_file, log_config_error_range from freqtrade.configuration.load_config import load_config_file, load_file, log_config_error_range
from freqtrade.constants import DEFAULT_DB_DRYRUN_URL, DEFAULT_DB_PROD_URL from freqtrade.constants import DEFAULT_DB_DRYRUN_URL, DEFAULT_DB_PROD_URL, ENV_VAR_PREFIX
from freqtrade.enums import RunMode from freqtrade.enums import RunMode
from freqtrade.exceptions import OperationalException from freqtrade.exceptions import OperationalException
from freqtrade.loggers import _set_loggers, setup_logging, setup_logging_pre from freqtrade.loggers import _set_loggers, setup_logging, setup_logging_pre
@ -1349,3 +1350,35 @@ def test_process_deprecated_ticker_interval(mocker, default_conf, caplog):
with pytest.raises(OperationalException, with pytest.raises(OperationalException,
match=r"Both 'timeframe' and 'ticker_interval' detected."): match=r"Both 'timeframe' and 'ticker_interval' detected."):
process_temporary_deprecated_settings(config) process_temporary_deprecated_settings(config)
def test_flat_vars_to_nested_dict(caplog):
test_args = {
'FREQTRADE__EXCHANGE__SOME_SETTING': 'true',
'FREQTRADE__EXCHANGE__SOME_FALSE_SETTING': 'false',
'FREQTRADE__EXCHANGE__CONFIG__whatever': 'sometime',
'FREQTRADE__ASK_STRATEGY__PRICE_SIDE': 'bid',
'FREQTRADE__ASK_STRATEGY__cccc': '500',
'FREQTRADE__STAKE_AMOUNT': '200.05',
'NOT_RELEVANT': '200.0', # Will be ignored
}
expected = {
'stake_amount': 200.05,
'ask_strategy': {
'price_side': 'bid',
'cccc': 500,
},
'exchange': {
'config': {
'whatever': 'sometime',
},
'some_setting': True,
'some_false_setting': False,
}
}
res = flat_vars_to_nested_dict(test_args, ENV_VAR_PREFIX)
assert res == expected
assert log_has("Loading variable 'FREQTRADE__EXCHANGE__SOME_SETTING'", caplog)
assert not log_has("Loading variable 'NOT_RELEVANT'", caplog)