Merge pull request #5350 from freqtrade/environment_vars

Add environment variable support

docs/configuration.md

@@ -11,6 +11,37 @@ Per default, the bot loads the configuration from the `config.json` file, locate
 
 You can specify a different configuration file used by the bot with the `-c/--config` command-line option.
 
+If you used the [Quick start](installation.md/#quick-start) method for installing 
+the bot, the installation script should have already created the default configuration file (`config.json`) for you.
+
+If the default configuration file is not created we recommend to use `freqtrade new-config --config config.json` to generate a basic configuration file.
+
+The Freqtrade configuration file is to be written in JSON format.
+
+Additionally to the standard JSON syntax, you may use one-line `// ...` and multi-line `/* ... */` comments in your configuration files and trailing commas in the lists of parameters.
+
+Do not worry if you are not familiar with JSON format -- simply open the configuration file with an editor of your choice, make some changes to the parameters you need, save your changes and, finally, restart the bot or, if it was previously stopped, run it again with the changes you made to the configuration. The bot validates the syntax of the configuration file at startup and will warn you if you made any errors editing it, pointing out problematic lines.
+
+### Environment variables
+
+Set options in the Freqtrade configuration via environment variables.
+This takes priority over the corresponding value in configuration or strategy.
+
+Environment variables must be prefixed with  `FREQTRADE__` to be loaded to the freqtrade configuration.
+
+`__` serves as level separator, so the format used should correspond to `FREQTRADE__{section}__{key}`.
+As such - an environment variable defined as  `export FREQTRADE__STAKE_AMOUNT=200` would result in `{stake_amount: 200}`.
+
+A more complex example might be `export FREQTRADE__EXCHANGE__KEY=<yourExchangeKey>` to keep your exchange key secret. This will move the value to the `exchange.key` section of the configuration.
+Using this scheme, all configuration settings will also be available as environment variables.
+
+Please note that Environment variables will overwrite corresponding settings in your configuration, but command line Arguments will always win.
+
+!!! Note
+    Environment variables detected are logged at startup - so if you can't find why a value is not what you think it should be based on the configuration, make sure it's not loaded from an environment variable.
+
+### Multiple configuration files
+
 Multiple configuration files can be specified and used by the bot or the bot can read its configuration parameters from the process standard input stream.
 
 !!! Tip "Use multiple configuration files to keep secrets secret"
@@ -22,17 +53,6 @@ Multiple configuration files can be specified and used by the bot or the bot can
     The 2nd file should only specify what you intend to override.
     If a key is in more than one of the configurations, then the "last specified configuration" wins (in the above example, `config-private.json`).
 
-If you used the [Quick start](installation.md/#quick-start) method for installing 
-the bot, the installation script should have already created the default configuration file (`config.json`) for you.
-
-If the default configuration file is not created we recommend you to use `freqtrade new-config --config config.json` to generate a basic configuration file.
-
-The Freqtrade configuration file is to be written in JSON format.
-
-Additionally to the standard JSON syntax, you may use one-line `// ...` and multi-line `/* ... */` comments in your configuration files and trailing commas in the lists of parameters.
-
-Do not worry if you are not familiar with JSON format -- simply open the configuration file with an editor of your choice, make some changes to the parameters you need, save your changes and, finally, restart the bot or, if it was previously stopped, run it again with the changes you made to the configuration. The bot validates the syntax of the configuration file at startup and will warn you if you made any errors editing it, pointing out problematic lines.
-
 ## Configuration parameters
 
 The table below will list all configuration parameters available.
@@ -41,6 +61,7 @@ Freqtrade can also load many options via command line (CLI) arguments (check out
 The prevalence for all Options is as follows:
 
 - CLI arguments override any other option
+- [Environment Variables](#environment-variables)
 - Configuration files are used in sequence (the last file wins) and override Strategy configurations.
 - Strategy configurations are only used if they are not set via configuration or command-line arguments. These options are marked with [Strategy Override](#parameters-in-the-strategy) in the below table.
 
@@ -526,9 +547,10 @@ Once you will be happy with your bot performance running in the Dry-run mode, yo
 
 ## Switch to production mode
 
-In production mode, the bot will engage your money. Be careful, since a wrong
-strategy can lose all your money. Be aware of what you are doing when
-you run it in production mode.
+In production mode, the bot will engage your money. Be careful, since a wrong strategy can lose all your money.
+Be aware of what you are doing when you run it in production mode.
+
+When switching to Production mode, please make sure to use a different / fresh database to avoid dry-run trades messing with your exchange money and eventually tainting your statistics.
 
 ### Setup your exchange account
 

freqtrade/configuration/configuration.py

@@ -11,6 +11,7 @@ from freqtrade import constants
 from freqtrade.configuration.check_exchange import check_exchange
 from freqtrade.configuration.deprecated_settings import process_temporary_deprecated_settings
 from freqtrade.configuration.directory_operations import create_datadir, create_userdata_dir
+from freqtrade.configuration.environment_vars import enironment_vars_to_dict
 from freqtrade.configuration.load_config import load_config_file, load_file
 from freqtrade.enums import NON_UTIL_MODES, TRADING_MODES, RunMode
 from freqtrade.exceptions import OperationalException
@@ -71,6 +72,11 @@ class Configuration:
 
             # Merge config options, overwriting old values
             config = deep_merge_dicts(load_config_file(path), config)
+
+        # Load environment variables
+        env_data = enironment_vars_to_dict()
+        config = deep_merge_dicts(env_data, config)
+
         config['config_files'] = files
         # Normalize config
         if 'internals' not in config:

freqtrade/configuration/environment_vars.py

@@ -0,0 +1,54 @@
+import logging
+import os
+from typing import Any, Dict
+
+from freqtrade.constants import ENV_VAR_PREFIX
+from freqtrade.misc import deep_merge_dicts
+
+
+logger = logging.getLogger(__name__)
+
+
+def get_var_typed(val):
+    try:
+        return int(val)
+    except ValueError:
+        try:
+            return float(val)
+        except ValueError:
+            if val.lower() in ('t', 'true'):
+                return True
+            elif val.lower() in ('f', 'false'):
+                return False
+    # keep as string
+    return val
+
+
+def flat_vars_to_nested_dict(env_dict: Dict[str, Any], prefix: str) -> Dict[str, Any]:
+    """
+    Environment variables must be prefixed with FREQTRADE.
+    FREQTRADE__{section}__{key}
+    :param env_dict: Dictionary to validate - usually os.environ
+    :param prefix: Prefix to consider (usually FREQTRADE__)
+    :return: Nested dict based on available and relevant variables.
+    """
+    relevant_vars: Dict[str, Any] = {}
+
+    for env_var, val in sorted(env_dict.items()):
+        if env_var.startswith(prefix):
+            logger.info(f"Loading variable '{env_var}'")
+            key = env_var.replace(prefix, '')
+            for k in reversed(key.split('__')):
+                val = {k.lower(): get_var_typed(val) if type(val) != dict else val}
+            relevant_vars = deep_merge_dicts(val, relevant_vars)
+
+    return relevant_vars
+
+
+def enironment_vars_to_dict() -> Dict[str, Any]:
+    """
+    Read environment variables and return a nested dict for relevant variables
+    Relevant variables must follow the FREQTRADE__{section}__{key} pattern
+    :return: Nested dict based on available and relevant variables.
+    """
+    return flat_vars_to_nested_dict(os.environ.copy(), ENV_VAR_PREFIX)

freqtrade/constants.py

@@ -47,6 +47,7 @@ USERPATH_STRATEGIES = 'strategies'
 USERPATH_NOTEBOOKS = 'notebooks'
 
 TELEGRAM_SETTING_OPTIONS = ['on', 'off', 'silent']
+ENV_VAR_PREFIX = 'FREQTRADE__'
 
 
 # Define decimals per coin for outputs

tests/test_configuration.py

@@ -18,8 +18,9 @@ from freqtrade.configuration.deprecated_settings import (check_conflicting_setti
                                                          process_deprecated_setting,
                                                          process_removed_setting,
                                                          process_temporary_deprecated_settings)
+from freqtrade.configuration.environment_vars import flat_vars_to_nested_dict
 from freqtrade.configuration.load_config import load_config_file, load_file, log_config_error_range
-from freqtrade.constants import DEFAULT_DB_DRYRUN_URL, DEFAULT_DB_PROD_URL
+from freqtrade.constants import DEFAULT_DB_DRYRUN_URL, DEFAULT_DB_PROD_URL, ENV_VAR_PREFIX
 from freqtrade.enums import RunMode
 from freqtrade.exceptions import OperationalException
 from freqtrade.loggers import _set_loggers, setup_logging, setup_logging_pre
@@ -1349,3 +1350,35 @@ def test_process_deprecated_ticker_interval(mocker, default_conf, caplog):
     with pytest.raises(OperationalException,
                        match=r"Both 'timeframe' and 'ticker_interval' detected."):
         process_temporary_deprecated_settings(config)
+
+
+def test_flat_vars_to_nested_dict(caplog):
+
+    test_args = {
+        'FREQTRADE__EXCHANGE__SOME_SETTING': 'true',
+        'FREQTRADE__EXCHANGE__SOME_FALSE_SETTING': 'false',
+        'FREQTRADE__EXCHANGE__CONFIG__whatever': 'sometime',
+        'FREQTRADE__ASK_STRATEGY__PRICE_SIDE': 'bid',
+        'FREQTRADE__ASK_STRATEGY__cccc': '500',
+        'FREQTRADE__STAKE_AMOUNT': '200.05',
+        'NOT_RELEVANT': '200.0',  # Will be ignored
+    }
+    expected = {
+        'stake_amount': 200.05,
+        'ask_strategy': {
+            'price_side': 'bid',
+            'cccc': 500,
+        },
+        'exchange': {
+            'config': {
+                'whatever': 'sometime',
+            },
+            'some_setting': True,
+            'some_false_setting': False,
+        }
+    }
+    res = flat_vars_to_nested_dict(test_args, ENV_VAR_PREFIX)
+    assert res == expected
+
+    assert log_has("Loading variable 'FREQTRADE__EXCHANGE__SOME_SETTING'", caplog)
+    assert not log_has("Loading variable 'NOT_RELEVANT'", caplog)