Merge pull request #1078 from creslinux/sandbox2

Allow sandbox API use on exchanges

README.md

@@ -50,6 +50,7 @@ hesitate to read the source code and understand the mechanism of this bot.
   - [Strategy Optimization](https://github.com/freqtrade/freqtrade/blob/develop/docs/bot-optimization.md)
   - [Backtesting](https://github.com/freqtrade/freqtrade/blob/develop/docs/backtesting.md)
   - [Hyperopt](https://github.com/freqtrade/freqtrade/blob/develop/docs/hyperopt.md)
+  - [Sandbox Testing](https://github.com/freqtrade/freqtrade/blob/develop/docs/sandbox-testing.md)
 - [Basic Usage](#basic-usage)
   - [Bot commands](#bot-commands)
   - [Telegram RPC commands](#telegram-rpc-commands)
@@ -62,6 +63,7 @@ hesitate to read the source code and understand the mechanism of this bot.
   - [Min hardware required](#min-hardware-required)
   - [Software requirements](#software-requirements)
  
+
 ## Quick start
 
 Freqtrade provides a Linux/macOS script to install all dependencies and help you to configure the bot.

docs/index.md

@@ -33,3 +33,4 @@ Pull-request. Do not hesitate to reach us on
 	- [Run tests & Check PEP8 compliance](https://github.com/freqtrade/freqtrade/blob/develop/CONTRIBUTING.md)
 - [FAQ](https://github.com/freqtrade/freqtrade/blob/develop/docs/faq.md)
     - [SQL cheatsheet](https://github.com/freqtrade/freqtrade/blob/develop/docs/sql_cheatsheet.md)
+- [Sandbox Testing](https://github.com/freqtrade/freqtrade/blob/develop/docs/sandbox-testing.md))

docs/sandbox-testing.md

@@ -0,0 +1,151 @@
+# Sandbox API testing
+Where an exchange provides a sandbox for risk-free integration, or end-to-end, testing CCXT provides access to these.
+
+This document is a *light overview of configuring Freqtrade and GDAX sandbox.
+This can be useful to developers and trader alike as Freqtrade is quite customisable.
+
+When testing your API connectivity, make sure to use the following URLs.
+***Website**
+https://public.sandbox.gdax.com
+***REST API**
+https://api-public.sandbox.gdax.com
+
+---
+# Configure a Sandbox account on Gdax
+Aim of this document section
+- An sanbox account
+- create 2FA (needed to create an API)
+- Add test 50BTC to account 
+- Create :
+- - API-KEY
+- - API-Secret
+- - API Password
+
+## Acccount
+
+This link will redirect to the sandbox main page to login / create account dialogues:
+https://public.sandbox.pro.coinbase.com/orders/
+
+After registration and Email confimation you wil be redirected into your sanbox account.  It is easy to verify you're in sandbox by checking the URL bar.
+> https://public.sandbox.pro.coinbase.com/
+
+## Enable 2Fa (a prerequisite to creating sandbox API Keys)
+From within sand box site select your profile, top right.
+>Or as a direct link: https://public.sandbox.pro.coinbase.com/profile
+
+From the menu panel to the left of the screen select 
+> Security: "*View or Update*"
+
+In the new site select "enable authenticator" as typical google Authenticator. 
+- open Google Authenticator on your phone
+- scan barcode 
+- enter your generated 2fa 
+
+## Enable API Access 
+From within sandbox select profile>api>create api-keys
+>or as a direct link: https://public.sandbox.pro.coinbase.com/profile/api
+
+Click on "create one" and ensure  **view** and **trade**  are "checked" and sumbit your 2Fa
+- **Copy and paste the Passphase** into a notepade this will be needed later
+- **Copy and paste the API Secret** popup into a notepad this will needed later
+- **Copy and paste the API Key** into a notepad this will needed later
+
+## Add 50 BTC test funds
+To add funds, use the web interface deposit and withdraw buttons.
+
+
+To begin select 'Wallets' from the top menu.
+> Or as a direct link: https://public.sandbox.pro.coinbase.com/wallets
+
+- Deposits (bottom left of screen)
+- - Deposit Funds Bitcoin 
+- - - Coinbase BTC Wallet 
+- - - - Max (50 BTC) 
+- - - - - Deposit
+
+*This process may be repeated for other currencies, ETH as example*
+---
+# Configure Freqtrade to use Gax Sandbox 
+
+The aim of this document section
+ - Enable sandbox URLs in Freqtrade
+ - Configure API 
+ - - secret
+ - - key
+ - - passphrase
+ 
+## Sandbox URLs
+Freqtrade makes use of CCXT which in turn provides a list of URLs to Freqtrade. 
+These include `['test']` and `['api']`. 
+- `[Test]` if available will point to an Exchanges sandbox. 
+- `[Api]` normally used, and resolves to live API target on the exchange 
+
+To make use of sandbox / test add "sandbox": true, to your config.json
+```
+  "exchange": {
+        "name": "gdax",
+        "sandbox": true,
+        "key": "5wowfxemogxeowo;heiohgmd",
+        "secret": "/ZMH1P62rCVmwefewrgcewX8nh4gob+lywxfwfxwwfxwfNsH1ySgvWCUR/w==",
+        "password": "1bkjfkhfhfu6sr",
+        "pair_whitelist": [
+            "BTC/USD"
+```
+Also insert your 
+- api-key  (noted earlier)
+- api-secret (noted earlier)
+- password (the passphrase - noted earlier)
+
+---
+## You should now be ready to test your sandbox!
+Ensure Freqtrade logs show the sandbox URL, and trades made are shown in sandbox.
+** Typically the  BTC/USD has the most activity in sandbox to test against. 
+
+## GDAX - Old Candles problem
+It is my experience that GDAX sandbox candles may be 20+- minutes out of date. This can cause trades to fail as one of Freqtrades safety checks 
+
+To disable this check, edit: 
+>strategy/interface.py
+Look for the following section:
+```
+      # Check if dataframe is out of date
+        signal_date = arrow.get(latest['date'])
+        interval_minutes = constants.TICKER_INTERVAL_MINUTES[interval]
+        if signal_date < (arrow.utcnow().shift(minutes=-(interval_minutes * 2 + 5))):
+            logger.warning(
+                'Outdated history for pair %s. Last tick is %s minutes old',
+                pair,
+                (arrow.utcnow() - signal_date).seconds // 60
+            )
+            return False, False
+```
+
+You could Hash out the entire check as follows:
+```
+       # # Check if dataframe is out of date
+        # signal_date = arrow.get(latest['date'])
+        # interval_minutes = constants.TICKER_INTERVAL_MINUTES[interval]
+        # if signal_date < (arrow.utcnow().shift(minutes=-(interval_minutes * 2 + 5))):
+        #     logger.warning(
+        #         'Outdated history for pair %s. Last tick is %s minutes old',
+        #         pair,
+        #         (arrow.utcnow() - signal_date).seconds // 60
+        #     )
+        #     return False, False
+ ```
+ 
+ Or inrease the timeout to offer a level of protection/alignment of this test to freqtrade in live.
+ 
+ As example, to allow an additional 30 minutes. "(interval_minutes * 2 + 5 + 30)"
+ ```
+      # Check if dataframe is out of date
+        signal_date = arrow.get(latest['date'])
+        interval_minutes = constants.TICKER_INTERVAL_MINUTES[interval]
+        if signal_date < (arrow.utcnow().shift(minutes=-(interval_minutes * 2 + 5 + 30))):
+            logger.warning(
+                'Outdated history for pair %s. Last tick is %s minutes old',
+                pair,
+                (arrow.utcnow() - signal_date).seconds // 60
+            )
+            return False, False
+```

freqtrade/constants.py

@@ -125,6 +125,7 @@ CONF_SCHEMA = {
             'type': 'object',
             'properties': {
                 'name': {'type': 'string'},
+                'sandbox': {'type': 'boolean'},
                 'key': {'type': 'string'},
                 'secret': {'type': 'string'},
                 'password': {'type': 'string'},

freqtrade/exchange/__init__.py

@@ -96,6 +96,8 @@ class Exchange(object):
         except (KeyError, AttributeError):
             raise OperationalException(f'Exchange {name} is not supported')
 
+        self.set_sandbox(api, exchange_config, name)
+
         return api
 
     @property
@@ -108,6 +110,16 @@ class Exchange(object):
         """exchange ccxt id"""
         return self._api.id
 
+    def set_sandbox(self, api, exchange_config: dict, name: str):
+        if exchange_config.get('sandbox'):
+            if api.urls.get('test'):
+                api.urls['api'] = api.urls['test']
+                logger.info("Enabled Sandbox API on %s", name)
+            else:
+                logger.warning(self, "No Sandbox URL in CCXT, exiting. "
+                                     "Please check your config.json")
+                raise OperationalException(f'Exchange {name} does not provide a sandbox api')
+
     def validate_pairs(self, pairs: List[str]) -> None:
         """
         Checks if all given pairs are tradable on the current exchange.

freqtrade/tests/exchange/test_exchange.py

@@ -98,6 +98,47 @@ def test_symbol_price_prec(default_conf, mocker):
     assert price == 2.3456
 
 
+def test_set_sandbox(default_conf, mocker):
+    """
+    Test working scenario
+    """
+    api_mock = MagicMock()
+    api_mock.load_markets = MagicMock(return_value={
+        'ETH/BTC': '', 'LTC/BTC': '', 'XRP/BTC': '', 'NEO/BTC': ''
+    })
+    url_mock = PropertyMock(return_value={'test': "api-public.sandbox.gdax.com",
+                                          'api': 'https://api.gdax.com'})
+    type(api_mock).urls = url_mock
+    mocker.patch('freqtrade.exchange.Exchange._init_ccxt', MagicMock(return_value=api_mock))
+    mocker.patch('freqtrade.exchange.Exchange.validate_timeframes', MagicMock())
+
+    exchange = Exchange(default_conf)
+    liveurl = exchange._api.urls['api']
+    default_conf['exchange']['sandbox'] = True
+    exchange.set_sandbox(exchange._api, default_conf['exchange'], 'Logname')
+    assert exchange._api.urls['api'] != liveurl
+
+
+def test_set_sandbox_exception(default_conf, mocker):
+    """
+    Test Fail scenario
+    """
+    api_mock = MagicMock()
+    api_mock.load_markets = MagicMock(return_value={
+        'ETH/BTC': '', 'LTC/BTC': '', 'XRP/BTC': '', 'NEO/BTC': ''
+    })
+    url_mock = PropertyMock(return_value={'api': 'https://api.gdax.com'})
+    type(api_mock).urls = url_mock
+
+    mocker.patch('freqtrade.exchange.Exchange._init_ccxt', MagicMock(return_value=api_mock))
+    mocker.patch('freqtrade.exchange.Exchange.validate_timeframes', MagicMock())
+
+    with pytest.raises(OperationalException, match=r'does not provide a sandbox api'):
+        exchange = Exchange(default_conf)
+        default_conf['exchange']['sandbox'] = True
+        exchange.set_sandbox(exchange._api, default_conf['exchange'], 'Logname')
+
+
 def test_validate_pairs(default_conf, mocker):
     api_mock = MagicMock()
     api_mock.load_markets = MagicMock(return_value={