[Akash // Sovryn Node] Deployment guideline (telegram, RSK wallet, secrets management) [rstm]

Hi Akash & Sovryn community,

Please find below Sovryn node deployment guideline in Akash Mainnet. The following topics are covered in this guideline:

  • Telegram bot and chat configuration
  • RSK blockchain wallet and keys export
  • Secure keys transmission using HashiCorp Vault (KV and cubbyhole engines)
  • Docker image creation and repository setup
  • Akash installation and account setup
  • Akash deployment instructions

Original guideline and configuration files can be found in our GitHub repo. Feel free to join to improve security and usability: sovryn-node-123/README.md at main · rustamabdullin/sovryn-node-123 · GitHub

Work to be done:

  • secure private key delivery from web3 wallet to Sovryn Node (arbitrary project in Akash). Suggest unified approach to secure secrets in Akash deployments

  • minor changes in Docker files

  • current Sovryn Node code has at least 10 high and critical severity vulnerabilities to be addressed https://github.com/rustamabdullin/sovryn-node-123/security/dependabot

  • hardening Sovryn Node image and Akash deployment (network access control, Docker security)

1. Prepare new repo

1.1 Create new repo, copy files from official repository GitHub - DistributedCollective/Sovryn-Node

git clone https://github.com/DistributedCollective/Sovryn-Node

1.2 Useful links:

2. Telegram bot and chat configuration

Sovryn node uses Telegram chat/bot to send notifications about new transactions and errors.

2.1 Use Telegram: Contact @BotFather to create new bot (/newbot command). Also you can use /help for more specified options to your bot and official telegram documentation

telegram01

2.2 Get API token. id:apiToken

i.e. 1876066462:AAF1nDRouq1rOqhjrJIo5eCWuC3-ktQ8iSI

2.3 Create new chat and add created telegram bot. Send some messages to newly created chat.

2.4 Use the following URL to get Chat ID starting with ("-") symbol: https://api.telegram.org/bot[BOT-API-TOKEN]/getUpdates

i.e. curl https://api.telegram.org/botBOT-API-TOKEN]/getUpdates and you’ll get response {"ok":true,"result":[]}

2.5 To receive notifications on telegram write API token to /secrets/telegram.js file

export default "[telegram-bot-token]";

and write Chat ID to /config/config_mainnet.js and /config/config_testnet.js files.

sovrynInternalTelegramId: -492690059,

3. Create wallet in RSK blockchain and export keys

To trade on Sovryn, you will need to set up a Web3 wallet that is compatible with the RSK chain (Rootstock).

Testnet Wallet setup

3.1 Go to the Metamask website and download the latest version of the Metamask Wallet extension.

3.2 Install and have the Metamask extension active in your browser.

3.3 Open Metamask and register on it. (Do not forget to save your recovery phrase!)

3.4 Click the circle in the upper right of the wallet → Settings → Networks → click the Add Network button and enter the following RSK Network settings.

3.5 Hit Save and make sure you are switched to the RSK Mainnet.

3.6 Request some tRBTC from https://faucet.rsk.co

3.7 Save your public key from Metamask and export your private key: → Account → Account Details → Export private key. That will be your credentials of the liquidator/rollover/arbitrage wallets credentials

Useful links:

4. Convert keys to Keystore v3 format

4.1 Update accounts.js file with the credentials of the liquidator/rollover/arbitrage wallets

You have 2 options

  • [Insecure] you can specify pKey instead of ks to just use the private key
  • [Secure] ks = encrypted keystore file in v3 standard. (Do not forget to save your keystore password!)

4.2 Install python3, pip3 and web3 libraries

pip3 install web3

4.3 Use the following simple python3 script to generate keystore v3 JSON

import web3
import json
from eth_account import Account

key = Account.encrypt("dde2ec361acec024e78587f605fb8f2f098aacb5c492393e7cca66a42f288664", "SecurePassword123$")

print(json.dumps(key))

Sample output:

{
  "address": "9af00e58040f2f0fbfb3acd542f7c5f1a4fabd70",
  "crypto": {
    "cipher": "aes-128-ctr",
    "cipherparams": {
      "iv": "28a54a14a930ba79d73f86b2ead959b8"
    },
    "ciphertext": "6ca58b069d23f178c067392fc34b50af02a403cbd7749e352ef5ee621eb5605b",
    "kdf": "scrypt",
    "kdfparams": {
      "dklen": 32,
      "n": 262144,
      "r": 1,
      "p": 8,
      "salt": "7d5519d5bddb35e4818f1ca7e368a459"
    },
    "mac": "86bf62840ae93e62d5acfdad8b24428eab2431d676afe7e95327101b731a5304"
  },
  "id": "29def9c7-46d6-421a-91e5-c86bfdeedac5",
  "version": 3
}

4.4 Update accounts.js file and protect password using HashiCorp vault (see instructions below)

export default {
    "liquidator": [{
        adr: "",
        ks: ""
    }],
    "rollover": [{
        adr: "",
        ks: ""
    }],
    "arbitrage": [{
        adr: "",
        ks: ""
    }],
}

5. Create Docker image and publish to repository

5.1 Review and update Dockerfile [Update ENV section]

5.2 Create new public DockerHub repo

5.3 Build your image

docker build -t [DockerHub account name]/[DockerHub repo name] . --no-cache

5.4 [Optional] Run your container in local Docker

docker run -p 3000:3000 [DockerHub account name]/[DockerHub repo name]:latest

sarasioux found that Akash may have cache issues during image rebuild/application redeploy. Use proper tagging to avoid such issues and to make sure you deploy proper version of your application.

5.5 Login to DockerHub with AccessKey and push your image

docker login --username [DockerHub account name]/[DockerHub repo name]
docker push [DockerHub account name]/[DockerHub repo name]:latest
1 Like

6. Install Akash and create account

6.1 Use the following guide to install Akash

6.2 Create new Akash account

Consider using the following bash script (update akash binary path or update PATH env variable).

#!/bin/bash

AKASH_KEY_NAME="[YOUR KEY NAME HERE"
AKASH_KEYRING_BACKEND="os"

/opt/homebrew/bin/akash --keyring-backend "$AKASH_KEYRING_BACKEND" keys add "$AKASH_KEY_NAME"

[IMPORTANT!!!] Save your account address and mnemonic phrase.

Useful links:

https://docs.akash.network/start/wallet

7. Akash deployment instructions

Consider using the following bash script. Properly update [VARIABLE VALUES]. Run commands one by one.

#!/bin/bash 

AKASH_NET="https://raw.githubusercontent.com/ovrclk/net/master/mainnet"
AKASH_VERSION="$(curl -s "$AKASH_NET/version.txt")"
export AKASH_CHAIN_ID="$(curl -s "$AKASH_NET/chain-id.txt")"
curl -s "$AKASH_NET/api-nodes.txt" 
AKASH_NODE="http://rpc.mainnet.akash.dual.systems:80"

AKASH_KEY_NAME="[AKASH ACCOUNT NAME]"
AKASH_KEYRING_BACKEND="os"
ACCOUNT_ADDRESS="[AKASH ACCOUNT ADDRESS]"

#STEP 0 - Create Certificate
#/opt/homebrew/bin/akash tx cert create client --chain-id $AKASH_CHAIN_ID --keyring-backend $AKASH_KEYRING_BACKEND --from $AKASH_KEY_NAME --node "http://rpc.mainnet.akash.dual.systems:80" --fees 5000uakt

#STEP 1 - Create deployment, get DSEQ into $DSEQ
#/opt/homebrew/bin/akash tx deployment create akash-sovryn-deploy.yml --from $AKASH_KEY_NAME --keyring-backend $AKASH_KEYRING_BACKEND --node "http://rpc.mainnet.akash.dual.systems:80" --chain-id $AKASH_CHAIN_ID -y --fees 5000uakt

#STEP 2 - Check BIDs
DSEQ=[DSEQ from previous STEP]
#/opt/homebrew/bin/akash query market bid list --owner=$ACCOUNT_ADDRESS --node $AKASH_NODE --dseq $DSEQ

#STEP 3 - Accept a bid by creating lease, get provider into $PROVIDER
DSEQ=[DSEQ from previous STEP]
GSEQ=[QSEQ from previous STEP]
OSEQ=[OSEQ from previous STEP]
PROVIDER="[PROVIDER from previous STEP]"
#/opt/homebrew/bin/akash tx market lease create --chain-id $AKASH_CHAIN_ID --node $AKASH_NODE --owner $ACCOUNT_ADDRESS --dseq $DSEQ --gseq $GSEQ --oseq $OSEQ --provider $PROVIDER --from $AKASH_KEY_NAME --fees 5000uakt --keyring-backend $AKASH_KEYRING_BACKEND

#STEP 4 - Check lease status
#/opt/homebrew/bin/akash query market lease list --owner $ACCOUNT_ADDRESS --node $AKASH_NODE --dseq $DSEQ

#STEP 5 - Upload our manifest, wait for spinup
#/opt/homebrew/bin/akash provider send-manifest akash-sovryn-deploy.yml --keyring-backend $AKASH_KEYRING_BACKEND --node $AKASH_NODE --from=$AKASH_KEY_NAME --provider=$PROVIDER --dseq $DSEQ --log_level=info --home ~/.akash

#STEP 6 - Check the lease status
#/opt/homebrew/bin/akash provider lease-status --node $AKASH_NODE --home ~/.akash --dseq $DSEQ --from $AKASH_KEY_NAME --provider $PROVIDER --keyring-backend $AKASH_KEYRING_BACKEND

#STEP 7 - Check logs
#/opt/homebrew/bin/akash provider lease-logs --dseq=$DSEQ --from=$ACCOUNT_ADDRESS --provider=$PROVIDER

#STEP 9 - Close deployment
#/opt/homebrew/bin/akash tx deployment close --node $AKASH_NODE --chain-id $AKASH_CHAIN_ID --dseq $DSEQ --owner $ACCOUNT_ADDRESS --from $AKASH_KEY_NAME --keyring-backend $AKASH_KEYRING_BACKEND -y --fees 5000uakt

8. Security. Securing private keys

8.1 Private key security

A private key is a sophisticated form of cryptography that allows a user to access their cryptocurrency.
Original Sovryn Node repository keeps private key [or private key password] in a clear text format in accounts.js file (or in *.yml file EVN section).

The following guidline aims to protect private key using HashiCorp Vault. The wrapped secret can be unwrapped using the single-use wrapping token. Even the user or the system created the initial token won’t see the original value.

Key principles are:

  • Avoid storing crypto wallet private key in a repo and in an image
  • Try to avoid using long lived Vault access tokens in a running Akash container

8.2 HashiCorp Cloud Platform (HCP) setup // Cloud Vault Cluster

Create a Vault Cluster in HCP (https://portal.cloud.hashicorp.com)

Official instructions are available here: Create a Vault Cluster on HCP | Vault - HashiCorp Learn

8.3 Secure private key delivery // Cubbyhole Response Wrapping

We use Vault’s cubbyhole response wrapping approach where the initial token is stored in the cubbyhole secrets engine. The wrapped secret can be unwrapped using the single-use wrapping token. Even the user or the system created the initial token won’t see the original value. The wrapping token is short-lived and can be revoked just like any other tokens so that the risk of unauthorized access can be minimized.

All secrets are namespaced under your token. If that token expires or is revoked, all the secrets in its cubbyhole are revoked as well.

It is not possible to reach into another token’s cubbyhole even as the root user. This is an important difference between the cubbyhole and the key/value secrets engine. The secrets in the key/value secrets engine are accessible to any token for as long as its policy allows it.

Benefits of using the response wrapping:

  • It provides cover by ensuring that the value being transmitted across the wire is not the actual secret. It’s a reference to the secret.
  • It provides malfeasance detection by ensuring that only a single party can ever unwrap the token and see what’s inside
  • It limits the lifetime of the secret exposure

8.4 Create Vault access token

  1. Install Vault CLI client.

Use official guideline. Install Vault | Vault by HashiCorp

  1. Define environment variables, authenticate and create new access token.
#!/bin/bash
# Export env variables for Vault
export VAULT_ADDR="https://vault-cluster.vault.[VAULT PUBLIC ADDRESS].aws.hashicorp.cloud:8200"
export VAULT_NAMESPACE="admin"

# Login via admin token
vault login [VAULT ADMIN TOKEN]

# Create a policy for the node
cat << EOF > node-policy.hcl
path "secret/data/dev" {
  capabilities = [ "read" ]
}
EOF

vault policy write node-policy node-policy.hcl
# Enable key/value v2 secrets engine at secret/ if it's not enabled already
 vault secrets enable -path=secret kv-v2

# Write some secret at secret/dev
vault kv put secret/dev private="my-private-data"

# Generating one-time token that'll be used on a node
ONETIME_TOKEN=`vault token create -use-limit=2 | grep -w token | awk '{print $2}'`

# Generating wrapping token that'll be used for retrieving the secret
WRAPPING_TOKEN=`vault token create -policy=node-policy -wrap-ttl=300 | grep -w "wrapping_token:" | awk '{print $2}'`

# Store wrapping token in a cubbyhole storage using newly generated token that'll expire in the next one use
VAULT_TOKEN="$ONETIME_TOKEN" vault write cubbyhole/private/access-token token="$WRAPPING_TOKEN"

# Copy this token to a node to get wrapping token
echo "Use this token on a node: $ONETIME_TOKEN"

Sample Vault secrets:

8.5 Update Docker image

################## NODE SIDE ##################
# Get wrapping token via one-time token
export VAULT_ADDR="https://vault-cluster.vault.[VAULT PUBLIC ADDRESS].aws.hashicorp.cloud:8200"
WRAPPING_TOKEN=`curl --header "X-Vault-Token: $ONETIME_TOKEN" \
    --header "X-Vault-Namespace: admin"  \
     $VAULT_ADDR/v1/cubbyhole/private/access-token | jq -r .data.token #| sed 's/"//g'`

# Unwrap the token
VAULT_TOKEN=`curl --header "X-Vault-Token: $WRAPPING_TOKEN" \
        --header "X-Vault-Namespace: admin" \
        --request POST \
        $VAULT_ADDR/v1/sys/wrapping/unwrap | jq -r .auth.client_token`

# Get the secret
curl --header "X-Vault-Token: $VAULT_TOKEN" \
        --header "X-Vault-Namespace: admin" \
        $VAULT_ADDR/v1/secret/data/dev | jq -r .data.data.private

8.6 Useful links:

Hi Akash and Sovryn users,

We are happy to share the following video where we deployed Sovryn node. Private key is protected by Vault.

Important notes:

  1. http://ce3ueoed7lbr3fak3igb8hc170.ingress.sjc1p0.mainnet.akashian.io - Sovryn Node
  2. Docker Hub - Docker repo. Image has only one time (or TTL 20min) Vault token. So we don’t keep sensitive information in config files or in Docker hub.
2 Likes