[DONE] Better Secrets management, for Dev / Stage / Prod

Feature Requests
#1

Following the patterns outlined for 12-Factor Apps would it be possible to move closer to having backing services credentials stored in the run-time environment and not the code ?

The desire is to have “get twitter api key” and not have to manage in code if its “get development twitter api key” or “get production twitter api key”

The suggestion from anvil support is as follows:

For example, if your development app id was 'ABCDEFGHIJKL', you could define a function called `load_secrets`:

def load_secrets():
  if anvil.app.id == "ABCDEFGHIJKL":
    data_key = anvil.secrets.get_secret('dev_data_key')
  else:
    data_key = anvil.secrets.get_secret('prod_data_key')
  return data_key

I suggest exploring options where the application code does not need to know which key id to look up (i.e. avoid “Get development twitter api key”)

Two suggestions:

(1) Prepend the app id the secrets section of the YAML


ABCDEFGHIJKL_secrets:
  my_api_key:
    value: {? '' : encrypted_key_value_for_env_1}
    type: secret

MNOPQRSTUVWXYZ_secrets:
  my_api_key:
    value: {? '' : encrypted_key_value_for_env_2}
    type: secret

then you can fetch the DEV anvil.yaml, via git, and include both secret libraries in a single yaml, and Anvil will load the right one at start up.

(2) Add AppId as a field in the Secrets yaml spec (variation on the above)

secrets:
  ABCDEFGHIJKL: 
      my_api_key:
         value: {? '' : encrypted_key_value_for_env_1}
        type: secret
  MNOPQRSTUVWXYZ: 
      my_api_key:
         value: {? '' : encrypted_key_value_for_env_2}
        type: secret

in both cases, the secret services loads the dictionary of encrypted values based on anvil.app_id and the application code is simply

anvil.secrets.get_secret('my_api_key')

Thanks for any consideration.

Cheers,
Tyler

1 Like
Do the Callable & http_endpoint annotations have an impact on logging?
#2

make an app that is specifically for secrets (SecretsApp) and import as dependency to the different versions of your app. In a server module in SecretsApp make a get_secret function and return anvil.secrets.get_secret(secret_name).

from your development or production app, wherever you need the secret do something like this

import SecretsApp.ServerModule as secrets_app

twitter_secret = secrets_app.get_secret('twitter_api_key')

just make sure that you have removed the secrets from your dev/prod apps because thats where anvil looks first before looking in dependencies.

2 Likes
#3

@joinlook

Thanks for the suggestion.

However, I don’t see how that addresses my issue.

I.e this is the scenario.

External MongoDB TEST instance, {"db_url": "testdb.com", "API_KEY": 123}
External MongoDB PRODUCTION instance, {"db_url": "livedb.com", "API_KEY": 456}

Anvil Dev App

import SecretsApp.ServerModule as secrets_app

mongo_db_credentials =  json.loads(secrets_app.get_secret('mongo_db_key'))
print( mongo_db_credentials['db_url'],  mongo_db_credentials['API_KEY'])

Anvil Production App

import SecretsApp.ServerModule as secrets_app

mongo_db_credentials = json.loads(secrets_app.get_secret('mongo_db_key'))

print( mongo_db_credentials['db_url'],  mongo_db_credentials['API_KEY'])

I am using a print statement as psuedo equivalent as connect to external service, for demonstration purposes.

I am not following how my Production Anvil app will get the production connection url and key value of 456 but my Development Anvil app will get connection url for testing and key value of 123

I am trying to avoid having my server module aware of the backing services it is connecting to (such as a mongo DB in this example). For Example, In case I add a third stage to the development cycle to not just be Dev -> Prod, but Dev-> UAT -> Production or whatever.

I don’t see how shared SecretsApp gets me there, from what I can see, the shared secrets app, can hold only 1 set of credentials. Can you connect the dots for me ?

Unless you do a version of the algorithm suggested by Anvil Support (in original post above), which is basically to pass in the runtime app_id and maintain the mapping to secrets in the shared secret_app.

To me, this architecturally fits best as a runtime configuration. The same codebase can be run in multiple environments (accomplished with multiple apps and git push to multiple remotes).

So it makes a cleaner deployment to say run this code (based on a git commit) and to have similar config push that says “use these credentials / secrets at runtime”.

FWIW, I think I’ll end up at the @joinlook suggestion, but pass in app_id …so

so shared secrets has

def get_secrets(_app_id):
  if _app_id == "ABCDEFGHIJKL":
    data_key = anvil.secrets.get_secret('dev_mongo_credential')
  else:
    data_key = anvil.secrets.get_secret('prod_mongo_credential')
  return data_key

then my code will as sketched out will get me there.

So something like this

Anvil Dev App

import SecretsApp.ServerModule as secrets_app

mongo_db_credentials =  json.loads(secrets_app.get_secret(anvil.app.id, 'mongo_db_key'))
print( mongo_db_credentials['db_url'],  mongo_db_credentials['API_KEY'])

Anvil Production App

import SecretsApp.ServerModule as secrets_app

mongo_db_credentials = json.loads(secrets_app.get_secret(anvil.app.id, 'mongo_db_key'))

print( mongo_db_credentials['db_url'],  mongo_db_credentials['API_KEY'])

However, I think there is a better solution design possible. (Here, “Better” == “Agnostic to all Python Code”)

1 Like
#4

Yep – we get what you’re driving at, and the feature request is definitely in our issue tracker!

2 Likes
#5

Reviving this thread to add that this is needed to manage secrets with external collaboration.

External collaborators might use a config.yaml outside the repo (sidesteps this issue) or they might use the Anvil Editor, which forces them to define new secrets or override the existing ones (stored in the anvil.yaml within the repo).

Currently, this issue seems to be solved for internal collaboration by either using the multi-environment feature in the Professional plan or emailing support to use the same encryption key for multiple apps on the same account.

However, this isn’t necessarily a good idea for external collaboration and open source collaboration, where collaborators are on different accounts.

I’d want secrets to be stored outside of the repo and I’d sacrifice the version control.

Alternatively, Tyler’s suggestion is quite clever - tying the secrets in the anvil.yaml to the App ID. That would probably be easier for Anvil to implement.

1 Like
#6

Looks like we got it!

3 Likes