Securing Kolla Ansible passwords with Hashicorp Vault

Written by Scott Solkhon (Cloud Engineer)

Kolla Ansible is a production-ready tool for deploying and maintaining containerised OpenStack clouds. Operators can easily deploy a vanilla OpenStack cloud with very little config changes out of the box and as requirements change, OpenStack configuration can be tweaked to suit your needs. All of this is deployed using the Kolla Ansible command line interface (cli), which wraps around Ansible playbooks and some custom Python scripts.

When running a production service you need to ensure that your credentials are kept safe from unwanted access. Using a centralised credential management service such as Hashicorp Vault, you are able to control the access to your credentials through the use of tokens, approles, and policies.

By combining Kolla Ansible’s deployment process and Hashicorp Vault’s credential management you can deploy a secure OpenStack cloud – potentially without having to write a single line of code yourself!

Generating Kolla passwords

The Kolla Ansible cli allows an operator to generate a full set of randomised passwords by using the ‘kolla-genpwd’ command:

$ cp kolla-ansible/etc/kolla/passwords.yml /etc/kolla/
$ kolla-genpwd
$ cat /etc/kolla/passwords.yml
aodh_database_password: acK1KZ1tulbzw3RjKrQC5zyxDrXMxKbHxYJR1ebX
aodh_keystone_password: 3NQDmG7PQPLV5NGg4onieMwAEoSGSDFb7fEJ5N5T
barbican_crypto_key: PugFHSE-U2cwLCqKojrltSuoGNWrzXD9gGk_XvP1Nbc=
barbican_database_password: lidQNGCxMnuXLNpggmtijYrRTAuXIBbdJoPCjtJx
barbican_keystone_password: eSacePFcfBxMs5fPysg44DEqzjwrPeMO8PbFaPKM
barbican_p11_password: ikO6saciMsYFGN5I17vmwPeOZvKLb0294fnCSeKH
...

In the above example, the ‘kolla-genpwd’ command takes a yaml file with a set of passwords that need to be generated and outputs the generated passwords into the same file. A template for the passwords needed in a typical Kolla Ansible deployment can be found in ‘kolla-ansible/etc/kolla/passwords.yml’ ¹.

Setting up Hashicorp Vault

The configuration and lockdown of your Vault policies and approles will largely depend on the deployment of your Hashicorp Vault server, but for the purpose of demonstration, I will include an example approle called “kolla”, which has write access into a key value (KV) secrets engine called “production”:

1) Administrator sets up the approle and policy:

$ vault auth enable approle
$ cat << EOF | vault policy write policy-kolla-ansible -
path "production/*" {
  capabilities = ["create", "read", "update", "delete", "list"]
}
EOF
$ vault write auth/approle/role/kolla 
    secret_id_ttl=10m 
    token_ttl=20m 
    token_max_ttl=30m 
    token_policies=policy-kolla-ansible

2) Operator generates a role-id and secret-id to authenticate to Vault:

$ vault read auth/approle/role/kolla/role-id
role_id     8f7ca1ff-8e5c-a924-3314-521dcbab304d

$ vault write -f auth/approle/role/kolla/secret-id
secret_id               fb932ccd-381e-188b-c1dd-a7ace9dd1be4
secret_id_accessor      079f7937-4697-40be-afaf-18bf63be230a
secret_id_ttl           10m

This approle could be securely stored in a deployment server, such as Jenkins, to allow your credential management to be fully automated.

Writing the passwords

Now that we have a set of passwords we need to write them into our Vault KV using the ‘kolla-writepwd’ command:

$ kolla-writepwd 
    --passwords /etc/kolla/passwords.yml 
    --vault-addr 'https://vault.example.com' 
    --vault-role-id 8f7ca1ff-8e5c-a924-3314-521dcbab304d 
    --vault-secret-id fb932ccd-381e-188b-c1dd-a7ace9dd1be4 
    --vault-mount-point production

In Vault this would look like:

$ vault kv list production/kolla_passwords
Keys
----
aodh_database_password
aodh_keystone_password
barbican_crypto_key
barbican_database_password
barbican_keystone_password
barbican_p11_password
...

$ vault kv get secret/kolla_passwords/aodh_database_password
====== Metadata ======
Key              Value
---              -----
created_time     2021-06-27T18:30:08.405201929Z
deletion_time    n/a
destroyed        false
version          1

====== Data ======
Key         Value
---         -----
password    acK1KZ1tulbzw3RjKrQC5zyxDrXMxKbHxYJR1ebX

Reading the passwords

Finally, when we want to update our Kolla Ansible deployment we can read the passwords back from Vault and generate a passwords.yml file using the ‘kolla-readpwd’ command:

$ cp kolla-ansible/etc/kolla/passwords.yml /etc/kolla/passwords.yml
$ kolla-readpwd 
    --passwords /etc/kolla/passwords.yml 
    --vault-addr 'https://vault.example.com' 
    --vault-role-id 8f7ca1ff-8e5c-a924-3314-521dcbab304d 
    --vault-secret-id fb932ccd-381e-188b-c1dd-a7ace9dd1be4 
    --vault-mount-point production

This will read the passwords from our Vault KV and populate the ‘/etc/kolla/passwords.yml’ file to look like the file we previously had to generate the passwords.

By combining Kolla Ansible and Hashicorp Vault we have a straightforward process for storing sensitive passwords for an OpenStack deployment.

The feature to support Hashicorp Vault for storing Kolla Ansible passwords was developed by G-Research as part of securing our private cloud. We work very closely with the open source community to regularly push back changes that will benefit the wider community. If you would like to see the code for this feature it is available on the link below:

https://github.com/openstack/kolla-ansible/commit/6bf74aa20d268f11f676a0e9affa92e3022b595d

References

[1] – https://github.com/openstack/kolla-ansible/blob/master/etc/kolla/passwords.yml