Deploying 1Password Connect and External Secrets Operator with Flux and not die trying
Recently I spent some time trying to deploy External Secrets operator + 1Password connect on my on-premise Kubernetes cluster to manage secrets on it. This is a small guide explaining how I was able to deploy it since I had a lot of issues and drawbacks along the way.
I already own a 1Password license so I figured it would not only be a nice tools to handle secrets and I also won't need to pay for it. 1Password provides their own full 1Password Kubernetes operator but I chose to go with the more generic and widely adopted External Secrets Operator, mainly because by using the latter I am able to also retrieve secrets from other sources in the future if I need to.
Because of this approach, I would only need to deploy the 1Password Connect server and not their whole operator. This application is the one that is actually going to be talking with the 1Password servers and retrieving the secrets. The operator will be just talking to this application in the cluster.
First approach to deployment
The first logical way to deploy something is to take a look at their docs.
1Password provides their own 1Password Connect deployment guide but here is where the problem starts.
If you follow the guide, you will end up creating two things:
1password-credentials.jsonfile: The file that has your credentials and is the one you 1Password Connect server will use to connect to your vaultsaccess-token: A token that is used for applications in you own cluster to be able to make requests to your Connect server
After creating this things, the guide explains how to deploy it via Helm and provide the base64 encoded values of the credentials file to the chart via the connect.credentials key. If using Helm it is a one-liner as stated in the docs:
helm install connect 1password/connect --set-file connect.credentials=1password-credentials.json
That being said, we can not use a file to provide secrets via Flux, so we need to look for alternatives on how to pass this secret to a HelmRelease
Trying to deploy it with Flux
One solution that would totally make sense is to try to create the secret manually and use that secret for the connect.credentials value in the HelmRelease created via Flux.
You can easily encode the content of the file by running:
cat 1password-credentials.json | base64 -w 0
And then put the content in a secret YAML file or just create the secret via the Kubernetes CLI. Then your HelmRelease would look something like:
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
name: 1password-connect
namespace: security
spec:
interval: 10m
chart:
spec:
chart: connect
version: 1.16.0
sourceRef:
kind: HelmRepository
name: 1password-charts
namespace: flux-system
valuesFrom:
- kind: Secret
name: op-credentials # The name you chose for the secret
valuesKey: op-credentials # The key that has the value
targetPath: connect.credentials_base64
However, if you have the same issue as me you might have realized that just simply does not work.
One first issue is that the HelmRelease reconciliation will simply not be succesful as it will struggle to read the secret. After browsing around a little bit, these two GitHub issues caught my attention:
They mention that the Helm chart expect the values of the file base64 encoded themselves. This means you would need to base64 encode the contents of the credentials file, and then base64 encode that again. Then put that double base64 encoded value in the secret.
Even after doing that, there are still issues. They HelmRelease might be successful but I started having issues when trying to do a request to the endpoints of the connect server, for example:
curl --location 'http://<node_ip>:<nodeport>/v1/vaults' --header 'Authorization: <the token you generated>'
I kept getting a 500 error. When inspecting the log, it was not the pods of the connect servers the ones that were having issues reading the secret. Apparently there was something wrong with the base64 encoding.
After reading the docs of the onepassword-connect Helm chart I realized that there is a value that you can pass to the chart that is not mentioned anywhere in other docs or GitHub issues. This value is connect.credentials_base64. This one was the key to finally make everything work.
How I ended up deploying it
This is the important part of this article. An step by step guide on how I was able to deploy this setup.
In case you want to take a look at a final version of this setup, you can check my Kubernetes homelab repo: crisszkutnik/home-lab@87c4a01
1. Base64 encode the contents of you 1password-credentials.json file twice
cat 1password-credentials.json | base64 -w 0 | base64 -w 0
2. Create a secret for that value
apiVersion: v1
kind: Secret
metadata:
name: op-credentials
namespace: security # Replace for whatever namespace you want
type: Opaque
data:
op-credentials: <base64-encoded-content>
3. Create a HelmRelease and set connect.credentials_base64
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
name: 1password-connect
namespace: security # Replace for whatever namespace you want
spec:
interval: 10m
chart:
spec:
chart: connect
version: 1.16.0
sourceRef:
kind: HelmRepository
name: 1password-charts
namespace: flux-system
valuesFrom:
- kind: Secret
name: op-credentials
valuesKey: op-credentials
targetPath: connect.credentials_base64
4. Create token secret for it to be used by the External Secrets operator
apiVersion: v1
kind: Secret
metadata:
name: external-secrets-operator-op-access-token
namespace: security # Replace for whatever namespace you want
type: Opaque
stringData:
token: <token>
5. Deploy your SecretStore or ClusterSecretStore that references your 1Password Connect deployment
apiVersion: external-secrets.io/v1beta1
kind: ClusterSecretStore
metadata:
name: 1password-secrets-store
namespace: security
spec:
provider:
onepassword:
connectHost: <1password-connect-cluster-url>
vaults:
# You should put all the vaults you want
# External secrets operator to check and
# the order you want the vaults to be inspected in
name_of_vault_1: 1
name_of_vault_2: 2
...
auth:
secretRef:
connectTokenSecretRef:
name: external-secrets-operator-op-access-token
key: token
namespace: security