⛄ Own your terraform state (with s3cmd)

Może Bałtyk May 5, 2025 9 min read Posts Devops Terraform Devops
On this page
⛄ Own your terraform state (with s3cmd)

What if you do not want to use Terraform cloud ?…

Recently I was reading this article about Terraform Cloud, and remembered that I went throught same issue when writing my github Workflows…

The issue…

When using CI, each job is a runner, so new triggered container for each step of the pipeline. So the terraform.tfstate is lost between the pipeline steps. In my case, I was deploying on Digital Ocean providers, willing to store the state on S3 bucket created at the start of the pipeline and destroyed at the end.

I wrote a documentation about s3cmd which cover 3 differents use cases (manual usage; in a github workflow; or mounting S3 bucket as a FS) in docs/storage/s3. The purpose of this article is not to repeat the documentation but rather underline the fact that we don’t need this dependence to Terraform Cloud and another account somewhere else, when s3cmd can help us to do the job.

The solution!

The big steps

  • Install and init a connection with s3cmd
  • Create a bucket
  • Use it as your terrafrom backend
  • Cleanup after if needed.

Inside a Github Workflows:

This is pretty usefull in a pipeline, but do not forget to include a job to cleanup everything once done…

  • Init the vars :
1env:
2  DO_PAT: ${{secrets.DIGITALOCEAN_ACCESS_TOKEN}}
3  AWS_ACCESS_KEY_ID: ${{secrets.DIGITALOCEAN_SPACES_ACCESS_TOKEN}}
4  AWS_SECRET_ACCESS_KEY: ${{secrets.DIGITALOCEAN_SPACES_SECRET_KEY}}
5  REGION: ${{secrets.DIGITALOCEAN_REGION}}
6  MOUNT_POINT: "/opt/rkub"
7  BUCKET: "rkub-github-action-${{ github.run_id }}"
  • Create a bucket directly on your provider :
 1    steps:
 2      - name: Set up S3cmd cli tool
 3        uses: s3-actions/s3cmd@main
 4        with:
 5          provider: digitalocean
 6          region: ${{secrets.DIGITALOCEAN_REGION}}
 7          access_key: ${{secrets.DIGITALOCEAN_SPACES_ACCESS_TOKEN}}
 8          secret_key: ${{secrets.DIGITALOCEAN_SPACES_SECRET_KEY}}
 9
10      - name: Create Space Bucket
11        run: |
12          ## sed -i -e 's/signature_v2.*$/signature_v2 = True/' ~/.s3cfg
13          ## sed -i -e 's/signature_v2.*$/signature_v2 = True/' /home/runner/work/_temp/s3cmd.conf
14          if [[ $BUCKET != "terraform-backend-github" ]]; then s3cmd mb s3://${BUCKET}; fi
15          sleep 10
  • Then use it as terraform/opentofu backend:
 1    steps:
 2      - name: Checkout files
 3        uses: actions/checkout@v4
 4
 5      - name: Setup Tofu
 6        uses: opentofu/setup-opentofu@v1
 7        with:
 8          tofu_version: "1.7.3"
 9
10      - name: Tofu Init
11        id: init
12        run: |
13          cd ./DO/infra
14          tofu init -backend-config="bucket=${BUCKET}"
15
16      - name: Tofu Validate
17        id: validate
18        run: |
19          cd ./DO/infra
20          tofu validate -no-color
21
22      - name: Tofu Plan
23        id: plan
24        run: |
25          cd ./DO/infra
26          tofu plan -out=terraform.tfplan \
27          -var "GITHUB_RUN_ID=$GITHUB_RUN_ID" \
28          -var "token=${DO_PAT}" \
29          -var "worker_count=${WORKER_COUNT}" \
30          -var "controller_count=${CONTROLLER_COUNT}" \
31          -var "instance_size=${SIZE}" \
32          -var "spaces_access_key_id=${{secrets.DIGITALOCEAN_SPACES_ACCESS_TOKEN}}" \
33          -var "spaces_access_key_secret=${{secrets.DIGITALOCEAN_SPACES_SECRET_KEY}}" \
34          -var "mount_point=${MOUNT_POINT}" \
35          -var "airgap=${AIRGAP}" \
36          -var "terraform_backend_bucket_name=${BUCKET}"
37        continue-on-error: true
38
39      - name: Tofu Plan Status
40        if: steps.plan.outcome == 'failure'
41        run: exit 1
42
43      - name: Tofu Apply
44        run: |
45          cd ./DO/infra
46          tofu apply terraform.tfplan

Doing so, I was able to pass the state throught the different jobs in my pipeline. Instead of doing one big job with all the step, I found this solution more elegent.

The full Pipeline

Below the full pipeline Github Workflow from the Rkub project:

  1---
  2name: Stage online install
  3
  4on:
  5  workflow_dispatch:
  6
  7env:
  8  DO_PAT: ${{secrets.DIGITALOCEAN_ACCESS_TOKEN}}
  9  AWS_ACCESS_KEY_ID: ${{secrets.DIGITALOCEAN_SPACES_ACCESS_TOKEN}}
 10  AWS_SECRET_ACCESS_KEY: ${{secrets.DIGITALOCEAN_SPACES_SECRET_KEY}}
 11  REGION: ${{secrets.DIGITALOCEAN_REGION}}
 12  MOUNT_POINT: "/opt/rkub"
 13  BUCKET: "rkub-github-action-${{ github.run_id }}"
 14  #BUCKET: "terraform-backend-github"
 15  CONTROLLER_COUNT: "1"
 16  WORKER_COUNT: "1"
 17  SIZE: "s-2vcpu-4gb"
 18  AIRGAP: "false"
 19
 20jobs:
 21  bucket:
 22    name: Bucket
 23    runs-on: ubuntu-latest
 24    timeout-minutes: 10
 25
 26    steps:
 27      - name: Set up S3cmd cli tool
 28        uses: s3-actions/s3cmd@main
 29        with:
 30          provider: digitalocean
 31          region: ${{secrets.DIGITALOCEAN_REGION}}
 32          access_key: ${{secrets.DIGITALOCEAN_SPACES_ACCESS_TOKEN}}
 33          secret_key: ${{secrets.DIGITALOCEAN_SPACES_SECRET_KEY}}
 34
 35      - name: Create Space Bucket
 36        run: |
 37          ## sed -i -e 's/signature_v2.*$/signature_v2 = True/' ~/.s3cfg
 38          ## sed -i -e 's/signature_v2.*$/signature_v2 = True/' /home/runner/work/_temp/s3cmd.conf
 39          if [[ $BUCKET != "terraform-backend-github" ]]; then s3cmd mb s3://${BUCKET}; fi
 40          sleep 10
 41
 42  deploy:
 43    name: Deploy
 44    runs-on: ubuntu-latest
 45    needs: [ Bucket ]
 46    timeout-minutes: 20
 47
 48    defaults:
 49      run:
 50        shell: bash
 51        working-directory: ./test
 52
 53    steps:
 54      - name: Checkout files
 55        uses: actions/checkout@v4
 56
 57      - name: Setup Tofu
 58        uses: opentofu/setup-opentofu@v1
 59        with:
 60          tofu_version: "1.7.3"
 61
 62      - name: Tofu Init
 63        id: init
 64        run: |
 65          cd ./DO/infra
 66          tofu init -backend-config="bucket=${BUCKET}"
 67
 68      - name: Tofu Validate
 69        id: validate
 70        run: |
 71          cd ./DO/infra
 72          tofu validate -no-color
 73
 74      - name: Tofu Plan
 75        id: plan
 76        run: |
 77          cd ./DO/infra
 78          tofu plan -out=terraform.tfplan \
 79          -var "GITHUB_RUN_ID=$GITHUB_RUN_ID" \
 80          -var "token=${DO_PAT}" \
 81          -var "worker_count=${WORKER_COUNT}" \
 82          -var "controller_count=${CONTROLLER_COUNT}" \
 83          -var "instance_size=${SIZE}" \
 84          -var "spaces_access_key_id=${{secrets.DIGITALOCEAN_SPACES_ACCESS_TOKEN}}" \
 85          -var "spaces_access_key_secret=${{secrets.DIGITALOCEAN_SPACES_SECRET_KEY}}" \
 86          -var "mount_point=${MOUNT_POINT}" \
 87          -var "airgap=${AIRGAP}" \
 88          -var "terraform_backend_bucket_name=${BUCKET}"
 89        continue-on-error: true
 90
 91      - name: Tofu Plan Status
 92        if: steps.plan.outcome == 'failure'
 93        run: exit 1
 94
 95      - name: Tofu Apply
 96        run: |
 97          cd ./DO/infra
 98          tofu apply terraform.tfplan
 99
100      # Save Artifacts
101      - name: Install s3fs-fuse on Ubuntu
102        run: |
103          sudo apt-get install -y s3fs
104
105      - name: Mount Space Bucket
106        run: |
107          echo "${{secrets.DIGITALOCEAN_SPACES_ACCESS_TOKEN}}:${{secrets.DIGITALOCEAN_SPACES_SECRET_KEY}}" > ./passwd-s3fs
108          chmod 600 ./passwd-s3fs
109          mkdir -p ${MOUNT_POINT}
110          s3fs ${BUCKET} ${MOUNT_POINT} -o url=https://${REGION}.digitaloceanspaces.com -o passwd_file=./passwd-s3fs
111          df -Th ${MOUNT_POINT}
112
113      - name: Save files
114        run: |
115          cp ${{ github.workspace }}/test/inventory/hosts.ini ${MOUNT_POINT}/hosts.ini
116          cp ${{ github.workspace }}/test/DO/infra/.key.private ${MOUNT_POINT}/.key.private
117
118  reachable:
119    name: Reachable
120    runs-on: ubuntu-latest
121    needs: [ Deploy ]
122    timeout-minutes: 10
123
124    defaults:
125      run:
126        shell: bash
127        working-directory: ./test
128
129    steps:
130      - name: Checkout files
131        uses: actions/checkout@v4
132
133      # Get Artifacts 
134      - name: Install s3fs-fuse on Ubuntu
135        run: |
136          sudo apt-get install -y s3fs
137
138      - name: Mount Space Bucket
139        run: |
140          echo "${{secrets.DIGITALOCEAN_SPACES_ACCESS_TOKEN}}:${{secrets.DIGITALOCEAN_SPACES_SECRET_KEY}}" > ./passwd-s3fs
141          chmod 600 ./passwd-s3fs
142          mkdir -p ${MOUNT_POINT}
143          s3fs ${BUCKET} ${MOUNT_POINT} -o url=https://${REGION}.digitaloceanspaces.com -o passwd_file=./passwd-s3fs
144          df -Th ${MOUNT_POINT}
145
146      - name: Get Artificats
147        run: |
148          cp ${MOUNT_POINT}/hosts.ini ${{ github.workspace }}/test/inventory/hosts.ini
149          cp ${MOUNT_POINT}/.key.private ${{ github.workspace }}/test/DO/infra/.key.private
150
151      # Test
152      - name: Set up Python
153        id: setup_python
154        uses: actions/setup-python@v5
155        with:
156          python-version: 3.12
157
158      - name: Install dependencies
159        run: |
160          python3 -m pip install --upgrade pip
161          python3 -m pip install "ansible-core>=2.15,<2.17"
162          ansible --version
163
164      - name: Test if reachable
165        run: |
166          ANSIBLE_HOST_KEY_CHECKING=False ansible RKE2_CLUSTER -m ping -u root
167
168      - name: Wait for cloud-init to finish
169        run: |
170          ANSIBLE_HOST_KEY_CHECKING=False ansible RKE2_CLUSTER -m shell -a "cloud-init status --wait" -u root -v
171
172  install:
173    name: Install
174    runs-on: ubuntu-latest
175    needs: [ Reachable ]
176    timeout-minutes: 60
177
178    defaults:
179      run:
180        shell: bash
181        working-directory: ./test
182
183    steps:
184      - name: Checkout files
185        uses: actions/checkout@v4
186
187      - name: Install requirements
188        run: |
189          cd ..
190          make prerequis
191          ansible --version
192
193      - name: Install dependencies
194        run: |
195          python3 -m pip install --upgrade pip
196          python3 -m pip install "ansible-core>=2.15,<2.17"
197          ansible --version
198
199      # Get Artifacts 
200      - name: Install s3fs-fuse on Ubuntu
201        run: |
202          sudo apt-get install -y s3fs
203
204      - name: Mount Space Bucket
205        run: |
206          echo "${{secrets.DIGITALOCEAN_SPACES_ACCESS_TOKEN}}:${{secrets.DIGITALOCEAN_SPACES_SECRET_KEY}}" > ./passwd-s3fs
207          chmod 600 ./passwd-s3fs
208          mkdir -p ${MOUNT_POINT}
209          s3fs ${BUCKET} ${MOUNT_POINT} -o url=https://${REGION}.digitaloceanspaces.com -o passwd_file=./passwd-s3fs
210          df -Th ${MOUNT_POINT}
211
212      - name: Get Artificats
213        run: |
214          cp ${MOUNT_POINT}/hosts.ini ${{ github.workspace }}/test/inventory/hosts.ini
215          cp ${MOUNT_POINT}/.key.private ${{ github.workspace }}/test/DO/infra/.key.private
216
217      # Install
218      - name: Run playbook install.yml
219        run: |
220          ANSIBLE_HOST_KEY_CHECKING=False ansible-playbook -u root playbooks/install.yml -e "airgap=false" -e "method=tarball"
221
222      #- name: Run playbook rancher.yml
223      #  run: |
224      #    ANSIBLE_HOST_KEY_CHECKING=False ansible-playbook -u root playbooks/rancher.yml
225
226      #- name: Run playbook longhorn.yml
227      #  run: |
228      #    ANSIBLE_HOST_KEY_CHECKING=False ansible-playbook -u root playbooks/longhorn.yml
229
230      #- name: Run playbook neuvector.yml
231      #  run: |
232      #    ANSIBLE_HOST_KEY_CHECKING=False ansible-playbook -u root playbooks/neuvector.yml
233
234  test:
235    name: Test
236    runs-on: ubuntu-latest
237    needs: [ Install ]
238    timeout-minutes: 10
239
240    defaults:
241      run:
242        shell: bash
243        working-directory: ./test
244
245    steps:
246      - name: Checkout files
247        uses: actions/checkout@v4
248
249      # Get Artifacts 
250      - name: Install s3fs-fuse on Ubuntu
251        run: |
252          sudo apt-get install -y s3fs
253
254      - name: Mount Space Bucket
255        run: |
256          echo "${{secrets.DIGITALOCEAN_SPACES_ACCESS_TOKEN}}:${{secrets.DIGITALOCEAN_SPACES_SECRET_KEY}}" > ./passwd-s3fs
257          chmod 600 ./passwd-s3fs
258          mkdir -p ${MOUNT_POINT}
259          s3fs ${BUCKET} ${MOUNT_POINT} -o url=https://${REGION}.digitaloceanspaces.com -o passwd_file=./passwd-s3fs
260          df -Th ${MOUNT_POINT}
261
262      - name: Get Artificats
263        run: |
264          cp ${MOUNT_POINT}/hosts.ini ${{ github.workspace }}/test/inventory/hosts.ini
265          cp ${MOUNT_POINT}/.key.private ${{ github.workspace }}/test/DO/infra/.key.private
266
267      # Test
268      - name: Install dependencies
269        run: |
270          python3 -m pip install --upgrade pip
271          python3 -m pip install "ansible-core>=2.15,<2.17"
272          python3 -m pip install -U pytest-testinfra pytest-sugar pytest
273          ansible --version
274
275      - name: Run Python Tests
276        run: |
277          export DEFAULT_PRIVATE_KEY_FILE=.key
278          python3 -m pytest --hosts=RKE2_CONTROLLERS --ansible-inventory=inventory/hosts.ini --force-ansible --connection=ansible basic_server_tests.py
279          python3 -m pytest --hosts=RKE2_WORKERS --ansible-inventory=inventory/hosts.ini --force-ansible --connection=ansible basic_agent_tests.py
280
281  delay:
282    name: Delay
283    runs-on: ubuntu-latest
284    needs: [ Test ]
285    if: always()
286
287    steps:
288      - name: Delay 10min
289        uses: whatnick/wait-action@master
290        with:
291          time: '600s'
292
293  cleanup:
294    name: Cleanup
295    runs-on: ubuntu-latest
296    needs: [ Delay ]
297    if: always()
298    timeout-minutes: 30
299
300    defaults:
301      run:
302        shell: bash
303        working-directory: ./test/DO/infra
304
305    steps:
306      - name: Checkout files
307        uses: actions/checkout@v4
308
309      - name: Setup Tofu
310        uses: opentofu/setup-opentofu@v1
311        with:
312          tofu_version: "1.7.3"
313
314      - name: Tofu Init
315        id: init
316        run: |
317          tofu init -backend-config="bucket=${BUCKET}"
318        continue-on-error: true
319
320      - name: Tofu plan delete stack
321        id: plan
322        run: |
323          tofu plan -destroy -out=terraform.tfplan \
324          -var "GITHUB_RUN_ID=$GITHUB_RUN_ID" \
325          -var "token=${DO_PAT}" \
326          -var "worker_count=${WORKER_COUNT}" \
327          -var "controller_count=${CONTROLLER_COUNT}" \
328          -var "instance_size=${SIZE}" \
329          -var "spaces_access_key_id=${{secrets.DIGITALOCEAN_SPACES_ACCESS_TOKEN}}" \
330          -var "spaces_access_key_secret=${{secrets.DIGITALOCEAN_SPACES_SECRET_KEY}}" \
331          -var "mount_point=${MOUNT_POINT}" \
332          -var "airgap=${AIRGAP}" \
333          -var "terraform_backend_bucket_name=${BUCKET}"
334        continue-on-error: true
335
336      - name: Tofu Apply
337        run: |
338          tofu apply terraform.tfplan
339        continue-on-error: true
340
341      - name: Set up S3cmd cli tool
342        uses: s3-actions/s3cmd@main
343        with:
344          provider: digitalocean
345          region: ${{secrets.DIGITALOCEAN_REGION}}
346          access_key: ${{secrets.DIGITALOCEAN_SPACES_ACCESS_TOKEN}}
347          secret_key: ${{secrets.DIGITALOCEAN_SPACES_SECRET_KEY}}
348
349      - name: Remove Space bucket
350        run: |
351          ## sed -i -e 's/signature_v2.*$/signature_v2 = True/' ~/.s3cfg
352          ## sed -i -e 's/signature_v2.*$/signature_v2 = True/' /home/runner/work/_temp/s3cmd.conf
353          if [[ $BUCKET != "terraform-backend-github" ]]; then s3cmd rb s3://${BUCKET} --recursive; fi
354          sleep 10
Thursday, January 15, 2026 Monday, May 5, 2025
🎉 The Beauty of WSL
🚀 The Homelab Journey
👺 The Bad, the Good and the Ugly Git
👺 The Bad, the Good and the Ugly Git
🚀 The Homelab Journey
🚀 The Homelab Journey
🎉 The Beauty of WSL
🎉 The Beauty of WSL
Terraform
Terraform
👷👮 Makefile VS Justfile
👷👮 Makefile VS Justfile
🎺 How to Create Custom Widget
🎺 How to Create Custom Widget
🌌 How to Create this Blog
🌌 How to Create this Blog
Projects
Projects
CEPH
CEPH
Collection
Collection