Browse Docs

S3 blockstorage

August 1, 2023 3 min read Docs Devops Cloud Storage
On this page

S3cmd command

S3cmd is a tool to handle blockstorage S3 type.

Install the command

 1# Ubuntu install 
 2sudo apt-get install s3cmd
 3
 4# Redhat install
 5sudo dnf install s3cmd
 6
 7# or from sources
 8wget https://sourceforge.net/projects/s3tools/files/s3cmd/2.2.0/s3cmd-2.2.0.tar.gz
 9tar xzf s3cmd-2.2.0.tar.gz
10cd s3cmd-2.2.0
11sudo python3 setup.py install

Configure it

  • From Cloud providers (for example DO):

  • Log in to the DigitalOcean Control Panel.

  • Navigate to API > Spaces Access Keys and generate a new key pair.

  • Note down the Access Key and Secret Key.

  • Auth and config:
 1s3cmd --configure 
 2
 3# Output
 4Access Key: xxxxxxxxxxxxxxxxxxxxxx
 5Secret Key: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
 6
 7Encryption password is used to protect your files from reading
 8by unauthorized persons while in transfer to S3
 9Encryption password: xxxxxxxxxx
10Path to GPG program [/usr/bin/gpg]:
11
12When using secure HTTPS protocol all communication with Amazon S3
13servers is protected from 3rd party eavesdropping. This method is
14slower than plain HTTP and can't be used if you're behind a proxy
15Use HTTPS protocol [No]: Yes
16
17New settings:
18  Access Key: xxxxxxxxxxxxxxxxxxxxxx
19  Secret Key: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
20  Encryption password: xxxxxxxxxx
21  Path to GPG program: /usr/bin/gpg
22  Use HTTPS protocol: True
23  HTTP Proxy server name:
24  HTTP Proxy server port: 0
25
26Test access with supplied credentials? [Y/n] Y
27Please wait, attempting to list all buckets...
28Success. Your access key and secret key worked fine :-)
29
30Now verifying that encryption works...
31Success. Encryption and decryption worked fine :-)
32
33Save settings? [y/N] y
34Configuration saved to '/root/.s3cfg'

Manage S3 buckets:

 1# List
 2s3cmd ls
 3
 4# Create if does not already exist
 5s3cmd info s3://terraform-backend-test || s3cmd mb s3://terraform-backend-test -v -d
 6
 7# Delete bucket 
 8s3cmd rb s3://terraform-backend-github-test --recursive
 9
10# Get size of the bucket
11s3cmd du s3://terraform-backend-test

Use s3cmd in a pipeline

  • in a github Workflows
 1jobs:
 2  backend:
 3    name: Backend
 4    runs-on: ubuntu-latest
 5
 6    steps:
 7      - name: Set up S3cmd cli tool
 8        uses: s3-actions/s3cmd@main
 9        with:
10          provider: digitalocean
11          region: FRA1
12          access_key: ${{secrets.DIGITALOCEAN_SPACES_ACCESS_TOKEN}}
13          secret_key: ${{secrets.DIGITALOCEAN_SPACES_SECRET_KEY}}
14
15      - name: Create Space Bucket
16        run: |
17          sed -i -e 's/signature_v2.*$/signature_v2 = True/' ~/.s3cfg
18          buck="github-action-${{ github.run_id }}"
19          s3cmd mb s3://$buck
20          sleep 10
21        continue-on-error: true

Mount S3 bucket as a FS - with s3fs

 1# conf epel repo on RHEL8 
 2cat < EOF > /etc/yum.repos.d/epel.repo
 3name = "Extra Packages for Enterprise Linux 8 - Release"
 4baseurl = "http://download.fedoraproject.org/pub/epel/8/Everything/$basearch"
 5enabled = true
 6failovermethod = "priority"
 7gpgcheck = true
 8gpgkey = "http://download.fedoraproject.org/pub/epel/RPM-GPG-KEY-EPEL-8"
 9EOF
10
11# install packages
12dnf install epel-release s3fs-fuse
13
14# Create passwd file
15echo "${spaces_access_key_id}:${spaces_access_key_secret}" > /etc/passwd-s3fs
16chown root:root /etc/passwd-s3fs
17chmod 0600 /etc/passwd-s3fs
18
19# Mount
20mkdir /mnt/point
21s3fs ${bucket_name} ${mount_point} -o url=https://${region}.digitaloceanspaces.com",
Thursday, January 15, 2026 Tuesday, August 1, 2023
CEPH