Azure MinIO gateway when using the GitLab chart | GitLab
Storage Account
Deploy MinIO to Web App on Linux
Conclusion
Reference
Azure MinIO gateway when using the GitLab chart
MinIO is an object storage server that exposes S3-compatible APIs and it has a gateway feature that allows proxying requests to Azure Blob Storage. To setup our gateway, we will make use of Azure’s Web App on Linux.
To get started, make sure you have installed Azure CLI and you are logged in (
az login
). Proceed to create a Resource group, if you don’t have one already:
az group create --name"gitlab-azure-minio"--location"WestUS"
Storage Account
Create a Storage account in your resource group, the name of the storage account must be globally unique:
First, we need to create an App Service Plan in the same resource group.
az appservice plan create \ --name"gitlab-azure-minio-app-plan"\ --is-linux\ --sku B1 \ --resource-group"gitlab-azure-minio"\ --location"WestUS"
Create a Web app configured with the
minio/minio
Docker container, the name you specify will be used in the URL of the web app:
az webapp create \ --name"gitlab-minio-app"\ --deployment-container-image-name"minio/minio"\ --plan"gitlab-azure-minio-app-plan"\ --resource-group"gitlab-azure-minio"
The Web app should now be accessible at
https://gitlab-minio-app.azurewebsites.net
.
Lastly, we need to set up the startup command and create environment variables that will store our storage account name and key for use by the web app, MINIO_ACCESS_KEY & MINIO_SECRET_KEY.
az webapp config appsettings set\ --settings"MINIO_ACCESS_KEY=gitlab-azure-minio-storage""MINIO_SECRET_KEY=h0tSyeTebs+...""PORT=9000"\ --name"gitlab-minio-app"\ --resource-group"gitlab-azure-minio"
You can proceed to use this gateway with any client with s3-compability. Your web application URL will be the
s3 endpoint
, storage account name will be your
accesskey
, and storage account key will be your
secretkey
.
Reference
This guide was adapted for posterity from Alessandro Segala’s blog post on same topic.
Configure the GitLab chart with an external object storage | GitLab
Azure Blob: URL [FILTERED] is blocked: Requests to the local network are not allowed
Configure the GitLab chart with an external object storage
GitLab relies on object storage for highly-available persistent data in Kubernetes.
By default, an S3-compatible storage solution named
minio
is deployed with the
chart. For production quality deployments, we recommend using a hosted
object storage solution like Google Cloud Storage or AWS S3.
To disable MinIO, set this option and then follow the related documentation below:
--set global.minio.enabled=false
An example of the full configuration
has been provided in the examples.
This documentation specifies usage of access and secret keys for AWS. It is also possible to use IAM roles.
S3 encryption
Introduced in GitLab 13.4.
GitLab supports Amazon KMS
to encrypt data stored in S3 buckets.
You can enable this in two ways:
In AWS, configure the S3 bucket to use default encryption.
In GitLab, enable server side encryption headers.
These two options are not mutually exclusive. You can set a default encryption
policy, but also enable server-side encryption headers to override those defaults.
See the GitLab documentation on encrypted S3 buckets
for more details.
Azure Blob Storage
Introduced in GitLab 13.4.
Direct support for Azure Blob storage is available for
uploaded attachments, CI job artifacts, LFS, and other object types supported via the consolidated settings. In previous GitLab versions, an Azure MinIO gateway was needed.
The Azure MinIO gateway is still needed for backups. Follow this issue
for more details.
GitLab does not support the Azure MinIO gateway as the storage for the Docker Registry.
Please refer to the corresponding Azure example when setting up the Docker Registry.
Although Azure uses the word container to denote a collection of blobs,
GitLab standardizes on the term bucket.
Azure Blob storage requires the use of the
consolidated object storage settings. A
single Azure storage account name and key must be used across multiple
Azure blob containers. Customizing individual
connection
settings by
object type (for example,
artifacts
,
uploads
, and so on) is not permitted.
To enable Azure Blob storage, see
rails.azurerm.yaml
as an example to define the Azure
connection
. You can load this as a
secret via:
Note
: The bucket name needs to be set both in the secret, and in
global.registry.bucket
. The secret is used in the registry server, and
the global is used by GitLab backups.
Create the secret per registry chart documentation on storage, then configure the chart to make use of this secret.
Examples for S3(S3 compatible storages, but Azure MinIO gateway not supported, see Azure Blob Storage), Azure and GCS drivers can be found in
examples/objectstorage
.
registry.s3.yaml
registry.gcs.yaml
registry.azure.yaml
Registry configuration
Decide on which storage service to use.
Copy appropriate file to
registry-storage.yaml
.
Edit with the correct values for the environment.
Follow registry chart documentation on storage for creating the secret.
Configuration of object storage for LFS, artifacts, uploads, packages, external
diffs, and pseudonymizer is done via the following keys:
global.appConfig.lfs
global.appConfig.artifacts
global.appConfig.uploads
global.appConfig.packages
global.appConfig.externalDiffs
global.appConfig.dependencyProxy
Note also that:
A different bucket is needed for each, otherwise performing a restore from
backup doesn’t function properly.
Storing MR diffs on external storage is not enabled by default, so,
for the object storage settings for
externalDiffs
to take effect,
global.appConfig.externalDiffs.enabled
key should have a
true
value.
The dependency proxy feature is not enabled by default, so,
for the object storage settings for
dependencyProxy
to take effect,
global.appConfig.dependencyProxy.enabled
key should have a
true
value.
See the charts/globals documentation on appConfig for full details.
Create the secret(s) per the connection details documentation, and then configure the chart to use the provided secrets. Note, the same secret can be used for all of them.
Examples for AWS (any S3 compatible like Azure using MinIO) and Google providers can be found in
examples/objectstorage
.
rails.s3.yaml
rails.gcs.yaml
rails.azure.yaml
rails.azurerm.yaml
appConfig configuration
Decide on which storage service to use.
Copy appropriate file to
rails.yaml
.
Edit with the correct values for the environment.
Follow connection details documentation for creating the secret.
Configure the chart as documented.
Backups
Backups are also stored in object storage, and must be configured to point
externally rather than the included MinIO service. The backup/restore procedure uses two separate buckets:
A bucket for storing backups (
global.appConfig.backups.bucket
)
A temporary bucket for preserving existing data during the restore process (
global.appConfig.backups.tmpBucket
)
AWS S3-compatible object storage systems and Google Cloud Storage are supported backends.
You can configure the backend type by setting
global.appConfig.backups.objectStorage.backend
to
s3
for AWS S3 or
gcs
for Google Cloud Storage.
You must also provide a connection configuration through the
gitlab.toolbox.backups.objectStorage.config
key.
When using Google Cloud Storage, the GCP project must be set with the
global.appConfig.backups.objectStorage.config.gcpProject
value.
See the backup/restore object storage documentation for full details.
To backup or restore files from the other object storage locations, the configuration file needs to be
configured to authenticate as a user with sufficient access to read/write to all GitLab buckets.
Backups storage example
Create the
storage.config
file:
On Amazon S3, the contents should be in the s3cmd configuration file format
[default] access_key=BOGUS_ACCESS_KEY secret_key=BOGUS_SECRET_KEY bucket_location=us-east-1 multipart_chunk_size_mb=128 # default is 15 (MB)
On Google Cloud Storage, you can create the file by creating a service account
with the
storage.admin
role and then
creating a service account key.
Below is an example of using the
gcloud
CLI to create the file.
[default] # Setup endpoint: hostname of the Web App host_base=https://your_minio_setup.azurewebsites.net host_bucket=https://your_minio_setup.azurewebsites.net # Leave as default bucket_location=us-west-1 use_https=True multipart_chunk_size_mb=128 # default is 15 (MB)
Azure Blob: URL [FILTERED] is blocked: Requests to the local network are not allowed
This happens when the Azure Blob hostname is resolved to a RFC1918 (local / private) IP address. As a workaround,
allow Outbound requests
for your Azure Blob hostname (
yourinstance.blob.core.windows.net
).
MinIO is an object storage server that exposes S3-compatible APIs.
MinIO can be deployed to several different platforms. To launch a new MinIO instance,
follow their Quickstart Guide.
Be sure to secure access to the MinIO server with TLS.
To connect GitLab to an external MinIO instance,
first create MinIO buckets for the GitLab application, using the bucket names
in this example configuration file.
Using the MinIO client, create the necessary buckets before use:
mc mb gitlab-registry-storage mc mb gitlab-lfs-storage mc mb gitlab-artifacts-storage mc mb gitlab-uploads-storage mc mb gitlab-packages-storage mc mb gitlab-backup-storage
Once the buckets have been created, GitLab can be configured to use the MinIO instance.
See example configuration in
rails.minio.yaml
and
registry.minio.yaml
in the examples folder.
The instructions here make use of the Omnibus GitLab package for Ubuntu. This package provides versions of the services that are guaranteed to be compatible with the charts’ services.
Create VM with Omnibus GitLab
Create a VM on your provider of choice, or locally. This was tested with VirtualBox, KVM, and Bhyve.
Ensure that the instance is reachable from the cluster.
Install Ubuntu Server onto the VM that you have created. Ensure that
openssh-server
is installed, and that all packages are up to date.
Configure networking and a hostname. Make note of the hostname/IP, and ensure it is both resolvable and reachable from your Kubernetes cluster.
Be sure firewall policies are in place to allow traffic.
Follow the installation instructions for Omnibus GitLab. When you perform the package installation,
do not
provide the
EXTERNAL_URL=
value. We do not want automatic configuration to occur, as we’ll provide a very specific configuration in the next step.
Configure Omnibus GitLab
Create a minimal
gitlab.rb
file to be placed at
/etc/gitlab/gitlab.rb
. Be
very
explicit about what is enabled on this node, use the contents below.
This example is not intended to provide Redis for scaling.
REDIS_PASSWORD
should be replaced with the value in the
gitlab-redis
secret.
# Listen on all addresses redis['bind'] = '0.0.0.0' # Set the defaul port, must be set. redis['port'] = 6379 # Set password, as in the secret `gitlab-redis` populated in Kubernetes redis['password'] = 'REDIS_PASSWORD'
After creating
gitlab.rb
, reconfigure the package with
gitlab-ctl reconfigure
.
After the task completes, check the running processes with
gitlab-ctl status
.
The output should appear similar to:
If you are connecting to a Redis HA cluster that has Sentinel servers
running, the
global.redis.host
attribute needs to be set to the name of
the Redis instance group (such as
mymaster
or
resque
), as
specified in the
sentinel.conf
. Sentinel servers can be referenced
using the
global.redis.sentinels[0].host
and
global.redis.sentinels[0].port
values for the
--set
flag. The index is zero based.
Use multiple Redis instances
GitLab supports splitting several of the resource intensive
Redis operations across multiple Redis instances. This chart supports distributing
those persistence classes to other Redis instances.
More detailed information on configuring the chart for using multiple Redis
instances can be found in the globals
documentation.
Specify secure Redis scheme (SSL)
To connect to Redis using SSL, use the
rediss
(note the double
s
) scheme parameter:
Configure the GitLab chart with FIPS-compliant images | GitLab
Sample values
Configure the GitLab chart with FIPS-compliant images
GitLab offers FIPS-compliant
versions of its images, allowing you to run GitLab on FIPS-enabled clusters.
Sample values
We provide an example for GitLab chart values in
examples/fips/values.yaml
which can help you to build a FIPS-compatible GitLab deployment.
Note the comment under the
nginx-ingress.controller
key that provides the
relevant configuration to use a FIPS-compatible NGINX Ingress Controller image. This image is
maintained in our NGINX Ingress Controller fork.