Azure MinIO gateway when using the GitLab chart | GitLab
Storage Account
Deploy MinIO to Web App on Linux
Conclusion
Reference
Azure MinIO gateway when using the GitLab chart
MinIO is an object storage server that exposes S3-compatible APIs and it has a gateway feature that allows proxying requests to Azure Blob Storage. To setup our gateway, we will make use of Azure’s Web App on Linux.
To get started, make sure you have installed Azure CLI and you are logged in (
az login
). Proceed to create a Resource group, if you don’t have one already:
az group create --name"gitlab-azure-minio"--location"WestUS"
Storage Account
Create a Storage account in your resource group, the name of the storage account must be globally unique:
First, we need to create an App Service Plan in the same resource group.
az appservice plan create \ --name"gitlab-azure-minio-app-plan"\ --is-linux\ --sku B1 \ --resource-group"gitlab-azure-minio"\ --location"WestUS"
Create a Web app configured with the
minio/minio
Docker container, the name you specify will be used in the URL of the web app:
az webapp create \ --name"gitlab-minio-app"\ --deployment-container-image-name"minio/minio"\ --plan"gitlab-azure-minio-app-plan"\ --resource-group"gitlab-azure-minio"
The Web app should now be accessible at
https://gitlab-minio-app.azurewebsites.net
.
Lastly, we need to set up the startup command and create environment variables that will store our storage account name and key for use by the web app, MINIO_ACCESS_KEY & MINIO_SECRET_KEY.
az webapp config appsettings set\ --settings"MINIO_ACCESS_KEY=gitlab-azure-minio-storage""MINIO_SECRET_KEY=h0tSyeTebs+...""PORT=9000"\ --name"gitlab-minio-app"\ --resource-group"gitlab-azure-minio"
You can proceed to use this gateway with any client with s3-compability. Your web application URL will be the
s3 endpoint
, storage account name will be your
accesskey
, and storage account key will be your
secretkey
.
Reference
This guide was adapted for posterity from Alessandro Segala’s blog post on same topic.
Stay Ahead in Today’s Competitive Market!
Unlock your company’s full potential with a Virtual Delivery Center (VDC). Gain specialized expertise, drive
seamless operations, and scale effortlessly for long-term success.
MinIO is an object storage server that exposes S3-compatible APIs.
MinIO can be deployed to several different platforms. To launch a new MinIO instance,
follow their Quickstart Guide.
Be sure to secure access to the MinIO server with TLS.
To connect GitLab to an external MinIO instance,
first create MinIO buckets for the GitLab application, using the bucket names
in this example configuration file.
Using the MinIO client, create the necessary buckets before use:
mc mb gitlab-registry-storage mc mb gitlab-lfs-storage mc mb gitlab-artifacts-storage mc mb gitlab-uploads-storage mc mb gitlab-packages-storage mc mb gitlab-backup-storage
Once the buckets have been created, GitLab can be configured to use the MinIO instance.
See example configuration in
rails.minio.yaml
and
registry.minio.yaml
in the examples folder.
Azure Blob: URL [FILTERED] is blocked: Requests to the local network are not allowed
Configure the GitLab chart with an external object storage
GitLab relies on object storage for highly-available persistent data in Kubernetes.
By default, an S3-compatible storage solution named
minio
is deployed with the
chart. For production quality deployments, we recommend using a hosted
object storage solution like Google Cloud Storage or AWS S3.
To disable MinIO, set this option and then follow the related documentation below:
--set global.minio.enabled=false
An example of the full configuration
has been provided in the examples.
This documentation specifies usage of access and secret keys for AWS. It is also possible to use IAM roles.
S3 encryption
Introduced in GitLab 13.4.
GitLab supports Amazon KMS
to encrypt data stored in S3 buckets.
You can enable this in two ways:
In AWS, configure the S3 bucket to use default encryption.
In GitLab, enable server side encryption headers.
These two options are not mutually exclusive. You can set a default encryption
policy, but also enable server-side encryption headers to override those defaults.
See the GitLab documentation on encrypted S3 buckets
for more details.
Azure Blob Storage
Introduced in GitLab 13.4.
Direct support for Azure Blob storage is available for
uploaded attachments, CI job artifacts, LFS, and other object types supported via the consolidated settings. In previous GitLab versions, an Azure MinIO gateway was needed.
The Azure MinIO gateway is still needed for backups. Follow this issue
for more details.
GitLab does not support the Azure MinIO gateway as the storage for the Docker Registry.
Please refer to the corresponding Azure example when setting up the Docker Registry.
Although Azure uses the word container to denote a collection of blobs,
GitLab standardizes on the term bucket.
Azure Blob storage requires the use of the
consolidated object storage settings. A
single Azure storage account name and key must be used across multiple
Azure blob containers. Customizing individual
connection
settings by
object type (for example,
artifacts
,
uploads
, and so on) is not permitted.
To enable Azure Blob storage, see
rails.azurerm.yaml
as an example to define the Azure
connection
. You can load this as a
secret via:
Note
: The bucket name needs to be set both in the secret, and in
global.registry.bucket
. The secret is used in the registry server, and
the global is used by GitLab backups.
Create the secret per registry chart documentation on storage, then configure the chart to make use of this secret.
Examples for S3(S3 compatible storages, but Azure MinIO gateway not supported, see Azure Blob Storage), Azure and GCS drivers can be found in
examples/objectstorage
.
registry.s3.yaml
registry.gcs.yaml
registry.azure.yaml
Registry configuration
Decide on which storage service to use.
Copy appropriate file to
registry-storage.yaml
.
Edit with the correct values for the environment.
Follow registry chart documentation on storage for creating the secret.
Configuration of object storage for LFS, artifacts, uploads, packages, external
diffs, and pseudonymizer is done via the following keys:
global.appConfig.lfs
global.appConfig.artifacts
global.appConfig.uploads
global.appConfig.packages
global.appConfig.externalDiffs
global.appConfig.dependencyProxy
Note also that:
A different bucket is needed for each, otherwise performing a restore from
backup doesn’t function properly.
Storing MR diffs on external storage is not enabled by default, so,
for the object storage settings for
externalDiffs
to take effect,
global.appConfig.externalDiffs.enabled
key should have a
true
value.
The dependency proxy feature is not enabled by default, so,
for the object storage settings for
dependencyProxy
to take effect,
global.appConfig.dependencyProxy.enabled
key should have a
true
value.
See the charts/globals documentation on appConfig for full details.
Create the secret(s) per the connection details documentation, and then configure the chart to use the provided secrets. Note, the same secret can be used for all of them.
Examples for AWS (any S3 compatible like Azure using MinIO) and Google providers can be found in
examples/objectstorage
.
rails.s3.yaml
rails.gcs.yaml
rails.azure.yaml
rails.azurerm.yaml
appConfig configuration
Decide on which storage service to use.
Copy appropriate file to
rails.yaml
.
Edit with the correct values for the environment.
Follow connection details documentation for creating the secret.
Configure the chart as documented.
Backups
Backups are also stored in object storage, and must be configured to point
externally rather than the included MinIO service. The backup/restore procedure uses two separate buckets:
A bucket for storing backups (
global.appConfig.backups.bucket
)
A temporary bucket for preserving existing data during the restore process (
global.appConfig.backups.tmpBucket
)
AWS S3-compatible object storage systems and Google Cloud Storage are supported backends.
You can configure the backend type by setting
global.appConfig.backups.objectStorage.backend
to
s3
for AWS S3 or
gcs
for Google Cloud Storage.
You must also provide a connection configuration through the
gitlab.toolbox.backups.objectStorage.config
key.
When using Google Cloud Storage, the GCP project must be set with the
global.appConfig.backups.objectStorage.config.gcpProject
value.
See the backup/restore object storage documentation for full details.
To backup or restore files from the other object storage locations, the configuration file needs to be
configured to authenticate as a user with sufficient access to read/write to all GitLab buckets.
Backups storage example
Create the
storage.config
file:
On Amazon S3, the contents should be in the s3cmd configuration file format
[default] access_key=BOGUS_ACCESS_KEY secret_key=BOGUS_SECRET_KEY bucket_location=us-east-1 multipart_chunk_size_mb=128 # default is 15 (MB)
On Google Cloud Storage, you can create the file by creating a service account
with the
storage.admin
role and then
creating a service account key.
Below is an example of using the
gcloud
CLI to create the file.
[default] # Setup endpoint: hostname of the Web App host_base=https://your_minio_setup.azurewebsites.net host_bucket=https://your_minio_setup.azurewebsites.net # Leave as default bucket_location=us-west-1 use_https=True multipart_chunk_size_mb=128 # default is 15 (MB)
Azure Blob: URL [FILTERED] is blocked: Requests to the local network are not allowed
This happens when the Azure Blob hostname is resolved to a RFC1918 (local / private) IP address. As a workaround,
allow Outbound requests
for your Azure Blob hostname (
yourinstance.blob.core.windows.net
).
The instructions here make use of the Omnibus GitLab package for Ubuntu. This package provides versions of the services that are guaranteed to be compatible with the charts’ services.
Create VM with Omnibus GitLab
Create a VM on your provider of choice, or locally. This was tested with VirtualBox, KVM, and Bhyve.
Ensure that the instance is reachable from the cluster.
Install Ubuntu Server onto the VM that you have created. Ensure that
openssh-server
is installed, and that all packages are up to date.
Configure networking and a hostname. Make note of the hostname/IP, and ensure it is both resolvable and reachable from your Kubernetes cluster.
Be sure firewall policies are in place to allow traffic.
Follow the installation instructions for Omnibus GitLab. When you perform the package installation,
do not
provide the
EXTERNAL_URL=
value. We do not want automatic configuration to occur, as we’ll provide a very specific configuration in the next step.
Configure Omnibus GitLab
Create a minimal
gitlab.rb
file to be placed at
/etc/gitlab/gitlab.rb
. Be
very
explicit about what is enabled on this node, use the contents below.
This example is not intended to provide Redis for scaling.
REDIS_PASSWORD
should be replaced with the value in the
gitlab-redis
secret.
# Listen on all addresses redis['bind'] = '0.0.0.0' # Set the defaul port, must be set. redis['port'] = 6379 # Set password, as in the secret `gitlab-redis` populated in Kubernetes redis['password'] = 'REDIS_PASSWORD'
After creating
gitlab.rb
, reconfigure the package with
gitlab-ctl reconfigure
.
After the task completes, check the running processes with
gitlab-ctl status
.
The output should appear similar to:
Use TLS between components of the GitLab chart | GitLab
Preparation
Generating certificates for internal use
Required certificate CN and SANs
Configuration
Result
Troubleshooting
Use TLS between components of the GitLab chart
The GitLab charts can use transport-layer security (TLS) between the various
components. This requires you to provide certificates for the services
you want to enable, and configure those services to make use of those
certificates and the certificate authority (CA) that signed them.
Preparation
Each chart has documentation regarding enabling TLS for that service, and the various
settings required to ensure that appropriate configuration.
Generating certificates for internal use
GitLab does not purport to provide high-grade PKI infrastructure, or certificate
authorities.
For the purposes of this documentation, we provide a
Proof of Concept
script
below, which makes use of Cloudflare’s CFSSL
to produce a self-signed Certificate Authority, and a wildcard certificate that can be
used for all services.
This script will:
Generate a CA key pair.
Sign a certificate meant to service all GitLab component service endpoints.
Create two Kubernetes Secret objects:
A secret of type
kuberetes.io/tls
which has the server certificate and key pair.
A secret of type
Opaque
which
only
contains the public certificate of the CA as
ca.crt
as need by NGINX Ingress.
Prerequisites:
Bash, or compatible shell.
cfssl
is available to your shell, and within
PATH
.
kubectl
is available, and configured to point to your Kubernetes cluster
where GitLab will later be installed.
Be sure to have created the namespace you wish to have these certificates
installed into before operating the script.
You may copy the content of this script to your computer, and make the resulting
file executable. We suggest
poc-gitlab-internal-tls.sh
.
#!/bin/bash set-e ############# ## make and change into a working directory pushd$(mktemp-d)
############# ## setup environment NAMESPACE=${NAMESPACE:-default} RELEASE=${RELEASE:-gitlab} ## stop if variable is unset beyond this point set-u ## known expected patterns for SAN CERT_SANS="*.${NAMESPACE}.svc,${RELEASE}-metrics.${NAMESPACE}.svc,*.${RELEASE}-gitaly.${NAMESPACE}.svc"
The various GitLab components speak to each other over their Service’s DNS names.
The Ingress objects generated by the GitLab chart must provide NGINX the
name to verify, when
tls.verify: true
(which is the default). As a result
of this, each GitLab component should receive a certificate with a SAN including
either their Service’s name, or a wildcard acceptable to the Kubernetes Service
DNS entry.
service-name.namespace.svc
*.namespace.svc
Failure to ensure these SANs within certificates
will
result in a non-functional
instance, and logs that can be quite cryptic, refering to “connection failure”
or “SSL verification failed”.
You can make use of
helm template
to retrieve a full list of all
Service object names, if needed. If your GitLab has been deployed without TLS,
you can query Kubernetes for those names:
kubectl -n ${NAMESPACE} get service -lrelease=${RELEASE}
Configuration
Example configurations can be found in examples/internal-tls.
For the purposes of this documentation, we have provided
shared-cert-values.yaml
which configures the GitLab components to consume the certificates generated with
the script above, in generating certificates for internal use.
Key items to configure:
Global Custom Certificate Authorities.
Per-component TLS for the service listeners.
(See each chart’s documentation, under charts/)
This process is greatly simplified by making use of YAML’s native anchor
functionality. A truncated snippet of
shared-cert-values.yaml
shows this:
When all components have been configured to provide TLS on their service
listeners, all communication between GitLab components will traverse the
network with TLS security, including connections from NGINX Ingress to
each GitLab component.
NGINX Ingress will terminate any
inbound
TLS, determine the appropriate
services to pass the traffic to, and then form a new TLS connection to
the GitLab component. When configured as shown here, it will also
verify
the certificates served by the GitLab components against the CA.
This can be verified by connecting to the Toolbox pod, and querying the
various compontent Services. One such example, connecting to the Webservice
Pod’s primary service port that NGINX Ingress uses:
$ kubectl -n ${NAMESPACE} get pod -lapp=toolbox,release=${RELEASE} NAME READY STATUS RESTARTS AGE gitlab-toolbox-5c447bfdb4-pfmpc 1/1 Running 0 65m $ kubectl exec -ti gitlab-toolbox-5c447bfdb4-pfmpc -c toolbox -- \ curl -Iv "https://gitlab-webservice-default.testing.svc:8181"
The output should be similar to following example:
If your GitLab instance appears unreachable from the browser, by rendering an
HTTP 503 error, NGINX Ingress is likely having a problem verifying the
certificates of the GitLab components.
You may work around this by temporarily setting
gitlab.webservice.workhorse.tls.verify
to
false
.
The NGINX Ingress controller can be connected to, and will evidence a message
in
nginx.conf
, regarding problems verifying the certificate(s).
Example content, where the Secret is not reachable:
# Location denied. Reason: "error obtaining certificate: local SSL certificate testing/gitlab-internal-tls-ca was not found" return 503;
Common problems that cause this:
CA certificate is not in a key named
ca.crt
within the Secret.
The Secret was not properly supplied, or may not exist within the Namespace.
