Welcome to Knowledge Base!

KB at your finger tips

This is one stop global knowledge base where you can learn about all the products, solutions and support features.

Configure the GitLab chart with an external NGINX Ingress Controller | GitLab

Configure the GitLab chart with an external NGINX Ingress Controller

This chart configures Ingress resources for use with the official
NGINX Ingress implementation. The
NGINX Ingress Controller is deployed as a part of this chart. If you want to
reuse an existing NGINX Ingress Controller already available in your cluster,
this guide will help.

TCP services in the external Ingress Controller

The GitLab Shell component requires TCP traffic to pass through on
port 22 (by default; this can be changed). Ingress does not directly support TCP services, so some additional configuration is necessary. Your NGINX Ingress Controller may have been deployed directly (i.e. with a Kubernetes spec file) or through the official Helm chart. The configuration of the TCP pass through will differ depending on the deployment approach.

Direct deployment

In a direct deployment, the NGINX Ingress Controller handles configuring TCP services with a
ConfigMap (see docs here).
Assuming your GitLab chart is deployed to the namespace gitlab and your Helm
release is named mygitlab, your ConfigMap should be something like this:

apiVersion: v1
kind: ConfigMap
name: tcp-configmap-example
22: "gitlab/mygitlab-gitlab-shell:22"

After you have that ConfigMap, you can enable it as described in the NGINX
Ingress Controller docs
using the --tcp-services-configmap option.

- /nginx-ingress-controller
- --tcp-services-configmap=gitlab/tcp-configmap-example

Finally make sure that the Service for your NGINX Ingress Controller is exposing
port 22 in addition to 80 and 443.

Helm deployment

If you have installed or will install the NGINX Ingress Controller via it’s Helm chart, then you will need to add a value to the chart via the command line:

--set tcp.22="gitlab/mygitlab-gitlab-shell:22"

or a values.yaml file:

22: "gitlab/mygitlab-gitlab-shell:22"

The format for the value is the same as describe above in the “Direct Deployment” section.

Customize the GitLab Ingress options

The NGINX Ingress Controller uses an annotation to mark which Ingress Controller
will service a particular Ingress (see docs).
You can configure the Ingress class to use with this chart using the
global.ingress.class setting. Make sure to set this in your Helm options.

--set global.ingress.class=myingressclass

While not necessarily required, if you’re using an external Ingress Controller, you will likely want to
disable the Ingress Controller that is deployed by default with this chart:

--set nginx-ingress.enabled=false

Custom certificate management

The full scope of your TLS options are documented elsewhere.

If you are using an external Ingress Controller, you may also be using an external cert-manager instance
or managing your certificates in some other custom manner. The full documentation around your TLS options is here,
however for the purposes of this discussion, here are the two values that would need to be set to disable the cert-manager chart and tell
the GitLab component charts to NOT look for the built in certificate resources:

--set certmanager.install=false
--set global.ingress.configureCertmanager=false
IAM roles for AWS when using the GitLab chart | GitLab

IAM roles for AWS when using the GitLab chart

The default configuration for external object storage in the charts uses access and secret keys.
It is also possible to use IAM roles in combination with kube2iam,
kiam, or IRSA.

IAM role

The IAM role will need read, write and list permissions on the S3 buckets. You can choose to have a role per bucket or combine them.

Chart configuration

IAM roles can be specified by adding annotations and changing the secrets, as specified below:


An IAM role can be specified via the annotations key:

--set registry.annotations."iam\.amazonaws\.com/role"=<role name>

When creating the registry-storage.yaml secret, omit the access and secret key:

bucket: gitlab-registry
v4auth: true
region: us-east-1

Note: If you provide the key pair, IAM role will be ignored. See AWS documentation for more details.

LFS, Artifacts, Uploads, Packages

For LFS, artifacts, uploads, and packages an IAM role can be specified via the annotations key in the webservice and sidekiq configuration:

--set gitlab.sidekiq.annotations."iam\.amazonaws\.com/role"=<role name>
--set gitlab.webservice.annotations."iam\.amazonaws\.com/role"=<role name>

For the object-storage.yaml secret, omit
the access and secret key. Because the GitLab Rails codebase uses Fog for S3
storage, the use_iam_profile
key should be added for Fog to use the role:

provider: AWS
use_iam_profile: true
region: us-east-1

Do NOT include endpoint in this configuration.
IRSA makes use of STS tokens, which use specialized endpoints.
When endpoint is provided, the AWS client will attempt
to send an AssumeRoleWithWebIdentity message to this endpoint and will fail.


The Toolbox configuration allows for annotations to be set to upload backups to S3:

--set gitlab.toolbox.annotations."iam\.amazonaws\.com/role"=<role name>

The s3cmd.config secret is to be created without the access and secret keys:

bucket_location = us-east-1

Using IAM roles for service accounts

If GitLab is running in an AWS EKS cluster (version 1.14 or greater), you can
use an AWS IAM role to authenticate to the S3 object storage without the need
of generating or storing access tokens. More information regarding using
IAM roles in an EKS cluster can be found in the
Introducing fine-grained IAM roles for service accounts
documentation from AWS.

Appropriate IRSA annotations for roles can be applied to ServiceAccounts throughout
this Helm chart in one of two ways:

  1. ServiceAccounts that have been pre-created as described in the above AWS documentation.
    This ensures the proper annotations on the ServiceAccount and the linked OIDC provider.
  2. Chart-generated ServiceAccounts with annotations defined. We allow for the configuration
    of annotations on ServiceAccounts both globally and on a per-chart basis.

To use IAM roles for ServiceAccounts in EKS clusters, the specific annotation must be arn:aws:iam::<ACCOUNT_ID>:role/<IAM_ROLE_NAME>.

To enable IAM roles for ServiceAccounts for GitLab running in an AWS EKS cluster, follow the instructions on
IAM roles for service accounts.

Using the backup-utility as specified in the backup documentation
does not properly copy the backup file to the S3 bucket. The backup-utility uses
the s3cmd to perform the copy of the backup file and it has a known
issue of not supporting OIDC authentication.
This has been resolved in their 2.2.0 release, which has been
merged into GitLab 14.4.

Workaround to perform backups before GitLab 14.4

If you are on a version earlier than 14.4, run the following command in your task-runner pod to sideload
the latest version of s3cmd. You can then run backup-utility as per usual.

pip install --upgrade s3cmd && export PATH="$(python3 -m site --user-base)/bin:${PATH}"

Using pre-created service accounts

Set the following options when the GitLab chart is deployed. It is important
to note that the ServiceAccount is enabled but not created.

enabled: true
create: false

Fine-grained ServiceAccounts control is also available:

create: false
name: gitlab-registry
create: false
name: gitlab-webservice
create: false
name: gitlab-sidekiq
create: false
name: gitlab-toolbox

Using chart-owned service accounts

The annotation can be applied to all ServiceAccounts
created by GitLab owned charts by configuring global.serviceAccount.annotations.

annotations: arn:aws:iam::xxxxxxxxxxxx:role/name

Annotations can also be added on a per ServiceAccount basis, but adding the matching
definition for each chart. These can be the same role, or individual roles.

annotations: arn:aws:iam::xxxxxxxxxxxx:role/gitlab-registry
annotations: arn:aws:iam::xxxxxxxxxxxx:role/gitlab
annotations: arn:aws:iam::xxxxxxxxxxxx:role/gitlab
annotations: arn:aws:iam::xxxxxxxxxxxx:role/gitlab-toolbox


You can test if the IAM role is correctly set up and that GitLab is
accessing S3 using the IAM role by logging into the toolbox pod and
using awscli (replace <namespace> with the namespace where GitLab is

kubectl exec -ti $(kubectl get pod -n <namespace> -lapp=toolbox -o jsonpath='{.items[0]}') -n <namespace> -- bash

With the awscli package installed, verify that you are able to communicate
with the AWS API:

aws sts get-caller-identity

A normal response showing the temporary user ID, account number and IAM
ARN (this will not be the IAM ARN for the role used to access S3) will be
returned if connection to the AWS API was successful. An unsuccessful
connection will require more troubleshooting to determine why the toolbox
pod is not able to communicate with the AWS APIs.

If connecting to the AWS APIs is successful, then the following command
will assume the IAM role that was created and verify that a STS token can
be retrieved for accessing S3. The AWS_ROLE_ARN and AWS_WEB_IDENTITY_TOKEN_FILE
variables are defined in the environment when IAM role annotation has been
added to the pod and do not require that they be defined:

aws sts assume-role-with-web-identity --role-arn $AWS_ROLE_ARN  --role-session-name gitlab --web-identity-token file://$AWS_WEB_IDENTITY_TOKEN_FILE

If the IAM role could not be assumed then an error message similar to the
following will be displayed:

An error occurred (AccessDenied) when calling the AssumeRoleWithWebIdentity operation: Not authorized to perform sts:AssumeRoleWithWebIdentity

Otherwise, the STS credentials and IAM role information will be displayed.

WebIdentityErr: failed to retrieve credentials

If you see this error in the logs, this suggests that endpoint has
been configured in your object-storage.yaml secret. Remove
this setting and restart the webservice and sidekiq pods.

Read article
Azure MinIO gateway when using the GitLab chart | GitLab

Azure MinIO gateway when using the GitLab chart

MinIO is an object storage server that exposes S3-compatible APIs and it has a gateway feature that allows proxying requests to Azure Blob Storage. To setup our gateway, we will make use of Azure’s Web App on Linux.

To get started, make sure you have installed Azure CLI and you are logged in (az login). Proceed to create a Resource group, if you don’t have one already:

az group create --name "gitlab-azure-minio" --location "WestUS"

Storage Account

Create a Storage account in your resource group, the name of the storage account must be globally unique:

az storage account create \
--name "gitlab-azure-minio-storage" \
--kind BlobStorage \
--sku Standard_LRS \
--access-tier Cool \
--resource-group "gitlab-azure-minio" \
--location "WestUS"

Retrieve the account key for the storage account:

az storage account show-connection-string \
--name "gitlab-azure-minio-storage" \
--resource-group "gitlab-azure-minio"

The output should be in the format:

"connectionString": "DefaultEndpointsProtocol=https;;AccountName=gitlab-azure-minio-storage;AccountKey=h0tSyeTebs+..."

Deploy MinIO to Web App on Linux

First, we need to create an App Service Plan in the same resource group.

az appservice plan create \
--name "gitlab-azure-minio-app-plan" \
--is-linux \
--sku B1 \
--resource-group "gitlab-azure-minio" \
--location "WestUS"

Create a Web app configured with the minio/minio Docker container, the name you specify will be used in the URL of the web app:

az webapp create \
--name "gitlab-minio-app" \
--deployment-container-image-name "minio/minio" \
--plan "gitlab-azure-minio-app-plan" \
--resource-group "gitlab-azure-minio"

The Web app should now be accessible at

Lastly, we need to set up the startup command and create environment variables that will store our storage account name and key for use by the web app, MINIO_ACCESS_KEY & MINIO_SECRET_KEY.

az webapp config appsettings set \
--settings "MINIO_ACCESS_KEY=gitlab-azure-minio-storage" "MINIO_SECRET_KEY=h0tSyeTebs+..." "PORT=9000" \
--name "gitlab-minio-app" \
--resource-group "gitlab-azure-minio"

# Startup command
az webapp config set \
--startup-file "gateway azure" \
--name "gitlab-minio-app" \
--resource-group "gitlab-azure-minio"


You can proceed to use this gateway with any client with s3-compability. Your web application URL will be the s3 endpoint, storage account name will be your accesskey, and storage account key will be your secretkey.


This guide was adapted for posterity from Alessandro Segala’s blog post on same topic.

Read article
Configure the GitLab chart with an external object storage | GitLab

Configure the GitLab chart with an external object storage

GitLab relies on object storage for highly-available persistent data in Kubernetes.
By default, an S3-compatible storage solution named minio is deployed with the
chart. For production quality deployments, we recommend using a hosted
object storage solution like Google Cloud Storage or AWS S3.

To disable MinIO, set this option and then follow the related documentation below:

--set global.minio.enabled=false

An example of the full configuration
has been provided in the examples.

This documentation specifies usage of access and secret keys for AWS. It is also possible to use IAM roles.

S3 encryption

Introduced in GitLab 13.4.

GitLab supports Amazon KMS
to encrypt data stored in S3 buckets.
You can enable this in two ways:

These two options are not mutually exclusive. You can set a default encryption
policy, but also enable server-side encryption headers to override those defaults.

See the GitLab documentation on encrypted S3 buckets
for more details.

Azure Blob Storage

Introduced in GitLab 13.4.

Direct support for Azure Blob storage is available for
uploaded attachments, CI job artifacts, LFS, and other object types supported via the consolidated settings. In previous GitLab versions, an Azure MinIO gateway was needed.

The Azure MinIO gateway is still needed for backups. Follow this issue
for more details.

GitLab does not support the Azure MinIO gateway as the storage for the Docker Registry.
Please refer to the corresponding Azure example when setting up the Docker Registry.

Although Azure uses the word container to denote a collection of blobs,
GitLab standardizes on the term bucket.

Azure Blob storage requires the use of the
consolidated object storage settings. A
single Azure storage account name and key must be used across multiple
Azure blob containers. Customizing individual connection settings by
object type (for example, artifacts, uploads, and so on) is not permitted.

To enable Azure Blob storage, see
as an example to define the Azure connection. You can load this as a
secret via:

kubectl create secret generic gitlab-rails-storage --from-file=connection=rails.azurerm.yml

Then, disable MinIO and set these global settings:

--set global.minio.enabled=false
--set global.appConfig.object_store.enabled=true
--set global.appConfig.object_store.connection.secret=gitlab-rails-storage

Be sure to create Azure containers for the default names or set the container names in the bucket configuration.

If you experience requests failing with Requests to the local network are not allowed,
see the Troubleshooting section.

Docker Registry images

Configuration of object storage for the registry chart is done via the key, and the global.registry.bucket key.

--set global.registry.bucket=bucket-name

Note: The bucket name needs to be set both in the secret, and in global.registry.bucket. The secret is used in the registry server, and
the global is used by GitLab backups.

Create the secret per registry chart documentation on storage, then configure the chart to make use of this secret.

Examples for S3(S3 compatible storages, but Azure MinIO gateway not supported, see Azure Blob Storage), Azure and GCS drivers can be found in

Registry configuration

  1. Decide on which storage service to use.
  2. Copy appropriate file to registry-storage.yaml.
  3. Edit with the correct values for the environment.
  4. Follow registry chart documentation on storage for creating the secret.
  5. Configure the chart as documented.

LFS, Artifacts, Uploads, Packages, External Diffs, Terraform State, Dependency Proxy

Configuration of object storage for LFS, artifacts, uploads, packages, external
diffs, and pseudonymizer is done via the following keys:

  • global.appConfig.lfs
  • global.appConfig.artifacts
  • global.appConfig.uploads
  • global.appConfig.packages
  • global.appConfig.externalDiffs
  • global.appConfig.dependencyProxy

Note also that:

  • A different bucket is needed for each, otherwise performing a restore from
    backup doesn’t function properly.
  • Storing MR diffs on external storage is not enabled by default, so,
    for the object storage settings for externalDiffs to take effect,
    global.appConfig.externalDiffs.enabled key should have a true value.
  • The dependency proxy feature is not enabled by default, so,
    for the object storage settings for dependencyProxy to take effect,
    global.appConfig.dependencyProxy.enabled key should have a true value.

Below is an example of the configuration options:

--set global.appConfig.lfs.bucket=gitlab-lfs-storage
--set global.appConfig.lfs.connection.secret=object-storage
--set global.appConfig.lfs.connection.key=connection

--set global.appConfig.artifacts.bucket=gitlab-artifacts-storage
--set global.appConfig.artifacts.connection.secret=object-storage
--set global.appConfig.artifacts.connection.key=connection

--set global.appConfig.uploads.bucket=gitlab-uploads-storage
--set global.appConfig.uploads.connection.secret=object-storage
--set global.appConfig.uploads.connection.key=connection

--set global.appConfig.packages.bucket=gitlab-packages-storage
--set global.appConfig.packages.connection.secret=object-storage
--set global.appConfig.packages.connection.key=connection

--set global.appConfig.externalDiffs.bucket=gitlab-externaldiffs-storage
--set global.appConfig.externalDiffs.connection.secret=object-storage
--set global.appConfig.externalDiffs.connection.key=connection

--set global.appConfig.terraformState.bucket=gitlab-terraform-state
--set global.appConfig.terraformState.connection.secret=object-storage
--set global.appConfig.terraformState.connection.key=connection

--set global.appConfig.dependencyProxy.bucket=gitlab-dependencyproxy-storage
--set global.appConfig.dependencyProxy.connection.secret=object-storage
--set global.appConfig.dependencyProxy.connection.key=connection

See the charts/globals documentation on appConfig for full details.

Create the secret(s) per the connection details documentation, and then configure the chart to use the provided secrets. Note, the same secret can be used for all of them.

Examples for AWS (any S3 compatible like Azure using MinIO) and Google providers can be found in

appConfig configuration

  1. Decide on which storage service to use.
  2. Copy appropriate file to rails.yaml.
  3. Edit with the correct values for the environment.
  4. Follow connection details documentation for creating the secret.
  5. Configure the chart as documented.


Backups are also stored in object storage, and must be configured to point
externally rather than the included MinIO service. The backup/restore procedure uses two separate buckets:

  • A bucket for storing backups (global.appConfig.backups.bucket)
  • A temporary bucket for preserving existing data during the restore process (global.appConfig.backups.tmpBucket)

AWS S3-compatible object storage systems and Google Cloud Storage are supported backends.
You can configure the backend type by setting global.appConfig.backups.objectStorage.backend
to s3 for AWS S3 or gcs for Google Cloud Storage.
You must also provide a connection configuration through the gitlab.toolbox.backups.objectStorage.config key.

When using Google Cloud Storage, the GCP project must be set with the global.appConfig.backups.objectStorage.config.gcpProject value.

For S3-compatible storage:

--set global.appConfig.backups.bucket=gitlab-backup-storage
--set global.appConfig.backups.tmpBucket=gitlab-tmp-storage
--set gitlab.toolbox.backups.objectStorage.config.secret=storage-config
--set gitlab.toolbox.backups.objectStorage.config.key=config

For Google Cloud Storage (GCS):

--set global.appConfig.backups.bucket=gitlab-backup-storage
--set global.appConfig.backups.tmpBucket=gitlab-tmp-storage
--set gitlab.toolbox.backups.objectStorage.backend=gcs
--set gitlab.toolbox.backups.objectStorage.config.gcpProject=my-gcp-project-id
--set gitlab.toolbox.backups.objectStorage.config.secret=storage-config
--set gitlab.toolbox.backups.objectStorage.config.key=config

See the backup/restore object storage documentation for full details.

To backup or restore files from the other object storage locations, the configuration file needs to be
configured to authenticate as a user with sufficient access to read/write to all GitLab buckets.

Backups storage example

  1. Create the storage.config file:

    • On Amazon S3, the contents should be in the s3cmd configuration file format

      access_key = BOGUS_ACCESS_KEY
      secret_key = BOGUS_SECRET_KEY
      bucket_location = us-east-1
      multipart_chunk_size_mb = 128 # default is 15 (MB)

    • On Google Cloud Storage, you can create the file by creating a service account
      with the storage.admin role and then
      creating a service account key.
      Below is an example of using the gcloud CLI to create the file.

      export PROJECT_ID=$(gcloud config get-value project)
      gcloud iam service-accounts create gitlab-gcs --display-name "Gitlab Cloud Storage"
      gcloud projects add-iam-policy-binding --role roles/storage.admin ${PROJECT_ID} --member=serviceAccount:gitlab-gcs@${PROJECT_ID}
      gcloud iam service-accounts keys create --iam-account gitlab-gcs@${PROJECT_ID} storage.config

    • On Azure Storage

      # Setup endpoint: hostname of the Web App
      host_base =
      host_bucket =
      # Leave as default
      bucket_location = us-west-1
      use_https = True
      multipart_chunk_size_mb = 128 # default is 15 (MB)

      # Setup access keys
      # Access Key = Azure Storage Account name
      access_key = BOGUS_ACCOUNT_NAME
      # Secret Key = Azure Storage Account Key
      secret_key = BOGUS_KEY

      # Use S3 v4 signature APIs
      signature_v2 = False

  2. Create the secret

    kubectl create secret generic storage-config --from-file=config=storage.config

Google Cloud CDN

Introduced in GitLab 15.5.

You can use Google Cloud CDN to cache
and fetch data from the artifacts bucket. This can help improve
performance and reduce network egress costs.

Configuration of Cloud CDN is done via the following keys:

  • global.appConfig.artifacts.cdn.secret

  • global.appConfig.artifacts.cdn.key (default is cdn)

To use Cloud CDN:

  1. Set up Cloud CDN to use the artifacts bucket as the backend.
  2. Create a key for signed URLs.
  3. Give the Cloud CDN service account permission to read from the bucket.
  4. Prepare a YAML file with the parameters using the example in rails.googlecdn.yaml.
    You will need to fill in the following information:

    • url: Base URL of the CDN host from step 1

    • key_name: Key name from step 2

    • key: The actual secret from step 2

  5. Load this YAML file into a Kubernetes secret under the cdn key. For example, to create a secret gitlab-rails-cdn:

     kubectl create secret generic gitlab-rails-cdn --from-file=cdn=rails.googlecdn.yml

  6. Set global.appConfig.artifacts.cdn.secret to gitlab-rails-cdn. If you’re setting this via a helm
    parameter, use:

     --set global.appConfig.artifacts.cdn.secret=gitlab-rails-cdn


Azure Blob: URL [FILTERED] is blocked: Requests to the local network are not allowed

This happens when the Azure Blob hostname is resolved to a RFC1918 (local / private) IP address. As a workaround,
allow Outbound requests
for your Azure Blob hostname (

Read article
Configure MinIO with the GitLab chart | GitLab

Configure MinIO with the GitLab chart

MinIO is an object storage server that exposes S3-compatible APIs.

MinIO can be deployed to several different platforms. To launch a new MinIO instance,
follow their Quickstart Guide.
Be sure to secure access to the MinIO server with TLS.

To connect GitLab to an external MinIO instance,
first create MinIO buckets for the GitLab application, using the bucket names
in this example configuration file.

Using the MinIO client, create the necessary buckets before use:

mc mb gitlab-registry-storage
mc mb gitlab-lfs-storage
mc mb gitlab-artifacts-storage
mc mb gitlab-uploads-storage
mc mb gitlab-packages-storage
mc mb gitlab-backup-storage

Once the buckets have been created, GitLab can be configured to use the MinIO instance.
See example configuration in rails.minio.yaml and
in the examples folder.

Read article
Setup standalone Redis | GitLab

Setup standalone Redis

The instructions here make use of the Omnibus GitLab package for Ubuntu. This package provides versions of the services that are guaranteed to be compatible with the charts’ services.

Create VM with Omnibus GitLab

Create a VM on your provider of choice, or locally. This was tested with VirtualBox, KVM, and Bhyve.
Ensure that the instance is reachable from the cluster.

Install Ubuntu Server onto the VM that you have created. Ensure that openssh-server is installed, and that all packages are up to date.
Configure networking and a hostname. Make note of the hostname/IP, and ensure it is both resolvable and reachable from your Kubernetes cluster.
Be sure firewall policies are in place to allow traffic.

Follow the installation instructions for Omnibus GitLab. When you perform the package installation, do not provide the EXTERNAL_URL= value. We do not want automatic configuration to occur, as we’ll provide a very specific configuration in the next step.

Configure Omnibus GitLab

Create a minimal gitlab.rb file to be placed at /etc/gitlab/gitlab.rb. Be very explicit about what is enabled on this node, use the contents below.

This example is not intended to provide Redis for scaling.
# Listen on all addresses
redis['bind'] = ''
# Set the defaul port, must be set.
redis['port'] = 6379
# Set password, as in the secret `gitlab-redis` populated in Kubernetes
redis['password'] = 'REDIS_PASSWORD'

## Disable everything else
gitlab_rails['enable'] = false
sidekiq['enable'] = false
registry['enable'] = false
gitaly['enable'] = false
gitlab_workhorse['enable'] = false
nginx['enable'] = false
prometheus_monitoring['enable'] = false
postgresql['enable'] = false

After creating gitlab.rb, reconfigure the package with gitlab-ctl reconfigure.
After the task completes, check the running processes with gitlab-ctl status.
The output should appear similar to:

# gitlab-ctl status
run: logrotate: (pid 4856) 1859s; run: log: (pid 31262) 77460s
run: redis: (pid 30562) 77637s; run: log: (pid 30561) 77637s
Read article

Signup now!

Login / Signup as

Post taks and get it done by experts

Find great opportunities and enjoy reading