Welcome to Knowledge Base!

KB at your finger tips

Book a Meeting to Avail the Services of Docker overtime

This is one stop global knowledge base where you can learn about all the products, solutions and support features.

Categories
All

Docker

Local cache

Local cache

The local cache store is a simple cache option that stores your cache as files in a directory on your filesystem, using an OCI image layout for the underlying directory structure. Local cache is a good choice if you’re just testing, or if you want the flexibility to self-manage a shared storage solution.

Note

This cache storage backend requires using a different driver than the default docker driver - see more information on selecting a driver here. To create a new driver (which can act as a simple drop-in replacement):

$ docker buildx create --use --driver=docker-container

Synopsis

$ docker buildx build --push -t <registry>/<image> \
  --cache-to type=local,dest=path/to/local/dir[,parameters...] \
  --cache-from type=local,src=path/to/local/dir .

The following table describes the available CSV parameters that you can pass to --cache-to and --cache-from .

Name Option Type Default Description
src cache-from String  Path of the local directory where cache gets imported from.
digest cache-from String  Digest of manifest to import, see cache versioning.
dest cache-to String  Path of the local directory where cache gets exported to.
mode cache-to min , max min Cache layers to export, see cache mode.
oci-mediatypes cache-to true , false true Use OCI media types in exported manifests, see OCI media types.
compression cache-to gzip , estargz , zstd gzip Compression type, see cache compression.
compression-level cache-to 0..22 Â Compression level, see cache compression.
force-compression cache-to true , false false Forcibly apply compression, see cache compression.

If the src cache doesn’t exist, then the cache import step will fail, but the build will continue.

Cache versioning

This section describes how versioning works for caches on a local filesystem, and how you can use the digest parameter to use older versions of cache.

If you inspect the cache directory manually, you can see the resulting OCI image layout:

$ ls cache
blobs  index.json  ingest
$ cat cache/index.json | jq
{
  "schemaVersion": 2,
  "manifests": [
    {
      "mediaType": "application/vnd.oci.image.index.v1+json",
      "digest": "sha256:6982c70595cb91769f61cd1e064cf5f41d5357387bab6b18c0164c5f98c1f707",
      "size": 1560,
      "annotations": {
        "org.opencontainers.image.ref.name": "latest"
      }
    }
  ]
}

Like other cache types, local cache gets replaced on export, by replacing the contents of the index.json file. However, previous caches will still be available in the blobs directory. These old caches are addressable by digest, and kept indefinitely. Therefore, the size of the local cache will continue to grow (see moby/buildkit#1896 for more information).

When importing cache using --cache-to , you can specify the digest parameter to force loading an older version of the cache, for example:

$ docker buildx build --push -t <registry>/<image> \
  --cache-to type=local,dest=path/to/local/dir \
  --cache-from type=local,ref=path/to/local/dir,digest=sha256:6982c70595cb91769f61cd1e064cf5f41d5357387bab6b18c0164c5f98c1f707 .

Further reading

For an introduction to caching see Optimizing builds with cache.

For more information on the local cache backend, see the BuildKit README.


Stay Ahead in Today’s Competitive Market!
Unlock your company’s full potential with a Virtual Delivery Center (VDC). Gain specialized expertise, drive seamless operations, and scale effortlessly for long-term success.

Book a Meeting to Avail the Services of Dockerovertime

Docker driver

Docker driver

The Buildx Docker driver is the default driver. It uses the BuildKit server components built directly into the Docker engine. The Docker driver requires no configuration.

Unlike the other drivers, builders using the Docker driver can’t be manually created. They’re only created automatically from the Docker context.

Images built with the Docker driver are automatically loaded to the local image store.

Synopsis

# The Docker driver is used by buildx by default
docker buildx build .

It’s not possible to configure which BuildKit version to use, or to pass any additional BuildKit parameters to a builder using the Docker driver. The BuildKit version and parameters are preset by the Docker engine internally.

If you need additional configuration and flexibility, consider using the Docker container driver.

Further reading

For more information on the Docker driver, see the buildx reference.

Read article

Local and tar exporters

Local and tar exporters

The local and tar exporters output the root filesystem of the build result into a local directory. They’re useful for producing artifacts that aren’t container images.

  • local exports files and directories.
  • tar exports the same, but bundles the export into a tarball.

Synopsis

Build a container image using the local exporter:

$ docker buildx build --output type=local[,parameters] .
$ docker buildx build --output type=tar[,parameters] .

The following table describes the available parameters:

Parameter Type Default Description
dest String  Path to copy files to

Further reading

For more information on the local or tar exporters, see the BuildKit README.

Read article

Registry cache

Registry cache

The registry cache storage can be thought of as an extension to the inline cache. Unlike the inline cache, the registry cache is entirely separate from the image, which allows for more flexible usage - registry -backed cache can do everything that the inline cache can do, and more:

  • Allows for separating the cache and resulting image artifacts so that you can distribute your final image without the cache inside.
  • It can efficiently cache multi-stage builds in max mode, instead of only the final stage.
  • It works with other exporters for more flexibility, instead of only the image exporter.

Note

This cache storage backend requires using a different driver than the default docker driver - see more information on selecting a driver here. To create a new driver (which can act as a simple drop-in replacement):

$ docker buildx create --use --driver=docker-container

Synopsis

Unlike the simpler inline cache, the registry cache supports several configuration parameters:

$ docker buildx build --push -t <registry>/<image> \
  --cache-to type=registry,ref=<registry>/<cache-image>[,parameters...] \
  --cache-from type=registry,ref=<registry>/<cache-image> .

The following table describes the available CSV parameters that you can pass to --cache-to and --cache-from .

Name Option Type Default Description
ref cache-to , cache-from String  Full name of the cache image to import.
dest cache-to String  Path of the local directory where cache gets exported to.
mode cache-to min , max min Cache layers to export, see cache mode.
oci-mediatypes cache-to true , false true Use OCI media types in exported manifests, see OCI media types.
compression cache-to gzip , estargz , zstd gzip Compression type, see cache compression.
compression-level cache-to 0..22 Â Compression level, see cache compression.
force-compression cache-to true , false false Forcibly apply compression, see cache compression.

You can choose any valid value for ref , as long as it’s not the same as the target location that you push your image to. You might choose different tags (e.g. foo/bar:latest and foo/bar:build-cache ), separate image names (e.g. foo/bar and foo/bar-cache ), or even different repositories (e.g. docker.io/foo/bar and ghcr.io/foo/bar ). It’s up to you to decide the strategy that you want to use for separating your image from your cache images.

If the --cache-from target doesn’t exist, then the cache import step will fail, but the build will continue.

Further reading

For an introduction to caching see Optimizing builds with cache.

For more information on the registry cache backend, see the BuildKit README.

Read article

Continuous integration with Docker

Continuous integration with Docker

Continuous Integration (CI) is the part of the development process where you’re looking to get your code changes merged with the main branch of the project. At this point, development teams run tests and builds to vet that the code changes don’t cause any unwanted or unexpected behaviors.

Git branches about to get merged

There are several uses for Docker at this stage of development, even if you don’t end up packaging your application as a container image.

Docker as a build environment

Containers are reproducible, isolated environments that yield predictable results. Building and testing your application in a Docker container makes it easier to prevent unexpected behaviors from occurring. Using a Dockerfile, you define the exact requirements for the build environment, including programming runtimes, operating system, binaries, and more.

Using Docker to manage your build environment also eases maintenance. For example, updating to a new version of a programming runtime can be as simple as changing a tag or digest in a Dockerfile. No need to SSH into a pet VM to manually reinstall a newer version and update the related configuration files.

Additionally, just as you expect third-party open source packages to be secure, the same should go for your build environment. You can scan and index a builder image, just like you would for any other containerized application.

The following links provide instructions for how you can get started using Docker for building your applications in CI:

  • GitHub Actions
  • GitLab
  • Circle CI
  • Render

Docker in Docker

You can also use a Dockerized build environment to build container images using Docker. That is, your build environment runs inside a container which itself is equipped to run Docker builds. This method is referred to as “Docker in Docker”.

Docker provides an official Docker image that you can use for this purpose.

What’s next

Docker maintains a set of official GitHub Actions that you can use to build, annotate, and push container images on the GitHub Actions platform. See Introduction to GitHub Actions to learn more and get started.

Read article

Docker container driver

Docker container driver

The buildx Docker container driver allows creation of a managed and customizable BuildKit environment in a dedicated Docker container.

Using the Docker container driver has a couple of advantages over the default Docker driver. For example:

  • Specify custom BuildKit versions to use.
  • Build multi-arch images, see QEMU
  • Advanced options for cache import and export

Synopsis

Run the following command to create a new builder, named container , that uses the Docker container driver:

$ docker buildx create \
  --name container \
  --driver=docker-container \
  --driver-opt=[key=value,...]
container

The following table describes the available driver-specific options that you can pass to --driver-opt :

Parameter Type Default Description
image String  Sets the image to use for running BuildKit.
network String  Sets the network mode for running the BuildKit container.
cgroup-parent String /docker/buildx Sets the cgroup parent of the BuildKit container if Docker is using the cgroupfs driver.
env.<key> String  Sets the environment variable key to the specified value in the BuildKit container.

Usage

When you run a build, Buildx pulls the specified image (by default, moby/buildkit ){:target=”blank” rel=”noopener” class=””}. When the container has started, Buildx submits the build submitted to the containerized build server.

$ docker buildx build -t <image> --builder=container .
WARNING: No output specified with docker-container driver. Build result will only remain in the build cache. To push result image into registry use --push or to load image into docker use --load
#1 [internal] booting buildkit
#1 pulling image moby/buildkit:buildx-stable-1
#1 pulling image moby/buildkit:buildx-stable-1 1.9s done
#1 creating container buildx_buildkit_container0
#1 creating container buildx_buildkit_container0 0.5s done
#1 DONE 2.4s
...

Loading to local image store

Unlike when using the default docker driver, images built with the docker-container driver must be explicitly loaded into the local image store. Use the --load flag:

$ docker buildx build --load -t <image> --builder=container .
...
 => exporting to oci image format                                                                                                      7.7s
 => => exporting layers                                                                                                                4.9s
 => => exporting manifest sha256:4e4ca161fa338be2c303445411900ebbc5fc086153a0b846ac12996960b479d3                                      0.0s
 => => exporting config sha256:adf3eec768a14b6e183a1010cb96d91155a82fd722a1091440c88f3747f1f53f                                        0.0s
 => => sending tarball                                                                                                                 2.8s
 => importing to docker

The image becomes available in the image store when the build finishes:

$ docker image ls
REPOSITORY                       TAG               IMAGE ID       CREATED             SIZE
<image>                          latest            adf3eec768a1   2 minutes ago       197MB

Cache persistence

The docker-container driver supports cache persistence, as it stores all the BuildKit state and related cache into a dedicated Docker volume.

To persist the docker-container driver’s cache, even after recreating the driver using docker buildx rm and docker buildx create , you can destroy the builder using the --keep-state flag:

For example, to create a builder named container and then remove it while persisting state:

# setup a builder
$ docker buildx create --name=container --driver=docker-container --use --bootstrap
container
$ docker buildx ls
NAME/NODE       DRIVER/ENDPOINT              STATUS   BUILDKIT PLATFORMS
container *     docker-container
  container0    desktop-linux                running  v0.10.5  linux/amd64
$ docker volume ls
DRIVER    VOLUME NAME
local     buildx_buildkit_container0_state

# remove the builder while persisting state
$ docker buildx rm --keep-state container
$ docker volume ls
DRIVER    VOLUME NAME
local     buildx_buildkit_container0_state

# the newly created driver with the same name will have all the state of the previous one!
$ docker buildx create --name=container --driver=docker-container --use --bootstrap
container

QEMU

The docker-container driver supports using QEMU (user mode) to build non-native platforms. Use the --platform flag to specify which architectures that you want to build for.

For example, to build a Linux image for amd64 and arm64 :

$ docker buildx build \
  --builder=container \
  --platform=linux/amd64,linux/arm64 \
  -t <registry>/<image> \
  --push .

Warning

QEMU performs full-system emulation of non-native platforms, which is much slower than native builds. Compute-heavy tasks like compilation and compression/decompression will likely take a large performance hit.

Custom network

You can customize the network that the builder container uses. This is useful if you need to use a specific network for your builds.

For example, let’s create a network named foonet :

$ docker network create foonet

Now create a docker-container builder that will use this network:

$ docker buildx create --use \
  --name mybuilder \
  --driver docker-container \
  --driver-opt "network=foonet"

Boot and inspect mybuilder :

$ docker buildx inspect --bootstrap

Inspect the builder container and see what network is being used:

$ docker inspect buildx_buildkit_mybuilder0 --format={{.NetworkSettings.Networks}}
map[foonet:0xc00018c0c0]

Further reading

For more information on the Docker container driver, see the buildx reference.

Read article