Speed up build times with this little Git trick

When building applications on build pipelines like GitHub Actions, Google Cloud Build, CircleCI, etc. every second counts. Here’s this small trick I use to speed up build times: when cloning the repo from its Git source, I instruct Git to do a shallow clone of the single branch it is building.

💡 If you’re running a prebuilt “git clone” step on your Build Pipeline it most likely already uses this trick. Be sure to double check though.

~

Shallow Cloning

When doing a git clone you, by default, get the entire history of the project along with that. Take this clone of Laravel for example:

$ git clone git@github.com:laravel/laravel.git
Cloning into 'laravel'...
remote: Enumerating objects: 19, done.
remote: Counting objects: 100% (19/19), done.
remote: Compressing objects: 100% (15/15), done.
remote: Total 32004 (delta 5), reused 11 (delta 3), pack-reused 31985
Receiving objects: 100% (32004/32004), 9.94 MiB | 6.98 MiB/s, done.
Resolving deltas: 100% (18934/18934), done.

That’s a whopping 32004 objects totalling ±10MiB that have been downloaded, even though the default Laravel branch only counts 66 files spread across 36 directories.

The objects contained in this ±10MiB make up the entire history of every file and folder the project. To build the project we don’t really need all that, as we’re only interested in the latest version of each file and folder. By leveraging the --depth argument of our git clone command, we can enforce just that. This is what we call Shallow Cloning.

$ git clone --depth 1 git@github.com:laravel/laravel.git
Cloning into 'laravel'...
remote: Enumerating objects: 108, done.
remote: Counting objects: 100% (108/108), done.
remote: Compressing objects: 100% (88/88), done.
remote: Total 108 (delta 6), reused 49 (delta 1), pack-reused 0
Receiving objects: 100% (108/108), 41.80 KiB | 535.00 KiB/s, done.
Resolving deltas: 100% (6/6), done.

That’s a much speedier clone with only 108 objects, totalling a mere ±40KiB!

🤔 You could argue that 10MiB worth of objects is still OK to clone, but think of scenarios where you have a big “monorepo” with plenty of branches … then you’ll be talking about hundreds of wasted MiBs, if not GiBs.

~

Single Branch Cloning

When building a project you’re building only one certain branch. Information about the other branches is irrelevant at that time. To directly clone one specific branch we can use the --branch option to target said branch. With that alone we won’t get there though, we we still need to discard information about other branches. This is where the --single-branch option comes into play:

$ git clone --branch 3.0 --single-branch git@github.com:laravel/laravel.git
Cloning into 'laravel'...
remote: Enumerating objects: 20392, done.
remote: Total 20392 (delta 0), reused 0 (delta 0), pack-reused 20392
Receiving objects: 100% (20392/20392), 5.79 MiB | 853.00 KiB/s, done.
Resolving deltas: 100% (12731/12731), done.

Here we’ve cloned only the 3.0 branch of Laravel, resulting in roughly 10000 fewer objects to be downloaded.

By checking the contents of git branch -a we can also verify that other branch info has not been fetched:

$ cd laravel
$ git branch -a
* 3.0
  remotes/origin/3.0

~

Shallow Cloning + Single Branch Cloning

By combining both we can download only the latest files of a specific branch. Since the use of --single-branch is implied when using --depth, we can drop the former and our command will look like this:

$ git clone --depth 1 --branch <branchname> <repo>

Here’s an example downloading the Laravel 3.0 branch:

$ git clone --depth 1 --branch 3.0 git@github.com:laravel/laravel.git
Cloning into 'laravel'...
remote: Enumerating objects: 545, done.
remote: Counting objects: 100% (545/545), done.
remote: Compressing objects: 100% (465/465), done.
remote: Total 545 (delta 78), reused 293 (delta 45), pack-reused 0
Receiving objects: 100% (545/545), 1.34 MiB | 832.00 KiB/s, done.
Resolving deltas: 100% (78/78), done.

~

With this in place you’ll see your build times drop by minutes, especially when working on a monorepo with many branches.

Did this help you out? Like what you see?
Thank me with a coffee.

I don't do this for profit but a small one-time donation would surely put a smile on my face. Thanks!

☕️ Buy me a Coffee (€3)

To stay in the loop you can follow @bramus or follow @bramusblog on Twitter.

🐳 Building Docker Images on Google Cloud Build? Check out this trick to enable caching.

Speed up your Docker builds in Google Cloud Build with Kaniko Cache

When building Docker images locally it will leverage its build cache:

When building an image, Docker steps through the instructions in your Dockerfile, executing each in the order specified. As each instruction is examined, Docker looks for an existing image in its cache that it can reuse, rather than creating a new (duplicate) image.

Therefore it is important that you carefully cater the order of your different Docker steps.

~

When using Google Cloud Build however, there – by default – is no cache to fall back to. As you’re paying for every second spent building it’d be handy to have some caching in place. Currently there are two options to do so:

  1. Using the --cache-from argument in your build config
  2. Using the Kaniko cache

⚠️ Note that the same rules as with the local cache layers apply for both scenarios: if you constantly change a layer in the earlier stages of your Docker build, it won’t be of much benefit.

~

Using the --cache-from argument (ref)

The easiest way to increase the speed of your Docker image build is by specifying a cached image that can be used for subsequent builds. You can specify the cached image by adding the --cache-from argument in your build config file, which will instruct Docker to build using that image as a cache source.

To make this work you’ll first need to pull the previously built image from the registry, and then refer to it using the --cache-from argument:

steps:
 - name: 'gcr.io/cloud-builders/docker'
   entrypoint: 'bash'
   args:
   - '-c'
   - |
     docker pull gcr.io/$PROJECT_ID/[IMAGE_NAME]:latest || exit 0
- name: 'gcr.io/cloud-builders/docker'
  args: [
            'build',
            '-t', 'gcr.io/$PROJECT_ID/[IMAGE_NAME]:latest',
            '--cache-from', 'gcr.io/$PROJECT_ID/[IMAGE_NAME]:latest',
            '.'
        ]
images: ['gcr.io/$PROJECT_ID/[IMAGE_NAME]:latest']

~

Using the Kaniko cache (ref)

Kaniko cache is a Cloud Build feature that caches container build artifacts by storing and indexing intermediate layers within a container image registry, such as Google’s own Container Registry, where it is available for use by subsequent builds.

To enable it, replace the cloud-builders/docker worker in your cloudbuild.yaml with the kaniko-project/executor.

steps:
- name: 'gcr.io/kaniko-project/executor:latest'
  args:
  - --destination=gcr.io/$PROJECT_ID/image
  - --cache=true
  - --cache-ttl=XXh

When using Kaniko, images are automatically pushed to Container Registry as soon as they are built. You don’t need to specify your images in the images attribute, as you would when using cloud-builders/docker.

Here’s a comparison of a first and second run:

From +8 minutes down to 55 seconds by one simple change to our cloudbuild.yaml 🙂

~

Did this help you out? Like what you see?
Thank me with a coffee.

I don't do this for profit but a small one-time donation would surely put a smile on my face. Thanks!

☕️ A thank you coffee for saving build time (€4)

To stay in the loop you can follow @bramus or follow @bramusblog on Twitter.

Google Cloud Build + Google Cloud Run: Fixing “ERROR: (gcloud.run.deploy) PERMISSION_DENIED: The caller does not have permission

Google Cloud Build is cool. Google Cloud Run is awesome. But when configuring Google Cloud Build to automatically deploy your built container to Google Cloud Run you might see this error:

ERROR: (gcloud.run.deploy) PERMISSION_DENIED: The caller does not have permission

If you’re seeing this error you forgot to set up the required IAM Permissions for the Cloud Build Service Account. Below I’ll show you the commands to fix this error.

~

As detailed in the Cloud Run documentation, a user needs the following permissions to deploy new Cloud Run services or revisions:

  1. run.services.create and run.services.update on the project level. Typically assigned through the roles/run.admin role.
  2. iam.serviceAccounts.actAs for the Cloud Run runtime service account. By default, this is PROJECT_NUMBER-compute@developer.gserviceaccount.com. The permission is typically assigned through the roles/iam.serviceAccountUser role.

By default – for security reasons – the Cloud Build Service Account does not have the permissions to manage Cloud Run, explaining why you’re getting errors.

~

You can set up these required permissions using the Google Cloud Console, yet I prefer to do this from the CLI using the Google Cloud SDK (gcloud), by invoking the commands below:

# Config
GC_PROJECT=your-gcp-project-id
GC_PROJECT_NUMBER=your-gcp-project-number

# Grant the Cloud Run Admin role to the Cloud Build service account
gcloud projects add-iam-policy-binding $GC_PROJECT \
  --member "serviceAccount:$GC_PROJECT_NUMBER@cloudbuild.gserviceaccount.com" \
  --role roles/run.admin

# Grant the IAM Service Account User role to the Cloud Build service account on the Cloud Run runtime service account
gcloud iam service-accounts add-iam-policy-binding \
  $GC_PROJECT_NUMBER-compute@developer.gserviceaccount.com \
  --member="serviceAccount:$GC_PROJECT_NUMBER@cloudbuild.gserviceaccount.com" \
  --role="roles/iam.serviceAccountUser"

💡 To know the values for GC_PROJECT(_ID) and GC_PROJECT_NUMBER, run gcloud projects list or go to the Home of your project inside Cloud Console.

After running these two commands, re-trigger a build and Google Cloud Build will be able to deploy your built container onto Google Cloud Run.

~

Did this help you out? Like what you see?
Thank me with a coffee.

I don't do this for profit but a small one-time donation would surely put a smile on my face. Thanks!

☕️ Buy me a Coffee (€4)

To stay in the loop you can follow @bramus or follow @bramusblog on Twitter.