Eventually this will be too slow as well and you will want to run multiple CI jobs with a matrix strategy. We started using the split plugin for this which lets you record test times and split them into balanced shards so each matrix job should take around the same amount of time. We always have to wait for all tests to complete in our CI, so we care that they are well-balanced since the longest shard is the limiting factor. The test durations are recorded in a .test_durations file that is committed into the repo. There will be drift over time so tests without known test duration will be assumed to have the arithmetic mean test duration from the known test durations.

Recently our test jobs had once again reached too high a duration and initially my plan was to just increase the number of shards again. However, even when doubling the shards I noticed only a minor speedup. When going through the logs of the recent test run I noticed that pytest startup took over 3 minutes:

• Python, uv setup and syncing dependencies: ~25 seconds
• Docker/test infrastructure: ~16 seconds
• Pytest startup/import/collection: ~3 minutes
• Actual test execution: ~3½ minutes

Digging into it, it turns out the major slow down came from each shard and each worker in each shard collecting all of the test files. At around 18k tests split across around 1.5k files, the collection of tests was starting to add up to almost half of the runtime of our CI test jobs.

This makes sense since the shards are balanced by individual test times so there is no guarantee collection can skip any files, so the plugin lets pytest collect all tests and then tells it which ones to deselect in pytest_collection_modifyitems.

This fine-grained balancing of test shards by individual test durations can be necessary if you have outlier tests concentrated in just a few test files. However in our codebase we can balance shards for our unit tests by test files instead of by individual tests because test files generally have a small number of tests and outliers aren’t too extreme (our longest measured test file takes 24.8s).

Figure 1: Histogram of test durations. X-axis is logarithmic. Longest test file took 24.8s, shortest under a ms. Bimodal distribution with means at 10ms and 1s.

Balancing by test shards allows us to use the pytest_ignore_collect hook to avoid collecting the entire test code base in each shard and instead only collect the assigned files. Here an abridged time line of the pytest_collect phases:

pytest_collection

• pytest_ignore_collect
  Our sharder rejects unassigned directories and files
  → rejected files are never imported
• pytest_collect_directory, pytest_collect_file, pytest_pycollect_makemodule, pytest_pycollect_makeitem, pytest_generate_tests
  → collects tests and generates the test objects
• pytest_collection_modifyitems, pytest_deselected
  → plugins apply filters, marks, reorders to collected items and finally lets plugins know about which tests have been deselected

Without file-based sharding test collection took 3:08 minutes:

13:59:56 GMT ~  uv run pytest -n 2 --dist=loadgroup -m "not e2e” --splits 8 --group 4 --splitting-algorithm least_duration
14:03:04 GMT ~ ========================= test session starts =========================

With file-based sharding it took 40s:

20:48:10 GMT ~ uv run pytest -n 2 --dist=loadgroup -m "not e2e" --fsplits 8 --fgroup 4
20:48:50 GMT ~ ========================= test session starts =========================

a tiny nitpick here - the second time stamp also includes the first test that was run, so there is variability introduced by that but we know that it’s not significant since our longest recorded test in test durations is 7s, so even if we double that we’re saving 2 minutes

With around 1.7k runs (not shards, full runs) of our unit test CI workflow last month, that amounts to roughly 57 hours of time saved. Of course developers usually don’t sit there waiting for CI to finish, so this is a purely theoretical number and the more realistic return is that it feels good in the cases when you are eager for the tests to finish so you can submit a PR for review.

Without this optimization your test setup scales worse and worse through sharding as your test codebase grows and your test shards spend more and more of their time collecting tests and less time actually running tests.

We built pytest-fsplit for this purpose. It’s built with pytest-xdist, pytest-split and nbval in mind. You could of course also use an alternative like rpytest or maelstrom but for us this little pytest plugin was the right fix.

Installation

pip install pytest-fsplit

Usage

First record durations from a complete, unsharded run:

pytest --fsplit-store-durations

Which produces the aforementioned .test_durations.

Then run each file shard separately similar to how you would with pytest-split:

pytest --fsplits 4 --fgroup 1
pytest --fsplits 4 --fgroup 2
pytest --fsplits 4 --fgroup 3
pytest --fsplits 4 --fgroup 4

I was inspired by shopify’s “The Journey to Cloud Development” article series to build such a system for our team. I especially liked how they talked through their iteration process all the way from very early experiments to finding the solution that worked for their developers. They do a great job explaining the decisions, failures and wins along the way.

While it was a great inspiration our path to sandboxes was different. Unlike the developers at shopify, we did not typically run into the issue of having too little compute for running our local dev setup. If anything we had too much compute power and workloads that looked fine on a local machine could fail in the cloud. At work we develop in a typical web application stack, a frontend, a backend API service, worker queues, cache and a database. We already had a stable sandbox with automated deploys, but our CD pipeline to said sandbox was gatekept by approval and typically done in larger releases rather than truly continuously. As a result, testing my changes in our real full stack required waiting until the sandbox branch was merged and deployed to our stable sandbox.

Being able to spin up such sandboxes also opens the door to letting agents test changes in full-stack environments for which we’d want that emulation of our production stack to be as close as possible.

Implementation

So I set out to scope and build a system that would allow us to easily create an ephemeral sandbox off of work in progress. The first question was how the system should be controlled: a signal sent from a local machine (i.e. via a script), a signal sent from a PR or from some other central resource like slack? I decided to tie it to PRs. This leaves a visible and centralized record, it ties it to an individual unit of work and it lets us easily get back to a sandbox if we need to.

I opted to go for a comment-driven workflow powered by GitHub actions because comments are easy to understand and carry the PR context with them. A /sandbox create comment would create a sandbox and similar commands exist to poll the status of a sandbox, to update one with newly pushed changes, and to destroy a sandbox when no longer needed.

Every sandbox should also be uniquely identifiable. Since we base each on a branch, we use the hash of that branch name to create a unique sandbox id.

The next question was around how we should provision the infrastructure. The most naive, faithful and complete option would be to create a 1-1 copy of the sandbox environment, entirely on its own VPC, gateway, with its own ALB, subnets, app workers, db server, etc. This isn’t feasible for cost reasons but also because AWS sets a per-account limit on the max number of resources per region, which is especially low for VPCs, Elastic IPs, and NAT gateways.

So instead I opted for a solution where we reuse most of the existing infrastructure in our sandbox environment. We use our unique sandbox ID to tag all associated resources. This makes it much easier to take them down. The only top level infrastructure resources we provision entirely newly are the app and worker queue tasks in ECS. We route them through the shared ALB, have them use the same REDIS cache instances but with their own queues.

The database required the biggest effort in all of this. I did consider seeding the database with our typical local seed script but decided against it. I wanted my colleagues to have a really familiar environment so they can easily reuse and inspect resources they have created. As a result I decided to use the existing stable sandbox db as the basis for populating ephemeral sandboxes.

Initially I tested out the naive method of copying the entire existing stable sandbox db on every ephemeral sandbox. The idea was that this would have resulted in the most up-to-date data in the fresh ephemeral sandbox db. Unsurprisingly this turned out to be prohibitively slow and would put a lot of strain on the db server. So instead I maintain a template db on the db server which is copied nightly from the stable sandbox db on the same server. When we provision a new ephemeral sandbox we clone from the template db and then apply migrations and security permissions. This is much faster and less strain on the db server. However, it also means that the ephemeral sandbox db lags behind the original sandbox db by the update cadence of the template db.

Impact

Our GTM lead’s Go To Market Lead - in charge of launching products to potential buyers eyes lit up when he realized we’d be able to quickly set up a demo from a feature branch to show a customer a fully-functioning custom integration they might have asked about, all without needing to have our GTM team setup a local coding environment. They just click on the frontend link and do their full demo. Our engineers have been using the sandboxes for demoing and more importantly sharing changes. It has been particularly nice for async review of large coupled frontend/backend reworks.

I myself use the sandboxes whenever I need to test how certain objects work in our cloud environment or when I develop a feature with a frontend component that I want feedback on.

Leftover crumbs

No project like this is perfect and of course right out the gate a few requests have landed.

The most consistent feedback I have gotten is that real-time syncing of the database would be nice. As soon as developers and GTM realized that all their content from stable sandboxes could be available, they asked how easy it would be to keep all the sandboxes in sync. Unfortunately in the current system I don’t see a good way to get true real-time syncing because we cannot expect the db schema to be the same between an ephemeral sandbox and the stable sandbox.

I have also been asked about the reverse, i.e. syncing changes from ephemeral sandboxes back to the stable sandbox. This is similarly challenging but slightly more realistic because we could potentially copy over data after the PR contents make it into the stable sandbox.

Surprisingly the startup time of 14 minutes for a fresh sandbox has not been subject to criticism. Developers find something else to do in the meantime. I have identified work to get about 6 minutes of potential time saves but they are more involved and startup time does not seem to be a big issue currently.

Scalability of ephemeral sandboxes isn’t a huge factor for us as a team right now but when it becomes more important, I will also have to revisit shaping our infra so it can more accurately dynamically scale to the demands of spinning up sandboxes, especially the db.

About the LocalStack-shaped elephant in the room

Why build all this when LocalStack exists? I frankly didn’t know about LocalStack’s sandbox support and only learned about it when Bill Easton was lauding them on LinkedIn. I certainly learned a lot about the wiring and internals of our AWS setup and the limitations of per-account AWS resources in the process and would definitely consider a solution like LocalStack if I was to do this again. That being said, the crunchiest parts of the project, e.g. generating, updating and maintaining the template DB, would have still been required.

Why not build this in Kubernetes

The sandboxing use case does sound like a perfect fit for kubernetes - spinning up and down sandboxes. Our main deployment does however not run on k8s, so if we wanted the sandboxes to model our production stack, we would have to move all of that over to k8s. Most of the difficulty in this process also did not come from starting/shutting down the AWS resources. The most difficult part was getting the database seeding right, and k8s doesn’t help with that. As a result the move to k8s did not seem worth it for our use case. When the time is right, we will revisit a port to k8s. Currently, I am considering moving most of our deployment to a parametrized CloudFormation template so we have an even easier time spinning up/down instances.

What’s next

It’s a running joke at work that my three values are observability, observability, observability. Since the logs from these sandboxes are tagged with the sandbox ID, it should be easy for us to make a nice dashboard with a switcher so we can observe any individual sandbox from the same dashboard instead of creating a new dashboard for each and flooding our dashboard section. Being able to see the backend activity visualized like that is really handy when testing new backend features and will be the next thing I build when I get the time.

This has been my favorite infra project so far because it really had me dig through all aspects of our infra, from the networking intricacies and zero-downtime deployment to db setup strategies. The best part has been the positive response and regular usage from the team, which is what makes me consider this a success.

Other than adding observability and fixing a few issues with the implementation, much like for the developers at shopify, user feedback and user data will tell me what direction I get to push this project next. There is also potential in using these sandboxes for letting agents experiment, but that and all the associated engineering changes might be the subject of a future writeup.

• Installation and setup
• Quick setup
• Minimal viable website
  • Starting jekyll server
  • Adding first content
• Updating preferences
  • Updating (S)CSS
  • Adding socials to the footer
  • Adding a feature image

On GitHub Pages you can host websites like this one for free. There are tons of tutorials out there, here I’m just sharing how I set up this very blog you’re reading.

By default GitHub Pages uses Jekyll for building your website but they will also accept already built static files or your own workflow for building your website.

In this post I am using Jekyll and the template alembic. I will show how and what I modified to make my site.

Installation and setup

We will be using Jekyll and bundler so we can locally preview the website while working on it.

To install Jekyll we’ll first install ruby. On Linux or Windows Subsystem for Linux (WSL) we install like this:

sudo apt install ruby-full build-essential zlib1g-dev

On a mac we do:

brew install ruby

Next we install bundler which we will use to manage the ruby plugins.

gem install bundler

We’ll make our project directory and call it my_jekyll_website

mkdir my_jekyll_website
cd my_jekyll_website

To setup our project we record our dependencies in the Gemfile.

Gemfile

source 'https://rubygems.org'
gem "github-pages", group: :jekyll_plugins
gem "jekyll-remote-theme"

We are using the github-pages plugin so we can test locally. The jekyll-remote-theme plugin lets us use themes other than those preconfigured for GitHub Pages. We will show in the section adding first content how to use a remote theme.

Next we’ll add jekyll to the Gemfile like so:

bundle add jekyll

This will pin a specific version of jekyll, in my case 3.10.

Now we install all the dependencies

bundle install

Quick Setup

The remainder of this post is about setting up elements piece by piece, explaining what they do and how to modify them. If you want to just get started right now and change things later, you can download the GitHub Pages with remote theme kit linked to on the theme’s GitHub. This kit uses reasonable defaults and enables most features from the get-go. You can still always come back to my post and read about how to further modify the theme for your needs.

Minimal viable website

Starting jekyll server

If we want jekyll to start serving a webpage from our directory, we run the following command.

bundle exec jekyll serve --watch

This will keep jekyll running and refreshing as we make changes. The initial output should show us something like this:

To use retry middleware with Faraday v2.0+, install `faraday-retry` gem
            Source: /home/myuser/my_jekyll_website
       Destination: /home/myuser/my_jekyll_website/_site
 Incremental build: disabled. Enable with --incremental
      Generating... 
                    done in 1.573 seconds.
 Auto-regeneration: enabled for '/home/myuser/my_jekyll_website'
    Server address: http://127.0.0.1:4000
  Server running... press ctrl-c to stop.

This also creates a directory called _site which jekyll has populated with an assets folder:

ls -la _site

total 12
drwxrwxr-x 3 tobsecret tobsecret 4096 Feb  5 22:33 .
drwxrwxr-x 3 tobsecret tobsecret 4096 Feb  5 22:33 ..
drwxrwxr-x 3 tobsecret tobsecret 4096 Feb  5 22:33 assets

If we now go to http://127.0.0.1:4000 we see jekyll is serving up the file structure of our _site directory:

This _site directory can have a lot of files in it, so we’d like to not have those show up when we use git for version control, so we’ll make a .gitignore file to ignore this and a few other artefacts that Jekyll can create:

.gitignore

_site
.jekyll-cache
.sass-cache

Adding first content

I am using a Jekyll template called Alembic for my blog which comes with a nice default setup. In our _config.yml we’ll tell Jekyll to use the jekyll-remote-theme plugin and specify the remote_theme: daviddarnes/alembic@main. In the navigation_header section we’ll specify two sections: Home and Blog. For our webpage this will generate a Home and a Blog link at the top of every page in the navigation header.

_config.yml

title: Tobsecret's homepage
description: Programming Blog & Dominion sets
plugins:
  - jekyll-remote-theme
remote_theme: daviddarnes/alembic@main

# 9. Site navigation
navigation_header:
- title: Home
  url: /
- title: Blog
  url: /blog/

The paths in the navigation_header section specify the path in our directory where jekyll should be looking for a file called index.md that has the content for the page.

So for our Home page it will look in /, so directly in our project directory.

index.md

---
layout: page
title: Home
---


---

Latest blog posts:
{% for post in site.posts limit:5 %}
- [{{ post.title }}]({{ post.url }}) — {{ post.date | date: "%Y-%m-%d" }}
{% endfor %}

layout: page is important since Alembic predefines the layouts for page and post, i.e. those pages will get the navigation header and other features of Alembic.

The little loop in the index.md grabs the latest 5 posts from the _posts folder by their date field.

If we refresh our page, it should now look like this: Notice the header now has two links, one to Home and one to Blog.

If we clicked on Blog we would get a 404: File Not Found.

To populate that page we’d have to make another page blog/index.md which I’d just copy what the template uses:

---
layout: page
title: Blog
permalink: /blog/
---


{% if site.posts.size == 0 %}
No posts yet. Create posts in the `_posts` folder.
{% else %}
<ul>
{% for post in site.posts %}
  <li><a href="{{ post.url }}">{{ post.title }}</a> — {{ post.date | date: "%Y-%m-%d" }}</li>
{% endfor %}
</ul>
{% endif %}

Now if we wanted to add our first post, so something actually shows up in the blog, we want to create a file _posts/2026-02-15-first-post.md:

---
layout: post
title: "Writing helps me think"
date: 2026-02-15 12:00:00 +0000
---
When I write about the software I write, I find myself looking up 
a lot more questions than when I am just getting things done.

And now our blog page is updated:

You can similarly also copy the Search section by adding a section in the navigation_header section of your _config.yml:

navigation_header:
- title: Home
  url: /
...
- title: Search
  url: /search/

and copying over the search/index.md:

---
title: Search
excerpt: "Search for a page or post you're looking for"
---

{% include site-search.html %}

Updating preferences

Updating (S)CSS

Usually to update files that the template has pre-defined, we have to provide an alternative copy of that file.

To update the site’s styling via SCSS everything goes through assets/styles.scss which looks like this in the template default:

---
title: false
styles: true
---
@import "alembic";

When Jekyll processes this Sass (.scss), @import "alembic" generates all of the style elements included with the template. This means if we want to change any of those, we have to import or explicitly call them before alembic is imported.

Let’s change the colors via a file _sass/modified_colors.scss. Putting it in the _sass folder means Jekyll will find it when we try to import it.

_sass/modified_colors.scss.

$linkColour: #61AFFF; // Modulex Medium Blue
$hoverColour: #0055BF; // Blue 
$accentColour: #61AFFF; // Modulex Medium Blue

We also have to modify assets/styles.scss to import the colors before alembic gets imported:

assets/styles.scss

---
title: false
styles: true
---
@import "modified_colors";
@import "alembic";

Any additional SCSS can be imported below alembic, e.g. for my dominion section I wrote _dominion.scss to standardize the formatting of the dominion cards and the containers for the expansions.

Adding an RSS feed

RSS feeds can be used to automatically get updated when a new post is published via an RSS reader like Feedly. In Jekyll we only have to add this little line to our _config.yml

plugins:
  ...
  - jekyll-feed

Now our RSS feed is automatically generated and available at <domain>/feed.xml (i.e. tobsecret.github.io/feed.xml).

Adding socials to the footer

By default the template sets up the footer to contain the same navigation as the header, or optionally a clickable link with some text. The template also features a way to include social links and having them displayed in an aside. For my blog I wanted these socials to instead show in the footer.

First let’s set up our socials in _config.yml as defined by the template:

social_links:
  LinkedIn: https://www.linkedin.com/in/tschraink
  GitHub: https://github.com/tobsecret
  RSS: /feed.xml

All we need to do is to create _includes/nav-footer.html with this content:

{% if site.navigation_footer %}
<nav class="nav  nav--footer">
  <ul class="list list--nav">
{% include nav-social.html %}
  </ul>
</nav>
{% else %}
  {% include nav-default.html %}
{% endif %}

The template provides its own _includes/nav-footer.html which instead of just {% include nav-social.html %} includes some logic for handling the text + external link the template would use.

We also have to enable this footer in the _config.yml like so:

navigation_footer: true

And we can now see the socials in our footer:

Adding a feature image

I spent some time to model and render my banner image in Blender and wanted to add it to each post. My banner is assets/banner.webp. We can add this for all posts by adding the following to our _config.yml:

# 5. Collections
defaults:
  -
    scope:
      path: ""
      type: "posts"
    values:
      layout: post # Set the default layout for posts
      feature_image: "/assets/banner.webp"
  -
    scope:
      path: ""
      type: "pages"
    values:
      layout: page # Set the default layout for pages
      feature_image: "/assets/banner.webp"

This won’t update immediately in your browser so you’ll have to interrupt your bundle exec jekyll serve --watch call with CTRL+C and then restart it. Refresh the page and you should see the banner at the top.

You can still manually override it in any particular page or post by specifying a feature_image in the markdown:

---
layout: page
title: Blog
permalink: /blog/
feature_image: /assets/some_other_banner.png
---