Product Update: Improved Git Reliability (Near-Zero Failure Rate)

TL;DR

We recently saw an increase in intermittent Git push errors during Full and Partial pushes.

The issue was not caused by customer content. It was caused by timing conflicts in the final GitHub delivery step, where multiple push jobs could attempt to update the same branch at nearly the same time.

We’ve now upgraded our GitHub “last mile” push logic to detect these conflicts, verify the real branch state, and automatically retry using the latest GitHub HEAD when needed.

The Full Breakdown

The Headless Hostman static deploy pipeline relies on GitHub for version control and the last mile of delivery to the live environment.

Historically, occasional GitHub edge cases during a Full or Partial push were rare. When they happened, the practical solution was usually simple: repush and try again.

In the last 90 days, this once-in-a-while error became more noticeable, so we reviewed and improved our GitHub API integration.

What was happening

Headless Hostman pushes your site to GitHub using GitHub’s API, not the traditional git push command line flow.

Under the hood, every push follows this general pattern:

1. Fetch the latest commit, also known as HEAD
2. Build a new file tree
3. Create a commit
4. Update the branch reference

The most important failure we identified was GitHub’s non-fast-forward protection.

A non-fast-forward error does not mean the content failed to generate, and it does not necessarily mean the files were wrong. It means the branch changed after our system fetched HEAD but before our system finished updating the branch.

In plain English:

• Push A starts and reads the current GitHub branch
• Push B starts around the same time and reads the same branch
• Push B finishes first and updates GitHub
• Push A then tries to update GitHub using the older branch state
• GitHub rejects Push A to prevent accidentally overwriting newer work

That rejection commonly appears as:

Update is not a fast forward

This is GitHub doing the safe thing. GitHub refuses non-fast-forward updates because applying them blindly could lose commits that were already added to the branch.

Why it became more noticeable

Concurrent jobs are not inherently bad. In fact, Headless Hostman has handled them successfully for years.

What changed is that our usage patterns have grown more demanding:

• More automated push events
• More cron-driven updates
• More batch jobs
• More multisite activity
• Larger pushes with more files
• Longer windows between fetching HEAD and updating the branch

Those factors increase the chance that another job updates the branch while a push is already in progress.

So the better way to describe the issue is not “GitHub broke” or “content failed.” It is this:

As our platform scaled, we began hitting GitHub’s normal fast-forward safety rules more often.

What we changed

We implemented a self-healing push system for the final GitHub update step.

1. Smarter verification after every branch update

After attempting to update the branch, we now immediately re-check the live GitHub branch.

This allows us to confirm whether our commit actually landed, even if the API response was incomplete, delayed, or not shaped the way we expected.

2. Automatic retry using the newest HEAD

If GitHub rejects the update because the branch moved, we now retry safely.

The retry process does this:

1. Fetches the newest GitHub HEAD
2. Fetches the correct base tree from that newest commit
3. Rebuilds our file tree on top of the updated branch state
4. Creates a new commit using the newest HEAD as the parent
5. Attempts the branch update again
6. Re-checks GitHub one final time to confirm success

This is effectively a lightweight automatic rebase for the GitHub API workflow.

3. Built-in safety controls

The retry only happens once.

That gives the system a chance to recover from normal timing conflicts without creating infinite loops or runaway API calls.

If the retry still fails, we log the exact failure stage and preserve the failed push snapshot so we can continue improving the system.

4. Better logging and observability

We added clearer logging around the final branch update step.

When a fast-forward conflict occurs, our logs now explicitly show that we detected the conflict and retried using the updated GitHub HEAD.

If the retry fails, we capture the attempted commit, the current GitHub HEAD, the tree items involved, and the GitHub response.

The result

Before these changes, intermittent GitHub push failures had become more frequent than we were comfortable with.

After the first round of improvements, failures dropped dramatically.

With this latest retry and verification logic in place, we expect these errors to return to “once in a blue moon” territory.

More importantly, when this edge case does occur, Headless Hostman now has a built-in recovery path instead of requiring a manual repush.

What this means for you

• Your Full and Partial pushes are significantly more reliable
• GitHub timing conflicts are now handled automatically
• Failed push logs are more detailed and actionable
• No action is required on your end

Final note

This issue is not unique to Headless Hostman. It is a known challenge when working directly with Git APIs at scale.

Git is designed to protect branch history. When two updates happen close together, GitHub may reject the older update rather than risk losing newer commits.

The difference now is that Headless Hostman detects that situation, adapts to it, and retries safely using the latest branch state.