Rotate the Secret First

You’ve just pushed code and then, horror of horrors, you see it: a live API key, a password, or a sensitive token staring back at you from the GitHub UI. We’ve all been there. The immediate instinct might be to panic and start rewriting history. But before you touch a single Git command, there’s one crucial, non-negotiable step: rotate or revoke the compromised secret.

The moment a secret hits a remote repository, especially a public one, assume it’s compromised. Automated bots are constantly scraping platforms like GitHub for credentials. By the time you notice the mistake, the secret might already be in the wild. Therefore, securing the system the secret protects is paramount. Revoke the old key, generate a new one, and update any services that rely on it. This must be done before you attempt to clean your Git history. This single step ensures that even if the leaked secret is used, its impact is immediately nullified.

Immediate Fix: The Quick Save

If you catch the mistake immediately after committing and pushing, you might be able to get away with a simpler approach. This is often referred to as a “quick save” or a “soft reset” for secrets.

The goal here is to rewrite the most recent commit. If the secret is only in the very last commit you pushed, you can amend that commit. First, ensure the secret is no longer in your working directory or staging area. Then, use the following commands:

# Remove the secret file from your working directory and staging area
git rm --cached path/to/your/secret/file

# Amend the last commit to remove the file from history
git commit --amend --no-edit

# Force push the rewritten history. Use with caution!
git push --force-with-lease origin main # or your branch name

The git rm --cached command removes the file from Git’s tracking but keeps it in your local filesystem. The git commit --amend --no-edit command rewrites the last commit without changing the commit message. Finally, git push --force-with-lease is critical. It overwrites the remote branch history with your locally rewritten history. The --force-with-lease option is safer than a plain --force as it checks if someone else has pushed to the branch since your last pull, preventing accidental overwrites of others’ work.

Deeper Clean: Rewriting Multiple Commits

What if the secret has been lurking in your history for several commits? In this scenario, the “quick save” isn’t enough. You need to rewrite a more extensive portion of your Git history. The most powerful tool for this is git filter-repo. This tool is the modern, recommended replacement for the older git filter-branch, which is known to be slow and complex.

git filter-repo allows you to rewrite history by removing files, changing file contents, and more. To remove a specific file throughout your repository’s history, you can use the following command:

# Install git-filter-repo if you haven't already
# pip install git-filter-repo

# Run filter-repo to remove a file from all history
git filter-repo --path path/to/your/secret/file --invert-paths

The --invert-paths flag tells git filter-repo to keep everything *except* the specified path. This effectively removes the file and all its history from the repository. It’s a thorough clean. After running this, you will have rewritten your repository’s history. All previous commit SHAs will change. This is a significant operation and requires a force push, just like the amend method, but on a much larger scale.

Important Note: Rewriting history, especially using tools like git filter-repo, is a destructive operation. It changes the SHA identifiers of all subsequent commits. If you are working on a shared branch, this will cause problems for your collaborators. You must coordinate with your team before performing such an operation. Ideally, this is done on a branch that only you have access to, or after coordinating a team-wide cleanup and re-pull.

Diagram illustrating Git commit history before and after scrubbing a secret file

The Nuclear Option: BFG Repo-Cleaner

For extremely large repositories or very sensitive situations, the BFG Repo-Cleaner is another powerful tool. It’s designed to be faster and simpler than git filter-branch for the specific task of removing large files or sensitive data.

To use BFG, you first need to download the JAR file. Then, you clone your repository using the --mirror option, which creates a bare repository clone. This is essential because BFG operates on the raw Git object database.

# Clone your repository as a bare mirror
git clone --mirror git@github.com:your-username/your-repo.git
cd your-repo.git

# Run BFG to remove the secret file (replace 'secret.txt' with your file name)
java -jar /path/to/bfg.jar --delete-files secret.txt

# Clean up the repository after BFG has run
git reflog expire --expire=now --all
git gc --prune=now --aggressive

# Push the cleaned history to the remote (force push required)
git push --force

BFG’s syntax is straightforward for removing files. After running BFG, the crucial steps are cleaning up the Git object database using git reflog expire and git gc. These commands remove the old, unreferenced objects that contained the secret. Finally, a force push is required to update the remote repository. Like git filter-repo, this rewrites history and requires careful coordination with your team.

Coordinating with Your Team

Rewriting Git history is not a trivial task, especially in a collaborative environment. If the secret was committed to a shared branch, or if other developers have based their work on commits that contain the secret, a simple force push can cause significant disruption.

Here’s a suggested workflow for team-wide cleanup:

  1. Communicate: Announce the incident and the planned cleanup to your team. Explain the risks of rewriting history.
  2. Rotate Secrets: Ensure all compromised secrets are rotated first.
  3. Choose a Tool: Decide whether to use git filter-repo or BFG Repo-Cleaner based on repository size and complexity.
  4. Cleanup on a New Branch: Perform the history rewrite on a fresh clone or a dedicated branch. This isolates the destructive operation.
  5. Coordinate Pulls: Instruct all team members to:

    • Fetch the latest changes.
    • Discard their local copies of the affected branch.
    • Re-clone the repository or check out the cleaned branch from the remote.

    This ensures everyone starts from the same, clean history. It’s like rebuilding a house foundation after discovering a structural flaw – everyone needs to move out and move back in once the repairs are done.

  6. Verify: After the push, have multiple team members confirm that the secret is no longer present in the history.

While rewriting Git history is a powerful capability, it should be used judiciously. For secrets that are truly sensitive and have been exposed, it is the correct, albeit disruptive, course of action. For less critical mistakes caught immediately, amending the last commit is often sufficient. The key takeaway is to always rotate secrets first, then clean your history, and communicate thoroughly with your team.