Advanced Git Internals: Packfiles & Delta Compression

At an architectural level, Git is a Directed Acyclic Graph (DAG) of cryptographic objects built on content-addressable storage. In a repository with years of history, storing every single file revision as an individual loose object in .git/objects would quickly consume millions of filesystem inodes and drastically degrade disk I/O performance.

To achieve unmatched speed and storage efficiency, Git utilizes a sophisticated packing subsystem based on Packfiles (.pack), Pack Indexes (.idx), Sliding-Window Delta Compression, and Reachability-Based Garbage Collection (git gc).

┌─────────────────────────────────────────────────────────────┐
│                 Loose Objects vs Packfile Compression       │
├──────────────────────────────┬──────────────────────────────┤
│ Loose Objects (.git/objects/)│ Packed Database (.pack/.idx) │
├──────────────────────────────┼──────────────────────────────┤
│ ├── 7f/8a9b0c... (1 file)    │ ├── pack-1a2b3c.pack         │
│ ├── 3c/4d5e1f... (1 file)    │ │   (Thousands of compressed │
│ ├── 9a/0b1c2d... (1 file)    │ │    objects with deltas)    │
│ └── ... (Millions of files)  │ └── pack-1a2b3c.idx          │
│                              │     (Fast binary search index│
│ Heavy filesystem inode cost. │      with O(log N) lookups)  │
└──────────────────────────────┴──────────────────────────────┘

1. Sliding-Window Delta Compression

When you commit a 10MB source code file with a 1-line modification, Git does not store two complete 10MB files in the long term. During repository optimization or network push:

Git sorts objects by filename and file size in a sliding window.
It identifies similar objects and encodes the latest version as a complete object (base) and previous revisions as compact binary Deltas (storing only byte insertions/deletions).
Storing deltas against the newest version optimizes for checking out the most recent code instantly without walking historical diff chains.

2. Packfiles and Pack Index Binary Trees

.pack File: A single concatenated binary archive containing thousands of zlib-compressed objects and deltas.
.idx File: A binary index containing a 256-entry first-level fan-out table and sorted SHA hashes, enabling Git to find the exact byte offset of any object in the .pack file in $O(\log N)$ time.

# Verify packfile integrity and inspect compression statistics
git verify-pack -v .git/objects/pack/pack-*.idx

3. Garbage Collection & Object Reachability (`git gc`)

Commits that are detached (no branch or tag points to them) and older than the reflogExpire threshold (default: 30–90 days) are considered Unreachable Objects.

# Optimize repository: packs loose objects, prunes unreachable objects
git gc --aggressive --prune=now

# Find dangling unreachable commits/blobs in local storage
git fsck --lost-found

Summary & Key Takeaways

Git packs loose objects into .pack archives indexed by .idx files for ultra-fast lookups.
Delta compression encodes historical revisions as binary differences against base objects.
git gc cleans up unreachable objects and compresses the repository object database.
Directed Acyclic Graph (DAG) structures guarantee cryptographic immutability across repository history.

Best Practices & Senior Guidance

Run git gc on Large Monorepos: Periodic maintenance packs loose objects and optimizes commit-graphs for rapid branch switching.
Never Manually Delete Files in .git/objects/pack/: Deleting pack files permanently corrupts the entire repository history.