ZeroS3: Building an S3-Compatible Content-Addressed Store in One Go File - Zero Dependency Hackathon 2026 - Hackathon Raptors
Written for the Zero Dependency 2026 Write-Up Side Quest, organized by Hackathon Raptors.
In August 2026, I entered the Zero Dependency Hackathon with a target that was deliberately uncomfortable: build an S3-compatible object store in Go without using a single third-party dependency. The easiest way to make that manageable would have been to keep narrowing the problem until it fit the constraint. A small server implementing the common S3 operations, Signature Version 4, and straightforward file-backed persistence would have been a reasonable submission, and it was not far from what I expected to build at the beginning.
ZeroS3 ended up going in a different direction. Instead of treating an S3 object as one opaque blob, the storage engine divides object data with content-defined chunking, stores the resulting chunks by SHA-256 in a content-addressed store, describes objects with immutable manifests, and separates those immutable structures from the append-only journal that determines what is visible in the namespace. Once that representation existed, features that had initially sounded like separate projects—deduplication, delta replication, peer-assisted repair, copy-on-write namespace forks, snapshots, and structural inspection—started reducing to different operations over the same underlying data.
The final result is ZeroS3, a self-hosted S3-compatible object store whose implementation lives in a single Go 1.27 source file with no third-party dependencies. Ordinary clients such as the AWS SDK, AWS CLI, and rclone can use its S3 interface without ZeroS3-specific client code. Underneath that familiar interface is a storage model that ended up becoming much more important than the protocol surface I had originally set out to implement.
The most interesting part of the project was therefore not getting S3 to work without an SDK. It was seeing what happened when the constraints forced the lower layers of the system to become explicit, and then finding that those lower-level decisions made some of the more ambitious parts of the project easier rather than harder.
- Source code: github.com/insightlabs38-pixel/zeros3
- Demo video: ZeroS3 demo on Vimeo
The repository contains the complete single-file implementation, first-party test suite, S3 compatibility documentation, standard-library dependency log, reproducible-build script, and generated zero-dependency evidence. The demo gives a shorter visual walkthrough of the storage architecture and shows structural reuse, delta replication, copy-on-write forks, snapshot restore, peer-assisted repair, and the final verification evidence.
A zero-dependency storage project loses more of the usual toolbox than it might seem at first. A conventional implementation could reasonably rely on separate packages for AWS signing, S3 protocol behavior, routing, content-defined chunking, UUID generation, structured persistence, embedded databases, worker pools, retries, or process lifecycle handling. ZeroS3 had to implement those behaviors directly or construct them from the Go standard library.
The resulting go.mod is intentionally uneventful:
module zeros3
go 1.27.0There is no require block, and the release binary can be built directly from the implementation file:
CGO_ENABLED=0 go build \
-trimpath \
-buildvcs=false \
-ldflags="-buildid=" \
-o zeros3 \
zeros3.goI also chose to pursue the hackathon's Single File bonus. The production implementation therefore remains in zeros3.go, with the organizer-approved first-party tests in zeros3_test.go. By the end of the project both files were large enough to need subsystem maps for navigation, but the number of files was never the most consequential restriction. The more important effect was that there was nowhere to delegate the fundamental behavior of the system. There was no embedded database quietly defining the crash model, no AWS library deciding how canonical paths should be represented, and no content-defined chunking package determining how object data was segmented.
That requirement to own the lower layers eventually shaped the entire project. Decisions about content identity, persistence, visibility, failure recovery, and network transfer were no longer implementation details hidden behind libraries; they became parts of the architecture that had to agree with one another.
One of the ideas that changed the direction of the project came from something I had been reading about before the architecture settled. I was looking into how Hugging Face handles very large collections of models and datasets and ended up reading about its adoption of Xet. Instead of treating every revision of a large file as a completely separate blob, Xet uses content-defined chunking and content-addressed storage to identify common data across files and revisions. Small changes can therefore leave most of the underlying content unchanged.
What originally interested me was the storage economics behind that design. Large model checkpoints, datasets, and other binary artifacts often contain substantial overlap between revisions, so storing and transferring them only at whole-file granularity throws away information that the storage system could use. Once content is divided along stable, content-dependent boundaries and identified independently of the file that contains it, two different files can share physical storage without having to know anything about each other's history.
While thinking about ZeroS3, I realized that the same family of ideas could be applied in an unusual place. Xet's production architecture ultimately persists data into object storage; ZeroS3 could invert that relationship and expose object storage as the interface while putting a content-aware representation underneath it. From a client's perspective, a PUT would still be an S3 PUT. Internally, however, the server would know that much of a supposedly new object might already exist.
I was not trying to reproduce Xet. Its production architecture, scale, workload, metadata structures, and integration with the Hugging Face Hub solve a different problem. The idea I wanted to borrow was more fundamental: content-defined boundaries combined with immutable content identity. If I could build that much using only the standard library, ZeroS3 could remain conventional on the outside while gaining structural information that an ordinary blob store does not possess.
That was the point where the project stopped looking like a small S3-compatible development server and started acquiring its own storage architecture.
The other major source of leverage was the development workflow. Claude Code was the primary implementation agent, while I used ChatGPT more heavily for architecture, milestone planning, scope decisions, adversarial review, and evaluating the evidence produced at each stage. That made it possible to build substantially more during the hackathon than I could have implemented manually in the same time, but raw code-generation speed was not the part of the workflow that mattered most as the project grew.
The harder problem became controlling context. ZeroS3 was becoming a large systems program while still living in one implementation file, and repeatedly asking an agent to read the entire development history before making a change would have become both expensive and unreliable. I instead kept one implementation writer active at a time and divided development into relatively narrow milestones. The repository, its tests, subsystem maps, and planning documents became the persistent state between sessions, while each new task carried only the architectural constraints and context relevant to that particular change.
A task could therefore be framed as adding peer-assisted repair while preserving the existing CAS integrity and publication invariants, rather than as a vague instruction to continue developing ZeroS3. Once the implementation was complete, I would review the result separately, looking for incorrect assumptions, missing adversarial cases, or places where the new feature had accidentally created a parallel mechanism instead of composing with the existing storage model. If a milestone exposed a new invariant, that invariant became part of the repository and the next task inherited it.
I also increasingly tested the built server from outside its own package. Real HTTP clients, actual process restarts, SIGKILL scenarios, concurrent writers, deliberate on-disk corruption, AWS clients, and rclone were useful precisely because they did not care what the implementation believed it had done. A unit test can accidentally encode the same assumption as the code it is testing; an ordinary client or an operating-system process boundary is much less cooperative.
By the final stages, the implementation had reached 12,765 lines in one Go source file and the first-party test file had grown beyond 27,000 lines. At that scale, “give an agent everything and ask it to reason globally” would have been a poor development model. Narrow tasks, persistent invariants, source maps, accumulated tests, and independent review made the project much more manageable.
The useful unit of context turned out not to be the conversation. It was the repository plus a sufficiently precise task. That development structure was a large part of why the scope could expand without the later milestones becoming disconnected experiments.
The simplest persistent object-store representation is also the obvious one:
bucket/key
↓
one file on disk
A PUT writes the file, a GET reads it, and a DELETE removes it. Metadata can live beside the object or inside a small index. For a lightweight S3-compatible server, that representation goes surprisingly far, and it would have been enough for a much smaller version of ZeroS3.
Its limitations become more visible when the questions stop being about individual requests and start being about relationships between objects. Suppose a 12 MiB object changes by only 4 KiB near the beginning. If each version is an opaque blob, the second object is simply another 12 MiB payload. A remote replica normally sees another 12 MiB object to transfer, a copy needs another logical object body, and a snapshot needs some separate mechanism to retain the earlier state. Localized repair either operates on the whole object or requires introducing a second lower-level representation alongside the first.
The storage engine knows almost nothing about the fact that the old and new objects are overwhelmingly identical. That was the ceiling I wanted to remove. Instead of making the S3 object the fundamental unit of payload storage, ZeroS3 moves one level down and makes content the fundamental unit.
The resulting architecture can be summarized like this:
flowchart TD
A[S3 object bytes] --> B[Content-defined chunking]
B --> C[SHA-256 content-addressed chunks]
C --> D[Immutable object manifest]
D --> E[Namespace visibility]
J[Append-only visibility journal] --> E
C --> F[Deduplication]
C --> G[Delta transfer]
C --> H[Peer repair]
D --> I[Copy-on-write forks]
D --> K[Snapshots]
D --> L[Diff and inspect]
Object data is divided into chunks, each chunk is identified by the SHA-256 digest of its contents, and those chunks live in a content-addressed store, or CAS. An immutable manifest records the ordered sequence of chunks required to reconstruct an object together with the relevant object metadata. Namespace visibility is managed separately through an append-only, checksummed journal.
The separation between immutable data and mutable visibility is one of the more important parts of the design. Once a CAS chunk exists, its contents do not change. Once an object manifest has been written, that manifest does not change either. Updating an object means constructing new immutable state and then changing which root the namespace publishes, rather than mutating the previous object representation in place.
That makes the visibility journal the commit boundary. During a write, new chunks can exist before the corresponding object becomes visible. If the process fails at that stage, some unreachable content may remain on disk until garbage collection removes it, but the namespace never claims that an incomplete object exists. The dangerous failure mode is the reverse: acknowledging an object as visible while some chunk or manifest required to reconstruct it has not yet crossed the appropriate durability boundary. ZeroS3 orders publication to avoid that state.
On recovery, the journal reconstructs the visible namespace while the immutable payload remains independently addressable by digest. The same distinction also gives garbage collection a clean model: live namespace entries, snapshots, and other durable roots determine which manifests and chunks remain reachable, while unreferenced content can eventually be reclaimed.
This arrangement matters because none of the higher-level capabilities use a second payload representation. Replication, repair, forks, snapshots, structural diff, and ordinary S3 operations all work over the same chunks, manifests, and visibility model. Most of ZeroS3's later functionality came from finding new ways to use those primitives rather than adding unrelated storage systems beside them.
Content addressing by itself does not preserve sharing particularly well across insertions and deletions. If every object is divided into fixed 64 KiB blocks, inserting a few bytes near the beginning changes the alignment of everything that follows. The underlying data may remain almost identical, but most of the resulting block hashes change because the block boundaries have shifted.
Content-defined chunking chooses boundaries based on the data itself rather than fixed byte offsets. ZeroS3 uses a Gear-style rolling fingerprint with a 16 KiB minimum chunk size, a target around 64 KiB, and a 256 KiB maximum. As the chunker scans a byte stream, it chooses boundaries when the rolling state satisfies its boundary condition while still respecting those size limits. A localized edit may disturb the chunks around the changed region, but after some distance the chunker tends to resynchronize with the original stream.
The practical result is that a small edit can remain a small structural change. In one of the final demo fixtures, I modified 4 KiB near the beginning of a 12 MiB object. The two versions contained 183 chunks, and ZeroS3 found 182 shared chunk identities:
shared unique chunk IDs: 182 / 183
structural reuse: 99.7%
Instead of treating the second 12 MiB version as wholly new data, the storage engine could see that almost all of its physical structure was already present. Deduplication was the most immediate use for that information, but it soon stopped being the most interesting one.
Every chunk in the CAS is named by its SHA-256 digest. If a chunk with that digest already exists, writing identical bytes again does not create another physical payload copy. An unrelated object, another revision, a forked namespace, or a snapshot can all reference the same immutable content.
Conceptually, two objects can converge on the same stored chunk without having any special relationship at the object layer:
object A ─┐
├──► chunk 4c9...
object B ─┘
The difficult part was not checking whether two hashes matched; it was keeping the publication and durability semantics correct around that reuse. A manifest cannot become visible before the chunks it references are durable. A failed operation can leave unreachable CAS content behind without damaging namespace state, but it must never create a visible root that depends on incomplete data.
This distinction between payload existence and namespace publication recurs throughout the rest of the design. Content addressing makes sharing cheap, but correctness still depends on being precise about when an operation is allowed to make a particular immutable root visible.
Once objects were manifests of content-addressed chunks, remote replication no longer had to begin by transferring an object body. The source can describe the chunks required by an object, the destination can determine which digests it already owns, and the network only needs to carry the difference.
At the simplest level:
source object:
[A] [B] [C] [D] [E]
destination CAS already has:
[A] [B] [D] [E]
payload that must move:
[C]
The destination does not need to know why it already possesses a chunk. The bytes may have arrived as part of another object, another revision, a previous replication, or a completely different namespace. If the digest is the same, the content is the same for the purpose of reconstruction.
ZeroS3's replication dry-run uses the same planning path as a real transfer, which means it can predict the missing payload before any chunks are sent. In one final fixture, the source object was 16 MiB and contained 238 chunks. The destination had never seen that exact object, but structurally related content meant it already owned 237 of the required chunks. The dry run predicted:
Would transfer: 77.66 KiB
The actual replication later reported:
Uploaded payload: 77.66 KiB
The predicted and real payload were identical. More importantly, the optimization did not depend on a special relationship between the two objects or on maintaining explicit delta chains. The storage representation itself contained enough information to discover the overlap.
That was one of the points where the content-addressed design stopped feeling like an internal storage optimization. The outside of the system still looked like S3, but the server could now exploit relationships between data that an ordinary opaque-blob representation would never expose.
Copy-on-write namespace forks followed from almost the same model. If payload and manifests are immutable, cloning a namespace does not require copying its object bodies. The new namespace can initially publish references to the same immutable roots:
before fork
prod ─────► manifest ─────► A B C
after fork
prod ─────┐
├────► same immutable content
fork ─────┘
Creating a fork can therefore add zero new CAS payload. In the demo fixture, ZeroS3 cloned 4 MiB of logical object data while reporting:
CAS payload bytes added: 0 B
After the fork existed, I modified only one of its objects. The edited version still shared 58 of 59 chunks with the source, or 97.9% structural reuse, while the original object's checksum remained unchanged. The namespace had diverged immediately, but physical payload diverged only where the content itself changed.
The first implementation of this feature also exposed one of the bugs that clarified the architecture. Structural sharing worked correctly, but the publication rule was too permissive. A fork was intended to be create-only at its destination, yet the initial path could overwrite an existing destination. Fixing that exposed a related resume case: after an interrupted operation, a retry could accept an already-published destination only when that state represented the exact object the original operation had intended to create.
The bug had very little to do with copying data. It was about the distinction between content identity and namespace authority. The fact that payload already exists does not mean an operation is entitled to publish a particular root under a particular name. Once that distinction was explicit, it became useful in replication, conditional writes, snapshots, and garbage collection as well.
Snapshots use the same immutable structures without introducing another object-body format. A snapshot records the roots representing the visible namespace at a particular point in time, and those roots keep the corresponding manifests and CAS chunks reachable even if the live namespace later changes or deletes them.
Restoring a snapshot can therefore republish existing content rather than rebuilding it from a separate archive. In the final demo, I deleted the live source namespace and restored its snapshot into a new namespace. The restore reported:
New CAS payload: 0 B
The payload had never needed to move. The snapshot's job was to preserve durable reachability to the immutable storage graph after the live namespace stopped referring to it.
That model also makes snapshot integrity relevant to garbage collection. If a durable snapshot descriptor is corrupt, ignoring it would be unsafe because the unreadable descriptor might still refer to content that must remain live. ZeroS3 therefore treats uncertainty conservatively rather than allowing garbage collection to infer that an unreadable root does not matter. The difficult part of snapshots was not copying bytes; it was defining which roots survive, how those roots become durable, and what failure behavior is safe when reachability cannot be established with confidence.
The obvious reason to hash chunks was deduplication, but the same digest later became useful as an integrity and repair primitive. If a stored chunk is corrupt, its manifest already says exactly which digest belongs at that position. ZeroS3 can ask a healthy peer for bytes corresponding to that digest, then independently hash the response and install it only if the result matches the expected SHA-256.
target CAS healthy peer
chunk abc
CORRUPT chunk abc
│ │
└──── request digest abc ──────┘
│
▼
SHA-256 verify
│
▼
atomic replace
For the final repair demo, I deliberately corrupted a CAS chunk on disk. Deep verification detected the corrupt content and failed. ZeroS3 fetched the required digest from a peer, independently verified the returned bytes, atomically replaced the damaged chunk, and then completed a second deep verification successfully. After the server restarted, an ordinary S3 GET reconstructed an object with the same SHA-256 as the original.
The same digest was now serving as storage identity, deduplication identity, and repair verification. Repair had not been part of the system I originally expected to build, but once content identity was already cryptographically defined, it no longer required inventing a separate integrity model.
The content-addressed storage model would have been much less useful if clients had to understand it. ZeroS3 therefore keeps the outside of the system deliberately ordinary. The final compatibility work included the AWS SDK, AWS CLI, and rclone using the S3 interface without ZeroS3-specific client behavior.
That meant implementing enough of S3's HTTP semantics and AWS Signature Version 4 to satisfy real clients rather than only hand-written requests. The cryptographic components themselves are straightforward standard-library primitives such as crypto/hmac and crypto/sha256; much of the difficulty lies instead in constructing exactly the request representation that AWS signing expects. Canonical paths, raw paths, query parameters, header normalization, payload hashes, signing-key derivation, and presigned requests all interact, and a superficially harmless normalization can produce a different signature.
One bug exposed this particularly well. Remote operations originally constructed URLs in a way that worked for ordinary object names but failed when keys contained characters such as %, #, or ?. A logical S3 object key, an escaped URL path, and the raw request target are not interchangeable representations, especially when the authentication scheme incorporates the path into a signature. Fixing the issue meant preserving the relevant raw and escaped path semantics through request construction instead of letting a later URL layer reinterpret the key.
It was a useful reminder of where compatibility work usually becomes difficult. Implementing an endpoint called PutObject is not the hard part; reproducing the behavior that ordinary clients quietly rely on at the edges is.
The final major performance change began with a question about remote delta replication. The protocol was already saving substantial bandwidth, but throughput under nontrivial latency was lower than I expected. My first suspicion was the planner: ZeroS3 had to compare chunk descriptors, determine what the destination already owned, and construct the missing transfer set, so negotiation seemed like a plausible place for optimization.
Measurement showed that assumption was wrong. Planning accounted for only around 0.4% of total wall time, which meant a more sophisticated negotiation structure would have had almost no effect on the problem I was actually seeing.
A controlled latency proxy was much more revealing. Adding ten milliseconds of latency per request caused throughput to collapse roughly in proportion to the number of missing chunk requests multiplied by that latency. The client was transferring missing chunks serially, so a protocol that saved a great deal of bandwidth could still spend most of its wall time waiting for request-response round trips.
That produced a slightly counterintuitive result: workloads with very high structural reuse could still have poor effective throughput when their relatively small amount of missing data was fragmented across several latency-bound HTTP requests. The system was saving bytes while wasting time.
The fix was bounded parallel chunk transport. ZeroS3 uses a controlled number of workers rather than launching an unbounded goroutine for every chunk, reuses a long-lived HTTP transport, propagates cancellation when an operation fails, and still keeps namespace publication behind the appropriate validation and commit boundary. Under a controlled ten-millisecond RTT with 256 MiB of missing payload, the measured results were:
| Transfer workers | Median time | Throughput | Speedup |
|---|---|---|---|
| 1 | 58.655 s | 4.36 MiB/s | 1.00× |
| 2 | 31.031 s | 8.25 MiB/s | 1.89× |
| 4 | 17.031 s | 15.03 MiB/s | 3.44× |
| 8 | 9.845 s | 26.00 MiB/s | 5.96× |
| 16 | 7.170 s | 35.70 MiB/s | 8.18× |
The scaling also makes the nature of the bottleneck clearer than the two endpoint measurements alone. Throughput improves substantially at every tested level of concurrency, with diminishing returns as other costs begin to matter. The normal default remains eight workers rather than sixteen; the experiment was intended to establish that serialized request latency was the bottleneck, not to claim that the largest tested worker count is universally optimal.
The benchmark itself also produced a useful failure before those measurements became trustworthy. Its deterministic fixture generator initially used an LCG whose selected seeds unexpectedly produced cross-fixture collisions, making supposedly unrelated data appear more structurally similar than intended. I corrected the fixture before relying on the results. Benchmark infrastructure is quite capable of producing persuasive evidence for the wrong conclusion if its inputs are not treated with the same suspicion as the program under test.
By the final stages of the hackathon, adding another ordinary S3 operation was no longer the most interesting architectural direction. ZeroS3 already covered the compatibility paths needed for normal AWS clients and rclone, while the underlying storage engine had expanded into delta replication, peer-assisted repair, copy-on-write namespaces, durable snapshots, conditional publication, structural inspection, and garbage collection over several kinds of durable roots.
The next major steps would begin changing the class of system rather than merely extending the current one. Multi-node high availability introduces distributed agreement over namespace visibility and failure recovery. Erasure coding introduces placement policy, reconstruction, and failure domains. A CAS operating at much greater scale would eventually benefit from packed chunk storage, persistent indexing, compaction, and different approaches to garbage collection. A serious IAM, STS, or KMS implementation would add authorization and key-management systems that are largely orthogonal to the content-addressed engine.
Those are plausible directions for ZeroS3, but I do not think of them as missing pieces required to complete the current idea. Each would introduce another authoritative subsystem with its own consistency rules and failure model. At that point the work becomes less about seeing how far the existing storage primitives compose and more about designing the next architectural layer around them.
There were also ideas closer to the current design that I chose not to pursue after measuring them. A Merkle-style hierarchy for replication negotiation sounded attractive in the abstract, for example, but once the existing planner measured at roughly 0.4% of runtime, increasing the sophistication of negotiation would have optimized the wrong part of the system. The point where I stopped was therefore not an artificial feature ceiling; it was the point where the most interesting next work would have required beginning a substantially different layer of the architecture.
The final first-party test file is more than twice the size of the production implementation. That was not a target I set at the beginning. It happened because, as more capabilities became different operations over the same CAS, manifests, journal, and reachability rules, failures increasingly crossed subsystem boundaries.
Garbage collection has to understand more than ordinary visible objects because snapshots can keep otherwise deleted content reachable. Forks depend on publication semantics as well as structural sharing. Replication must handle destination mutation between planning and commit. Repair cannot trust the peer from which it retrieves a replacement chunk. Conditional S3 operations have to make their condition check and publication atomic with respect to competing writers. Crash recovery and shutdown behavior need to be tested through actual process boundaries rather than inferred from functions that happen to pass in-process tests.
The frozen candidate completed 738 internal tests, passed the Go race detector, and was clean under go vet and gofmt. I also exercised the running binary through process-level and client-level scenarios covering real restarts, SIGKILL recovery cases, multipart persistence, concurrent conditional writers, replication conflicts, corrupted CAS payload, incorrect repair responses, copy-on-write isolation, snapshot reachability, and interrupted operations. Across the final compatibility and adversarial runs, 2,734 checks passed with zero failures.
Rclone provided one of the more useful final interoperability tests because it exercised the server as an ordinary external client rather than through a purpose-built API. The frozen candidate accepted a fresh 1 GiB upload through rclone's normal multipart path, receiving 205 real parts. The downloaded object matched the source SHA-256, and the same integrity check passed again after restarting the server.
I also ran the same frozen AWS SDK compatibility probe against both ZeroS3 and s3rver 3.7.1, changing only the connection and credential configuration required by each target. Both completed all 14 checks successfully. That does not imply that the two systems have identical S3 compatibility surfaces, but it provides a concrete comparison for the behavior that the probe actually tested.
The verification work eventually became less about accumulating a large test count and more about protecting the invariants that several parts of the storage engine shared. Once replication, snapshots, repair, forks, and garbage collection all depend on the same underlying representation, a test of one subsystem can also be a test of assumptions made elsewhere.
A fully standard-library Go binary with CGO disabled also made reproducibility a property that could be tested directly. For the frozen release candidate, the verification script copied zeros3.go and go.mod into two independent directories at different absolute filesystem paths and built each copy using the same controlled command:
CGO_ENABLED=0 go build \
-trimpath \
-buildvcs=false \
-ldflags="-buildid=" \
-o zeros3 \
zeros3.goThe two resulting binaries produced the following SHA-256 values:
copy A:
d62ab3c821eef02acc2379a4e21c70e190b1b029e29c59d918b4c592ae82dbe7
copy B:
d62ab3c821eef02acc2379a4e21c70e190b1b029e29c59d918b4c592ae82dbe7
The outputs were byte-identical, and the same hash matched the directly built final candidate. The verified environment used Go 1.27.0 on Linux/amd64, and the reproducibility script remains part of the public project so the comparison can be repeated rather than taken as a claim.
The zero-dependency requirement unquestionably made some parts of ZeroS3 harder. Signature Version 4 had to be understood rather than delegated. Persistence required an explicit crash and publication model instead of relying on a database transaction. Content-defined chunking had to become part of the implementation rather than arriving through a library. HTTP lifecycle behavior, integrity checking, concurrency, and metadata persistence all had to be built from relatively low-level primitives.
What I did not expect was that those constraints would push the project toward a representation that made other parts of the system easier to compose. I had originally encountered the core idea while reading about storage infrastructure built for a very different environment. Hugging Face's use of Xet interested me because content-aware storage changes the economics of very large collections of related artifacts. ZeroS3 took that idea in another direction: S3 remained the external interface, while content-defined chunking and content addressing moved underneath it.
Once the store had content-defined chunks, immutable content identity, immutable manifests, and an explicit visibility boundary, several capabilities that initially sounded unrelated began to collapse into the same model. Replication became transfer of a missing digest set. Repair became retrieving bytes for a known digest and verifying them independently. Forks became additional namespaces publishing shared immutable roots. Snapshots became durable roots over the same graph. Structural diff and inspection became ways of exposing information that the storage engine already possessed.
There was a similar effect in the development process. AI agents made it possible to implement and test much more code during the hackathon, but their usefulness depended on reducing large goals into constrained pieces with persistent invariants, independent review, and evidence that survived between sessions. The project benefited from two kinds of leverage at once: architectural leverage from choosing primitives that composed well, and development leverage from organizing agent work so each stage could safely build on the last.
The final implementation is 12,765 lines of Go in one source file, so ZeroS3 did not remain small in any literal sense. What matters more to me is that most of that growth continued to fit the same architecture instead of accumulating as unrelated subsystems.
If I rebuilt ZeroS3 without the hackathon constraints, that is the part I would keep.
The content-addressed design in ZeroS3 was partly inspired by reading about Xet and its use on the Hugging Face Hub. These are the references that were most relevant to that part of the project:
- Xet is on the Hugging Face Hub — background on Hugging Face's move to Xet-backed storage and why content-aware storage is useful for large models and datasets.
- Xet: Deduplication — an overview of how Xet uses content-defined chunking and content-addressed storage to deduplicate data across files and revisions.
- Xet: Content-Defined Chunking — the more technical description of Xet's Gear-based chunking algorithm and chunk-boundary behavior.
ZeroS3 does not attempt to reproduce Xet's architecture. These references were the starting point for the idea of putting content-defined chunking and content addressing underneath an otherwise conventional S3 interface.