fastcdc-go

fastcdc-go splits byte slices and streams into content-defined chunks with the FastCDC 2020 algorithm. Its validated Chunker is immutable and safe for concurrent use, with small APIs for in-memory and streaming input.

The module requires Go 1.23 or later.

Install

go get github.com/SaveTheRbtz/fastcdc-go

Byte slices

Use Chunks when the complete input is already in memory:

chunker, err := fastcdc.New(fastcdc.Config{AverageSize: 1 << 20})
if err != nil {
	log.Fatal(err)
}

for offset, chunk := range chunker.Chunks(data) {
	fmt.Printf("offset=%d size=%d\n", offset, len(chunk))
}

Chunks can be ranged over repeatedly and reads the input during each pass. Do not mutate the input while ranging. Each chunk aliases the input slice and has its capacity clipped to its length; mutating either view changes the same storage.

Cut(data) is the lower-level operation. It returns the length of the first chunk, or zero for empty input. It treats data as a complete input, so it returns a short final chunk immediately. Do not use Cut on an incomplete stream fragment; use Reader instead.

To loop over Cut manually, process data[:n] and continue with data[n:]. Chunks and Reader handle this for you.

Streams

Create an independent Reader for each stream:

reader := chunker.NewReader(src)
for {
	offset := reader.InputOffset()
	chunk, err := reader.Next()
	if err == io.EOF {
		break
	}
	if err != nil {
		return err
	}

	if err := consume(offset, chunk); err != nil {
		return err
	}
}

Next returns non-empty chunks and reports io.EOF only after returning the final buffered bytes. The returned slice borrows reader-owned storage and remains valid only until the next call to Next or Reset, even if that call returns an error. Clone a chunk before retaining it:

saved := bytes.Clone(chunk)

The reader may consume input beyond the last returned chunk. InputOffset reports the logical end of that chunk, not the underlying source position. Abandoning the reader or calling Reset can therefore discard bytes already read from the source. The reader never closes its source.

If the source returns a non-EOF error, the reader treats the successfully read prefix as a complete, truncated stream. Next returns its complete chunks and remaining short tail before returning the source error. The error is terminal: later calls return the same error until Reset selects a new source. Repeated (0, nil) reads eventually produce terminal io.ErrNoProgress.

Reader is not safe for concurrent use and must not be copied. Reset reuses its chunk buffer, discards buffered input and the terminal error, and sets InputOffset back to zero.

Configuration

AverageSize is the only required field:

Field	Rules and default
AverageSize	Power of two from 256 B through 4 MiB; required
MinSize	Positive and less than AverageSize; default AverageSize / 4
MaxSize	Greater than AverageSize and at most 16 MiB; default AverageSize * 4
Normalization	Default NormalizationLevel1

AverageSize selects the target scale; it does not guarantee the arithmetic mean of produced chunks. The observed mean also depends on normalization and input data.

Normalization changes the boundary mask below and above the requested average so chunk sizes cluster more tightly around it:

Value	Behavior
NormalizationNone	Disable normalization
NormalizationLevel1	Moderate normalization; also selected by the zero value
NormalizationLevel2	Narrower size distribution
NormalizationLevel3	Narrowest size distribution

Higher levels trade a tighter distribution for different chunk boundaries. All configuration fields are part of the chunking format: use the same values where identical boundaries are required.

A Chunker contains only validated, immutable state. Share it freely between goroutines, and create a separate Reader for each stream.

Performance

On an Intel Core i5-10210U with Go 1.26.5 on Linux/amd64, Reader processed a 1,660,723,200-byte tar archive of Linux v7.1 in 1.626 seconds. The configuration used a 1 MiB average chunk size and the derived defaults: 256 KiB minimum, 4 MiB maximum, and normalization level 1.

The input was in the operating-system page cache. The process was pinned to one hardware thread with GOMAXPROCS=1; the CPU used its performance governor and the NMI watchdog was disabled. Each of ten samples read the complete archive once. The benchmark deliberately omitted the race detector, whose instrumentation changes performance.

benchstat metric	Result
Time	1.626 s/op ± 1%
Throughput	974.2 MiB/s ± 1%
Allocated bytes	272 B/op ± 0%
Allocations	5 allocs/op ± 0%

Reader's reusable 4 MiB chunk buffer was allocated before measurement. The reported allocations include opening and closing the input file and resetting the reader for one complete pass.

The input was the archive of Linux commit 8cd9520d35a6c38db6567e97dd93b1f11f185dc6 (tag v7.1), with SHA-256 b5470460e136e037c3f90579b6ec9b61c53e6f833b1acc37c0329466ffea0b6e. Reproduce the measurement with:

go install golang.org/x/perf/cmd/benchstat@v0.0.0-20260709024250-82a0b07e230d

git -C "$LINUX_REPO" archive --format=tar \
  --output=/tmp/linux-v7.1.tar \
  8cd9520d35a6c38db6567e97dd93b1f11f185dc6

taskset -c 3 env GOMAXPROCS=1 FASTCDC_BENCH_FILE=/tmp/linux-v7.1.tar \
  go test -run '^$' -bench '^BenchmarkReaderFile$' \
  -benchmem -benchtime=1x -count=10 > reader.bench
benchstat reader.bench

Command-line tool

Install the included boundary-inspection tool:

go install github.com/SaveTheRbtz/fastcdc-go/cmd/fastcdc@latest
fastcdc -file archive.img -avg 1048576

It prints each chunk's offset and size. Pass -csv for CSV output, -normalization=1, 2, or 3 to select a level, or -normalization=-1 to disable normalization.

Analysis

The reproducible analysis records deterministic chunk-size distributions, Linux source-history dedup ratios at two chunk sizes, and an exact source-boundary oracle comparison. The supporting command emits canonical CSV; plots are generated separately with the checked-in gnuplot script.

No normalization	Level 1

Level 2	Level 3

Compatibility

This version implements the 2020 algorithm and its canonical Gear table. It does not preserve the API or chunk boundaries of earlier fastcdc-go releases.

References

• Wen Xia et al., FastCDC: a Fast and Efficient Content-Defined Chunking Approach for Data Deduplication, IEEE Transactions on Parallel and Distributed Systems, 2020.
• Wen Xia et al., FastCDC: a Fast and Efficient Content-Defined Chunking Approach for Data Deduplication, USENIX ATC, 2016.

License

Apache-2.0. See LICENSE.