CC-MAIN-2026-30

announcement_todo.md

TODO

• final_steps/update_url_index.md
• inal_steps/overview_and_statistics_tables.md
• webgraph step 4
• https://github.com/commoncrawl/crawl-guide/pull/82

Blog post

This crawl comes with a series of improvements and changes.

WARC-Record-ID Now Uses Type 7 UUID

We switch from using Type 4 to Type 7 UUIDs for WARC record IDs.

Type 7 UUIDs are defined by RFC 9562 (published in 2024 and updating RFC 4122) and combine a Unix timestamp (epoch seconds) with a random component. One example of a Type 7 WARC record ID:

WARC/1.0
WARC-Type: response
WARC-Date: 2026-07-10T22:25:47Z
WARC-Record-ID: <urn:uuid:019f4e23-0ae1-7128-99c9-625a3048e3b1>
Content-Length: 27888
Content-Type: application/http; msgtype=response
WARC-Warcinfo-ID: <urn:uuid:019f4e20-cbc0-7088-8251-ecc9a008540a>
WARC-Concurrent-To: <urn:uuid:019f4e23-0ae1-71ae-a89b-b0992fcec9b8>
WARC-IP-Address: 2620:cb:2000::1
WARC-Target-URI: https://commoncrawl.org/
WARC-Protocol: h2
WARC-Protocol: tls/1.3
WARC-Cipher-Suite: TLS_AES_128_GCM_SHA256
WARC-Payload-Digest: sha1:BYSGGYPUHJIF4ZIOWKMPPLYUJ5U5YJQO
WARC-Block-Digest: sha1:5OM6GSLZYKMJJ5MPXWM5BUVWW43GGZZU
WARC-Identified-Payload-Type: text/html

The following Python snippets demonstrates how the UUID can be parsed to extract the timestamp:

import datetime as dt
import uuid

record_id = uuid.UUID('urn:uuid:019f4e23-0ae1-7128-99c9-625a3048e3b1')
dt.datetime.fromtimestamp(record_id.time / 1000).isoformat()
# returns '2026-07-11T00:25:47.233000'

As you can see, the timestamp in the in the UUID mirrors the WARC-Date header. Of course, the UUIDs in the WARC response records follow the same pattern. For further details about this change, please see the corresponding issue on GitHub.

WARC-Record-ID Included in URL Indexes

The record ID of the WARC response records is now contained in our URL indexes without the decorating <urn:uuid:>:

• The CDXJ index stores the UUID in the field recordid .
• The URL index has a new column warc_record_id.

In order to use the new column in the URL index with Amazon Athena, you need to use the updated table schema defined by the cc-index-create-table-flat.sql statement.

The column warc_record_id is a binary column. You can cast from and to the data type UUID:

select cast(warc_record_id as uuid)
from ccindex
where url = 'https://commoncrawl.org/'
  and url_host_name_reversed = 'org.commoncrawl'
  and crawl = 'CC-MAIN-2026-30';

To look-up a record ID:

select url, fetch_time
from ccindex
where cast(warc_record_id as uuid) = uuid '019f4e23-0ae1-7128-99c9-625a3048e3b1'
  and url_host_name_reversed = 'org.commoncrawl'
  and crawl = 'CC-MAIN-2026-30';

Also DuckDb provides a UUID data type. You can use the new column the exact same way as in Amazon Athena.

More details about the new column can be found in the pull request cc-index-table#42 on GitHub.

The URL Index Switches to ZStandard Parquet Compression

The Parquet files of the URL index, starting with the current crawl's partition (crawl=CC-MAIN-2026-30) are compressed using ZStandard compression.

Given that most Parquet readers and query engines support ZStandard nowadays, no changes are required to query the recent partitions of the URL index. Additional information can be found in this issue report on GitHub.

Statistics of IP Address and HTTP Protocol Versions

We started publishing statistics of IP address and HTTP protocol versions observed by our crawler CCBot. The new metrics are found on the Crawl Stats site along with other crawler metrics.