Common Crawl Web Graph Embeddings: A Public, Reproducible Release
Web-link structure is difficult to turn into a reusable machine-learning input; this release packages host vectors with reproducible pipeline code and browser-based tools for exploring the map and nearest neighbors.
Read the paper ↗What it is
- Reproducible release The host vectors are published alongside the code for preparing the graph, training the embeddings, and exporting the results.
- Interactive exploration A browser map gives a broad visual view, while hostname search finds nearby sites using the full released representation.
Why it matters
You can download and use the vectors, inspect how they were produced, and test whether their neighborhoods look sensible without first building the whole pipeline or an exploration interface yourself.
Before getting into the release details, here is the short version: Common Crawl has turned links between websites into reusable vectors for hosts, then published both the vectors and the code that produces them. It also gives you browser tools to look around the representation instead of taking a large data file on faith.
-
Usability: A web graph tells you which sites link to which, but most machine-learning workflows want a numerical representation they can load and compare. The canonical example here is a host: instead of handling its entire neighborhood of links every time, you can work with its released vector. That is handy for finding similar sites or feeding the representation into another model, though a vector is only a view of the site's link structure.
-
Reproducibility: A dataset without its construction path leaves you guessing how raw graph data became vectors. The release makes the pipeline public, from downloading and preparing the graph through training and export. In practical terms, that is like sharing the recipe as well as the meal; the analogy stops short because running the recipe still requires the relevant data and compute.
-
Inspection: A huge vector table is hard to understand by itself. The canonical example is looking up a site you already know: a two-dimensional map gives you a broad sense of where it sits, and nearest-neighbor search lists sites close to it in the full representation. The map is a projection, so apparent distance on screen is not the same thing as a full-space neighbor query; and neighbors can be noisy for hubs.
So the real question was never just whether the vectors can be downloaded—it was whether you can inspect them and reproduce their production. The map projects all released hosts into a view you can pan and zoom, with tiles streamed as you explore. You can recolour that view using link count, topic, language, or content-quality labels. The nearest-neighbor tool answers a more direct question: give it a hostname and it returns nearby sites in the full 128-dimensional space. Which view should you trust? Neither as ground truth: one compresses the representation for display, and the other can be noisy for hubs.
One useful detail is that the release is explicitly an experiment, not a commitment to publish embeddings on a regular schedule. The vectors and pipeline make it possible to try applications and assess the data, while the browser tools lower the cost of an initial sanity check. I like that the release makes scrutiny part of the product rather than an afterthought. What remains open is whether people find the representation useful enough to motivate future versions and improvements.
Read the deep dive
Why People Wanted It
Common Crawl had published Web Graph datasets for years: maps of which sites link to which others. But a graph is not yet a ready-to-use numerical input for machine learning. If you want to compare sites, train a classifier, or explore link structure, you need a representation you can actually load and inspect. The real question was not only how to make such a representation, but how to release it so other people can use it, reproduce its construction, and judge what it captures.
The Core Move
Release the host vectors, the end-to-end pipeline that produces them, and browser tools that let people inspect the result rather than treating the embeddings as an opaque download. The source describes each host as a 128-dimensional vector learned from hyperlinks alone; it gives no mathematical formula for the embedding objective.
Why the Release Includes the Pipeline
Now the version for readers who want more of the machinery: reproducibility here means making the path from raw graph to exported vectors public, not just sharing a finished table. The published pipeline covers downloading and preparing raw graph data, training, and exporting. This lets a user inspect the steps that connect a Web Graph snapshot to the released representation, rather than guessing how the data was made.
The release is an experiment, not a promise of a recurring product. The vectors are available from S3 and Hugging Face, and the source code is on GitHub. The particular embedding described comes from a host-level graph assembled from three crawl months. Those details make the release usable and inspectable; they do not by themselves establish that future snapshots will be published on the same schedule.
Why Two Browser Tools Answer Different Questions
Here is the part that gets technical in terms of how you inspect the data: the map and nearest-neighbor search expose different views of the same representation. The map projects the hosts onto a two-dimensional plane for panning and zooming, with tiles streamed as you explore; its display can be recoloured by link count, topic, language, or content-quality labels. The projection is an overview, not the full representation, so a visible neighborhood on the map is not a substitute for comparing vectors in their original space.
Nearest-neighbor search instead takes a hostname and returns nearby sites in the full 128-dimensional space. That gives you a direct way to check whether a familiar site's neighbors seem plausible. Hubs can have noisy nearest neighbors, so neither a colorful map nor a search result should be mistaken for a definitive classification. What do you need to see before trusting a pattern in either view?
Minimal Causal Code
This deliberately plain pseudocode shows the release pipeline's causal shape, not its implementation details. The source does not give a formula for training, so this loop does not invent one:
raw_graph = download_graph_snapshot()
host_graph = prepare_host_level_graph(raw_graph)
vectors = train_link_embedding(host_graph)
export_vectors_and_metadata(vectors)
The key is that each stage is visible: graph data goes in, a host-level graph is prepared, vectors are trained, and the result is exported. The published code is the place to inspect the actual operations.
What You Gain, and What You Lose
You gain a downloadable representation, public end-to-end code, and two ways to explore it without first building your own inspection interface. Together these make the release easier to apply and easier to question. The map gives a broad visual orientation; nearest-neighbor search gives a direct local comparison in the full vector space.
You do not get a guarantee that the release will become a regular series: it is explicitly experimental. The map is a two-dimensional projection, and nearest neighbors can be noisy for hubs. The vectors are also a link-derived view, not a complete description of a website. Treat the release as a useful starting point for inspection and application, not as a final account of the web.
A useful embedding release shares not just vectors, but the path to reproduce them and tools to inspect what they mean.