Common Crawl - Blog - Web Graph Embeddings: An Experimental Dataset Release

The short version

The problem

The web has a huge map of which sites link to which, but it is hard to use that map directly in machine-learning systems. Researchers and developers lack a ready-made numerical description of sites based only on their link patterns.

The idea

The paper releases a compact list of numbers for each of 52.9 million sites, learned from links between sites rather than from their text or names. Think of it as placing sites near others with similar link patterns; the release also includes code and browser tools to inspect and use these representations.

What it got

The representations help predict site topics, reaching a score of 0.38 compared with 0.06 from guessing by a site's ending, such as “.com” or “.org.” They also identify language with 0.67 overall accuracy and provide a spam signal that can complement an existing link-based detector.