Calibrated Typed Decisions for Entity, Relation, and Event Extraction

The short version

The problem

Finding names, links, and events in text is difficult when the subject matter or the rules for what counts as an answer change. Existing tools may miss information or return words that were not actually in the text.

The idea

CALON is one model that can be given a list of things to look for and their descriptions, then find the matching text spans and connections between them. For example, it can mark “Alice” as a person and “Acme” as an organization, then identify that Alice works for Acme; during training it also learns from rewards that favor correct choices over wrong types and unsupported guesses.

What it got

On 40 tests, CALON did better than the compared systems on most of them and led every reported test of relations between text spans. A correction also lowered average confidence-calibration error from 0.25 to 0.09 for relations, with little change in accuracy.