Contrastive pre-training for language tasks
Number of patents in Portfolio can not be more than 2000
United States of America Patent
Stats
-
Feb 27, 2024
Grant Date -
Jan 19, 2023
app pub date -
Sep 19, 2022
filing date -
Sep 19, 2022
priority date (Note) -
In Force
status (Latency Note)
Importance
Abstract
Systems and methods are provided that train a machine-learned language encoding model through the use of a contrastive learning task. In particular, the present disclosure describes a contrastive learning task where the encoder learns to distinguish input tokens from plausible alternatives. In some implementations, on each training example the proposed method masks out some subset (e.g., 15%) of the original input tokens, replaces the masked tokens with samples from a “generator” (e.g., which may be a small masked language model), and then trains the encoder to predict whether each token comes from the original data or is a replacement produced by the generator.
First Claim
Family
International Classification(s)
- [Classification Symbol]
- [Patents Count]
Cited Art Landscape
Patent Citation Ranking
Advertisement
Advertisement
Advertisement
Recipient Email Address
Recipient Email Address
Comment
Recipient Email Address
Add to Portfolio(s)
To add this patent to one, or more, of your portfolios, simply click the add button.
This Patent is in these Portfolios:
Add to additional portfolios:
Last Refreshed On:
Changes done successfully