When wget encounter several urlA,urlB,urlC sharing the same digest hash1 in the input .cdx file of --warc-dedup, it only registers the last (urlC,hash1) pair as potential candidate for deduplication.

When crawling, it rejects (urlA,hash1) and (urlB,hash1) as candidates for a revisit record, even if found as-is during the crawl.

Some steps to reproduce against a toy example, we first build a cdx file:

% wget 'http://perdu.com' 'https://perdu.com' --delete-after --warc-file=testA --warc-cdx=on

We can visualize the two (url,digest) pairs sharing the same digest:

% cat testA.cdx

Then we can verify that deduplication works for the second pair:

% wget 'https://perdu.com' --delete-after --warc-file=testB --warc-dedup=testA.cdx

And we see that deduplication fails for the first pair:

% wget 'http://perdu.com' --delete-after --warc-file=testC --warc-dedup=testA.cdx

We can confirm that only testB got its revisit record:

% zgrep revisit *.warc.gz

In practice, this cause that most frequent files are the ones most commonly NOT deduplicated. This is noticeable during repeated crawls of websites that mostly don't change: the first crawl will download the whole website, and all subsequent crawls will only download files for which deduplication failed (typically fonts, http/https siblings, resources present at several URLs of the website...)

AFAIK current implementation use digest as key of an in-memory hash table, instead of using the (url,digest) pair.