MoCo and Contrastive Learning

Basic ideas in Contrastive Learning and Kaiming’s improvment in MoCo.

0 Contrastive Learning

The overview position of CL A slide from Yang LeCun

To do CL is supposed to be easiler than SL. Human is easy to distinguish objects rather than identify objects. The idea is similar to training embedding with triplets. So with postive and negative samples, you are trained to distinguish them too.

1 Memory Bank

A earlier paper proposed memory bank idea.

2 Momentum Contrastive

Kaiming combined the end-to-end and memory bank and proposed momentum contrastive.

The loss is defined as InfoNCE Try to meet both alignment and uniformaity goal.

Comparing to Memory Bank, it’s similar to momentum in BP learning.