Contrastive Learning
Visualize how contrastive learning pulls similar examples together and pushes dissimilar ones apart in embedding space.
By Riano · Published · Updated · Report a correction
Contrastive Learning: Predict the Pull and Push
How can a representation bring related examples together without giving every example a manually written category? Contrastive methods compare positive pairs with alternatives. Two augmented views of the same image can supply a positive pair; other images provide negatives. A learned encoder changes its representations through an objective, rather than through literal physical forces.
This interactive model uses 75 seeded points in five predefined color groups. Same-color points attract and different colors repel. The colors are supplied to the demonstration, not discovered labels. Its two-dimensional forces offer an experiment about attraction, separation and sensitivity. They are an analogy, not a trained SimCLR network or an exact InfoNCE gradient.
A 100-unit pair that you can calculate first
Let a be Pull Force, p be Push Force, d the distance between two points, and τ the Temperature. The toy uses attraction magnitude A = ad for a same-color pair and repulsion magnitude R = p(1000/d) exp(−d/(200τ)) for different colors. Each acts along the line between the points. The displayed reference pair always uses d = 100, independently of the changing cloud.
- At the defaults a = 0.05, p = 0.02 and τ = 0.5, attraction is 0.05 × 100 = 5.000.
- The repulsion multiplier is 1000/100 = 10, and the exponent is −100/(200 × 0.5) = −1. Therefore R = 0.02 × 10 × exp(−1) ≈ 0.073576.
- Reduce Temperature to 0.1. Now R = 0.2 × exp(−5) ≈ 0.001348, while attraction remains 5.000. Increasing Temperature to 2.0 instead gives R = 0.2 × exp(−0.25) ≈ 0.155760.
The characteristic decay distance is 200τ. Lower temperature makes this toy's repulsion decay over a shorter distance and decreases its magnitude at any fixed positive separation. It does not make the fixed-distance force stronger. The reference bars update while paused, have separately labeled maxima of 10 and 1, and should not be compared as if equal bar lengths meant equal forces.
These are pair-force magnitudes, not the displacement of a point in the full simulation. Each point also feels all other pairs, a central attraction of 0.01 times its offset toward the center, damping, and the frame boundary.
The actual contrastive objective is different
A normalized contrastive objective uses scores s for an anchor's positive and negative representations. For a positive score s+, negative scores sk, and temperature τ, a one-positive loss has the form:
As a separate paper calculation, take one positive score 1, one negative score 0 and τ = 0.5. The positive probability is exp(2)/(exp(2) + 1) ≈ 0.880797 and L = log(1 + exp(−2)) ≈ 0.126928. This score example is not an input to this module: the controls change a force analogy, not cosine similarities or a neural loss. Temperature changes relative softmax weights in the objective; its effect is not the same as substituting τ in the toy's distance kernel.
Change a rate, then take one step
Open the Contrastive Learning interactive module. Before each comparison predict whether the reference attraction or repulsion should change.
- Press Reset Embeddings. Set Temperature (τ) to 0.5, Pull Force (Positive) to 0.05 and Push Force (Negative) to 0.02. Confirm Steps is 0 and the reference readouts are 5.000 and 0.073576. Reset restores the seeded positions and stops motion; it preserves your chosen rates.
- While paused, change Temperature to 0.1. Observe the shorter push reference bar and 0.001348 readout. No point motion is implied by changing a rate while paused.
- Set Pull Force to 0.10. Attraction becomes 10.000; the repulsion readout is unchanged. Change Push Force to 0.10 with Temperature back at 0.5: repulsion becomes 0.367879.
- Press Step once. The cloud advances exactly one 1/60-second update and Steps becomes 1. Press again to inspect a second bounded update. Every force for an update uses the previous positions, so the array order does not turn the update into sequential point motion.
- Press Start Learning to run, and Pause Learning to stop. These are the visible states of the stable Toggle learning button. Step is disabled during continuous motion. Drag the embedding view or use its arrow keys to pan; Reset view or Home resets navigation without resetting Steps.
To compare rates fairly, reset positions before each run and use the same number of Steps. The fixed seed and fixed step remove two sources of variation, but a short trajectory still does not establish eventual cluster quality.
Where the analogy stops
No encoder, image augmentation pipeline, learned projection head or optimizer is implemented. Five colors are predefined. The display is a two-dimensional toy space, not a projection of measured high-dimensional embeddings. Do not infer representation quality, benchmark accuracy or real model convergence from compact colored clusters.
A step uses Δt = 1/60, v' = 0.8(v + FΔt) and x' = x + 50v'Δt, with the same update for y. Positions are confined to x in [32, 768] and y in [32, 568]; a velocity component is stopped at its wall. The initial cloud is fitted to these same padded bounds. Wall contacts can distort clustering. Pairs closer than 0.1 units are skipped to avoid division by zero. Those numerical choices and damping are features of this illustration, not parts of the cited learning algorithms.
Continuous playback uses fixed updates and caps accumulated wall-clock time after a suspended tab. Pausing, resizing or moving the view does not regenerate points. Rounded readouts show six decimals for repulsion and three for attraction; very small positive forces can look like zero at display precision. A panned view may deliberately move points off-screen; Reset view restores the fitted frame.
From the illustration to representation learning
Contrastive Predictive Coding introduced a predictive representation-learning framework with a contrastive objective and negative sampling. SimCLR later studied image augmentations, a nonlinear projection head and a normalized temperature-scaled objective. Their methods explain why the positive/negative comparison matters; neither source specifies Riano's damping or force kernel. Learned representations can support retrieval or downstream prediction, but the toy demonstrates neither evaluation task.
Primary sources and what they support
Read these papers for the representation-learning objectives and experimental settings. Their positive and negative samples motivate this illustration; the seeded positions, force law, damping and boundary rules are separate choices made for this module.
- Chen et al., A Simple Framework for Contrastive Learning of Visual Representations (2020). Section 2.1 and Equation 1 define SimCLR's normalized contrastive objective; the experiments study augmentation, projection heads and temperature. They do not describe the physical simulation here.
- van den Oord, Li and Vinyals, Representation Learning with Contrastive Predictive Coding (2018). Section 2.3 introduces the InfoNCE objective for distinguishing a positive sample from negatives. The independent reference-force calculation above belongs to this module, not to the paper.
Experience it interactively
Adjust parameters, observe in real time, and build deep intuition with Riano’s interactive Contrastive Learning module.
Try Contrastive Learning on Riano →