Affinity learning refers to the problem of learning a function that takes a pair of data points as the input and outputs an affinity or similarity score. It closely relates to metric learning and finds itself in many applications, such as ranking or retrieval, clustering, face verification, and recommender systems. We present a generic affinity learning technique, namely the diffusion process, that is capable of learning context-aware affinity. A diffusion process takes an input affinity matrix and outputs an updated one. It can be interpreted as a markov random walk on the input affinity graph, where the affinity value defines the transition probability between nodes. By stochastically walking from one node to another, the diffusion process propagates pairwise affinity to all neighbour nodes visited, and thus updates affinity values using the contextual information. Being a generic affinity learning technique, the diffusion process is applied to various machine learning problems, including unsupervised clustering, semi-supervised learning, and representation learning with graph convolutional networks. Extensive experimental results demonstrate the capability of the diffusion process on learning accurate pairwise affinity and being a flexible tool to address a wide variety of learning problems.