Born in Toronto, Colville moved as a boy to Amherst, Nova Scotia with his family. After his studies at Mount Allison University, he served in the army from 1942 to 1946, working as a military artist from 1944 to 1946. He then taught at university, but left in 1963 to devote himself completely to painting. In the 1950s, his approach became associated with that of certain American artists, such as Andrew Wyeth, who are considered regionalists. His paintings are characterized by a latent anxiety; an example is the well known work Child and Dog, in which the juxtaposition of a blond child and a large black dog with pronounced claws creates a feeling of unease. Nearly a third of Colville's works involve animals, particularly domestic animals. The famous image of a couple crossing the Straits of Northumberland, To Prince Edward Island, reveals a number of themes that recur in Colville's work: means of transportation, the sea, the relationship between a couple; but he also uses the woman with the binoculars to illustrate the power inherent in a gaze, the dynamic that exists between the person looking and the person being looked at, who in turn is looking back at the gazer.
Over his long career, Colville's fame grew and he received many honours. Major retrospectives of his work were held at the Art Gallery of Ontario in 1983 and the Montreal Museum of Fine Arts in 1994. He was made a Companion of the Order of Canada in 1982, and won a Governor General's Visual and Media Arts Award in 2003.