The History and Technology of Image Colorization

Colorization is the process of adding color to monochrome images. What began as a painstaking manual craft in the early 20th century is now a one-click AI operation. This article traces the technical evolution and explains the algorithms behind modern colorization services.

1. The Hand-Painted Era (1900-1960)

Early colorization was done frame by frame by skilled artists using fine brushes and transparent dyes. Studios like Warner Bros. employed teams of colorists who would paint each cel by hand. A feature length animated film required 50,000 to 100,000 painted cels. The work was highly skilled and took weeks per reel.

For photographs, the process was different. Color tinting used chemical baths to give a sepia, blue, or green cast to the entire image. Hand coloring, popular in the 1910s-1930s, used small brushes and oil paints to add color to specific regions. Studio portraits and postcards were the primary products.

2. The Electronic Era (1960-1990)

Analog video opened the door to electronic colorization. The first commercial electronic colorization system was developed by CBS in the 1960s. A technician would identify regions of similar luminance and assign a color from a palette. The system stored a "color map" alongside the luminance signal and combined them in playback.

These systems produced decent results for cartoons but were not feasible for live-action content. A single 30-minute show took two weeks to colorize, and the result looked flat and artificial compared to native color.

3. The First Digital Algorithms (1990-2010)

Computer vision researchers began publishing on automated colorization in the 1990s. Early approaches used a small dictionary of reference colors. The algorithm would look up a matching patch in the reference and transfer the color. This worked for textures but failed on complex scenes.

The landmark 2002 paper by Welsh et al. introduced a "color transfer" technique that matched luminance patterns between a reference color image and a target grayscale image. The method produced credible results for landscape photos with similar content to the reference, but required careful selection of the reference image.

4. The Deep Learning Revolution (2012-2020)

The ImageNet competition in 2012 demonstrated that deep convolutional networks could classify images with unprecedented accuracy. Researchers quickly applied similar architectures to colorization. The 2016 paper "Colorful Image Colorization" by Zhang et al. at Berkeley introduced a CNN that learned to predict color distributions rather than specific colors, allowing the model to output plausible but diverse results.

By 2018, the results were good enough to be deployed in commercial software. Adobe Photoshop added an "AI Colorize" feature in 2020 that used a similar architecture. The results were good but not great: skin tones were often off, skies were oversaturated, and complex patterns sometimes turned into mush.

5. The Transformer and Diffusion Era (2020-2026)

Two architectural innovations drove the next leap. First, Vision Transformers replaced the convolutional backbone, allowing the model to consider global context. A patch of grass in the foreground is influenced by the sky above, the shadow direction, and the season implied by foliage. This global reasoning improved color consistency dramatically.

Second, diffusion models (the same family that powers Stable Diffusion and DALL-E) added a new capability: they could colorize a single image while conditioning on a text prompt. A user can now upload a black and white family photo and write "1940s summer afternoon" to get a result that matches the implied era. The model has been trained on millions of captioned images and has learned the visual conventions of different time periods.

6. The Pipeline We Use

Our colorization pipeline is a hybrid:

1. Pre-processing: We normalize the input to a fixed resolution and apply gentle denoising. Old photos often have silver grain that confuses the model.
2. Global pass: A Vision Transformer predicts the overall color palette and lighting direction. This is what gives the result a coherent "mood".
3. Local pass: A diffusion model refines individual regions, conditioned on the global pass. Skin tones, fabric, foliage, and sky get specialized treatment.
4. Post-processing: We apply a temporal consistency check for video colorization and a perceptual quality boost for stills.

7. Why Colorization Is Harder Than It Looks

A common misconception is that colorization is a simple "lookup" problem: find a similar object in a reference image and copy its color. The reality is that color is not determined by object identity alone. A red brick wall in the late afternoon sun is a different red than the same wall at noon. A white shirt in a wedding photo is a different white than the same shirt in a funeral photo. The model has to infer the lighting, time of day, cultural context, and even the photographer's intent.

This is why colorization models sometimes produce surprising results. A photo of a 1950s American street scene might come back with European-looking architecture because the model has been exposed to a wider range of training data. We are constantly expanding and rebalancing our training set to address this.

8. The Ethics of Colorization

Colorization is not a neutral operation. It can change the emotional impact of an image. A black and white photo of a civil rights protest feels different once color is added. A portrait of a Holocaust survivor feels different in color. The technology is the same, but the meaning shifts.

We have a content policy that prohibits colorization of sensitive historical material without explicit educational or documentary context. We also watermark our AI-colorized outputs in the metadata, so a viewer can see in the file properties that the colors were algorithmically generated rather than captured in camera.

9. Future Directions

The frontier in 2026 is video colorization at full frame rate. Early results from the labs suggest this is achievable within two years. We are also working on guided colorization, where a user can paint a single reference region (say, the color of a wedding dress) and the model propagates that constraint across the rest of the image. This kind of artist-in-the-loop workflow is what professional colorists have been requesting for years.

Conclusion

Colorization is a useful case study in how AI transforms creative work. The core algorithmic problems have been understood for decades; what changed is the scale of training data and the expressiveness of the model architecture. The remaining challenges are not technical but ethical: deciding when colorization is appropriate and how to be transparent about the artificial nature of the result.