Colorization using Neural Network Ensemble

Abstract. This paper investigates into the colorization problem which converts a grayscale image to a colorful version. This is a difficult problem and normally requires manual adjustment to achieve artifact-free quality. For instance, it normally requires human-labelled color scribbles on the grayscale target image or a careful selection of colorful reference images. The recent learning-based colorization techniques automatically colorize a grayscale image using a single neural network. Since different scenes usually have distinct color styles, it is difficult to accurately capture the color characteristics using a single neural network. We propose a mixture learning model representing the presence of sub-color-style within an overall image dataset. We therefore ensemble multiple neural networks to obtain better color estimation performance than could be obtained from any of the constituent neural network alone. A two-step colorization strategy is utilized as an adaptive color style clustering followed by a neural network ensemble. To ensure artifact-free quality, a joint bilateral filtering based post-processing step is proposed. Numerous experiments demonstrate that our method generates high-quality results comparable with state-of-the-art algorithms.