Fast color transfer based on image statistics

Leave a comment

Color transfer is a long-standing problem that seeks to transfer the color style of a reference image onto a source image. By using different references, one can alter the color style without changing the original image content in order to emulate different illumination, weather conditions, scene materials, or even artistic color effects. It can be used for color correction for rendered imagery, artistic style transfer, turning a day scene into a night scene, image analogies.

Color transfer task defines the following inputs:

  • A content image —  the image we want to transfer the color palette to;
  • A style image —  the image we want to transfer the color palette from;
  • A generated image  —  the image that contains the final result.

A lot of research over the last years has focused on this task. The best results were obtained using deep learning techniques inspired by the idea of semantic-guided transfer. In this case, the network architecture VGG-19 is used for extracting content feature maps. But the number of weight parameters are quite large and the models are very heavy (over 533MB). Due to its size, the deploying VGG is a difficult task especially if you are looking to deploy a model to run locally on mobile. Therefore, we need a fast statistical method that you can easily deploy on mobile platforms.

We present a new method of color transfer based on simple statistics. The main idea of the proposed method is to sort separately content and style image pixels by its brightness, get some statistics from 5-10%, 24-26%, 49-51%, 74-76% and 99-99.5% quantiles of the brightness distribution. We choose the transformation to match the covariance and mean of quantile distribution.

In particular, let x_i = (R, G, B)^T be a source image pixel. Each pixel is transformed as:

x_{S'} = Ax_s + b

where A is a 3×3 matrix and b is a three-dimensional vector. New data points x_{S'} will have the desired mean and covariance:

µ_{S'} = \frac{1}{n} \sum_{i}^{}x^i

Σ _{S'} = \sum_{i}^{} (y_i - µ_i)(y_i - µ_i)^T

We went to find some A and b to satisfy the matching of the style and result image statistics. Hence, the transform is:

x_{S'} = Σ_S^{\frac{1}{2}}Σ_C^{-\frac{1}{2}}(x - µ_C) + µ_S

To compute the square root of the matrix we use a 3D color matching formulations. First, let the eigenvalue decomposition of a covariance matrix be A = UΣU^T . Then, we define a matrix square-root as: A^{\frac{1}{2}} = UΣ{\frac{1}{2}}U^T . Then, the transformation is given by A = Σ_S^{\frac{1}{2}}Σ_C^{\frac{1}{2}} .

At the next step we need histogram equalization and use it to improve the contrast of the source image. Then, we apply linear interpolation to find the covariance matrix and mean for each pixel and calculate new pixel values. And at the last step, we need to clip any values that fall outside the range [0, 255] and merge the channels back together.

However, our method still has limitations. It may mismatch some regions which exist in the source but not in the reference, and thus cause the incorrect color transfer. Our color transfer is suitable for semantically similar images. So, it may lead to some unnatural color effects for image pairs without a semantic relationship, such as the yellow water in the 3rd failure example

This simple algorithm can be easily implemented using different rendering frameworks. You can speed up transformation calculation using GPU unit and achieve 30 fps for 600px*400px frames on the Apple A10 Fusion GPU.

About the author. Anna Khanko is a Machine Learning researcher at She received her MS in Computer Science at the National Technical University of Ukraine where she was doing the research on Natural Language Processing. These days, Anna continues to work on NLP and Computer Vision areas, she developed an information retrieval system, chatbots, and image processing applications. Anna is always keen to learn new things and broaden her professional horizons.

Adoption of AI in Healthcare, insights from HIMSS19

comment 1
Artificial Intelligence / Healthcare

 Last month, we’ve attended HIMSS19. If you are not familiar with the event, Healthcare information and Management Systems Society (HIMSS) is an annual healthcare information conference, which brings together more than 45,000 healthcare and information technology professionals from all over the world. 

Artificial Intelligence was a big topic at HIMSS this year. On Monday, we attended “Machine Learning and AI for Healthcare” pre-conference event where health systems, technology companies and clinicians discussed their efforts to utilize AI and Machine Learning to improve healthcare system, boost medical outcomes and reduce costs.

Read More

Seven surprising applications of AI

Leave a comment
Artificial Intelligence / Machine Learning

Everyone is talking about AI applications for healthcare, financial services and self-driving cars. At the same time, creative occupations were previously understood to be immune from the disruptions of AI due to the high levels of intuition and gut instinct, difficult to replicate by complex algorithms, but that is changing now. In this post, I want to talk about 7 lesser-known ways Artificial Intelligence is changing the world around us. Some of these applications might surprise you. Read More

How Artificial Intelligence is Changing Digital Marketing

Leave a comment
Artificial Intelligence / Big Data / chatbot

Artificial Intelligence is changing many industries and Digital Marketing is no exception. Customers today expect personalized and consistent experience across every channel. For the second edition of the “State of Connected Customer” report, Salesforce Research surveyed over 6,700 consumers globally to better understand the mindset of the modern customer. The report revealed that 80% of customers think that the experience a company provides is as important as its products or services. 57% have stopped buying from the company because a competitor provided a better experience.

Savvy marketers know this and rely on Artificial Intelligence to analyze their data to gain a deeper understanding of consumers’ needs, then use these insights to help their organizations deliver improved customer experience. Read More AI Q&A Series: Jackie Snow, MIT Tech Review

Leave a comment
Artificial Intelligence / Big Data
fashion digamma ai MIT AI Q&A Series: Jackie Snow, MIT Tech Review

1. We are in the very early stages of AI in history. Style is still so complex and there is this sense that AI is trending towards an intelligent assistant that will help us look like we shop at Saks Fifth Avenue regularly and yet do it under budget. In what areas do you believe AI will help consumers and what areas of the style and retail experience do you believe humans still need to do themselves?

I would agree that style is still too complex for AI to get a grip on. So, there’s a long way to go before AI is putting together an outfit for me. Right now, with everything that we’re seeing in the fashion world, I think it’s really geared towards predicting what sort of items a consumer might like to buy. That doesn’t really have anything to do with style. In the meantime, I do think we’re going to have AI that can help surface a lot of different items that we may not necessarily be exposed to through the online shopping environments that are available to us right now. Read More Partners With the Institute of Mathematics at the National Academy of Sciences of Ukraine

Leave a comment
Artificial Intelligence / Machine Learning
Institute of Mathematics

Our team here at is very excited to announce our partnership with the Institute of Mathematics of the National Academy of Sciences of Ukraine in Kyiv.

Through the Kyiv Academic University’s Dual Education Program, we will working with Masters and PhD students from the Department of Mathematics to tackle key machine learning projects together.

With mathematics and statistics as the foundational underpinnings of algorithms that are powering the emerging AI economy, the mathematics field — and researchers in the sector — are experiencing a flood of interest from the private sector.

Through this initiative, we will have an opportunity to work with some of the brightest mathematical minds at the University — which has a long-standing history of exceptional achievements in the field —  and provide an environment for mathematics graduate students to apply their research and knowledge to solving real-life problems using machine learning methodologies. Read More CEO Q&A: Amar Chokhawala, CEO of Reflektion

Leave a comment
Artificial Intelligence / deep learning / Machine Learning
Reflektion CEO Q&A Series: Amar Chokhawala, CEO of Reflektion

Reflektion seeks to give retailers a deeper understanding of customer intent. How does your company do this from a technology perspective?

In today’s world, everyone is sharing everything. So, the customer’s intent almost starts with temporal signals. When I’m looking for a place to eat in the morning, that means I’m looking mostly at restaurants which are actually serving a breakfast or lunch. Here’s an example of how the temporal aspect comes into play. Imagine a three year old—if you asked her what an “apple” is, she would say it was a fruit. If you posed that same question to a thirteen year old, she would say that it’s the technology company. So, in order to understand customer intent in retail, we need to build a comprehensive user profile that takes into account this temporal dimension. Because if you don’t know the consumer, then you won’t know what the intent of that particular consumer is. Our technology is capable of building those user profiles. We focus on the user first and, to date, have built 600 million plus profiles. That allows us to gain a deeper understanding of customer intent. We use machine learning because, unlike other technology, it works more effectively with more data. The more data you have, the better the model performs. Read More

Why Companies Need To Tackle Big Data With Machine Learning

Leave a comment
Big Data / Machine Learning
big data

By 2018, 50,000 gigabytes of data will be created per second. A significant amount of that data will be stored in corporate server farms. A report from IDG found that “[m]anaging unstructured data is growing as a challenge – rising from 31 percent in 2015 to 45 percent in 2016.” IDC’s Digital Universe report found that “the amount of data stored in the world’s IT systems” doubles every two years. On one hand, many companies are eager to start confronting the challenge of big data. But according to an IDG Enterprise report, 90% of the companies surveyed reported running into major problems when implementing or developing their big data initiatives. No wonder an October 2016 report from Gartner found that most companies who attempted to implement a big data project were mostly stuck in the pilot stage. The challenge of what do to with big data is daunting for many companies. However, machine learning has an important role to play in “solving” big data. Read More