Transforming photos into art, automatically: classic algorithms, neural style transfer, GAN, … and everything in between

Hello! I’m Ricardo Corin, and for the last decade or so, I’ve been interested on turning photos into art. Being a software developer, and having no artistic talent nor patience for being an actual painter or graphic designer, the idea of using computers to automatically turn photos and images into art really appeals me. I’ve been building apps for this end for many years now (at some point I’ve formed a company that was doing it fulltime, with several amazing people… sadly nowadays I’m consulting at a different field so I do it mostly on my free time). But my itch for this problem and the different approaches is still very much alive in me (which surprised me, it’s lasting much longer than I thought I would be still interested, really… but here I am).

Also in the last decade the field (which encompasses both academic research and commercial applications, both desktop and mobile) has seen dramatic improvements in terms of output quality and realistic results; it moved from classic algorithmic approaches, which were fairly basic painterly algorithms constrained basically by old and limited hardware, to nowadays fancy Machine Learning techniques that use powerful GPUs, like neural style transfer and GANs. And I moved along these techniques too, making photo apps along the way: Superphoto was a first app that used classical algorithms, then Painnt and Neuralstyle.art use neural style transfer. Neural style really made a leap and improved the results significantly, the output paintings can be really good!

Neural Style

Let’s see some neural style transfer examples. Take this beautiful sea lion photo:

Image in public domain, from https://all-free-download.com/free-photos/download/sea-lion-in-water_197564.html

Let’s use neuralstyle.art to style it. The magic of neural style is that we can choose any style as we like, it’s just a matter of downloading a painting with the desired style and that’s it. For example, this style:

Style photo

Gives the following sea lion stylized:

Stylized sea lion using the above style

Here are some other samples:

Neural style is really a magical technology! It allows any artist to explore and create some awesome art. But, it has some drawbacks:

1. it’s difficult to use: it’s hard to choose a style that works well, and it’s hard to tweak neural network parameters to obtain good results
2. it’s computationally expensive: the original method, of Gatys etal, takes a long time to compute, and requires powerful GPUs
3. it’s difficult to make HD: the method works on whole-images, so one needs to pool several GPUs together to make enough memory to process images in HD (15MP+ resolution).
4. other neural style approaches work fast and with less memory (like the mobile app Painnt), but they also lack in the above items.

SDFX: GAN used in photo to art

GANs are attractive because they can generate realistic images from scratch. There’s for example StyleGAN, which was used in the famous https://thispersondoesnotexist.com/ site. It’s not clear how can we use GANs to style a photo, similarly to neural style transfer, and also avoid all the issues with style transfer. We need a tool that users can really control when generating art, for example the detail & brush parameters of the outcome, and a tool that can really scale in output size without needing supercomputers or really expensive GPU setups.

For the last year, I’ve been working on a new app, called SDFX. SDFX has a collection of photo filters, some classic and some vector ones, but lately I’ve added a new filter called “Brushstrokes”. This filter takes a different approach to turn a photo into a realistic painting: it “simulates” an actual painting by starting on a blank canvas and simulating each brushstroke on the canvas. This way, the canvas can be arbitrarily large without requiring more memory, it will simply take some more time because more brushstrokes will be needed, just like a real painting. Also, controlling each brushstroke allows to vary it’s size or detail, depending if we’re in an important area (like the eyes of a subject in a portrait), or using a large brush if we’re in the background or some other unimportant area.

So, you ask, where’s the use of AI and machine learning here? The trick is that each brushstroke is generated by a specialized painting GAN network. So, each brushstroke is a unique sample for covering the needed space, and this results in a realistic painting. Have a look at the above sea lion, using SDFX’s Brushstrokes style:

Sea lion painted with SDFX

Since we paint each brushstroke, we can see the app working in real time:

This is just the beginning, and future versions of the app will for example use face detection and saliency to paint with more detail on the important areas. Also, one can use different brushes (which each one being a GAN) to paint differently, just like a painter switches brushes when painting. Of course, other effects/styles will come to SDFX, not necessarily painting ones, for example pencil sketching or etching are certainly possible. On the other hand, this approach is not as flexible as neural style in the sense that each effect needs coding, as opposed to neural style where just a new image gives a new style. But the advantages are appealing: HD, and user control, and no expensive hardware needed.

SDFX and neural style both use AI and machine learning, but in a different (and crucial) way: Neural style approaches let the AI be the ultimate boss and fully generate the outcome result. This makes it usage simple, but difficult to control: it behaves like a “black box” that works for a while and outputs the result, but we’ve no idea what happens inside — this is actually a common issue with AI and other methods called “deep learning”. On the other hand, SDFX uses AI but not in “master mode”, but rather uses AI as a tool to make a “smart/realistic brush”, but it reserves the “painter” part to a more classical algorithm. This in my opinion is the best of both worlds, because now we understand what each style is doing, and can give the user control on the outcome… however, we use the magic of AI to simulate painting brushstrokes which would be very difficulty to program by hand. Being “in charge”, SDFX can also leverage other AI tools to improve the outcome, for example use Saliency or face landmarks to detect areas of the input image and select different brushes accordingly when painting.

Transforming photos into art automatically is a nice space to be around because it is at the intersection of technology and art/creativity, and it requires some skill in both areas. It’s not uncommon to see papers or apps that claim to do for example a new neural style transfer very quickly or with some super advantage, but the stylistic outcomes are very ugly. With SDFX, I more tend to not care if the result is very fast or uses very little memory, if the result is better or the user can control more stuff I don’t mind. Some users complain that the apps don’t provide immediate results or sometimes crash when doing HD, but I prefer to wait a bit of time and get a good result than getting immediate results that look awful. It’s a matter of taste, I guess :)

Thanks for reading! and stay tuned for updates!

ps. If you made it this far and have a Mac or an iPhone/iPad, you may want to check out SDFX.