Transforming images with artistic style

Introduction

A powerful means of expression is art that captivates our senses and awakens our emotions. In this advanced era of generative artificial intelligence (ai), a new avenue has emerged to combine the realms of creativity and technology. An interesting and trendy application of generative ai is style transfer, a technique that allows us to transform the visual style of an image or video. In this blog, we will explore the role of generative ai in style transfer, explore its concept, implementation and possible implications.

Learning objectives

• Understand what style transfer is and how it combines artistic styles with content.
• Learn to implement style transfer techniques on our own.
• Understand the applications of style transfer in multiple industries.

This article was published as part of the Data Science Blogathon.

Understanding style transfer

In essence, style transfer seeks to bridge the gap between artistic style and content. Style transfer is based on the fusion principle, which extracts the style from one image and applies it to another to combine the content of one image with the aesthetic qualities of another and generate a completely new image. Basically, it relies on deep learning algorithms, specifically convolutional neural networks (CNN), to perform this style transfer process.

Implementation: Revealing the Magic

First, we need to explore some of the key techniques to understand the implementation of style transfer. Let’s understand the basic techniques followed by the code.

Preprocessing: Input images are generated by resizing them to the desired size and normalizing their pixel values. In this preprocessing step, we need to collect and modify the input images.

Neural network architecture: A pre-trained CNN (often a VGG-19 or similar model) is used as the basis for style transfer. This network has layers that capture the low- and high-level features of the image.

Presentation of contents: The image content representation is generated by passing the image through selected layers of your CNN and extracting feature maps. This representation captures the content of the image but ignores its particular style.

Style expression: A technique called Gram matrix calculation is used to extract style from an image. Compute correlations between feature maps in different layers to obtain the statistical properties that define the style.

Loss function: The loss function is defined as the weighted sum of the content loss, the style loss, and the total variation loss. Content leakage measures the difference between the content representation of the input image and the content representation of the generated image. Style leakage quantifies the style mismatch between the style reference and the generated images. Total variation loss promotes spatial smoothness in the resulting image.

The artistic implications

The transfer of style has opened up interesting possibilities in art and design. It allows artists, photographers and enthusiasts to experiment with different styles, pushing the limits of visual expression. Additionally, style transfer can serve as a tool for creative inspiration, allowing artists to explore new aesthetics and reimagine traditional art forms.

Real world applications

The transfer of style extends beyond the realm of artistic expression. It has found practical applications in industries such as advertising, fashion and entertainment. Brands can take advantage of style transfer to create visually appealing ads or apply different styles to clothing designs. Additionally, style transfer can be used by the film and gaming industries to achieve unique visual effects and immersive experiences.

Ethical considerations

As with any technological advancement, style transfer comes with ethical considerations. Simply manipulating visual content using style transfer algorithms raises concerns about copyright infringement, misinformation, and potential abuse. As technology advances, it is important to address these concerns and establish ethical guidelines.

Code

Simplified implementation of style passing using the TensorFlow library in Python:

import tensorflow as tensor
import numpy as np
from PIL import Image

# Load the pre-trained VGG-19 model

vgg_model = tensor.keras.applications.VGG19(weights="imagenet", include_top=False)

# Define the layers for content and style representations

c_layers = ('b5_conv2')

s_layers = ('b1_conv1', 'b2_conv1', 'b3_conv1', 'b4_conv1', 'b5_conv1')

# Function to preprocess the input image

def preprocess_image(image_path):

    img = tensor.keras.preprocessing.image.load_img(image_path)

    img = tensor.keras.preprocessing.image.img_to_array(img)

    img = np.exp_dims(img, axis=0)

    img = tensor.keras.applications.vgg19.preprocess_input(img)

    return img

# Function to unprocess the generated image

def deprocess_image(img):

    img = img.reshape((img.shape(1), img.shape(2), 3))

    img += (103.939, 116.779, 123.68)  # Undo VGG19 preprocessing

    img = np.clip(img, 0, 255).astype('uint8')

    return img

Here, we are extracting features from middle layers.

def get_feature_representations(model, content_img, style_img):

    content_outputs = model(content_img)

    style_outputs = model(style_img)

    content_feat = (c_layer(0) for content_layer in content_outputs(len(style_layers):))

    style_features = (s_layer(0) for style_layer in style_outputs(:len(style_layers)))

    return content_feat, style_features

# Function to calculate content loss.

def content_loss(content_features, generated_features):

    loss = tensor.add_n((tensor.reduce_mean(tensor.square(content_features(i) -
            generated_features(i))) for i in range(len(content_features))))

    return loss

# Function to calculate style loss.

def style_loss(style_features, generated_features):

    loss = tensor.add_n((tensor.reduce_mean(tensor.square(gram_matrix
           (style_features(i)) - gram_matrix(generated_features(i)))) 
            for i in range(len(style_features))))

    return loss

Function to calculate the Gram matrix

def gram_matrix(input_tensor):

    result = tensor. linalg.einsum('bijc,bijd->bcd', input_tensor, input_tensor)

    input_shape = tensor.shape(input_tensor)

    num_locations = tensor.cast(input_shape(1) * input_shape(2), tensor.float32)

    return result / (num_locations)

# Function to calculate the total variation loss for spatial smoothness

def total_variation_loss(img):

    x_var = tensor.reduce_mean(tensor.square(img(:, :-1, :) - img(:, 1:, :)))

    y_var = tensor.reduce_mean(tensor.square(img(:-1, :, :) - img(1:, :, :)))

    loss = x_var + y_var

    return loss

# Function to perform style transfer

def style_transfer(content_image_path, style_image_path, num_iterations=1000, 
        content_weight=1e3, style_weight=1e-2, variation_weight=30):

    content_image = preprocess_image(content_image_path)

    style_image = preprocess_image(style_image_path)

    generated_image = tensor.Variable(content_image, dtype=tensor.float32)

    opt = tensor.optimizers.Adam(learning_rate=5, beta_1=0.99, epsilon=1e-1)

    for i in range(num_iterations):

        with tensor.GradientTape() as tape:

            content_features, style_features = get_feature_representations(vgg_model, 
                                               content_image, generated_image)

            content_loss_value = content_weight * content_loss(content_features, style_features)

            style_loss_value = style_weight * style_loss(style_features, generated_features)

            tv_loss_value = variation_weight * total_variation_loss(generated_image)

            total_loss = content_loss_value + style_loss_value + tv_loss_value

        gradients = tape.gradient(total_loss, generated_image)

        opt.apply_gradients(((gradients, generated_image)))

        generated_image.assign(tensor.clip_by_value(generated_image, 0.0, 255.0))

        if i % 100 == 0:

            print("Iteration:", i, "Loss:", total_loss)
    
            # Save the generated image

            generated_image = deprocess_image(generated_image.numpy())

            generated_image = Image.fromarray(generated_image)

            generated_image.save("generated_image.jpg")
    

Conclusion

To push the limits of creativity and imagination, generative ai shows its potential by combining art with technology and proving that the combination is a game-changer. Whether as a tool for artistic expression or a catalyst for innovation, style transfer shows the extraordinary possibilities when art and ai intertwine, redefining the artistic landscape for years to come.

Key takeaways

• Style transfer is an interesting generative ai application that allows us to transform the visual style of an image or video.
• It uses deep learning algorithms, or convolutional neural networks (CNN), to perform this style transfer process.
• Brands can take advantage of style transfer to create visually appealing ads or apply different styles to clothing designs.

Frequent questions

The media shown in this article is not the property of Analytics Vidhya and is used at the author’s discretion.

Related