How art is generated from Algorithms?
By definition, Algorithmic art is a type of art, primarily visual, in which the design is produced by an algorithm. But what are Algorithms? Algorithms are a set of instructions followed by a computer to perform an action/computation. In the context of neural art, algorithms act as a detailed recipe for the design and execution of an artwork. AI can hugely boost our creative work and take ideas to a new horizon. The most widespread forms of AI art centre on processing photos and identifying features like colour, texture, and text.AIs learn your style and preferences and then work at huge scales very quickly. However, “the creativity is still done by people: they are the ones making the connections” or “synthesizing” ideas. In conclusion, AI art happens when machines and humans come together.
This article will provide an overview of various examples of algorithmic arts.
The topics that will be covered are:
- Deep dream
- Neural Style transfer
Deep Dream
Artificial Neural Networks have sparked tremendous interest and progress in image classification. Even though these networks are widely used, we know very little about what goes into each of these layers.
How do we train a neural network?… By simply showing many examples of what we want them to learn in the expectation that they will grasp the essence of the subject at hand but it is not always the case!
Each layer of the neural network deals with various levels of complexity. Lower layers produce simple patterns and are sensitive to basic features like edges and their orientations whereas in higher layers even objects tend to emerge.
In a nutshell, we ask the network:
“Whatever you see there, I want more of it!”.
How does it work?
Deep Dream over-interprets and enhances the patterns it sees in an image. If the network sees something in the cloud that maybe looks like a dog then it tries to make it look more like a dog until they seem to appear out of nowhere from the input image.
This results in a dream-like image. This process was dubbed “Inceptionism”, a reference to InceptionNet and the movie “Inception”!!
The idea of Deep Dream is to choose a layer from a pre-trained model (InceptionV3 in this case) and then enhance the features incorporated in this layer onto our input image. There are 11 layers in InceptionV3 from ‘mixed0’ to ‘mixed10’.
We can select any layers depending upon the complexity of patterns that we want to see. Here we have selected ‘mixed3’ and ‘mixed5’, you can try out different layers and observe different levels of complexity of dream-like, trippy images formed.
The activations in the selected layers are added to determine the loss. Each layer’s contribution to the loss is normalized, preventing larger layers from outweighing smaller layers.
Walk Through the code
// Import necessary libraries
Now, let’s start by taking an image in which we want the patterns and dream-like structures to be enhanced.
// Download an image and read it into the NumPy array
// Normalize the image
// Now choose a pre-trained model and the layers on the network that would be enhanced (InceptionV3 has been used here)
// Calculate Loss
The loss calculated from here is then maximized by gradient ascent to enhance the pattern or features that we bring out from our input image.
Now specify the iterations and step size.
Play around with the layers that have been chosen from the InceptionV3 pre-trained model and generate some cool “dream-ified” images!!!
Implemented on a small scene from “Inception”
Neural Style Transfer
Ever imagined how your ordinary photo might look if it were painted by some famous artist? Well then, the Style Transfer technique is what you are looking for to turn your desire into reality!
Neural Style transfer builds on blending the content image to a style reference image such that the content is painted in a specific style. This technique employs a pre-trained Convolutional Neural Network for feature extraction and separation of content and style representations from an image. Any NST network has two inputs: Content image and Style image. The content image is recreated as a newly generated image which is the only trainable variable in the neural network.
Image style transfer is a growing topic of research and aims to boost the artistic skills of every person. It is currently being used in diversified domains like gaming, virtual reality, photo and video editors etc.
How does it work?
Style Transfer uses a pre-trained Convolutional Neural Network with added loss functions, in order to transfer the style from one image to another to create a new image with the features we want to add.
Training a style transfer model requires two networks:
i. Pre-trained feature extractor.
ii. Transfer network.
The model cannot interpret the images on its own. These have to be transformed into raw pixels and supplied to the model. After the image is fed to the model, the deep neural network(eg.- ImageNet-VGG) is employed to extract the features of the content and style image at various layers. These multilayered features are used to compute the respective content and style losses and the final stylized transfer image is generated using the combined loss.
Loss Functions
We define two different types of loss functions to get the desired output. Optimizing this function helps us obtain the desired result which has a balance of content and style.
a) Content loss
It helps to establish a correlation between the content image and the generated image. This function helps retain features of the content image in the generated image. This way the final output image has content similar to the provided content image but has added style features from the style image.
Content loss is calculated by Euclidean distance between the respective intermediate higher-level feature representation of input image(x) and content image(p) at layer l.
This helps us to deduce that images having the same content should also have similar activations in the higher layers.
b) Style Loss
Style loss measures the difference between the low-level features of the newly generated image and the base image. It penalizes the output features ( like colour and texture) when it deviates from the original style image.
The generated outputs are averaged over every value in the feature map to calculate the Gram matrix. This matrix measures the style at each layer and helps to identify which regions of the image are related. Hence different style features which are correlated can exist simultaneously. We make gram matrices of both the style and generated images and then compute the difference between the two.
Below is the formula for the gram matrix :
The style loss is the squared distance between the gram matrix of the generated image and the gram matrix of the style image (Frobenius norm of the matrix). Below is the formula to calculate the Frobenius norm of a matrix :
Now we have computed both the loss functions. Therefore to calculate the final loss we will compute a weighted summation of both the computed content and style losses.
where α and β are user-defined hyperparameters. These hyperparameters help us to control the amount of content and style in the generated image.
STYLE IMAGE :
CONTENT IMAGE:
FINAL IMAGE:
Code:
In this article, we have discussed what algorithmic art is and how it is generated. Further, we discussed two examples of the same. Hope this brief introduction to the mystical world of art through algorithms will inspire you to deep-dive into algorithmic art. We will be returning with more art-generating techniques soon!