Two meters ap-art

What if our artistic heritage would have been affected by social distancing? Let's explore it by using Deep Learning techniques.

Introduction

Since we started working from home and self-isolating, we thought about what the effects of social distancing will be once we will be back to "normality". But, instead of trying to predict the impact of social distancing in the near future and how people will interact each other after COVID-19, we imagined what the effects would have been if self-isolation, quarantine and two meters apart rules would have been applied constantly over the centuries and had an impact on how artists represented the world.

Keep in mind, social distancing is not an invention of this COVID-19 age. It helped communities in the past to survive infections spreads. In October 1918 the city of St. Louis banned public gatherings to reduce the spread of contagions during the flu pandemic. Closures of parks, schools, theatres and other public places contributed to keep the infection rate under control on several other cases.

The idea for this article came when I saw someone on Instagram posting "A Sunday Afternoon on the Island of La Grande Jatte" (1886) by Georges Seurat. The painting represents an idyllic location at the gates of Paris, where people escape from the city and enjoy fresh air under a tree or an umbrella. The scene is quite dense of people, so I asked myself: what if there was social distancing to be respected at that time? I did a quick retouch, masking some of the figures on the painting and applying either clone stamp or context-aware fill from Photoshop and I came with the following result:

Definitely less crowded as a scene, isn't? With the team we wanted to explore more about the impact that social distancing may have had in our visual cultural heritage, but the "traditional" retouch tools need a lot of manual work to remove human figures and reconstruct the missing background. This is exactly the kind of challenge we were excited to take with Deep Learning! So, we selected some famous and iconic paintings or photographs and we applied a AI technique we knew well: Image Inpainting. Bear in mind, we used a bit of ironic sense of what social distance could have done in the past, so don't take our decisions and outcomes too much seriously!

Image Inpainting vs Context-aware fill

Image Inpainting is the technique that allows to fill a masked area of an image with some generated content that the system predicts should replace most likely, based on the scene. We are gonna see shortly how. There is a fundamental difference from classical context-aware filling tool that you can find on Photoshop or other tools: Image Inpainting AI models learned from data to generalise how to fill previously unseen content, based on the "experience" has gained during the training. Context-aware filling uses the unmasked part of the image to complete the masked area, nothing comes from other data than that. This is a limit as we are forced to "recycle" all the existing content.

In the following comparison, Rembrandt's Musical Allegory (1626). If we pay attention to the context-aware result we will see that the mask area has been filled by pieces of content taken from the rest of the image, while the inpainting technique tries to "imagine" how that empty space should look like without the musician that has been removed. But let's discuss the scene now! The scene is made by a singer with two musicians while an older woman listens on. With a hypothetical social distancing in the 17th century, we imagined that the Golden Age of dutch art would have suffered a lot, partially deprived of the vibrant life that was the trademark of that period.

But inpainting is not the solution for everything. It may happen that actually the system learns to replace a mask with exactly what the shape of the mask suggests, instead of trying to remove the missing object, it can just replace it with something generated, and the results were quite interesting. For example, we compared our inpainting code with the NVIDIA inpainting online tool and a context-aware filling with Photoshop. by masking the silhouette of Leonardo's Gioconda (1506). This time the idea was to imagine art with all the iconic figures absent from the scene, and a text saying just "Stay Home". A concept similar to what we saw elsewhere (examples like Leonardo's Last Supper with Jesus and the Apostles in videoconference). While our code solution was not able to fill completely the mask with the background, NVIDIA solution replaced Mona Lisa with a new face, "overfitting" the original silhouette.

We were not satisfied, therefore we opted for a context-aware filling that cloned the background elements. Here what we got:

Our approach

So, how our AI system worked? We are not gonna dive too deeply into the Deep Learning models needed to achieve our results, we want to keep the focus on the art side rather than talk about the technicalities (there are a lot of good papers + code resources about the topic at Papers with Code). But still, it's good to have a quick overview of how the system is structured.

We designed an AI system made essentially by two steps: object detection and image inpainting, with a selective refinement in the middle. We chose YOLOv3 algorithm for the object detection step, because it's fast and accurate enough in recognising people. The image inpainting task, at least in its "supervised" form, is based on the the assumption that a system has been trained with batches of images that are showed in their original form and partially masked. So the Deep Learning network, based on convolutional layers, learns the "pattern" between masked and unmasked images, and once it has been trained, it tries to predict how to fill any previously unseen masked image. For our implementation of image inpainting, we can say that we used Contextual Attention and Gate Convolution. Let's show the process by means of a concrete example:

We tried to minimise human intervention on our choices and we expected to be challenged by the complexity of art imagery. Let's see what happened.

Some results

Eric Salomon, The Hague (1930). German photojournalist Eric Salomon was the first reporter able to capture what happened behind the scenes of a political meeting. Salomon shot the Foreign Ministers exhausted after a long day of negotiations during the 1930 meeting for the German WWI reparations, which took place in The Hague. If social distancing was applied in that year, politicians may have preferred to stay isolated in separate rooms. The result of the inpainting operation looks more like a foyer at the theatre than a summit.

Grant Wood, American Gothic (1930). Wood's best know work, America Gothic is the quintessential of the rural American Midwest. People think it represents a couple of farmers, but it's actually a father and his daughter. The model were Wood's sister and their dentist. In times of social distancing, would have been possible to see both figures together outside?

Edward Hopper, Nighthawks (1942). The reason behind one of the most popular paintings in American art is debated. As everyone sees loneliness looking through the window of the diner at night, Hopper actually said that Nighthawks has more to do with the possibility to be a predator at night than with loneliness. Performing AI inpainting, the system automatically removed the two "predators" (that Hopper himself modelled for both of them) and kept only the woman (who was modelled by Josephine, Hopper's wife) and the diner's owner. Now the result definitely expresses loneliness in a world affected by social distancing.

Vincent Van Gogh, The Potato Eaters (1885). Van Gogh considered this painting his masterpiece when he completed it. It represents the de Groots family. We believe it would have been difficult for the dutch painter to access their cottage at all, if there was a social distancing in act, but we imagined that actually less members of the family agreed to be painted.

Sandro Botticelli, The Birth of Venus (1486). We can honestly say that the reason why we chose Primavera (as it's best known) was not because we thought that ancient Greek gods should bother about social distancing, but because Zephyr, the wind god, blows at her, an action that simply would look inappropriate during any social distancing context and therefore considered a taboo.

Rembrandt, The Night Watch (1642). Another iconic work from Rembrandt, this painting, famous for the dynamism of its characters, represents the military company led by captain Frans Bannick Cocq (in black) and his lieutenant Willem van Ruytenburch (in white). Without that animated background made of arquebusiers and a woman carrying a chicken, the impact of the military couple in the middle would be way less effective.

Conclusions

We played rewriting (actually repainting) a bit of art history by using Deep Learning techniques like object detection and image inpainting. The results varied as we expected. Fortunately social distancing was not so impactful over the centuries and humanity has been able to create timeless masterpieces despite quarantines and lockdowns. We all hope to go back to the normality soon and meet again. In the meantime, stay safe.