Designing an artificial intelligence system based on citizen science to recognize architectural styles of monuments

A multidisciplinary team made up of experts from the Andalusian Interuniversity Institute for Data Science and Computational Intelligence (DaSCI), the University of Granada’s group ‘Soft Computing and Intelligent Information Systems (SCI2S)’ together with Fundación Descubre has designed a computer system based on artificial intelligence combining mathematical algorithms, artistic descriptions and citizen participation to identify architectural features in monuments.

Multidisciplinary team of MonuMAI project.

Specifically, the designers have configured an algorithm that, with the collaboration of anonymous people who provide photographs to the system, detects architectural elements in these images. In order to do this, the algorithm first performs a general reading of the photograph and divides it into different parts. Then it processes all the information it finds in each of these sections and scans it in search of known objects. If these are detected, the system points out, processes and identifies this data.

To start working on identifying the structural and decorative parts of an architectural work, the experts have shown the system examples of supervised images. They have thereby created a database that already contains about 4,000 elements. «This fieldwork is still active, as it is an open resource bank where the algorithm, designed by researchers and science communicators, is learning, improving and updating from new photographs sent by anyone from their mobile phone. The participation of the public in this project is essential because they contribute a large share of the materials with which the system works”, explains Teresa Cruz, one of the authors of this work and director of Fundación Descubre.

As set out in the article entitled ‘MonuMAI: Dataset, deep learning pipeline and citizen science based app for monumental heritage taxonomy and classification’ and published in the journal Neurocomputing, this system is based on what is known as ‘deep learning’. This method of learning works with artificial neural networks, in the same way as the human brain does. «The procedure is the same as that used when teaching a child to identify objects through observation. He or she is shown a cup and the word ‘cup’ is named, so that when they see that object at another time and in another context they recognise it, know what it is and associate it with that particular term,» explains to Fundación Descubre Alberto Castillo, researcher at the University of Granada and co-author of the study.

This algorithm is currently programmed mainly to offer information on the architectural style identified from colour, structure, shape, background or texture, while providing data on proportions and symmetries. However, its performance will depend on the prior definition given to it, on the labelling established a priori by the experts in order for the algorithm to supply detailed and precise information.

Another special feature of this algorithm is its immediacy. The estimated time to process an image is 0.15 seconds and the waiting time for it to collect the specific information of the identified architectural element is less than two seconds. «These are very short fractions of time, but we want to reduce them so that the descriptive data of the image reach the user in real time,» says Castillo.

Hospital Real, Granada.

In this work, mathematics is related to artificial intelligence in two ways. «On the one hand, it supports the visual recognition algorithms and the methods with which we work. And, at the same time, it is part of the reasoning methods that allow each artistic style to be classified,” clarifies Álvaro Martínez Sevilla, researcher at the DaSCI and director of the project ‘Paseos Matemáticos’.

This multidisciplinary study results from the combination of four areas -mathematics, art, artificial intelligence and citizen collaboration- that are very different, but at the same time complementary within a system like this. «Here is an example of an interdisciplinary work where artificial intelligence is placed at the service of citizens to better appreciate aspects of their heritage, in this particular case, monuments,» states Francisco Herrera, Professor of Computer Science and Artificial Intelligence and director of DaSCI.

MonuMAI, a heritage ‘tracking’ app

In order to test the usability and performance of this algorithm, the experts have developed a web application for android and iOS mobile phones and a website called ‘MonuMAI’, an acronym that unifies the fields of study and work on which this system is set up: monuments, mathematics and artificial intelligence.

Both platforms feed off each other because the website displays information generated by the mobile application, specifically the processed images of the monuments, and at the same time, the app collects all the contents of the website related to each area of research, the explanation of the project and also the photographs of the users.

Specifically, the app consists of a library containing all the elements registered in it and two algorithms responsible for detecting these objects and providing information about them. One reports on the architectural style and the other reports on the elements in the image. «These features make MonuMAI the first app in the world that extracts expert knowledge and offers it directly to the end user,» says Castillo.

Once the application has been downloaded, anyone interested can upload their own photograph of a particular monument. This allows the public to participate actively and directly in this collaborative project of citizen science where they act as citizen-researchers, thereby contributing to improve the system because the algorithm continues to learn more and more. They also provide up-to-date information with continuous input of data, in this case, in the form of images.

MonuMAI has a web application for android and iOS mobile phones and a website.

As soon as the server automatically receives and processes this information, it is returned with details about the architectural elements of the styles it knows. So far, the model identifies elements of artistic styles from all over Europe, including those from Medieval and Modern Ages Andalusia, such as Hispano-Muslim, Renaissance, Baroque or Gothic art, and even objects found in monuments in France and Italy.

In addition, this application provides detailed information accredited by Andalusian art and mathematics researchers. «Unlike other apps or programs with similar purposes, this tool does not redirect to other websites that offer explanatory content, but offers its own data. And that information has been generated, in fact, by an artificial intelligence that has learned from the expert knowledge we have provided,» Castillo points out.

In order to share all this knowledge with the public, the experts have initiated a website where images sent by other users can be viewed. «You can see the different architectural elements with just the click of a button, as if it were an online art library,» says Castillo.

The objective of this team is to continue improving the system and provide it with additional functions. In particular, it is studying the possibility of designing new algorithms that recommend visiting monuments close to those consulted in MonuMAI.

This study has received funding from the University of Granada and Fundación Descubre.

Spanish’s version: Diseñan un sistema de inteligencia artificial basado en ciencia ciudadana para reconocer estilos arquitectónicos en monumentos

References

Alberto Lamas; SihamTabik; Policarpo Cruz; Rosana Montes; Álvaro Martínez-Sevilla; Teresa Cruz y Francisco Herrera: ‘MonuMAI: Dataset, deep learning pipeline and citizen science based app for monumental heritage taxonomy and classification’. Neurocomputing. 2020.

Más información:

#CienciaDirecta, agencia de noticias de ciencia andaluza, financiada por la Consejería de Transformación Económica, Industria, Conocimiento y Universidades de la Junta de Andalucía.

Teléfono: 954 232 349

E-mail: comunicacion@fundaciondescubre.es