Inserting and Encircling Two complementary immersive strategies for mixed-reality applied to cultural heritage *

To accomplish the aims of a three-year research project we are developing, connected to cultural heritage, we became interested in the fusion of Virtual Reality and Augmented Reality, two emergent development fields that gave birth to what was coined as Mixed Reality. Both dimensions have intricate connections with hardware and software improvements related with the so called ``4th Industrial Revolution''.Virtual Reality (VR), an interactive experience generated by a computer, takes place inside of simulated environments, which can be analogous to the real world or which can be created as imaginary contexts. On the other hand, Augmented Reality (AR) is always based in an interactive experience inside a tangible environment where the elements of that reality are nurtured with digital information, across several senses, to empathize certain aspects of reality. Our research combines both VR and AR to empathize sensory and intellectual experience. To do so, several senses, mainly visual and auditory, are stimulated.We therefore explore two Case-Studies from our research project in order to show two different strategies. The intention of both situations is to create immersive mixed reality environments where the fusion of the digital and analogue elements can be persistently sustained by the visual outputs.


INTRODUCTION
The Santa Cruz monastery, in Coimbra, of the Canons Regular of St. Augustine, founded in 1131, was one of the major religious houses of Portugal before the reli-gious orders were extinguished, by decree of the new liberal government, in 1834. The following neglect of the building and its application to the most diverse uses led to the progressive obliteration of significant artistic and architectonic legacies. Nevertheless, remarkable architectonic features can still be seen today, such as the late gothic church, the church's chancel (that hosts the tombs of the first two Portuguese Kings), the late gothic cloister and refectory, the renaissance Manga fountain, the mannerist vestry and the baroque sanctuary. (see figure 1) Figure 1 Aerial view of the Santa Cruz Monastery in downtown Coimbra (Photo: Filipe Jorge/ Argumentum, 2003) In fact, during the late 19th and early 20th centuries, significant parts of the monastery were destroyed such as the monastic façade and entrance cloister (replaced by the Town Hall building of 1876-79), the extensive renaissance dormitory, the Manga cloister (which delimited the remaining fountain) and the medieval/baroque bell-tower. Evidently, the scale and monumentality of the monastic complex during the later years of its original function, was largely affected, giving to contemporary citizens and visitors a totally different perception of the architectural ensemble. This is why we thought of setting up a research project that could put together history of art and of architecture with the new technologies, par-ticularly in the fields of Virtual and Augmented Reality. Our initial intention was submitted to a successful application process and is now being funded, as a three-year research project, by the Portuguese Science and Technology Foundation ( FCT-Fundação para a Ciência e a Tecnologia ) under the name "Santa Cruz -3D digital reconstitution of the Monastery of Santa Cruz in 1834".
The research project supports itself on previous historical research by several authors working in related fields, some of them directly involved in the project itself, such as archaeologists (Alarcão, 2011), art historians (Dias & Coutinho, 2003;Craveiro, 2011), photography historians (Ramires, 2001) and architects (Lobo, 2006;Couto, 2014), which will account for the physical state of the monastic complex in the mid-19th century, before the main destructive actions took place.
On the other hand, we became interested in the fusion of Virtual Reality and Augmented Reality, two emergent development fields that gave birth to what was coined as Mixed Reality (Milgram and Kishino, 1994). Thus, the environments created can be analogous to the real world and perceived as extensions of reality. In this project digitally recreated architectural and sculptural data can assist the existing environment with missed architectural and artistic elements, masking important areas of the existing environment with information from other epochs or contexts. This new/old information will be consistently interrelated with the real environment in an immersive experience.
The primary value of Mixed Reality technologies applied to our research project is that they will bring components of the digital world into a person's perception of the real world, not just as a simple display of data, but through the integration of immersive sensations that are perceived as natural parts of a given environment. Main overlaps that we are aiming to build are the elevation of the original cloister surroundings around the remaining Manga fountain (in order to re-insert this magnificent Renaissance architectural piece into its original built envi-ronment), the reconstitution of the monastic Renaissance façade beside the remaining church façade, and the insertion of the magnificent human scale sculptural terracotta ensemble of the "Last Supper" (executed by Hodart between 1530 and 1534 and today at the Machado de Castro National Museum -see figure 2) into its original setting, presiding over the monastery's still standing refectory hall.
From this work, other developments can be attained in the near future, such as going further back in time, to the original Romanesque church and cloister, which were substituted in the early 16th century by the actual church and cloister structures.

MIXED REALITY AND CULTURAL CREATIVE INDUSTRIES
Today, Cultural and Creative Industries are globally recognized as one of the most relevant economic factors of growth and job creation. According to Fo-rum D'Avignon & TERA consultants (2014), it is vital to study and promote the uniqueness of European cultural heritage sites, taking into consideration that these industries represent around 5% of the European Union's GDP and that roughly 10% of the European Union's GDP comes from International Tourism activities.
It's also relevant to mention that UNESCO World Heritage sites compromise almost half of global tourism revenues from cultural products related with the local monuments, arts and crafts. The Santa Cruz monastery is not a World Heritage Site in itself but belongs to the Property Area of the University of Coimbra which was recently granted with the UNESCO World Heritage classification in 2013. [1] The rapid arrival of 3D digitization and the associated shift in the way people apprehend reality is forcing the Creative Cultural Industries (OECD designation) to develop new growth strategies to improve this almost trillionaire activity. The CCI's boast €558 billion in value, added to national GDP's in 2011 (Forum D'Avignon & TERA consultants, 2014).
Simultaneously, contemporary authors/artists such as Maurice Conti and Zenka (Jenny Carden) suggest that we are at the dawn of a new age of human history defined by the way we apprehend knowledge. This last revolution, based in the intimate connection between information, automation and our perceptive capabilities, is connecting us to realities far beyond our natural senses, giving birth to new exploratory fields such as Virtual and Augmented Reality, characterized by cognitive augmentations anchored in our basic sensory system, providing an interactive experience of a real environment where the elements of reality are nurtured with digital information, across several senses, to empathize certain aspects of reality. The interconnection between intrinsic and extrinsic elements of virtual and real worlds allows computational systems to help us to observe and think, based on the way our nervous system was developed and pre-wired, as well as allowing us to communicate symbolically in several cultural contexts.
Unlike Augmented Reality, Mixed Reality is not based on the simple superimposition of layers of information to create enriched contents. Mixed reality works mainly with tridimensional maps of the surroundings to interconnect the real and the virtual elements inside a physical paradigm that leads us to a better understanding of the reality or, if we prefer, to a better illusion.

TECHNICAL SPECIFICS: THE VR TURING TEST EXPERIMENT
Among innumerous aspects related with the search of fidelity in the creation of the Mixed Reality environments, we have focused our attention in the following aspects of space interpretation: • Parallax effect (slicing several pairs of spherical views and programming the process to remake them). • Dropped shadows (coherence and hierarchy). • Illumination (intensity and color).
• Focal distance. The focal point can be part of the storytelling technic, like it is common practice in photography or cinema. The elements that we want the observer to be focused on are the ones we will (literally) focus on at the final cut. • Fog or atmosphere density. • Soundscape based on the interaction between the sound heard in real space and the introduced sound environment. • Coherent sound directions (auditory sense is much faster to be perceived by our brain than the visual stimulus, giving coherence, or not, to the visual stimulus).
Additional sensory feedbacks will also be explored, such as audition and also touch -or, generalizing, the "haptic", as defined by Gibson (1968): "the sensibility of the individual to the world adjacent to his body, by use of his body". The stereoscopical footage and laser scanning are made taking into account the scale and the position of the observer in order to make the physical connection with the remnants of the context and to avoid the sensation of being too big, too small, disembodied or disintegrated. As we have been mentioning the aim is to create immersive mixed reality environments where the fusion of the digital and analogue elements could be persistently sustained by the visual outputs. Renshaw, Sonnenfeld and Meyers proposed in 2006, at the Human Factors and Ergonomics Society Annual Meeting, the ground rules for a future development of a Turing type test for Virtual Reality in order to pursue the objectives of the imitation game created by Alan Turing in 1950. This challenge, initially focused on aspects related with Artificial Intelligence, also promotes studies of how humans construct a judgment about the veracity and tangibility of their routine observations. Accordingly, a Virtual Reality Turing Test was shaped and used as a reference for the accomplishment of created immersive environments. The observers submitted to the VR Turing Test were, firstly, the visitors of prototype displays developed in three workshops in Coimbra and Aalborg, Denmark (see figure 3), where digital versions of human scale sculptures of Hodart's Last Supper are taken from their actual environment -the Machado de Cas-tro Museum -and abruptly inserted into other architectural spaces with scarce relation with the primordial context (the Santa Cruz refectory). Reactions from observers have been photographed and videorecorded in order to verify the momentarily accomplishment of the VR Turing Test. Instinct reactions such as surprise or distress, as well as other more complex responses such as the verification of the physicality of the sculptures, were verified.
This accomplishment was possible thanks to the introduction of three accurate processes to capture and recreate artistic and architectural elements as inserted, or encircling, components: 1. The first process, and the most traditional one, is based in stereo photographic captures. In this case, architectural and sculptural elements are captured with UHD photographic cameras, from several different positions, in order to create parallax effects. Furthermore, captures can be dome-shaped to create full immersion. 2. The second process starts with accurate and long-distance laser scans that are used to create point clouds. The Cartesian data provided by the laser scan, as well as all the materialsrelated information, is used into geometric analyses. The ensuing models are used to create photo-realistic objects and environments. 3. The last method applied is based in photogrammetric analysis. In this case, hundreds of photos are needed to provide a 3D accurate model. This method is especially valuable when the elements to represent are too small or too complex to be examined by a wide-ranging laser scanner. Nevertheless, this method was the one less explored so far, taking into account that we are continuously discovering new techniques to improve laser scanning strategies and processing methods.

INSERTING AND ENCIRCLING
These three methods are being combined in two different ways, in order to recreate high quality mixed reality outputs. In both cases, instead of leaving to the 3D glasses the effort of reading the 3D environment in real time and rendering it, we anticipate the interpretation of the environment with a 3D laser scanning and UHD 360º photo captures combined to provide both 3D vector information and stereo images. These techniques and technologies give us the opportunity to make several geometrical and image refinements that can be given to the final observer as a high quality stereoscopic 360º immersive experience. Consequently, two different Case-Studies can be highlighted from our research project in order to show these two main strategies developed to create Mixed Reality environments. The first concerns the INSERTING of 3D models inside a real environment. In fact, we intend to reintroduce Hodart's 13 terracotta human scale figures that constitute the Last Supper, now at the Machado de Castro National Museum (and set in large room, according to a tentative disposition), into its original setting, a much constricted space (today destroyed) that existed upon the Eastern top wall of the still extant monastic refectory hall, in downtown Coimbra. In the process we will try to reconstitute the original layout of the 13 figures (see figure 4), the table, the benches and the space where the whole composition was displayed, with the help of art historians and experts in Renaissance perspective.
Technically, we have made point-cloud surveys of the sculptures at the Museum (see again figure 2), which have been subsequently worked upon in order to create 3D model meshes of Christ and the apostles.
In parallel we will create a 360º parallax scenario of the Eastern end of the refectory hall. For this we have taken several UHD spherical photos according to the directions and spots where we want to intensify the observer's experience of immersive parallax perspectives. A 3D point cloud scan of the refectory has also been done (see figure 5). This was made in order to create the geometrical frame for the UHD renders needed to complete 8K 360º photographed environments.
A digital virtual scenario of the space where the figures were originally displayed will also be recreated, including the virtual completion of the classical opening of the scene, the remnants of which still exist, integrated in the refectory's Eastern top wall. Furthermore, to simulate several compositional hypotheses, the digital sculptures are being scaled and 3D printed to be tangibly tested before being integrated inside the photographed and digitalized scenarios. Finally, all these elements will be creatively mixed in video. Hence, this reconstitution, which would be extremely difficult to explain by any means of codification or verbalization, can easily reach effectiveness when translated into an immersive scenario. In this sense, our project also combines explicit and tacit knowledge.
Furthermore, this idea to let the biblical "Last Supper" be experienced as realistically as possible was attempted by Hodart himself with his original sculptural recreation, nearly five hundred years ago. With the technology and knowledge of his time, and alike the immersive big paints drawn in perspective views during the Renaissance (such as Leonardo's own "Last Supper" at the refectory of Santa Maria delle Grazie monastery in Milan), the intention was there to experience this canonical context with the maximum of fidelity.
In the second situation the strategy is inverted. The still standing Manga tempietto and fountain (designed by the sculptor and architect Jean de Rouen, 1533) was originally set at the center of the third cloister of the Monastery of Santa Cruz. However, the surrounding structures have been destroyed during the late 19th century and early 20th century (see figure  6).  This second approach is composed by a stereoscopically captured architectural element (the still standing Manga tempietto and fountain) inserted into its "original" virtual environment -the ENCIRCLING 3D model (see figure 7).
All the lost surrounding elements, including the lost four cloister façades and the monastic baroque bell tower (which was visible from inside the cloister) were reconstituted and modelled in 3D using a Rhinoceros 6.0 application, based on old photographs, drawings, plans, written descriptions and other evidence, which contributed to recreate the former cloister's environment as accurately as possible.

CONCLUDING
The aim of the Santa Cruz research project is to allow for a renewed understanding of the Santa Cruz monastery's erased past through the use of Virtual Reconstructions of its lost architectural heritage that can be perceived through Augmented and Virtual Reality platforms and devices. These will be available for visitors of the modern building and site, through the establishment of an array of vantage "hotspots".
As we have mentioned, other sensory feedbacks like the auditory and the haptic will also be explored. In parallel, to enhance sensory and intellectual experience, our project will also combine explicit and tacit knowledge as we have referred.
Nevertheless, our estimate is that the most direct output will be provided by the use of personal Smartphones, allowing visitors to download and watch up to 8K 3D 360º contents, of the lost heritage (or of the sculptures kept at the museum), matching them with the still standing architectural elements which can be seen on site (see figure 8), providing the means for an extensive personal contact with almost lost specific historical contexts.