Implementing Augmented Reality for the Holographic Assembly of a Modular Shading Device

The development of innovative digital design and fabrication tools for material processing and manufacturing of complex and non-standard forms, apart from their advantages, have brought a number of challenges. These might be related to the effectiveness and sustainable potential of implementation associated with environmental, cost and time-related parameters, particularly in cases of large number of elements construction and complex assembly. Augmented Reality (AR) is an emerging technology with great potential for implementation in the construction industry, since it can enhance the real world with additional digital information, and thus, can assist towards manufacture and assemble of these particular systems. This study presents an AR methodology for assembling a modular shading device and discusses the advantages and disadvantages that this application can bring to the Architecture, Engineering and Construction (AEC) industry by taking into account precision and construction time issues based on the handling of the process by skilled and unskilled users/workers. Our aim is to investigate the potential implementation of AR in the assembly, and consequently, in the construction process as a whole. Also, this study aims at exploring existing constraints of the technology and suggests ways of improvement.


INTRODUCTION
Augmented Reality (AR) provides a mixed environment where digital information is superimposed onto the user's view in physical world (Chi et al. 2013) whereas in Virtual Reality (VR) the users are placed in an immersive environment that is entirely computergenerated [1]. In recent years, there has been an increasing trend in the use of AR technology in AEC industry, as it provides the possibility of construction visualization, allowing decisions to be made [2]. Specifically, it can be used for visualizing buildings on site, for rendered 3D visualization of interiors, furniture, structures, etc. In addition, the technological ability to overlay digital information onto real world environment, allows its implementation in manufacture and construction activities, mostly in cases of complex and non-standard building components and overall structures. In such cases, the accuracy in the implementation of the large amount of information is a demanding process that cannot be solely undertaken either by skilled or by unskilled users/workers.
Specifically, AR can assist manufacturers, builders and generally construction agents in the inspection and making of more complex forms of physical components by following the shape of the virtual ones through AR glasses. Although, with limited use, in literature review works discuss AR potential and applicability in the near future (Chi et al. 2013). Examples of integration within a digital design and construction framework can be found in several cases, for instance in the example where a cost-effective interactive AR tool is used in the physical development of free-form modular surfaces (Fazel and Izadi 2018). In addition, integration of technology is observed in a unified framework using automation and robots [3]. In the project (Johns 2012), the process of designing a curve on the floor by the user is followed by its translation into a free-form wall, which is visible through the application of a low-tech augmented reality system based on a mobile device. Then, the design is send to an industrial robotic arm for fabrication. In addition, in examples such as (Johns et al 2014), the design and fabrication sequence is protracted through an augmented material concept that allows interactive involvement of users.
The development of open source software and hardware towards this direction, such as the Fologram plug-in [4] and the HoloLens glasses [5] respectively, opens up new frontiers of AR technology research to the wider community of architects and builders in a relatively simple and fast way. Therefore, the application of AR in architecture and construction can broaden the spectrum of possibilities in design as well as can speed up the process without the necessity for implementing expensive fabrication tools and time-consuming programing and control processes. Although, AR technology appears to be in use, its application is mostly found in an early stage of imple-mentation with few examples. This does not provide a clear picture of AR capabilities in the construction industry, especially in cases of complex form construction, where a large number of standard or customized components need to be assembled in a specific manner.
This paper aims to deepen into this research direction through a suggested methodology for holographic assembly that can be used to assist people involved in physical prototyping or constructionscale tasks. This is tested in an experiment of AR implementation in real scale and particularly in the assembly of a shading devise consisting of modules, which are placed and interlocked in specific manner in order to formulate the overall system. Through this experimentation, consideration is given to the ability of specific technology to be applied, taking into account precision and construction time issues that are based on the process of handling by skilled or unskilled users/workers. In addition, existing constraints are detected at both technological and methodological levels and suggestions are made for possible improvement of available technology.
In the following section, the methodology introduced in this work is briefly discussed. Then, an overview of the experimental study is presented and a number of tests for the holographic assembly of the specific modular shading device is demonstrated, which involves skilled and unskilled users/workers, evaluating at the same time the accuracy of the technology and the construction time associated in each case. This is followed by discussion on the results and finally, conclusion and suggestions for further development are given.

OVERVIEW OF APPLIED METHODOLOGY
The suggested methodology aims to establish a framework where AR can be applied in prototyping and construction-scale applications of complex assemblies, either in cases of non-conventional morphologies, where systems consist of large number of variable components or regular systems that involves complex interlocking logics. Such systems and their Tab-hold-move for grid origin control construction logic require the introduction of technologies, both at the production and assembly level, which in case of conventional applications, would cause many failures and would be economically unprofitable. The choice of AR technology, as an alternative solution, aims to investigate its advantages and disadvantages and clarify its role within this research framework. An initial summary of its benefits shows that AR can provide many possibilities like to superimpose virtual elements in space with their accompanied information projected in 2D or 3D view, to animate virtual elements in space, to 'teleport' in different spaces providing the feeling of physical presence of users in another location [6] and to allow an ease hand gesturing without extra devices. Within this scope, AR technology would be an advantageous alternative, which is worth to be explored further.
In order to achieve the above objectives, the current exploration focuses on available technology, and particularly on the use of Fologram plugin [4] embedded in the parametric environment of Grasshopper plug-in [7] for Rhino [8] with the HoloLens glasses [5]. This combination of technologies provides, on the one hand, a parametric design environment where complex components can be de-veloped and controlled in digital world, and on the other hand, it allows interactivity in physical word, by enhancing and superimposing the digital results and their accompanied digital information onto real world. Particularly, through the AR devices the users can navigate and interact with digital objects projected in physical space using the AR glasses, their gaze and hand gestures.
Hand gesturing plays an important role during the suggested process, because the users as acting agents, need to control their view and the way digital components and information associated with them appear in their immediate front view through gestures. At the same time, the users/workers need to physically assemble the structural components of the proposed structure, providing a proper balance between different hand and bodily motions. Within this framework, hand gesture possibilities have been thorough examined, mainly in order to establish an understanding and communication between the users/workers and the projected digital information. Figure 1 demonstrates selected hand gestures and associated steps in the process of activating, specifying and calibrating digital elements in the physical space.

Figure 2
The workflow of the suggested AR assembly procedure that includes: A. The physical scene, digital overlapping and holographic assembly, B. User/worker and AR glasses, and C. Computer control and associated software The current investigation deals with a specific construction-scale activity, although other tasks associated with the fabrication process of complex systems, using the same applied methodology, can be examined in future studies. Particularly, the goal of this study is to investigate precision and construction time results of a modular shading system, which is assembled on an existing opening by skilled and un-skilled users/workers, using the suggested methodology. Figure 2 briefly demonstrates the flowchart of the AR-based holographic assembly process that incorporates digital parametric design and control as well as physical and digital overlaying and assembling.

Shading device development overview
The experimental work introduced in this paper deals with the investigation of the possibilities provided by the implementation of AR technology in the construction phase of a shading building system. However, its special form and assembly logic, which involves assembly and disassembly process, can be extended to operational level, with AR technology being used to adjust the data according to the lighting comfort conditions of the interior space. This might be a process guided by augmented technology. In particular, the proposed shading system consisting of primary and secondary modules as well as joint parts, can be placed on facades or window frames of existing or new buildings. The system is acting as shading device to control the amount of light entering the interior based on seasonal environmental conditions (Figure 3). The morphology of the unit and of the overall system is based on a honeycomb-like structure in the form of hexagonal pattern made up from two standard Vshape modules, which are acting as primary system members, a secondary V-shape acting as cell divider and a number of joint parts (Figure 4). The joints are used for keeping the primary units together, both to shape each hexagon but also to connect the adjacent ones in a unified and rigid system. The secondary units are placed inside the hexagonal cells without further joining. In this way, the users/workers who are occupying the interior space can freely add or remove modules, depending on the amount of lighting that enters the space. Figure 5 demonstrates a preliminary fabrication test and the assembly logic.
As it has been already described, the holographic assembly process involves the development of a parametric definition of the suggested system, which for this experimental study, is based on an identical hexagonal patter that is repeated in a certain manner, controlling at the same time the assembly sequence. Despite the canonical nature of the specific morphology, this experimental study contains particular characteristics that could be considered as rea-sons for using AR technology as opposed to a conventional assembly process, and include large number of elements and complex assembly logic that requires placing and interlocking in specific manner in order to formulate the overall system ( Figure 6).

Experimental set up and holographic execution
Subsequently, the Fologram plug-in is used to stream the design information to the HoloLens glasses, offering the opportunity to the users/workers to compose digital data with the physical set up of opening where the hexagonal pattern is assembled. Also, it allows them to interact with the projected pattern through gestures, or to modify and test the different design possibilities in real time by their direct visualization in real physical space. In addition, the process enables the control of the next unit to be placed and generally the assembly and construction sequence. Apart from the positioning of structure, the technology enables an ease transformation by dismantling and reassembly of the system or an extension of it, based on environmental conditions requirements. The experimental scene is set up by selecting a room with dimensions of 6m x 3m and a window with dimensions 1.84 x 0.90m, on which the system is assembled ( Figure  7; Figure 8). The user/worker, through the HoloLens glasses, can perceive the virtual structure on the window and at the same time can move within the given space to a further point for a clearer perspective of the whole structure or closer to the window for placing the modular parts and the joints. The user/worker is also able to drag his/her arm in front of the AR Figure 6 Parametric development of the suggested shading system and basic assembly sequence glasses in order to adjust the right frame of scanning and make visible the first unit to be placed. In every sequence of new unit placement and joint connection, the user/worker air taps and drags his/her arm for the next module to appear on the virtual model. This process is kept repeated until the whole structure is completed (Figure 9).

EXPERIMENTAL STUDIES
A series of experiments have been conducted in order to test the feasibility of the process, especially in terms of its usefulness to be implemented by skilled or unskilled users/workers and in cases where the overall structure under investigation consists of a large number of components with complex assembly sequence. As it has been stated, this paper demonstrates experimental study results based on the use of the proposed shading device, which was described in previous sections. The experimental studies are distinguished into two basic test categories: a. Preliminary tests aiming to assess the performance of the process that includes ease of assembly and construction time with experimentation conducted by students of architecture, b. Advanced tests aiming to assess the performance of the process and particularly construction time by skilled and unskilled users with architecture or other background. Towards this direction, information and results derived from the tests regarding precision and construction time are analyzed and discussed.

Preliminary tests (PTs)
Participants involved in the preliminary tests were 3, with age between 21 and 25 years old. The internal (In.) participant was a student of architecture, who was involved in the development of AR system. The other two users (A and B) were also architects but unskilled in terms of construction and assembly tasks. After a brief introduction to the AR system, hand gesture control and a summary of assembly tasks, each participant conducted the test with and without AR glasses.

Figure 9
Screenshots of the assembly execution of modules using AR glasses. The pictures are the results of video recording taken by camera mounted at the AR glasses The analysis of results obtained from the preliminary tests shows that unskilled users/workers can benefit from the use of AR glasses, both at the level of faults reduction and at the rate of increase in the assembly completion speed. Also, a reduction in the assembly time of specific number of components was observed compared to the results derived without AR glasses ( Figure 10).

Advanced tests (ATs)
For the second category of tests, 6 participants were involved, with age between 20 and 25 years old. Users A, B, and F were students of architecture and users C, D, and E students with different background, all of them unskilled in terms of the construction and assembly. Each of the participant conducted the tests with the aim to fulfill the assembly of overall system, whose construction sequence was divided into 4 completion steps, 1st layer, 2nd layer, half structure and full structure. Also, each participant repeated the test twice: with AR and without AR glasses. Results were associated with completion time in each step. Table 1 and Table 2 demonstrate corresponding results with and without AR.
The analysis of results obtained from the advanced tests shows that unskilled users/workers, both with background in architecture and in other fields, can take advantage of the AR glasses during complex assembly, particularly during first layering of components. Also, the comparison between tests with and without AR shows that in the first case, the time needed to accomplish the task is, in general, less than the time needed in the second case. This shows that the AR might be useful towards learning curve improvement, especially during the beginning of the process (Figure 11).
In addition, the results show that the best assembly time performance was achieved by user B and D, with architectural background in the first case and with non-architectural background in the second case. The largest assembly time was performed by user C, a result that was out of the given time limits (60 minutes approximately). In both cases, results show that tests with AR achieved to reduce the as-

Figure 10
Assembly time for the completion of 8, 36 and 60 components by user A and user B with AR and without AR Table 1 Results of preliminary tests by 6 users with AR glasses sembly time required, compared with tests without AR, especially during the two first layers (Figure 12).

Discussion on constraints
The results obtained during preliminary and advanced experimentation are accompanied by a series of observations regarding limitations of existing AR technology. Such discussion could provide feedback for technological and procedural further improvement. These observations are summarized as follows: • Obstructions have been observed regarding the accurate assembly of components due to the overlapping of hybrid components on the existing physical structure in each assembly sequence; • The size of frame/scope through which the visualization is projected is found to be limited, which means that the users/workers often have to rotate their heads to view all parts of a visual object in front of them. This also meant that to view the whole of a virtual structure at once, a considerable distance (in this  Table 2 Results of preliminary tests by 6 users without AR glasses Figure 11 Assembly time performance for the completion of 1st and 2nd layer of the shading system by users A-F case 6m) is required between the users/workers wearing the glasses and the structure; • Inaccuracy has been observed regarding overlaying between the virtual objects and the physical environment when live streaming or video recording is occurred. On the contrary, this has not been observed when the users/workers wear the AR glasses, in which case he/she is experiencing an accurate overlaying of the two worlds. It seems that this could be due to the difference between the position of the users/workers eyes and the position of the camera recording.

CONCLUSION
The aim of this work is to explore the possibilities offered by the use of AR technology in architecture with particular emphasis on construction process of complex morphologies or complex assembly tasks. In particular, this work investigates the use of AR Figure 12 Assembly time performance for user C and D. In the first case, the user failed to fulfill the given task on time, showing an increasing of assembly time performance. In the second case, the best performance among all users is observed. Also, the graph shows gradual balancing of assembly time with and without AR as medium to assist the assembly of a shading device, which consists of primary and secondary components and joint parts, assembled together in a specific sequence. Results are derived based on a series of experimental tests that involved users with architectural or other background, as well as skilled or unskilled users regarding construction and assembly competences. The results show that AR usage can improve the users' assembly time performance, especially during the first layers of assembly.
In conclusion, the results of this study show that the process of holographic assembly through the specific technology can be possible, enhancing at the same time the ability of users/workers to be involved in complex construction tasks. Also, it provides the opportunity for AR technology to be introduced in construction industry in order to eliminate construction time and cost. Nevertheless, AR technology, as new and promising direction of investigation in construction industry, requires further improvements, both in terms of technological readiness and applications. This work will continue towards the application of specific technology in other experimental studies, both towards fabrication and assembly of non-conventional systems, aiming to evaluate their construction time and cost performance.