Body Architectures Real time data visualization and responsive immersive environments

This project sets up a design framework that promotes augmenting the human body's interactions exploring methods for merging and blending the users of physical and virtual environments, through the design of wearable devices that are embedded with sensors and actuators. This allows for haptic and visual feedback through the use of data that reflects changes in the surrounding physical environment, and visualized in the immersive Virtual Reality (VR) environment. We consider the Body Architectures project to serve as mechanisms for augmenting the body in relation to the virtual architecture. These wearable devices serve to bring a hyper-awareness to our senses, as closed-loop cybernetic systems that utilize `digitized' biometric and environmental data through the use of 3D scanning technologies and cloud point models, virtual reality visualization, sensing technologies, and actuation. The design of Body Architectures relies on hybrid design, transdisciplinary collaborations, to explore new possibilities for wearable body architectures that evolve human-machine-environment interactions, and create hyper awareness of the temporal, atmospheric qualities that make up our experience of space, as `sensorial envelopes' (Lally 2014).


INTRODUCTION
Our bodies are complex systems that are dynamic and kinetic, allowing us to navigate through the world, experience our surroundings, and communicate with others through sensing and data feedback. In order to understand our surrounding environment, we rely on our senses; sight, hearing, touch, taste, and smell. However, the limitations of our human senses do not allow us to perceive all of the invisible, intangible, ephemeral, multi-layered environmental and biometric phenomena that coalesce and form dense networks that are important to take into account in multiple fields of design.
Typically, when individuals are immersed within Figure 1 User wearing the body architecture, wearable embedded with sensors and actuators, and Oculus Rift headset (inset), and a screenshot of the responsive, Virtual Reality (VR) environment.
a Virtual Reality (VR) environment, they are cut off visually from their surrounding physical environment. There is a distinct separation between the user's physical surroundings and the immersive virtual space that they are experiencing. Body Architectures is an experimental research project, with interactive wearable devices that venture to embed the physical world seamlessly with the virtual world by capitalizing on VR, ubiquitous computing, sensing and actuating technologies. (Figure 1). There has been a wide and well documented use of Virtual and Augmented Reality technologies during conceptual design from configuration of space, to visualization as well as collaborative decisionmaking (Milanovoc, J, et al. 2017) Existing studies focused either on the benefit of Virtual Reality for students to understand structure and construction, on the comparison of different VR systems (Gill, L., et al. 2013), on the implementation of a new working environment or framework for designers, on the evaluation of remote design collaboration, and also on the integration of VR in the curriculum for design courses at architecture schools (Eloy, S et al. 2015). Our approach to the use of VR technology integrates the use of data collection through sensors embed-ded in wearable devices. Our team designed and fabricated two unique wearable devices that capture different types of data and that augment the human body's interactions with real-world and virtualworld environments. This project sets up a framework that allows this data to be collected from our surrounding environment, visualized, and output as a virtual representation of the spaces in which we live and work. This data is sent and mapped in real time into a digital representation of the physical environment. Through VR, the user of the wearable device can experience visualize representations of the data being collected, which has an effect on the geometries and representation within the digital, immersive environment. The workflow serves as mechanisms that serve to bring a hyper-awareness to our senses, as closed-loop cybernetic systems that utilize 'digitized' biometric and environmental data through the use of 3D scanning and point cloud models, virtual reality visualization, sensing technologies, and actuation. The design of these body architectures relies on hybrid design, transdisciplinary collaborations, to explore new possibilities for wearable body architectures that evolve human machine-environment interactions.

METHODOLOGY
The wearable devices focus on two themes; the first wearable device captures environmental data, in this case carbon monoxide (CO) levels, which is related with issues of health and air quality concerns. The second wearable device is related with body proxemics, which is the study of space, how we use it, how it makes us feel more or less comfortable, and how we arrange objects and ourselves in relation to space. The data from the sensors embedded in the wearables, is used to actuate different behaviors in the virtual reality environment. The first wearable uses an air quality sensing device that is programmed to relay data levels, which serves as input parameters of the virtual model, resulting in changes of color and scale according to the air quality levels it is sensing. The second wearable integrates a proximity sensor, again, serving as input parameters for the virtual model, resulting in changes in opacity and transparency. This project aims to experiment with interactive wearable devices that venture to embed the physical world seamlessly with the virtual world by capitalizing on virtual reality, ubiquitous computing, and sensing and actuating technologies.

3D SCANNING
The project incorporates digitizing methods using two different types of 3D scanning technologies to scan and create 3D models of both the built environment (large scale) and the human body (small scale) Digitizing refers to the translation of analog, physical objects and built environments into digital, virtual models, typically through the use of scanning technologies. Scanners allow for the detection of many elements that makes up our physical environment, including objects, people, buildings, landscapes, etc., and convert this information into digital values which are used to generate 3D models.
In order to scan the large scale, expansive space, and the built environment, which includes the interior and exterior of the New Lab building, and the exterior of other structures in the Brooklyn Navy Yards, a LIDAR (light, detection, and ranging) scanner was used to create point cloud models. (Figure 2). Point cloud models are data sets of points (often millions of points) that are produced by 3D Laser Scanners to create 3D models of physical objects and forms, such as buildings, landscapes, and urban environments. 3D laser scanners create models that use points within an XYZ coordinate system to create the point cloud. The extreme accuracy and precision of the point cloud models support their typical application as surveying tool, to survey existing buildings and sites. The point cloud model can also serve as data to create Mesh and NURBS surface models, and used for other purposes, such as the production of drawings, renderings, and BIM models. In this project, the point cloud model was used to create the immersive environment and abstract visualization and representation of the immersive VR space. This was achieved in collaboration with the 3D Laser Scanning and Architectural Surveying firm, MYND Workshop, based in New York, NY.

Figure 2
Visualizations of the point cloud model of the New Lab, Brooklyn Navy Yards, Brooklyn, New York, created using a LIDAR scanner. One point perspective projection (top) and aerial perspective projection (bottom) views.
Mobile scanning devices use the camera and sensors of smart phones and tablets to create photogrammetric data, which is turned into a 3D digital model.
In this project, a Structure Sensor and iPad mini tablet were used to scan various human bodies, and generate mesh models, which are used to create a parametric model of the wearable device, that can be adjusted to custom fit individual bodies. (Figure 3).

WEARABLE DESIGN AND FABRICATION
The two wearable devices, or body architectures, designed for this project consist of silicone membranes tailored to fit the body of the user, and embedded with sensors, actuators, microcontrollers, and electronics. The data collected from the sensors is transmitted to effect the VR environment using wireless technology. The formation of the silicone membranes began with the scanned 3D model of the body. The wearable devices were modeled as sinuous NURBS surfaces, contoured to hug and fit the body. In this sense, the wearable devices are intended to serve as a second skin, an elastic, flesh-like surface that augments the body, through the extension of sensing capabilities. The sinuous surfaces were unrolled in the 3D model to create flat, planar surfaces, which were printed and used as templates to create molds for casting silicone. The elastic and pliability of the planar silicone sheets, allowed for the surface to be wrapped around the sinuous curves of the body, to match the curved surfaces in the 3D model. The wearable devices were formed with two types of silicone produced from the company SmoothOn; Ecoflex 30 and Dragon Skin. Each of these silicone materials exhibited various levels of elasticity, known as shore hardness, to create areas that stretch more than others within the surface. (Figure 4).

Figure 4
Silicone casting of the wearable device (top), embedded sensors, actuators, and electronics (bottom).
A computationally generated pattern was produced to create ornamentation; providing both aesthetic and utilitarian qualities. The pattern was created as a branching pattern, using an L system algorithm to create various versions of the design, ranging from low density to high density. The selected pattern was digitally fabricated and laser cut out of an acrylic panel, which was embedded in the silicone cast. Once removed from the cured cast, the pattern created an aesthetic effect similar to a tattoo, resulting in linework within a malleable, flesh-like surface, and a performative quality, by providing channels to mask and embed the electronics and wiring. (Figure 5).
This project aims to experiment with interactive wearable devices that venture to embed the physical world seamlessly with the virtual world by capitalizing on virtual reality, ubiquitous computing, and sensing and actuating technologies. The proposed wearables capture data that represents human's physical interaction with real-world environment and use the collected data to manipulate the virtual world.

Figure 5
The computationally generated pattern and digital fabrication sequence.

PHYSICAL COMPUTING
Our team created two wearable devices capable of sensing information around us. In particular, focusing in proximity and air quality. For this, we used an Ultrasonic Range Sensor and a MQ-5 Gas Sensor. Both sensors are commonly available and are simple to interface with Arduino compatible boards.
The Ultrasonic Range Sensor emits an ultrasonic 30°angle beam, as the ultrasonic signals collide against objects in the beam, they are reflected back to the sensor. Through this process, the sensor is capable of measuring distances from 3-400 cm with a maximal resolution of 3 mm. The MQ-5 Gas sensor, is able to detect the concentration of a multitude ses, including: H2, LPG, CH4, CO and Alcohol. We decided to focus only into CO due to its dangerous nature. Carbon Monoxide (CO) is an odorless and colorless gas that can be lethal in high concentrations, it is often the product of burning fuels, stoves, lanterns or fireplaces. As indicated by the US Department of Health & Human Services, Center for Disease Control and Prevention, "Each year, more than 400 Americans die from unintentional CO poisoning not linked to fires, more than 20,000 visit the emergency room, and more than 4,000 are hospitalized. " [1].
In addition to sensing environmental phenomena, the project incorporates actuators to output information as haptic and visual experiences. This occurs in both the real-world and virtual-world. The real-world actuation includes mini, vibrating disks and EL wire lighting that are embedded in the wearable. The virtual-world actuation includes changes in the geometry scale and colors within the 3D model experienced within the VR environment. (Figure 6).
Understanding the importance of movement and spatial freedom, we designed the wearables devices considering minimal intrusion into the person wearing them. This not only applies to the material selection, and form of the wearable devices, but to the embedded micro-controllers, sensors, actuators, and communication protocols. In order for the person to freely explore the space while transmitting live information back to the main computer, we selected the Arduino compatible board ESP32. This economic micro-controller has its own wifi chip, allowing it to connect to wireless networks to transmit and receive information. If desired, this board could directly connect to an IOT server to directly log data, further bridging our bodies to the architecture.
Among the abundant communication protocols out there, we chose OSC due to its ubiquity among computer programs and its ability to cross between platforms, systems, and programs. Open Sound Con-

Figure 6
Diagram of the body architecture, responsive workflow within each wearable device, illustrating the sensing and actuation feedback. trol (OSC) is a network communication protocol that allows the simple and reliable transfer of information over UDP/IP. Originally, it was created for transferring musical data, however, due to its flexibility, and universality, it is often used to send other types of data. In our case, the wearable devices can send live environmental information directly into the VR software and vice versa.The ESP32 microcontroller was programmed using the Arduino IDE, we took advantage of well documented libraries written by the Arduino open source community, in particular, Yotam Mann's "oscuino" for the OSC communication.

VIRTUAL REALITY AND RESPONSIVE DE-SIGN
Body Architectures aims to experiment with interactive wearable devices that venture to embed the physical world seamlessly with the virtual world by capitalizing on virtual reality, ubiquitous computing, sensing and actuating technologies. To achieve our goal as well as the desired look and feel of the piece, we combined various software and tools such as Processing, Rhino, Arduino, various sensors and the Oculus Rift. The challenge of the project was finding among available techniques, the ones that can work together and turn numbers into engaging and meaningful visual attributes. In order to successfully unite all of these technologies, Body Architectures was developed using Unreal Engine 4 by Epic Studios, a game engine with a complete suite of integrated tools for developers, designers, and artists to create, build games, simulations, and visualizations. Besides being a very powerful tool used by professionals and well-known studios, Unreal Engine is free to use for non-commercial projects and encourages such initiatives.
Within Unreal Engine, we've created customized blueprint scripts, a node-based visual language, to translate the data coming from the wearable devices into visual elements in the virtual environment. Addi-tionally, there were two important plugins for Unreal Engine that were implemented to make this possible: "Pointcloud Plugin" and "UE4-OSC plugin". Both were developed by members of the Unreal Engine community and free to use. The Pointcloud Plugin was used to help with importing, processing, and rendering of the large point cloud model. With this plugin, the cumbersome process of working with the point cloud file was more straightforward. The Pointcloud Plugin enables us to simply drag and drop the threedimensional scanned environments into Unreal Engine without having to write any C++ programming language to do so.The UE4-OSC plugin main purpose was to receive live data from Processing that was collected from the wearable devices via the same network. It utilizes an Open Sound Control protocol to communicate between computers and other multimedia devices that is optimized for modern networking technology. In this project, it binds the transmitted binaries data from the sensors to the rendered simulated model in the game engine. Here we assigned streaming parameters to materials and textures in order to create interactivity and responsive visual attributes.In designing the virtual environment and interactivity of Body Architectures we created two separate virtual scenes for each wearable devices in Unreal Engine. Both scenes consist of the environment point cloud model, a spherical, threedimensional structure that visually reflects the live data, constant values printed on the left side for debugging purposes, and an avatar that allows us to move around the space and see the designed environment in 1:1 scale in virtual reality.
The first scene explores the concept of proximity. The wearable device detects the presence of nearby objects or person without any physical contact. The closer the user is to a wall or a person, the more opaque and apparent the three-dimensional, spherical structure will become in the virtual scene. On the other hand, if the user has no obstruction in his/her path, the spherical structure will become more transparent. In this way, the user in virtual reality is able to understand their surroundings in a different way while exploring the rendered space. Besides the visual changes in opacity, this wearable acts as a secondary eye while the user is exploring the rendered virtual space and closed off to the physical world. It created a sensory illusion to the user's brain to navigate safely and explore freely the virtual world surpassing technology limitations, particularly since the VR headset is tethered and restrained to four meters by four meters physical boundaries. The second scene is designed to examine pollution with the focus on the level of carbon dioxide in the air. If the wearable device detects a high number of carbon dioxide in the air, the egg-like structure will turn pink. But if there is a low number of carbon monoxide detected, the model will gradually change color to blue. The color purple would mean the air is neutral. This wearable device represents a digital warning system to the body, an artificial man-made sensorial organ that is beyond the traditional five senses, sight, hearing, taste, smell, and touch. Because carbon dioxide is odorless and colorless, it is interesting to use it as a subject matter. By giving the collected CO values literal color representation, it peels away the barrier of human inability to recognize bad from good air and prompts us to be conscious of our own surroundings. (Figure 7).

CONCLUSION
This paper attempts to describe and demonstrate new methods for data collection and visualization through experimental processes in VR that utilize sensors and physical computing technologies. These methods are intended to provide a research and pedagogical framework that expands upon conventional VR applications while promoting a new paradigm for creating and teaching data feedback visualization between physical and virtual environments. The project is also intended to expand pedagogies in architectural visualization by integrating mobile data collection and designing wearable devices for the collection of data.   (left to right) user with wearable device illustrating air quality sensor, and the resulting real-time, color changes visualized in the VR environment.
The project described in this paper uses technologies and workflows that integrate sensors and data to create immersive experiences that incorporate the ephemeral, environmental phenomena, that define and contribute to architectural spaces and experiences, as methods for visualizing atmospheric qualities. These methods allow for visualizations of sensed atmospheric data, abstract experimentations, and responsive virtual environments. Through experimental and alternative methods in using sensors to capture data and utilize this data in visualizations, the role of VR experiences can continue to evolve, pro-ducing new modes of representation through the use of VR visualization and immersion, computational processes, and digital technologies. (Figures 8  and 9).