Molecular visualization and 2D interaction in virtual reality
Abstract
Efficient molecular visualization in a virtual reality environment, requires fast and high-quality rendering of geometrical objects, as well as intuitive interaction and information visualization. Furthermore, it should also take into consideration, the challenges posed by the limitations of virtual reality, which are caused by world space occlusion, and lack of traditional interaction and selection tools. This work aims to contribute in these areas, by providing ray-casting based rendering of impostor geometry objects inside Unity, a 2D navigation and selection technique without the need to use controller-based ray casting, and an occlusion-free world space interaction panel. In this work, we will render high-quality sphere and cylinder impostor objects through ray casting, to be used as the basis for the space-filling and ball-and-stick visualization of molecular models. These impostor objects are built inside the Unity game engine, compatible with Unity's pipeline, lighting and post-processing effects. We also implement a selection technique that allows the user to select and jump to neighboring objects without the need to ray cast into them, that facilitates selection in cluttered regions. With this technique we will visualize atom distances, bond angles, and torsion angles. Finally, we implement a world space interaction panel, that disregards occlusion to ease interaction. All of the above are concluded in a Unity powered application for molecular visualization and interaction in virtual reality.
Full text
Molecular Visualization and 2D Interaction in Virtual Reality Author: Konstantinos Kazatzis Supervisor: Pere-Pau Vázquez ViRVIG Group Universitat Politècnica de Catalunya A master’s thesis on Computer Graphics and Virtual Reality submitted to: Facultat d'Informàtica de Barcelona Universitat Politècnica de Catalunya 25/6/2020 .
2 Abstract Efficient molecular visualization in a virtual reality environment, requires fast and high-quality rendering of geometrical objects, as well as intuitive interaction and information visualization. Furthermore, it should also take into consideration, the challenges posed by the limitations of virtual reality, which are caused by world space occlusion, and lack of traditional interaction and selection tools. This work aims to contribute in these areas, by providing ray-casting based rendering of impostor geometry objects inside Unity, a 2D navigation and selection technique without the need to use controller-based ray casting, and an occlusion-free world space interaction panel. In this work, we will render high-quality sphere and cylinder impostor objects through ray casting, to be used as the basis for the space-filling and ball-and-stick visualization of molecular models. These impostor objects are built inside the Unity game engine, compatible with Unity’s pipeline, lighting and postprocessing effects. We also implement a selection technique that allows the user to select and jump to neighboring objects without the need to ray cast into them, that facilitates selection in cluttered regions. With this technique we will visualize atom distances, bond angles, and torsion angles. Finally, we implement a world space interaction panel, that disregards occlusion to ease interaction. All of the above are concluded in a Unity powered application for molecular visualization and interaction in virtual reality.
3 Table of Contents 1. Introduction ...................................................................................................................................7 1.1 Motivation and addressed problems .................................................................................................. 7 1.2 Previous work...................................................................................................................................... 8 1.3 Contributions ...................................................................................................................................... 8 1.4 COVID-19 ............................................................................................................................................. 9 2. Visualizing molecular models ........................................................................................................ 11 2.1 Space filling model ............................................................................................................................ 11 Covalent Radius ................................................................................................................................... 11 Van der Waals radius .......................................................................................................................... 12 2.2 Ball-and-stick model .......................................................................................................................... 12 2.3 Residues and chains .......................................................................................................................... 13 2.4 Model Information visualized ........................................................................................................... 14 3. Rendering .................................................................................................................................... 15 3.1 Sphere Impostors .............................................................................................................................. 15 Unlit Vertex Shader ............................................................................................................................. 16 Unlit Fragment Shader ........................................................................................................................ 17 Fragment shader output semantics .................................................................................................... 18 3.2 Cylinder Impostors ............................................................................................................................ 19 3.3 Deferred rendering ........................................................................................................................... 21 G-Buffer content and Physically based rendering .............................................................................. 22 3.4 Instancing and material property blocks .......................................................................................... 24 Adding instancing support .................................................................................................................. 24 Per-instance data ................................................................................................................................ 25 3.5 Forward Rendering pass ................................................................................................................... 26 3.5 Post processing stack ........................................................................................................................ 27 Ambient Occlusion .............................................................................................................................. 27 Silhouette highlighting ........................................................................................................................ 28 3.6 Transparency, UI and rendering order .............................................................................................. 30 Transparent impostors ........................................................................................................................ 30 User Interface ..................................................................................................................................... 31 Full rendering order ............................................................................................................................ 32 3.7 View frustum culling ......................................................................................................................... 32
4 3.8 Unity Virtual Reality Rendering ......................................................................................................... 33 Single-Pass Instanced Stereo XR shader support ................................................................................ 33 3.9 Unity prefabs ..................................................................................................................................... 34 3.10 Performance and Unity evaluation ................................................................................................. 34 Performance ....................................................................................................................................... 34 Unity advantages and disadvantages ................................................................................................. 36 4. Interaction ................................................................................................................................... 39 4.1 3D selection....................................................................................................................................... 39 4.2 Information panel ............................................................................................................................. 40 4.3 Visualizing residues and chains ......................................................................................................... 41 4.4 2D atom selection ............................................................................................................................. 42 Mapping to 2D .................................................................................................................................... 44 Arrow direction indicators .................................................................................................................. 46 Color coding directions ....................................................................................................................... 46 4.5 Atom distances.................................................................................................................................. 47 4.6 Bond angles ....................................................................................................................................... 48 4.7 Torsion angles ................................................................................................................................... 49 Selecting atoms ................................................................................................................................... 50 Calculating the dihedral angle ............................................................................................................ 50 Visualizing the dihedral angle ............................................................................................................. 50 4.8 Translating and rotating .................................................................................................................... 51 4.9 World space widget .......................................................................................................................... 52 Modes of interaction and visualization parameters ........................................................................... 52 Virtual buttons implementation ......................................................................................................... 53 4.10 Model and World UI position .......................................................................................................... 55 4.11 HTC Vive Controller button mapping .............................................................................................. 56 5. Usability experiments ................................................................................................................... 59 6. Conclusions and further work ....................................................................................................... 61 Contributions .......................................................................................................................................... 61 Further work ........................................................................................................................................... 61 Expand in the molecular visualization area ........................................................................................ 61 Expand in the interaction area ............................................................................................................ 61 Expand in the molecular rendering area............................................................................................. 62
5 Bibliography .................................................................................................................................... 63 Appendix ......................................................................................................................................... 65 A: Ray-Sphere intersection ..................................................................................................................... 65 B: Ray-Cylinder intersection .................................................................................................................... 65 C: Sphere impostor Unity deferred and forward shader ........................................................................ 66 D: Phong shading functions .................................................................................................................... 72 E: Border highlighting post processing effect Unity shader ................................................................... 74 F: Stereo instancing Unity built-in matrix variables ................................................................................ 75 Table of Figures Figure 1: 1tes molecular model with CPK coloring, covalent radius and no post processing .................... 12 Figure 2: 1tes molecular model with CPK coloring, van der waals radius and no post processing ............ 12 Figure 3: 4f0h molecular model, space filling model .................................................................................. 13 Figure 4: 4f0h molecular model, ball-and-stick .......................................................................................... 13 Figure 5: 1tes molecular model, space filling (left) and ball-and-stick (right), depth cue comparison ...... 13 Figure 6: Visualization of a residue (highlighted) and a chain (colored green) .......................................... 14 Figure 7: Visualization of atom distances, bond angles, and torsion angles .............................................. 14 Figure 8: Cube subdivided spheres ............................................................................................................. 15 Figure 9: Sphere impostor in Unity ............................................................................................................. 16 Figure 10: Real(blue) and impostor spheres(pink), depth testing .............................................................. 19 Figure 11: A cube geometry (orange) that covers an impostor cylinder .................................................... 19 Figure 12: Cylinder representation ............................................................................................................. 20 Figure 13: Impostor spheres and cylinders, no post processing................................................................. 21 Figure 14: Impostor sphere shader with default shadows ......................................................................... 22 Figure 15: Physically based rendering impostor sphere(pink) and Unity sphere (blue), from left to right: glossy, metallic, our look ............................................................................................................................. 23 Figure 16: Physically based rendering impostor cylinder(pink) and Unity cylinder(blue), from left to right, glossy, metallic, our look ............................................................................................................................. 24 Figure 17: Left G-Buffer rendering without instancing, Right G-Buffer rendering with instancing ............ 26 Figure 18: Fog post processing effect (left), depth of field effect (right) ................................................... 27 Figure 19: Ambient Occlusion with impostor (pink) and Unity spheres (blue) .......................................... 27 Figure 20: Ball-and-stick visualization, Multi-Scale Volumetric Obscurance AO ........................................ 28 Figure 21: Ambient Occlusion parameters ................................................................................................. 28 Figure 22: Highlighted group of atoms ....................................................................................................... 29 Figure 23: Alpha channel of G-Buffer Render Target 2 ............................................................................... 29 Figure 24: Post processing stack ................................................................................................................. 30 Figure 25: Spheres impostors, view frustrum culling problem ................................................................... 32 Figure 26: Unity profiler for 1S5L ................................................................................................................ 36 Figure 27: Debugging rendering order using the Frame Debugger ............................................................ 36 Figure 28: Atom distance and bond angle visualization ............................................................................. 39
6 Figure 29: Atom distance in space filling model, and torsion angle visualization ...................................... 39 Figure 30: Information panel ...................................................................................................................... 40 Figure 31: Calculating horizontal position of the information panel .......................................................... 40 Figure 32: Residue visualization in ball-and-stick model (left), and space-filling model (right) ................. 41 Figure 33: Atom chain visualization in ball-and-stick model (left) and space-filling model (right) ............ 42 Figure 34: 2D navigation within close atoms .............................................................................................. 42 Figure 35: Atom selection with and without rendering a sphere to indicate the region of selection ....... 43 Figure 36: Atom navigation using arrows in ball-and-stick model ............................................................. 44 Figure 37: Atom navigation using color coding in space filling model ....................................................... 44 Figure 38: Top down look of the new coordinate system .......................................................................... 44 Figure 39: View of the new coordinate system from the perspective of the camera ................................ 45 Figure 40: 2D direction based on the XY angle of the atom in the new coordinate system ...................... 45 Figure 41: Arrow indication in the space filling model ............................................................................... 46 Figure 42: Comparison between the color circle and arrows in the ball-and-stick model ......................... 47 Figure 43: Atom distance visualization, ball-and-stick(left), space filling with covalent radius (right) ...... 47 Figure 44: Visualization of angle between bonds(left), vector calculation(right) ...................................... 48 Figure 45: Torsion angle visualization (up), selected atoms order (down) ................................................ 49 Figure 46: Torsion angle visualization, aligned view ................................................................................... 49 Figure 47: Torsion plane construction ........................................................................................................ 51 Figure 48: Torsion angle arc rendering ....................................................................................................... 51 Figure 49: World space interaction panel ................................................................................................... 52 Figure 50: The state machine for the atom visualization virtual button .................................................... 54 Figure 51: The state machine for the atom exploration mode virtual button ........................................... 54 Figure 52: The state machine of the virtual button for increasing the radius of the navigation sphere ... 54 Figure 53: Camera and floor position ......................................................................................................... 55 Figure 54: World panel position ................................................................................................................. 56 Figure 55: Possible mapping of interaction operation onto the right controller ....................................... 57 Figure 56: Possible mapping of model translation operation onto the left controller .............................. 57 List of Abbreviations VR Virtual Reality HMD Head Mounted Display PDB Protein Data Bank GPU Graphics Processing Unit CPU Central Processing Unit AO Ambient Occlusion XR Unity Virtual Reality modes
7 1. Introduction Molecular visualization is the discipline that uses visualization techniques to represent molecular models, and facilitate interaction and information extraction from the elements that construct the model, for example, atoms, bonds and secondary structures, in order to help biologists get a deeper understanding of these elements. To represent a molecular model, one can choose one of several visualization methods, like the space-filling model, or the ball-and-stick model, that allow atom and bond level interaction, or one can provide a more abstract representation by re-constructing the surface of the secondary structures, such as alpha helixes and beta sheets. Furthermore, modern 3D visualization powered by computer graphics technology, can provide a more faithful representation when paired with realistic and real-time rendering, to ease interaction. Towards increasing the fidelity of the visualization, Virtual Reality goes one step further, by providing stereoscopic 3D rendering and high field of view displays, that allow the exploration of much larger models. However, the current technology of VR devices, and more specifically head mounted displays, usually offer lower resolution displays compared to a common monitor display, which can be the cause of several problems when it comes to interacting and visualizing information. Furthermore, the user lacks the traditional interaction tools, the keyboard and mouse, which makes the interaction with the scene more complex. 1.1 Motivation and addressed problems One of the drawbacks of molecular visualization in VR, is that VR requires fast and realistic rendering. Common VR devices are Head Mounted Displays which consist of one screen per eye with usually lower resolution compared to common monitor displays, but with much higher refresh rate. Refresh rate is a very crucial parameter in VR, since a 3D visualization with low refresh rate can be the cause of many problems, for example, motion sickness. The standard for VR rendering is around 90 frames per second refresh rate per eye, whereas a common monitor display can get away with less than 60 frames per second. Another drawback of molecular visualization in VR, is that the user does not have the traditional interaction tools that is accustomed to use, the keyboard and mouse, but rather uses custom hand-held controller devices unique to each HMD brand, that usually work with ray casting or hand-extension techniques. Such techniques can be very inefficient when it comes to selecting small objects in a complex scene. Furthermore, a common problem in VR environments, is information visualization, and world interaction. In a desktop monitor display, it is relatively straightforward to create a screen-space panel, with screen space buttons and text elements, to store information and change parameters of the visualization using the mouse. However, VR scenes do not have screen space elements, and the hand-held controllers used can only have up to a certain number of buttons. As a result, VR scenes tend to have world space panels, with virtual buttons and text, placed within the scene next to the model. One of the problems with world space information and interaction panels, is occlusion. A world space element can be easily hidden, depending on the position of the model and the user. Furthermore, the user must also be able to interact with the panel at any time, in order to change parameters of the visualization. In VR, this problem is amplified, due to lower resolution displays.
8 1.2 Previous work There has already been significant research in the area of 3D selection inside a Virtual Reality environment. Several proposals have been made towards enhancing ray casting to ease the selection of small and occluded objects, like the proposal to replace the selection ray with a selection cone with a variable apex angle [1], or to incorporate a two handed technique to define the pointing direction [2]. Other researchers have proposed eye-rooted techniques, meaning, ray casting is enhanced by using a ray that originates from the user’s position and viewing angle. Such techniques use a combination of rays, to facilitate selection and visual feedback about the selected object [3]. Such techniques also combine a variety of methods, for example, the cone selection with variable apex angle. Most recently, there has been the work of F. Argelaguet and C. Andujar [4], who used a combination of eye-rooted and hand-rooted techniques, with an eye-rooted ray for selection, and a hand-rooted ray for visual feedback, to facilitate the selection of occluded objects, and solve the occlusion mismatch between the set of visible objects between the eye and the hand. While the above techniques significantly enhance simple ray casting, they are not suitable for such cluttered scenes as the ones with large molecules. As a result, we have investigated new interaction techniques to facilitate accurate selection in those environments that use a pointer for the initial selection, and the navigation pad for fine selection. 1.3 Contributions The contributions of this work come in three areas. To render the scene, we are going to use Unity, which is a very common end-to-end tool, mainly used for creating video games. However, Unity provides excellent support for all VR devices out of the box, has a wide range of tools and assets offered to ease development, and already implements the “heavy lifting” of a rendering pipeline, while at the same time offering the ability for custom shaders, lightning and post processing effects. Towards that, we are going to implement Unity shaders for realistic and fast rendering of geometry objects, impostor spheres and impostor cylinders, fully integrated within Unity’s lightning calculations and factory post processing effects, ready to be instantiated in a Unity scene. This part, as well as all other parts of the rendering pipeline, are covered in the chapter Rendering. Next, we are going to implement a novel selection technique, that allows the user to move around the scene, and select neighboring objects, without the need to continuously use a 3D selection technique, but rather utilizes 2D input. Then, we are going to illustrate the effectiveness of this technique by visualizing distances between atoms, angles formed between bonds, and torsion angles formed between atoms. This part is covered in the chapter Interaction. The last contribution is a simple approach to a world space panel, that facilitates interaction, even when the panel is occluded behind elements in the scene. This world space panel will use virtual buttons with images rather than text, and will allow the user to bring it forward for interaction, ignoring everything else in the world. This part is covered in the section Interaction: World space widget. In the last chapter of the work, Usability experiments, we will explore possible experiments that involve real users, that could be carried out in order to validate the usability of the system.
9 1.4 COVID-19 Given the unique circumstances created during the 2019-2 semester, this last chapter, as well as all other sub-chapters that involve equipment that was not available, for example, the HTC Vive VR headset, or a VR capable computer, are focused mainly on how we could plan to use such equipment and how we could design and conduct such experiments. Everything is implemented with VR in mind, from rendering to interaction, by adapting the solutions as realistically as possible to a VR environment.
16 Figure 9: Sphere impostor in Unity To achieve this effect in Unity, we are going to define a Vertex/Fragment shader pipeline. Unity calls these shaders, Unlit shaders, in order to differentiate them with Surface shaders. Surface shaders, as the name suggests, are used to perform basic coloring algorithms on the surface of an object, without worrying too much about vertices and triangles and lightning. Since we want to actually change the geometry of the rendered fragments, we have to use Unlit shaders. Then, we are going to define a Unity Material and set that this material is implemented by our shader, and attach the material on a quad in the scene. The uniform data of the shader or the Properties of the shader as Unity calls them, like the radius and the color, will be exposed as material attributes. Unlit Vertex Shader We first generate a Unity Quad object. A unity quad is a quad plane that spans from (-0.5, -0.5, 0) to (0.5, 0.5, 0), facing down the -z axis. By passing this geometry through the vertex shader, we can use the vertices to create our camera-oriented quad in view space. struct appdata { /* Object space position */ float4 vertex : POSITION; }; struct v2f { /* clip space position */ float4 pos : SV_POSITION; /* view space position */ float4 view_pos : TEXCOORD0; }; v2f vert(appdata input) { v2f output; output.view_pos = mul(UNITY_MATRIX_MV, float4(0.0, 0.0, 0.0, 1.0)) + BOX_CORRECTION * float4(input.vertex.x, input.vertex.y, 0.0, 0.0) * 2.0f * float4(radius, radius, 1.0, 1.0); output.pos = mul(UNITY_MATRIX_P, output.view_pos); return output; } Unity provides three possible inputs to a vertex shader, appdata_base, appdata_tan, appdata_full [7]. Since only the position of the object vertices is needed, we are going to use the basic input struct.
17 The output of the vertex shader can be any struct that contains an attribute of type SV_POSITION. Unity is going to use this attribute as the clip space coordinates. We are also going to use an extra attribute, the view space position, since we are going to need it in the fragment shader. To calculate the cameraoriented, view-space vertex positions, we are going to use the following formula: 𝑣𝑖𝑒𝑤𝑆𝑝𝑎𝑐𝑒𝑃𝑜𝑠=[0,0,0,1]∗𝑀∗𝑉+[𝑥,𝑦,0,0]∗2∗(𝑟,𝑟,1,1) where M = Model matrix, V = View matrix, and x, y are the input object-space vertex coordinates. Since we render a quad from -0.5 to 0.5 in the xy plane, the above formula will multiply by 2 to make the vertices span from -1 to 1, multiply the result of this with the sphere radius, and then add this vertex to the view space position of the model. That way we generate a camera-oriented quad with size equal to the size of the sphere. We are also going to multiply with a BOX_CORRECTION factor. This factor is used to compensate for the projection of the quad to the screen. Unlit Fragment Shader fragment_output frag(v2f input) : COLOR { /* Compute real fragment world position and normal */ float3 normal_world, position_world; ImpostorSphere(mul(UNITY_MATRIX_I_V, input.view_pos), radius, position_world, normal_world); /* Calculate depth */ float4 clip = mul(UNITY_MATRIX_VP, float4(position_world, 1.0f)); float z_value = clip.z / clip.w; /* Set depth, and calculate color */ /* .... */ } The function ImpostorSphere, will take as input the fragment’s world position and the radius of the sphere, and will calculate the position and the normal that this fragment should have. To do that, ImpostorSphere will cast a ray from the camera towards the fragment, and find the closest intersection point with the perfect geometrical sphere that we want to render. If such point does not exist, it will clip the fragment using Unity’s clip(float) method. To represent a sphere we need two attributes, a center and a radius. The ImpostorSphere function will assume that the sphere is placed at (0,0,0) in object space. That way the position is encoded inside the Model matrix which lowers the amount of shader properties. The Sphere-Ray intersection algorithm is a basic sphere-line intersection algorithm that solves the seconddegree equation. You can find the full code for this function in Appendix A: Ray-Sphere intersection. By calculating the closest intersection point to the sphere, we essentially perform back-face culling for the sphere. If we want to do front-face culling, we can use the farthest intersection of the ray with the geometrical sphere. After we calculate the world space position that we want this fragment to have, we transform it to clip space, divide with the w coordinate, and calculate the new depth of that fragment. We are going to give this depth value to Unity to perform depth testing.
18 After we calculate the geometry of the fragment and set the depth, we will proceed to calculate color and set the fragment’s shader output. Fragment shader output semantics In order for Unity to accurately perform depth testing, we have to set the new fragment depth, which is calculated from the clip space position of the intersection point. Unity allows the fragment shader to return multiple values, either to be used by us, or to be used by Unity, and calls these outputs Fragment shader output semantics [8]. One semantic that we can define, is multiple render targets SV_TargetN. This tool is useful when we want to render to multiple places for example textures, in order to perform custom post processing effects. Another output semantic is SV_DEPTH, which allows us to set the depth for that fragment. We are going to use both: struct fragment_output { half4 diffuse : SV_Target0; half4 specular : SV_Target1; half4 normal_world : SV_Target2; half4 emission : SV_Target3; }; fragment_output frag(v2f input, out float outDepth : SV_Depth) : COLOR { /* Calculate impostor*/ /* ... */ /* Calculate depth */ float4 clip = mul(UNITY_MATRIX_VP, float4(position_world, 1.0f)); float z_value = clip.z / clip.w; outDepth = z_value; /* ... */ /* Set output parameters */ fragment_output o; /* Set values to o */ /* ... */ return o; } All we have to do is set the calculated z_value to the SV_DEPTH variable, and then Unity will use it with whatever depth encoding formula they have to perform depth testing. We can also see the Multi-Target Rendering output that the fragment shader has, which will be explained in more depth in the Deferred Rendering section. In the following image, we can see impostor spheres in pink, and unity geometry spheres in blue:
19 Figure 10: Real(blue) and impostor spheres(pink), depth testing 3.2 Cylinder Impostors Now that we have seen how sphere impostors are rendered, we can generalize this approach to basically render any kind of geometrical structure, like cylinders, cones, etc. The two things that we have to change is, the initial geometry rendered that defines the fragments on the screen that will render the geometrical structure, and the ray casting algorithm that calculates the actual fragment position. For sphere impostors, rendering a camera-oriented quad was enough, since a sphere looks the same, geometrically, from every angle. For a cylinder impostor however, this is not true. As a result, we are going to render a simple cube with size and position that tightly covers the desired cylinder object. We are going to use a single base mesh, an object-space cube that ‘sits’ on top of the 𝑥𝑧 plane, and has size 1 across every dimension. This is a cube that tightly covers a cylinder that sits on top of the 𝑥𝑧 plane, has radius 0.5, and height 1 with direction along the y axis. We can use this single mesh object, along with different translation, scaling and rotation matrices to place the cylinder anywhere in the scene. That way, the Unlit Vertex Shader is very simple, and just transforms the vertices to clip space. Figure 11: A cube geometry (orange) that covers an impostor cylinder The Unlit Fragment Shader has to calculate the actual fragment position by performing ray casting from the camera to the fragment world position, against the desired geometrical cylinder. In order to represent a geometrical cylinder, we define the following properties: 𝐶 ∈𝑅3, 𝑒 ∈𝑅3,||𝑒 ||= 1, 𝑟>0, ℎ >0, where 𝐶 is the center of the cylinder, 𝑒 defines the direction, and 𝑟 and ℎ are the width and the height. To lower the amount of properties sent to the shader, the shader assumes that 𝐶 is (0,0,0)
20 in object-space, and 𝑒 is the object-space y axis transformed to world space. We will also use the scaling that the object has in the y direction as the height of the cylinder. That way, the only additional property sent to the shader is the radius, since everything else is encoded within the Model matrix. Like in the sphere impostors, a correction factor must be applied in the size of cube rendered, to compensate for the projection onto the screen. This correction is applied as a scale to the x and z components of the cube Figure 12: Cylinder representation The Cylinder-Ray intersection algorithm implemented calculates the intersection point assuming an infinite cylinder, and does the following: Assume that the ray is defined by the equation: 𝑃+𝜆∗𝑣 , and the cylinder axis is defined by the equation: 𝐶+𝜇∗ 𝑒 . First, we will find the point 𝐴 along the ray that is closest to the cylinder axis 𝑒 . To do that, assume the direction vector defined by, a random point on the ray 𝐴, and a random point on the axis 𝑀: 𝐴−𝑀 , and we will solve the system: (𝐴−𝑀 )∗𝑣 =0,(𝐴−𝑀 )∗𝑒 =0. This system has a solution if the ray direction and the cylinder direction are not parallel. By solving the system, we get the value 𝜆, that gives us the point 𝛢. Then, we will calculate the distance 𝑑 of this point to the cylinder axis. If this distance 𝑑 is greater than the radius, then there is no intersection. Otherwise, there is. We are going to calculate the point B, by using the PB distance. The value 𝜆 that defines point A, is also the distance between A and P. If the plane defined by 𝐴,𝑃,𝑀 was perpendicular to the cylinder axis, then 𝐴𝐵 distance can be found by using the Pythagorean theorem directly. Since that is not generally the case, we have to scale the 𝐴𝐵 value with the angle inclination. Thus: |𝑃𝐵|= 𝜆′=|𝐴𝑃|−|𝐴𝐵|= 𝜆− √𝑟2−𝑑2 √1−(𝑒 ∗𝑣 )2 This 𝜆′ will give the intersection point 𝐵, 𝐵 =𝑃+𝜆′∗𝑣 , for the infinite cylinder. After this point is calculated, we clip it based on the cylinder height, and calculate the normal. You can find the full code for this calculation in Appendix B: Ray-Cylinder intersection.
21 In general, we would also want to calculate the intersection points on the top and bottom of the cylinder, assuming planes perpendicular to the cylinder direction. However, since we are going to use this cylinder as a bond between atoms, this is not needed. Figure 13: Impostor spheres and cylinders, no post processing 3.3 Deferred rendering Deferred rendering or deferred shading, is a technique in which while passing through the geometry of the scene, we don’t draw directly on the screen, but we rather draw/store all the necessary information onto texture buffers. These texture buffers are called the G-Buffer. The G-Buffer will hold all the data that we need in order to perform the lightning calculations, for example, position, normal, color, etc., for each fragment and in the final pass, the color for each fragment will be calculated by drawing a simple quad that covers the screen. Deferred rendering is mainly used when we have many lights on the scene, since we can reduce the contribution of each light to a small area of screen fragments, and not for all rendered objects. Another advantage of deferred rendering, is that it allows us to perform more complicated post processing effects, that with forward rendering we couldn’t. In our case, by using the deferred rendering path, we also have the advantage that we can use Unity’s Physically based rendering. If we were to use forward rendering, we would have to perform lighting calculations ourselves, which can be a significant downgrade from Unity’s lightning pipeline. Furthermore, if we were to use forward rendering, we would have to implement all the post processing effects ourselves, since the impostor geometry is calculated in the fragment shader. That means that if we were to use Unity’s post processing pipeline, all the effects would be calculated and applied in the scene as if we are drawing quads and cubes for spheres and cylinders. The main disadvantage of deferred rendering is that it’s not possible to handle semi-transparent objects, and transparency needs to be handled with a different rendering pipeline. But since our main goal does not necessarily include any transparency, this does not have any impact on our scenes. This is not such a big drawback, since Unity allows both deferred and forward rendering in the same scene. Another disadvantage is that the G-Buffer and its size is usually a bottleneck in performance. Many graphics cards have a limited amount of render targets available, and limited memory and memory bandwidth available. However, most graphics cards have no problem running deferred rendering pipelines. Another disadvantage that comes from the limited memory of the G-Buffer is that we can’t have multiple materials
22 on an object. The above disadvantages are not significant enough to force us to use forward rendering for our scene. Furthermore, as mentioned in the previous paragraph, Unity allows different rendering queues based on the type of geometry rendered, and when the rendered geometry is set to transparent, then the forward rendering queue is used by default. As a result, we are also going to define forward rendering shaders that implement a simple Phong Shading model that we are going to use to render transparent spheres or cylinders. G-Buffer content and Physically based rendering To define that a shader is to be used in a deferred rendering pass, we apply the tag: Tags { "LightMode" = "Deferred" } Shader tags is a way for a shader to tell Unity how to use the shader, in terms of which rendering queue to use [9], and which lighting pass to use [10]. When a shader that unity needs does not exist, Unity is going to use the fallback shader. For example, if we were to enable shadows in our scene, this would be the result, since we haven’t defined a ShadowCaster lightning pass, and Unity will use the default one: Figure 14: Impostor sphere shader with default shadows Implementing a ShadowCaster pass for a sphere or cylinder impostor is straightforward, we apply the same ray casting algorithm, but from the point of view of the light, and the only output is the depth. Let’s examine now the contents of the G-Buffer [11]: • Render target 0: ARGB32 format: Diffuse color (RGB), occlusion (A) • Render target 1: ARGB32 format: Specular color (RGB), roughness (A) • Render target 2: ARGB2101010 format: World space normal (RGB), unused(A) • Render target 3: ARGB2101010 (non-HDR) or ARGBHalf (HDR) format: Emission + lightning + lightmaps + reflection probes buffer • Depth and stencil buffer • Render target 4: ARGB32 format: shadowmask or distance shadowmask if enabled We have already seen the in the previous chapter that we are going to use the first four textures (depth and stencil buffer handled by Unity): struct fragment_output { half4 diffuse : SV_Target0; half4 specular : SV_Target1; half4 normal_world : SV_Target2; half4 emission : SV_Target3;
23 }; In the first texture, in the RGB components we store the diffuse color value of the object. In the A channel we store the occlusion factor. Since we are not going to use this feature, we set its value to 1. In the second render texture we store the specular color (RGB), and in the A channel we store the glossiness of the object. The third render texture uses 10 bits for the RGB channels, and 2 bits for the unused A channel. We store in the RGB channels the world space normal encoded as a regular normal map. In the alpha channel of this texture we still store the border highlighting information, which we are going to use for object highlighting as a Post Processing effect. In the fourth texture we will only store the ambient component of the object: fragment_output frag(v2f input, out float outDepth : SV_Depth) : COLOR { /* Calculate impostor geometry */ /* Set depth */ /* ... */ /* Get albedo and is_highlighted values for this object */ /* ... */ /* Calculate diffuse and specular component from using Unity's Physically based rendering pipeline */ half3 specular; half specularMonochrome; half3 diffuseColor = DiffuseAndSpecularFromMetallic(albedo, metallic, specular, specularMonochrome); /* Set output parameters */ fragment_output o; o.diffuse = float4(diffuseColor, 1); o.specular = half4(specular, gloss); o.normal_world.xyz = normal_world * 0.5f + 0.5f; o.normal_world.w = is_highlighted; o.emission.xyz = ambient_factor * diffuseColor; return o; } We don’t store the world space position of the fragment in the G-Buffer, since Unity will calculate that using the depth of the fragment. By using Unity’s function DiffuseAndSpecularFromMetallic() we can calculate the diffuse and specular components from the metallic value, and then set them in the G-Buffer textures. This will allow Unity to calculate the final color using physically based rendering in the final pass. Figure 15: Physically based rendering impostor sphere(pink) and Unity sphere (blue), from left to right: glossy, metallic, our look
24 Figure 16: Physically based rendering impostor cylinder(pink) and Unity cylinder(blue), from left to right, glossy, metallic, our look We opted for the look in the outmost right image in Figure 15, 16, which is zero metallic component and zero glossiness. 3.4 Instancing and material property blocks Drawing thousands of objects in a scene can have a significant impact in performance, since the CPU must circle through all of them and draw them one by one. Unity has two ways to reduce this cost called batching and instancing. Batching, which happens automatically, either static or dynamic, groups together the meshes of all small enough objects into one huge mesh, and then draws this mesh. Static batching happens automatically to all objects marked as static, if they are small enough [12]. Dynamic batching has to be enabled. However, batching does not help us with impostors, since we actually use the vertices of each individual quad in the vertex shader to generate the camera-oriented quad. As a result, in order to reduce the number of calls we can only use instancing. Instancing is a rendering technique that allows us to draw the same exact mesh with different parameters, using a single draw call [13]. In Unity, two objects can be rendered with a single draw call, if and only if, they have the same exact mesh, material, and material attributes as well. However, material attributes can be different in each instance, without breaking instancing, with a technique that Unity calls material property blocks, which we are going to use to set different properties, like colors, radiuses, etc. to the instantiated objects. Since both impostor spheres and impostor cylinders use the same base mesh, we can batch all impostor draw calls through instancing. Adding instancing support To add instancing support to the Impostor shader, we have to add a couple of things. We have to add a new preprocessor directive to the shader (#pragma multi_compile_instancing) to inform Unity that this shader supports instancing. Next, we have to add the instance id to the shader. This instance id will allow us to get per instance data. struct appdata { /* Instance ID */ UNITY_VERTEX_INPUT_INSTANCE_ID /* Object space position */ float4 vertex : POSITION; }; struct v2f { UNITY_VERTEX_INPUT_INSTANCE_ID /* clip space position */ float4 pos : SV_POSITION; /* view space position */ float4 view_pos : TEXCOORD0;
25 }; Currently, only the model matrix is different for each instantiated object, but we will add more detailed properties later. We also have to change the vertex shader to be aware of their instance id and set up the fragment shader instance id. v2f vert(appdata input) { v2f output; UNITY_SETUP_INSTANCE_ID(input); UNITY_TRANSFER_INSTANCE_ID(input, output); /* Calculate camera-oriented quad */ /* ... */ return output; } fragment_output frag(v2f input, out float outDepth : SV_Depth) : COLOR { /* Set up instance id */ UNITY_SETUP_INSTANCE_ID(input); /* Rest of fragment shader code */ /* ... */ } The impostor shaders now support instancing. We only have to enable it through the Unity Editor. However, all instances must have the same attributes: radius, color, etc. To change that we have to add support for material property blocks. Per-instance data If we were to change the properties of a material (for example color) attached to an object after instantiating the object, then Unity would duplicate the material for that object, and as a result, break batching since the new object has now a different material. Furthermore, if we were to change the shared material properties for that object, then all instances would see the same change. As a result, we have to use something that Unity calls Material Property Block. Material property blocks are small structs that contain data, which we can set to the mesh renderer attached to the object, to override the set material properties. For sphere impostors, we only want to be able to change the color of spheres, whether or not a sphere is highlighted, and the radius of the sphere. As a result, we want 5 floating point parameters. Since material property blocks only support parameters of type Color(float, float, float, float), we are going to use one parameter to store the color of the sphere to the RGB channels and the is_highlighted field to the alpha channel, and another parameter to store the radius, the ambient factor, and the metallic and glossiness factors, even though we are not going to use them. For cylinder impostors we want to be able to change the color, since the color could indicate a different type of bond, the radius and the height of the cylinders, and also the highlighting information. Since the
32 The scene will have two user interfaces, the first one is an always visible information panel that is rendered on the top left of the camera view. This panel is the last thing rendered with the 3010 queue. The second user interface, is an actual world space UI, at a specific position in the world, that can be brought forward and rendered on top of the geometry, with the push of a button. When rendered normally in the world, this UI will use the 3000 queue, and when rendered on top of the geometry, it will use the 3004 queue. Full rendering order In order to compose the final scene, we only have to make sure that the post processing effects, like Ambient Occlusion and border highlighting are drawn before the forward rendering step, and our Transparent queue is drawn last. As a result, the full rendering order is the following: 1. Fill the GBuffer with the main scene geometry 2. Calculate Ambient Occlusion 3. Unity lightning calculations 4. Apply the border highlighting effect 5. Render the transparent queue 3.7 View frustum culling View frustum culling is a CPU technique that culls geometry that falls outside the virtual camera’s view frustum, since if rendered, it will not end up on the screen anyway. However, this view frustrum culling algorithm is not going to work properly for sphere impostors. As far as Unity is concerned, we have no spheres in the scene, but quads, and more specifically, quads that face a certain direction. Figure 25: Spheres impostors, view frustrum culling problem When looking at a quad down the z axis, view frustum calculation will be performed correctly, since the sphere is “contained” within the quad geometry. However, when looking at the sphere from the side of the quad, the orientation of the quad only changes in the vertex shader. View frustum culling will be performed with the original z facing quad. This produces an ugly effect of spheres being culled on the edges of the camera’s frustum while still being visible. For example, in Figure 25, if the camera rotates just a bit to the right, the sphere will be culled. This is not a problem for cylinder impostors, since the cube geometry tightly covers all the cylinder geometry generated in the fragment shader. To avoid this effect on spheres, we will perform view frustum culling using a slightly higher field of view, and render the geometry with the default field of view. This will surely add some extra geometry being rendered, but the alternative of rotating all the quads in the CPU is strictly prohibitive.
33 To achieve the above effect, Unity allows us to attach a script to the virtual camera and declare the OnPreCull() function. void Start() { /* Grab components */ cam = GetComponent<Camera>(); cam_old_fov = cam.fieldOfView; } private void OnPreCull() { /* Change the field of view used for culling */ cam.fieldOfView = cam_old_fov + fov_addition; } private void OnPreRender() { /* Set back old field of view for rendering */ cam.fieldOfView = cam_old_fov; } 3.8 Unity Virtual Reality Rendering Rendering for Virtual Reality devices, is supported in Unity with two modes, multi pass rendering, and single pass instanced stereo rendering. Multi pass rendering works by passing and drawing the scene twice, one for each eye, and it is supported by default for all Unity scenes and custom shaders. However, since the objects in the scene are drawn twice, the GPU and CPU times are expected to be at least doubled in multi pass XR rendering. The other technique is, single pass instanced stereo rendering which is a much faster VR rendering technique, since the scene is passed and drawn one time. This technique takes advantage of two features. Instancing, which allows the renderer to draw the same mesh multiple times, and stereo rendering, which allows the renderer to pass and cull the scene one time, and render it into a single packed texture for both eyes. However, not all Unity shaders automatically support this technique and since it’s a more complex pipeline, not all graphics cards support it. Single-Pass Instanced Stereo XR shader support To add support to Unity custom shaders, we have to make sure that unlit shaders, post processing effects, and screen-space shaders correctly calculate vertex transformations to clip space, and correctly access the packed render texture for reading, since Unity is going to use a single double-width render texture for both eyes. Unity outlines the changes that must be made in the following articles [16], [17]. Since a VR ready computer capable of this technique was not available, the following paragraphs outline the steps that could be made to support it. When performing stereo rendering, unlit (vertex/fragment) shaders must differentiate between the left and right eye, when transforming the vertices to clip space, since the left and right eye use different view matrices. Unity suggests that these vertex transformations to clip space are done through the standard UnityWorldToClipPos(in float3 pos) call. The only shader that is impacted from this, is the sphere impostor, since the sphere impostor shader calculates the camera oriented quad vertices in the vertex program in view space. However, if we examine the Unity built-in shaders, we can see that the UnityWorldToClipPos() function applies the standard transformation that our shaders do as well, by using the built-in View and Projection matrices, Appendix F: Stereo instancing Unity built-in matrix variables. This means that the sphere impostor vertex program that uses these matrices as well, does not need any change. If that is not
34 the case, we could abandon the calculation of the quad oriented vertices in the vertex program, and follow the same path like in the cylinder impostor, that is, render a geometry that tightly covers the perfect geometrical sphere, which is a cube for the sphere impostor. With this change, we can directly use the above function. Post-processing effects must also change how they access the packed render texture. The only change that must be applied is when reading texel data, and use Unity’s built-in function for transformation of uv coordinates UnityStereoScreenSpaceUVAdjust(). To add support for stereo instancing, apart for the macros needed for regular instancing, we must also add to the vertex output data structure the UNITY_VERTEX_OUTPUT_STEREO macro. To know if a shader renders for the left or the right eye, Unity offers the variable unity_StereoEyeIndex, which has a different value for each eye. With this value, and with the necessary macros to set it up, the shader can differentiate between the two eyes. 3.9 Unity prefabs The impostor sphere object, and the impostor cylinder object, can now be packed under a Unity prefab, which is an object ready to be instantiated in the world. Both the sphere and the cylinder impostor, expose an interface that allows the user to change their parameters (color, radius, highlighting, transparency etc.) by passing these changes through the material blocks, and thus at all times support instancing. However, the above prefabs have the following limitations: First, shadow casting and shadow receiving is not implemented. Shadow casting requires the definition of a ShadowCaster pass, which has been explained briefly in the start of the Deferred rendering chapter. Shadow receiving requires some more careful handling, since the actual impostor fragments are generated in the fragment shader, and thus, we cannot simply use the built-in helper functions: LIGHT_ATTENUATION and TRANSFER_VERTEX_TO_FRAGMENT as they are. Impostors can support shadow receiving, if we carefully transform the world position of the fragment in the fragment shader, to the coordinate space defined by the light, by slightly tweaking the above macros. Another limitation of these prefabs is that the forward rendering pass is not compatible with factory post processing effects. 3.10 Performance and Unity evaluation In the first part of this section, the performance benefits from using the impostor techniques will be explored, while in the second section are demonstrated the advantages and disadvantages of using Unity. Performance In order to evaluate the performance benefit of using impostors to render spheres and cylinders, we spawn the same scene, one time by using Unity spheres and cylinders, and another time using impostors, for the following models: • 1TES 1435 atoms, 1092 bonds • 4F0H: 5030 atoms, 3895 bonds • 1S5L: 45945 atoms, 42644 bonds
35 All benchmarks were run in a low-end laptop computer with the following specs: i5-8250U, NVIDIA MX150 2GB, in 1920x1080 resolution, since high performance Desktop computers were not available. The results are the following: 1TES Triangles rendered CPU main (ms) Render thread (ms) FPS No impostors, No Instancing 1.2M 5 2 200.1 No impostors, Instancing 1.2M 4.3 0.8 230.2 Impostors, No instancing 18.5K 6.1 3.7 161.5 Impostors, Instancing 18.5K 4.8 1.1 210.4 4FOH Triangles rendered CPU main (ms) Render thread (ms) FPS No impostors, No Instancing 4.2M 14.7 5.6 68 No impostors, Instancing 4.2M 12.9 2 77.4 Impostors, No instancing 63.0K 18.3 10.2 54.7 Impostors, Instancing 63.0K 13.9 1.7 72.1 1S5L Triangles rendered CPU main (ms) Render thread (ms) FPS No impostors, No Instancing 38.7M 332.8 222.4 3.0 No impostors, Instancing 38.7M 126.8 14.5 7.9 Impostors, No instancing 605.3K 225.9 118.3 4.4 Impostors, Instancing 604.5K 131.6 12.1 7.6 The key outtakes from the above numbers is that, first, if the scene has thousands of objects for rendering, instancing, or any other technique for render call batching is a must. Second, impostors will only yield a performance benefit when the GPU is not able to render the geometry. For example, in the small model, the GPU is able to render 1.2M triangles, and thus, impostors don’t improve the performance. However, as the models get larger, the impostors scale better in rendering times. Of course, by using impostors, we also render spherical geometry with much higher fidelity. The main problem however, in all executions, is the CPU main time. In Unity, having thousands of objects in the scene, even if they are marked as static, even if they have no Update() function attached, will worsen performance. For example, the following image is a snapshot of the Unity profiler, for a specific frame when rendering the 1S5L model, with impostors and instancing:
36 Figure 26: Unity profiler for 1S5L In the above image, we can see how much time rendering and culling require. For that specific frame, CPU main time was 153.3 ms, while the GPU time was 14.6 ms, which means that the bottleneck in performance is managing thousands of objects in the scene. In order to increase performance, we would have to employ techniques that aim to lower the number of objects in the scene, for example, partitioning the world. Furthermore, given that the Unity GameObject is a relatively heavy structure, we could try to use a more lightweight game engine, or even create a custom one, that does the bare minimum required. Unity advantages and disadvantages Using Unity comes with a long list of advantages, where the most important ones are debugging superiority and development speed. When writing the shaders implemented, and managing the order of rendering, one of the most valuable features was the Frame Debugger. In the frame debugger, one can carefully examine the order in which render calls are issued, the parameters that the shaders have, and the results of the rendering in a single place. Figure 27: Debugging rendering order using the Frame Debugger
37 When it comes to development speed, one can jump right into the implementation of his work, while Unity takes care of Input, Output, disk IO, VR peripherals, and also takes care of several other “invisible” features. For example, if we were to implement our own custom engine, we would definitely have to implement several acceleration data structures, to ease culling, scene traversal and ray casting. Furthermore, we would also have to implement a complete rendering pipeline that involves both deferred, post processing effects, and UI forward rendering. The disadvantages of Unity in this project, can be summed up to the fact that there are several restrictions when using the engine. For example, Unity is very centralized around the GameObject structure. For example, it is not possible to have colliders to perform ray casting, without having GameObjects, and it is not straightforward to render the same GameObject multiple times with different shaders. Another disadvantage of writing Unity shaders is that, Unlit shaders, that is Vertex/Fragment shaders, don’t have extensive documentation, and many times one must rely on diving into the code of the Unity built-in shaders.
38
39 4. Interaction Interacting with a Virtual Reality scene, usually involves one or more 3D selection techniques, with the most common being, especially in Head Mounted Displays, ray casting. However, ray casting a specific object with a hand-held controller, can sometimes be very challenging, especially in complex and cluttered areas. As a result, this second part of the work, will be devoted on a novel navigation technique within a molecular scene, that does not need ray casting. This navigation technique will instead use simple 2D direction input, that can be given with the arrows in a keyboard, or with a touchpad in a controller, that will allow the user to select atoms without ray casting to them. We are going to then use this navigation technique to visualize properties of the molecular scene, like the distances between atoms, the angles between bonds, and the torsion angles formed by multiple atoms. Figure 28: Atom distance and bond angle visualization Figure 29: Atom distance in space filling model, and torsion angle visualization 4.1 3D selection To select an object in a VR environment, we have to employ a 3D selection technique. F. Argelaguet and C. Andujar in [18] provide a thorough list of 3D selection techniques, along with their advantages and disadvantages. In this work, we mainly focus away from 3D selection, and thus, we will build upon a simple 3D ray casting technique.
40 To be able to ray cast into the scene and get back the closest intersection, either if it is a sphere, or a bond, or a UI element, we are going to attach Unity colliders onto the impostor objects. On the sphere impostor we attach a Sphere Collider, in the Cylindrical impostor we attach a capsule collider and in the UI elements we attach Box Colliders. Capsule colliders in bonds, have a slightly bigger size to ease ray casting. 4.2 Information panel The information panel, is a world space canvas, specifically mapped to always be on the top left of the camera view, that displays information about the selected atom. Figure 30: Information panel The information panel displays the type of the current atom selected, and the residue that it belongs to in the first row, in the second row it displays the exact atom name within the residue given the molecule’s and atom’s nomenclature, as well as the secondary structure that this atom belongs to. The third and fourth row display the occupancy and temperature factor of the atom, as parsed from the .pdb file. The torsion atoms section is used to mark atoms when visualizing torsion angles. This object is solely for the purpose of information visualization, and the user will not have to interact with it in any shape or form. The information panel is always rendered on top of everything else in the scene and it’s the last thing rendered in the transparent queue, with the Z-Test set to AlwaysPass. To map the position of the world space panel on the top left of the camera, we make the panel a child of the camera object, and calculate the position of the label in the local coordinate system defined by the camera, with the following algorithm: Figure 31: Calculating horizontal position of the information panel
41 𝐿𝑅 =(z distance)∗tan(𝐻𝐹𝑜𝑉 2), 𝐿𝑎𝑏𝑒𝑙.𝑥 =−𝐿𝑅 2+ℎ𝑜𝑟𝑖𝑧𝑜𝑛𝑡𝑎𝑙 𝑜𝑓𝑓𝑠𝑒𝑡+𝑝𝑎𝑛𝑒𝑙 𝑤𝑜𝑟𝑙𝑑 𝑤𝑖𝑑𝑡ℎ 2 With a similar equation with the above, and using the vertical field of view, the height of the panel and a vertical offset, we can calculate the y coordinate of the position of the label. The z coordinate is fixed at z distance. After setting the position, the panel is rotated to face the camera and since the object is a child of the camera, the initial position and orientation is preserved along the movement of the camera. With these calculations, we can precisely control the position of panel in the camera view, and “zoom in” or “zoom out” the panel by changing the z distance parameter. Furthermore, since the calculation is automatic, porting the application in different resolutions is much easier, since we only have to adjust the z distance parameter, because going in lower resolution will “zoom in” the label, and higher resolutions will have the opposite effect. 4.3 Visualizing residues and chains With the use of ray casting, the user can visualize residues and chains within the molecular model. To visualize residues, we highlight all the atoms that belong to the residue of the atom currently ray-casted by the user. While this technique works well to pinpoint small structures within a big group of molecules, it is not very effective to visualize bigger structures like chains. Towards that, to visualize chains, we change the color of all the atoms that belong to the chain of the currently ray casted atom, and at the same, highlight the atoms that belong to the same residue. In both cases, the information of the ray casted atom is displayed on the information panel. Figure 32: Residue visualization in ball-and-stick model (left), and space-filling model (right)
48 distance number, is a quad always facing the camera, whose position is calculated with the cross product between the vector that connects the camera and the middle of the distance, and the vector that connects the two atoms. The position of the label is static in the world, i.e. does not change depending on the camera position, but rather calculated once. In contrast, the position of the direction arrows in the 2D navigation, are constantly calculated to be in front of the atom. Furthermore, since the position of the number label is not guaranteed to be visible and not occluded, the rendering ignores depth. 4.6 Bond angles The user can select consecutive bonds and display the angle between them. The angle between bonds is an important piece of information, as it is part of the molecule geometry that influences many properties of a substance, for example, reactivity, polarity, phase of matter, color, magnetism and biological activity. The bonds that the user can select and display the angle between, must have the same origin, so that the angle value is meaningful. Figure 44: Visualization of angle between bonds(left), vector calculation(right) To select consecutive bonds, we can adapt the 2D selection technique used previously for navigation. Furthermore, since the bond angle only makes sense between bonds with a common origin, we don’t need to have the additional mark button, but we can have the bond angle displayed continuously between the last two bonds visited during 2D navigation. The molecular angle is always displayed in the [0 – 180) range, as a result, we have to always make sure to display the angle in that range and draw an arc that always connects the smallest angle of two (instead of its complementary), no matter in which order the bonds are selected. To do that, we calculate the direction vectors of the bonds, 𝑋 and 𝑊 in Figure 44, always with the regard to the same origin, which is their common atom, disregarding the orientation of the 3D objects that encapsulate the two bonds. That way, the angle can be calculated as: 𝑎𝑛𝑔𝑙𝑒= 𝑎𝑟𝑐𝑐𝑜𝑠( 𝑋 ∗𝑊 |𝑋 |∗|𝑊 |) With the direction vectors 𝑋 and 𝑊 , and the above angle value, we can apply the following formula to draw the arc: 𝑍 =𝑋 ×𝑊 , 𝑌 =𝑍 ×𝑋
49 𝑃(𝜃)=𝑅∗cos(𝜃)∗𝑋 +𝑅∗sin(𝜃)∗𝑌 where 𝜃 ∈[0, 𝑎𝑛𝑔𝑙𝑒], and 𝑅 is the desired radius of the arc. The above formula gives us points along the arc in the local coordinate space of the arc object, which is placed at the point of intersection of the two direction vectors 𝑋 and 𝑊 . To draw the arc, we use a resolution of 30 points along the arc connected with a line, using Unity’s LineRenderer tool. To have the arc rotated and moved in accordance to the whole model, we have to make sure to always transform the local coordinate space arc points to world space coordinates, every time there is a translation or rotation of the parent object, since Unity’s LineRenderer draws lines whose coordinates are in world space. The position of the angle number label, is calculated by using the vector that connects the arc origin with the middle of the arc, i.e. 𝑃(𝜃 2), and expanding along that direction using the required radius of the arc, and multiplication factor of 1.8. The rendering of the label is done by a custom shader that takes into consideration Z-Testing, since Unity’s default 3D text rendering shader does not. 4.7 Torsion angles A torsion angle, is the angle defined as a particular example of a dihedral angle, describing the geometric relation of two parts of a molecule joined by a chemical bond [19]. Every set of three non colinear atoms in space define a plane. Thus, a group of four consecutively-bonded atoms, A-B-C-D, define two planes, A-B-C and B-C-D. The torsion angle then, is the dihedral angle between these two planes. Figure 45: Torsion angle visualization (up), selected atoms order (down) Figure 46: Torsion angle visualization, aligned view
50 In its absolute value, the angle is between 0° and 180°, however, the torsion angle between groups A and D is considered to be positive if the bond A-B is rotated in a clockwise direction through less than 180° in order to eclipse the bond C-D. A negative torsion angle requires rotation in the opposite, counterclockwise sense. The symmetric applies for the angle between D and A. The torsion angle is a very important piece of information in stereochemistry, and it is used widely in molecular conformation [20], as a result, we want to be able to calculate it and visualize it effectively. Selecting atoms To select the atoms that will form the torsion angle, we can use the same selection technique that we used in the previous chapters. Just like in the bond distance visualization, the user can navigate through the atoms in the scene freely, and use the mark button to mark atoms for the torsion angle. Since the order here matters, we are going to display the marked atoms in the information panel, in the order in which they are marked. Once the user has selected four atoms, the torsion angle will be calculated and displayed automatically. Calculating the dihedral angle The order in which the atoms are selected is important, since in an 𝐴,𝐵,𝐶,𝐷 configuration, the 𝐵,𝐶 atoms are the ones that participate in the formation of both planes, and also define the axis of rotation for the torsion angle. As a result, if the user has selected the atoms in the above order, the calculation of the torsion angle is done with the following formulas: 𝑛1 =𝑃𝑙𝑎𝑛𝑒𝑁𝑜𝑟𝑚𝑎𝑙(𝐴,𝐵,𝐶), 𝑛2 =𝑃𝑙𝑎𝑛𝑒𝑁𝑜𝑟𝑚𝑎𝑙(𝐵,𝐶,𝐷) 𝑎𝑛𝑔𝑙𝑒 =arccos(𝑛1 ∗𝑛2 ), 𝑠𝑖𝑔𝑛 =𝑠𝑖𝑔𝑛(𝑛2 ∗(𝐴−𝐵)) Visualizing the dihedral angle To visualize the dihedral angle, we are going to render transparent planes that represent the planes defined by the atoms, we are going to highlight the atoms that form the dihedral angle, and we are also going to draw an arc, with the same algorithm like the bond angle. To render the transparent planes, we are going to build the meshes of the planes from scratch, i.e. we will calculate the world space coordinates of the plane vertices, using the world space coordinates of the atoms. This is going to give us more control to the position, scale and orientation of the meshes, compared to using Unity’s plane mesh object, and trying to position and orientate each plane according to the atoms. Thus, we calculate the world space coordinates of the points 𝑃𝐴,𝑃𝐵,𝑃𝐶,𝑃𝐷, in Figure 47, and construct two back and two front facing triangles for a single plane. This algorithm is run symmetrically for the atoms C-B-A and B-C-D, so that Atom1 and Atom2 always refer to the axis atoms.
51 Figure 47: Torsion plane construction To draw the arc, we use the same arc object used to visualize the bond angle, by defining the following 𝑋 and 𝑊 directions, originating from the middle of the atoms that form the common axis: Figure 48: Torsion angle arc rendering 4.8 Translating and rotating Translating and rotating the whole model is supported for all of the above interaction methods. Since rendering the geometry requires no additional transformation, other than the Model matrix computed by Unity, we need not re-calculate anything in order for the impostor objects to be rotated or translated in real-time. We simply group all atoms and bonds under a single parent object called Atoms. The 2D navigation technique, re-maps the atoms within the spherical region to the new coordinate system in each frame by default, since the 2D direction changes based on the camera position and orientation. As a result, the 2D directions also respond to changes in rotation and position of the object itself. Text labels, arrow direction labels, color circles and torsion planes, are Unity objects grouped under the same parent, and as a result, their rotation is handled by Unity.
52 The only thing that we must make sure is rotated and translated correctly along with the Atoms object, are lines drawn through Unity’s LineRenderer object, since this object requires that the coordinates are given in world space. Thus, every time a line is drawn, we calculate the points of the line in the local object space, and at real-time, check if there has been a change in the Model matrix of that object, and if it has, re-transform the local object space points to world space, and pass them to the LineRenderer. 4.9 World space widget In order to facilitate interaction, the scene will also have another object to allow the user to change the parameters of the visualization. This panel, is an actual world space panel, i.e. has a specific position in the world unchanged, and has world space virtual buttons with which the user can interact with, and change modes of interaction. Figure 49: World space interaction panel This widget, has the ability to be drawn in front of everything else in the scene, in order to facilitate interaction in cluttered environments. The ability to bring forth this widget, can me mapped into a physical button. When that button is pushed, this panel is drawn on top of everything else in the scene, except for the information panel, and the user can perform ray casting only against that panel. That way, the user can change parameters and modes of interaction, even when the panel is occluded. Modes of interaction and visualization parameters There are three configurable options: This option is used to change the visualization model of the atoms. The user can interact with it with ray casting, and in real time, change the visualization between space filling and ball-and-stick. The default is ball-and-stick model. This option is used to change the method used to display the direction that must be chosen, when navigating between atoms or bonds. The default is arrow direction labels. This button is used to change the exploration method between visualizing chains, and visualizing residues. The default is visualizing residues. and four modes of interaction:
53 General atom exploration: In this mode, the user can ray cast towards the atoms of the model, in any combination of model visualization and chain or residues visualization, and when an atom is selected, navigate through the neighbors while visualizing the information of the currently selected atom. Atom distances: In this mode, the user can ray cast towards the atoms of the model, in any combination of options, and when an atom is selected, navigate and mark atoms for distances, as explained in the atom distances section. Bond angles: In this mode, the user can ray cast towards the bonds of the model, and when a bond is selected, navigate within the bonds while displaying the angle between them, as explained in the bond angles section. This mode is not compatible with the space filling visualization, or the chain or residue visualization. Torsion angle: In this mode, the user can ray cast towards the atoms of the model, in any combination of options, and when an atom is selected, navigate and mark atoms for torsion angle freely, as explained in the torsion angle section. In general, the user can configure any of the above parameters and modes, in any combination. For example, the user can measure atom distances either in the space filling or the ball-and-stick model. Navigate within bonds and atoms by either using the arrow directions or the color circle. The visualization of chains or residues refers to the first ray casting call in any mode, before selecting an atom. For example, when in the atom distances mode, and while ray casting towards the model before selecting an atom, then the user can either visualize residues or chains. Of course, there are some incompatible options, for example, the bond angles mode does not make any sense along with the space-filling mode, since there are no bonds. Nonetheless, the user is free to interact with the model in that way. Furthermore, in order to minimize the ray casting operations, when the user switches between modes, the previously selected atom will remain. For example, switching from atom distances to torsion angles, will keep the currently selected atom, and the user can start visualizing torsion angles without a selecting a new atom, by just navigating to the neighbors with 2D controls. When the change of mode involves change between selected atom and selected bond, then the nearest object in either case is fetched, if such exists. The final option in this panel gives the ability to the user to change the radius of the selection sphere for 2D navigation in real-time, and thus, reach possibly unreachable atoms without ray casting to them. Virtual buttons implementation Each image in the panel, is a button that interacts with ray casting, and implements its own state machine to facilitate the interaction, based on the following interface: public abstract class ButtonEvent : MonoBehaviour { public abstract void RayCastHoverOff(); public abstract void RayCastHover(); public abstract void RayCastHit(); } The RayCastHover() function is called when a world space ray collides with the button, the RayCastHoverOff() is used to monitor if the ray is still colliding with the virtual button, and the RayCastHit() is used to signal that the button was hit.
54 There are three kinds of buttons. The first row of buttons, which are independent from each other, have four different states, one for each option, and their highlighted version. The state machine about the atom visualization button can be seen in the following image. The other two implement a similar one. Figure 50: The state machine for the atom visualization virtual button The second kind of button is the current mode buttons, which are not independent with each other, since the user can select any of them, in any order, and the other buttons must react to the change. An example of the state machine that they implement can be seen in the following image. The rest of the mode buttons implement a similar one. Figure 51: The state machine for the atom exploration mode virtual button The third kind of button, are the buttons used to change the radius of the selection, whose state machine is very simple: Figure 52: The state machine of the virtual button for increasing the radius of the navigation sphere
55 4.10 Model and World UI position Since the application can read any input .pdb file, the initial position of the model inside the scene, as well as the initial position and rotation of the camera, must be calculated for each input model. To calculate the initial position of the model, we calculate the axis aligned bounding box of the input atoms, with size (xsize, ysize, zsize), and place the model on top of the virtual floor, so that the bounding box barely touches the floor, using a small epsilon value. The initial position of the camera is placed in front of the model looking towards the model in the z axis, at distance from the floor that is greater than 4∗𝑦𝑠𝑖𝑧𝑒 6 ⁄, where 𝑦𝑠𝑖𝑧𝑒 is the size of the bounding box in the y axis. By using this distance value, the camera in the initial position, is slightly higher than the center of the bounding box. The World UI panel is placed on the right of the model, by using the size that the bounding box has in the x axis, and adding a constant factor. Figure 53: Camera and floor position
56 Figure 54: World panel position The disadvantage of the above technique is that if the model is too large along the y axis, the user might get a weird feeling from the virtual floor being too far away. An alternative to this is, cap the distance that the virtual camera can have from the virtual floor on a certain distance value. This value has to be calculated of course by using a HMD. 4.11 HTC Vive Controller button mapping In this chapter we are going to explore a possible mapping of operations into the physical buttons offered from the HTC VIVE controllers. We essentially need the following input for faster interaction: Input Operation Selection button Select an atom during ray casting, and when navigating between neighboring atoms and bonds, and for interacting with the world UI 2D directional input Used to choose a direction when navigating between neighboring atoms and bonds Mark button Used to mark an atom in atom distances mode, and in torsion angle mode Discard button Used to exit the 2D selection, and start ray casting again World UI button Used to bring forward the world UI panel Apart from the above operations, we can also map basic translate and rotate operations onto the controllers, so that the user can move the molecular model in space. For example: Input Operation 2D directional input Move the whole model up, down, left, right in world space 2 Buttons When the user has selected an atom, move the model towards or away from the user Rotational input Rotate the whole model For the first table of interaction operations, we can use the right controller, and for the second table of movement operations we can use the left controller.
57 Figure 55: Possible mapping of interaction operation onto the right controller Figure 56: Possible mapping of model translation operation onto the left controller As far as the rotation of the model is concerned, we can map a rotation button onto a physical button, and when this button is pushed, we can detect changes of rotation of the left controller, and apply these rotations onto the model. That way, the rotation of the model could be controlled from the user’s hand rotations.
64 [16] "Unity Documentation, Single Pass Stereo Rendering," Unity, [Online]. Available: https://docs.unity3d.com/Manual/SinglePassStereoRendering.html. [Accessed 24 5 2020]. [17] "Unity Documentation, Single Pass Instancing," Unity, [Online]. Available: https://docs.unity3d.com/Manual/SinglePassInstancing.html. [Accessed 24 5 2020]. [18] F. Argelaguet and C. Andujar, "A survey of 3D object selection techniques for virtual environments," Computer Graphics, vol. 37, pp. 121-136, 2013. [19] "Compendium of Chemical Terminology, 2nd ed, "Gold Book"," IUPAC, 1997. [20] E. Anslyn and D. Dougherty, Modern Physical Organic Chemistry, 2006. [21] Baaden-Chavent-Martine, "UnityMol," [Online]. Available: http://www.baaden.ibpc.fr/umol/. [Accessed 15 6 2020]. [22] USFC, "ChimeraX," [Online]. Available: https://www.cgl.ucsf.edu/chimerax/. [Accessed 15 6 2020]. [23] M. Chavent, A. Vanel, A. Tek, B. Levy, S. Robert, B. Raffin and M. Baaden, "GPU‐accelerated atom and dynamic bond visualization using hyperballs: A unified algorithm for balls, sticks, and hyperboloids," Journal of Computational Chemistry, vol. 32, no. 13, p. 2924‐2935, 2011.
65 Appendix A: Ray-Sphere intersection void ImpostorSphere(float3 fragment_position_worldspace, float sphere_radius, inout float3 position_worldspace, inout float3 normal_worldspace, bool back_face = false) { float3 fragment_pos = fragment_position_worldspace; /* World space position of the center of the sphere, object space (0,0,0) */ float3 sphere_center = mul(UNITY_MATRIX_M, float4(0.0, 0.0, 0.0, 1.0)).xyz; /* A is the origin of the ray */ float3 A = _WorldSpaceCameraPos.xyz; /* B is the direction of the ray */ float3 B = normalize(fragment_pos - _WorldSpaceCameraPos.xyz); /* C is the sphere center */ float3 C = sphere_center; /* Solve the ray, sphere intersection problem */ float a = dot(B, B); float b = 2.0f * dot(B, A - C); float c = dot(A - C, A - C) - (sphere_radius * sphere_radius); /* Calculate delta */ float delta = (b * b) - (4.0f * a * c); /* Clip fragment if delta is negative */ clip(delta); /* If not, calculate world space position of intersection point */ delta = sqrt(delta); float t1 = (0.5f) * (-b + delta) / a; float t2 = (0.5f) * (-b - delta) / a; /* Get the closest point */ float t; if (!back_face) t = min(t1, t2); else t = max(t1, t2); position_worldspace = A + B * t; normal_worldspace = normalize(position_worldspace - sphere_center); } B: Ray-Cylinder intersection void ImpostorCylinder2(float3 fragment_position_worldspace, float3 cylinder_direction_worldspace, float cylinder_radius, float cylinder_height, inout float3 position_worldspace, inout float3 normal_worldspace) { /* Calculate geometry cylinder data, assume cylinder center is at (0,0,0) object space */ float3 C = mul(UNITY_MATRIX_M, float4(0.0, 0.0, 0.0, 1.0)).xyz; float3 e = cylinder_direction_worldspace; float r = cylinder_radius;
66 /* Calculate ray */ float3 P = _WorldSpaceCameraPos.xyz; float3 v = normalize(fragment_position_worldspace - _WorldSpaceCameraPos.xyz); /* Precalculate some values used more than twice */ float CPe = dot(C - P, e); float CPv = dot(C - P, v); float ev = dot(e, v); /* Calculate lambda for the point A that is closest on the cylinder axis */ float lambda = (ev / (pow(ev, 2) - 1)) * (CPe - CPv / ev); /* Calulcate point A */ float3 A = P + lambda * v; /* Calculate the distance of that point to the cylinder axis */ float d = GetDistanceFromPointToLine(A, C, e); /* If negative, discard fragment */ clip(r - d); /* Calculate the intersection point lambda value */ float l_prime = lambda - sqrt((pow(r, 2) - pow(d, 2)) / (1 - pow(ev,2))); /* Calulcate intersection point */ position_worldspace = P + l_prime * v; /* Calculate normal of that intersection point project that point on the cylinder axis, clip the projection based on the height required and then calculate normal using the projection to find the corresponding point on the axis */ float projection = dot(position_worldspace - C, e); clip(projection); clip(cylinder_height - projection); normal_worldspace = normalize(position_worldspace - (C + projection * e)); } C: Sphere impostor Unity deferred and forward shader Shader "Custom/ImpostorSphere" { Properties{ _Albedo("Albedo", Color) = (1, 0, 0.8, 0) _RadiusAndShading("_RadiusAndShading", Color) = (0.07, 0.7, 0, 0) } SubShader { Tags { "Queue" = "Geometry" } Pass { /* Base forward rendering shader, executed when the rendering mode is set to forward, and with the directional light as input */ Tags { "LightMode" = "ForwardBase" }
67 /* This shader will be used for transparent rendering, with specific rendering order */ Blend SrcAlpha OneMinusSrcAlpha ZWrite Off CGPROGRAM #pragma vertex vert #pragma fragment frag #pragma multi_compile_instancing #include "UnityCG.cginc" #include "Lightning.cginc" #include "Impostor.cginc" #include "AutoLight.cginc" #define BOX_CORRECTION 1.5 /* Provided by Unity */ uniform float4 _LightColor0; struct appdata { /* Instance ID */ UNITY_VERTEX_INPUT_INSTANCE_ID /* Object space position */ float4 vertex : POSITION; }; struct v2f { UNITY_VERTEX_INPUT_INSTANCE_ID /* clip space position */ float4 pos : SV_POSITION; /* view space position */ float4 view_pos : TEXCOORD0; /* Single pass instanced rendering */ UNITY_VERTEX_OUTPUT_STEREO }; /* Unpack extra instance properties */ UNITY_INSTANCING_BUFFER_START(Props) UNITY_DEFINE_INSTANCED_PROP(float4, _Albedo) UNITY_DEFINE_INSTANCED_PROP(float4, _RadiusAndShading) UNITY_INSTANCING_BUFFER_END(Props) v2f vert(appdata input) { v2f output; UNITY_SETUP_INSTANCE_ID(input); UNITY_INITIALIZE_OUTPUT(v2f, output); UNITY_TRANSFER_INSTANCE_ID(input, output); UNITY_INITIALIZE_VERTEX_OUTPUT_STEREO(output); float radius = UNITY_ACCESS_INSTANCED_PROP(Props, _RadiusAndShading).r; /* Transform standard quad geometry to face the camera */ /* Multiply the width of the quad with the box correction */
68 /* Multiply with 2 sinxe the standard quad geometry goes from -0.5 to 0.5 and we want the standard sphere to have radius 1 */ output.view_pos = mul(UNITY_MATRIX_MV, float4(0.0, 0.0, 0.0, 1.0)) + BOX_CORRECTION * float4(input.vertex.x, input.vertex.y, 0.0, 0.0) * 2.0f * float4(radius, radius, 1.0, 1.0); output.pos = mul(UNITY_MATRIX_P, output.view_pos); return output; } float4 frag(v2f input, out float outDepth : SV_Depth) : COLOR { /* Set up instance id */ UNITY_SETUP_INSTANCE_ID(input); UNITY_SETUP_STEREO_EYE_INDEX_POST_VERTEX(input); float4 radius_and_shading = UNITY_ACCESS_INSTANCED_PROP(Props, _RadiusAndShading); float radius = radius_and_shading.r; float ambient_factor = radius_and_shading.g; /* Compute real fragment world position and normal */ float3 normal_world, position_world; ImpostorSphere(mul(UNITY_MATRIX_I_V, input.view_pos), radius, position_world, normal_world); /* Calculate depth */ float4 clip = mul(UNITY_MATRIX_VP, float4(position_world, 1.0f)); float z_value = clip.z / clip.w; outDepth = z_value; float3 view_direction = normalize(position_world - _WorldSpaceCameraPos.xyz); /* Phong shading */ DirectionalLight light; light.direction = -_WorldSpaceLightPos0.xyz; light.ambient_factor = ambient_factor; light.diffuse_color = _LightColor0; float4 albedo = UNITY_ACCESS_INSTANCED_PROP(Props, _Albedo); float3 color = DirectionalLightColor(light, normal_world, view_direction, albedo.xyz); return half4(color, 0.35); } ENDCG } Pass { /* Base forward rendering shader, executed when the rendering mode is set to forward, and with a point light as input */ Tags { "LightMode" = "ForwardAdd" } Blend One One
69 CGPROGRAM #pragma vertex vert #pragma fragment frag #pragma multi_compile_instancing #include "Lightning.cginc" #include "Impostor.cginc" #include "UnityCG.cginc" #define BOX_CORRECTION 1.5 /* Provided by Unity */ uniform float4 _LightColor0; struct appdata { /* Instance ID */ UNITY_VERTEX_INPUT_INSTANCE_ID /* Object space position */ float4 vertex : POSITION; }; struct v2f { UNITY_VERTEX_INPUT_INSTANCE_ID /* clip space position */ float4 pos : SV_POSITION; /* view space position */ float4 view_pos : TEXCOORD0; /* Single pass instanced rendering */ UNITY_VERTEX_OUTPUT_STEREO }; UNITY_INSTANCING_BUFFER_START(Props) UNITY_DEFINE_INSTANCED_PROP(float4, _Albedo) UNITY_DEFINE_INSTANCED_PROP(float4, _RadiusAndShading) UNITY_INSTANCING_BUFFER_END(Props) v2f vert(appdata input) { v2f output; UNITY_SETUP_INSTANCE_ID(input); UNITY_INITIALIZE_OUTPUT(v2f, output); UNITY_TRANSFER_INSTANCE_ID(input, output); UNITY_INITIALIZE_VERTEX_OUTPUT_STEREO(output); float radius = UNITY_ACCESS_INSTANCED_PROP(Props, _RadiusAndShading).r; output.view_pos = mul(UNITY_MATRIX_MV, float4(0.0, 0.0, 0.0, 1.0)) + BOX_CORRECTION * float4(input.vertex.x, input.vertex.y, 0.0, 0.0) * 2.0f * float4(radius, radius, 1.0, 1.0); output.pos = mul(UNITY_MATRIX_P, output.view_pos); return output; } float4 frag(v2f input, out float outDepth : SV_Depth) : COLOR
70 { /* Set up instance id */ UNITY_SETUP_INSTANCE_ID(input); UNITY_SETUP_STEREO_EYE_INDEX_POST_VERTEX(input); float4 radius_and_shading = UNITY_ACCESS_INSTANCED_PROP(Props, _RadiusAndShading); float radius = radius_and_shading.r; float ambient_factor = radius_and_shading.g; float3 normal_world, position_world; ImpostorSphere(mul(UNITY_MATRIX_I_V, input.view_pos), radius, position_world, normal_world); float4 clip = mul(UNITY_MATRIX_VP, float4(position_world, 1.0f)); float z_value = clip.z / clip.w; outDepth = z_value; float3 view_direction = normalize(position_world - _WorldSpaceCameraPos.xyz); PointLight light; light.position = _WorldSpaceLightPos0.xyz; light.ambient_factor = ambient_factor; light.diffuse_color = _LightColor0; float4 albedo = UNITY_ACCESS_INSTANCED_PROP(Props, _Albedo); float3 color = PointLightColor(light, position_world, normal_world, view_direction, albedo.xyz); return half4(color, 0.35); } ENDCG } Pass { /* Deferred rendering shader, executed when the rendering mode is set to deferred */ Tags { "LightMode" = "Deferred" } CGPROGRAM #pragma target 3.0 /* Exclude GPU that don't support Multi Target Rendering */ #pragma exclude_renderers nomrt /* Define the vertex and fragment shader programs */ #pragma vertex vert #pragma fragment frag /* Add multi compiling support for instancing */ #pragma multi_compile_instancing #include "UnityCG.cginc" #include "Impostor.cginc" #include "UnityPBSLighting.cginc"
71 #define BOX_CORRECTION 1.5 struct appdata { /* Instance ID */ UNITY_VERTEX_INPUT_INSTANCE_ID /* Object space position */ float4 vertex : POSITION; }; struct v2f { UNITY_VERTEX_INPUT_INSTANCE_ID /* clip space position */ float4 pos : SV_POSITION; /* view space position */ float4 view_pos : TEXCOORD0; /* Single pass instanced rendering */ UNITY_VERTEX_OUTPUT_STEREO }; struct fragment_output { half4 diffuse : SV_Target0; half4 specular : SV_Target1; half4 normal_world : SV_Target2; half4 emission : SV_Target3; }; /* Unpack extra instance properties */ UNITY_INSTANCING_BUFFER_START(Props) UNITY_DEFINE_INSTANCED_PROP(float4, _Albedo) UNITY_DEFINE_INSTANCED_PROP(float4, _RadiusAndShading) UNITY_INSTANCING_BUFFER_END(Props) v2f vert(appdata input) { v2f output; UNITY_SETUP_INSTANCE_ID(input); UNITY_INITIALIZE_OUTPUT(v2f, output); UNITY_TRANSFER_INSTANCE_ID(input, output); UNITY_INITIALIZE_VERTEX_OUTPUT_STEREO(output); float radius = UNITY_ACCESS_INSTANCED_PROP(Props, _RadiusAndShading); output.view_pos = mul(UNITY_MATRIX_MV, float4(0.0, 0.0, 0.0, 1.0)) + BOX_CORRECTION * float4(input.vertex.x, input.vertex.y, 0.0, 0.0) * 2.0f * float4(radius, radius, 1.0, 1.0); output.pos = mul(UNITY_MATRIX_P, output.view_pos); return output; } fragment_output frag(v2f input, out float outDepth : SV_Depth) : COLOR { /* Set up instance id */ UNITY_SETUP_INSTANCE_ID(input); UNITY_SETUP_STEREO_EYE_INDEX_POST_VERTEX(input);
72 float4 radius_and_shading = UNITY_ACCESS_INSTANCED_PROP(Props, _RadiusAndShading); float radius = radius_and_shading.r; float ambient_factor = radius_and_shading.g; float metallic = radius_and_shading.b; float gloss = radius_and_shading.a; /* Compute real fragment world position and normal */ float3 normal_world, position_world; ImpostorSphere(mul(UNITY_MATRIX_I_V, input.view_pos), radius, position_world, normal_world); /* Calculate depth */ float4 clip = mul(UNITY_MATRIX_VP, float4(position_world, 1.0f)); float z_value = clip.z / clip.w; outDepth = z_value; /* Calculate albedo for this instance */ float4 albedo = UNITY_ACCESS_INSTANCED_PROP(Props, _Albedo); float is_highlighted = albedo.w; /* Calculate diffuse and specular component from using Unity's Physically based rendering pipeline */ half3 specular; half specularMonochrome; half3 diffuseColor = DiffuseAndSpecularFromMetallic(albedo.xyz, metallic, specular, specularMonochrome); /* Set output parameters */ fragment_output o; o.diffuse = float4(diffuseColor, 1); o.specular = half4(specular, gloss); o.normal_world.xyz = normal_world * 0.5f + 0.5f; o.normal_world.w = is_highlighted; o.emission.xyz = ambient_factor * diffuseColor; return o; } ENDCG } } Fallback "Diffuse" } D: Phong shading functions #ifndef __Lightning_cginc__ #define __Lightning_cginc__ /* Phong shading parameters used in forward rendering */ static float _Shininess = 32; static float _SpecularIntensity = 0.2f; struct DirectionalLight { float3 direction; float ambient_factor;
73 float3 diffuse_color; }; struct PointLight { float3 position; float3 ambient_factor; float3 diffuse_color; }; /* Calculate linear attenuation between the position of the fragment and the light */ float Attenuation(float3 fragment_position, float3 light_position) { float3 vert = light_position - fragment_position; float distance_inv = 1.0 / length(vert); return lerp(0.0, 1.0, distance_inv); } /* Calculate directional light color contribution */ float3 DirectionalLightColor(DirectionalLight light, float3 fragment_normal, float3 view_direction, float3 fragment_color) { /* Ambient component */ float3 light_ambient = light.ambient_factor * (fragment_color * light.diffuse_color); /* Diffuse component */ float3 light_direction_inv = normalize(-light.direction); float light_diffuse_strength = dot(fragment_normal, light_direction_inv); float3 light_diffuse = max(light_diffuse_strength, 0.0f) * (fragment_color * light.diffuse_color); float3 light_specular = float3(0, 0, 0); if (light_diffuse_strength > 0) { /* Specular component */ /* Find the reflected vector from the light towards the surface normal */ float3 light_reflect_vector = reflect(light_direction_inv, fragment_normal); float light_specular_strength = pow(max(dot(view_direction, light_reflect_vector), 0.0), _Shininess); light_specular = light.diffuse_color * light_specular_strength * _SpecularIntensity; } return light_ambient + light_diffuse + light_specular; } float3 PointLightColor(PointLight light, float3 fragment_position, float3 fragment_normal, float3 view_direction, float3 fragment_color) { /* Ambient component */ float3 light_ambient = light.ambient_factor * (fragment_color * light.diffuse_color); /* Calculate diffuse component */ float3 light_direction_inv = normalize(light.position - fragment_position);