tomires/mvr.md

## mvr.md

      
    Raw
  

              mvr.md
            
          
    1 - introduction


continuum of reality a virtuality

real world
mixed reality

augumented reality
augumented virtuality


virtual reality


challenges

VR

creation of believable 3D assets
achieving high framerate


AR

real world analysis
seamless addition of virtual objects into real world


human senses in XR

principle

virtual world generator -> display -> sense organ -> brain


audatory (speaker)

surround speaker setups
stereo, binaural recording
spatial sound in videogames


visual (screen, projector)

world-fixed displays (CAVE)
head-mounted displays (VR headset)
handheld displays (cellphone)


haptic

various actuators
vibration, pressure, temperature


problems in VR

multimodalities

vestibulo-ocular reflex
discrepancy between movement sensation and visual input
motion sickness


input devices

traditional

game controller
keyboard/mouse


power glove
full body tracking

mocap
Kinect


3DOF

Wiimote
Oculus Go controller


6DOF

Oculus Touch, Vive wand


finger tracking

Leap Motion


tracking

intertial measurement unit (IMU)

estimates orientation
gyroscope

3D angular velocity
integrates over time
introduces drift error


accelometer

3D linear acceletarion


magnetometer

measures sum of all magnetic fields (building, environment, Earth,...)
corrects drift error


positional tracking

inside-out, outside-in
computer vision algorithms
depth cameras
electromagnetic field tracking


motion in VR

mirrors user's movement in real world
through an interface

game controller
teleportation


alternate world generators

game engines
world representations

X3D, VRML


2 - CG overview


gimbal lock - situation when two out of three gimbals are in the same plane resulting in the loss of one degree of freedom
3D rotation is not commutative
viewing transformations

local frame
global frame
(camera separation - L/R)
camera transform
normalized viewport frame
(distortion fix for HMD)
screen frame - individual pixels


3D geometry representations

basic primitives

cube, sphere, cylinder,...


mesh

vertex, edge, face


volumetric representation (voxels)

value in a grid
analogous to pixels


2D objects

sprites

billboards - sprites always oriented towards camera


lights

types of lights

directional

does not have an origin, constant intensity


point

intensity depends on distance from origin


area

computationally demanding, we calculate shadows


local lightning model

diffuse reflection (matte surface)
specular reflection (glossy surface)
ambient light (constant for entire scene)


shading

constant - flat shading
color interpolation (Gorraud)
normal vector interpolation (Phong)
physics based

attempt to mimic reality
energy conservation, surface self-occlusion


optimization

culling - occlusion, backface
clipping - near, far clipping plane
single/multi-pass


3 - interaction


should imitate real world
design around limitations of VR
interaction fidelity

realistic interactions

ex. holding tennis racket
key in surgical applications, training, etc.


magical interactions

natural physical movements are exarragated


non-realistic interactions

can increase performance, enjoyment
often causes less fatigue


biomechanical symmetry

degree to which body movements in VR correspond to real world
walking in room-scale vs using a joystick


input veracity

degree to which an input device measures user's movement
accuracy, precision, latency


proprioception

physical sense of the pose and motion of body and limbs


egocentric interaction

relative to body's reference frame


exocentric interaction

viewing and manipulating virtual model from outside of it
'god view'


stabilizing cues

objects placed in the periphery to reduce motion sickness
ex. plane cockpit, car dashboard


torso reference frame

can be used for UI, not as intrusive as a HUD
does not require tracked controllers


hand reference frames

visual representation of controller
left and right hands can serve different functions


head reference frame

used as the center camera for mixed reality applications
heads-up displays

UI elements should be small, few, not too close


eye reference frames

only used for specific applications
ex. scope of a gun


gestures

movement of body or body part
communicates spatial and symbolic information


visual-physical conflict

stop VR hand if penetration is shallow
alternatively show correct locations
ghosting, audio cues, force feedback


4 - interaction patterns


interaction patterns

generalized high-level interaction concept
implementation independent
from user's point of view


interaction technique

specific, implementation dependent
walking pattern

room-scale tracking, cybershoes, arm movements


types of patterns

selection

clutching - defining control space
pointing pattern (raycasting)
eye-hand visibility mismatch
occlusion and framing

head crusher, sticky finger


direct hand manipulation
proxy pattern

mapping from local object to remote
tracked physical props


3D tool pattern

direct manipulation with tool that manipulates object in the world


viewport control

real walking
redirected walking (slightly overemphasize head rotation)
walking in place / treadmill
3D multi-touch

simultaneous modification of position, orientation and scale using two hands


automated

teleportation, forced movement


indirect control patterns

speech recognition


5 - 3D reconstruction


process of converting real world into a 3D model
types

3D modelling

requires human work, very precise


laser scanner

expensive equipment, dense point cloud


depth camera

real-time performance, low resolution, limited distance


photogrammetry

computer assisted modelling

slow, but faster than manual modelling


automatic reconstruction

cheap and fast


acquiring photographs - challenges

perspective distortion
non-unique anchors
low dynamic range
occlusion


3D laser scanners

time of flight

long range - up to kilometers
slower, stable fixation


triangulation

short range - meters
faster, can be handheld


RGBd cameras

single depth camera can result in occlusion issues
structured light - projector + camera
time-of-flight camera - camera + light (measures speed of light)


stereo RGB

depth is calculated in post using stereo separation


photogrammetry

capture object from multiple angles, find common points
geometry is created based on changes between camera images
color value is derived from original photographs


6 - augumented reality


types of AR

retinal projection
glasses / HMD

optical see-through

safer (falls back to unoccluded vision)
does not reduce resolution of real world
easier optical blending


video see-through

wider FoV
matching delays of real and virtual views


handheld device
spatial

i.e. videomapping


display characteristics

spatial resolution in pixels
fill factor (space between pixels)
persistence - time the pixel remains illuminated
latency, response time
color gamut, contrast


world spatial understanding

extend real world with known objects (markers)

flat, 2D images


recognize natural markers

use depth data
keypoint extraction


build real world model from scratch and update it

simultaneous localization and mapping (SLAM)
track interest points with respect to previous frame
utilizes IMU


user tracking

GPS, WiFi beacons, gyroscope, accelometer, optical tracking,...


guidelines

if visualization is impacted by tracking error, the user should be able to see it
implement occlusion


7 - X3D


Virtual Reality Modelling Language (VRML)

declarative format for 3D data exchange
support in NN 2.0, MSIE 2.0
VRML 2 (97) - interactive behaviour, scripting
X3D - XML notation


scene is a DAG
instancing

<Shape DEF='BALL'>
<Shape USE='BALL'>


group nodes

<Shape> - contains geometry and appearance of an object

definitions for basic primitives (<Sphere/>)
extrusion
text
point set
elevation grid
indexed face set


<Inline> - container for external file
<Group> - groups nodes
<Transform> - transforms child nodes

scaleOrientation - rotation before scaling
scale
center - rotation pivot
rotation
translation


level of detail

pairing of models with distances


prototyping capabilities
interaction example

<TouchSensor DEF='lightswitch'>
<PointLight DEF='light'>
<ROUTE fromNode='lightswitch' fromField='isActive' toNode='light' toField='on'>


dynamics

sensor -> logic -> timer -> engine -> target
X3D: 8 sensors -> JS -> TimeSensor -> 6 interpolators -> any node in scene


sensors

manipulators - plane sensor, sphere sensor, cylinder sensor
collision, visibility sensor, proximity sensor
time sensor + interpolator -> animation


8 - sound in VR


interaural time difference

one ear is reached faster than the other


interaural intensity difference

one ear gets less energy than the other


head-related transfer function

measured for individual users


sound localization

repeat the sound / enable user to trigger them multiple times
use known sounds with wide frequency spectrum


complexities

reverberation by environment

high complexity of computation
ignore certain sound bite after X reflections?


sound refraction

sound gets around obstacles, gets muffled


diegetic ambience

sound of environment without any events happening


diegetic sounds

can be explained by processes inside the world
ex. footsteps of a person we see, dialogue


non-diegetic sounds

cannot be explained by processes in the world
ex. voiceover, music, UI sounds


performance optimization

reduce number of sounds, bake ambience into one track
distance culling

do not play one-shot sounds when too far
be ready to start playback of sustained sounds (ambience, music) when player enters neighbouring zone


volume culling

mute softer sounds when they are overpowered by louder sounds


9 - light and optics


visible light - wavelength of 380-750nm
refraction

diffraction - around corner
refraction - light transmission through material

refractive index - angle inside material depends on speed of light in medium


absorption
reflection - specular/diffuse


spherical lens

directs parallel rays of light into a focal point
parallel rays coming at an angle get directed elsewhere on the focal plane
focal distance

1/f = (n2-n1)[1/R1 - 1/R2]


diopter (m^-1)

converging/diverging power of lens
when combining lenses, overall power is a sum of parts
human eye - ca. 60D


lens abberations

optical distortion

barel distortion, pincushion distortion


chromatic abberation

speed of propagation of light depends on frequency
solution - compound lens, software compensation


astigmatism

light does not focus evenly on focal point


coma

off-axis point sources appearing distorted


10 - tracking


estimation of rotation and translation

head (HMD)
eyes (-> foveated rendering)
hands, fingers (-> interaction)
objects in physical space


differential nature of data from IMUs introduces drift error

fix by using another sensor to provide world reference

tilt - 'up reference' using accelometer
yaw - magnetometer


apply corrections slowly to avoid simulation sickness


positional tracking

inside-out, outside-in
natural features
LEDs


11 - human vision and latency


delay

result of tracking, application logic, rendering, display and synchronization delays
optimizations

light count, shadow detail, level of detail, MSAA, prebake lighting


eye movement - yaw, pitch, small roll

saccades - rapid jerks in motion, approx. 900°/s
smooth pursuit
vestibulo-ocular reflex

keeps image stability by countering head motion


vergence

convergence / divergence based on object distance