Spatial Audio
Introduction
The Google Virtual Reality (VR) SDK features a best-in-class audio rendering engine that is highly optimized for mobile VR. The goal of the engine is to give listeners a truly realistic spatial audio experience by replicating how sound waves interact with the environment and the listener's head and ears.
空間音頻
介紹
谷歌的虛擬現實(VR)SDK特性最佳音頻渲染引擎高度優化的移動虛擬現實。引擎的目的是給聽衆一個真正現實空間音頻體驗通過複製聲波與環境交互和偵聽器的頭和耳朵。
How Spatial Audio works
Spatial Audio is a powerful tool that you can use to control user attention. You can present sounds from any direction to draw a listener's attention and give them cues on where to look next. But most importantly, Spatial Audio is essential for providing a believable VR experience. When VR users detect a mismatch between their senses, the illusion of being in another world breaks down.
The Google VR SDK simulates the main audio cues humans use to localize sounds:
Interaural time differences.
Interaural level differences.
Spectral filtering done by our outer ears.
空間音頻是如何工作的
空間音頻是一種強大的工具,您可以使用它們來控制用戶的關注。你可以從任何方向呈現聲音吸引聽衆的注意力,給他們暗示在接下來去哪裏看。但最重要的是,空間音頻提供可信的虛擬現實的體驗至關重要。當虛擬現實用戶檢測他們的感官之間的不匹配,在另一個世界分解的假象。
谷歌VR SDK模擬主音頻提示人類使用本地化的聲音:
兩耳的時間差異。
兩耳的水平差異。
光譜過濾由外耳。
Interaural time differences: When a sound wave hits a person's head, it takes a different amount of time to reach the listener's left and right ears. This time difference varies depending on where the sound source is in relationship to the listener's head. The farther to the left or right side of the head the object is located, the larger this time difference is.
兩耳時間差異:當一個聲波撞擊一個人的頭,需要不同的時間到達聽者的左和右耳朵。這時差變化取決於聲源在關係到偵聽器的頭。越遠的左邊或者右邊的頭對象所在地,更大的區別。
Interaural level differences: For higher frequencies, humans are unable to discern the time of arrival of sound waves. When a sound source lies to one side of the head, the ear on the opposite side lies within the head's acoustic shadow. Above about 1.5 kHz, we mainly use level (volume) differences between our ears to tell which direction sounds are coming from.
Spectral filtering: Sounds coming from different directions bounce off the inside of the outer ears in different ways. The outer ears modify the sound's frequencies in unique ways depending on the direction of the sound. These changes in frequency are what humans use to determine the elevation of a sound source.
耳間水平差異:頻率越高,人類都無法辨別聲波的到達時間。當聲源位於頭部的一側,另一側的耳朵在頭部的聲影。約1.5 kHz以上,我們主要使用水平(體積)之間的差異我們的耳朵告訴聲音是來自哪個方向。
光譜過濾:來自不同方向的聲音反彈外耳的內部以不同的方式。外耳修改聲音的頻率以獨特的方式取決於聲音的方向。這些變化的頻率是人類用來確定聲源的高程。
Spatial Audio in Google VR
To simulate sound waves coming from virtual objects, we use a technology known as ambisonics to envelop the listener's head in a sphere of sound. The Google VR audio system surrounds the listener with a high number of virtual loudspeakers to reproduce sound waves coming from any direction in the listener's environment. The denser the array of virtual loudspeakers, the higher the accuracy of the synthesized sound waves.
在谷歌虛擬現實空間音頻
模擬聲波來自虛擬對象,我們使用一種技術稱爲ambisonics信封偵聽器的聲音在一個球體。谷歌VR音頻系統圍繞着偵聽器與大量的虛擬揚聲器繁殖聲波來自任何方向的偵聽器的環境。數組的密度虛擬揚聲器,合成聲波的準確性就越高。
Virtual loudspeakers are made possible through the use of head-related transfer functions (HRTFs). The cues discussed in the previous section are captured within these HRTFs. When audio is played through HRTFs over headphones, the listener is fooled into thinking the sound is located at a particular point in 3D space.
In the real world, as sound waves travel through the air, they bounce off of every surface in our environment, resulting in a complex mix of reflections. The Google VR SDK breaks this complex set of sound waves down into three components:
Direct sound
Early reflection
Late reverb
虛擬揚聲器成爲可能通過使用head-related轉移函數(頭)。在前一節中討論的線索在這些頭捕獲。打了頭在耳機音頻時,聽衆是傻到以爲聲音位於某一特定點在3 d空間。
在現實世界中,當聲波穿過空氣,每一個在我們的環境中表面被彈開,導致一個複雜的混合反射。谷歌VR SDK將這組複雜的聲波分爲三部分:
直接的聲音
早期反射
晚混響
The first wave that hits our ears is the direct sound which has travelled directly from the source to the listener. The farther a sound source is from the listener, the less energy it has resulting in a lower volume than closer sounds.
第一波襲擊我們的耳朵是直接的聲音直接從源到偵聽器。越遠的聲源是傾聽者,導致較低的更少的能量體積比聽起來。
The first few reflected waves that arrive at your ears are known as the early reflections. These give the listener an impression of the size and shape of the room they are in. The Google VR SDK spatializes the early reflections in real time and then creates new, artificial sources for each of them.
最初幾個反射波到達你的耳朵被稱爲早期的倒影。這些給聽者的印象房間的大小和形狀。谷歌VR SDK spatializes早期實時反射,然後創建新的人工來源。
Over time, the density of reflections arriving at your ears builds more and more until the individual waves are indistinguishable. This is what we refer to as the late reverb. The Google VR SDK has a powerful built-in reverb engine that can be used to very closely match the sound of real rooms. If you change the size of the room or the surface materials of the walls around you, the reverb engine reacts in real time and adjusts the sound waves to match the new conditions.
The Google VR audio system can also simulate the ways in which sound waves traveling between the source and listener are blocked by objects in between. The Google VR audio system simulates these occlusion effects by treating high and low frequency components differently, with high-frequencies being blocked more than low frequencies. This mimics what is happening in the real-world.
隨着時間的推移,反射的密度到達耳朵構建越來越多,直到個體是沒有區別的。這就是我們稱爲混響。谷歌VR SDK具有強大的內置混響引擎,可用於緊密匹配的聲音真正的房間。如果你改變房間的大小或你周圍的牆壁的表面材料,混響引擎實時反應和調整聲波以匹配新的情況。
谷歌VR音頻系統還可以模擬聲波的源和偵聽器被對象之間的旅行。谷歌的VR音頻系統模擬這些閉塞效應治療高和低頻率成分不同,高頻區被屏蔽比低頻率。這模仿真實世界發生的事情。
Directivity
Closely related to the effect of occlusion is a sound object’s directivity pattern. A directivity pattern is a shape or pattern that describes the way in which sound emanates from a source in different directions. For example, if you walk in a circle around someone playing a guitar, it sounds much louder from the front (where the strings and sound hole are) than from behind. When you are behind, the body of the guitar and the person holding it occlude the sound coming from the strings.
隨着時間的推移,反射的密度到達耳朵構建越來越多,直到個體是沒有區別的。這就是我們稱爲混響。谷歌VR SDK具有強大的內置混響引擎,可用於緊密匹配的聲音真正的房間。如果你改變房間的大小或表面
方向性
與阻塞的影響密切相關的是聲音對象的指向性圖案。一個方向性的模式是一種形狀或模式,描述了聲音的方式散發從源在不同的方向。舉個例子,如果你走在一個圓圈周圍有人彈吉他,聽起來聲音從前面和音孔(字符串)從後面。當你背後,吉他,拿着它的人的身體擋住來自琴絃的聲音。
With the GVR audio system, a user can change the shape of a directivity pattern for an object and mimic the non-uniform ways in which real-world objects emit sound. There are two available parameters:
隨着時間的推移,反射的密度到達耳朵構建越來越多,直到個體是沒有區別的。這就是我們稱爲混響。谷歌VR SDK具有強大的內置混響引擎,可用於緊密匹配的聲音真正的房間。如果你改變房間的大小或表面
GVR音頻系統,用戶可以改變對象的指向性圖案的形狀,模擬非均勻的方式真實世界的物體發出的聲音。有兩個可用的參數:
Alpha: Changes the shape of the sound emission pattern.
Sharpness: Controls how wide or narrow the emission pattern is.
Head movements and sound
By moving our heads, we can perceive the relative changes in all of the time level and frequency cues. This helps us to localize sounds more accurately.
α:聲發射模式的形狀變化。
清晰度:控制寬或窄的發射模式。
頭部動作和聲音
通過移動我們的頭,我們可以把所有的時間和頻率的相對變化的暗示。這有助於我們更準確地定位聲音。
When a user moves his head in VR, his head-mounted display tracks the movements. The Google VR SDK uses rotation information to rotate sounds around inside the virtual loudspeaker array in the opposite direction of head movement. In this way, virtual sounds stay locked in position.
當用戶移動他的頭在虛擬現實,他頭戴顯示設備跟蹤運動。谷歌VR SDK使用旋轉信息在虛擬揚聲器陣列內部旋轉聲音頭運動的方向相反。通過這種方式,虛擬聲音保持鎖定的位置。
Design Tips
GVR Audio Room
For each part of your GVR experience, you should first determine if you need a GVR Audio Room. Audio Rooms provide early reflections and reverb, which help make the sound more realistic when there are nearby walls or structures. They are—not surprisingly—most useful when your scene takes place in an actual room. For outdoor scenes, an Audio Room can feel less natural, because you may have only one reflective surface (the ground).
You have full control over the amount of reverb and the material of the surfaces, so take care to match the room sound to the environment.
對於每個GVR經驗的一部分,你應該首先確定你需要一個GVR音頻的房間。音頻房間提供早期反射和混響,使聲音更現實當附近的牆壁或結構。他們不是時,多數有用你的場景發生在一個實際的房間。對於戶外場景,一個音頻的房間可以感覺不那麼自然,因爲你可能只有一個反射面(地面)。
你有完全控制混響的數量和材料的表面,所以照顧比賽房間的聲音環境。
Atmospheric sounds
For producing the general ambience of a scene, like the wind in the trees, ocean waves, and birds, there are two choices for sound playback.
GVR Audio Sources.
GVR Audio Sources are the most flexible and work best for objects that move dynamically or that users might interact with. For example, if you attach an Audio Source to a bird, the listener hears the sound of the bird change naturally as it flies near, and then farther away. You can sprinkle Audio Sources throughout the environment to create the general ambience.
大氣的聲音
生產一個場景的氛圍,就像風在樹上,海浪,和鳥類,有兩個選擇聲音回放。
GVR音頻源。
GVR音頻來源是最靈活和最適合動態移動的對象,或者,用戶可能會相互作用。例如,如果您將一個音頻源到一隻鳥,聽衆聽到鳥的聲音改變自然,因爲它附近的蒼蠅,然後更遠。您可以在音頻資源在整個環境來創建氛圍。
GVR SoundField Sources.
GVR SoundField Sources play back ambisonic files that let you hear audio from every direction. This is similar to how skybox or 360 photos work. Since ambisonic files only respond to head rotation, they work best as sounds in the distance.
GVR SoundField來源。
GVR SoundField回放ambisonic來源文件,讓你聽到聲音從各個方向。這類似於天空體或360張照片是如何工作的。因爲ambisonic文件只響應頭旋轉,他們在遠處工作最好的聲音。
Animate the sound source
If you want to command the listener's attention, but a sound source is out of view, you can animate the position of the sound. This enables the listener to pinpoint sounds much more quickly.
動畫聲音的來源
如果你想命令聽衆的注意力,但是一個聲源的視圖,您可以動畫聲音的位置。這使聽者能夠查明聽起來更快。
Repeat the sound
To help the listener pinpoint a sound, play it more than once. This is why, for example, your phone's ringtone is not a single beep. If it was, you would have a hard time finding it, and you might
not even be sure it was your phone. You can achieve the same effect by using sounds that comprise many distinct elements.
重複的聲音
幫助聽衆查明一個聲音,不止一次。這就是爲什麼,例如,你手機的鈴聲不是一個單一的嗶嗶聲。如果是的話,你會很難找到它,你甚至可能不確定這是你的電話。你可以使用聲音來達到同樣的效果,包括許多不同的元素。
Use more complex sounds
You should avoid using overly quiet sounds, sounds lacking in high frequencies, or simple tones like a sine wave beep. Instead, craft sounds that have sufficient volume levels, are complex, and contain a full spectrum of frequencies.
使用更復雜的聲音
您應該避免使用過於安靜的聲音,聽起來缺乏高頻率,或者簡單的音調像一個正弦波嗶嗶聲。相反,工藝的聲音,有足夠的音量水平,很複雜,包含一個全譜的頻率。
Tips for crafting sounds
Audio Source sounds
For Audio Sources, make sure the sound files you use are monophonic and don't include reverb.
SoundField Source sounds (ambisonic sounds)
For SoundField Source sounds, we currently support first-order ambisonic files. These files are more complex than Audio Source files, and the tools and libraries that support them are still in the early stages.
With digital audio workstation (DAW) software and a plugin such as Ambix, you can create ambisonic files two ways:
Using monophonic files, place sounds on a virtual sphere around the user. You can move the sounds around and add effects to them.
小貼士制定的聲音
音頻源的聲音
音頻源,確保您使用的聲音文件是單聲部的,不包括混響。
SoundField來源的聲音(ambisonic聲音)
SoundField聲音來源,我們目前支持一階ambisonic文件。更復雜的音頻源文件,這些文件和支持他們的工具和庫還在早期階段。
數字音頻工作站(寒鴉)Ambix等軟件和插件,您可以創建ambisonic文件兩種方式:
使用單聲部的文件,聽起來在一個虛擬的領域用戶的地方。你可以移動的聲音和添加效果。
Use an ambisonic microphone like the SoundField ST450, TetraMic, or Zoom H2n to capture the sound of an environment in 3D. You can load the captured sound into the Ambix plugin and run effects on it, rotate it if needed, and then mix.
Check your work
After you get your VR experiences up and running, make sure to check your work. You want to ensure that what you see matches what you hear. For example, if you can hear the sound of ocean waves crashing, but the ocean looks frozen, it takes away from the sense of realism.
Be sure to visit all the places your users will go in your VR experience to confirm that everything sounds natural. In experiences where users can roam freely, they love putting their ear right up to sound sources.
使用一個ambisonic麥克風像SoundField ST450,TetraMic或變焦H2n捕捉3 d環境的聲音。你可以捕獲的聲音加載到Ambix插件和運行效果,如果需要旋轉,然後混合。
檢查你的工作
VR啓動和運行經驗,後你一定要檢查你的工作。你想確保你所看到的與你所聽到的。例如,如果你可以聽到海浪的聲音崩潰,但海洋看起來凍結,遠離現實的感覺。
一定要參觀所有的地方你的用戶將會在你的虛擬現實經驗證實這一切聽起來自然。在體驗,用戶可以自由行走,他們喜歡把他們的耳朵,聲音來源。
Tom Merton/Getty Images
Take extra care to ensure that the sounds you're using are of high quality, are at a clear but comfortable volume, and adjust realistically with any movement. Because most of your users will be listening through headphones, make sure to test your sound on a variety of headphones, not laptops or desktop speakers.
湯姆·默頓/蓋蒂圖片社
聽起來格外小心,確保你使用的是高質量的,在一個清晰但舒適的體積,並調整實際上與任何運動。因爲大多數用戶將通過耳機聽,一定要測試你的聲音在各種耳機,不是筆記本或桌面揚聲器。