创新背景

音乐艺术以旋律和词作构成人们听觉上的享受，以声波形式和人体产生反应。但人的身体器官不是单独存在的，多种知觉器官互相联系影响，构成了完整的生理机能，进而影响人的心理。

 

创新过程

人听到歌曲时往往会在脑海中联想与歌曲相关的场景，无论是模仿还是边做其他事边听音乐，所有与音乐相关的动作都会在耳朵实际听到的内容中发挥作用。挪威的音乐研究员Tejaswinee Kelkar表示，音乐不仅是一个好的歌词或旋律，它其实是听众所感受到的一切之间的相互作用。
Kelkar研究了与音乐相关的形状，并通过研究人们在听音乐时做出的手势，认为影响一个人听力经验感受的因素有很多，包括手势、面部表情和腿的位置。

歌唱欧洲音乐和印度音乐使用的手势是不同的。印度音乐经常用手势帮助孩子学习唱歌是很常见的，上台表演时注意力更多地在歌曲本身和手势影响，而非台下的观众。歌唱者专注于自己和排练房间内的联系。手势就像一定意义上的联想和音乐记忆，塑造歌唱者和听众使用音乐的方式。
音乐以不同的模式在人的所有感官上播放，研究主要注重它的空间性。听觉具有时间的前后限制，发音需要一个一个往外出，而音乐的空间模式则在于其他感知上的联想。

通过让参与者多次听一首歌的同一段音乐，绘制或解释听到音乐时脑海中想象的内容，研究了解到音乐是如何在空间上被感知的。有的参与者将音乐描述或绘制为经过他们脑海的波浪，有的则在注意到旋律中的某一段不断重复出现，想象出一个圆圈。
另外一个实验则让参与者们以自己认为和音乐符合的方式移动手臂，利用动作捕捉跟踪技术来记录一切，搜索模式。一部分参与者努力画出音乐上升或下降的轮廓，反映了音高、音色等音乐特征。
以往的语言研究者已经证明大脑会在人视觉和听觉内容之间建立联系，麦格克效应就是一种感性的认知现象，表现出在语音感知过程中听觉和视觉之间的相互作用。有时人的听觉判断会受到视觉的影响，从而产生误听。当视觉看到的一种声音与耳朵听到的另一种声音不匹配时，会让人们神秘的察觉到第三种声音。

面部表情对音乐感知的影响就是典型的麦格克效应，比如歌手表情的不同会影响人们对旋律的看法。如果歌手的模仿方式不同，两个音调之间的间隔可能听起来不同。

旋律和节奏是最常用于识别音乐的方式，而与旋律相关的形状也具有同样的功能。1975年丹尼斯·帕森斯在《曲调和音乐主题目录》中就提出股票根据轮廓搜索旋律的想法。他根据旋律轮廓（音高的上下移动）对大约15000首古典作品进行了编目。因此，基于音乐轮廓的识别被称为“帕森斯密码”。将与旋律相关的不同动作记录输入到人工智能中，可以开发先进的音乐识别技术，通过扫描动作就能找到想要的歌曲。

声音、文本和空间是音乐包含的模式。听同一首歌是否想象跳舞的感受是不同的，体验音乐会的方式不断变化，有的安静端坐聆听，有的观赏舞蹈感受，有的参与跳舞感知，都表明了音乐具有不同模式，即多模态性。每个人的所见所闻会影响对音乐的看法，反之亦然。

 

创新关键点

从不同的模式研究音乐和人的感官之间的相互作用，发现音乐与多重感知之间的联系。

 

The interaction of music with multiple perceptions

When people hear a song, they tend to associate scenes related to the song in their minds, whether it is imitating or listening to music while doing something else, and all the music-related actions will play a role in what the ear actually hears. Tejaswinee Kelkar, a music researcher in Norway, said that music is not only a good lyric or melody, it is actually an interaction between everything that the listener feels.

Kelkar studied shapes associated with music, and by studying the gestures people make when listening to music, he believes that there are many factors that affect a person's listening experience, including gestures, facial expressions, and the position of the legs.

The gestures used to sing European music and Indian music are different. It is common for Indian music to use gestures to help children learn to sing, and when performing on stage, the attention is more on the song itself and the influence of gestures than on the audience. The singer focused on his connection to the rehearsal room. Gestures are like associations and musical memories in a sense that shape the way the singer and the listener use the music.

Music is played in different modes on all human senses, and research focuses mainly on its spatiality. Hearing has a time limit, pronunciation needs to go out one by one, and the spatial mode of music lies in other perceptual associations.

By having participants listen to the same piece of music from a song multiple times, drawing or interpreting what is imagined in their minds when they hear the music, the study learns how music is perceived spatially. Some participants described or drew the music as waves passing through their minds, while others imagined a circle while noticing that a certain segment of the melody was constantly repeating.

Another experiment involved participants moving their arms in a way they thought fit the music, using motion capture tracking technology to record everything and search for patterns. Some participants worked hard to draw the outline of the rise or fall of the music, reflecting musical characteristics such as pitch and timbre.

Previous language researchers have shown that the brain makes a connection between human visual and auditory content, and the McGurk effect is a perceptual cognitive phenomenon that shows the interaction between hearing and vision in the process of speech perception. Sometimes people's auditory judgments are affected by vision, resulting in misauditation. When one sound seen visually does not match another sound heard by the ear, it mysteriously detects a third sound.

The effect of facial expressions on musical perception is typical of the McGurk effect, such as the difference in a singer's expression affecting people's perception of the melody. If the singer imitates differently, the interval between the two tones may sound different.

Melody and rhythm are the most commonly used ways to identify music, and shapes associated with melody have the same function. In 1975, Dennis Parsons proposed the idea of stocks searching for melodies based on outlines in the Catalogue of Tunes and Musical Themes. He cataloged about 15,000 classical works based on melodic outlines (pitch movements up and down). Therefore, the recognition based on the outline of the music is called the "Parsons code". By entering different motion records related to melodies into artificial intelligence, advanced music recognition technology can be developed to find the song you want by scanning the movements.

Sound, text, and space are the modes that music contains. Listening to the same song whether you imagine the feeling of dancing is different, the way of experiencing the concert is constantly changing, some sit quietly and listen, some watch the dance feeling, some participate in the dance perception, all indicate that the music has different modes, that is, multi-modality. What everyone sees and hears affects how they think about music, and vice versa.