MARINE SOUNDSCAPE ECOSYSTEMS AT RISK:
A DATA-DRIVEN SPATIAL-SOUND INTERACTIVE EXPERIENCE TO RAISE AWARENESS
Pier C. ALFEO • 1, Alessio MASTRORILLO • 2, Giuseppa BUSCAINO • 3, Maria C. WANG • 4, and Emilia TAPPREST • 5
1 • Music Conservatory "San Pietro a Majella", Napoli, Italy
2 • Music Conservatory "Giuseppe Verdi", Ravenna, Italy
3 • CNR–National Research Council, Italy
4 • Research Cluster "Ecological Art Practices", NICHE - Ca’ Foscari, Venezia, Italy
5 • NVISIBLE.STUDIO, Amsterdam, The Netherlands
Abstract
This paper introduces the immersive interactive installation 504 Ways Of Listening and its underlying software Data Driven Multiverse Manipulation to explore new pathways to public awareness towards anthropogenic noise pollution in marine environments. By developing a system for mapping complex datasets into three dimensional sensory environments, this installation probes new perspectives on how marine soundscapes can be interpreted, represented and experienced.
Data Driven Multiverse Manipulation is a state-of-theart framework designed to explore and decode the hidden acoustic structures of underwater ecosystems. Through this software, this installation processes passive acoustic recordings dataset provided by the National Research Council of Italy (CNR) bioacoustics laboratory using adaptive normalization algorithms, statistical analysis, dimensionality reduction, quantization techniques, interaction design and High-Order Ambisonics to organize the data within an interactive multi-dimensional acoustic space.
In response to the growing ecological urgency posed by anthropogenic noise in marine ecosystems, 504 Ways Of Listening advocates for environmental awareness through the practice of listening, promoting new forms of engagement with the ocean’s acoustic narratives and the ecological issues they reveal.
1. INTRODUCTION
1.1 The Significance of Natural Soundscapes and the Impact of Anthropogenic Noise in Marine Environments
1.2 Citizen engagement
1.3 504 WAYS OF LISTENING
2. METHOD
2.1 Framework overview
2.2 Data origin
2.3 Normalisation
2.4 Segmentation
2.5 Analysis
2.6 Dimensionality Reductionand and Standardization
2.7 RealtimeAmbisonicsNavigation
3. DISCUSSION
3.1 Results
3.2 Limitations
3.3 Further developments
4. CONCLUSION
5. ACKNOWLEDGEMENT
6. REFERENCES
INTRODUCTION
The Significance of Natural Soundscapes and the Impact of Anthropogenic Noise in Marine Environments
Sound plays a pivotal role in the lives of many marine species, serving as a primary means of communication, navigation, and environmental awareness. Many marine organisms rely on acoustic signals to locate mates, detect predators, and coordinate group behaviors. The natural symphony of an ecosystem, often referred to as its soundscape, is thus integral to the health and functionality of its inhabitants.
However, the intrusion of anthropogenic (human-made) noise has increasingly disrupted these natural soundscapes, posing significant challenges to marine life. Activities such as shipping, construction, and energy exploration introduce substantial underwater noise. A study modeling the underwater sound of floating offshore wind farms in the Central Mediterranean Sea highlighted concerns about the potential impacts of such noise on marine life, emphasizing the need for careful assessment and mitigation strategies. Other studies explored how different anthropogenic noise sources impact marine organisms, revealing that high-intensity noise emissions from anthropogenic activities may significantly interfere with embryonic development. Moreover, research has shown that human-generated noise from both shipping traffic and oil and gas exploration activities may affect the distribution and behavior of marine species.
The consequences of altered soundscapes are profound. Anthropogenic noise can mask crucial biological sounds, leading to increased stress, disrupted mating calls, and impaired predator-prey interactions. In some cases, species may abandon habitats that have become acoustically inhospitable, leading to reduced biodiversity. Research on the acoustic environment of marine mammals has shown that noise pollution affects their ability to communicate and detect environmental cues, further highlighting the importance of preserving natural soundscapes. The study of fish bioacoustic diversity in Fiji's back-reef habitats demonstrated the importance of passive acoustic monitoring in understanding community dynamics and highlighted how noise pollution could disrupt these delicate ecosystems. Additionally, noise interference with dolphin communication contributes to the documented negative impacts on animal fitness.
CITIZEN ENGAGEMENT
Mitigating the impact of anthropogenic noise requires a multifaceted approach: beyond the adoption of quieter technologies, the implementation of acoustic regulations, and the establishment of protected areas, it is essential to act on a cultural level by promoting a renewed culture of listening and sound awareness. Sound, in fact, is not only a medium for environmental analysis: it is a shared material, a common territory between humans and non-humans, and can become a powerful tool for reconnecting with the marine environment.
Our ocean will be determinant in many of the things that happen in the choices that we have ahead, on where we live and how our economy moves, our ecosystems, our energy and how we deal with some of the risks that come from the climate change that we are going to have to adapt to in the future. So, our understanding of the Ocean is central to our understanding of our own prospects as a society and as an economy
John Bell
European Commission--Digital Twin Ocean
With a 2030 target, the EU Mission Restore our Ocean and Waters aims to protect and restore the health of our ocean and waters through research and innovation, citizen engagement, and blue investments. The Mission’s new approach addresses the ocean and waters as a unified system and plays a key role in achieving climate neutrality and nature restoration. In line with this mission, our work is positioned within the dimension of sensory and cultural engagement: a form of active participation that recognizes listening as a political, aesthetic, and epistemological practice - capable of generating shared ecological value.
Acoustic pollution has already profoundly transformed marine and terrestrial ecosystems, as well as our perceptual habits - altering languages, communication, and, in urban contexts, even musical production, as demonstrated by the phenomenon of musical pollution. In this light, the practice of active listening becomes an act of ecological citizenship. The issue of musical pollution must be assessed from this broader perspective of the relationship between humans and their sound environment - both in terms of the health of individuals and ecosystems, and in terms of the value of the sound environment as a public domain, a common good, an integral part of the ecosystem that must be protected.
To foster awareness around these issues and to promote listening as a form of care, we are developing a deep, blindfolded, and interactive sonic experience - an immersive installation - that uses scientific underwater passive acoustic data and new technologies, placing them at the service of the common good.
504 WAYS OF LISTENING
In this paper, we introduce 504 Ways of Listening (504) by Author1, a multimedia project comprising an immer- sive installation and its software framework, Data Driven Multiverse Manipulation (DDMM) . This interactive artwork is conceived to be navigated by a single performer at a time, through a control station positioned within an Ambisonics speaker dome, which is shared with an audience of listeners. The Ambisonics technique envelops the listeners in a three-dimensional sonic sphere, rendering the depths of the underwater world palpably present, enabling spatialized playback of underwater recordings in a way that mimics the real-world relations of acoustic features into an immersive 3D field. The performer, seated at the center, manipulates this acoustic environment through embodied interaction. In this context, sound is not merely a conduit for information, it becomes an experiential medium for navigating the dynamics and layered complex- ity of marine acoustic environments.
Thanks to the Italian National Research Council (CNR) and its high-fidelity hydrophone recordings, we have access to extensive datasets from otherwise inaccessible marine environments. These recordings provide rare access to underwater acoustic ecologies that are otherwise diffi- cult to perceive, forming the empirical basis for our installation’s audio content and structure. The development unfolds in two phases: one asynchronous, focused on preparing and analyzing the data and one real-time, ded- icated to sound synthesis and interactive control—detailed in the Methodology section. Interaction with 504 unfolds in a blindfolded state, with the performer’s vision entirelyblinded. Through a Virtual Reality (VR) headset, repur- posed solely as a gyroscopic sensor, the performer deter- mines their direction of observation. Seated on a rotating chair, they operate a digital Braille keyboard, where the hexagrammatic keys become directional controls within the acoustic universe.
504 invites reflection on how acoustic recordings - whether environmental, biological, or relational - can transcend their role as mere data and be reimagined as cultural material. Recent studies have shown, for instance, that the songs of humpback whales exhibit statistical patterns comparable to those of human language, suggesting that these soundscapes are not simply natural noise, but complex semiotic systems to be listened to, perceived, and revered.
Can art allow us to feel our way into such systems - those that rationality alone may fail to grasp - through practices of deep, embodied listening?
DATA DRIVEN MULTIVERSE MANIPULATION
Data Driven Multiverse Manipulation (DDMM) is a software framework, developed by Author1 and Author2, structured in two main phases: an offline phase, primarily dedicated to the analysis of audio recordings (dataset), and a real-time phase focused on managing the Ambisonics sphere and reading the analysis data.
Framework overview
This framework was developed using the programming languages Python and MaxMSP, along with the REAPER software.
The choice of Python as the programming language for the offline operations stems from its strong suitability for dataset analysis, including audio data, thanks to its extensive ecosystem of specialized libraries, such as NumPy, Librosa, Pydub, and SciPy, which we employed. Additionally, Python features a simple and readable syntax and offers widely adopted tools for Machine Learning and Artificial Intelligence, which are among the future developments planned for DDMM.
MaxMSP proves to be the optimal choice for real-time operations, as it is a powerful environment for creating interactive applications in the audio field, from music to multimedia arts. It features a visual programming environment that is immediate and modifiable in real time, valued for its flexibility in processing audio and MIDI signals and in the interactive control of sound, video, and physical devices.
Its modular and performance-oriented nature makes it both a creative and technical tool. Moreover, it includes powerful libraries for handling complex multichannel techniques, such as Ambisonics via Spat~ by IRCAM, and it offers the ability to write sample-level optimized algorithms using Gen~, a MaxMSP subsystem that compiles code to reduce latency and improve efficiency.
Within the digital audio workstation REAPER, the decoding phase of the Ambisonics signals generated with Spat~ is managed using the IEM audio plugin suite. This phase is efficiently handled by REAPER, which is particularly lightweight, versatile, and well-suited for flexible multichannel signal routing.
The following sections detail the DDMM processes developed in Python, for the offline normalization, segmentation, analysis, dimensionality reduction and standardization, and the real-time processes developed in MaxMSP and REAPER for Ambisonics navigation.
