\Organoid_Protonoesis 1

A neurobiological-data-driven interactive transmodal installation that gives form and expression to the intricate relationship between brain-cell organoid firings, and AI image generation.

© transLAB, MAT, University of California, Santa Barbara & Kosik Neurobiology Lab, University of California, Santa Barbara, 2024

research team:

transLAB (Marcos Novak, Director) Nefeli Manoudaki Iason Paterakis, Diarmid Flatley, Ryan Millett, Jazer Sibley Schwartz

Kosik Neurobiology Lab (Ken Kosik, Director) Tjitse van der Molen, Eve Bodnia

SBCAST (Santa Barbara Center for Art, Science, and Technology) (Alan Macy, Director)

Organoid_Protonoesis 1 is a neurobiological-data-driven interactive transmodal installation that gives form and expression to the intricate relationship between brain-cell organoid firings, and AI image generation. It is a work-in-progress that extends previous efforts such as the transLAB’s Synaptic Time Tunnel (SIGGRAPH 2023) in fusing transmodal media arts, neurobiology and cognitive science, and artificial intelligence and machine learning.

What is a brain organoid? A brain organoid is a three-dimensional organ model produced in the lab from pluripotent stem cells. These self-organized structures contain a multitude of organ-specific cells, mimicking aspects of early brain development and functionality, making them valuable for studying neurodevelopmental processes and diseases​.

This project introduces a real-time interface between spontaneous brain organoid activity and external stimuli, enabling unique and novel interactions between human intellect and emergent and self-organizing neural connectivity and activation. Neuronal firing data, converted to hyperedges within hypergraphs, are transformed into computational architectural morphologies, generative spatialized sound, and AI virtual entities.

Organoid_Protonoesis 1 showcases the potential of integrating real-time generative AI with biological systems, inviting participants to explore and reflect on the evolving boundaries of cognition, signal processing, and contemporary media. The current state of the project establishes a real-time communication between the brain organois in the Kosik Neurobiology Lab and the installation interface allowing for the development of a closed-loop interface between the brain organoids, human activiity, and an Artificial Intelligence Diffusion model.

Organoid_Protonoesis 1 is a collaboration between the MAT transLAB and the Kosik Neurobiology Lab, with additional support from the MAT AlloPlex Studio/Lab at SBCAST.

EXHIBITIONS:

MAT End of the Year Show @ California NanoSystems Institute (CNSI) Elings Hall, MAT Program, University of California, Santa Barbara, California / June 2024 (excerpt)

MAT- End of the Year Show @ SBCAST, Santa Barbara, California / June 2024 (excerpt)

\“Alloplex BioAffective Worlds XR: Hydnum”

Extending Transmodal Worldmaking beyond Russel's Circumplex Model of Affect.

© transLAB, MAT, University of California, Santa Barbara / SBCAST 2022

research team: Iason Paterakis, Nefeli Manoudaki, Diarmid Flatley, Pau Roselló Diaz, Alan Macy

supervisor: Marcos Novak

“Hydnum” is an extended reality data-driven interactive installation that immerses the visitor in a transmodal continuum of scent, music, form, poetry, space, and light. This data-driven piece is based on research connecting virtual and physical elements to emotional affect and cross-sensory stimulus through human biodata. Hydnum extends James Russell's "Circumplex Model of Affect and Motivational State" into transmodal worldmaking in extended reality. 

Hydnum is a transmodal / symbiotic system consisting of a prototype medical scent delivery system, a cyber-physical artifact, an extended Reality experience, and a generative audio component. The biodata visualizations also extend into the surrounding area by using projection mapping. The interactive spatial system can transmit scents based on the user's real-time psychophysiological measures (heart rate and heart rate variability data).

A modular system of tubes, 3D printed outputs, and brackets emerges from the six scent modules of the BIOPAC modular scent delivery system and expands into the surrounding space, introducing a prominent tree-shaped structure. Our approach adopts the characteristics of a nonlinear network, usually referred to as "Rhizome" after the Ancient Greek word "ῥίζωμα", "rhízōma".

Hydnum is the result of the ongoing collaboration between the transLAB, BIOPAC systems, and SBCAST (Santa Barbara Center for Art, Science, and Technology). Hydnum is part of the <Alloplex: BioAffective Worlds XR Extending Transmodal Worldmaking beyond Russel's Circumplex Model of Affect> research initiative.

The installation has been presented twice during the End of the Year Show of the Media Arts and Technology Program, of the University of California, Santa Barbara.

The first event took place in Elings Hall on Dec 27, 2022, and the second event was held at the Santa Barbara Center of Arts and Technology (SBCAST) on June 2nd, 2022.

Contribution: Concept Design, VR Content Creation (Unreal Engine), Installation Design, Hardware Setup, Projection Mapping

Keywords: #Extended Reality, #Transmedia, #Scent Art, #Hydnum, #Spatial Design, #Transplex, #Digital #Futures, #Biodata, #Scentification, #Sonification

EXHIBITIONS:

UCSB, Media Arts and Technology program- End of the Year Show @ Santa Barbara Center for Art, Science, and Technology (SBCAST), California, Santa Barbara / June 2022

UCSB, Media Arts and Technology program- End of the Year Show @ California NanoSystems Institute (CNSI) Elings Hall, MAT Program, University of California, Santa Barbara, California, Santa Barbara / May 2022

\Greece 2021 “Desire for Freedom”

© TUC TIE LAB 2021, “Greece 2021” Committee, ADAF

In June 2021, the most extensive projection mapping installation in Europe was simultaneously brought to life across 18 cities in Greece.

”On Saturday, June 12, and Sunday, June 13, 2021, the Greek Revolution "came to life" throughout Greece as a major event of the "Greece 2021" Committee. "DESIRE FOR FREEDOM" was screened on iconic buildings of 18 cities, using the technique of projection mapping. It is a visual narrative of the Struggle of 1821 through relevant works of art.

Athens (the Hellenic Parliament building), Alexandroupolis (Zarifeios Pedagogical Academy), Heraklion (the building of the Region of Crete), Thessaloniki (Royal Theatre), Ioannina (Judicial Hall), Kalamata (Visual Arts Workshop - Art Center), Corfu (Old Town Hall), Lamia (Municipal Theatre), Larissa (Club of Guard Officers), Messolonghi (Municipal Gallery), Naoussa (Town Hall), Nafplio (Town Hall), Piraeus (Municipal Theater of Piraeus), Rhodes (Town Hall), Syros (Apollo Theatre), Tripoli (Judicial Hall), Chania (Mikis Theodorakis Theatre), and Chios (Castle Wall - Porta Maggiore).

The narrative of the Revolution in a digital application, too.

About a month after the projections, a digital application began operating, which, through augmented reality, enables citizens and visitors of the 18 cities to watch the audiovisual narrative, which was shown on June 12 and 13 using their mobile phones or tablets.”

text source: https://greece2021.gr/en/provoles.html

links: https://www.youtube.com/watch?v=KZRjxncHSek

https://youtube.com/watch?t=804s&v=f4CWKtsJlxY

AR application on the App Store / Google Play

background video source: ©ADAF

\“Osmosis”

XR urban fluctuations using AI

© transLAB, MAT, University of California, Santa Barbara / SBCAST 2023-2024

research team -E03: Iason Paterakis, Nefeli Manoudaki (Visuals) Ryan Millett (Audio)

supervisor: Marcos Novak

“AI Worlds XR” is an extended reality (XR) installation that immerses the visitor in a transmodal continuum of AI, music, form, and space. The AI-generated forms express a continuous dialogue between the microcosms and the macrocosms. The piece is based on research connecting 2D AI images, biological forms (cells, and cosmic objects (Nebulae, star clusters, etc.). The biological forms are inspired by the German biologist Ernst Haeckel and his graphic work on biological forms.

VR, XR, and “Allotopes” with music and particle systems are formed inside a VR cosmos following a symbiotic principle. Rigid and familiar forms collide with natural and bioinspired structures to form an Allotope in Virtual Reality. The visualizations also extend into the surrounding area by using projection mapping. This research was expressed through VR and projection mapping in the transLAB during the end-of-the-quarter celebration of Tranvergence.

Hand drawings —generated with Procreate— were used along with text prompts to initiate the dialogues between AI Diffusion systems such as MidJourney and DALL-E. The 2D AI-generated images were then transformed into 3D forms using Blender’s shader nodes, and NVIDIA’s NERFs. A MAX MSP patch translated the images into spatial sound which was played back through the transLAB’s 16.2 sound system.

“AI Worlds” is based on the ongoing research initiative of <Alloplex: BioAffective Worlds XR Extending Transmodal Worldmaking beyond Russel's Circumplex Model of Affect>.

research team: Iason Paterakis, Nefeli Manoudaki

supervisor: Marcos Novak

Contribution: Concept Design, VR Content Creation (Unreal Engine 5.1), Installation Design, Projection Mapping, Sound Design

Software: Unreal Engine 5.1, Max MSP, Blender 3.3, Touchdesigner, Midjourney, Dall-E, NVIDIA NeRFs, MadMapper, Procreate

Keywords: #Extended Reality, #Transmedia, #Artificial Intelligence, #XR, #VR, #Allotopes, #Biological, #Digital #Futures, #Sonification

\"ETHO" (ἔθω) VR ”

an ML gallery in VR about the consequences of the war

© transLAB, MAT, University of California, Santa Barbara/ MIT XR Hack, Massachusetts Institute of Technology, Cambridge, Massachusetts

Etho refers to:

"... anyone afraid to talk about the war. To preserve history and remember the consequences and the impact of war. To raise awareness about the current hostile situation in Ukraine.

Etho, is the ancient Greek word that refers to the notion of a nation. Etho is offering an alternative reality that has been deprived of anything valuable and is composed of cultural ruins….”

“Etho” is a VR gallery related to the war in Ukraine. Aiming to introduce a strong feeling of immersion and detachment from our version of physical reality we used Virtual Reality technologies combined with physical interactive objects.

The artifacts are exhibited in a Virtual Apocalyptic Space surrounded by a fractal dome that contains life, culture, and the human. The user is immersed in the dark space containing remnants of a previous civilization. A high-level Machine Learning algorithm has processed the tangible remnants and projected them on the vertical spatial elements via virtual projection devices. The audio stimuli are granulated fragments from audio clips that refer to war and conflict.

More importantly, the piece is proposing a full embodiment of a fragmented cosmos. It is a balance between the safety of our western boundary and the remnants of a demolished civilization and culture.

research team: Iason Paterakis, Nefeli Manoudaki, Emma Zhao, Dillan Hoyos

Contribution: Concept Design, VR Content Creation (Unreal Engine 4.27)

Software: Unreal Engine 4.27, Max MSP, Blender 3.0, Procreate

Keywords: #Virtual Reality, #Transmedia, #Machine Learning, #XR, #VR, #Allotopes, #Virtualgallery, #Metaverse

Lorenz Attractor

Parameters:

σ = 10, ρ = 28, β = 8/3

System:

dx / dt = σ(-x + y)

dy / dt = -xz + ρx - y

dz / dt = xy - βz

Den Tsucs Attractor

Parameters:

a = 40, c = 0.833, d = 0.5, e = 0.65, f = 20

System:

dx / dt = α(y -x) + dxz

dy / dt = fy - xz

dz / dt = cz + xy - ex2

Chen - Lee Attractor

Parameters:

α = 5, β = -10, δ = -0.38

System:

dx / dt = αx - yz

dy / dt = βy + xz

dz / dt = δz + x

Allorenz Attractor

Parameters:

ρ = 3.94, σ = 8.79, β = 0.33

System:

dx / dt = yz

dy / dt = ρ(x-y)

dz / dt = (σ - β )xy - (1 - β) x2

The original thought came from the “Kinect imprint” of the user's figure, the composition concerns the existence of the body in a different virtual space. The captured figures appear in different virtual spaces - windows, through projection mapping technology. These interactive showcases-cells, symbolize the existential liberation from the physical world and its “entrapment” in a parallel digital environment.

The example of the kinectv2 library for processing was utilized as a basis for the experiment. Background videos were created in Cinema4d and were triggered when the user approached the sensor. Finally, the Processing program was integrated with Resolume via the Spout library for processing.

@7th Chania Film Festival

Project team: Iason Paterakis, Angelos Klothakis

Supervisor: Angelos Floros

Contribution: Concept Design, Content Creation (C4D, AE, Processing), Installation Design, Hardware Setup, Projection Mapping