Shared Workspace And Memory for Brain-Inspired Transformers (SWAMBIT)
Subproject: Transformer models and medical applications
Goal. Large monolithic transformer systems such as ChatGPT offer great practical benefits, at the price of scaling laws that create huge demands in terms of computation. This project proposes research into reducing the computational needs of Machine Learning models by means of a novel architecture inspired by the mammalian brain, focusing on sparsity, modularity and global feedback controllers. Our architecture consists of Specialist Modules (cortical macro-columns), a Shared Global Workspace (thalamus) and a Memory Bench (hippocampus). We use small transformers as building blocks for our multi-modal agent architecture. It is scalable and maintainable, as new transformer modules can be added or updated without disturbing others. It is energy efficient during training and inference as the activation of the basic building blocks is sparse.
Outcomes. Our modular architecture is far more energy-efficient than monolithic approaches. It offers business opportunities not only to the biggest players but also to smaller companies. Modules can be combined from different sources and custom domain specific specialists can be trained at low cost. Our versatile agents can be applied to a variety of tasks combining different modalities.
Projektleitung:
Prof. Dr. Felix Alexander Gers, BHT
Partner:
Prof. Dr.-Ing. habil. Alexander Löser, BHT
Prof. Dr. Matthew Larkum, HU Berlin, Institut für Biologie
Associate Partner:
Prof. Dr. Walter Senn, Universität Bern, Institute for Physiology / Computational Neuroscience Group
Förderung:
BMFTR
Projektlaufzeit:
1.12.2025-30.11.2028
Kontakt:
Felix.Gers (@) bht-berlin.de
Alexander.Loeser (@) bht-berlin.de (Data Science)
