Skip to content
Home Publications Publications
Preprints
2025
2024
Dissecting the Interplay of Attention Paths in a Statistical Mechanics Theory of Transformers Kazuki Irie , Haim SompolinskyNeurIPS ),MoEUT: Mixture-of-Experts Universal Transformers Kazuki Irie , Christopher Potts, Christopher D. ManningNeurIPS SwitchHead: Accelerating Transformers with Mixture-of-Experts Attention Kazuki Irie , Jürgen SchmidhuberNeurIPS Neural representational geometry of concepts in large language modelsKazuki Irie , Haim SompolinskyNeurIPS Workshop on Symmetry and Geometry in Neural Representations (NeurReps), December 2024
Self-Organising Neural Discrete Representation Learning à la Kohonen Kazuki Irie* , Róbert Csordás*, Jürgen SchmidhuberICANN ), September 2024Exploring the Promise and Limits of Real-Time Recurrent Learning ICLR
2023
Practical Computational Power of Linear Transformers and Their Recurrent and Self-Referential Extensions Kazuki Irie , Róbert Csordás, Jürgen SchmidhuberEMNLP ), Short paper, December 2023Approximating Two-Layer Feedforward Networks for Efficient Transformers Kazuki Irie , Jürgen SchmidhuberEMNLP-Findings ), December 2023Contrastive Training of Complex-Valued Autoencoders for Object Discovery Kazuki Irie , Jürgen SchmidhuberNeurIPS ), December 2023Mindstorms in Natural Language-Based Societies of Mind Kazuki Irie , Louis Kirsch, Bing Li, Guohao Li, Shuming Liu, Jinjie Mai, Piotr Piękos, Aditya Ramesh, Imanol Schlag, Weimin Shi, Aleksandar Stanić, Wenyi Wang, Yuhui Wang, Mengmeng Xu, Deng-Ping Fan, Bernard Ghanem, Jürgen SchmidhuberNeurIPS Workshop on Robustness of Few-shot and Zero-shot Learning in Foundation Models (R0-FoMo), December 2023Best Paper Award @ R0-FoMo Workshop Topological Neural Discrete Representation Learning à la Kohonen Kazuki Irie* , Róbert Csordás*, Jürgen SchmidhuberICML Workshop on Sampling and Optimization in Discrete Space, July 2023 Later version published at ICANN 2024 Accelerating Neural Self-Improvement via Bootstrapping Kazuki Irie , Jürgen SchmidhuberICLR Workshop on Mathematical and Empirical Understanding of Foundation Models (ME-FoMo), May 2023Images as Weight Matrices: Sequential Image Generation Through Synaptic Learning Rules Kazuki Irie , Jürgen SchmidhuberICLR ), May 2023
2022
CTL++: Evaluating Generalization on Never-Seen Compositional Patterns of Known Functions, and Compatibility of Neural Representations Kazuki Irie , Jürgen SchmidhuberEMNLP ), Short paper, December 2022Neural Differential Equations for Learning to Program Neural Nets Through Continuous Learning Rules Kazuki Irie , Francesco Faccio, Jürgen SchmidhuberNeurIPS ), November 2022Learning to Control Rapidly Changing Synaptic Connections: An Alternative Type of Memory in Sequence Processing Artificial Neural Networks Kazuki Irie , Jürgen SchmidhuberNeurIPS Workshop on Memory in Artificial and Real Intelligence (MemARI), November 2022The Dual Form of Neural Networks Revisited: Connecting Test Time Predictions to Training Patterns via Spotlights of Attention Kazuki Irie* , Róbert Csordás*, Jürgen SchmidhuberICML ), July 2022A Modern Self-Referential Weight Matrix That Learns to Modify Itsel Kazuki Irie , Imanol Schlag, Róbert Csordás, Jürgen SchmidhuberICML ), July 2022The Neural Data Router: Adaptive Control Flow in Transformers Improves Systematic Generalization Kazuki Irie , Jürgen SchmidhuberICLR ), April 2022Unsupervised Learning of Temporal Abstractions using Slot-based Transformers Kazuki Irie , Jürgen Schmidhuber, Sjoerd van SteenkisteNeural Computation, March 2023
2021
Going Beyond Linear Transformers with Recurrent Fast Weight Programmers Kazuki Irie* , Imanol Schlag*, Róbert Csordás, Jürgen SchmidhuberNeurIPS ), December 2021The Devil is in the Detail: Simple Tricks Improve Systematic Generalization of Transformers Kazuki Irie , Jürgen SchmidhuberEMNLP ), November 2021Linear Transformers Are Secretly Fast Weight Programmers Kazuki Irie* , Jürgen SchmidhuberICML ), July 2021A Modern Self-Referential Weight Matrix That Learns to Modify Itself Kazuki Irie , Imanol Schlag, Róbert Csordás, Jürgen SchmidhuberNeurIPS Workshop on Deep Reinforcement Learning (DeepRL), December 2021 Later version published at ICML 2022 Improving Baselines in the Wild Kazuki Irie , Imanol Schlag, Róbert Csordás, Jürgen SchmidhuberNeurIPS Workshop on Distribution Shifts (DistShift), December 2021Learning Adaptive Control Flow in Transformers for Improved Systematic Generalization Kazuki Irie , Jürgen SchmidhuberNeurIPS Workshop on Advances in Programming Languages and Neurosymbolic Systems (AIPLANS), December 2021Later version published at ICLR 2022 Unsupervised Learning of Temporal Abstractions using Slot-based Transformers Kazuki Irie , Jürgen Schmidhuber, Sjoerd van SteenkisteNeurIPS Workshop on Deep Reinforcement Learning (DeepRL) & Workshop on Offline Reinforcement Learning (OfflineRL), December 2021Later version published in Neural Computation Training and Generating Neural Networks in Compressed Weight Space Kazuki Irie , Jürgen SchmidhuberICLR 2021 Workshop on Neural Compression, May 2021
Publications before 2020 (Language Modeling & Speech Recognition)
Advancing Neural Language Modeling in Automatic Speech Recognition K. Irie PhD Thesis , Computer Science Department, RWTH Aachen University, Aachen, Germany, May 2020The RWTH ASR system for TED-LIUM release 2: Improving Hybrid HMM with SpecAugment K. Irie , M. Kitza, R. Schlüter, and H. NeyICASSP ), May 2020Domain Robust, Fast, and Compact Neural Language Models K. Irie , P. Golik, E. Beck, and H. NeyICASSP ), May 2020How Much Self-Attention Do We Need? Trading Attention for Feed-Forward Layers K. Irie , A. Gerstenberger, R. Schlüter, and H. NeyICASSP ), May 2020. [slides]
Training Language Models for Long-Span Cross-Sentence Evaluation K. Irie , A. Zeyer, R. Schlüter, and H. NeyASRU ), December 2019. [poster] A comparison of Transformer and LSTM encoder decoder models for ASR K. Irie , R. Schlüter, and H. NeyASRU ), December 2019Language Modeling with Deep Transformers K. Irie , A. Zeyer, R. Schlüter, and H. NeyInterspeech , September 2019. [slides] Best Student Paper Award On the Choice of Modeling Unit for Sequence-to-Sequence Speech Recognition K. Irie , R. Prabhavalkar, A. Kannan, A. Bruguier, D. Rybach, and P. NguyenInterspeech , September 2019. [slides] Google internship outcome RWTH ASR Systems for LibriSpeech: Hybrid vs Attention K. Irie , M. Kitza, W. Michel, A. Zeyer, R. Schlüter, and H. NeyInterspeech , September 2019Lingvo: a Modular and Scalable Framework for Sequence-to-Sequence Modeling et al. Google internship outcome
Investigation on Estimation of Sentence Probability By Combining Forward, Backward and Bi-directional LSTM-RNNs K. Irie , Z. Lei, L. Deng, R. Schlüter, and H. NeyInterspeech , September 2018Improved training of end-to-end attention models for speech recognition K. Irie , R. Schlüter, and H. NeyInterspeech , September 2018RADMM: Recurrent Adaptive Mixture Model with Applications to Domain Robust Language Modeling K. Irie , S. Kumar, M. Nirschl, and H. LiaoICASSP ), April 2018. [slides] Google internship outcome Prediction of LSTM-RNN Full Context States as a Subtask for N-gram Feedforward Language Models K. Irie , Z. Lei, R. Schlüter, and H. NeyICASSP ), April 2018Best Student Paper Award & IEEE Spoken Language Processing Student Travel Grant Award
The 2016 RWTH Keyword Search System for Low-Resource Languages K. Irie , E. Beck, R. Schlüter, and H. NeySPECOM ), Lecture Notes in Computer Science, September 2017Investigations on byte-level convolutional neural networks for language modeling in low resource speech recognition K. Irie , P. Golik, R. Schlüter, and H. NeyICASSP ), March 2017
LSTM, GRU, Highway and a Bit of Attention: An Empirical Overview for Language Modeling in Speech Recognition K. Irie , Z. Tüske, T. Alkhouli, R. Schlüter, and H. NeyInterspeech , September 2016The RWTH/UPB/FORTH System Combination for the 4th CHiME Challenge Evaluation K. Irie , A. Zeyer, M. Kitza, P. Golik, I. Kulikov, L. Drude, R. Schlüter, H. Ney, R. Haeb-Umbach, and A. Mouchtaris CHiME ), September 2016Automatic Speech Recognition Based on Neural Networks K. Irie , Z. Tüske, and A. ZeyerSPECOM ), Lecture Notes in Computer Science, August 2016Investigation on log-linear interpolation of multi-domain neural network language model K. Irie , R. Schlüter, and H. NeyICASSP ), March 2016.
