Putting AI in your Ears with 3D Neural Networks

It’s difficult to communicate with someone when neither of you speak the same language; apps and online tools like Google Translate can help to bridge the gap, but they hardly make for a natural conversation. However, the rise of artificial intelligence (AI) has opened the door for the development of speech-to-speech technology, where spoken language is translated instantly in real time. To bring this idea closer to reality, a collaboration of European researchers led by Professor Cristell Maneux at the University of Bordeaux have proposed concepts for a 3D artificial neural network accelerator. This ultra-compact, highly efficient design could enable the construction of standalone earpieces capable of translating spoken language instantly, with no need for internet access.

Breaking Down Language Barriers with Speech-to-Speech Translation

As our world becomes ever more complex and interconnected, effective communication is more important than ever. In today’s globalised economy, businesses, research collaborations, and even casual conversations often involve speakers of several different languages. This diversity can be enriching, but it can also create barriers to communicating effectively.

The dream of speech-to-speech translation is to break these barriers down entirely. Speech-to-speech refers to the direct conversion of speech in one language to speech in another. This can involve several steps including:

• Automatic Speech Recognition (ASR), where raw spoken language is converted to a digital form such as text,
• Machine Translation (MT), where the generated text is translated to another language,
• and Text-to-Speech Synthesis (TTS), where the translated text is converted into natural-sounding speech.

A device capable of speech-to-speech translation would enable natural conversations between multiple individuals speaking different languages, by translating in real-time. You could speak to someone in your native language and speech-to-speech technology would immediately translate your words as you spoke them to your correspondent’s native language. This has obvious applications to industries such as tourism, but it could also play a vital role in international trade and diplomacy, where it’s vital that communication is precise with minimal loss of meaning.

The Technical Challenges of Constructing EdgeAI Devices

The rise of AI has already facilitated the creation of speech-to-speech technologies. But one drawback of the ones currently available on the market is that they don’t perform the translation themselves. They rely on large, energy-devouring supercomputers with thousands of graphics processing units (GPUs). Typical AI tasks require lots of computing resources, far more than an individual earpiece could supply. It’s these machines which process the speech data, translate it, and then communicate the results via the cloud back to the earpieces. There are several key issues with this way of operating. First of all, since all the conversation data is being sent to the AI model’s servers, there may be concerns over privacy and data protection. Secondly, with so much data being transferred both via the cloud and within the AI model’s own machinery, large quantities of energy are consumed.

The alternative would be an Edge AI device – a small device capable of performing AI tasks by itself without connection to a datacentre. They’re called ‘Edge’ devices because they would operate independently of the large AI models like ChatGPT, on the edge of the AI sphere, much like a satellite orbiting the Earth. Nevertheless, current computer architecture is too inefficient for Edge AI devices to be realistic. This is what a pan-European research team coordinated by the University of Bordeaux wants to change.

The FVLLMONTI project, led by Prof Cristell Maneux, aims to develop an Edge AI device specifically for speech-to-speech translation. This would be a small, lightweight earpiece capable of translating speech from one language to another, without any need for internet access or connection to a supercomputer. The FVLLMONTI team involves experts in nanotechnology and electronic engineering from universities across France, Germany, Austria, and Switzerland. One of their most important results has been the development of a new class of neural network accelerators. These are computational units specifically designed to perform the complex tasks needed by AI with much greater efficiency.

Computing in 3D with the Neural Network Compute Cube

One of the most promising ideas the team have explored is the development of three-dimensional computing architectures. In traditional computers, components are spread out over a 2D processing unit. Thanks to decades of development, 2D designs are extremely reliable and cost-efficient, but they simply aren’t designed to efficiently handle the kind of neural network tasks that are used in AI technologies.

One of the biggest challenges the team identified is the volume of data that needs to be transferred between the computer’s processor and memory. For AI tasks, the amount of data that needs to be transferred is orders of magnitude larger than for traditional computations and, when you’re confined to just two dimensions, this uses up lots of time and energy. Since compact devices, such as earpieces, have significant limitations when it comes to space, power, and battery life, increased efficiency is a paramount necessity. To address this, the FVLLMONTI researchers have been investigating the potential of vertically stacked units. By layering computing elements vertically, as well as horizontally, they found that the speed and efficiency with which a device can perform AI tasks can be vastly improved.

To this end, the FVLLMONTI team detailed the specifications for a new fundamental computing unit they call the Neural Network Compute Cube (N²C²).

The Beating Heart of the Machine: The Systolic Array

To test the potential of their N²C² units, the team began to investigate how they could be combined in a way that can perform matrix calculations effectively. Working with matrices, which are large arrays of numbers, is the fundamental basis of AI operations. The FVLLMONTI group started by arranging N²C² units into a structure known as a systolic array. The term ‘systolic’ comes from the name for the rhythmic beats of our hearts. Like a heart, the systolic array operates in ‘pulses’, whereupon data flows through successive N²C² units in the grid.

Combining the N²C² units in a systolic array offers several key advantages. Systolic arrays enable higher processing speeds by having the units work together in parallel. Whereas it would take a traditional processor more time to deal with calculations involving larger matrices, systolic arrays perform the tasks in constant time. This means that systolic arrays are ideally suited for dealing with the matrix-heavy computations characteristic of AI tasks. The FVLLMONTI team reported impressive speed-ups when N²C² units were used to perform tasks. A 4 x 4 arrangement of N²C² units achieved speeds of over 13 times the baseline, with larger 8 x 8 and 16 x 16 arrangements offering speeds of around 37- and 90-fold the baseline.

Unlocking the N²C² Units’ Full Potential

But a systolic array can be constructed from older, classic technology. The specific design of the N²C² units offers great potential for more impressive results. For a start, the systolic array the team used to test the units was still a 2D structure, but one of their main draws is that they can be stacked in 3D too. The team estimates that this alone would offer an additional 10x speed-up.

Another key design feature of the N²C² units is that they are built using Vertical Gate-All-Around Nanowire Field-Effect Transistors (VNWFETs). A FET is effectively a tiny switch used in traditional computer central processing units (CPUs) or GPUs to control the flow of electricity through the circuit board. Nowadays, FETs are mostly planar and spread out only in two dimensions. On the other hand, a VNWFET is where the switch —or gate— surrounds the entire unit, enabling them to be stacked in both horizontal and vertical directions. This overcomes one of the limitations of traditional FETs, which is that the gate length cannot be reduced much below 10nm. With VNWFETs, the gate is aligned vertically, meaning that horizontal spread of hardware can be reduced. This can lead to ultra-compact processing units with an estimated extra 10x gain in performance.

The N²C² units were also designed to contain ferroelectric elements, which is what the F in FVLLMONTI stands for. Ferroelectricity means that units can store some of their own data non-volatilely — that is, they have memory which is not lost when the device is switched off. This has obvious benefits: it reduces the distance over which data needs to be transferred, improving speed, and also helps to reuse data during computations, saving energy. The FVLLMONTI team estimate that this could yield a further 20x boost in performance.

Designing Software and Hardware Together for Greater Performance

An additional avenue the team explored was to investigate how the N²C² hardware and speech-to-speech translation software could be co-optimised to improve energy efficiency and speed. The team employed a strategy called ‘structured pruning’, which compresses data in the neural network by removing blocks with low value, called as such when they have limited impact on the final result. This method of pruning synergises with the underlying systolic array structure, meaning that significant speedups and improvements in energy efficiency can be achieved.

The research team found that each of the tasks involved in speech-to-speech translation could be pruned to improve speed: Automatic Speech Recognition tasks could be pruned by up to 15%, and Machine Translation tasks by up to 85%, with performance reduction of less than 10%. This pruning led to speed improvements of up to 74-fold for a block of 32 x 32 N²C² units. In addition, the energy consumption was reduced by between 22% and 34%. These results offer encouragement that the team’s N²C² architecture has the potential to provide the speed and efficiency needed for the development of an Edge AI device like a speech-to-speech translation earpiece.

Towards Speech-to-Speech Translation on Edge AI Devices

The FVLLMONTI team’s proposed N²C² unit is an exciting development towards the realisation of Edge AI devices capable of performing AI tasks locally. The latest results from the FVLLMONTI project indicate that this alternative architecture can drastically improve the efficiency in performing speech-to-speech related AI tasks. With the fundamentals of the N²C² unit established, the team are now looking to take the next steps to build a functional, compact earpiece that could break down language barriers instantly, and with no need for an internet connection.

Moreover, beyond speech-to-speech translation, the FVLLMONTI team’s work has the potential to revolutionise computing in much broader ways. The development of alternative computing architecture based on vertical nanowire FET technologies will be critical for the construction of all sorts of chips capable of performing a variety of AI tasks efficiently.