The Technology Behind Intel's Neural Processing Units: Accelerating AI

In a recent discussion, Michael Langan, a leader within Intel’s neural processing unit (NPU) team, shared insights about how NPUs serve as foundational elements in advancing artificial intelligence (AI). This conversation, underscored by the booming interest in AI technology, shed light on the essential role that NPUs play in supporting and speeding up AI computations.

With the growing prevalence of AI in various sectors, news about breakthrough technologies or new funding initiatives is ever-present. However, the complexity of AI requires powerful hardware designed specifically to handle the computational demands that AI applications entail. This is where NPUs, also referred to as AI accelerators, come into play. They are specially crafted hardware components that aim to replicate the processing abilities of the human brain to expedite the computations necessary for AI models.

Intel's NPU Development

During a conversation at the Midas conference in November 2024, Michael Langan explained that he has been part of Intel for 14 years and now leads their NPU IP (Intellectual Property) team based in Ireland. This team is imperative for developing technology that is crucial for a wide variety of client devices, including laptops and desktops. The NPU division contributes significantly to Intel's revenue, which stands at approximately $30 billion yearly.

The global NPU IP team consists of around 500 members, with origins traceable to the Irish start-up Movidius, acquired by Intel in 2016. Since the inception of NPUs, Langan pointed out that the turning point for the technology arose in 2012 when convolutional neural networks were introduced. These networks are a popular deep learning architecture widely utilized for image recognition and computer vision tasks. A major breakthrough came in 2017 with the publication of the paper ‘Attention is All You Need’ by Google, which introduced transformer architecture. Langan noted that this was a revolutionary moment for AI, giving rise to large language models (LLMs) like ChatGPT, which operate on the fundamental principles developed in that paper.

Hardware, Software, and Challenges

Langan elaborated on the diverse functions within Intel’s NPU infrastructure, including hardware design in Verilog RTL and extensive verification processes. Their design capabilities are adaptable, utilizing both TSMC and Intel process technologies, which allows the architecture to be implemented in various applications. The software aspect is just as critical, with a dedicated compiler team in Ireland working on optimizing AI compilers, a rapidly evolving area in the tech world.

As companies launch new models and features, the challenge of keeping pace with change has become a significant concern for the NPU team. Previously, the onus was on internal teams to educate customers about new capabilities, but now, it is the customers themselves who are bringing new needs and applications to Intel.

Talent Acquisition and Future Developments

Another pressing issue is the shortage of specialized talent needed for NPU development. The demand for experts skilled in areas like deep learning hardware and AI compilers is high. To address this, Intel has initiated an internship program over a decade ago to cultivate relationships with universities and develop a pipeline of top-tier talent. Langan remarked on the exceptional quality of candidates emerging from Irish institutions, which are now recognized globally.

Looking ahead, Langan noted that while current focus remains on AI models and their hardware/software environments, the question of what the next architectural breakthrough beyond transformers could be looms large. New variants are continually emerging, dubbed 'transformer killers,' such as Mamba and Hymba, which aim to enhance training efficiency while reducing power consumption. Intel is vigilant and aims to incorporate these advancements in future hardware developments, ensuring they are equipped to lead in the evolving landscape of AI technology.

Intel, NPU, AI