The AI Hardware Shift in IT Devices

In a new Super-Sector Analysis from Citi Research, a team of analysts led by Peter Lee predicts that AI servers will become so efficient and compact that one day people will carry them as easily as they do a smartphone, with such devices occupying a variety of form factors. We see portable AI servers emerging as key applications and becoming significant drivers of demand for on-device AI semiconductors.

Early computers were confined to laboratories and utilized for limited purposes, such as the military or national defense. With the emergence of the personal computer in the 1980s, computing entered everyday life; in the 2000s, laptops and smartphones allowed people to carry their own devices. We anticipate a similar transition in AI computing from centralized server-based infrastructure to personal on-device AI servers, kicking off an era in which most consumers own a portable, personal AI server.

Traditionally, AI-trained models required heavy hardware investment. But the emergence of distilled AI models such as DeepSeek has been a disruptive signal for the industry given DeepSeek’s significantly lower training and development cost; benchmark performance comparable with cutting-edge AI models; and open-source accessibility. DeepSeek’s approach has significantly advanced the field by showing how strategic combinations of reinforcement learnings and knowledge distillation can dramatically improve reasoning capabilities across various model sizes. We see these advances as accelerating the advancement of AI models that can be shrunk to operate in edge AI devices. As a result, we project the emergence of on-device AI demand, which should drive changes to computing structure and growth of semiconductors.

In the Von Neumann computing structure, instructions and data share the same memory space. This architecture has been widely adopted across PCs, smartphones and TVs; we think it will ultimately evolve into AI server-like architectures where DRAMs are placed adjacent to a neural processing unit (NPU)/tensor processing unit (TPU) to maximize AI processing efficiency.

Future computing architectures

We foresee the emergence of portable AI servers triggering three main architectural shifts in on-device AI products.

The first architecture direction is adding an AI kit to conventional Von Neumann architecture. This is the most convenient way to upgrade without changing the architecture of conventional IT devices. But we don’t think this fundamentally solves that existing architecture’s limitations, and it lacks the optimization needed for AI computing.

The second architecture direction involves the use of near memory or LPDDDR6 — the next generation of high-speed, low-power synchronous DRAM — with increased input/output and bandwidth near NPU and TPU. This is a more advanced and promising architecture than the first option, offering a more AI-friendly data path that improves computing efficiency.
The third architecture direction places DRAMs directly adjacent to NPU/TPU, similar to Nvidia’s AI server architecture. This offers the best performance, but at the highest cost.

We believe PC suppliers are considering the first architecture, while key industry participants are also exploring the second and third options. We expect a significant reduction in AI model sizes, accelerating architectural changes in portable devices where AI models can be embedded directly into edge devices. As a result, we believe mini AI servers will emerge as key applications, and we expect smartphones to take the lead in terms of architecture change. 

Technology directions

Edge devices require low power consumption and better thermal efficiency to become viable portable AI servers. With limited network throughput in edge AI environments, portable edge-device AI servers must optimize cost efficiency and energy efficiency to fully operate AI models.

Therefore, future semiconductors for mini AI servers will likely be designed with mobile-device-like characteristics such as low power consumption and cost efficiency. Toward that goal, we anticipate three major hardware architecture schemes:

 1.    We expect broader adoption of heterogeneous integration, with high-end mobile application processors combining processors, graphics processing units, static random access memory and other components.
2.    We expect that the adoption of mobile AI DRAMs will continue to expand, with low-power DRAMs becoming the mainstream choice by 2028, effectively increasing bandwidth and serving as an alternative to high bandwidth memory chips.
3.    We expect die-to-die integration to be widely adopted. This should make hybrid bonding technology, which is the most efficient technology for such integration, increasingly important. Hybrid bonding is more expensive than current thermocompression bonding.