NUS Chiplet Design Targets the Real Cost of AI: Memory, Not Maths

AI Summary

Researchers at the National University of Singapore (NUS) have published a technical paper called CHIPSMORE, describing a new accelerator design for running large language models (LLMs) more efficiently. Instead of treating memory chips as passive storage, the design embeds computation directly into memory (compute-in-memory, or CIM) and into the wiring that connects chip modules together (compute-in-interconnect). The architecture is built from chiplets — small modular chip blocks assembled like Lego — and is designed to handle many user requests at once while supporting both standard base models and lightweight customised versions known as low-rank adaptation (LoRA). For Malaysian businesses, the significance is indirect but real: inference cost — the ongoing cost of running AI models — is the biggest line item in AI adoption, and this research attacks the exact bottleneck (memory data movement) driving that cost. It also lands close to home, since Malaysia's Penang corridor specialises in the advanced packaging that chiplet designs depend on.

Key Takeaways

• The core insight: LLM inference is slowed by moving data between memory and processor, not by raw computing power. CHIPSMORE moves the compute to where the data already sits.
• The design supports "multi-mode" operation — one base model plus multiple LoRA-adapted variants — which mirrors how businesses actually deploy AI today: one general model, many task-specific customisations.
• "Multi-request" acceleration targets concurrent users, the real workload of any production AI service, from a customer-service chatbot to an internal document assistant.
• Chiplet-based design ties directly to advanced packaging and assembly — the segment of the semiconductor value chain where Penang and Kulim are strongest.
• This is a research paper, not a product. Commercial silicon built on these ideas is likely years away, so treat it as a signal of direction, not a procurement decision.

What Happened

A team at the National University of Singapore published a technical paper titled "CHIPSMORE: Compute-in-Interconnect and -Memory Chiplets for Multi-Mode Multi-Request LLM Inference Acceleration," reported by industry publication Semiconductor Engineering. The paper describes an accelerator — specialised hardware for running AI models — built from chiplets. Chiplets are small, function-specific chip blocks manufactured separately and assembled into one package, an approach the industry has adopted because cramming everything onto a single large silicon die has become expensive and yield-prone.

Two ideas distinguish the design. First, compute-in-memory (CIM): placing processing capability inside or right next to the memory that stores model data. Second, compute-in-interconnect: embedding processing into the wiring that shuttles data between chiplets. The paper states the accelerator integrates both, and that it supports "multi-mode" inference — meaning it can run a standard base model as well as low-rank adaptation (LoRA) variants, which are small add-on weight modules used to customise a base model cheaply — and "multi-request" inference, meaning it is built to serve many requests arriving at the same time.

To understand why anyone would do this, picture a restaurant kitchen. In a conventional computer, the chef (processor) keeps walking to the pantry (memory) to fetch ingredients, and the walking — not the cooking — dominates the time. LLM inference is famously pantry-bound: model weights live in memory, and reading them out dominates energy and latency. CIM is the equivalent of installing small cooking stations inside the pantry itself, so ingredients are prepared where they are stored. Compute-in-interconnect adds prep stations along the corridor between pantries, so work happens while ingredients move.

Why It Matters

Most conversations about AI progress fixate on model training and model cleverness. For businesses, the dominant cost is inference — the ongoing expense of running a model every time a customer, employee, or agent queries it. Training happens once; inference happens millions of times. Any architecture that lowers the cost per query, or raises the number of queries a given piece of hardware can serve, changes the economics of every AI product built on top of it. That is the stakes here. This is analysis on my part, not a claim from the paper, but the paper's framing — multi-request, multi-mode — is clearly aimed at serving workloads, not benchmark stunts.

The multi-mode support deserves attention because of how enterprises actually use LLMs. Almost nobody fine-tunes a full model from scratch anymore. The common pattern is one frozen base model plus several small LoRA adapters: one tuned for legal document review, one for Bahasa Malaysia customer chat, one for invoice extraction. Hardware that treats these adapters as first-class citizens — rather than requiring a separate copy of the model for each — maps directly onto how Malaysian banks, telcos, and government agencies are structuring their AI deployments.

The bigger signal is where cutting-edge AI silicon research is happening. A Southeast Asian university publishing on compute-in-memory chiplet integration for LLM serving, in a venue like Semiconductor Engineering's coverage, reflects the region's move up the semiconductor value chain. It also reflects an industry-wide consensus that the "memory wall" — the gap between processor speed and memory bandwidth — is now the central constraint in AI hardware. When multiple research groups converge on the same bottleneck, that bottleneck usually defines the next decade of products.

What This Means for Malaysia

The research comes from Singapore, but the subject matter sits squarely in Malaysia's industrial lane. Chiplet architectures live or die on packaging — the assembly, interconnection, and testing of multiple dies into one working unit. Malaysia, through Penang and Kulim, is one of the world's largest centres for back-end semiconductor operations, hosting major multinationals in assembly, test, and increasingly advanced packaging. If compute-in-interconnect and CIM chiplet designs move from papers to products, demand shifts further toward exactly the capabilities Malaysian facilities and their engineers provide. The National Semiconductor Strategy's push to move Malaysia up the value chain is aimed at precisely this transition.

There is also a talent and research angle. If NUS is producing this class of work, Malaysian universities and research institutes — and bodies like MDEC working on AI talent pipelines — have a nearby benchmark, plus potential collaboration partners across the Causeway. Regional supply chains mean a Singapore research breakthrough rarely stays a Singapore story; the design, packaging, and manufacturing ripple effects reach Johor and Penang quickly.

For Malaysian AI adopters, the practical relevance is cost trajectory. Cheaper inference eventually flows through to cloud API pricing and to the feasibility of running models on local infrastructure for PDPA-sensitive workloads. When inference hardware gets more efficient per ringgit, the case for onshore, in-country AI processing — rather than sending data to overseas endpoints — gets stronger. That matters for regulated sectors: banking, healthcare, and government.

How Your Business Can Use This

You will not buy a CHIPSMORE chip this year, and no vendor roadmap in the source material suggests otherwise. The value to you now is strategic planning, and there are three concrete moves.

First, budget on cost-per-query, not cost-per-project. List every AI use case your organisation runs or plans — chat support, document summarisation, lead scoring — and estimate monthly query volumes. When next-generation inference hardware arrives, your savings scale with query volume, so the use cases worth building today are the high-volume, repetitive ones. A Penang manufacturer processing supplier emails daily, or a KL e-commerce operator handling product questions, benefits most.

Second, structure your AI deployments around the base-model-plus-adapters pattern this hardware is designed for. Even on today's GPUs, keeping one base model and adding small LoRA adapters for specific tasks (a legal-contract adapter, a Malay-language customer-service adapter) is cheaper than maintaining multiple full fine-tunes. Adopting that pattern now means your architecture is aligned with where the hardware is heading, and migration later becomes simpler.

Third, if you are in the semiconductor-adjacent economy — equipment suppliers, EMS firms, precision engineering in Penang or Kulim — track compute-in-memory and chiplet interconnect skills. Engineers familiar with heterogeneous integration, testing multi-die packages, and interconnect reliability will be in demand as designs like this mature.

The Agentic AI Angle

Agentic AI — systems that plan, call tools, and execute multi-step tasks with minimal supervision — multiplies inference demand in a way simple chatbots never did. A single agent completing "review this contract, flag risks, draft a response" might make twenty or more model calls behind the scenes. Multiply that across a company's daily workflows and you can see the problem: agent economics collapse if each step is slow or expensive. This is my analysis, but it is the reason "multi-request" acceleration matters more each year. Hardware designed for many concurrent requests is hardware designed for fleets of agents.

The multi-mode angle matters for agents too. A well-built agent pipeline uses different tuned capabilities per step — extraction for one call, reasoning for another, generation for a third. An architecture that serves one base model with multiple LoRA adapters resident on the same silicon lets an orchestrator route each step to a specialised adapter without reloading anything. Concretely, a Malaysian bank could run a fraud-detection agent, a compliance-review agent, and a customer-reply agent on shared infrastructure, each drawing on its own adapter, served as concurrent request streams. That is the workload shape this paper targets.

Risks and Limitations

Be clear about what this is: a research paper. No commercial timeline, no vendor commitment, and no benchmark numbers appear in the source material — so any claim of "X times faster" you see elsewhere about CIM chiplets deserves scepticism. Compute-in-memory also carries known engineering challenges the field has not fully solved: precision and variability of in-memory arithmetic, manufacturing yield, and software compatibility with mainstream AI frameworks like PyTorch. These are why CIM has been "five years away" for a while now.

There is also ecosystem risk. Chiplet adoption depends on interconnect standards, packaging supply chains, and tooling maturing together. A design can be technically sound and still fail commercially if the surrounding stack is not ready