Graphon Says Its ‘Intelligence Layer’ Will Lighten the Load on AI Models

Original Article
AI models have scaled to incredible size, but still face limits on the amount of data they can process at once. As a result, companies are sitting on massive amounts of data that their AI can’t fully understand, according to Arbaaz Khan, a founder and CEO of Graphon AI.

Khan, a former Amazon senior applied scientist who developed customer-service platform models, says he has created a new way to address that problem. Graphon is designed to make large language models more capable by creating a so-called intelligence layer that sits between data and the LLM.

The idea is to map the relationships across all sorts of data, from video to documents and systems, and real-world data, instead of having the LLM do it. And he says the new approach—based on applying smaller models to smaller chunks of data—is cheaper than processing all of the information in a massive LLM over and over again.

It’s an early-stage company, but reflective of an environment in which experienced developers who break off from established labs to test big ideas on their own are able to scale startups at an increasingly rapid pace.

Graphon said it emerged from stealth today with $8.3 million in seed funding to build its class of AI infrastructure. It said the round was led by Arvind Gupta of Novera Ventures, with participation from Perplexity Fund, Samsung Next, GS Futures, Hitachi Ventures, Gaia Ventures, B37 Ventures and Aurum Partners. The company is based in San Francisco.

“It’s a fundamental new technology as opposed to something that can make AI a bit more efficient,” Gupta said.

Even the most advanced LLMs are limited to processing millions of tokens at a time, while organizations “hold trillions of tokens across documents, videos, logs and databases,” Graphon said. Approaches such as retrieval augmented generation, or RAG, can tap relevant organizational data, but can’t reason about the way that data connects, according to Graphon.

Graphon said its system is designed to work with “effectively unlimited” data and that it is compatible with any foundation model or agent framework, making them more efficient. 

Learning from the robots

Khan drew inspiration from his earlier work in robotics, which he studied as a doctoral student at the University of Pennsylvania. If a robot is maneuvering in a defined space, such as a suburban cul-de-sac or an office hallway, an engineer can use knowledge of that real-world structure to reduce the computation that the robot employs as it understands its operating environment.

Khan applied a similar idea to data, drawing on the mathematical concept of the graphon from which the company takes its name. In social media, he said, a graphon can look at a massive user graph, identify disparate users and group them into “neighborhoods” based on shared relational properties, such as two people who go to the dog park on Sundays or have mutual friends who host dinner parties.

Transformers, the technology underlying the LLM, process text by converting every word into a number or token, and expend immense time and power figuring out which words are related. The transformer does all of that work repeatedly against a vast “context,” or the totality of the information a model can handle at one time in working memory.

Khan created a separate intelligence layer where the relational analysis can be done outside of the LLM.

“We’ll go build this big relational representation that will use the property of the graphon and will find these similar ‘neighborhoods’ of data and that is what is going to feed the model, instead of having the model do all of the heavy lifting of looking at all of the data,” Khan said.

It’s also a massive savings in terms of efficiency and compute, according to Khan. The approach uses a relatively small model to process tiny chunks of data. “So it’s a lot more efficient to run this 200 million [parameter model] a thousand times than it is to try and run like a 5 trillion [parameter model] for one hour,” he said.

_{Ally Kim, vice president at Korean conglomerate GS, has incorporated Graphon AI’s intelligence layer into its work. Steven Rosenbush/WSJ Leadership Institute}

The ability to work with larger volumes of data will be helpful as companies look beyond the application of AI to text, and unlock insights from voice and video.

GS, the Korean conglomerate, has employed Graphon within its 52g initiative, which focuses on digital and AI transformation, design thinking, prototyping and user experience across GS’s many lines of business, from convenience stores to oil refining. GS Futures, a Graphon investor, is an investment arm of the GS conglomerate.

GS Vice President Ally Kim, who leads 52g, said the team used Graphon to improve analysis of closed-circuit television recordings that monitor construction sites for safety compliance. And instead of having people spend hours watching raw video footage to vet candidates for a GS-sponsored soccer team, it used Graphon to more efficiently analyze player movements, strengths and weaknesses across various situations.

“We really need to expand our knowledge scope to multimodalities, like voice or video or other contexts. Graphon can be good support,” she said during a meeting at company headquarters in Seoul.