The Evolution and Future Prospects of Blockchain Data Indexing Technology

1. Introduction

From the earliest blockchain applications to the flourishing financial, gaming, and social dApps today, the blockchain ecosystem has undergone tremendous changes. In this process, the data sources that dApp interactions rely on have gradually become the focus of industry attention.

In 2024, the integration of AI and Web3 has become a hot topic. In the field of artificial intelligence, data is like the source of life for its growth and evolution. Just as plants need sunlight and water to thrive, AI systems also rely on vast amounts of data to continuously learn and think. Without data support, even the most sophisticated AI algorithms find it difficult to demonstrate their expected intelligence and effectiveness.

This article will delve into the development history of Blockchain data accessibility, analyze the evolution of data indexing technologies in the industry, and compare several major data indexing protocols, with a particular focus on how emerging protocols leverage AI technology to optimize data services and product architecture.

2. The Evolution of Data Indexing: From Nodes to Full-Chain Database

2.1 Data Source: Blockchain Node

Blockchain is often described as a decentralized ledger. Nodes are the foundation of the entire network, responsible for recording, storing, and disseminating all on-chain transaction data. Each node keeps a complete copy of the blockchain data, ensuring the decentralized nature of the network. However, for ordinary users, building and maintaining a node is not an easy task, as it requires specialized skills and involves high hardware and bandwidth costs. Additionally, the query capabilities of ordinary nodes are limited, making it difficult to meet the needs of developers.

To solve this problem, RPC node providers have emerged. They bear the costs and management of nodes and provide data services through RPC endpoints. Although public RPC endpoints are free, they have rate limits that may affect the user experience of dApps. Private RPC endpoints offer better performance, but they are less efficient for complex queries and difficult to scale and achieve cross-network compatibility. Nevertheless, the standardized API interfaces of node providers lower the threshold for users to access on-chain data, laying the foundation for subsequent data parsing and application.

2.2 Data Parsing: From Raw Data to Usable Data

The raw data provided by blockchain nodes is often encrypted and encoded, which ensures the integrity and security of the data but increases the difficulty of parsing. For ordinary users and developers, directly handling this data requires a significant amount of technical knowledge and computational resources.

The data parsing process becomes particularly important in this context. By converting complex raw data into a more understandable and operable format, users can utilize this data more intuitively. The quality of the parsing directly affects the efficiency and effectiveness of blockchain data applications, making it a key link in the entire data indexing process.

2.3 Development of Data Indexers

With the explosive growth of blockchain data, the demand for data indexers has become increasingly prominent. The main function of an indexer is to organize on-chain data and store it in a database for querying. They index blockchain data and provide SQL-like query language interfaces (such as GraphQL), making the data readily available. This unified query interface allows developers to quickly and accurately retrieve the information they need, greatly simplifying the entire process.

Different types of indexers each have their advantages:

1. Full Node Indexer: Extract data directly from full nodes to ensure data completeness and accuracy, but requires a large amount of storage and processing power.
2. Lightweight Indexer: Relies on full nodes to obtain specific data, reducing storage requirements but may increase query time.
3. Dedicated Indexer: Optimized for specific types of data or Blockchain, such as NFT data or DeFi transactions.
4. Aggregated Indexer: Extracts data from multiple blockchains and sources, including off-chain information, providing a unified query interface, suitable for multi-chain dApps.

Currently, the storage requirements for Ethereum archive nodes vary from 3TB to 13.5TB across different clients, and continue to increase as the Blockchain grows. In the face of such a massive amount of data, mainstream indexing protocols not only support multi-chain indexing but also customize data parsing frameworks for different application needs.

The emergence of indexers has significantly improved data indexing and query efficiency. Compared to traditional RPC endpoints, indexers can efficiently handle large volumes of data and support complex queries and data filtering. Some indexers also support aggregating multi-chain data sources, avoiding the issue of multi-chain dApps needing to deploy multiple APIs. By operating in a distributed manner, indexers provide stronger security and performance, reducing the risk of interruptions that centralized RPC providers may cause.

2.4 Full-Chain Database: Aligning to Stream Priority

As the project scale expands, standardized APIs struggle to meet the increasingly complex query requirements, such as searching, cross-chain access, or off-chain data mapping. The "stream-first" approach in modern data pipeline architecture has become a solution to overcome the limitations of traditional batch processing, enabling real-time data processing and analysis.

Blockchain data service providers are also moving towards building data streams. Traditional indexer service providers have launched real-time blockchain data stream products, such as Substreams of a certain protocol and Mirror of a certain company. At the same time, emerging service providers like a certain data platform and a certain protocol also offer real-time data lakes generated based on blockchain.

These services are designed to address the need for real-time analysis of Blockchain transactions and to provide comprehensive query capabilities. By re-examining on-chain data management from the perspective of modern data pipelines, we can explore more possibilities for data storage and utilization. Viewing indexers like Subgraph and Ethereum ETL as data flows rather than final outputs opens up new possibilities for customizing high-performance datasets.

3. The Combination of AI and Databases: A Comparison of Major Protocols

3.1 A certain decentralized indexing protocol

The protocol provides multi-chain data indexing and querying services through a decentralized network of nodes. Its core products include a data query execution market and a data index caching market, serving users' querying needs.

The foundational data structure of the protocol is the "subgraph", which defines how to extract and transform data from the Blockchain into a queryable format. The network consists of four roles: indexers, curators, delegators, and developers, ensuring the system operates through economic incentives.

The protocol has recently made breakthroughs in AI applications. The core development team of the ecosystem has developed several AI tools, such as a dynamic pricing mechanism, resource allocation optimizer, and natural language query tool, enhancing the system's intelligence and user-friendliness.

3.2 A Full Blockchain Data Network

This is a platform that integrates all Blockchain data, offering features such as a real-time data lake, dual-chain architecture, innovative data format standards, and a cryptographic world model.

The platform is built on a certain technology execution layer, forming a parallel dual-chain architecture with a certain consensus algorithm, enhancing the programmability and composability of cross-chain data. The platform introduces a new data format standard called "manuscripts," optimizing the structuring and utilization of data in the encryption industry.

The platform combines AI model technology to create an AI model capable of understanding, predicting Blockchain transactions, and interacting with them. A basic version of the model has been launched for public use, which is based on technology developed by a certain company, integrating on-chain and off-chain data with temporal and spatial activities to deeply explore the potential value and patterns of on-chain data.

3.3 A Verifiable Computing Layer

The project aims to create a verifiable computing layer that extends zero-knowledge proofs on a decentralized data warehouse, providing reliable data processing for smart contracts, large language models, and enterprises.

The project introduces innovative zero-knowledge proof technology, ensuring that SQL queries executed on decentralized data warehouses are tamper-proof and verifiable. This technology changes the traditional way blockchain networks rely on consensus mechanisms to verify data authenticity, enhancing the overall performance of the system.

The project collaborates with a large tech company's AI laboratory to develop generative AI tools that simplify the process for users to process blockchain data through natural language. Users can input natural language queries, and the AI automatically converts them into SQL and executes the queries to present the final results.

Conclusion and Outlook

The blockchain data indexing technology has evolved from the source of node data, to data parsing and indexing, and then to AI-enabled full-chain data services. This process has continuously improved the efficiency and accuracy of data access, providing users with a more intelligent experience.

In the future, with the development of new technologies such as AI technology and zero-knowledge proofs, blockchain data services will become further intelligent and secure. As infrastructure, these services will continue to provide important support for industry progress and innovation.