SubQuery

SubQuery is a fast, flexible, and reliable open-source data decentralised infrastructure network. The SubQuery Data indexer is a open-source data indexer that provides you with custom APIs for your web3 project across all of our supported chains.

Preparation

1、Nodejs

Download from Github

curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.39.1/install.sh | bash

Remember to add nvm to the shell environment.

Select node version

nvm use 20

2、Docker Compose

Download from Github

sudo curl -L https://github.com/docker/compose/releases/download/v2.25.0/docker-compose-linux-x86_64 -o /usr/local/bin/docker-compose

Execution permission

sudo chmod +x /usr/local/bin/docker-compose

Check the version

docker-compose --version

About SubQuery

1. Install the SubQuery CLI

Install SubQuery CLI globally on your terminal by using NPM. We do not encourage the use of yarn global for installing @subql/cli due to its poor dependency management. This may lead to multiple errors.

# NPM
npm install -g @subql/cli

# Test that it was installed correctly
subql --help

2. Initialise a new SubQuery Project

Run the following command inside the directory that you want to create a SubQuery project in:

subql init

You'll be asked certain questions as you proceed ahead:

  • Project name: A project name for your SubQuery project.

  • Network family: The layer-1 blockchain network family that this SubQuery project will index. Use the arrow keys to select from the available options (scroll down as there are multiple pages).

  • Network: The specific network that this SubQuery project will index. Use the arrow keys to select from the available options (scroll down as there are multiple pages).

  • Template project: Select a SubQuery template project that will provide a starting point in the development. For some networks we provide multiple examples.

  • RPC endpoint: Provide an HTTP or websocket URL to a running RPC endpoint, which will be used by default for this project. You can use public endpoints for different networks, your own private dedicated node, or just use the default endpoint. This RPC node must have the entire state of the data that you wish to index, so we recommend an archive node.

  • Git repository: Provide a Git URL to a repo that this SubQuery project will be hosted in.

  • Authors: Enter the owner of this SubQuery project here (e.g. your name!) or accept the provided default.

  • Description: Provide a short paragraph about your project that describes what data it contains and what users can do with it, or accept the provided default.

  • Version: Enter a custom version number or use the default (1.0.0).

  • License: Provide the software license for this project or accept the default (MIT).

Let’s look at an example:

Ethereum Project Scaffolding

You can generate a project from a JSON ABIs to save you time when creating your project in EVM chains. Please see EVM Project Scaffolding

After you complete the initialisation process, you will see a folder with your project name created inside the directory. Please note that the contents of this directory should be near identical to what's listed in the Directory Structure.

Finally, run the following command to install the new project’s dependencies from within the new project's directory.

cd PROJECT_NAME
npm install

You have now initialised your first SubQuery project with just a few simple steps. Let’s now customise the standard template project for a specific blockchain of interest.

You may want to refer to the command line arguments used in SubQuery. It will help you understand the commands better.

3. Run a template

We use Ethereum packages, runtimes, and handlers (e.g. @subql/node-ethereum, ethereum/Runtime, and ethereum/*Hander) for BEVM Canary Network. Since BEVM is an EVM-compatible layer-2 scaling solution, we can use the core Ethereum framework to index it.

Your Project Manifest File

The Project Manifest file is an entry point to your project. It defines most of the details on how SubQuery will index and transform the chain data.

For EVM chains, there are three types of mapping handlers (and you can have more than one in each project):

  • BlockHanders: On each and every block, run a mapping function

  • TransactionHandlers: On each and every transaction that matches optional filter criteria, run a mapping function

  • LogHanders: On each and every log that matches optional filter criteria, run a mapping function

As we are indexing all transfers and approvals from the Wrapped BTC contract on BEVM Canary Network, the first step is to import the contract abi definition which can be obtained from from any standard ERC-20 contractopen in new window. Copy the entire contract ABI and save it as a file called erc20.abi.json in the /abis directory.

Update the datasources section as follows:

{
  dataSources: [
    {
      kind: EthereumDatasourceKind.Runtime,
      startBlock: 9680021,
      options: {
        // Must be a key of assets
        abi: "erc20",
        // This is the contract address for Wrapped BTC https://scan-canary.bevm.io/address/0x09Ff8E49D0EA411A3422ed95E8f5497D4241F532
        address: "0x09Ff8E49D0EA411A3422ed95E8f5497D4241F532",
      },
      assets: new Map([["erc20", { file: "./abis/erc20.abi.json" }]]),
      mapping: {
        file: "./dist/index.js",
        handlers: [
          {
            kind: EthereumHandlerKind.Call,
            handler: "handleTransaction",
            filter: {
              /**
               * The function can either be the function fragment or signature
               * function: '0x095ea7b3'
               * function: '0x7ff36ab500000000000000000000000000000000000000000000000000000000'
               */
              function: "approve(address spender, uint256 rawAmount)",
            },
          },
          {
            kind: EthereumHandlerKind.Event,
            handler: "handleLog",
            filter: {
              /**
               * Follows standard log filters https://docs.ethers.io/v5/concepts/events/
               * address: "0x60781C2586D68229fde47564546784ab3fACA982"
               */
              topics: [
                "Transfer(address indexed from, address indexed to, uint256 amount)",
              ],
            },
          },
        ],
      },
    },
  ],
}

The above code indicates that you will be running a handleTransaction mapping function whenever there is a approve method being called on any transaction from the Wrapped BTC Tokenopen in new window on BEVM's Canary Network.

The code also indicates that you will be running a handleLog mapping function whenever there is a Transfer event being emitted from the Wrapped BTC Tokenopen in new window on BEVM's Canary Network.

Check out our Manifest File documentation to get more information about the Project Manifest (project.ts) file.

Update Your GraphQL Schema File

The schema.graphql file determines the shape of your data from SubQuery due to the mechanism of the GraphQL query language. Hence, updating the GraphQL Schema file is the perfect place to start. It allows you to define your end goal right at the start.

Remove all existing entities and update the schema.graphql file as follows. Here you can see we are indexing block information such as the id, blockHeight, transfer receiver and transfer sender along with an approvals and all of the attributes related to them (such as owner and spender etc.).

type Transfer @entity {
  id: ID! # Transaction hash
  blockHeight: BigInt
  to: String!
  from: String!
  value: BigInt!
  contractAddress: String!
}

type Approval @entity {
  id: ID! # Transaction hash
  blockHeight: BigInt
  owner: String!
  spender: String!
  value: BigInt!
  contractAddress: String!
}

Importantly, these relationships can not only establish one-to-many connections but also extend to include many-to-many associations. To delve deeper into entity relationships, you can refer to this section. If you prefer a more example-based approach, our dedicated Hero Course Module can provide further insights.

SubQuery simplifies and ensures type-safety when working with GraphQL entities, smart contracts, events, transactions, and logs. The SubQuery CLI will generate types based on your project's GraphQL schema and any contract ABIs included in the data sources.

npm run-script codegen

This action will generate a new directory (or update the existing one) named src/types. Inside this directory, you will find automatically generated entity classes corresponding to each type defined in your schema.graphql. These classes facilitate type-safe operations for loading, reading, and writing entity fields. You can learn more about this process in the GraphQL Schema section.

It will also generate a class for every contract event, offering convenient access to event parameters, as well as information about the block and transaction from which the event originated. You can find detailed information on how this is achieved in the EVM Codegen from ABIs section. All of these types are stored in the src/types/abi-interfaces and src/types/contracts directories.

You can conveniently import all these types:

import { Approval, Transfer } from "../types";
import {
  ApproveTransaction,
  TransferLog,
} from "../types/abi-interfaces/Erc20Abi";

Check out the GraphQL Schema documentation to get in-depth information on schema.graphql file.

Now that you have made essential changes to the GraphQL Schema file, let’s proceed ahead with the Mapping Function’s configuration.

Add a Mapping Function

Mapping functions define how blockchain data is transformed into the optimised GraphQL entities that we previously defined in the schema.graphql file.

Navigate to the default mapping function in the src/mappings directory. You will be able to see two exported functions handleLog and handleTransaction:

import { Approval, Transfer } from "../types";
import {
  ApproveTransaction,
  TransferLog,
} from "../types/abi-interfaces/Erc20Abi";
import assert from "assert";

export async function handleLog(log: TransferLog): Promise<void> {
  logger.info(`New transfer transaction log at block ${log.blockNumber}`);
  assert(log.args, "No log.args");

  const transaction = Transfer.create({
    id: log.transactionHash,
    blockHeight: BigInt(log.blockNumber),
    to: log.args.to,
    from: log.args.from,
    value: log.args.value.toBigInt(),
    contractAddress: log.address,
  });

  await transaction.save();
}

export async function handleTransaction(tx: ApproveTransaction): Promise<void> {
  logger.info(`New Approval transaction at block ${tx.blockNumber}`);
  assert(tx.args, "No tx.args");

  const approval = Approval.create({
    id: tx.hash,
    owner: tx.from,
    spender: await tx.args[0],
    value: BigInt(await tx.args[1].toString()),
    contractAddress: tx.to,
  });

  await approval.save();
}

The handleLog function receives a log parameter of type TransferLog which includes log data in the payload. We extract this data and then save this to the store using the .save() function (Note that SubQuery will automatically save this to the database).

The handleTransaction function receives a tx parameter of type ApproveTransaction which includes transaction data in the payload. We extract this data and then save this to the store using the .save() function (Note that SubQuery will automatically save this to the database).

For more information on mapping functions, please refer to our Mappings documentation.

Build Your Project

Next, build your work to run your new SubQuery project. Run the build command from the project's root directory as given here:

npm run-script build

Whenever you make changes to your mapping functions, you must rebuild your project.

Now, you are ready to run your first SubQuery project. Let’s check out the process of running your project in detail.

Whenever you create a new SubQuery Project, first, you must run it locally on your computer and test it and using Docker is the easiest and quickiest way to do this.

Run Your Project Locally with Docker

The docker-compose.yml file defines all the configurations that control how a SubQuery node runs. For a new project, which you have just initialised, you won't need to change anything.

However, visit the Running SubQuery Locally to get more information on the file and the settings.

Run the following command under the project directory:

npm run-script start:docker

It may take a few minutes to download the required images and start the various nodes and Postgres databases.

Query your Project

Next, let's query our project. Follow these three simple steps to query your SubQuery project:

  1. Open your browser and head to http://localhost:3000.

  2. You will see a GraphQL playground in the browser and the schemas which are ready to query.

  3. Find the Docs tab on the right side of the playground which should open a documentation drawer. This documentation is automatically generated and it helps you find what entities and methods you can query.

Try the following queries to understand how it works for your new SubQuery starter project. Don’t forget to learn more about the GraphQL Query language.

# Write your query or mutation here
{
  query {
    transfers(first: 5, orderBy: VALUE_DESC) {
      totalCount
      nodes {
        id
        blockHeight
        from
        to
        value
        contractAddress
      }
    }
  }
}

You will see the result similar to below:

{
  "data": {
    "query": {
      "transfers": {
        "totalCount": 5,
        "nodes": [
          {
            "id": "0x625fd9f365a1601486c4176bc34cf0fdf04bf06b2393fd5dd43e8dd7a62d9ec5",
            "blockHeight": "53",
            "from": "0x0000000000000000000000000000000000000000",
            "to": "0xb680c8F33f058163185AB6121F7582BAb57Ef8a7",
            "value": "1000000000000000000000000",
            "contractAddress": "0x28687c2A4638149745A0999D523f813f63b4786F"
          },
          {
            "id": "0x32057c64d795a7f919925082b9cdc885e307e3a4590377154d746beadc557d3e",
            "blockHeight": "62",
            "from": "0xb680c8F33f058163185AB6121F7582BAb57Ef8a7",
            "to": "0xCa8c45FE7FEDc3922266A1964cD8B8D29946A6A7",
            "value": "300000000000000000000",
            "contractAddress": "0x28687c2A4638149745A0999D523f813f63b4786F"
          },
          {
            "id": "0xc591997f3217f6dfb6d4dad244126ad4ce245234fe452339b5ba8ad4d4264bdc",
            "blockHeight": "66",
            "from": "0xCa8c45FE7FEDc3922266A1964cD8B8D29946A6A7",
            "to": "0xb21aBf688A6bE0975134a41e73bf2c8Da111fF0d",
            "value": "50000000000000000000",
            "contractAddress": "0x28687c2A4638149745A0999D523f813f63b4786F"
          },
          {
            "id": "0x1e29daac0434ad4936391e7ba439146ecd9ff9d65869436d466a8e48963e420a",
            "blockHeight": "67",
            "from": "0xCa8c45FE7FEDc3922266A1964cD8B8D29946A6A7",
            "to": "0xe42A2ADF3BEe1c195f4D72410421ad7908388A6a",
            "value": "50000000000000000000",
            "contractAddress": "0x28687c2A4638149745A0999D523f813f63b4786F"
          },
          {
            "id": "0x73e95b32fe50daf7d0480a7dbd3005fcf22007ebff82fc6fa06a0c606783a0e3",
            "blockHeight": "68",
            "from": "0xe42A2ADF3BEe1c195f4D72410421ad7908388A6a",
            "to": "0x6F715c294Dd78BB11aeB0817B44E2a0b06e3A0B4",
            "value": "1000000000000000000",
            "contractAddress": "0x28687c2A4638149745A0999D523f813f63b4786F"
          }
        ]
      }
    }
  }
}

The final code of this project can be found hereopen in new window.

The final code of this project can be found here.

5. Make Changes to Your Project

There are 3 important files that need to be modified. These are:

  1. The GraphQL Schema in schema.graphql.

  2. The Project Manifest in project.ts.

  3. The Mapping functions in src/mappings/ directory.

6. EVM Project Scaffolding

Scaffolding saves time during SubQuery project creation by automatically generating typescript facades for EVM transactions, logs, and types.

When you are initalising a new project using the subql init command, SubQuery will give you the option to set up a scaffolded SubQuery project based on your JSON ABI. If you select a compatible network type (EVM), it will prompt:

? Do you want to generate scaffolding with an existing abi contract?

For example, to create the Ethereum Gravatar indexer, download the Gravity ABI contract JSON from Etherscanopen in new window, save it as Gravity.json, and then run the following:

Once completed, you will have a scaffold project structure from your chosen ABI functions/events.

You can read more about this feature in Project Scaffolding

PreviousIndexerNextAccount Abstraction

Last updated