Tech

The graph - 區塊鏈數據索引協議

Graph node 搭建、subgraph 撰寫及部署

前言

開開心心學 the graph

從 subgraph 撰寫、搭建本地 graph node 到部署 subgraph

Introduction

The Graph is a decentralized protocol for indexing and querying data from blockchains, starting with Ethereum.

The graph 為一個去中心化索引協議，用於索引和查詢區塊鏈的數據。

It makes it possible to query data that is difficult to query directly.

The graph 使那些難以直接查詢的數據變成可能。

舉例來說，我們想要取得某個用戶在 Uniswap 在各個 pool 有多少資產，過去可以透過連接鏈上 endpoint、發起 RPC call、調用合約接口達成。具體來說：

  1. 取得總共有幾個 pool
  2. 取得所有 pool 的合約地址
  3. 遍歷所有 pool 合約取得該用戶的 balance

但是這麼做有幾個缺點：

  • endpoint 的 request 時間可能較長
  • 需發出多個 RPC call，畢竟可是有好幾百個 pool…

以上例子利用 the graph 的話只需要一個 query 就能達成:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19

{
  users(where: {id:"0x4de7009354f32b950052d4556695c04449820849"}) {
    id
    liquidityPositions{
      liquidityTokenBalance
      pair {
        token0 {
          symbol
          totalSupply
        }

        token1 {
          symbol
          totalSupply
        }
      }
    }
  }
}

可以搭配 Debank 看看結果

The graph 除了可以解決某些情境的效能瓶頸，有時候根據合約接口甚至沒辦法取得我們需要的數據，所以需要根據需求提前構建索引，the graph 則會根據規則索引鏈上數據，使用者可以透過 GraphQL API 查詢數據。

題外話， multicall 可以參考兔子哥：https://github.com/banteg/multicall.py

The graph protocol

The Graph 根據子圖描述（subgraph manifest）來學習什麼以及如何索引鏈上數據。Subgraph manifest 定義了子圖的 data source (智能合約)，以及此合約要處理的 event，還有如何將 event 數據 mapping 到資料庫。

查詢及索引都是透過 graph node 完成，啟動後會連接區塊鏈節點及 IPFS。

以下舉個例子：

  1. 定義了一個會記錄某 token 所有 transfer event 的子圖。
  2. 用戶透過一筆 transaction 轉了 token 給另一個人（transfer)。
  3. 當合約處理此交易的時候觸發了 transfer event。
  4. Graph node 持續掃描區塊。
  5. 當某區塊包含 transfer event，graph node 會執行 mapping 處理程序，並將處理好的資料儲存。
  6. 透過 GraphQL API 可以查詢到索引下來的數據。

Graph node 部署

Running a graph node using Docker

https://github.com/graphprotocol/graph-node/tree/master/docker

注意，使用 Macbook M1 請見此連結重新 build image: https://github.com/graphprotocol/graph-node/tree/v0.26.0/docker#running-graph-node-on-an-macbook-m1

1
2
3

git clone https://github.com/graphprotocol/graph-node.git
cd docker
docker-compose up

Running a local graph node

rust 安装

1

curl --proto '=https' --tlsv1.2 https://sh.rustup.rs -sSf | sh

Postgresql 安装

1

brew install postgresql

IPFS 安裝

1

brew install ipfs

Start Up an IPFS Node

1
2

ipfs init
ipfs daemon

Create the Postgres databas

1
2
3

initdb -D .postgres
pg_ctl -D .postgres -l logfile start
createdb graph-node

Start the Graph Node

1
2
3
4
5
6

git clone https://github.com/graphprotocol/graph-node.git
cd graph-node
cargo run -p graph-node --release -- \
  --postgres-url postgresql://USERNAME:PASSWORD@127.0.0.1:5432/graph-node \
  --ethereum-rpc mainnet:<ETHEREUM_RPC> \
  --ipfs 127.0.0.1:5001

Try your OS username as USERNAME and PASSWORD

Subgraph

以 WETH token 為例： https://github.com/ashirleyshe/subgraph/tree/main/weth-erc20

於 Subgraph studio 的部署：https://api.studio.thegraph.com/query/27930/weth-subgraph/v0.1.1/graphql

需要寫的部分：

  1. subgraph.yaml: subgraph manifest
  2. schema.graphql: GraphQL schema 定義子圖資料(entity)如何儲存及如何透過 GraphQL query
  3. AssemblyScript Mappings: AssemblyScript code 將 event data 轉成 schema 中的 entity

Reference: https://docs.openzeppelin.com/subgraphs/0.1.x/

使用 Openzeppelin subgraph library

當要支援多個 erc20 時，編寫 manifest 可以藉由套件完成(@amxx/graphprotocol-utils):

  1. 先準備config，裡面填寫 datasource 也就是 token address，範例：https://github.com/OpenZeppelin/openzeppelin-subgraphs/blob/main/configs/top-erc20.json

  2. 生成 schema 及 manifest

1
2
3
4
5

npx graph-compiler \
  --config sample.json \
  --include node_modules/@openzeppelin/subgraphs/src/datasources \
  --export-schema \
  --export-subgraph

部署 subgraph

package.json

在 script 中增加 “create-local” “remove-local” “deploy-local” 方便後續部署

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

{
  "name": "gravity",
  "license": "UNLICENSED",
  "scripts": {
    "codegen": "graph codegen",
    "build": "graph build",
    "deploy": "graph deploy --node https://api.studio.thegraph.com/deploy/ gravity",
    "create-local": "graph create --node http://127.0.0.1:8020/ gravity",
    "remove-local": "graph remove --node http://127.0.0.1:8020/ gravity",
    "deploy-local": "graph deploy --node http://127.0.0.1:8020/ --ipfs http://127.0.0.1:5001 gravity"
  },
  "dependencies": {
    "@graphprotocol/graph-cli": "0.29.2",
    "@graphprotocol/graph-ts": "0.26.0"
  }
}

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

yarn

# Code Generation
yarn codegen

# Build
yarn build

yarn create-local
yarn deploy-local

完成 subgraph 部署后，便會從某個區塊開始構建索引

Checking Subgraph health

https://thegraph.com/docs/hostedservice/deploy-subgraph-hosted#checking-subgraph-health

subgraph sync status 可以在這邊看到：預設 port 8030/subgraph

Example query:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22

{
  indexingStatusForCurrentVersion(subgraphName: "org/subgraph") {
    synced
    health
    fatalError {
      message
      block {
        number
        hash
      }
      handler
    }
    chains {
      chainHeadBlock {
        number
      }
      latestBlock {
        number
      }
    }
  }
}

Performance Improvement

一些坑

HTTP error creating the subgraph: ECONNREFUSED

1
2
3
4
5
6
7

➜  weth-erc20 yarn create-local
yarn run v1.22.19
warning ../package.json: No license field
$ graph create --node http://localhost:8020/ weth-erc20
✖ HTTP error creating the subgraph: ECONNREFUSED
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

→ localhost 改成 127.0.0.1 即可

Failed to get entities from store: column c.block$ does not exist

1
2
3
4
5
6
7

{
  "errors": [
    {
      "message": "Failed to get entities from store: column c.block$ does not exist, query = /* qid: 431b3891025ffb2d-6dc8c4ef99f8cc6a */\nselect 'ERC20Deposit' as entity, to_jsonb(c.*) as data from (select  * \n  from \"sgd33\".\"erc20_deposit\" c\n where c.block$ <= $1\n\n order by \"id\"\n limit 100) c -- binds: [4752680]"
    }
  ]
}

→ graph node bug 新版本已修復

https://github.com/graphprotocol/graph-node/issues/3553

graph-ts graph-cli 版本

每個 subgraph 要額外定義，每個版本可能不一樣TT

Reference

comments powered by Disqus