lily_setup.md

Lily Mainnet Maintenance

Hardware Minimum Requirements

You'll need a machine with at least 24 cores, 512GB RAM, and a 1TB (or bigger) hard drive with 10,000+ IOPS.

Installation Installation Dependencies

Instal related packages

$ apt update
$ apt install -y build-essential clang ocl-icd-opencl-dev ocl-icd-libopencl1 jq libhwloc-dev g++ gcc libc6-dev pkg-config aria2

Install Go

$ wget https://go.dev/dl/go1.22.9.linux-amd64.tar.gz
$ sudo tar -C /usr/local -xzf go1.22.9.linux-amd64.tar.gz
$ export PATH=$PATH:/usr/local/go/bin

Install Lily and Compile

Download Lily

$ git clone https://github.com/filecoin-project/lily.git
$ export CGO\_ENABLED=1

Compile

$ make clean all \# for mainnet
$ make clean calibnet \# for calibration
$ sudo cp lily /usr/local/bin/

Configure sysctl

We are using the systemctl to control lily

Create Config

Create sysctl config

touch /etc/sysctl.conf

setting:

net.core.somaxconn = 32768
net.core.netdev\_max\_backlog = 32768
net.ipv4.tcp\_max\_syn\_backlog = 16384
net.core.rmem\_max = 2500000
net.core.wmem\_max = 2500000

run

$ sudo sysctl -p

Configure Database

Install TimescaleDB with Docker Compose

services:
 timescaledb:
 container_name: timescaledb
 image: timescale/timescaledb:2.14.2-pg15
 environment:
 - POSTGRES_PASSWORD=password
 volumes:
 - /data/timescaledb:/var/lib/postgresql/data
 healthcheck:
 test: ["CMD-SHELL", "pg_isready -U postgres"]
 interval: 10s
 timeout: 5s
 retries: 5
 network_mode: "host"
 restart: always
 shm_size: 2g

Create Role and Database

$ docker exec -it timescaledb bash
$ psql -U postgres -c "CREATE ROLE lily WITH PASSWORD
'lily-password' LOGIN CREATEDB"
$ psql -U lily -c "CREATE DATABASE lily" postgres
$ exit

Initialization

$ export
LILY_DB=postgres://$PGUSER:$PGPASSWORD@$PGHOST:$PGPOR
T/$PGDATABASE?sslmode=disable
$ /usr/local/bin/lily migrate --db "$LILY_DB" --latest

Configure Lily

$ mkdir -p /etc/lily
$ cat /etc/lily/config.toml

[API]
 ListenAddress = "/ip4/0.0.0.0/tcp/1234/http"
[Backup]
[Libp2p]
 ListenAddresses = ["/ip4/0.0.0.0/tcp/1347"]
[Pubsub]
[Client]
[Metrics]
[Chainstore]
 EnableSplitstore = true
 [Chainstore.Splitstore]
 ColdStoreType = "discard"
[Fevm]
 EnableEthRPC = true
[Events]
 EnableActorEventsAPI = true
[Storage]
 [Storage.Postgresql]
 [Storage.Postgresql.Database1]
 URL = "postgres://lily:password@127.0.0.1::5432/lily?sslmode=disable"
 ApplicationName = "lily"
 SchemaName = "public"
 PoolSize = 20
 AllowUpsert = true

Run Lily

Start Daemon

$ export LILY_REPO=/data/lily
$ export LILY_CONFIG=/etc/lily/config.toml
$ export GOLOG_FILE=/data/lily/lily.log
$ export LOTUS_VM_ENABLE_TRACING=1
$ lily daemon &
$ lily sync wait

Watch Jobs

$ export
tasks="actor,actor_state,block_header,block_message,builtin_actor_
event,chain_economics,chain_economics_v2,chain_power,chain_rew
ard,data_cap_balance,derived_gas_outputs,fevm_actor_stats,id_addr
esses,market_deal_proposal,market_deal_state,message,message_g
as_economy,miner_beneficiary,miner_fee_debt,miner_info,miner_loc
ked_fund,miner_sector_deal_v2,miner_sector_event,miner_sector_inf
os_v7,multisig_approvals,multisig_transaction,parsed_message,pow
er_actor_claim,receipt,verified_registry_verifier,vm_messages"
$ lily run --restart-on-failure --storage=Database1 --tasks="$tasks"
watch --confidence=3

Fill Missing Data

Walk

$ lily job run --storage=Database1 --tasks=$tasks walk --from=$min --to=$max

Gapfill

$ lily job run --storage=Database1 --tasks=$tasks find --from=$min --to=$max
$ lily job run --storage=Database1 --tasks=$tasks fill --from=$min --to=$max

Export Data

Perform Gapfill Before Exporting

Option 1: Provide the Starboard team with a TimescaleDB account and restrict access via an IP whitelist to allow them to connect from specified IP addresses.

Option 2: Output the parquet format. Install pg2parquet to export data.

$ tables='
actors
actor_states
block_headers
block_messages
builtin_actor_events
chain_economics
chain_economics_v2
chain_powers
chain_rewards
data_cap_balances
derived_gas_outputs
fevm_actor_stats
id_addresses
market_deal_proposals
market_deal_states
message_gas_economy
messages
miner_beneficiaries
miner_fee_debts
miner_infos
miner_locked_funds
miner_sector_events
miner_sector_deals_v2
miner_sector_infos_v7
multisig_approvals
multisig_transactions
parsed_messages
power_actor_claims
receipts
verified_registry_verifiers
vm_messages
'
$ for table in $tables; \
do \
pg2parquet export  -q \
"select * from $table where height between $min_height and $max_height" \
--compression zstd \
--compression-level 9 \
--output-file $backup_dir/$backup_file \
--host $PGHOST \
--dbname $PGDATABASE \
--quiet; \
done