DIA Nexus Documentation
  • Nexus Overview
  • Intro to Nexus
    • How it works
    • Nexus vs. Lumina
    • Integrated Chains
  • Data products
    • Token Price Feeds
    • RWA Price Feeds
    • Randomness
    • Fair-value Feeds
  • How-to Guides
    • Fetch Price Data
      • Solidity
      • Vyper
      • Demo Oracles
      • Chain-specific Guide
        • Aleph Zero
        • Alephium
        • Edu Chain
        • Hydration
        • Kadena
        • LUKSO
        • Somnia
        • Stacks
        • Superseed
        • XRP Ledger (XRPL)
    • Generate Randomness
      • Solidity
      • WASM
      • Demo Oracles
      • Chain-specific Guide
        • Alephium
    • Migrate to DIA
    • Fund the Oracle
    • Build a Scraper
      • Development Cluster Stack
      • DIA Test‐Space with Docker Compose
      • DIA Test‐Space with Minikube
      • Add a new exchange scraper
      • Add a new foreign scraper
      • Add a new liquidity scraper
      • Additional notes
  • Request a Custom Oracle
  • Reference
    • Architecture
      • Data sourcing
      • Data computation
      • Data delivery
    • APIs
      • Token prices
        • RestAPI
          • Request Samples
        • GraphQL
          • Request Samples
      • RWA prices
    • Pricing Methodologies
      • IR: Interquartile Range Filter
      • MAIR: Moving Average with Interquartile Range Filter
      • VWAP: Volume Weighted Average Price
      • VWAPIR: Volume Weighted Average Price with Interquartile Range Filter
      • LST fair price
    • Data Sources
      • CEXes Data
      • DEXes Data
    • Smart Contracts
      • DIAOracleV2.sol
      • DIARandomOracle.sol
    • Randomness Protocol
  • Resources
    • Audits
    • Community & Support
    • Security Bounty Program
    • Research
      • Return Rates in Crypto Farming
      • Crypto Volatility Index
      • Compounded Rates
      • Polkadot Medianizer
    • T&C
      • Licence Agreement
      • Contributor Covenant Code of Conduct
      • Disclaimer
Powered by GitBook
On this page
  • Prepare
  • Scrapers
  • CEX (Kraken as example)
  • DEX (CurveFi as example)
  • Clean
  • Advanced usage
Export as PDF
  1. How-to Guides
  2. Build a Scraper

DIA Test‐Space with Docker Compose

Developing, Testing, and Building the DIA platform with Docker Compose

Prepare

# Start data store services
(
  cd deployments/local/exchange-scraper
  docker compose -f docker-compose.yml up --build --force-recreate -d
)

# Define env variables needed
export USE_ENV=true
export INFLUXURL=http://localhost:8086
export INFLUXUSER=test
export INFLUXPASSWORD=testtest
export POSTGRES_USER=postgres
export POSTGRES_PASSWORD=password
export POSTGRES_HOST=localhost
export POSTGRES_DB=postgres
export REDISURL=localhost:6379

# Set config directory
export DIA_CONFIG_DIR=./config

# Set local directory as Go dependency
go mod edit -replace "github.com/diadata-org/diadata=$(pwd)" cmd/exchange-scrapers/collector/go.mod

Scrapers

CEX (Kraken as example)

Prepare

# Initialize the exchange and blockchain metadata
go run ./cmd/services/blockchainservice/main.go

# Prepare pairs metadata
go run ./cmd/services/pairDiscoveryService/main.go -exchange=Kraken -mode=verification

Run the scraper

go run ./cmd/exchange-scrapers/collector/collector.go -exchange=Kraken -mode=storeTrades

DEX (CurveFi as example)

Prepare

# Set needed RPC variables
export ETHEREUM_URI_REST=https://rpc.ankr.com/eth
export ETHEREUM_URI_WS=wss://eth-mainnet.g.alchemy.com/v2/7X9hL_lpF-Utavuv1aT49SWkWu9oYZTq

# Prepare the assets metadata
go run ./cmd/assetCollectionService/main.go -source=assetlists -secret=eth_assets -caching=true
go run ./cmd/assetCollectionService/main.go -source=assetlists -secret=non_eth_assets -caching=true
go run ./cmd/assetCollectionService/main.go -source=assetlists -secret=fiat_assets -caching=true
go run ./cmd/assetCollectionService/main.go -source=Curvefi -caching=true

# Prepare and populate pools
go run ./cmd/liquidityScraper/main.go -exchange=Curvefi

Run the scraper

go run ./cmd/exchange-scrapers/collector/collector.go -exchange=Curvefi -mode=storeTrades

Clean

unset USE_ENV
unset INFLUXURL
unset INFLUXUSER
unset INFLUXPASSWORD
unset POSTGRES_USER
unset POSTGRES_PASSWORD
unset POSTGRES_HOST
unset POSTGRES_DB
unset REDISURL
unset DIA_CONFIG_DIR
unset ETHEREUM_URI_REST
unset ETHEREUM_URI_WS

Advanced usage

Access services between Docker Composes

On prepare step if you want to connect services between other Compose files, create a deployments/local/exchange-scraper/docker-compose.override.yml file:

version: "3.3"

services:
  influxdb:
    networks:
      - dia-bridge

networks:
  dia-bridge:
    driver: bridge

And instead of run the services using this command:

# Start data store services
(
  cd deployments/local/exchange-scraper
  docker compose -f docker-compose.yml -f docker-compose.override.yml up --build --force-recreate -d
)

Now in other Compose file just use like this:

version: "3"

networks:
  exchange-scraper_dia-bridge:
    external: true

services:
  service-x:
    image: service-x-image
    networks:
      - exchange-scraper_dia-bridge

Now in Docker context, service-x can resolve the http://influxdb:8086 hostname

PreviousDevelopment Cluster StackNextDIA Test‐Space with Minikube

Last updated 2 months ago