DIA Nexus Documentation
  • Nexus Overview
  • Intro to Nexus
    • How it works
    • Nexus vs. Lumina
    • Integrated Chains
  • Data products
    • Token Price Feeds
    • RWA Price Feeds
    • Randomness
    • Fair-value Feeds
  • How-to Guides
    • Fetch Price Data
      • Solidity
      • Vyper
      • Demo Oracles
      • Chain-specific Guide
        • Aleph Zero
        • Alephium
        • Edu Chain
        • Hydration
        • Kadena
        • LUKSO
        • Somnia
        • Stacks
        • Superseed
        • XRP Ledger (XRPL)
    • Generate Randomness
      • Solidity
      • WASM
      • Demo Oracles
      • Chain-specific Guide
        • Alephium
    • Migrate to DIA
    • Fund the Oracle
    • Build a Scraper
      • Development Cluster Stack
      • DIA Test‐Space with Docker Compose
      • DIA Test‐Space with Minikube
      • Add a new exchange scraper
      • Add a new foreign scraper
      • Add a new liquidity scraper
      • Additional notes
  • Request a Custom Oracle
  • Reference
    • Architecture
      • Data sourcing
      • Data computation
      • Data delivery
    • APIs
      • Token prices
        • RestAPI
          • Request Samples
        • GraphQL
          • Request Samples
      • RWA prices
    • Pricing Methodologies
      • IR: Interquartile Range Filter
      • MAIR: Moving Average with Interquartile Range Filter
      • VWAP: Volume Weighted Average Price
      • VWAPIR: Volume Weighted Average Price with Interquartile Range Filter
      • LST fair price
    • Data Sources
      • CEXes Data
      • DEXes Data
    • Smart Contracts
      • DIAOracleV2.sol
      • DIARandomOracle.sol
    • Randomness Protocol
  • Resources
    • Audits
    • Community & Support
    • Security Bounty Program
    • Research
      • Return Rates in Crypto Farming
      • Crypto Volatility Index
      • Compounded Rates
      • Polkadot Medianizer
    • T&C
      • Licence Agreement
      • Contributor Covenant Code of Conduct
      • Disclaimer
Powered by GitBook
On this page
  • Access data endpoints and UIs
  • Contribute to DIA main package
  • Debug Influx writes
  • Use a custom pre-populated image data
Export as PDF
  1. How-to Guides
  2. Build a Scraper

Additional notes

PreviousAdd a new liquidity scraperNextRequest a Custom Oracle

Access data endpoints and UIs

Kubernetes Dashboard

To start the Kubernetes Dashboard, use the following command:

minikube dashboard --url=true --port=8083

Once the dashboard service is ready, you can visit the web interface in your browser.

Data stores

Forward the ports of the data stores services to localhost:

  • PostgreSQL Database: kubectl port-forward deployment/postgres 5432:5432

  • Redis Cache: kubectl port-forward deployment/redis 6379:6379

  • InfluxDB Database: kubectl port-forward deployment/influx 8086:8086

Other routes available after forward:

  • REST Server: port 8081

  • GraphQL: port 1111, and

  • Kafka: port 8080, and

  • Grafana: port 3000, and

Contribute to DIA main package

go mod edit -replace github.com/diadata-org/diadata=/mnt/env-context

Debug Influx writes

To debug InfluxDB writes, just change points in batch to see more frequent writes to influx (pkg/model/db.go):

influxMaxPointsInBatch = 10

Use a custom pre-populated image data

A .testenv.local file at root folder is required to pull private images from Docker:

data_docker_registry=docker.io
data_docker_username=my_user
data_docker_password="my_password"
data_docker_email=my_email@domain.com

Change image of pre-populated data from file

Web UI
AKHQ Web UI
Web UI
deployments/k8s-yaml/data-postgres-prepopulated.yaml