Data sourcing

Granular trade data sourcing

By leveraging a network of WebSockets and decentralized node providers, DIA Nexus retrieves token trade ticks and metadata from exchanges. This, in turn, represents an enormous data availability (≈15bn trades/day) to, later on, enable DIA to capture the highest data precision and build flexible and customizable price feeds.

Trades for every asset pair get recorded and collected into blocks of 120 seconds into the DIA Nexus platform. For each block, a range of filters are applied. These are functions to get a single price point from the collection of trades in a block. For example, the Moving Average filter returns the volume-weighted moving average of the trades block.

Filters are calculated for each asset pair on each exchange individually, as well as for each asset on all exchanges combined. This combined filter result represents the result closest to the true "whole market" that can be determined by this system. Learn more about token filters here.

Type of data sources and their aggregation

DIA Nexus retrieves trading data from as close to the data source as possible in order to ensure the highest precision possible. The data retrieval is processed through a series of asset collectors:

• For centralised exchanges such as Coinbase, Kraken or Binance, DIA scrapers collect trades directly from the exchange’s databases. The data collection happens in retrieval periods from 1 to 7 seconds and varies from exchange to exchange. All trading data is retrieved through Rest APIs or WebSocket APIs, developed and maintained by the respective trading platform. More on CEX trade data collection.

• For decentralised exchanges, DIA collects data from decentralized exchanges on various blockchains by subscribing to swap events in the corresponding liquidity pools – LPs. In contrast to centralized exchanges, in decentralized exchanges, it is possible to retrieve trading data directly from the respective blockchain by subscribing to the corresponding pool events. Examples of sources are Uniswap, curve.finance or PancakeSwap. More on DEX trade data collection.

• For cross-chain bridges, we collect the swaps, enabling us to port prices and map assets across blockchains. Cross-Chain bridges allow for swapping a crypto asset on a given blockchain to the corresponding asset on a different blockchain. For instance, swap USDC on Ethereum to USDC on Arbitrum. Cross-Chain bridges can be seen as ranging somewhere between centralized and decentralized: Parts of the complete bridging transactions can be seen publicly on the respective blockchain. However, by its very nature, the bridging itself is done by a centralized entity.

On oracle feed manipulation and risk

One of the biggest risks for DeFi and Web3 applications is not having enough sources for its oracle and using a feed that relies on one single market. These single-source oracles, as explained in the beginning, are relatively easily exposed to price manipulations and man-in-the-middle attacks.

DIA is designed to avoid these scenarios by leveraging multiple markets as sources, including CEXs and DEXs and enabling the customisation of methodologies and sources to cater best for the needs of specific use cases. So far in DeFi’s (short) history, a failure in the oracle node network has never been the reason for a loss of funds, but attacks on the markets that provide the price data to the sources. Take for instance the example of the Cream Finance attack in October 2021.

DIA prioritises the transparency aspect of its data and oracle services and enables any dApp developer and user to know precisely how the data feeding the application is built, what data sources are been used, and what methodologies are been applied.