Bulk Data Collection Example

This example demonstrates how someone can upload timestamps and automate the collection of data from NTerminal with Splunk Enterprise.

Data collection

• Random timestamps are taken.

  • They are collected in a .csv file under the column header timestamp
  • timestamps are numbered and in %Y-%m-%dT%H:%M:%S.%Q" format
  • The example table can be seen below:

  number	timestamp
  1	2019-02-11T12:00:00.000+00:00
  2	2019-01-11T12:00:00.000+00:00

• The .csv file is uploaded to splunk (via Settings>Lookups>"+Add New" under “Lookup table files”)

  • Note: “.csv” must be included in the file name
    • Our example file is named example_timestamp.csv
  • We can view this lookup table under “Datasets” or by typing | inputlookup example_timestamp.csv into the search head

• Financial specifications

  • A symbol is chosen, this example uses bitcoin
  • Exchanges and Bases for pricing information are determined

Search Query

The following search will filter for trade events (using the index=financial sourcetype=ohlcv data source) meeting the financial specifications.

It then pulls the uploaded timestamp file, converts the %Y-%m-%dT%H:%M:%S.%Q" format into Unix Epoch time and rounds to the whole number. Candelstick events are then limited to those with a _time value of those timestamps.

index=financial sourcetype=ohlcv
symbol=BTC
market_venue IN (COINBASE, KRAKEN, BITSTAMP, COINBASE, GEMINI, BINANCE, HITBTC, COINBENE)
base IN (USD, PAX, USDT)
([| inputlookup example_timestamp.csv
| eval timestamp=strptime(timestamp, "%Y-%m-%dT%H:%M:%S.%Q"), timestamp = mvindex(split(timestamp, "."), 0), timestamp = "_time=".timestamp
| stats values(timestamp) as times
| eval times = mvjoin(times, " OR ")
| return $times])
| fields _time close market_venue
| stats latest(close) as usd_price by market_venue _time

The 1m close price at each timestamp is returned per market_venue. Below are the results for our example dataset.

These results can then be exported by following the instruction on the “Export Data” page.