Bulk Data Collection Example

This example demonstrates how someone can upload timestamps and automate the collection of data from NTerminal with Splunk Enterprise.

Data collection

  • Random timestamps are taken.

    • They are collected in a .csv file under the column header timestamp
    • timestamps are numbered and in %Y-%m-%dT%H:%M:%S.%Q" format
    • The example table can be seen below:
    numbertimestamp
    12019-02-11T12:00:00.000+00:00
    22019-01-11T12:00:00.000+00:00
  • The .csv file is uploaded to splunk (via Settings>Lookups>"+Add New" under “Lookup table files”)

    • Note: “.csv” must be included in the file name
      • Our example file is named example_timestamp.csv
    • We can view this lookup table under “Datasets” or by typing | inputlookup example_timestamp.csv into the search head
  • Financial specifications

    • A symbol is chosen, this example uses bitcoin
    • Exchanges and Bases for pricing information are determined

Search Query

The following search will filter for trade events (using the index=financial sourcetype=ohlcv data source) meeting the financial specifications.

It then pulls the uploaded timestamp file, converts the %Y-%m-%dT%H:%M:%S.%Q" format into Unix Epoch time and rounds to the whole number. Candelstick events are then limited to those with a _time value of those timestamps.

index=financial sourcetype=ohlcv
symbol=BTC
market_venue IN (COINBASE, KRAKEN, BITSTAMP, COINBASE, GEMINI, BINANCE, HITBTC, COINBENE)
base IN (USD, PAX, USDT)
([| inputlookup example_timestamp.csv
| eval timestamp=strptime(timestamp, "%Y-%m-%dT%H:%M:%S.%Q"), timestamp = mvindex(split(timestamp, "."), 0), timestamp = "_time=".timestamp
| stats values(timestamp) as times
| eval times = mvjoin(times, " OR ")
| return $times])
| fields _time close market_venue
| stats latest(close) as usd_price by market_venue _time

The 1m close price at each timestamp is returned per market_venue. Below are the results for our example dataset.

market_venue_timeusd_price
BINANCE2019-01-11 12:003589.810000000
BINANCE2019-02-11 12:003584.230000000
BITSTAMP2019-01-11 12:003635.010000000
BITSTAMP2019-02-11 12:003583.000000000
COINBASE2019-01-11 12:003634.570000000
COINBASE2019-02-11 12:003585.010000000
GEMINI2019-01-11 12:003636.810000000
GEMINI2019-02-11 12:003583.990000000
HITBTC2019-01-11 12:003622.920000000
HITBTC2019-02-11 12:003626.760000000
KRAKEN2019-01-11 12:003632.900000000
KRAKEN2019-02-11 12:003588.000000000

These results can then be exported by following the instruction on the “Export Data” page.