Bulk Data Collection Example

This example demonstrates how someone can upload timestamps and automate the collection of data from NTerminal with Splunk Enterprise.

Data collection

  • Random timestamps are taken.

    • They are collected in a .csv file under the column header timestamp
    • timestamps are numbered and in %Y-%m-%dT%H:%M:%S.%Q" format
    • The example table can be seen below:
    number timestamp
    1 2019-02-11T12:00:00.000+00:00
    2 2019-01-11T12:00:00.000+00:00

  • The .csv file is uploaded to splunk (via Settings>Lookups>"+Add New" under “Lookup table files”)

    • Note: “.csv” must be included in the file name
      • Our example file is named example_timestamp.csv
    • We can view this lookup table under “Datasets” or by typing | inputlookup example_timestamp.csv into the search head

  • Financial specifications

    • A symbol is chosen, this example uses bitcoin
    • Exchanges and Bases for pricing information are determined

Search Query

The following search will filter for trade events (using the index=financial sourcetype=ohlcv data source) meeting the financial specifications.

It then pulls the uploaded timestamp file, converts the %Y-%m-%dT%H:%M:%S.%Q" format into Unix Epoch time and rounds to the whole number. Candelstick events are then limited to those with a _time value of those timestamps.

index=financial sourcetype=ohlcv
symbol=BTC
market_venue IN (COINBASE, KRAKEN, BITSTAMP, COINBASE, GEMINI, BINANCE, HITBTC, COINBENE)
base IN (USD, PAX, USDT)
([| inputlookup example_timestamp.csv
| eval timestamp=strptime(timestamp, "%Y-%m-%dT%H:%M:%S.%Q"), timestamp = mvindex(split(timestamp, "."), 0), timestamp = "_time=".timestamp
| stats values(timestamp) as times
| eval times = mvjoin(times, " OR ")
| return $times])
| fields _time close market_venue
| stats latest(close) as usd_price by market_venue _time

The 1m close price at each timestamp is returned per market_venue. Below are the results for our example dataset.

market_venue _time usd_price
BINANCE 2019-01-11 12:00 3589.810000000
BINANCE 2019-02-11 12:00 3584.230000000
BITSTAMP 2019-01-11 12:00 3635.010000000
BITSTAMP 2019-02-11 12:00 3583.000000000
COINBASE 2019-01-11 12:00 3634.570000000
COINBASE 2019-02-11 12:00 3585.010000000
GEMINI 2019-01-11 12:00 3636.810000000
GEMINI 2019-02-11 12:00 3583.990000000
HITBTC 2019-01-11 12:00 3622.920000000
HITBTC 2019-02-11 12:00 3626.760000000
KRAKEN 2019-01-11 12:00 3632.900000000
KRAKEN 2019-02-11 12:00 3588.000000000

These results can then be exported by following the instruction on the “Export Data” page.