Tickstore

As the name suggests, tickstore is used for storing stream of ticks from financial exchanges. For example, this could be the bid/offers to an exchange.

It's designed for read heavy loads and realtime, continuous ticks of data and you are expected to do the batching on your side, generally done with a message queue like kafka / redis queue etc.

Reading and Writing data with Tickstore

Sample tick:

    sample_ticks = [
    {
            'ASK': 1545.25,
            'ASKSIZE': 1002.0,
            'BID': 1545.0,
            'BIDSIZE': 55.0,
            'CUMVOL': 2187387.0,
            'DELETED_TIME': 0,
            'INSTRTYPE': 'FUT',
            'PRICE': 1545.0,
            'SIZE': 1.0,
            'TICK_STATUS': 0,
            'TRADEHIGH': 1561.75,
            'TRADELOW': 1537.25,
            'index': 1185076787070
        },
        {
            'CUMVOL': 354.0,
            'DELETED_TIME': 0,
            'PRICE': 1543.75,
            'SIZE': 354.0,
            'TRADEHIGH': 1543.75,
            'TRADELOW': 1543.75,
            'index': 1185141600600
        }
    ]

Writing and reading to tickstore

tickstore_lib.write('FEED::SYMBOL', sample_ticks)

df = tickstore_lib.read('FEED::SYMBOL', columns=['BID', 'ASK', 'PRICE'])

Another example with datetime index with tz_info

    data = [{'A': 120, 'D': 1}, {'A': 122, 'B': 2.0}, {'A': 3, 'B': 3.0, 'D': 1}]
    tick_index = [dt(2013, 6, 1, 12, 00, tzinfo=mktz('UTC')),
                  dt(2013, 6, 1, 11, 00, tzinfo=mktz('UTC')),  # Out-of-order
                  dt(2013, 6, 1, 13, 00, tzinfo=mktz('UTC'))]
    data = pd.DataFrame(data, index=tick_index)

    tickstore_lib._chunk_size = 3
    tickstore_lib.write('SYM', data)
    tickstore_lib.read('SYM', columns=None)

Usecases

  • Storing billions of ticks in a compressed way with fast querying by date ranges.
  • Customizable chunk sizes. The default is 100k, which should fit easily in a single mongo doc for fast reads.
  • Structured to work with financial tick data stored on a per symbol basis. Generally used with kafka / redis queue or some sort of message broker for streaming data.

See James's talk for more details