Torq

TorQ Integration Runbook (Sravz)

This document captures the TorQ/KDB work completed in this repo so far, including fixes, runtime commands, SSL work, and websocket feed integration notes.

Environment

  • Local compose file: localhost-docker-compose.yaml
  • TorQ container name pattern: sravz-torq-1
  • KDB runtime: KDB-X 5.0 (Community)
  • TorQ core in container: /opt/backend-q/deploy/TorQ/latest
  • TorQ app in container: /opt/backend-q/deploy/TorQApp/latest

Major Fixes Completed

1) KDB-X .Q.lim[] compatibility bug in TorQ core

Issue:

  • TorQ checked connection limit with 0W=.Q.lim[][\conns]`.
  • In KDB-X CE, .Q.lim[] returns a dict (cur|..., lim|...) and this comparison failed.

Fix:

  • Extract limit scalar from nested dict: x[\conns][`lim]`.

Files updated:

  • backend-q/TorQ/config/settings/default.q
  • backend-q/TorQ/config/settings/segmentedchainedtickerplant.q
  • Matching app override restored in:
    • deploy/TorQApp/v1.12.1/TorQ-Finance-Starter-Pack-1.12.1/appconfig/settings/segmentedchainedtickerplant.q

Result:

  • Discovery/connection limit logic now behaves correctly under KDB-X CE.

2) Runtime script update for realtime stack

Change:

  • Removed discovery process from realtime start helper.

File:

  • backend-q/docker/torq-realtime.sh

Current realtime helper starts:

  • stp1 sctp1 feed1 gateway1

3) Install scripts moved under backend-q

Moved:

  • install_kdb.sh -> backend-q/install_kdb.sh
  • installtorqapp.sh -> backend-q/installtorqapp.sh

Adjusted:

  • backend-q/Dockerfile copy paths updated accordingly.

4) Finance Starter Pack setenv.sh CA bundle fix

Issue:

  • Legacy script downloaded cacert.pm and exported it as CA bundle.
  • That caused OpenSSL certificate load errors.

Fix in app script:

  • Prefer existing CA bundle from env/system.
  • Fallback download uses https://curl.se/ca/cacert.pem.

File:

  • backend-q/TorQ-Finance-Starter-Pack/setenv.sh

Important deployment note:

  • In container, /opt/backend-q/deploy/bin/setenv.sh may be a copied stale file.
  • Local compose now mounts deploy-aware wrapper to that path.

5) Local deploy-aware setenv wrapper

Added:

  • backend-q/docker/setenv-deploy.sh

Purpose:

  • Enforces deploy path variables (TORQHOME, KDBAPPCODE, etc.)
  • Creates log directory safely
  • Applies corrected CA bundle behavior

Mounted by local compose:

  • localhost-docker-compose.yaml

6) Local TLS file preparation for q websocket client

Added:

  • backend-q/docker/prepare_q_tls.sh

Purpose:

  • Splits cert bundle into:
    • server-key.pem
    • server-crt.pem
    • cacert.pem

Added Make target:

  • make -C backend-q prep-tls

Local compose mounts these to:

  • /usr/lib/ssl/server-key.pem
  • /usr/lib/ssl/server-crt.pem
  • /usr/lib/ssl/cacert.pem

7) Websocket feed implementation

Primary file:

  • backend-q/TorQ-Finance-Starter-Pack/code/tick/feed_ws.q

Capabilities now:

  • Uses q websocket client handshake API (not hopen ws shortcut)
  • Works with two sockets in one process:
    • trades: /ws/us
    • quotes: /ws/us-quote
  • Handle-based routing via .z.w
  • Subscribes after per-socket 200 Authorized
  • Publishes to TorQ tables:
    • trade
    • quote
  • Publishes side-channel raw table for provider timestamp preservation:
    • eodwsraw

Schema update:

  • Added eodwsraw in backend-q/TorQ-Finance-Starter-Pack/database.q

Side-Channel Table

Added table:

  • eodwsraw

Goal:

  • Keep provider timestamp (t) exactly, alongside normalized fields and raw payload text.

Fields:

  • time (ingest timestamp)
  • providerts (converted from provider epoch ms)
  • sym
  • kind (trade/quote)
  • price, bid, ask, size, bsize, asize
  • raw
  • src

Commands

Build / run local torq service

From repo root:

docker compose -f localhost-docker-compose.yaml up -d torq

Prepare local TLS files

make -C backend-q prep-tls

TorQ process summary

make summary-backend-q

Enter torq container shell

docker exec -it sravz-torq-1 bash

Start websocket feed process manually

Inside container:

source /opt/backend-q/deploy/bin/setenv.sh
q "$TORQHOME/torq.q" -proctype feed -procname feed_ws1 -p 6098 -debug -noredirect -trap -load "$KDBAPPCODE/tick/feed_ws.q"

Known good direct handshake probe (inside container)

.z.ws:{0N!"ws msg";0N!x;}
req:"GET /ws/us?api_token=demo HTTP/1.1\r\nHost: ws.eodhistoricaldata.com:443\r\n\r\n"
res:.[{[u;r] u r};(`$":wss://ws.eodhistoricaldata.com:443";req);{[e] e}]
show type res
show res

Common Troubleshooting Notes

/torq.q. OS reports: No such file or directory

Cause:

  • TORQHOME unset in shell.

Fix:

  • source setenv, or use absolute path.

connection failed: type / websocket open issues

Cause:

  • Wrong argument type or connection form.

Fix:

  • Use q websocket handshake API:
    • (`$":wss://host:443") "GET ..."

X509_load_cert_crl_file_ex:no certificate or crl found

Cause:

  • Invalid CA file path/content (historically from cacert.pm).

Fix:

  • Ensure SSL_CA_CERT_FILE points to a real PEM bundle.
  • Use corrected deploy setenv and mounted CA bundle.

TorQ logger type failures

Cause:

  • Passing list/mixed types instead of single char vector in .lg.* message.

Fix:

  • Flatten log text before .lg.o/.lg.e calls.

onmsg failed: rank

Cause:

  • Trap function arity mismatch in .[f;args;e].

Fix:

  • Ensure f lambda arg list matches passed args exactly.

EOD subscribe validation errors

Observed:

  • Action and symbols should be string

Fix:

  • Send:
    • {"action":"subscribe","symbols":"AMZN,TSLA"}

Quote endpoint payload shape

Confirmed /ws/us-quote payload keys:

  • s, ap, as, bp, bs, t

Current quote mapper supports these aliases and common alternates.

Scaling Guidance (600 tickers)

Provider constraint:

  • 50 tickers per websocket connection.

Recommendation:

  • Minimum 12 connections for 600 symbols.
  • Prefer 14-16 connections for headroom and rebalance.
  • Shard symbols deterministically across sockets.
  • Run multiple feed processes (not one giant process) for isolation.

For a full feed handler design, see:

  • docs-hugo/content/docs/Tech/kdb/feed_handler.md
Follow Us: