TorQ Integration Runbook (Sravz)

This document captures the TorQ/KDB work completed in this repo so far, including fixes, runtime commands, SSL work, and websocket feed integration notes.

Environment

• Local compose file: localhost-docker-compose.yaml
• TorQ container name pattern: sravz-torq-1
• KDB runtime: KDB-X 5.0 (Community)
• TorQ core in container: /opt/backend-q/deploy/TorQ/latest
• TorQ app in container: /opt/backend-q/deploy/TorQApp/latest

Major Fixes Completed

1) KDB-X .Q.lim[] compatibility bug in TorQ core

Issue:

• TorQ checked connection limit with 0W=.Q.lim[][\conns]`.
• In KDB-X CE, .Q.lim[] returns a dict (cur|..., lim|...) and this comparison failed.

Fix:

• Extract limit scalar from nested dict: x[\conns][`lim]`.

Files updated:

• backend-q/TorQ/config/settings/default.q
• backend-q/TorQ/config/settings/segmentedchainedtickerplant.q
• Matching app override restored in:
  • deploy/TorQApp/v1.12.1/TorQ-Finance-Starter-Pack-1.12.1/appconfig/settings/segmentedchainedtickerplant.q

Result:

• Discovery/connection limit logic now behaves correctly under KDB-X CE.

2) Runtime script update for realtime stack

Change:

• Removed discovery process from realtime start helper.

File:

• backend-q/docker/torq-realtime.sh

Current realtime helper starts:

• stp1 sctp1 feed1 gateway1

3) Install scripts moved under backend-q

Moved:

• install_kdb.sh -> backend-q/install_kdb.sh
• installtorqapp.sh -> backend-q/installtorqapp.sh

Adjusted:

• backend-q/Dockerfile copy paths updated accordingly.

4) Finance Starter Pack setenv.sh CA bundle fix

Issue:

• Legacy script downloaded cacert.pm and exported it as CA bundle.
• That caused OpenSSL certificate load errors.

Fix in app script:

• Prefer existing CA bundle from env/system.
• Fallback download uses https://curl.se/ca/cacert.pem.

File:

• backend-q/TorQ-Finance-Starter-Pack/setenv.sh

Important deployment note:

• In container, /opt/backend-q/deploy/bin/setenv.sh may be a copied stale file.
• Local compose now mounts deploy-aware wrapper to that path.

5) Local deploy-aware setenv wrapper

Added:

• backend-q/docker/setenv-deploy.sh

Purpose:

• Enforces deploy path variables (TORQHOME, KDBAPPCODE, etc.)
• Creates log directory safely
• Applies corrected CA bundle behavior

Mounted by local compose:

• localhost-docker-compose.yaml

6) Local TLS file preparation for q websocket client

Added:

• backend-q/docker/prepare_q_tls.sh

Purpose:

• Splits cert bundle into:
  • server-key.pem
  • server-crt.pem
  • cacert.pem

Added Make target:

• make -C backend-q prep-tls

Local compose mounts these to:

• /usr/lib/ssl/server-key.pem
• /usr/lib/ssl/server-crt.pem
• /usr/lib/ssl/cacert.pem

7) Websocket feed implementation

Primary file:

• backend-q/TorQ-Finance-Starter-Pack/code/tick/feed_ws.q

Capabilities now:

• Uses q websocket client handshake API (not hopen ws shortcut)
• Works with two sockets in one process:
  • trades: /ws/us
  • quotes: /ws/us-quote
• Handle-based routing via .z.w
• Subscribes after per-socket 200 Authorized
• Publishes to TorQ tables:
  • trade
  • quote
• Publishes side-channel raw table for provider timestamp preservation:
  • eodwsraw

Schema update:

• Added eodwsraw in backend-q/TorQ-Finance-Starter-Pack/database.q

Side-Channel Table

Added table:

• eodwsraw

Goal:

• Keep provider timestamp (t) exactly, alongside normalized fields and raw payload text.

Fields:

• time (ingest timestamp)
• providerts (converted from provider epoch ms)
• sym
• kind (trade/quote)
• price, bid, ask, size, bsize, asize
• raw
• src

Commands

Build / run local torq service

From repo root:

docker compose -f localhost-docker-compose.yaml up -d torq

Prepare local TLS files

make -C backend-q prep-tls

TorQ process summary

make summary-backend-q

Enter torq container shell

docker exec -it sravz-torq-1 bash

Start websocket feed process manually

Inside container:

source /opt/backend-q/deploy/bin/setenv.sh
q "$TORQHOME/torq.q" -proctype feed -procname feed_ws1 -p 6098 -debug -noredirect -trap -load "$KDBAPPCODE/tick/feed_ws.q"

Known good direct handshake probe (inside container)

.z.ws:{0N!"ws msg";0N!x;}
req:"GET /ws/us?api_token=demo HTTP/1.1\r\nHost: ws.eodhistoricaldata.com:443\r\n\r\n"
res:.[{[u;r] u r};(`$":wss://ws.eodhistoricaldata.com:443";req);{[e] e}]
show type res
show res

Common Troubleshooting Notes

/torq.q. OS reports: No such file or directory

Cause:

• TORQHOME unset in shell.

Fix:

• source setenv, or use absolute path.

connection failed: type / websocket open issues

Cause:

• Wrong argument type or connection form.

Fix:

• Use q websocket handshake API:
  • (`$":wss://host:443") "GET ..."

X509_load_cert_crl_file_ex:no certificate or crl found

Cause:

• Invalid CA file path/content (historically from cacert.pm).

Fix:

• Ensure SSL_CA_CERT_FILE points to a real PEM bundle.
• Use corrected deploy setenv and mounted CA bundle.

TorQ logger type failures

Cause:

• Passing list/mixed types instead of single char vector in .lg.* message.

Fix:

• Flatten log text before .lg.o/.lg.e calls.

onmsg failed: rank

Cause:

• Trap function arity mismatch in .[f;args;e].

Fix:

• Ensure f lambda arg list matches passed args exactly.

EOD subscribe validation errors

Observed:

• Action and symbols should be string

Fix:

• Send:
  • {"action":"subscribe","symbols":"AMZN,TSLA"}

Quote endpoint payload shape

Confirmed /ws/us-quote payload keys:

• s, ap, as, bp, bs, t

Current quote mapper supports these aliases and common alternates.

Scaling Guidance (600 tickers)

Provider constraint:

• 50 tickers per websocket connection.

Recommendation:

• Minimum 12 connections for 600 symbols.
• Prefer 14-16 connections for headroom and rebalance.
• Shard symbols deterministically across sockets.
• Run multiple feed processes (not one giant process) for isolation.

For a full feed handler design, see:

• docs-hugo/content/docs/Tech/kdb/feed_handler.md

9) TorQ Main Components: What To Say In Interviews

Use this section when asked, “How do the major TorQ processes work together?”

10) Mental model

Use this section to explain TorQ as a resilient, decoupled data pipeline in interviews.

10.1 Unifying pitch: The resilient conveyor belt

“TorQ organizes kdb+ as an asynchronous conveyor belt. Components are decoupled so that a crash, query spike, or downstream slowdown does not stall front-door market data ingestion.”

10.2 Component value and mental anchors

10.3 Interview close

“TorQ scales by separation of concerns: ingest and journaling are protected, fan-out is isolated, real-time analytics stay in memory, history is persisted separately, and replay/observability provide operational resilience.”