/
2019-11-18 kdb+ Project Meeting Minutes

2019-11-18 kdb+ Project Meeting Minutes

Jan 03, 2020

Meeting minutes status: Approved (at 2019-12-02 meeting)

Table of Contents

Date/Time

  • November 18, 2019. 11am ET

Attendees

Name

Organisation

Github ID 

Name

Organisation

Github ID 

@Alvin Shih

Morgan Stanley

alvin-c-shih

@Rob Underwood (Deactivated)

FINOS

brooklynrob

Jonny Press

AquaQ

 

Peter Gyorok

Morgan Stanley

gyorokpeter

Timofey Beletskiy

JP Morgan

 

Action Items

Tasks Identified and Assigned

Alok to reach out to BAML and GS
@Alvin Shih to send out conventions that get used for qualifying names.

Task Report from the Last Meeting

(In the task report select the meetings notes from the LAST meeting in the 'Space(s) and page(s)' field to auto-populate this table– remove this note before publishing)

Decisions Made

 

Agenda

Time

Item

Who

Notes from the Meeting

Time

Item

Who

Notes from the Meeting

5 min

Convene & roll call.

All

 

5 min

Review and approve minutes from last meeting.

All

  • Approved by acclamation.

10 min

HDB Well-Formedness.md document is live under "technical-documentation".

Aaron to go over what it is and what it isn't.  Hopefully opens the door to shared development of HDB utils.

All

  • Aaron unable to make the call.

  • Good discussions about extensions that go beyond Kx functionality.

    • Ideas that have been implemented at Morgan Stanley:

      • Sparse HDB

        • If new table is introduced, backfilling many years of empty table segments is time consuming and makes schema changes more expensive.

      • Staging Directory

        • When a table segment is only partially written due to I/O speed limitations, the HDB as a whole is technically not well-formed.

        • Staging a table on the same mount point and moving it atomically when writes complete could avoid problems with intra-day HDB reloads.

      • Partial HDB Reload

        • Intraday writedown (w.q) often implies intraday reload.  Loading of HDBs is I/O intensive.  A means of loading for a target date range could help tremendously.

    • JP Morgan interested in extensions that support archiving in a standard way.

      • Perhaps recent data run on more performant storage, whereas older data would be hosted on slower, cheaper drives.

  • Perhaps worth creating an rfc/ directory in the repository to get the details out in a shareable format.

5 min

GitHub repository now has reviewers.

All

  • Should be able to turn reviews / approvals around more quickly.

  • Hope to add one or two more based on strong contributors.

5 min

Managed to merge a couple of branches to main (with the help of Magit):

Hoping it makes it easier for others to improve on them.

All

  • timer pull request to be merged soon.

5 min

Update on AquaQ's effort to FINOS-ize memusage.q.

All

  • memusage pull request is up.

  • Question arose about how documentation should be maintained.

    • AquaQ is familiar with GitHub microsites as they use those already.

    • Suggestion that they talk to Eric Griswald to see what (if anything) is different with microsites under FINOS compared to what AquaQ is doing already.

5 min

AOB

All

 

Action Items

@Jason Foster-Bey to start follow-up thread re:structlog
Aaron Davies to send out HDB consistency doc.
Peter Gyorok to put "include" on GitHub.
@Alvin Shih to put together qstructlog PoC.
Aaron Davies to get Symphony access to discuss authz_ro with @Jason Foster-Bey.
Aaron Davies to get approvals for pushing kdb+ WG-related work to GitHub.