/
2019-10-07 kdb+ Project Meeting Minutes

2019-10-07 kdb+ Project Meeting Minutes

Dec 02, 2019

Meeting minutes status: Approved (at 2019-11-18 meeting)

Table of Contents

Date/Time

  • October  7, 2019. 11am ET

Attendees

Name

Organisation

Github ID 

Name

Organisation

Github ID 

@Alvin Shih

Morgan Stanley

alvin-c-shih

Timofey Beletskiy

JP Morgan

 

Aaron Davies

Morgan Stanley

adavies42

@Aitana Myohl

FINOS

aitana16

Alok Dutt

Citi

 

Jonny Press

AquaQ

 

Gary Davies

AquaQ

 

@James McLeod (Unlicensed)

FINOS

mcleo-d

Kai Zeng

UBS

 

Action Items

Tasks Identified and Assigned

Alok to reach out to BAML and GS
@Alvin Shih to send out conventions that get used for qualifying names.

Task Report from the Last Meeting

(In the task report select the meetings notes from the LAST meeting in the 'Space(s) and page(s)' field to auto-populate this table– remove this note before publishing)

Decisions Made

 

Agenda

Time

Item

Who

Notes from the Meeting

Time

Item

Who

Notes from the Meeting

5 min

Convene & roll call.

All

 

5 min

Review and approve minutes from last meeting.

All

20 min

Overview of TorQ and 4 components that AquaQ could break out as contributions to FINOS.

 

All

  • AquaQ offered to FINOS-ify 4 components as contributions.

    • Agreement from AquaQ and folks on the call that it'd be better to start with a few components rather than trying to bring in TorQ all at once.

  • In terms of Enterprise kdb+ sites using TorQ, a team in JPMC is starting to use it.

  • How can FINOS be useful to AquaQ / TorQ?

    • AquaQ developed TorQ to avoid rewriting utilities / framework for every new client.

    • Goals aligned with those of FINOS kdb+ Project.

    • FINOS allows Enterprise users to contribute back and / or provide a channel to discuss requirements.

25 min

Look at AquaQ's memusage.q in more detail.

All

  • Attendees decided to take a closer look at memusage.q first.

  • For compound columns longer than a specified threshold, sampling is used to estimate the space consumed by the subvectors.

    • Can increase the threshold to avoid sampling.

    • Might be useful to return an indication of whether sampling was employed.

  • memusage.q also tries to account for memory utilization due to attributes.

  • Seems like it could be useful during development.

  • AquaQ said it's useful for diagnosing memory issues on live systems as well.

    • Caller from MS said it would be useful in cases where users are stashing results on a shared q process.

  • A wrapper to find large values could be included.

    • Need to take care to handle \d / namespaces properly.

    • Some "by name" functions in q are inconsistent in how namespaces are handled.

      • Example was "set" vs "::".

  • How to FINOS-ify?

    • Suggested putting the functions in .finos.objutil since memory consumption is just one facet of an object that a developer might care about.

    • Others might include recursive traversal (related to finding large objects) and unpacking projections.

    • Mentioned that global variable like "threshold" after which statistical sampling is employed should have getter / setter to support namespace aliasing.

  • Caller from Citi seemed enthusiastic about trying it out.

5 min

AOB

All

  • James McLeod introduced himself as new FINOS Director of Community based in London.

    • Might facilitate meeting up with AquaQ representatives.

Action Items

@Jason Foster-Bey to start follow-up thread re:structlog
Aaron Davies to send out HDB consistency doc.
Peter Gyorok to put "include" on GitHub.
@Alvin Shih to put together qstructlog PoC.
Aaron Davies to get Symphony access to discuss authz_ro with @Jason Foster-Bey.
Aaron Davies to get approvals for pushing kdb+ WG-related work to GitHub.