Expose filing language metric similarity in universe selection - #31
Merged
jaredbroad merged 1 commit intoJun 24, 2026
Conversation
The point-in-time BrainCompanyFilingLanguageMetrics readers populate the ReportSentiment / RiskFactorsStatementSentiment / ManagementDiscussion... Similarity values via the two-argument BrainCompanyFilingLanguageMetrics.Parse, but the universe selection path did not: - universe.py wrote only the 33 sentiment metrics (datum[3:36]) to the universe file, dropping the trailing similarity-difference columns. - BrainCompanyFilingLanguageMetricsUniverse.Reader called the metrics-only Parse overload, so .Similarity stayed null. As a result BrainCompanyFilingLanguageMetricsUniverseAll/10K never exposed similarity.all (or the other similarity fields), even though the point-in-time data type did via add_data. Fix: - universe.py: write datum[3:53] so the universe file carries the same trailing report/RF/MD similarity columns as the point-in-time file. - BrainCompanyFilingLanguageMetricsUniverse.Reader: pass the similarity columns to Parse using the same offsets as BrainCompanyFilingLanguageMetricsBase.Reader (Skip(39).Take(7), Skip(46).Take(3), Skip(49)). Existing metric offsets and Value/Symbol parsing are unchanged. The generic base reader is shared by both the "all" and "10k" universes, so both are fixed.
jaredbroad
approved these changes
Jun 24, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Expose the filing language metric similarity values (
Similarity.Alland the related fields) inBrainCompanyFilingLanguageMetricsuniverse selection (BrainCompanyFilingLanguageMetricsUniverseAllandUniverse10K). They were already available point in time viaadd_databut were missing from universe selection.Related Issue
Resolves #32
Motivation and Context
BrainCompanyFilingLanguageMetrics.Parseonly attachesSimilaritywhen given its second argument. On the universe path the similarity columns were never written to the universe file (universe.pykept onlydatum[3:36]) and the universeReaderused the metrics onlyParseoverload, soSimilaritywas always null. The fix writes the trailing similarity columns and parses them in the universeReaderwith the same offsets asBrainCompanyFilingLanguageMetricsBase.Reader(Skip(39).Take(7),Skip(46).Take(3),Skip(49)), bringing universe selection in line with the point in time type. The universe file schema gains columns, so deployed universe data needs reprocessing.Requires Documentation Change
N/A
How Has This Been Tested?
dotnet testpasses (24 of 24). Verified that the universe column offsets line up with the point in time reader, soSIMILARITY_ALLmaps toSimilarity.Allfor the report, risk factor, and management discussion sections.Types of changes
Checklist: