Expected Behavior
BrainCompanyFilingLanguageMetricsUniverseAll and BrainCompanyFilingLanguageMetricsUniverse10K should expose the filing language metric similarity values during universe selection, for example d.ReportSentiment.Similarity.All (and the risk factor and management discussion similarity blocks). These values are already available point in time via add_data.
Actual Behavior
During universe selection the Similarity object is always null, so report_sentiment.similarity.all and the other similarity fields cannot be read, even though the point in time data type exposes them.
Root Cause
Two things on the universe path drop the data:
DataProcessing/universe.py writes only the 33 sentiment metric columns (datum[3:36]) to the universe file, dropping the trailing similarity columns.
BrainCompanyFilingLanguageMetricsUniverse.Reader calls the metrics only BrainCompanyFilingLanguageMetrics.Parse overload, so Similarity is never populated.
Reproducing the Problem
Add the universe with add_universe(BrainCompanyFilingLanguageMetricsUniverseAll, ...) and read d.ReportSentiment.Similarity.All for any selected datum. It is null, whereas the same field is populated when the symbol is subscribed point in time with add_data(BrainCompanyFilingLanguageMetricsAll, ...).
Expected Behavior
BrainCompanyFilingLanguageMetricsUniverseAllandBrainCompanyFilingLanguageMetricsUniverse10Kshould expose the filing language metric similarity values during universe selection, for exampled.ReportSentiment.Similarity.All(and the risk factor and management discussion similarity blocks). These values are already available point in time viaadd_data.Actual Behavior
During universe selection the
Similarityobject is always null, soreport_sentiment.similarity.alland the other similarity fields cannot be read, even though the point in time data type exposes them.Root Cause
Two things on the universe path drop the data:
DataProcessing/universe.pywrites only the 33 sentiment metric columns (datum[3:36]) to the universe file, dropping the trailing similarity columns.BrainCompanyFilingLanguageMetricsUniverse.Readercalls the metrics onlyBrainCompanyFilingLanguageMetrics.Parseoverload, soSimilarityis never populated.Reproducing the Problem
Add the universe with
add_universe(BrainCompanyFilingLanguageMetricsUniverseAll, ...)and readd.ReportSentiment.Similarity.Allfor any selected datum. It is null, whereas the same field is populated when the symbol is subscribed point in time withadd_data(BrainCompanyFilingLanguageMetricsAll, ...).