Repository navigation
[Feature Request] Frequency merge operations #263
Description
Merge operations for Maps
Data can be sorted into bins of predefined width using the Frequency or EmpiricalDistribution module. If two datasets are binned and should be merged afterwards, several merging strategies are possible. A simple merge of freqA and freqB is straightforward with keys that are present in freqA and freqB are replaced with the values of freqB.
let a =
[("k1",1);("k2",3)]
|> Map.ofList
let b =
[("k2",1);("k3",4)]
|> Map.ofList
merge a bresults in the following combination with ("k2",3) from a being replaced by ("k2",1) from b:
val it: Map<float,int> = map [("k1", 1); ("k2", 1); ("k3", 4)]Generic formulation of merge operations
I'm in the process of adding a generic function that gets an additional function that handles key duplicates. E.g.:
add a bresulting in the combination of a and b with ("k2",3) from a being added to ("k2",1) from b:
val it: Map<float,int> = map [("k1", 1); ("k2", 4); ("k3", 4)]While this is trivial, I'm not sure how to handle a subtraction. Should the result from subtract a b result in:
- a)
val it: Map<float,int> = map [("k1", 1); ("k2", 2); ("k3", 4)]- counts from
aare subtracted by the corresponding values frombif keys are present in both maps - here the values of
athat are not present inbare untouched
- counts from
- b)
val it: Map<float,int> = map [("k1", 1); ("k2", 2); ("k3", -4)]- counts from
aare subtracted by the values fromb, even for keys that are not present ina
- counts from
The latter option (b) makes no sense to me since frequency counts should not be negative, but I cannot think of applications in which the result of (a) makes any sense. Maybe the subtract function is not the best to start with because in this post they implemented (a) with addition and multiplication examples. Especially for the addition, a and b would give the correct result and I think it is intuitive to just apply the function to values of keys that are present in both maps.
@HarryMcCarney, do you know use cases that use subtract? Do you have any thoughts about this? I would suggest to add version (a) to Frequency as well as EmpiricalDistribution
Additional remark: When applied to continuous data bandwidths must be equal, to not merge counts from overlapping bins!
Activity
add map merge
add map merge tests
add distribution merge documentation
bvenn commentedon Apr 27, 2023
A difficulty that I'm not sure how to tackle is the required bandwidth equality on continuous data. If you want to add the following histograms:
Histograms (regardless if they are
FrequenciesorEmpiricalDistributions) that should be merged, have to have the same bandwidth. For categorical data this is no issue!Solution
add bandwidth test
bvenn commentedon Apr 27, 2023
For now I decided to go with an unsatisfactory hybrid of (B) and (C). I added a parameter that requests the user to specify if the maps are based on equal binning or if it is categorical data. If its continuous data with unequal binning, the merge fails with a description explaining the issue.
In future a procedure could be implemented that dissect both maps and creates a new one with a new binning. If my understanding is correct the bandwidth must be double the maximal bandwidth that is observed in the input maps.
Merge pull request #264 from fslaborg/#263-add-map-merge
github-actions commentedon Apr 19, 2026
Note
The pull request was not created — a fallback review issue was created instead due to protected file changes: #379
🤖 This is an automated response from Repo Assist.
Checking in: it looks like this issue has been fully resolved! The
Frequencymodule already containsmergeBy,merge,add, andsubtract(all with theequalBandwidthOrNominalguard), matching the hybrid approach described in the comments above. TheMap.mergeBybackbone is also in place.Could this issue be closed? If there are remaining concerns (e.g. the future "rebinning" path for unequal-bandwidth continuous histograms), those could be tracked separately.