Skip to content

Lazy-mode for loading Sortinganalyzer - #4623

Merged
chrishalcrow merged 17 commits into
SpikeInterface:mainfrom
alejoe91:lazy-extensions
Jul 22, 2026
Merged

Lazy-mode for loading Sortinganalyzer#4623
chrishalcrow merged 17 commits into
SpikeInterface:mainfrom
alejoe91:lazy-extensions

Conversation

@alejoe91

@alejoe91 alejoe91 commented Jun 22, 2026

Copy link
Copy Markdown
Member

The SortingAnalyzer, especially for chronic data, can be very memory hungry. This PR adds a lazy mode at load that:

  • Reads the spike vector in lazy mode (memmaps memory folder, virtual memmap for zarr - TODO)
  • Reads vector-extensions as memmaps or keeps them as zarr array
  • Implements a memmap-like ZarrSpikeVector

NOTE: in lazy mode, save/select/merge/split operations are not permitted!

@alejoe91 alejoe91 added core Changes to core module performance Performance issues/improvements labels Jun 22, 2026
Comment thread src/spikeinterface/core/sortingfolder.py
@alejoe91
alejoe91 marked this pull request as ready for review June 24, 2026 15:19
@alejoe91
alejoe91 requested a review from samuelgarcia June 24, 2026 15:23
@alejoe91 alejoe91 added this to the 0.105.0 milestone Jul 15, 2026
@chrishalcrow

chrishalcrow commented Jul 21, 2026

Copy link
Copy Markdown
Member

Hello, when I do a get_data() on any of the spike-vector extensions (e.g. spikeamplitudes) on a zarr'd analyzer I get an error here:

return all_data.copy() # return a copy to avoid modification

as follows:

Traceback (most recent call last):
  File "<python-input-10>", line 1, in <module>
    sa.get_extension('spike_amplitudes').get_data()
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/home/nolanlab/fromgit/spikeinterface/src/spikeinterface/core/sortinganalyzer.py", line 3465, in get_data
    return self._get_data(*args, **kwargs)
           ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
  File "/home/nolanlab/fromgit/spikeinterface/src/spikeinterface/core/analyzer_extension_core.py", line 1622, in _get_data
    return all_data.copy()  # return a copy to avoid modification
           ^^^^^^^^^^^^^
AttributeError: 'Array' object has no attribute 'copy'

@alejoe91

Copy link
Copy Markdown
Member Author

.get_data()

So when it's lazy we should force copy=False. Does it make sense?

Comment thread src/spikeinterface/core/sortinganalyzer.py Outdated
Comment thread src/spikeinterface/core/sortingfolder.py
Co-authored-by: Chris Halcrow <57948917+chrishalcrow@users.noreply.github.com>

@chrishalcrow chrishalcrow left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Works for me, and doesnt't change any defaults.
Note: it's pretty conservative. If you use lazy mode, you can't save/recompute/anything!

@chrishalcrow
chrishalcrow merged commit fe501e8 into SpikeInterface:main Jul 22, 2026
17 checks passed
@samuelgarcia

Copy link
Copy Markdown
Member

I would have been very happy to discuss this PR in detail before the merge
For perf reason mainly.
The ZarrSpikeVector would have deserve some deep benchmark no ?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

core Changes to core module performance Performance issues/improvements

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants