IntanRawIO: Fix memmap order for Intan one-file-per-stream

[zm711]

Fixes #1556

Data before fix:

Data after fix:

Basically, since Intan was designed to be easily read with Matlab (their example work is to use fread which is column-major reading) we should make our memmap with order='F'.

@h-mayorquin want to look over this quickly and @alejoe91 I think this can be a quick review as long as everyone is agreed. See the linked issue for more information.

Can you test this @Tevin-yue?

NOTE: this is simulated data so it should look like a repeating motif of simulated "spikes".

[samuelgarcia]

why not

Suggested change

	file_path, dtype=stream_datatype, mode="r", shape=(num_channels, n_samples), order='F',
	file_path, dtype=stream_datatype, mode="r", shape=(n_samples, num_channels), order='C',

[zm711]

Good point. Let me simplify. We originally were reimplementing the Matlab function from the Intan docs, so there is also a .T of the map that we need to remove.

[samuelgarcia]

oups good catch!!

[h-mayorquin]

I don't have your experimental expertise so this is for my learning. How do you know based on your plots (before and after fix) that your fix is right?

[zm711]

I don't have your experimental expertise so this is for my learning. How do you know based on your plots (before and after fix) that your fix is right?

The values are really weird on the axis (almost like we are summing spikes ie the samples and channels values are mixed up). And for intan simulated data it should like a regularly spiking (regular in the non FS/RS way) neuron so it should have a nice downward followed by upward deflections on a background of some level of noise. The "bad" graph looks to sinusoidal (again from in appropriate summing of the voltages).

so this meant either we scaled wrong. I double checked and we weren't. Or we loaded the data in wrong (easy to mix up samples and channels when making the memmap). Since we did the documentation from intan (I think this part was me so mea culpa), I did the F-major style but didn't specify. Sam's tip was just flip the sample and channel to allow us to read the C style.

So to summarize I would accept amplifier/wideband data to be noisy with spikes in it. Not weird sinusoidal things.

Or do you have an additional question related to this?

[zm711]

Maybe this will help you @h-mayorquin if I plot all channels instead of one channel.

You see we have sinusoidal noise (because this is simulated data. In real data it wouldn't be so clean ahah) with a bunch of spikes! (colors are channels in this graph).

And for your reference here is the before fix:

[h-mayorquin]

LGTM

Thanks for the diagrams.

I am trying to think on whether we could make a list of heuristics to check when we make a change to avoid doing mistakes like this I wonder.

As I am not a very graphical person I also wrote some code to explain the nature of the error so it remains here for future reference:

num_channels = 3
num_samples = 2
# This is according to the documention how to the data is layoued A_{channel}(sample)
data = np.asarray(["A1(0)", "A2(0)", "A3(0)", "A1(1)", "A2(1)", "A3(1)"], dtype='U5')
data.tofile("testing_order.bin")


data_memmap = np.memmap(
    "testing_order.bin", 
    dtype='U5', 
    mode='r', 
    shape=(num_channels, num_samples), 
    order='C'
)

print("Normal")
print(data_memmap)
print("Transposed")
print(data_memmap.T)

Normal
[['A1(0)' 'A2(0)']
 ['A3(0)' 'A1(1)']
 ['A2(1)' 'A3(1)']]
Transposed
[['A1(0)' 'A3(0)' 'A2(1)']
 ['A2(0)' 'A1(1)' 'A3(1)']]

You can see that the channels are scrambled

The fix is like this:

num_channels = 3
num_samples = 2
# This is according to the documention how to the data is layoued A_{channel}(sample)
data = np.asarray(["A1(0)", "A2(0)", "A3(0)", "A1(1)", "A2(1)", "A3(1)"], dtype='U5')
data.tofile("testing_order.bin")


data_memmap = np.memmap(
    "testing_order.bin", 
    dtype='U5', 
    mode='r', 
    shape=(num_samples, num_channels), 
    order='C'
)

print(data_memmap)

[['A1(0)' 'A2(0)' 'A3(0)']
 ['A1(1)' 'A2(1)' 'A3(1)']]

[zm711]

Honestly dude you should propose that to Numpy! They just link to the wikipedia to explain this, but at least with your example you show what is actually happening with Numpy code. That's great!

@alejoe91 or @samuelgarcia. this is good to go now :) Tested and everyone is happy!