Activity

1. nbargnesi

   added

   bug

   on Feb 24, 2015
2. abargnesi commented on Feb 24, 2015

   

   abargnesi

   on Feb 24, 2015

   Member

   More actions

   Annotation (and namespace) search doesn't do partial matches unless you use wildcards. This implies that the search parameter is a pass-through for the underlying FTS mechanism (SQLite FTS4). Here are some example requests:

   EFO Example

   GET http://next.belframework.org/api/annotations/efo/values?filter={%22category%22:%22fts%22,%22name%22:%22search%22,%22value%22:%22*cell%20line*%22}

   Uberon

   GET http://next.belframework.org/api/annotations/uberon/values?filter={%22category%22:%22fts%22,%22name%22:%22search%22,%22value%22:%22*lung*%22}

   Currently species annotations are not searchable. A fix for this is forthcoming.
3. abargnesi

   added

   in progress

   on Feb 24, 2015
4. nbargnesi commented on Feb 24, 2015

   

   nbargnesi

   on Feb 24, 2015

   MemberAuthor

   More actions

   It appears to do partial matches now w/out wildcards. Can I leave them out?
5. abargnesi commented on Feb 24, 2015

   

   abargnesi

   on Feb 24, 2015

   Member

   More actions

   The wildcards are required. You might be matching on the whole token value. For example searching on tiss will not find results with the tissue token. You will have to use either the whole term or wildcard:

   http://next.belframework.org/api/annotations/values?filter={"category":"fts","name":"search","value":"tissue"}
   
   http://next.belframework.org/api/annotations/values?filter={"category":"fts","name":"search","value":"tiss*"}
6. nbargnesi commented on Feb 24, 2015

   

   nbargnesi

   on Feb 24, 2015

   MemberAuthor

   More actions

   How is this request/response possible?

   request:

   next.belframework.org/api/annotations/values?size=1&filter={"category":"fts","name":"search","value":"9"}

   response:

   {
     "annotation_values": [
       {
         "identifier": "0001045",
         "name": "11-5.2.1.9 cell",
         "type": "AnnotationConcept"
       }
     ]
   }

   Here's the actual GET line:

   GET /api/annotations/values?size=1&filter=%7B%22category%22:%22fts%22,%22name%22:%22search%22,%22value%22:%229%22%7D HTTP/1.1
   
7. abargnesi commented on Feb 24, 2015

   

   abargnesi

   on Feb 24, 2015

   Member

   More actions

   The result is possible because 9 is a separate token within it. The SQLite FTS4 table treats . as a token separator and not a token character.

   The token characters are defined as space = , - ( ) within the table creation. I should be able to add the period as a token character. Have you seen other non-alphanumeric characters apart from these?
8. nbargnesi commented on Feb 25, 2015

   

   nbargnesi

   on Feb 25, 2015

   MemberAuthor

   More actions

   You may want to consider taking more reponsibility from the client and exposing simple search capabilities for now. This will get us to good enough. We can go for robustness at a later point and take on the added complexity then.

   If you decide to go the robust route now, there are at least three points that need addressing:

   1. Fail to match leading *, e.g.: *ateral fails to match lateral
   2. Fail to match both leading and trailing *, e.g.: *atera* fails to match lateral
   3. Unexpected results when we are performing searches and match other data.

   Some amplifying information on (3) as well - if we search ell and match electrosensory lateral line lobe, it needs to be apparent why we matched it.

   examples

   search by token

   ...Creveld

   Using Creveld alone gets expected results:

   filter:

   {"category":"fts","name":"search","value":"Creveld"}

   response:

   {
     "identifier": "12714",
     "name": "Ellis-Van Creveld syndrome",
     "type": "BiologicalProcessConcept"
   }

   ...Ellis-Van

   Or Ellis-Van:

   filter:

   {"category":"fts","name":"search","value":"Ellis-Van"}

   response:

   {
     "identifier": "12714",
     "name": "Ellis-Van Creveld syndrome",
     "type": "BiologicalProcessConcept"
   }

   ...syndrome

   Okay, now we're getting somewhere. How about syndrome:

   filter:

   {"category":"fts","name":"search","value":"syndrome"}

   response:

   {
     "identifier": "0050120",
     "name": "hemophagocytic lymphohistiocytosis",
     "type": "BiologicalProcessConcept"
   }

   Okay, hemophagocytic lymphohistiocytosis is a type of syndrome, but how do I know that from the search result?

   ...lateral

   This one makes sense:

   filter:

   {"category":"fts","name":"search","value":"lateral"}

   response:

   {
     "identifier": "0008021",
     "name": "anterior lateral line ganglion neuron",
     "type": "CellAnnotationConcept"
   }

   ...latera*

   So does this (note the trailing *):

   filter:

   {"category":"fts","name":"search","value":"latera*"}

   response:

   {
     "identifier": "0008021",
     "name": "anterior lateral line ganglion neuron",
     "type": "CellAnnotationConcept"
   }

   ...*ateral

   This doesn't (leading *):

   filter:

   {"category":"fts","name":"search","value":"*ateral"}

   response:

   HTTP/1.1 404 Not Found
   

   ...atera

   Or this (leading and trailing *):

   filter:

   {"category":"fts","name":"search","value":"*atera*"}

   response:

   HTTP/1.1 404 Not Found
   

   multiple wildcards

   Multiple wildcards support is nice:

   filter:

   {"category":"fts","name":"search","value":"posterior lateral * ganglion *"}

   response:

   {
     "identifier": "1000245",
     "name": "posterior lateral line ganglion neuron",
     "type": "CellAnnotationConcept"
   }

   weird results

   Try ell:

   filter:

   {"category":"fts","name":"search","value":"ell"}

   response:

   {
   "identifier": "2002105",
   "name": "electrosensory lateral line lobe",
   "type": "AnatomyAnnotationConcept"
   }

   Not sure why I'm getting this. I suspect ell is an abbreviation, but it's not apparent.
9. abargnesi commented on Feb 25, 2015

   

   abargnesi

   on Feb 25, 2015

   Member

   More actions

   Thanks for the write-up with examples. Very thorough and provides a nice set to test with.

   The major problems seem to be:

   1. Suffix and inner matches cannot be found.
   2. Does not return where in the result the search matched (e.g. name, title, synonym and which part matched).

   To address (1) I will have to index suffixes of identifiers, names, titles, and synonyms. For example the term carcinoma would index using carcinoma arcinoma rcinoma cinoma inoma noma oma ma aAn approach using SQLite that seems reasonable is explained here. I am in the middle of this work.

   For (2) I will be able to provide which field(s) match and where (position range), but this will take longer.
10. abargnesi commented on Feb 26, 2015

    

    abargnesi

    on Feb 26, 2015

    Member

    More actions

    Closing. Split work among a few tickets.
11. abargnesi

    closed this as completedon Feb 26, 2015
12. abargnesi

    removed

    in progress

    on Feb 26, 2015