Implement Tokenizer op - #31

Merged

Dmitri Smirnov (yuslepukhin) merged 12 commits into

dmitrism/implement_tokenizer

Dec 6, 2018

Commits on Dec 4, 2018

Implement separator tokenizer with TST.
Dmitri Smirnov
committed
d12bc6b
Add utf8 util test
Dmitri Smirnov
committed
2a5e9dd
For empty output produce [C] -> [C][0], [N][C] -> [N][C][0]
Dmitri Smirnov
committed
c683c31
Augument TST search with match conflict resolution in favor of
Dmitri Smirnov
committed
0d490ec
Address MAcOS build error.
Dmitri Smirnov
committed
ef59eaf
Adjust error message
Dmitri Smirnov
committed
c52985f
Address review comments.
Dmitri Smirnov
committed
0a89353
Remove nested loops.
Dmitri Smirnov
committed
afdf923
Remove 3rd party utf8 validation code.
Dmitri Smirnov
committed
e23ba6f

Commits on Dec 5, 2018

Address review comments part I.
Dmitri Smirnov
committed
845853c
Move padding outside start/end markers.
Dmitri Smirnov
committed
cf120f8

Commits on Dec 6, 2018

Fix a common prefix bug reported by Xavier.
Dmitri Smirnov
committed
fb054d4