 Reece H. Dunn
|
a902f451d8
|
tests/tokenizer.test: Support printing the tokens from a provided file, making it easy to investigate tokenizer issues.
|
2017-04-29 10:13:16 +01:00 |
|
|
Reece H. Dunn
|
d093513b65
|
tokenizer.c: Add an options parameter to the tokenizer_reset API.
|
2017-04-28 07:48:55 +01:00 |
|
|
Reece H. Dunn
|
c41ac642fa
|
tokenizer.c: Tokenise Zp codepoints as paragraphs.
|
2017-04-28 07:44:22 +01:00 |
|
|
Reece H. Dunn
|
f3ea6f68f3
|
tokenizer.c: Tokenise U+000B [VERTICAL TAB (VT)] as whitespace, not as newlines.
|
2017-04-28 07:37:00 +01:00 |
|
|
Reece H. Dunn
|
fc7a4e6701
|
tokenizer.c: Recognise U+000C [FORM FEED (FF)] as a newline codepoint.
|
2017-04-28 07:31:05 +01:00 |
|
|
Reece H. Dunn
|
d2d718d700
|
tokenizer.c: Tokenize line separator codepoints as newline tokens.
|
2017-04-28 07:16:38 +01:00 |
|
|
Reece H. Dunn
|
bf45e7ce36
|
tokenizer.c: Recognise U+0085 [NEW LINE (NEL)] as a newline codepoint.
|
2017-04-28 07:08:02 +01:00 |
|
|
Reece H. Dunn
|
df6ca7a22c
|
tokenizer.c: Support whitespace tokens.
|
2017-04-27 07:59:29 +01:00 |
|
|
Reece H. Dunn
|
8f0dae6a38
|
tokenizer.c: Support windows newlines.
|
2017-04-26 07:52:33 +01:00 |
|
|
Reece H. Dunn
|
1c8ed9c190
|
tokenizer.c: Support mac newlines.
|
2017-04-26 07:39:06 +01:00 |
|
|
Reece H. Dunn
|
7602c9ac18
|
tokenizer.c: Support linux newlines.
|
2017-04-26 07:34:57 +01:00 |
|
|
Reece H. Dunn
|
bce44316bb
|
Create a basic tokenizer API using a structure that mirrors the TtsTokenizer interface in the tts-dev-studio project.
|
2017-04-25 07:55:10 +01:00 |
|
|
Reece H. Dunn
|
5c6bc0e556
|
Armenian emphasis mark (U+055B) is used for interjections, so treat it as an exclamation mark.
|
2017-04-22 18:54:21 +01:00 |
|
|
Reece H. Dunn
|
1c4ce3dcd3
|
tokenizer.c: create and use a clause_type_from_codepoint function, with tests.
|
2017-04-19 22:33:32 +01:00 |
|