 Reece H. Dunn
|
fc7a4e6701
|
tokenizer.c: Recognise U+000C [FORM FEED (FF)] as a newline codepoint.
|
2017-04-28 07:31:05 +01:00 |
|
|
Reece H. Dunn
|
d2d718d700
|
tokenizer.c: Tokenize line separator codepoints as newline tokens.
|
2017-04-28 07:16:38 +01:00 |
|
|
Reece H. Dunn
|
bf45e7ce36
|
tokenizer.c: Recognise U+0085 [NEW LINE (NEL)] as a newline codepoint.
|
2017-04-28 07:08:02 +01:00 |
|
|
Reece H. Dunn
|
df6ca7a22c
|
tokenizer.c: Support whitespace tokens.
|
2017-04-27 07:59:29 +01:00 |
|
|
Reece H. Dunn
|
539edac795
|
tokenizer.c: Create a codepoint_type helper function to classify codepoints for the tokenizer.
|
2017-04-27 07:12:27 +01:00 |
|
|
Reece H. Dunn
|
8f0dae6a38
|
tokenizer.c: Support windows newlines.
|
2017-04-26 07:52:33 +01:00 |
|
|
Reece H. Dunn
|
1c8ed9c190
|
tokenizer.c: Support mac newlines.
|
2017-04-26 07:39:06 +01:00 |
|
|
Reece H. Dunn
|
7602c9ac18
|
tokenizer.c: Support linux newlines.
|
2017-04-26 07:34:57 +01:00 |
|
|
Reece H. Dunn
|
bce44316bb
|
Create a basic tokenizer API using a structure that mirrors the TtsTokenizer interface in the tts-dev-studio project.
|
2017-04-25 07:55:10 +01:00 |
|
|
Reece H. Dunn
|
3cc53d98f4
|
Add ucd.h to tokenizer.c to provide the definition of the ucd_category identifier for the emscripten build.
|
2017-04-24 18:49:09 +01:00 |
|
|
Reece H. Dunn
|
61d668c0cb
|
ucd-tools: Inverted_Terminal_Punctuation eSpeakNG extended property support; use in clause_type_from_codepoint.
|
2017-04-22 19:09:49 +01:00 |
|
|
Reece H. Dunn
|
5c6bc0e556
|
Armenian emphasis mark (U+055B) is used for interjections, so treat it as an exclamation mark.
|
2017-04-22 18:54:21 +01:00 |
|
|
Reece H. Dunn
|
bc13173ac4
|
ucd-tools: Punctuation_In_Word eSpeakNG extended property support; use in clause_type_from_codepoint.
|
2017-04-22 18:52:46 +01:00 |
|
|
Reece H. Dunn
|
1131d0924b
|
ucd-tools: Optional_Space_After eSpeakNG extended property support; use in clause_type_from_codepoint.
|
2017-04-22 18:17:39 +01:00 |
|
|
Reece H. Dunn
|
b932f3c493
|
ucd-tools: Extended_Dash eSpeakNG extended property support; use in clause_type_from_codepoint.
|
2017-04-22 15:55:44 +01:00 |
|
|
Reece H. Dunn
|
3100ca9d1b
|
Use ucd_properties to implement clause_type_from_codepoint for supported types.
|
2017-04-22 15:20:19 +01:00 |
|
|
Reece H. Dunn
|
1c4ce3dcd3
|
tokenizer.c: create and use a clause_type_from_codepoint function, with tests.
|
2017-04-19 22:33:32 +01:00 |
|