8 Commits

Include renames

Author	SHA1	Message	Date
Reece H. Dunn	bf45e7ce36	tokenizer.c: Recognise U+0085 [NEW LINE (NEL)] as a newline codepoint.	2017-04-28 07:08:02 +01:00
Reece H. Dunn	df6ca7a22c	tokenizer.c: Support whitespace tokens.	2017-04-27 07:59:29 +01:00
Reece H. Dunn	8f0dae6a38	tokenizer.c: Support windows newlines.	2017-04-26 07:52:33 +01:00
Reece H. Dunn	1c8ed9c190	tokenizer.c: Support mac newlines.	2017-04-26 07:39:06 +01:00
Reece H. Dunn	7602c9ac18	tokenizer.c: Support linux newlines.	2017-04-26 07:34:57 +01:00
Reece H. Dunn	bce44316bb	Create a basic tokenizer API using a structure that mirrors the TtsTokenizer interface in the tts-dev-studio project.	2017-04-25 07:55:10 +01:00
Reece H. Dunn	5c6bc0e556	Armenian emphasis mark (U+055B) is used for interjections, so treat it as an exclamation mark.	2017-04-22 18:54:21 +01:00
Reece H. Dunn	1c4ce3dcd3	tokenizer.c: create and use a clause_type_from_codepoint function, with tests.	2017-04-19 22:33:32 +01:00