Commit Graph
512 Commits
Author SHA1 Message Date
Reece H. Dunn d2d718d700 tokenizer.c: Tokenize line separator codepoints as newline tokens. 2017-04-28 07:16:38 +01:00
Reece H. Dunn bf45e7ce36 tokenizer.c: Recognise U+0085 [NEW LINE (NEL)] as a newline codepoint. 2017-04-28 07:08:02 +01:00
Reece H. Dunn df6ca7a22c tokenizer.c: Support whitespace tokens. 2017-04-27 07:59:29 +01:00
Reece H. Dunn 539edac795 tokenizer.c: Create a codepoint_type helper function to classify codepoints for the tokenizer. 2017-04-27 07:12:27 +01:00
Reece H. Dunn 8f0dae6a38 tokenizer.c: Support windows newlines. 2017-04-26 07:52:33 +01:00
Reece H. Dunn b897ff5aa8 encoding.c: Support calling peekc past the end of the buffer. This makes calling peekc easier. 2017-04-26 07:48:34 +01:00
Reece H. Dunn 3f692f498b encoding.c: Implement a peekc API. 2017-04-26 07:45:21 +01:00
Reece H. Dunn 1c8ed9c190 tokenizer.c: Support mac newlines. 2017-04-26 07:39:06 +01:00
Reece H. Dunn 7602c9ac18 tokenizer.c: Support linux newlines. 2017-04-26 07:34:57 +01:00
Reece H. Dunn bce44316bb Create a basic tokenizer API using a structure that mirrors the TtsTokenizer interface in the tts-dev-studio project. 2017-04-25 07:55:10 +01:00
Reece H. Dunn 3cc53d98f4 Add ucd.h to tokenizer.c to provide the definition of the ucd_category identifier for the emscripten build. 2017-04-24 18:49:09 +01:00
Reece H. Dunn 61d668c0cb ucd-tools: Inverted_Terminal_Punctuation eSpeakNG extended property support; use in clause_type_from_codepoint. 2017-04-22 19:09:49 +01:00
Reece H. Dunn 5c6bc0e556 Armenian emphasis mark (U+055B) is used for interjections, so treat it as an exclamation mark. 2017-04-22 18:54:21 +01:00
Reece H. Dunn bc13173ac4 ucd-tools: Punctuation_In_Word eSpeakNG extended property support; use in clause_type_from_codepoint. 2017-04-22 18:52:46 +01:00
Reece H. Dunn 1131d0924b ucd-tools: Optional_Space_After eSpeakNG extended property support; use in clause_type_from_codepoint. 2017-04-22 18:17:39 +01:00
Reece H. Dunn b932f3c493 ucd-tools: Extended_Dash eSpeakNG extended property support; use in clause_type_from_codepoint. 2017-04-22 15:55:44 +01:00
Reece H. Dunn 3100ca9d1b Use ucd_properties to implement clause_type_from_codepoint for supported types. 2017-04-22 15:20:19 +01:00
Reece H. Dunn 1c4ce3dcd3 tokenizer.c: create and use a clause_type_from_codepoint function, with tests. 2017-04-19 22:33:32 +01:00
Reece H. Dunn 92f703d98b Use defines instead of hard-coded numbers for more clause logic. 2017-04-18 07:14:45 +01:00
Reece H. Dunn 8749891069 Better specify the CLAUSE_ flags returned by ReadClause. 2017-04-17 22:04:45 +01:00
Reece H. Dunn e4e1e4db0a TranslateWord: remove the unused add_plural_suffix variable. 2017-04-06 19:39:03 +01:00
Reece H. Dunn 62d4aff9a9 Remove the now unused option_multibyte variable. 2017-04-01 15:12:49 +01:00
Reece H. Dunn ec8a7b810f Use the text decoder object at the top-level Synthesize/espeak_TextToPhonemes call, not in TranslateClause. 2017-04-01 15:11:01 +01:00
Reece H. Dunn b3e0fbc8ed encoding.c: Create a text_decoder_decode_string_multibyte helper to work with the espeakCHARS_* flags. 2017-04-01 13:36:18 +01:00
Reece H. Dunn 9dabf64680 encoding.c: Support determining the string length for length < 0. 2017-04-01 13:23:09 +01:00
Reece H. Dunn b5ed1f28a5 encoding.c: Don't crash if NULL is passed as the string to the decode APIs. 2017-04-01 13:00:00 +01:00
Reece H. Dunn d167d5649b encoding.c: Implement support for the auto-detected character set (utf-8 + codepoint-encoding). 2017-03-31 22:30:27 +01:00
Reece H. Dunn be480c12de Make TranslateClause return 'const void *' to preserve constness. 2017-03-31 20:23:22 +01:00
Reece H. Dunn 6451917bde encoding.c: Fix text_decoder_get_buffer at EOF. 2017-03-31 20:21:28 +01:00
Reece H. Dunn 7c16ac543c Use the text decoder API in readclause.c. 2017-03-31 13:17:37 +01:00
Reece H. Dunn 8933185de4 Remove the unused f_in argument to the Read/Translate/SpeakNextClause functions. 2017-03-31 07:13:36 +01:00
Reece H. Dunn 0b0661cef0 Use the encoding.c tables for 8-bit encodings.
1.  Store the encoding enumeration values in the Translation
    object, instead of the charset table.

2.  Use the encoding.c charset table data instead of the ones
    in translate.c.

3.  Remove the charset language file option -- it is only used
    in the Arabic language file, but is used incorrectly there.

4.  Specify ISO 8859-6 for the 8-bit encoding for Arabic instead
    of UTF-8, so that espeakCHARS_8BIT and espeakCHARS_AUTO work
    correctly for Arabic.
2017-03-31 00:01:14 +01:00
Reece H. Dunn a714c0554b encoding.c: Use a codepage table to implement ISO-8859-1. 2017-03-30 23:35:00 +01:00
Reece H. Dunn db314438de encoding.c: don't generate warnings in text_decoder_decode_string. 2017-03-30 23:10:33 +01:00
Reece H. Dunn 6a0b5e4ae1 encoding.c: Support using wchar_t strings with the text decoder API. 2017-03-30 22:52:38 +01:00
Reece H. Dunn 2de0368d39 encoding.c: Store the start/end positions as 'const utf8_t *'. 2017-03-30 22:45:01 +01:00
Reece H. Dunn b74f756f00 encoding.c: Support the ISO-10646-UCS-2 encoding. 2017-03-30 22:33:22 +01:00
Reece H. Dunn fa5d31a8af encoding.c: Support the UTF-8 encoding. 2017-03-30 21:17:56 +01:00
Reece H. Dunn 27ff1f6b69 encoding.c: Use 16 bits for the codepage encodings as they are all in the Basic Multilingual Plane. 2017-03-29 20:33:32 +01:00
Reece H. Dunn 2499610433 encoding.c: Support the ISCII encoding. 2017-03-29 08:01:24 +01:00
Reece H. Dunn 39f3ea54cf encoding.c: Support the KOI8-R encoding. 2017-03-29 07:49:48 +01:00
Reece H. Dunn b8a1006dd8 encoding.c: Support the ISO 8859-16 encoding. 2017-03-29 07:42:00 +01:00
Reece H. Dunn 166e815723 encoding.c: Support the ISO 8859-15 encoding. 2017-03-29 07:36:42 +01:00
Reece H. Dunn 91e054ec7c encoding.c: Fix the ISO 8859 encoding names with date suffices. 2017-03-29 07:27:30 +01:00
Reece H. Dunn 0235c42652 encoding.c: Support the ISO 8859-14 encoding. 2017-03-29 07:19:16 +01:00
Reece H. Dunn 24faceab57 encoding.c: Support the ISO 8859-13 encoding. 2017-03-28 23:26:33 +01:00
Reece H. Dunn 495c0aed20 encoding.c: Support the ISO 8859-11 encoding. 2017-03-28 23:13:41 +01:00
Reece H. Dunn 84f20f8bb8 encoding.c: Support the ISO 8859-10 encoding. 2017-03-28 22:50:50 +01:00
Reece H. Dunn 1d2d0acd95 encoding.c: Use a lookup table to find the appropriate decoder. 2017-03-28 21:38:58 +01:00
Reece H. Dunn 0421f127e8 encoding.c: Support the ISO 8859-9 encoding. 2017-03-28 21:02:22 +01:00