Reece H. Dunn
d2d718d700
tokenizer.c: Tokenize line separator codepoints as newline tokens.
2017-04-28 07:16:38 +01:00
Reece H. Dunn
bf45e7ce36
tokenizer.c: Recognise U+0085 [NEW LINE (NEL)] as a newline codepoint.
2017-04-28 07:08:02 +01:00
Reece H. Dunn
df6ca7a22c
tokenizer.c: Support whitespace tokens.
2017-04-27 07:59:29 +01:00
Reece H. Dunn
539edac795
tokenizer.c: Create a codepoint_type helper function to classify codepoints for the tokenizer.
2017-04-27 07:12:27 +01:00
Reece H. Dunn
8f0dae6a38
tokenizer.c: Support windows newlines.
2017-04-26 07:52:33 +01:00
Reece H. Dunn
b897ff5aa8
encoding.c: Support calling peekc past the end of the buffer. This makes calling peekc easier.
2017-04-26 07:48:34 +01:00
Reece H. Dunn
3f692f498b
encoding.c: Implement a peekc API.
2017-04-26 07:45:21 +01:00
Reece H. Dunn
1c8ed9c190
tokenizer.c: Support mac newlines.
2017-04-26 07:39:06 +01:00
Reece H. Dunn
7602c9ac18
tokenizer.c: Support linux newlines.
2017-04-26 07:34:57 +01:00
Reece H. Dunn
bce44316bb
Create a basic tokenizer API using a structure that mirrors the TtsTokenizer interface in the tts-dev-studio project.
2017-04-25 07:55:10 +01:00
Reece H. Dunn
3cc53d98f4
Add ucd.h to tokenizer.c to provide the definition of the ucd_category identifier for the emscripten build.
2017-04-24 18:49:09 +01:00
Reece H. Dunn
61d668c0cb
ucd-tools: Inverted_Terminal_Punctuation eSpeakNG extended property support; use in clause_type_from_codepoint.
2017-04-22 19:09:49 +01:00
Reece H. Dunn
5c6bc0e556
Armenian emphasis mark (U+055B) is used for interjections, so treat it as an exclamation mark.
2017-04-22 18:54:21 +01:00
Reece H. Dunn
bc13173ac4
ucd-tools: Punctuation_In_Word eSpeakNG extended property support; use in clause_type_from_codepoint.
2017-04-22 18:52:46 +01:00
Reece H. Dunn
1131d0924b
ucd-tools: Optional_Space_After eSpeakNG extended property support; use in clause_type_from_codepoint.
2017-04-22 18:17:39 +01:00
Reece H. Dunn
b932f3c493
ucd-tools: Extended_Dash eSpeakNG extended property support; use in clause_type_from_codepoint.
2017-04-22 15:55:44 +01:00
Reece H. Dunn
3100ca9d1b
Use ucd_properties to implement clause_type_from_codepoint for supported types.
2017-04-22 15:20:19 +01:00
Reece H. Dunn
1c4ce3dcd3
tokenizer.c: create and use a clause_type_from_codepoint function, with tests.
2017-04-19 22:33:32 +01:00
Reece H. Dunn
92f703d98b
Use defines instead of hard-coded numbers for more clause logic.
2017-04-18 07:14:45 +01:00
Reece H. Dunn
8749891069
Better specify the CLAUSE_ flags returned by ReadClause.
2017-04-17 22:04:45 +01:00
Reece H. Dunn
e4e1e4db0a
TranslateWord: remove the unused add_plural_suffix variable.
2017-04-06 19:39:03 +01:00
Reece H. Dunn
62d4aff9a9
Remove the now unused option_multibyte variable.
2017-04-01 15:12:49 +01:00
Reece H. Dunn
ec8a7b810f
Use the text decoder object at the top-level Synthesize/espeak_TextToPhonemes call, not in TranslateClause.
2017-04-01 15:11:01 +01:00
Reece H. Dunn
b3e0fbc8ed
encoding.c: Create a text_decoder_decode_string_multibyte helper to work with the espeakCHARS_* flags.
2017-04-01 13:36:18 +01:00
Reece H. Dunn
9dabf64680
encoding.c: Support determining the string length for length < 0.
2017-04-01 13:23:09 +01:00
Reece H. Dunn
b5ed1f28a5
encoding.c: Don't crash if NULL is passed as the string to the decode APIs.
2017-04-01 13:00:00 +01:00
Reece H. Dunn
d167d5649b
encoding.c: Implement support for the auto-detected character set (utf-8 + codepoint-encoding).
2017-03-31 22:30:27 +01:00
Reece H. Dunn
be480c12de
Make TranslateClause return 'const void *' to preserve constness.
2017-03-31 20:23:22 +01:00
Reece H. Dunn
6451917bde
encoding.c: Fix text_decoder_get_buffer at EOF.
2017-03-31 20:21:28 +01:00
Reece H. Dunn
7c16ac543c
Use the text decoder API in readclause.c.
2017-03-31 13:17:37 +01:00
Reece H. Dunn
8933185de4
Remove the unused f_in argument to the Read/Translate/SpeakNextClause functions.
2017-03-31 07:13:36 +01:00
Reece H. Dunn
0b0661cef0
Use the encoding.c tables for 8-bit encodings.
...
1. Store the encoding enumeration values in the Translation
object, instead of the charset table.
2. Use the encoding.c charset table data instead of the ones
in translate.c.
3. Remove the charset language file option -- it is only used
in the Arabic language file, but is used incorrectly there.
4. Specify ISO 8859-6 for the 8-bit encoding for Arabic instead
of UTF-8, so that espeakCHARS_8BIT and espeakCHARS_AUTO work
correctly for Arabic.
2017-03-31 00:01:14 +01:00
Reece H. Dunn
a714c0554b
encoding.c: Use a codepage table to implement ISO-8859-1.
2017-03-30 23:35:00 +01:00
Reece H. Dunn
db314438de
encoding.c: don't generate warnings in text_decoder_decode_string.
2017-03-30 23:10:33 +01:00
Reece H. Dunn
6a0b5e4ae1
encoding.c: Support using wchar_t strings with the text decoder API.
2017-03-30 22:52:38 +01:00
Reece H. Dunn
2de0368d39
encoding.c: Store the start/end positions as 'const utf8_t *'.
2017-03-30 22:45:01 +01:00
Reece H. Dunn
b74f756f00
encoding.c: Support the ISO-10646-UCS-2 encoding.
2017-03-30 22:33:22 +01:00
Reece H. Dunn
fa5d31a8af
encoding.c: Support the UTF-8 encoding.
2017-03-30 21:17:56 +01:00
Reece H. Dunn
27ff1f6b69
encoding.c: Use 16 bits for the codepage encodings as they are all in the Basic Multilingual Plane.
2017-03-29 20:33:32 +01:00
Reece H. Dunn
2499610433
encoding.c: Support the ISCII encoding.
2017-03-29 08:01:24 +01:00
Reece H. Dunn
39f3ea54cf
encoding.c: Support the KOI8-R encoding.
2017-03-29 07:49:48 +01:00
Reece H. Dunn
b8a1006dd8
encoding.c: Support the ISO 8859-16 encoding.
2017-03-29 07:42:00 +01:00
Reece H. Dunn
166e815723
encoding.c: Support the ISO 8859-15 encoding.
2017-03-29 07:36:42 +01:00
Reece H. Dunn
91e054ec7c
encoding.c: Fix the ISO 8859 encoding names with date suffices.
2017-03-29 07:27:30 +01:00
Reece H. Dunn
0235c42652
encoding.c: Support the ISO 8859-14 encoding.
2017-03-29 07:19:16 +01:00
Reece H. Dunn
24faceab57
encoding.c: Support the ISO 8859-13 encoding.
2017-03-28 23:26:33 +01:00
Reece H. Dunn
495c0aed20
encoding.c: Support the ISO 8859-11 encoding.
2017-03-28 23:13:41 +01:00
Reece H. Dunn
84f20f8bb8
encoding.c: Support the ISO 8859-10 encoding.
2017-03-28 22:50:50 +01:00
Reece H. Dunn
1d2d0acd95
encoding.c: Use a lookup table to find the appropriate decoder.
2017-03-28 21:38:58 +01:00
Reece H. Dunn
0421f127e8
encoding.c: Support the ISO 8859-9 encoding.
2017-03-28 21:02:22 +01:00