Commit Graph
2794 Commits
Author SHA1 Message Date
Reece H. Dunn a902f451d8 tests/tokenizer.test: Support printing the tokens from a provided file, making it easy to investigate tokenizer issues. 2017-04-29 10:13:16 +01:00
Reece H. Dunn d093513b65 tokenizer.c: Add an options parameter to the tokenizer_reset API. 2017-04-28 07:48:55 +01:00
Reece H. Dunn c41ac642fa tokenizer.c: Tokenise Zp codepoints as paragraphs. 2017-04-28 07:44:22 +01:00
Reece H. Dunn f3ea6f68f3 tokenizer.c: Tokenise U+000B [VERTICAL TAB (VT)] as whitespace, not as newlines. 2017-04-28 07:37:00 +01:00
Reece H. Dunn fc7a4e6701 tokenizer.c: Recognise U+000C [FORM FEED (FF)] as a newline codepoint. 2017-04-28 07:31:05 +01:00
Reece H. Dunn d2d718d700 tokenizer.c: Tokenize line separator codepoints as newline tokens. 2017-04-28 07:16:38 +01:00
Reece H. Dunn bf45e7ce36 tokenizer.c: Recognise U+0085 [NEW LINE (NEL)] as a newline codepoint. 2017-04-28 07:08:02 +01:00
Reece H. Dunn df6ca7a22c tokenizer.c: Support whitespace tokens. 2017-04-27 07:59:29 +01:00
Reece H. Dunn 539edac795 tokenizer.c: Create a codepoint_type helper function to classify codepoints for the tokenizer. 2017-04-27 07:12:27 +01:00
Reece H. Dunn 7b1243679f Update the document for the new fr-CH French accent. 2017-04-26 19:02:34 +01:00
Reece H. Dunn 4bc3f15e79 Generalize the exclusion of Windows batch files. 2017-04-26 18:52:00 +01:00
claude beazley c05e3898a4 Adding Swiss French Variant
Creating  the language  variant , swiss french. Primarily for counting
as Swiss French uses huitante  for 80 and ,  like the Belgians septante
eand nonante for  70 and 90.
2017-04-26 12:19:17 +02:00
Reece H. Dunn 8f0dae6a38 tokenizer.c: Support windows newlines. 2017-04-26 07:52:33 +01:00
Reece H. Dunn b897ff5aa8 encoding.c: Support calling peekc past the end of the buffer. This makes calling peekc easier. 2017-04-26 07:48:34 +01:00
Reece H. Dunn 3f692f498b encoding.c: Implement a peekc API. 2017-04-26 07:45:21 +01:00
Reece H. Dunn 1c8ed9c190 tokenizer.c: Support mac newlines. 2017-04-26 07:39:06 +01:00
Reece H. Dunn 7602c9ac18 tokenizer.c: Support linux newlines. 2017-04-26 07:34:57 +01:00
Reece H. Dunn bce44316bb Create a basic tokenizer API using a structure that mirrors the TtsTokenizer interface in the tts-dev-studio project. 2017-04-25 07:55:10 +01:00
Reece H. Dunn 3cc53d98f4 Add ucd.h to tokenizer.c to provide the definition of the ucd_category identifier for the emscripten build. 2017-04-24 18:49:09 +01:00
Reece H. Dunn ee61cc4358 Fix running 'make clean' when gradle is not present. Gradle is used for the Android build and is not needed when just building eSpeak NG on Linux/BSD systems. 2017-04-24 18:37:06 +01:00
Reece H. Dunn a72199f714 Run the tests as part of the Travis build. 2017-04-22 19:14:32 +01:00
Reece H. Dunn 61d668c0cb ucd-tools: Inverted_Terminal_Punctuation eSpeakNG extended property support; use in clause_type_from_codepoint. 2017-04-22 19:09:49 +01:00
Reece H. Dunn 5c6bc0e556 Armenian emphasis mark (U+055B) is used for interjections, so treat it as an exclamation mark. 2017-04-22 18:54:21 +01:00
Reece H. Dunn bc13173ac4 ucd-tools: Punctuation_In_Word eSpeakNG extended property support; use in clause_type_from_codepoint. 2017-04-22 18:52:46 +01:00
Reece H. Dunn 1131d0924b ucd-tools: Optional_Space_After eSpeakNG extended property support; use in clause_type_from_codepoint. 2017-04-22 18:17:39 +01:00
Reece H. Dunn b932f3c493 ucd-tools: Extended_Dash eSpeakNG extended property support; use in clause_type_from_codepoint. 2017-04-22 15:55:44 +01:00
Reece H. Dunn 3100ca9d1b Use ucd_properties to implement clause_type_from_codepoint for supported types. 2017-04-22 15:20:19 +01:00
Reece H. Dunn be86091088 Use #defines for the ESPEAKNG_PROPERTY_ constants, so they can be used in things like switch expressions. 2017-04-22 14:33:51 +01:00
Reece H. Dunn af88820954 Merge commit 'd18d98b92ca42bf7b098d3d1fd873d9b0ee82e00' 2017-04-22 14:31:07 +01:00
Reece H. Dunn d18d98b92c Use #defines for the UCD_PROPERTY_ constants, so they can be used in things like switch expressions. 2017-04-22 14:28:41 +01:00
Reece H. Dunn 1c4ce3dcd3 tokenizer.c: create and use a clause_type_from_codepoint function, with tests. 2017-04-19 22:33:32 +01:00
Reece H. Dunn 92f703d98b Use defines instead of hard-coded numbers for more clause logic. 2017-04-18 07:14:45 +01:00
Reece H. Dunn 8749891069 Better specify the CLAUSE_ flags returned by ReadClause. 2017-04-17 22:04:45 +01:00
Reece H. Dunn 2e375362c4 ucd-tools: Paragraph_Separator eSpeakNG extended property support. 2017-04-17 20:20:42 +01:00
Reece H. Dunn 7fc4f5ece2 ucd-tools: Ellipsis eSpeakNG extended property support. 2017-04-17 17:35:26 +01:00
Reece H. Dunn 8d8c8b3b56 ucd-tools: Semi_Colon eSpeakNG extended property support. 2017-04-17 17:17:52 +01:00
Reece H. Dunn 9869ee051e ucd-tools: Colon eSpeakNG extended property support. 2017-04-17 16:18:57 +01:00
Reece H. Dunn 9ef03b8ac8 ucd-tools: Comma eSpeakNG extended property support. 2017-04-17 15:14:02 +01:00
Reece H. Dunn 5017153d62 ucd-tools: Exclamation_Mark eSpeakNG extended property support. 2017-04-17 13:47:50 +01:00
Reece H. Dunn 31d66fddb1 ucd-tools: Question_Mark eSpeakNG extended property support. 2017-04-17 12:50:04 +01:00
Reece H. Dunn 07cd2b12e1 ucd-tools: Full_Stop eSpeakNG extended property support. 2017-04-17 11:45:21 +01:00
Reece H. Dunn 9bbb16a74c Add ucd-tools proplist.c to the build. 2017-04-16 15:04:15 +01:00
Reece H. Dunn 2615242f05 Merge commit 'ff018e33df5763dc0ff1644c9479ee2351bee45e' 2017-04-16 15:02:08 +01:00
Reece H. Dunn ff018e33df Support compiling ucd.h with a strict C11 compiler. 2017-04-16 15:01:37 +01:00
Reece H. Dunn 1a4b9d087b Add the new PropList API to the README file. 2017-04-16 14:44:55 +01:00
Reece H. Dunn 691457e98d Add Prepended_Concatenation_Mark support from PropList.txt. 2017-04-16 14:36:27 +01:00
Reece H. Dunn 429b8f3629 Add Pattern_Syntax support from PropList.txt. 2017-04-16 14:17:43 +01:00
Reece H. Dunn d91a249c14 Add Pattern_White_Space support from PropList.txt. 2017-04-16 11:37:14 +01:00
Reece H. Dunn eecd7984c5 Add Variation_Selector support from PropList.txt. 2017-04-16 11:26:05 +01:00
Reece H. Dunn f7f228bc0c Add Sentence_Terminal support from PropList.txt. 2017-04-16 11:05:59 +01:00