Commit Graph
1357 Commits
Author SHA1 Message Date
Reece H. Dunn d093513b65 tokenizer.c: Add an options parameter to the tokenizer_reset API. 2017-04-28 07:48:55 +01:00
Reece H. Dunn c41ac642fa tokenizer.c: Tokenise Zp codepoints as paragraphs. 2017-04-28 07:44:22 +01:00
Reece H. Dunn fc7a4e6701 tokenizer.c: Recognise U+000C [FORM FEED (FF)] as a newline codepoint. 2017-04-28 07:31:05 +01:00
Reece H. Dunn d2d718d700 tokenizer.c: Tokenize line separator codepoints as newline tokens. 2017-04-28 07:16:38 +01:00
Reece H. Dunn bf45e7ce36 tokenizer.c: Recognise U+0085 [NEW LINE (NEL)] as a newline codepoint. 2017-04-28 07:08:02 +01:00
Reece H. Dunn df6ca7a22c tokenizer.c: Support whitespace tokens. 2017-04-27 07:59:29 +01:00
Reece H. Dunn 539edac795 tokenizer.c: Create a codepoint_type helper function to classify codepoints for the tokenizer. 2017-04-27 07:12:27 +01:00
Reece H. Dunn 8f0dae6a38 tokenizer.c: Support windows newlines. 2017-04-26 07:52:33 +01:00
Reece H. Dunn b897ff5aa8 encoding.c: Support calling peekc past the end of the buffer. This makes calling peekc easier. 2017-04-26 07:48:34 +01:00
Reece H. Dunn 3f692f498b encoding.c: Implement a peekc API. 2017-04-26 07:45:21 +01:00
Reece H. Dunn 1c8ed9c190 tokenizer.c: Support mac newlines. 2017-04-26 07:39:06 +01:00
Reece H. Dunn 7602c9ac18 tokenizer.c: Support linux newlines. 2017-04-26 07:34:57 +01:00
Reece H. Dunn bce44316bb Create a basic tokenizer API using a structure that mirrors the TtsTokenizer interface in the tts-dev-studio project. 2017-04-25 07:55:10 +01:00
Reece H. Dunn 3cc53d98f4 Add ucd.h to tokenizer.c to provide the definition of the ucd_category identifier for the emscripten build. 2017-04-24 18:49:09 +01:00
Reece H. Dunn 61d668c0cb ucd-tools: Inverted_Terminal_Punctuation eSpeakNG extended property support; use in clause_type_from_codepoint. 2017-04-22 19:09:49 +01:00
Reece H. Dunn 5c6bc0e556 Armenian emphasis mark (U+055B) is used for interjections, so treat it as an exclamation mark. 2017-04-22 18:54:21 +01:00
Reece H. Dunn bc13173ac4 ucd-tools: Punctuation_In_Word eSpeakNG extended property support; use in clause_type_from_codepoint. 2017-04-22 18:52:46 +01:00
Reece H. Dunn 1131d0924b ucd-tools: Optional_Space_After eSpeakNG extended property support; use in clause_type_from_codepoint. 2017-04-22 18:17:39 +01:00
Reece H. Dunn b932f3c493 ucd-tools: Extended_Dash eSpeakNG extended property support; use in clause_type_from_codepoint. 2017-04-22 15:55:44 +01:00
Reece H. Dunn 3100ca9d1b Use ucd_properties to implement clause_type_from_codepoint for supported types. 2017-04-22 15:20:19 +01:00
Reece H. Dunn be86091088 Use #defines for the ESPEAKNG_PROPERTY_ constants, so they can be used in things like switch expressions. 2017-04-22 14:33:51 +01:00
Reece H. Dunn af88820954 Merge commit 'd18d98b92ca42bf7b098d3d1fd873d9b0ee82e00' 2017-04-22 14:31:07 +01:00
Reece H. Dunn d18d98b92c Use #defines for the UCD_PROPERTY_ constants, so they can be used in things like switch expressions. 2017-04-22 14:28:41 +01:00
Reece H. Dunn 1c4ce3dcd3 tokenizer.c: create and use a clause_type_from_codepoint function, with tests. 2017-04-19 22:33:32 +01:00
Reece H. Dunn 92f703d98b Use defines instead of hard-coded numbers for more clause logic. 2017-04-18 07:14:45 +01:00
Reece H. Dunn 8749891069 Better specify the CLAUSE_ flags returned by ReadClause. 2017-04-17 22:04:45 +01:00
Reece H. Dunn 2e375362c4 ucd-tools: Paragraph_Separator eSpeakNG extended property support. 2017-04-17 20:20:42 +01:00
Reece H. Dunn 7fc4f5ece2 ucd-tools: Ellipsis eSpeakNG extended property support. 2017-04-17 17:35:26 +01:00
Reece H. Dunn 8d8c8b3b56 ucd-tools: Semi_Colon eSpeakNG extended property support. 2017-04-17 17:17:52 +01:00
Reece H. Dunn 9869ee051e ucd-tools: Colon eSpeakNG extended property support. 2017-04-17 16:18:57 +01:00
Reece H. Dunn 9ef03b8ac8 ucd-tools: Comma eSpeakNG extended property support. 2017-04-17 15:14:02 +01:00
Reece H. Dunn 5017153d62 ucd-tools: Exclamation_Mark eSpeakNG extended property support. 2017-04-17 13:47:50 +01:00
Reece H. Dunn 31d66fddb1 ucd-tools: Question_Mark eSpeakNG extended property support. 2017-04-17 12:50:04 +01:00
Reece H. Dunn 07cd2b12e1 ucd-tools: Full_Stop eSpeakNG extended property support. 2017-04-17 11:45:21 +01:00
Reece H. Dunn 9bbb16a74c Add ucd-tools proplist.c to the build. 2017-04-16 15:04:15 +01:00
Reece H. Dunn 2615242f05 Merge commit 'ff018e33df5763dc0ff1644c9479ee2351bee45e' 2017-04-16 15:02:08 +01:00
Reece H. Dunn ff018e33df Support compiling ucd.h with a strict C11 compiler. 2017-04-16 15:01:37 +01:00
Reece H. Dunn 691457e98d Add Prepended_Concatenation_Mark support from PropList.txt. 2017-04-16 14:36:27 +01:00
Reece H. Dunn 429b8f3629 Add Pattern_Syntax support from PropList.txt. 2017-04-16 14:17:43 +01:00
Reece H. Dunn d91a249c14 Add Pattern_White_Space support from PropList.txt. 2017-04-16 11:37:14 +01:00
Reece H. Dunn eecd7984c5 Add Variation_Selector support from PropList.txt. 2017-04-16 11:26:05 +01:00
Reece H. Dunn f7f228bc0c Add Sentence_Terminal support from PropList.txt. 2017-04-16 11:05:59 +01:00
Reece H. Dunn 628dfb8887 Add Other_ID_Continue support from PropList.txt. 2017-04-16 09:43:03 +01:00
Reece H. Dunn 90a3615c48 Add Other_ID_Start support from PropList.txt. 2017-04-16 09:34:31 +01:00
Reece H. Dunn a3f811aac1 Add Logical_Order_Exception support from PropList.txt. 2017-04-16 09:29:02 +01:00
Reece H. Dunn 66b2404ce3 Add Soft_Dotted support from PropList.txt. 2017-04-16 09:09:03 +01:00
Reece H. Dunn 423aa813aa Add Deprecated support from PropList.txt. 2017-04-15 12:25:00 +01:00
Reece H. Dunn e18564899c Add Other_Default_Ignorable_Code_Point support from PropList.txt. 2017-04-15 11:55:23 +01:00
Reece H. Dunn 2459a4fa8f Add Unified_Ideograph support from PropList.txt. 2017-04-14 20:58:31 +01:00
Reece H. Dunn 4ce8b61180 Extend ucd_property to 64-bits to allow all properties to be specified. 2017-04-14 20:12:02 +01:00