Reece H. Dunn
a902f451d8
tests/tokenizer.test: Support printing the tokens from a provided file, making it easy to investigate tokenizer issues.
2017-04-29 10:13:16 +01:00
Reece H. Dunn
d093513b65
tokenizer.c: Add an options parameter to the tokenizer_reset API.
2017-04-28 07:48:55 +01:00
Reece H. Dunn
c41ac642fa
tokenizer.c: Tokenise Zp codepoints as paragraphs.
2017-04-28 07:44:22 +01:00
Reece H. Dunn
f3ea6f68f3
tokenizer.c: Tokenise U+000B [VERTICAL TAB (VT)] as whitespace, not as newlines.
2017-04-28 07:37:00 +01:00
Reece H. Dunn
fc7a4e6701
tokenizer.c: Recognise U+000C [FORM FEED (FF)] as a newline codepoint.
2017-04-28 07:31:05 +01:00
Reece H. Dunn
d2d718d700
tokenizer.c: Tokenize line separator codepoints as newline tokens.
2017-04-28 07:16:38 +01:00
Reece H. Dunn
bf45e7ce36
tokenizer.c: Recognise U+0085 [NEW LINE (NEL)] as a newline codepoint.
2017-04-28 07:08:02 +01:00
Reece H. Dunn
df6ca7a22c
tokenizer.c: Support whitespace tokens.
2017-04-27 07:59:29 +01:00
Reece H. Dunn
539edac795
tokenizer.c: Create a codepoint_type helper function to classify codepoints for the tokenizer.
2017-04-27 07:12:27 +01:00
Reece H. Dunn
7b1243679f
Update the document for the new fr-CH French accent.
2017-04-26 19:02:34 +01:00
Reece H. Dunn
4bc3f15e79
Generalize the exclusion of Windows batch files.
2017-04-26 18:52:00 +01:00
claude beazley
c05e3898a4
Adding Swiss French Variant
...
Creating the language variant , swiss french. Primarily for counting
as Swiss French uses huitante for 80 and , like the Belgians septante
eand nonante for 70 and 90.
2017-04-26 12:19:17 +02:00
Reece H. Dunn
8f0dae6a38
tokenizer.c: Support windows newlines.
2017-04-26 07:52:33 +01:00
Reece H. Dunn
b897ff5aa8
encoding.c: Support calling peekc past the end of the buffer. This makes calling peekc easier.
2017-04-26 07:48:34 +01:00
Reece H. Dunn
3f692f498b
encoding.c: Implement a peekc API.
2017-04-26 07:45:21 +01:00
Reece H. Dunn
1c8ed9c190
tokenizer.c: Support mac newlines.
2017-04-26 07:39:06 +01:00
Reece H. Dunn
7602c9ac18
tokenizer.c: Support linux newlines.
2017-04-26 07:34:57 +01:00
Reece H. Dunn
bce44316bb
Create a basic tokenizer API using a structure that mirrors the TtsTokenizer interface in the tts-dev-studio project.
2017-04-25 07:55:10 +01:00
Reece H. Dunn
3cc53d98f4
Add ucd.h to tokenizer.c to provide the definition of the ucd_category identifier for the emscripten build.
2017-04-24 18:49:09 +01:00
Reece H. Dunn
ee61cc4358
Fix running 'make clean' when gradle is not present. Gradle is used for the Android build and is not needed when just building eSpeak NG on Linux/BSD systems.
2017-04-24 18:37:06 +01:00
Reece H. Dunn
a72199f714
Run the tests as part of the Travis build.
2017-04-22 19:14:32 +01:00
Reece H. Dunn
61d668c0cb
ucd-tools: Inverted_Terminal_Punctuation eSpeakNG extended property support; use in clause_type_from_codepoint.
2017-04-22 19:09:49 +01:00
Reece H. Dunn
5c6bc0e556
Armenian emphasis mark (U+055B) is used for interjections, so treat it as an exclamation mark.
2017-04-22 18:54:21 +01:00
Reece H. Dunn
bc13173ac4
ucd-tools: Punctuation_In_Word eSpeakNG extended property support; use in clause_type_from_codepoint.
2017-04-22 18:52:46 +01:00
Reece H. Dunn
1131d0924b
ucd-tools: Optional_Space_After eSpeakNG extended property support; use in clause_type_from_codepoint.
2017-04-22 18:17:39 +01:00
Reece H. Dunn
b932f3c493
ucd-tools: Extended_Dash eSpeakNG extended property support; use in clause_type_from_codepoint.
2017-04-22 15:55:44 +01:00
Reece H. Dunn
3100ca9d1b
Use ucd_properties to implement clause_type_from_codepoint for supported types.
2017-04-22 15:20:19 +01:00
Reece H. Dunn
be86091088
Use #defines for the ESPEAKNG_PROPERTY_ constants, so they can be used in things like switch expressions.
2017-04-22 14:33:51 +01:00
Reece H. Dunn
af88820954
Merge commit 'd18d98b92ca42bf7b098d3d1fd873d9b0ee82e00'
2017-04-22 14:31:07 +01:00
Reece H. Dunn
d18d98b92c
Use #defines for the UCD_PROPERTY_ constants, so they can be used in things like switch expressions.
2017-04-22 14:28:41 +01:00
Reece H. Dunn
1c4ce3dcd3
tokenizer.c: create and use a clause_type_from_codepoint function, with tests.
2017-04-19 22:33:32 +01:00
Reece H. Dunn
92f703d98b
Use defines instead of hard-coded numbers for more clause logic.
2017-04-18 07:14:45 +01:00
Reece H. Dunn
8749891069
Better specify the CLAUSE_ flags returned by ReadClause.
2017-04-17 22:04:45 +01:00
Reece H. Dunn
2e375362c4
ucd-tools: Paragraph_Separator eSpeakNG extended property support.
2017-04-17 20:20:42 +01:00
Reece H. Dunn
7fc4f5ece2
ucd-tools: Ellipsis eSpeakNG extended property support.
2017-04-17 17:35:26 +01:00
Reece H. Dunn
8d8c8b3b56
ucd-tools: Semi_Colon eSpeakNG extended property support.
2017-04-17 17:17:52 +01:00
Reece H. Dunn
9869ee051e
ucd-tools: Colon eSpeakNG extended property support.
2017-04-17 16:18:57 +01:00
Reece H. Dunn
9ef03b8ac8
ucd-tools: Comma eSpeakNG extended property support.
2017-04-17 15:14:02 +01:00
Reece H. Dunn
5017153d62
ucd-tools: Exclamation_Mark eSpeakNG extended property support.
2017-04-17 13:47:50 +01:00
Reece H. Dunn
31d66fddb1
ucd-tools: Question_Mark eSpeakNG extended property support.
2017-04-17 12:50:04 +01:00
Reece H. Dunn
07cd2b12e1
ucd-tools: Full_Stop eSpeakNG extended property support.
2017-04-17 11:45:21 +01:00
Reece H. Dunn
9bbb16a74c
Add ucd-tools proplist.c to the build.
2017-04-16 15:04:15 +01:00
Reece H. Dunn
2615242f05
Merge commit 'ff018e33df5763dc0ff1644c9479ee2351bee45e'
2017-04-16 15:02:08 +01:00
Reece H. Dunn
ff018e33df
Support compiling ucd.h with a strict C11 compiler.
2017-04-16 15:01:37 +01:00
Reece H. Dunn
1a4b9d087b
Add the new PropList API to the README file.
2017-04-16 14:44:55 +01:00
Reece H. Dunn
691457e98d
Add Prepended_Concatenation_Mark support from PropList.txt.
2017-04-16 14:36:27 +01:00
Reece H. Dunn
429b8f3629
Add Pattern_Syntax support from PropList.txt.
2017-04-16 14:17:43 +01:00
Reece H. Dunn
d91a249c14
Add Pattern_White_Space support from PropList.txt.
2017-04-16 11:37:14 +01:00
Reece H. Dunn
eecd7984c5
Add Variation_Selector support from PropList.txt.
2017-04-16 11:26:05 +01:00
Reece H. Dunn
f7f228bc0c
Add Sentence_Terminal support from PropList.txt.
2017-04-16 11:05:59 +01:00