 Reece H. Dunn
|
5030ff95cc
|
automake: don't make the tests print 'done', have the make rule print 'PASSED' instead.
|
2017-05-09 08:01:21 +01:00 |
|
|
Reece H. Dunn
|
8821c9e361
|
Split out the readclause tests so the tokenizer tests can use public-only APIs.
|
2017-05-09 07:44:42 +01:00 |
|
|
Reece H. Dunn
|
dd90d3812d
|
tokenizer.c: Support general symbol tokens.
|
2017-05-03 08:08:40 +01:00 |
|
|
Reece H. Dunn
|
786575c6ed
|
tokenizer.c: Support general punctuation tokens.
|
2017-05-03 07:56:53 +01:00 |
|
|
Reece H. Dunn
|
683579f403
|
Make the tokenizer.h API public.
|
2017-05-02 18:52:06 +01:00 |
|
|
Reece H. Dunn
|
9af96da469
|
Make the encoding.h API public.
|
2017-05-02 18:40:16 +01:00 |
|
|
Reece H. Dunn
|
55bfbb4754
|
tokenizer.c: Support ellipsis tokens.
|
2017-05-02 07:28:29 +01:00 |
|
|
Reece H. Dunn
|
b847df63b5
|
tokenizer.c: Support semicolon tokens.
|
2017-05-01 19:38:37 +01:00 |
|
|
Reece H. Dunn
|
af7e8fc5a3
|
tokenizer.c: Support colon tokens.
|
2017-05-01 19:22:48 +01:00 |
|
|
Reece H. Dunn
|
7560070dcd
|
tokenizer.c: Support comma tokens.
|
2017-05-01 19:14:47 +01:00 |
|
|
Reece H. Dunn
|
c9199cfacb
|
tokenizer.c: Support exclamation mark tokens.
|
2017-05-01 19:00:40 +01:00 |
|
|
Reece H. Dunn
|
128ceaff6a
|
tokenizer.c: Support question mark tokens.
|
2017-05-01 17:50:54 +01:00 |
|
|
Reece H. Dunn
|
8f62e18324
|
tokenizer.c: Support full stop tokens.
|
2017-05-01 17:45:55 +01:00 |
|
|
Reece H. Dunn
|
d50f3f2fa5
|
tokenizer.c: Support word tokens.
|
2017-05-01 09:37:23 +01:00 |
|
|
Reece H. Dunn
|
a902f451d8
|
tests/tokenizer.test: Support printing the tokens from a provided file, making it easy to investigate tokenizer issues.
|
2017-04-29 10:13:16 +01:00 |
|
|
Reece H. Dunn
|
d093513b65
|
tokenizer.c: Add an options parameter to the tokenizer_reset API.
|
2017-04-28 07:48:55 +01:00 |
|
|
Reece H. Dunn
|
c41ac642fa
|
tokenizer.c: Tokenise Zp codepoints as paragraphs.
|
2017-04-28 07:44:22 +01:00 |
|
|
Reece H. Dunn
|
f3ea6f68f3
|
tokenizer.c: Tokenise U+000B [VERTICAL TAB (VT)] as whitespace, not as newlines.
|
2017-04-28 07:37:00 +01:00 |
|
|
Reece H. Dunn
|
fc7a4e6701
|
tokenizer.c: Recognise U+000C [FORM FEED (FF)] as a newline codepoint.
|
2017-04-28 07:31:05 +01:00 |
|
|
Reece H. Dunn
|
d2d718d700
|
tokenizer.c: Tokenize line separator codepoints as newline tokens.
|
2017-04-28 07:16:38 +01:00 |
|
|
Reece H. Dunn
|
bf45e7ce36
|
tokenizer.c: Recognise U+0085 [NEW LINE (NEL)] as a newline codepoint.
|
2017-04-28 07:08:02 +01:00 |
|
|
Reece H. Dunn
|
df6ca7a22c
|
tokenizer.c: Support whitespace tokens.
|
2017-04-27 07:59:29 +01:00 |
|
|
Reece H. Dunn
|
8f0dae6a38
|
tokenizer.c: Support windows newlines.
|
2017-04-26 07:52:33 +01:00 |
|
|
Reece H. Dunn
|
1c8ed9c190
|
tokenizer.c: Support mac newlines.
|
2017-04-26 07:39:06 +01:00 |
|
|
Reece H. Dunn
|
7602c9ac18
|
tokenizer.c: Support linux newlines.
|
2017-04-26 07:34:57 +01:00 |
|
|
Reece H. Dunn
|
bce44316bb
|
Create a basic tokenizer API using a structure that mirrors the TtsTokenizer interface in the tts-dev-studio project.
|
2017-04-25 07:55:10 +01:00 |
|
|
Reece H. Dunn
|
5c6bc0e556
|
Armenian emphasis mark (U+055B) is used for interjections, so treat it as an exclamation mark.
|
2017-04-22 18:54:21 +01:00 |
|
|
Reece H. Dunn
|
1c4ce3dcd3
|
tokenizer.c: create and use a clause_type_from_codepoint function, with tests.
|
2017-04-19 22:33:32 +01:00 |
|