Commit Graph
562 Commits
Author SHA1 Message Date
Reece H. Dunn 22270bd259 Fix reading hash table entries > 128.
This is related to ebfa320956, but
when reading the hash chain entry length, not writing it.

If char is signed, then before this change the length would be
negative, causing problems loading the dictionary.
2017-07-26 21:11:44 +01:00
Reece H. Dunn ebfa320956 Fix storing the line length in the hash chain.
The length is stored as the first byte in the output from
compile_line. As the data pointer is char, if char is signed then
length could be negative resulting in undefined behaviour. This
commit fixes the issue by reading and writing that byte as a
uint8_t.

This bug was caused by 2a00ca79f6.
Previously, the entries could only be a maximum of 128 bytes, and
would not be negative on platforms with signed chars. That
commit was made to support long emoji entries, especially for
non-Latin languages where the utf-8 representations could be
longer than 128 bytes.

This change also adds some documentation to make it clearer what
is going on. NOTE: The code should really be using actual struct
objects instead of writing to opaque char buffers.

Reported by Reef Turner <reef@nvaccess.org>
2017-07-26 20:44:27 +01:00
Reece H. Dunn e7ac4b819d hash_counts is never used, so remove it.
Reported by Reef Turner <reef@nvaccess.org>
2017-07-26 20:21:05 +01:00
Reece H. Dunn 0383e3525a Merge commit '0994206f710a4defc1eecfb78ab70ff57c58fcda' 2017-07-26 19:53:11 +01:00
Reece H. Dunn 921229259d Use int to store the value from GetFileLength.
This fixes the clang warning:

    comparison of unsigned expression < 0 is always false
    [-Wtautological-compare]

Reported by Reef Turner <reef@nvaccess.org>
2017-07-26 19:50:03 +01:00
Reef Turner 0994206f71 address buffer overrun when dict listx file has no trailling newline
See ar_listx as an example.
2017-07-25 14:29:45 +08:00
yorkie 949de3f80f build: fix duplicated symbol when linking 2017-06-15 18:10:54 +08:00
Reece H. Dunn 2a00ca79f6 Increase dict_line to support long emoji entries. 2017-06-04 19:47:30 +01:00
Reece H. Dunn d5d980862e Guard against overflowing the dict_line bounds when compiling dictionary files. 2017-06-04 19:38:40 +01:00
Reece H. Dunn 416bc66dca Fix pronouncing replacement text that is dependent on start of word rules. 2017-05-29 11:53:33 +01:00
Reece H. Dunn 7accbcd498 Only update the sample rate if the mbrola voice was successfully loaded. 2017-05-28 10:41:16 +01:00
Reece H. Dunn e41a78d819 Report the actual error from LoadMbrolaTable when loading the mbrola voice. 2017-05-28 10:38:15 +01:00
Reece H. Dunn 8dde51ca87 Remove the now unused emoji_zwj specifier. 2017-05-25 07:23:21 +01:00
Reece H. Dunn fbd2e679fc Don't crash if loading the default voice fails. 2017-05-21 12:44:10 +01:00
Reece H. Dunn dced9f58fd Clean up the decoder memory in espeak_Terminate. 2017-05-21 12:07:15 +01:00
Reece H. Dunn 63e2826753 Ensure that the old translator is not used after calling espeak_Terminate. 2017-05-21 12:01:46 +01:00
Reece H. Dunn ae6116bf97 Fix calling espeak_Synth without first setting a voice, with tests. 2017-05-21 10:47:47 +01:00
Reece H. Dunn 6ab1314843 Move outbuf_size to where it is actually used. 2017-05-21 10:02:27 +01:00
Reece H. Dunn 9fc2efb1fa Don't re-initialize my_audio if espeak_ng_InitializeOutput is called multiple times. 2017-05-21 09:49:35 +01:00
Reece H. Dunn ee07eaa733 Support capitalized words in text-based replacements. This is important in languages like German that differentiate pronunciation on case. Even English does this (e.g. Polish/polish). 2017-05-19 07:13:22 +01:00
Reece H. Dunn 7e85b36b4d en: (emoji) -- Unicode Emoji 4.0 (Emoji Roles) 2017-05-18 07:16:05 +01:00
Reece H. Dunn a4b79f104b Merge remote-tracking branch 'valdisvi/master' 2017-05-16 08:03:04 +01:00
Valdis Vitolins aea34ae5d9 Issue #100 Manipuri: Fix for wrongly pronounced numbers 2017-05-15 23:30:24 +03:00
chrislm 3b0771df5a Decimal numbers in Spanish 2017-05-15 11:46:51 +02:00
Reece H. Dunn dcc43bc30a Support _emoji dictionary lists for symbol and emoji pronunciations. 2017-05-14 11:05:43 +01:00
Reece H. Dunn fe14c83372 Support multi-word text replacements in dictionary list files. 2017-05-14 11:03:33 +01:00
Reece H. Dunn 05cab4f2dc Extend compile_line to include pronunciation data after a space (e.g. multi-word pronunciation replacements). 2017-05-14 08:39:29 +01:00
Reece H. Dunn 48ca2239bb Fix non-Latin character languages falling back to English when reading Latin characters. 2017-05-12 07:35:27 +01:00
Reece H. Dunn 94376c2d5f Use an enum and named values for the steps in compile_line to make the logic easier to read. 2017-05-11 19:40:43 +01:00
Reece H. Dunn efd1df3206 Build a test version of libespeak-ng that exposes the internal APIs. 2017-05-09 07:37:56 +01:00
Reece H. Dunn 85801fc1e3 Remove the now unused dictdialect functionality from the code. 2017-05-07 15:17:47 +01:00
Reece H. Dunn 2f8f125c68 Remove voice/language support for alphabet2.
This is not used by any of the espeak-ng voices and languages.
Additionally, this functionality would be superceded by support
for specifying the language used by different scripts in the
language argument on the command line.
2017-05-07 15:10:42 +01:00
Reece H. Dunn dd90d3812d tokenizer.c: Support general symbol tokens. 2017-05-03 08:08:40 +01:00
Reece H. Dunn 786575c6ed tokenizer.c: Support general punctuation tokens. 2017-05-03 07:56:53 +01:00
Reece H. Dunn 0705844bf8 tokenizer.c: Move general category classification that does not override property behaviour to the end, for generic classification. 2017-05-02 18:58:24 +01:00
Reece H. Dunn 683579f403 Make the tokenizer.h API public. 2017-05-02 18:52:06 +01:00
Reece H. Dunn 9af96da469 Make the encoding.h API public. 2017-05-02 18:40:16 +01:00
Reece H. Dunn 55bfbb4754 tokenizer.c: Support ellipsis tokens. 2017-05-02 07:28:29 +01:00
Reece H. Dunn b847df63b5 tokenizer.c: Support semicolon tokens. 2017-05-01 19:38:37 +01:00
Reece H. Dunn af7e8fc5a3 tokenizer.c: Support colon tokens. 2017-05-01 19:22:48 +01:00
Reece H. Dunn 7560070dcd tokenizer.c: Support comma tokens. 2017-05-01 19:14:47 +01:00
Reece H. Dunn c9199cfacb tokenizer.c: Support exclamation mark tokens. 2017-05-01 19:00:40 +01:00
Reece H. Dunn 128ceaff6a tokenizer.c: Support question mark tokens. 2017-05-01 17:50:54 +01:00
Reece H. Dunn 8f62e18324 tokenizer.c: Support full stop tokens. 2017-05-01 17:45:55 +01:00
Reece H. Dunn 0bbc9e9730 Merge remote-tracking branch 'Christianlm/master' 2017-05-01 17:32:31 +01:00
chrislm 5d8bb74169 IT: new improvements tested on april 2017
reduced length to 160 for unstressed syllables

Added some exceptions to the italian dictionaries
2017-05-01 11:06:19 +02:00
Reece H. Dunn d50f3f2fa5 tokenizer.c: Support word tokens. 2017-05-01 09:37:23 +01:00
Reece H. Dunn d093513b65 tokenizer.c: Add an options parameter to the tokenizer_reset API. 2017-04-28 07:48:55 +01:00
Reece H. Dunn c41ac642fa tokenizer.c: Tokenise Zp codepoints as paragraphs. 2017-04-28 07:44:22 +01:00
Reece H. Dunn fc7a4e6701 tokenizer.c: Recognise U+000C [FORM FEED (FF)] as a newline codepoint. 2017-04-28 07:31:05 +01:00