[Corpora-List] Q: Hyphenation removal

Jordi Porta Zamorano jordi.porta at uam.es
Fri Aug 17 18:23:30 CEST 2012

Dear Ronald,

Perhaps the data-driven solution described in http://www.euppublishing.com/doi/abs/10.3366/cor.2011.0010 could help you.

Best, --Jordi

On Thursday, 16 August 2012, Roland Schäfer wrote:

> Dear list members,
> are there any tools to remove hard-coded "hyphe- nation" from texts (or
> papers describing principled solutions to the problem). The
> tool/solution should ideally:
> * work even if the line break after the hyphen has been removed,
> * differentiate with near-perfect accuracy between actual hyphenation
> and other superficially identical graphematic constructions involving
> hyphens (like German truncated compound coordination as in "Bus- und
> Bahnticket" for "Busticket und Bahnticket"),
> * (consequently:) be trainable (ideally in an unsupervised way) on
> arbitrary languages which use some specifiable set of characters to
> indicate hyphenation,
> * possibly also detect cases of hyphenation which are not written with a
> space (as in "hyphe-nation") or additional spaces (as in "hyphe - nation"),
> * process UTF-8 or at least some Unicode encoding,
> * be open-source/patent-free and available in the form of a library or
> command line tool (e.g., not a GUI tool/part of some OCR product/web
> service).
> Of course, references which only partially match these requirements are
> also highly appreciated.
> Thanks a lot.
> Regards
> Roland
> _______________________________________________
> UNSUBSCRIBE from this page: http://mailman.uib.no/options/corpora
> Corpora mailing list
> Corpora at uib.no <javascript:;>
> http://mailman.uib.no/listinfo/corpora
-------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: text/html Size: 2279 bytes Desc: not available URL: <https://mailman.uib.no/public/corpora/attachments/20120817/365f3462/attachment.txt>

More information about the Corpora mailing list