RandLM ("randomised language modelling") is yet another language model for the open source translation system Moses. However, it is designed to be very space-efficient indeed: depending upon settings, it can represent an SRILM language model in about 1/10 of the space. The code can be used to estimate LMs either from raw text (similar to SRILM's "ngram-count") or else can be used to load pre-built ARPA files. Best compression results are obtained when building LMs from raw text.
You can get the code here:
(This is the first public release and there are sure to be bugs)
Read the files:
for Moses integration and:
for general information on building the release.
Note that Moses can support SRILM and RandLM LMs at the same time --just use
/configure --with-randlm=/path/to/randlm --with-randlm=/path/to/srilm
If you want to read more about this, then look at our ACL and EMNLP papers:
David Talbot and Miles Osborne. Smoothed Bloom filter language models: Tera-Scale LMs on the Cheap. EMNLP, Prague, Czech Republic 2007. http://www.iccs.informatics.ed.ac.uk/~osborne/papers/emnlp07.pdf
David Talbot and Miles Osborne. Randomised Language Modelling for Statistical Machine Translation. ACL, Prague, Czech Republic 2007. http://www.iccs.informatics.ed.ac.uk/~osborne/papers/acl07.pdf
-- The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336.