Python port of Mikolov's word2phrase.c from the word2vec toolkit
Given a document or documents this program attempts to learn phrases. It does so by progressively joining adjacent pairs of words with an '_' character. You can then run the code multiple times to create multiword phrases.
For more detail on this check out: https://code.google.com/p/word2vec/ and Mikolov's paper: http://arxiv.org/abs/1310.4546
The OCaml directory contains a simple ocaml implementation of the code.
Example using the text8 corpus used in Mikolov's experiments:
$ time python word2phrase.py --train=text8 --output=text8-phrase-py --min-count=5 --threshold=500.0
# Now count instances of phrases. We separate phrases with an underscore
$ cat text8-phrase-py| tr ' ' '\n' | grep '_' | python wordcount.py 20
9913 united_states
7100 th_century
6761 external_links
4942 new_york
3147 rather_than
2371 united_kingdom
2184 prime_minister
1602 soviet_union
1507 civil_war
1464 main_article
1266 no_longer
1247 science_fiction
1100 don_t
1095 new_zealand
1069 hong_kong
1067 http_www
1019 north_america
998 los_angeles
959 roman_catholic
940 air_force