Compiling dictionaries

keyvi dictionaries can be compiled from the command line, either with the keyvi command that ships with the python package or with the C++ keyvicompiler tool.

The keyvi command line tool

After pip install keyvi the keyvi command is available:

keyvi compile input.tsv output.kv key-only

The last argument is the dictionary type, one of json, string, int, completion or key-only. Compiler parameters can be passed with --param, e.g. --param memory_limit_mb=2048.

To dump a dictionary:

keyvi dump output.kv dump.txt

To show statistics:

keyvi stats output.kv

The keyvicompiler tool

When building keyvi from source (see the C++ readme), the keyvicompiler binary is built as well:

keyvicompiler -h
keyvi compiler options::
  -h [ --help ]                         Display this help message
  -v [ --version ]                      Display the version number
  -i [ --input-file ] arg               input file
  -o [ --output-file ] arg              output file
  -m [ --memory-limit ] arg             amount of main memory to use
  -d [ --dictionary-type ] arg (=integer)
                                        type of dictionary (integer (default),
                                        string, key-only, json, completion)
  -p [ --parameter ] arg (=EMPTY)       An option; format is -p xxx=yyy
  --manifest arg                        manifest to be embedded

To compile simply run for example:

keyvicompiler -i test.txt -o test.kv

Input formats

type	details
integer	Expects integer values (tab separated from the key)
key-only	Expects no values, just keys (e.g. for filters)
string	Expects string values (tab separated from key)
json	Expects valid json values (dumped as string)
completion	Expects integer weights (tab separated from the key)

Memory limit

The level of compression can depend on the amount of memory. If you create large dictionaries it might make a difference to allow the compiler the usage of more memory. This will not only speedup the sort phase but might also lead to smaller dictionaries in the end. Per default the memory limit is set to 1 GB.

Note: The memory limit just sets the amount of memory the keyvi compiler can use for its minimization hashtables, in addition keyvi dictionary compiler needs more memory to persist the data.

Dumping keyvi dictionaries

You can use keyviinspector to dump the data:

keyviinspector -i test.kv -o ../test-dump.txt

or to show statistics:

keyviinspector -i test.kv -s