Speech:Exps 0288 019

Description
Authors: Team Cap

Date: 4/21/16

Purpose: copied /mnt/main/local/bin/sphinx3_align and /mnt/main/scripts/train/scripts_pl/04.vtln_align to 019/bin. Believe missing these is what caused the failure of 017

Details:
 * Train configuration
 * Corpus: Switchboard
 * Length: 30hr
 * $CFG_FINAL_NUM_DENSITIES = 16 (density)
 * $CFG_N_TIED_STATES = 4000(senones)
 * $CFG_FORCEDALIGN = 'yes' -- required for below
 * $CFG_VTLN = ’yes'
 * $CFG_VTLN_START = 0.70
 * $CFG_VTLN_END = 1.40
 * $CFG_VTLN_STEP = 0.05
 * copied /mnt/main/local/bin/sphinx3_align and /mnt/main/scripts/train/scripts_pl/04.vtln_align to 019/bin.
 * Believe missing these is what caused the failure of 017


 * Timeline
 * Train start: 12:45PM on 4/21/16
 * Train end: AM on 4/21/16 (Total time: ) (errors in 017.html: )
 * Decode start: on 4/21/16
 * Decode end: on 4/21/16 (Total time: )


 * Decode configuration
 * Decoding on: /mnt/main/corpus/switchboard/30hr/test/trans/dev.trans (~5 hours)
 * Decoding at: 4000 senones to match the senone count in the train configuration

Results: WER: % Unseen

UPDATE:

EMAIL 1: mike,

Team cap is trying to add vocal tract length normalization to our experiments. 0288/017 and 019 failed (due to what we believed was missing bin files).

We believe it is due to a directory (vtnlnout/.070/) not being made:

019.html file: [jax472@caesar 019]$ tail -10 019.html ../scripts_pl/make_feats.pl Log File completed Phase 6: Running force alignment in 2 parts Force alignment starting: (2 of 2) sphinx3_align Log File Force alignment starting: (1 of 2) sphinx3_align Log File This step had 1 ERROR messages and 0 WARNING messages. Please check the log file for details. completed This step had 1 ERROR messages and 0 WARNING messages. Please check the log file for details. completed Failed in part 1 Failed in part 1

[jax472@caesar 04.vtln_align]$ tail -5 019.1.vtln.log -wdsegdir /mnt/main/Exp/0288/019/vtlnout/0.70,CTL -wlen 0.025625 2.562500e-02

SYSTEM_ERROR: "main_align.c", line 974: Failed to open file /mnt/main/Exp/0288/019/vtlnout/0.70/019.alignedtranscripts.1 for writing; No such file or directory Thu Apr 21 13:23:37 2016

found the vtln script here: jax472@miraculix 04.vtln_align]$ pwd /mnt/main/scripts/train/scripts_pl/04.vtln_align [jax472@miraculix 04.vtln_align]$ ls slave_align.pl vtln_align.pl

Phase 6 starts in vtln_align.pl at line 230. Further scanning through vtln_align shows that the directory appears to never be made.

A google search led me to the cmusphinx github: https://github.com/cmusphinx/sphinxtrain

Under the scripts there, they have "12.vtln_align" last updated 2 years ago. Under that, in the vtln_align.pl file on line 61 & 62: my $outdir = "$ST::CFG_BASE_DIR/vtlnout/$warp"; mkdir($outdir,0777);

where $warp would be the .70 value we need. Appears that this makes the directory we need.

our 04.vtln_align directory was last modified in Apr 2012. Looks like we may have and outdated version where VTLN is not working.

googling the System_error message i found: https://sourceforge.net/p/cmusphinx/discussion/help/thread/7c8f824c/#bf9f

This appears to show that prior to a 2014 fix, VTLN was broke. However the author of that post also claims the VTLN is not supported in decoding, it may not work. The following post from the original post author says that he got it working though.

Thoughts on this? It appears it is possible to get working and does improve performance, but would require some work with updating sphinxtrain.

EMAIL 2: Here is what you can do:

Take Miraculix and remove the /usr/local link to /mnt/main/local you'll have to recreate the /usr/local directory (it might actually just have been renamed to /usr/local-OFF unmount /mnt/main umount -a should do it You need to do this because some sphinx scripts will go into /mnt/main/root and we don't want that stuff touched either.

Plug in the network wire from brutus set up DHCP Download and install anything from sphinx that you want note, you can also copy some (or all) of the executables from /mnt/main/local to /usr/local before unmounting /mnt/main. you can also copy all of /mnt/main/root to (say) /usr/local/root perhaps copy /man/main/install to /root/install Once you have configured your new version of Sphinx that allows you to do LDA as well as VTL, and it is set up to use /usr/local as well as /usr/local/root, you can then mount /mnt/main again to get back the experiment directories and on Asterix you will be able to run your own modified executables.

Please document this (privately if you have to) so that in the future we could for instance just grab your new executables and replace ours. If you move all the root stuff into /usr/local/root then when we move all of the /usr/local stuff into /mnt/main/local we will effectively have moved /mnt/main/root to /mnt/main/local/root so that works really nicely for future upgrades.

This way you won't be hampered by the rule of not changing Caesar but you also don't unfairly hamper the other team.

BTW, please inform Brenden Collins to use Asterix instead of Miraculix to run his three parallel decode jobs. I'd rather keep Asterix clean since that is usually our golden drone machine so I'd like to have that be the template for the others (i.e. the comic book is called Asterix afterall).

Let me know what you think and please be careful with /mnt/main and be sure it is unmounted on Mircaulix before doing any installs.

Mike