Speech:Exps 0039

Description
Author: Jake (Joseph) Gallagher on Majestix Date: 3/26/2013

Purpose: Begin the process of understanding how an experiment is setup and also how to start a train and/or troubleshoot a train.

Details: I chose to run a Mini train test for this first run. The newer instructions helped facilitate a smoother transition from experiment setup to train initiation. Getting the proper folders created did not reveal any problems and I was able to get the structure made and appropriate files copied or created in the directory. Making sure I had everything ordered properly, I then began to start the train and ran into a common problem with words missing from the dictionary. Although there were some troubleshooting instructions to fix this, I found myself slightly confused. However, once my group met in class I was able to clear up the process and get myself ready for the train. Results: The train started as it should (per Drew's confirmation) and I watched the execution of the various modules. After about 5-8 minutes of processing I received this error:

MODULE: 45 Prune Trees Phase 1: Tree Pruning FATAL: "main.c", line 167: Unable to open /mnt/main/Exp/0039/trees/0039.unpruned/D-1.dtree for reading; No such file or directory MODULE: 50 Training Context dependent models Phase 1: Cleaning up directories: accumulator...logs...qmanager... Phase 2: Copy CI to CD initialize Phase 3: Forward-Backward Baum welch starting for 1 Gaussian(s), iteration: 1 (1 of 1) 0% FATAL_ERROR: "main.c", line 1054: initialization failed

Failed to start bw Only 0 parts of 1 of Baum Welch were successfully completed Parts 1 failed to run! Training failed in iteration 1 Something failed: (/mnt/main/Exp/0039/scripts_pl/50.cd_hmm_tied/slave_convg.pl)

Starting with the error in an unavailable file known as D-1.dtree I copied this same file from another experiment in an attempt to boot leg a fix. Once I did I started the train again and received a simlar error but this time regarding a dtree file known as ER0-0.dtree which from first glance appears to be in the proper location.

MODULE: 45 Prune Trees Phase 1: Tree Pruning FATAL: "main.c", line 167: Unable to open /mnt/main/Exp/0039/trees/0039.unpruned/ERO-0.dtree for reading; No such file or directory MODULE: 50 Training Context dependent models Phase 1: Cleaning up directories: accumulator...logs...qmanager... Phase 2: Copy CI to CD initialize Phase 3: Forward-Backward Baum welch starting for 1 Gaussian(s), iteration: 1 (1 of 1) 0% FATAL_ERROR: "main.c", line 1054: initialization failed

Failed to start bw Only 0 parts of 1 of Baum Welch were successfully completed Parts 1 failed to run! Training failed in iteration 1 Something failed: (/mnt/main/Exp/0039/scripts_pl/50.cd_hmm_tied/slave_convg.pl)

Seeing that the error with the dtree file does not halt the process I am now looking to see what the errors are following the dtree file error. To be continued...

After further group discussion and failed attempts at rerunning the mini train I am putting this mini train on hold until we progress further in understanding where the error is occurring.