Speech:Summer 2012 Cedric Woodbury Log


 * Home
 * Semesters
 * Summer 2012

Week Ending June 18, 2012

 * Task:
 * Read Speech Project Proposal and Report
 * Scan through student logs
 * Read Information Page
 * Get openSUSE 11.3 installed on my laptop
 * Setup the network bridge
 * Research a backup solution
 * Implement a backup solution that runs as a service


 * Results:
 * Completed assigned reading tasks and got up to speed on the project
 * Installed openSUSE 11.4 on my laptop
 * Network bridge was implemented on Caesar and the 9 client machines were configured to support it.
 * Researched several cloud backup solutions. Dropbox looks like an attractive option
 * Got Dropbox running as a service - see concerns however.


 * Plan:
 * Finish implementing Dropbox so it is fully automated
 * Get automount working
 * Meet with Prof. Jonas to discuss additional tasks.
 * Concerns:
 * I had to install openSUSE 11.4 as 11.3 did not play nicely with my laptop's hardware
 * Dropbox is working as a service. However, I have to be logged in to start it manually.  I need to get it so it runs when the computer starts.
 * There was a hiccup when saving the network configuration on majestix. I was not able to access the machine afterwards.  Hopefully it just needs a reboot.  Prof. Jonas will check the server closet next week.

Week ending June 25, 2012

 * Task:
 * Set up Dropbox on Caesar so that it will run automatically in the background
 * Add Cron tasks to the 9 client machines to re-mount the network shares every 6 hours.

0 0,6,12,18 * * * mount -a
 * Results:
 * Dropbox implemented on Caesar. The service has to be started manually, but will continue to run until the computer is rebooted.
 * Added the following crontab entry for root on each machine:

Next will do this. None this week.
 * Plan:
 * Concerns:

Week ending July 24, 2012

 * Task:
 * Attempt to Run a train on obelix. I will run a train using the train1 folder under SphinxTools-1.0 folder.  This has already been run, but it will be a proof of concept test for me to make sure I understand the process.


 * Results:
 * Ran into a problem where sox wasn't recognized on Obelix. Sox is installed on caesar.  Prof Jonas fixed this so all computers can see it.
 * I followed the directions for running a train here.  I created experiment 0007.
 * I had to make some corrections to the directions, but I was able to get to the point where I could execute the RunAll.pl script.
 * While the train appeared to be successful, after meeting with prof Jonas, he noticed that it did not get all the way through the process. Not sure why, but it may be due to the fact that the data set I am using is not large enough.


 * Plan:
 * I will run a train using the train2 folder under /mnt/main/old/train This one got all the way through.  I will set it up under experiment 0008.  If I am able to run it successfully then I will try another train on my own with a larger data set.
 * Need to modify the GenTrans.pl script to make it more efficient. It is copying over the sph files it needs to a temp directory, but we don't need that step.  All the sph files are in /mnt/main/corpus/dist/Switchboard/flat as aliases.
 * Concerns:
 * Concerned that my train did not finish completely. I hope that it is only due to a small data set.  If not, then this may get a bit more complex.
 * Also noticed that setup_SphinxTrain.pl has to be run from the SphinxTools-1.0 directory. It will not work if executed from the scripts folder.  Need to look into that.  I will try and fix it if I have time after completing my other tasks.

Week ending July 31, 2012

 * Task:
 * Attempt to run a train using the successful train2 configuration under the old/train folder.
 * Build a Language Model.
 * Attempt to run a decode.
 * Update instructions for running a train, building a language model and running a decode.
 * Try to simplify the process of prepping a case to run a train by writing a script that completes most of the tasks automatically.
 * Attempt to modify the genTrans.pl script to make it more efficient.

$CFG_HMM_TYPE = '.cont.'; #Sphinx III $CFG_HMM_TYPE = '.semi.'; # Sphinx II The config file used by the successful train2 experiment folder had the second line commented out. However, the config file that setup_Sphinx.pl creates had the first line commented out, which is why my trains were not fully completing the files that I needed.
 * Results:
 * Had to add a symbolic link for csh pointing to tcsh on each of the computers so some of the C scripts would not fail.
 * Was not able to complete a train using the train2 data, even when copying everything over exactly.
 * I was able to determine why the training session was not creating the .cont output files and was creating .semi files instead. It turns out that there are two lines in the sphinx_train.cfg file:
 * After noting the above issue I was able to successfully run a train on experiments 0009 and 0010.
 * Built a Language Model for experiments 0009 and 0010.
 * Ran a decode on 0009 and 0010 (which appeared to be successful).
 * Updated the step-by-step instructions with corrections and as well as creating instructions for building the language model and decoding process.


 * Plan:
 * Write a script that generates an equivalent _train.trans file from decode.log with the and tags inserted.
 * Install and get sclite to work. This is the scoring tool that will take as input the truth (.trans) and our hypothesized results (the decode log).
 * If I have time, try to write that script that simplifies the process of setting up an experiment.
 * If I have time, try to make the genTrans.pl script more efficient.
 * Concerns:
 * While the training sessions were successful there were many many errors that were recorded to the logs. More investigation needs to be done to determine if these errors are cause for concern or can be ignored.

Week ending August 7th, 2012

 * Task:
 * Modify the genTrans.pl to do the following:
 * Eliminate the need for a tempWav directory.
 * Make it so that it pulls the transcript from the corpus/switchboard directory
 * Remove a redundant sox command that is not needed
 * Have it only use one temp file for conversion rather than making one temp file for each sph file being coverted
 * Create a script that will pull the predicted phrases out of the decode.log file, and save them in a new file that has the same format as the input transcript used for training.
 * Compile and install SCTK.

(mkdir -p bin) (cd src; make all) make[1]: Entering directory `/mnt/main/root/stck/sctk-2.4.0/src' (cd asclite; make all) make[2]: Entering directory `/mnt/main/root/stck/sctk-2.4.0/src/asclite' (cd core; make all) make[3]: Entering directory `/mnt/main/root/stck/sctk-2.4.0/src/asclite/core' g++ -c -DPACKAGE_NAME=\"\" -DPACKAGE_TARNAME=\"\" -DPACKAGE_VERSION=\"\" -DPACKAGE_STRING=\"\" -DPACKAGE_BUGREPORT=\"\" -DHAVE_LIBM=1 -DSTDC_HEADERS=1 -DHAVE_SYS_TYPES_H=1 -DHAVE_SYS_STAT_H=1 -DHAVE_STDLIB_H=1 -DHAVE_STRING_H=1 -DHAVE_MEMORY_H=1 -DHAVE_STRINGS_H=1 -DHAVE_INTTYPES_H=1 -DHAVE_STDINT_H=1 -DHAVE_UNISTD_H=1 -DHAVE_UNISTD_H=1 -DHAVE_STRING_H=1 -DHAVE_STDIO_H=1 -DHAVE_STDLIB_H=1 -DHAVE_MATH_H=1 -DHAVE_TIME_H=1 -DHAVE_ASSERT_H=1 -DHAVE_FLOAT_H=1 -DHAVE_LIMITS_H=1  -I. -g -Os main.cpp In file included from main.cpp:20:0: recording.h:122:29: error: âFilter::Filterâ cannot appear in a constant-expression recording.h:122:36: error: template argument 2 is invalid recording.h:122:36: error: template argument 4 is invalid make[3]: *** [main.o] Error 1 make[3]: Leaving directory `/mnt/main/root/stck/sctk-2.4.0/src/asclite/core' make[2]: *** [all] Error 2 make[2]: Leaving directory `/mnt/main/root/stck/sctk-2.4.0/src/asclite' make[1]: *** [all] Error 2 make[1]: Leaving directory `/mnt/main/root/stck/sctk-2.4.0/src' make: *** [all] Error 2
 * Results:
 * Successfully made the modifications to the genTrans.pl script
 * Successfully created a new script to parse the decode.log file.
 * Was not able to compile SCTK on caesar - get the following error:


 * I downloaded, compiled and installed gcc v4.4.7 and was able to successfully compile SCTK on my laptop.


 * Plan:
 * I will discuss options with Prof Jonas on Tuesday as he has concerns about using another gcc compiler as it may break other programs


 * Concerns:
 * Was not able to compile SCTK using the version of gcc installed to caesar.

Week ending August 15th, 2012

 * Notes from last weeks meeting with Prof Jonas:
 * Prof Jonas was able to get SCTK to compile. There was a problem with the asclite component which he commented out.  Installation instructions can be found  here.


 * Task:
 * Attempt to run a Train and Decode, and scoring on an hours worth of dialog - Will be Exp 0012
 * This will require writing a script that counts how much time has elapsed based on the duration of each spoken phrase.
 * Transcript/Sph files will be under /mnt/main/corpus/switchboard/mini/train
 * After training has completed run a decode and scoring on the next 5 minutes of dialog, using the Acoustic Model built in 0012 - this will be Exp 0013.
 * Transcript/Sph files will be under /mnt/main/corpus/switchboard/mini/dev
 * Set up another decode and scoring using the next 5 minutes after 0013, again using the Acoustic Model from 0012.
 * Transcript/Sph files will be under /mnt/main/corpus/switchboard/mini/eval


 * Results:
 * I was able to complete Experiment 0012. Details can be found here.
 * I was able to set up the experiment in the normal way by following the steps used to create previous experiments. However, this time I needed to create and use the transcript and wav files located under /mnt/main/corpus/switchboard/mini/train.  I created a new script called createTranscript.pl which allowed me to create a transcript based on a specified length of spoken dialog.  Once that was created and placed in the trans dir, I made links to the sph files in the wav dir.
 * After that was completed I created two scripts, pruneDictionary.pl and dictionary.pl that created a custom dictionary containing only the words used in that transcript.  From there I attempted to run the train and decode and scoring using sclite.
 * I ran into some problems at first with training. Mainly there were some slang/misspelled words that were not in the dictionary.  I did my best to construct the phones for those words and added them in the dictionary.  Running it again got further, but it failed again because it didn't like one of the vowel phones.  Apparently the vowel phones need to have an emphasis (can't have just AY, needs to be AY0, AY1, etc) unlike some of the other phones.  I corrected the entry I made that had this phone in it, and re-ran the train successfully.
 * After building the Language Model I ran the decode without any problems.
 * I attempted to use sclite to score the results, but ran into a few problems. The transcripts have some duplicate entry lines and it doesn't like that.  I removed the duplicate entry lines and was able to run sclite and save the results to scoring.log under the etc directory.
 * I was able to successfully run a train, decode and scoring on a new section of transcript. The first experiments used the same transcripts.  Now with this experiment I was able to work out the processes needed to run a train on any section of transcript I want.
 * I was not able to complete Experiments 0013 and 0014.


 * Plan:
 * Need to create and run Experiments 0013 and 0014.


 * Concerns:
 * I was not able to get all of my tasks completed due to a very busy work week.

Week ending August 22nd, 2012

 * Task:
 * Create Exp 0013
 * Create Exp 0014
 * Document scripts and changes


 * Results:
 * Exp 0013 was created successfully. Decoding and scoring were successful.  I created a dev folder in the corpus/switchboard/mini folder.  This experiment used 5 minutes of the transcript starting after the hour of transcript used for Exp 0012.  The decode used the acoustic model created by experiment 0012.
 * Exp 0014 was created successfully. Decoding and scoring were successful.  I created an eval folder in the corpus/switchboard/mini folder.  This experiment used 5 minutes of the transcript starting after what experiment 0013 used.
 * Created perl script copySph.pl which analyzes a transcript and pulls in the needed sph files.
 * Modified run_decode.pl to allow a user to specify the location of an acoustic model to use.


 * Plan:
 * Create an experiment that uses 9 hours of dialog.
 * Create a transcript that uses an hour of dialog for the mini/dev corpus folder.
 * Create a transcript that uses an hour of dialog for the mini/eval corpus folder.
 * Document a start to finish process for running experiments that uses the scripts and info generated from this semester.

None at this time.
 * Concerns:

Week ending August 29nd, 2012

 * Task:
 * Create Exp 0015
 * Create Exp 0016
 * Create Exp 0017


 * Results:
 * Exp 0015 was created successfully. Decoding and scoring were successful.  I created a dev folder in the corpus/switchboard/mini folder.  This experiment used an hour of the transcript starting after the hour of transcript used for Exp 0012.  The decode used the acoustic model created by experiment 0012.
 * Exp 0016 was created successfully. Decoding and scoring were successful.  I created an eval folder in the corpus/switchboard/mini folder.  This experiment used an hour of the transcript starting after what experiment 0013 used.
 * I began work on Exp 0017 which uses 10 hours of dialog. After attempting the first training run, there were about 181 words that were not in the dictionary and I began work on adding the phones to the dictionary for these missing words


 * Plan:
 * Continue work on adding the missing words to the dictionary and attempt to run the train again

None at this time.
 * Concerns: