Speech:Spring 2012 Ted Pulkowski Log


 * Home
 * Semesters
 * Spring 2012
 * Proposal
 * Report

Week Ending February 6th, 2012

 * Task:

Tbd


 * Results:

Tbd


 * Plan:

Tbd


 * Concerns:

Tbd

Week Ending February 13th, 2012

 * Task:


 * Install Ubuntu for a dual-boot on my laptop to get re-used to using linux/unix
 * Get familiar with the foss website
 * Search for Sphinx 3
 * Learn to how use Sphinx
 * Read the other log files


 * Results:


 * Tuesday (2-7-2012)
 * Read the wiki on speech
 * Read the other log files


 * Wednesday (2-8-2012)
 * After searching most of the day, I believe I found Sphinx3. Here is the link I found. I followed the instructions under Sphinx3 v. 0.7 trunk
 * http://sphinx.subwiki.com/sphinx/index.php/Main_Page


 * Friday (2-10-2012)
 * After talking with my group on tuesday, I decided that instead of Ubuntu, I needed openSUSE because that is what is currently running on the servers.
 * Read the log files this morning


 * Monday (2-10-2012)
 * Read some of the official Sphinx wiki to begin to learn how it works.
 * http://cmusphinx.sourceforge.net/wiki/
 * Read the log files this morning


 * Plan:


 * As of Friday (2-10-2012)
 * I intend to learn about how to use Sphinx3.


 * Concerns:


 * I'm a little overwhelmed with the amount of information on the Sphinx wiki. Taking it in will be hard.

Week Ending February 20th, 2012

 * Task:


 * As of Thursday (2-16-2012)
 * Re-install openSUSE on my laptop because it's not booting properly for some reason.
 * Re-install Sphinx 3 along with any other packages necessary to have it work correctly
 * Finish reading the Sphinx wiki
 * Read the log files of the other people on my team


 * Results:


 * Thursday (2-16-2012)
 * Re-installed openSUSE
 * Re-installed Sphinx 3
 * Re-installed any other required pacakges
 * Read my team's log files


 * Saturday (2-18-2012)
 * I finished reading the sphinx wiki.
 * Read my team's log files this morning


 * Monday (2-20-2012)
 * Read my team's log files
 * Read up on various linux/unix commands.


 * Concerns:


 * As of Thursday (2-16-2012)
 * None at the moment

Week Ending February 27th, 2012

 * Task:


 * As of Tuesday (2-21-2012)
 * Read the current draft of the proposal
 * Read the log files of the other people on my team


 * As of Monday (2-27-2012)
 * Re-Read the proposal to see what might have been added/updated to it.
 * Add my portion of Language Modeling to the proposal.
 * Read the log files of the other people on my team


 * Results:


 * Tuesday (2-21-2012)
 * As a side note, I came to the conclusion that the reason that I had to re-install openSUSE was that I had too many different applications and libraries installed. This caused my installation to boot, but the screen was always black and rendered it useless.


 * Friday (2-24-2012)
 * I read the current draft of the proposal.
 * I read my team's log files.


 * Monday (2-27-2012)
 * I re-read the proposal to check for updates.
 * I added to the language modeling section of the proposal to state what the steps are in order to generate a language model.
 * http://foss.unh.edu/wiki/index.php/Speech:Spring_2012_Proposal
 * I read Nick's log files from last spring to help me in writing my piece of the proposal.
 * http://foss.unh.edu/wiki/index.php/Speech:Spring_2011_Nick_Log
 * I read my team's log files.


 * Concerns:


 * As of Monday (2-27-2012)
 * That it will be difficult to locate the missing perl script ParseTranscript.perl, which is supposed to be found in the /media/data/trans directory on Caesar.

Week Ending March 5th, 2012

 * Task:


 * As of Tuesday (2-28-2012)
 * Edit my piece of the proposal
 * Locate the ParseTranscript.perl script
 * Understand and figure out how the following scripts work:
 * CreateLanguageModelFromText.perl
 * ParseTranscript.perl
 * Read my team's log files


 * Results:


 * Tuesday (2-28-2012)
 * Edited my piece of the proposal to make it look more pleasing.
 * With help from James, I located the ParseTranscript.perl in the following directory:
 * /media/data/Switchboard/transcripts/ICSI\ Transcriptions/trans
 * Nick had mentioned that this script was supposed be in the same directory as the CreateLanguageModelFromText.perl script, so I copied it to the /media/data/trans directory for the moment.
 * The ParseTranscript.perl is used to strip all non-word (numbers and symbols) characters from a transcript file.


 * Friday (3-2-2012)
 * Read my team's log files this morning.


 * Monday (3-2-2012)
 * Discovered that to give lm_create.pl the text file it needs I needed to give this command:
 * # perl lm_create.pl trans.text (trans.text being an example transcipt file)
 * With this file, Parse transcript is called to strip the file of all unwanted text and writes the result to tmp.text.
 * The next step to create a language model is to create the tmp.wfreq file. This file is a record for the number of occurrences for each word that is contained within the parsed transcript file.
 * The third file that is created is a vocab file. This file lists alphabetically all the words that are contained within the transcript file.
 * The last three files that are created are the 3 gram file, and the arpa and the binary formats of the language model. Unfortunately these are not simple text files.
 * Read my team's log files this morning.


 * Plan:


 * As of Monday (3-5-2012)
 * Since the 3 gram file, and the arpa and binary formats of the language model are unreadable as a simple text file, my plan by March 7 is to be able to interpret these somehow.


 * Concerns:


 * As of Tuesday (2-28-2012)
 * None at this time.

Week Ending March 12th, 2012

 * Task:


 * As of Tuesday (3-6-2012)
 * Look at the lm_create.pl script to determine which commands are called.
 * See what each of the commands that are called in the script does.


 * Results:


 * Tuesday (2-28-2012)
 * Found an html version of the CMU toolkit documentation:
 * http://www.speech.cs.cmu.edu/SLM/toolkit_documentation.html#text2idngram
 * The command that creates the tmp.idngram file is: "text2idngram -vocab tmp.vocab -n 3 <$inFilename> tmp.idngram". text2idngram maps each word in the text stream to a short integer as soon as it has been read, thus enabling more n-grams to be stored and sorted in memory.
 * The command that creates the tmp.arpa and the tmp.binlm versions of the language model is: "idngram2lm -idngram tmp.idngram -vocab tmp.vocab -arpa tmp.arpa" for the arpa version, and: "idngram2lm -idngram tmp.idngram -vocab tmp.vocab -binary tmp.binlm"
 * The arpa version of the language model is the version of the model that is


 * Plan:

Tbd


 * Concerns:


 * As of Tuesday (2-28-2012)
 * None at the moment

Week Ending March 19th, 2012

 * Task:


 * As of Monday (3-19-2012)
 * To find out why each of the seven files are required to generate a language model
 * Read my team's log files


 * Results:


 * Sunday (3-18-2012)
 * Read my team's log files


 * Monday (3-19-2012)
 * The input text file contains the raw, unfiltered transcript.
 * ParseTranscript.perl removes all unwanted text from the file, creating a readable transcript.
 * The tmp.wfreq file is a list of all the words from the transcript and the number of occurrences of each word.
 * The tmp.vocab file is an alphabetical list of all the words found within the transcript.
 * The tmp.idngram file is a diagram for the specific type of language model.
 * The tmp.arpa file is the readable text version of the 3-gram language model.
 * The tmp.binlm file is the version that is used by sphinx.


 * Plan:


 * As of Monday (3-19-2012)
 * My plan is to run the CreateLanguageModelFromText.perl and the ParseTranscript.perl and figure out how all the files that are created interact with each other.


 * Concerns:


 * As of Monday (3-18-2012)
 * None at this time

Week Ending March 26th, 2012

 * Task:


 * As of Tuesday (3-20-2012)
 * To find out how each of the seven files interact with each other to generate the language model.
 * Read my team's log files


 * Results:


 * Tuesday (3-20-2012)
 * Based on the documentation from cmu toolkit website, the five executable commands that are used to create the language use the output from the previous command as its input. The exception being the text2wfreq command uses the raw transcript file for its input. The idngram2lm command is used twice in order to create the binary version which is read by the computer, and the arpa version which is created so the users can read the language model. The following is a brief description of what each command does and its input and output files:
 * text2wfreq: 
 * Input: Raw transcript text file (Updated on Monday (3-26-2012))
 * Output: A file that lists every word that is found in the transcript and the number of occurrences of each word. (Updated on Monday (3-26-2012))
 * What it does: Uses a hash table to efficiently count the words. The words in the list are out of order due to the randomness of the hash. (Updated on Monday (3-26-2012))
 * wfreq2vocab: 
 * Input: Word unigram file produced from text2wfreq (Updated on Monday (3-26-2012))
 * Output: Vocab file (Updated on Monday (3-26-2012))
 * What it does: Produces an alphabetically ordered list of all the words from the transcript.(Updated on Monday (3-26-2012))
 * text2idngram -vocab tmp.vocab -n 3: 
 * Input: Parsed transcript file and vocab file (Updated on Monday (3-26-2012))
 * Output: File that lists all id n-grams that occurs in the text file, and the total number of occurrences of each. (Updated on Monday (3-26-2012))
 * What it does: Matches each word from the transcript with an integer to enable more n-grams to be stored and sorted in memory. If the "-write_ascii" command is used, an ascii version of the file is created instead of the binary version that sphinx needs to create the language model. (Updated on Monday (3-26-2012))
 * idngram2lm -idngram tmp.idngram -vocab tmp.vocab -arpa: 
 * Input: Id n-gram file (either the binary or the ascii versions) and a vocab file. (Updated on Monday (3-26-2012))
 * Output: Language model file in arpa format. (Updated on Monday (3-26-2012))
 * What it does: Creates a language model using the idngram and vocab files. If using the "-arpa .arpa" command, a readable language model file is created. If using the "-binary .binlm" command, a binary file is created that is used by the "evallm" command. (Updated on Monday (3-26-2012))
 * idngram2lm -idngram tmp.idngram -vocab tmp.vocab -binary: 
 * Input: Id n-gram file (either the binary or the ascii versions) and a vocab file. (Updated on Monday (3-26-2012))
 * Output: Language model file in binary format. (Updated on Monday (3-26-2012))
 * What it does: Creates a language model using the idngram and vocab files. If using the "-binary .binlm" command, a binary file is created that is used by the "evallm" command. (Updated on Monday (3-26-2012))
 * http://www.speech.cs.cmu.edu/SLM/toolkit_documentation.html
 * Read my team's log files


 * Plan:


 * As of Tuesday (3-20-2012)
 * TBD


 * Concerns:


 * As of Tuesday (3-20-2012)
 * None at this time

Week Ending April 2nd, 2012

 * Task:


 * As of Tuesday (3-27-2012)
 * Enter in the information on how the lm_create.pl and ParseTranscript.perl scripts work into the information section on the wiki
 * Read my team's log files.


 * Results:


 * Tuesday (3-27-2012)
 * Entered in the information for the lm_create.pl script into the information section on the wiki.
 * http://foss.unh.edu/wiki/index.php/Speech:Models_LM_Build
 * Read my team's log files.


 * Wednesday (3-28-2012)
 * Entered in the information for the ParseTranscript.perl script into the information section on the wiki.
 * http://foss.unh.edu/wiki/index.php/Speech:Models_LM_Build
 * The ParseTranscript.perl and lm_create.pl scripts are the only two scripts that I am aware of at this time that create a language model.
 * Read my team's log files.


 * Sunday (4-1-2012)
 * Read my team's log files.


 * Plan:


 * As of Tuesday (3-27-2012)
 * To enter in the information for the ParseTranscript.perl script into the information section on the wiki by tomorrow night.


 * Concerns:

Tbd

Week Ending April 9th, 2012

 * Task:


 * As of Tuesday (4-3-2012)
 * Split into three groups of five people each. My group is Aaron J., Bethany R.,
 * Explain what exactly is in the table in the language modelling section on the information page using a couple of paragraphs.
 * Read the team's log files


 * Results:


 * Saturday (4-7-2012)
 * Read my team's log files.


 * Monday (4-9-2012)
 * Sent an email yesterday morning to my team asking what was discussed at the skype meeting, but I still haven't received a response.
 * Read my team's log files to see that they were having trouble running the train on friday, but managed to fix it.


 * Plan:


 * As of Saturday (4-7-2012)
 * Since I missed the skype meeting, I'm going to find out what was discussed.


 * As of Monday (4-9-2012)
 * I plan to ask Aaron J. tomorrow how to run train.


 * Concerns:


 * As of Monday (4-9-2012)
 * I think I'm a little behind, but only because I missed the team meeting on friday. I'm going to talk to my team in class tomorrow to get caught up.

Week Ending April 16th, 2012

 * Task:


 * As of Monday (4-16-2012)
 * Add a clear explanation for each of the scripts used in creating a language model.
 * Read my team's log files.


 * Results:


 * Saturday (4-14-2012)
 * Read my team's log files.
 * Sunday (4-15-2012)
 * Read my team's log files to check for any updates.
 * Monday (4-16-2012)
 * Added a clear explanation on the steps to create language model and details on what each script does to aid in the process.
 * Read my team's log files to check for any updates.


 * Plan:

Tbd


 * Concerns:


 * As of Monday (4-16-2012)
 * Other than I'm getting really busy due to it being the end of the semester, I have no concerns.

Week Ending April 23rd, 2012

 * Task:


 * As of Sunday (4-22-2012)
 * Read my team's log files.


 * Results:


 * Sunday (4-22-2012)
 * On Wednesday Aaron, Brandon, and I had a Skype meeting to discuss what we were going to do for the week.
 * On Friday Aaron, Brandon, and I had a Skype meeting to figure out and discuss how to run a mini train.
 * Before the Skype meeting on Friday, I copied the needed files from Caesar to Miraculix using the steps as noted in the april 24th group 4 log file.
 * Since I had yet to write the steps I used to copy the files into my group's log, I did so.
 * Read my team's log files including the April 24 group log files.


 * Monday (4-23-2012)
 * I tried a few things to run a mini train, but I have been unable to successfully run it.
 * Read my team's log files including the April 24 group log files.


 * Plan:

Tbd


 * Concerns:


 * I am concerned that there is not enough time to understand how to do a train.

Week Ending April 30th, 2012

 * Task:


 * As of Tuesday (4-24-2012)
 * Split into new groups with the following people in my group:
 * Brice R
 * Aaron J
 * Johnny M
 * Jonathan S
 * Brandon M
 * Chad
 * Attend the URC
 * Read my teammate's log files as well as my group log file


 * As of Saturday (4-28-2012)
 * Read my teammate's log files as well as my group log file


 * As of Sunday (4-28-2012)
 * Go through all the scripts that are mentioned in run_all.pl to see what directories those scripts are looking for
 * Fix an issue with the verify_all.pl script
 * Read my teammate's log files as well as my group log file


 * Results:


 * Wednesday (4-25-2012)
 * Helped out at the URC


 * Saturday (4-28-2012)
 * Read my teammate's log files as well as my group log file


 * Sunday (4-29-2012)
 * Read my group's log file to see if I had a response to the note that I left last night asking what I could do to help.
 * Jonathan mentioned that there was an issue with the verify_all.pl script on automatix. On the line: use lib catdir(dirname($0), updir, 'lib'); the $0 needed to be changed to /mnt/main/Exp/0001/script_pl/00.verify. I did this by copying the script from automatix to my laptop using a program called WinSCP then editing the file, and copying it back to automatix.
 * Began to look through the scripts that are called from run_all.pl looking for directory paths.


 * Monday (4-30-2012)
 * Posted the exact command I used to execute the RunAll.pl script, along with the error I received.
 * Tried looking for where the directory for the sphinx_train.cfg file gets declared.
 * Checked for updates on the group log file.


 * Plan:


 * As of Saturday (4-28-2012)
 * I'm not too sure how best to use my time, so I plan to leave a message on my group's wiki page to see where I can help.


 * As of Sunday (4-29-2012)
 * I plan to continue looking through the scripts that are called from run_all.pl looking for directory paths.


 * As of Monday (4-30-2012)
 * I plan to continue looking for where the directory for the sphinx_train.cfg file gets declared. If I find it I will post it to the group log file.


 * Concerns:

Tbd

Week Ending May 7th, 2012

 * Task:


 * As of Friday (5-4-2012)
 * To figure out why the RunAll.pl script is not working properly.
 * Read the group and personal log files for updates.


 * Results:


 * Friday (5-4-2012)
 * On Tuesday I found the solution to the line 48 compilation problem on a forum. The solution, as well as a link to the forum, is noted in the modeling group's log file.
 * I believe I found the section of code that is causing verify_all.pl to fail. There is a LogWarning on line 382, but I'm not sure where it gets logged to. I put the code that fails on the group log in hopes that someone might know.
 * After thinking about the issue, I added the phone SIL to 0001.phone. That seemed to solve the current problem, however, it is now telling me that there is an error on line 52 of /mnt/main/Exp/0001/scripts_pl/01.vector_quantize/slave.VQ.pl.
 * Read the group log file.


 * Saturday (5-5-2012)
 * Read the group log file to check for any updates.


 * Sunday (5-6-2012)
 * Read the group log file to check for any updates.


 * Plan:


 * As of Saturday (5-5-2012)
 * I plan to keep looking into the phone SIL problem.


 * Concerns:

Tbd