Speech:Spring 2017 Alexander Turner Log

From Openitware
Jump to: navigation, search


Week Ending February 7th, 2017

2/5: Checking In

2/6: Checking In

Task

2/7: Read previous logs to research modeling. Work with team to fill out draft proposal.

Results

2/7:

Plan

2/7:

Concerns

2/7:

Week Ending February 14, 2017

2/13: Checking In

2/15: Checking IN

Task

2/12: Research the RNNLM Toolkit, what it does, how it works, and how to use it. The plan for us is to implement a Recurrent Neural Network(RNN) based Language Model(LM) and the current understanding is that the RNNLM toolkit is compatible with Sphinx 3.

2/14: Investigate how to use the RNNLM Toolkit. Try training and testing language models using sample data. This will be done locally instead of on Caesar.

Results

2/12: I watched the video provided my group : https://www.youtube.com/watch?v=pQ3JUVR5XHw Unfortunately, the video was very poorly done and I didn't get much from watching it. I did however look into where I could find the toolkit and figure out how to use it.

Everything associated with the toolkit can be found here: http://www.fit.vutbr.cz/~imikolov/rnnlm/ . The site will say that is has been relocated to another site but it doesn't appear that the other site is active.

The site provides the latest version of the toolkit, data, and examples of how to train. Here is one of the examples that I found :

#!/bin/bash
#This is simple example how to use rnnlm tool for training and testing rnn-based language models
#Check 'example.output' how the output should look like
#SRILM toolkit must be installed for combination with ngram model to work properly
make clean
make
rm model
rm model.output.txt
#rnn model is trained here
time ./rnnlm -train train -valid valid -rnnlm model -hidden 15 -rand-seed 1 -debug 2 -class 100 -bptt 4 -bptt-block 10 -direct-order 3 -direct 2 -binary
#ngram model is trained here, using SRILM tools
ngram-count -text train -order 5 -lm templm -kndiscount -interpolate -gt3min 1 -gt4min 1
ngram -lm templm -order 5 -ppl test -debug 2 > temp.ppl
gcc convert.c -O2 -o convert
./convert <temp.ppl >srilm.txt
#combination of both models is performed here
time ./rnnlm -rnnlm model -test test -lm-prob srilm.txt -lambda 0.5 

2/14: First, I downloaded the latest version (rnnlm-0.3e) of the toolkit from the site: http://www.fit.vutbr.cz/~imikolov/rnnlm/ . Second, I needed to compile the tool because they can only provide source code. Luckily there was a makefile included. However, this makefile was using a linux version of g++ to compile which wouldn't work on mac. In the makefile, I replaced the line "x86_64-linux-g++-4.6" with just "g++". After first compilation try, I was getting an error that the function exp10() did not exist. I went into the corresponding c++ library and commented out every line containing this function. I don't believe it would affect the output of the core functionality due to it being use only for logging. After doing this, I compiled again and it worked. I looked in the FAQ provided, ran the train script on the sample data, and a model was successfully created.Note I discovered another solution for the exp10() issue that is more elegant than just commenting it out. Instead, replace each instance of exp10(params...); to pow((double)10,params...);. This will give you the same output and may give us useful information for debugging. Example of use in the file:

// Old function
fprintf(flog, "PPL net: %f\n", exp10(-logp/(real)wordcn));
// New function 
fprintf(flog, "PPL net: %f\n", pow((double)10, (-logp/(real)wordcn)));  
Plan

2/12: Dig further into the RNNLM Toolkit and try some of the provided examples. Sample data is provided so there is no need to wait for this tool to be installed onto Caesar.

2/14: Discuss my findings and provide instructions to my group and figure out what the next step is with the tool kit. I should also dig into the SRILM toolkit used to make n-gram models. the RNNLM Toolkit is able to combine the RNN model and n-gram model together. I'm not sure what the benefit of this is, I would have to do more research.

Concerns

2/12: I'm mostly concerned about changing the current implementation to use an RNN-based LM. I'm afraid that we'll get everything working and find out that RNN-based LM's are not compatible with Sphinx 3.

2/14: I have the same concerns as before.

Week Ending February 21, 2017

2/18: Checking in

2/20: Checking in

Task

2/19: Create and run an experiment myself. We have just gotten the drone Idefix assigned to us so I will test running on the drone. I want to get the experience of running an experiment myself so next time I can work on making the changes to improve the results. It looks like my task will be trying to improve the Language Model by switching to an RNN-based LM. We will do so by using the RNNLM toolkit.

2/21: The plan today is to complete the experiment on the 5 hour corpus. I should be able to pick back up right where I left off granted there were no issues.

Results

2/19: I began to run a train on the 5 hr corpus and I let it run while I walk away. When I came back, PuTTy had given me a network failed exception for some reason. I logged back into Caesar and found that there were still processes running under my username. I will let it continue to run and perform the next two steps next time I get a chance.

2/21: The first step was to create the language model for the 5 hour corpus. This required retrieving the transcripts from the 5 hour directory. When I went to do this, I noticed that the 5 hour corpus had been deleted! I brought it up to the group and it turns out it was just a misunderstanding. The 5 hour corpus was a data set that another student had created this semester but it wasn't functioning properly. So I didn't know that I wasn't supposed to run a train on that corpus. The next step for me was to start the process all over again with the 30 hour corpus. This took a lot longer but got the job done. I finished the experiment and posted the results generated. I did however run into some issues creating the Language Model and decoding on idefix. When I ran the script to generate the Language model, it only created half of the required files which in turn made the other steps fail. I returned to Caesar and ran the scripts and every file was created successfully. I'm not sure what caused this error, there was nothing that jumped out to me on the screen.

Plan

2/19: Finish the experiment on the 5 hour corpus. This entails creating the language model and running the decode.

2/21: Now that I am familiar with how to run an experiment, I would like to start looking into how I could incorporate the RNNLM toolkit into it. This would basically be changing the entire second step of the experiment process. The plan is to do this and hopefully be able to generate better results.

Concerns

2/19: I'm not sure if the error I saw was relevant to the training or just a error I got because my computer went into sleep mode.

2/21: I would like to find out why certain steps of the experiment failed while running on idefix. I would also like to find out how much the procedure will change when the RNN-based language model is implemented.

Week Ending February 28, 2017

2/23: Checking In

2/24: Checking In

Task

2/27: Run an experiment, but switch out the Language Model for an RNN-based LM. This may require extensive research.

2/28: System granted our machine internet access and we were able to get gcc and g++ installed just on our machine. Today the plan is to get the RNNLM toolkit installed on Idefix and continue working on my experiment.

Results

2/27: I had originally built the RNNLM toolkit on my machine and pushed the tool to Idefix. When I tried to execute it, an error was returned indicating that it wasn't in the right format. So right now, i just have an experiment sitting at the second step until I can figure out a way to get the RNNLM toolkit to work. I would compile it on Idefix, but gcc or g++ is not installed on the machine.

2/28: I was able to install the RNNLM toolkit pretty easily now that we have a compiler. I just pushed the source to Idefix, complied the executable, moved that to /usr/local/bin, and then rehashed so now it is available from everywhere. I looked into how the toolkit can be used and this is what I found:

Recurrent neural network based language modeling toolkit v 0.3d
Options:
----
Parameters for training phase:
  -train <file>
Use text data from <file> to train rnnlm model
  -class <int>
Will use specified amount of classes to decompose vocabulary; default is 100
  -old-classes
This will use old algorithm to compute classes, which results in slower models but can be a bit more precise
  -rnnlm <file>
Use <file> to store rnnlm model
  -binary
Rnnlm model will be saved in binary format (default is plain text)
  -valid <file>
Use <file> as validation data
  -alpha <float>
Set starting learning rate; default is 0.1
  -beta <float>
Set L2 regularization parameter; default is 1e-7
  -hidden <int>
Set size of hidden layer; default is 30
  -compression <int>
Set size of compression layer; default is 0 (not used)
  -direct <int>
Sets size of the hash for direct connections with n-gram features in millions; default is 0
  -direct-order <int>
Sets the n-gram order for direct connections (max 20); default is 3
  -bptt <int>
Set amount of steps to propagate error back in time; default is 0 (equal to simple RNN)
  -bptt-block <int>
Specifies amount of time steps after which the error is backpropagated through time in block mode (default 10, update at each time step = 1)
  -one-iter
Will cause training to perform exactly one iteration over training data (useful for adapting final models on different data etc.)
  -anti-kasparek <int>
Model will be saved during training after processing specified amount of words
  -min-improvement <float>
Set minimal relative entropy improvement for training convergence; default is 1.003
  -gradient-cutoff <float>
Set maximal absolute gradient value (to improve training stability, use lower values; default is 15, to turn off use 0)
----
Parameters for testing phase:
  -rnnlm <file>
Read rnnlm model from <file>
  -test <file>
Use <file> as test data to report perplexity
  -lm-prob
Use other LM probabilities for linear interpolation with rnnlm model; see examples at the rnnlm webpage
  -lambda <float>
Set parameter for linear interpolation of rnnlm and other lm; default weight of rnnlm is 0.75
  -dynamic <float>
Set learning rate for dynamic model updates during testing phase; default is 0 (static model)
----
Additional parameters:
  -gen <int>
Generate specified amount of words given distribution from current model
  -independent
Will erase history at end of each sentence (if used for training, this switch should be used also for testing & rescoring)
----
Examples:
rnnlm -train train -rnnlm model -valid valid -hidden 50
rnnlm -rnnlm model -test test


Plan

2/27: Get gcc or g++ installed on Idefix so that I can compile the RNNLM toolkit. We could either wait for the tools group to get it on Caesar, or wait until the systems group can grant us internet access on the machine.

2/28:

Concerns

2/27: I'm not sure how long it will be until we have a compiler to use on Idefix.

2/28:

Week Ending March 7, 2017

Task


Results


Plan


Concerns


Week Ending March 21, 2017

Spring Break
Task


Results


Plan


Concerns

Week Ending March 28, 2017

3/25: Checking In

3/26: Checking In

Task

3/27: Attempt to run experiment on 30 hr corpus using RNNLM tool kit. Greg has already been able to accomplish this but I want to familiarize myself with the steps that he's come up with. It looks like we'll basically be using the RNN model to generate sentences to train the n-gram model on. Also, I want to check in on how the empire group is doing with the new machines. I created experiments to be run on each machine to show that they are both functional.

3/28: Finish the experiment using the RNNLM. These are the steps that I will be using (credit to Greg)

$ rnnlm -train trans_parsed -rnnlm model -valid trans_parsed -hidden 50
$ rnnlm -gen 1000000 -rnnlm model >> rnn_trans
$ rm -rf trans_parsed 
$ cp -i rnn_trans trans_parsed
$ ./lm_create.pl trans_parsed
Results

3/27: I ran into an issue while trying to add an experiment to foss. It turns out the addExp.pl was edited and broken by someone but never fixed. I took it upon myself to see what was wrong with the script and fix it myself. I figured that would be much quicker than trying to ask someone else to do it. It turned out that there was one line missing that would normally stop the script to wait for input. After that I was able to run the train on the 30 hr corpus. I checked on my groups progress and they were both able to run experiments using the 30 hr corpus. They did however run into errors while using the 5 hr corpus which we are still investigating.

3/28:


Plan

3/27: Finished the experiment using the RNNLM toolkit and figure out why the 5 hr corpus is failing on Asterix. It's probable that we just need to copy over /usr/local/. It's also possible that the people before us that used the machine may have messed something up. I also need to read through Greg's logs to determine what the next steps are to use the RNNLM toolkit.

3/28: If we do decide to continue with the RNNLM toolkit, I would either like to create a new script for the new RNN stuff or change lm_create to take in a -rnn flag.

Concerns

3/27: N/A

3:28: N/A

Week Ending April 4, 2017

Task


Results


Plan


Concerns


Week Ending April 11, 2017

4/8: Checking In

4/10: Checking In

Task

4/9: Work on the poster for the URC due this Monday.

4/11: Get RNNLM Toolkit installed on the Empire machines. Run 5-hr experiments on both machines.

Results

4/9: We completed the URC poster as a group. We covered the main components of speech modeling

4/11: I got RNNLM installed onto asterix but couldn't log into miraculix for some reason. I will try to figure that out in next week' meeting. I did not run the 5 hour experiments because I think it may be more important now to focus on LDA and improving the WER by changing the parameters for Sphinx.

Plan

4/9: We will submit the URC to Jonas before the Monday.

4/11: Continue making experiments with LDA and changing Sphinx parameters.

Concerns

4/9: N/A , I think the poster came out great.

4/11: I can't get onto miraculix which is concerning.

Week Ending April 18, 2017

4/13: Checking In

Task

4/12: Start 300 hour experiment on Miraculix with certain parameters.

4/14: Create the language model and run the decode on the 300 hr experiment.

4/16: Score the experiment and compare the results to the baseline we've created without LDA.

Results

4/12: The 300 hour train was started. It may be a day or two before I can create the language model and start a decode. This experiment will be using the baseline configurations from last year + LDA.

4/14: I created the language model with no problems but running the decode gave me issues. I kept getting an error indicating that the script couldn't find one of the shared object libraries. I re-synced /usr/local from Caesar and I still the got the same issue. I ended up just running the decode on Caesar but I'm not sure if it will work properly since Caesar only has Python 2.6.

4/16: The score I got from the experiment was over 7% worse than the baseline we ran without LDA. I may have to redo the decode on a drone with Python 2.7.

Plan

4/12: Create the language model and decode once the train has complete.

4/14: Score the 300 hr exp if and when the decode completes.

4/16: Redo the decode on experiment 0300/009 on a drone with Python 2.7

Concerns

4/14: N/A

4/14: I'm afraid that since I am running the decode on Caesar, that it won't come out right. LDA depends on Python 2.7 where Caesar only has 2.6.

4/16: Since the results were worse, I'm curious if it was because I ran the decode on Caesar. I also hope the error I got previously isn't a road block.

Week Ending May 2, 2017

4/29: Checking In

4/30: Checking In

Task

5/1: Work on documentation for the team report.

5/2: Communicate with team to figure out what we can do in our last week in efforts to improve results. We know were behind right now so we need to do something different.

Results

5/1: I did not have edit access to the report. I have instead looked through our best experiments and wrote some things in my own document.

5/2: We came up with some suggestions that may work and I will try them out. I am not going to put them in these logs because they are top secret. If the results improve, then it will be reflected in the final document.

Plan

5/1: I will transfer the documentation to the master document once I get edit rights.

5/2: Try new experiments using the suggestion provided by the team.

Concerns

5/1: N/A

5/2: Other groups have tried improve the thing I will be working on and they came up with error. Hopefully that does not block my progress.

Week Ending May 9, 2017

5/4: Checking In

5/5: Checking In

Task

5/3: Trying last minute configuration changes

5/6: Check on the experiment that i started a few days ago

5/7: Working on team documentation

Results

5/3: We have one more week so I am trying one last experiment in efforts to improve the WER

5/6: The experiment failed to train. This could be from the configuration change but I am not sure.

5/7: Worked on team documentation

Plan

5/3: The experiment is training, I will check back in later.

5/6: There is not enough time to try another configuration, we will have to go with the best results that we had last week.

5/7: Hand in documentation for Wednesday

Concerns

5/3: N/A

5/6: I'm not sure why the experiment failed.

5/7: N/A