Speech:Spring 2015 Nicholas Tello Log


 * Home
 * Semesters
 * Spring 2015
 * Proposal
 * Report
 * Information - General Project Information
 * Experiments - List of speech experiments

Week Ending February 3, 2015
01/30 - Read others logs 02/01 - Read over the information concerning Media Wiki left from past experiment group as well as experiment with what I could. 02/02 - Read others logs 02/03 - Read over the first 1/3 of the scripts from the past experiment group and evaluated them by determining which ones we will keep and which we will not.
 * Task:

2/01 - Discovered a blank page under the Discussion link from the following page: http://foss.unh.edu/projects/index.php/Speech:Spring_2015 Read through the Project Wiki Documentation page. Noted a few spelling/gramar errors in need of correction. Also found that the "How Do I Create A Page" description will need modification to match the new process. Found the information concerning links and the code behind the wiki especially helpful and have marked them for future refrence material. The 'Special Pages' will need to be edited to match the new wiki operations, specifically concerning the 'template' links. API seems strait forward and easy to use.
 * Results:

02/03 - The 'buildData' script will not be usable as it does not conform to the new model that the experiment group is moving in. The 'checkTrain' script will be kept and used. The 'clone_exp.pl' script will be able to be used but might need a bit of editing. The 'convert.pl' script will be able to be used and should be quite helpful. The 'copySph.pl' script will be able to be used as well and will also be useful when used in tandem with the 'convert.pl' script. The 'createTranscript.pl' script should be able to be used and will be able to be used. The 'createSubTranscript.pl' script should be able to be used and operates very effectively if combined with the 'createTranscript.pl' both inside a larger script that we might be able to write.


 * Plan:

1) Familiarize myself with the way the wiki API works and come to a firm understanding 2) Look over the first third of the scripts from the past Experiments group and determine which scripts we will keep and which we will toss 2/01 - I worry about how to link the media wiki API up to the servers for automation. Will continue to look into the advanced mechanics of the API. Perhaps a script will be able to link the two together effectively.
 * Concerns:

Week Ending February 10, 2015
02/07 - Read others logs 02/08 - Read others logs 02/09 - Read through three different downloadable/usable pearl API's that will work in correspondence with the media wiki built in API. 01/10 - Read through the remaining scripts from previous semesters and determined which could be reused and which couldn't.
 * Task:

02/09 - Read through the CPAN site and documentation. Documentation isn't the best but the API seems solid. It has a command: "action => 'upload'" The command allows us to use a file designated by filename from the built in media wiki API that allows the pearl API to grab the file and then convert it to an array that we might be able to send over to Ceasar to execute the creation of the new experiment on the servers. The second one I looked at I don't know if it is worth pursuing. The API came from pearlmediawikiapi.wiki.com. I looked through the Documentation for the API, or the lack there of. The documentation page on their website has a statement saying that there is inline documentation inside the code, and that the documentation is to be uploaded at a later date, but I do not know if we have the time to wait for that to happen, nor do we want to have to read through all the comments in the code so that we can understand which commands are used to do what. The third API i looked at was from wiki.servicenow.com. The third one also had a lack of documentation but on the bring side it did have a line of code: perldoc ServiceNow     that will bring up the documentation for the API. I did read through the rest of the information on the site that concerns the API. It has a good flowchart that explains how the API operates and has a list of all the commands the API includes. Based on how the commands looks in the chart, this API should accomplish what we need as well but I will need to discuss it with my partners to decide which we would like to use.
 * Results:

02/10 - I read through the remaining scripts. The following scripts seem to be usable: dictionary, find, gen_errors, Im_create, parseDecode, pruneDictionary, train_01, train_02, trans_time, updateDict,

The following scripts look to possibly need to be reworked: generateFeats, genFileIDs, GenTrans, master_run_train, prepareExperiment, prepareExperiment2, prepareExperiment3,

The following scripts look like they will not be usable: Could not find any

I have sent these off to my other teammates for further deliberation on what exactly we will do with these.

Keep up to date on other team members logs
 * Plan:

Come up with ideas on which pearl API we wish to use and distribute it to the rest of the group for consultation

Sort through the remaining scripts
 * Concerns:

Week Ending February 17, 2015
02/12 - Run a experiment from start to finish including all steps and take notes on the process
 * Task:

01/12 - Ran experiment 0262. Is still running the train. Will continue work on this once it finishes, but I do not think it will be tonight. Notes taken thus far are as follows: 1) Launch Terminal 2) Log into caesar using: ssh npa23@caesar.unh.edu 3) Go to root lv of caesar 4) Navigate to directory 'mnt' 5) Navigate to directory 'main' 6) Navigate to directory 'Exp' 7) Create directory for new experiment '0262' 8) Navigate to directory of new experiment '0262' 9) ran scripts from directory: /mnt/main/scripts/user/prepareExperiment3.pl first_5hr/train	Note: See Wiki page for explanation on how this script works 10) ran scripts from directory: /mnt/main/scripts/user/generateFeats2.pl	Note: See Wiki page for explanation on how this script works 11) ran scripts from directory: nohup scripts_pl/RunAll.pl . &	Note: See Wiki page for explanation on how this script works
 * Results:

Will continue to monitor its progress and will continue the process once it finishes. Checked on others logs while I was here.

02/15 - Read others logs

02/16 - Read others logs

02/17 - Finished experiment 0262, with a result of 49.6% sum/average word error rate after a 5hr train. The steps i took above and beyond what was previously logged this week are as follows:

12) created new directory named: “LM” 13) changed directory to: “LM” 14) ran: cp -i /mnt/main/corpus/switchboard/first_5hr/train/trans/train.trans trans_unedited	Note: This copies over the transcript used from gentians.pl 15) ran: /mnt/main/corpus/switchboard/dist/Switchboard/transcripts/ICSI_Transcriptions/trans/icsi/ParseTranscript.perl trans_unedited trans_parsed Note: This is to prepare the transcript 16) ran: cp -i /mnt/main/scripts/user/lm_create.pl .	Note: This copies the script that is used to create the language model 17) ran: ./lm_create.pl trans_parsed Note: This executes the previously mentioned script and concludes the creation of the language model. 18) created new directory under base experiment directory called DECODE using ‘mkdir DECODE’ 19) changed directory into DECODE 20) ran: cp -i /mnt/main/scripts/user/run_decode2.pl .	Note: This is to create a copy of the run_decode2.pl file in our new directory 21) ran: nohup run_decode2.pl 0262 0262 1000 Note: this is to execute the script and passes the parameters of 0262 (being the experiment number and then 0262 again as there are no sub experiments yet with a senome count of 1000 (the default) 22) ran: /mnt/main/scripts/user/parseDecode.pl decode.log ../etc/hyp.trans Note: this is to create the hypothesis transcript. It will return a error that we should investigate into possibly fixing? Not sure if this falls under our groups responsibilities but worth mentioning. 23) changed directories to ‘etc’ 24) ran: sclite -r0262 _train.trans -h hyp.trans -i swb >> scoring.log Note: This is done to give us a output file, in this case ‘scoring.log’ from the sclite program that is grading the process. 25) ran: cat scoring.log	Note: This was so that I could visually see what the results of the experiment were. In the end this experiment resulted in a 49.6 Sum/Avg word error rate. Create a new experiment from scratch and run it to completion to get a better understanding on how the process works.
 * Plan:
 * Concerns:

I am unfamiliar with both unix and this process so this will be a learning experience.

Week Ending February 24, 2015
02-21 Read Others Logs 02-22 Read Others Logs 02-23 Read through the master script and identified the different portions of the script that I can take out and split up into multiple seperate functions. Then added these into a rough draft of the new experiment process.
 * Task:
 * Results:


 * Plan:


 * Concerns:

Week Ending March 3, 2015
02/28 - Read Logs 03/01 - Read Logs 02/02 - Created a new version of the master script. It is smaller and the first part of splitting up the master script in the experiment creation process. 02/04 - Solved syntax error and began work on fixing file location before being contacted by Sam Sweet. 02/02 - The new script continues to crash on line 59. Will continue to debug and investigate what the issue might be. 02/03 - Error was being caused by a syntax error. Upon fixing this error I found that the script had issues performing due to locating files. Before I got too far into this, I spoke with Sam over email. He said he had read my logs and wanted me to hold off on perusing this any further as he has made edits to the file and I will need to see the difference before I can peruse this further for fear of reversing his work.
 * Task:
 * Results:


 * Plan:


 * Concerns:

Week Ending March 10, 2015
03/07 - Read Others Logs 03/08 - Read Others Logs 03/09 - Read through the createWikiExperiment.pl file Morgan sent to me and created a guide on how to use it under the Experiment Setup guide. 03/10 - Read through the createWiki_Sub_Experiment.pl file Morgan sent to me and created a guide on how to use it under the Experiment Setup guide.
 * Task:

03/09 - Created a comprehensive guide on how to use the new createWikiExperiment.pl file. It can be found under the Experiment Setup guide and reads as follows:
 * Results:

Before running the train, you need to add the experiment to the master experiment log, as well as add the experiment on the media wiki. You will do this by executing the createWikiExperiment.pl script.

Once executed, the script will prompt you to enter your login info for the domain. After you enter this information, the script will check to see what the last experiment created was by its experiment number. It will then increment this and assign you the next available number.

< Check that this is the correct number. If someone creates a experiment without using this script and skips a number or many, the auto increment will continue after the previously created experiment, even if it was entered incorrectly.>

The script will then ask you what your experiment name is. After entering this, the script will prompt you to enter the author's name, in this case that is your name. Finally, the script will ask you to enter a brief description of what your experiment hopes to accomplish or test. The script will fill in all other information automatically, and if this process is done correctly, the script will return with a message saying "Your experiment number is (your experiment number). Please go to Caesar and make a directory for this experiment."

03/10 - Created a comprehensive guide on how to use the new createWiki_Sub_Experiment.pl file. It can be found under the Experiment Setup guide and reads as follows:

If you wish to make a sub experiment under the experiment directory created with the previous steps, you will need to use the createWiki_Sub_Experiment.pl script.

Upon launching this script, you will be prompted with "please enter your username->". Enter this data and hit enter After entering your user name, the script will prompt you with "please enter your password->". Enter this data and hit enter

The script will then authenticate you and prompt you with "please enter the main experiment number (Ex: 0268)->". Enter the experiment number here that you wish to create a sub experiment for and then hit enter.

The script will automatically create the next available sub experiment, starting with sub experiment #001 and incrementing up from there. You will then be prompted with a message reading "what is your sub-experiment's name?->". Enter this data and then hit enter. You will then be prompted with a message reading "please enter the author's name->". Enter this data and then hit enter. You will then be prompted with a message reading "Please enter a brief description of your sub-experiment->". Enter this data and then hit enter.

The script will then return with a message reading, "Your sub-experiment number is (###). Please go to Caesar and make a directory for this sub-experiment."

Log into Caesar and create this directory.
 * Plan:


 * Concerns:

Week Ending March 24, 2015

 * Task:


 * Results:

3/16: Read through others logs.

3/20: Read through some other logs.

3/21: Caesar is still down, so I am still unable to work on my tasks.

3/22: Caesar is still down? Trying to find out whats going on.


 * Plan:


 * Concerns:

Week Ending March 31, 2015
03/28 - Read Others Logs 03/29 - Read others logs and linked up my accounts so as to make successful connection to caesar. 03/30 - Read through and analyzed the setup_SphinxTrain.pl script 03/31 - Read through and analyzed the generate_feats.pl script
 * Task:

03/30 - See team Patriots private page. 03/31 - See team Patriots private page.
 * Results:


 * Plan:


 * Concerns:

Week Ending April 7, 2015
04/04 - Read Others Logs 04/05 - Read Others Logs 04/06 - Created new sub experiment. See pats page.
 * Task:
 * Results:


 * Plan:


 * Concerns:

Week Ending April 14, 2015
10/15 - Read Others Logs 10/16 - Read Others Logs
 * Task:


 * Results:


 * Plan:


 * Concerns:

Week Ending April 21, 2015

 * Task:


 * Results:


 * Plan:


 * Concerns:

Week Ending April 28, 2015

 * Task:


 * Results:


 * Plan:


 * Concerns:

Week Ending May 5, 2015

 * Task:


 * Results:


 * Plan:


 * Concerns: