Speech:Spring 2018 Stephen Thibault Log


 * Home
 * Semesters
 * Spring 2018
 * Proposal
 * Report
 * Information - General Project Information
 * Experiments - List of speech experiments

1FEB18
Created sub-experiment 001 in 0303 in Caesar.
 * Task:

Sub-experiment 001 shows as created.
 * Results:

Research how to perform an experiment and what tools, files, scripts, etc. are required.
 * Plan:

Talking with a number of people from the class that we will somehow screw up Caesar and break something.
 * Concerns:

2FEB18
Read Basic Concepts of Speech Recognition, the CMUSphinx Tutorial For Developers and Training an Acoustic Model for CMUSphinx.
 * Task:

Learned about speech structure, recognition process, acoustic and language models, building a dictionary and a language model, when to train and not, data preparation and setting up training scripts.
 * Results:

Attempt to run first experiment this weekend.
 * Plan:

No new concerns at this time. However, I feel like I should have one.
 * Concerns:

3FEB18
Created dir 001 within 001 for first experiment to run. Within 001/001, built files and directory structure, then modified the train configuration, generated feats data, and then ran the train.
 * Task:

Didn't appear to have any issues building the directory structure and files, modifying the train configuration or generating the feats data, but something failed in verify_all in the scripts.
 * Results:

Re-research Sphinx train configuration to try and determine the source of that error and how to fix it.
 * Plan:

That I am barrelling ahead without having a better understanding of how everything works together.
 * Concerns:

4FEB18
Reviewed log entries. Changed Modeling Group Member Logs page to reflect group member's names linked to the respective individual log entry page.
 * Task:

Spring 2018 Modeling Group page now has Group Member Logs section reflecting group member's names which are linked to each member's individual log page.
 * Results:

6FEB18
1: Determined that the experiment run on Saturday did indeed not work. Worked with fellow project members to determine correct procedure. Ran into some issues as I was not running it on a drone server from the correct location. Once the correct location was determined, was able to successfully run a train that made it to MODULE: 99.
 * Task:

2: Updated Modeling Group page Task entry with what Professor Jonas stated should be our way forward. Recommended all other groups do the same via Discord. Also recommended all groups link their individual log pages to their names located in their group pages as Software has already done.

1: Even though the train ran successfully to MODULE: 99, it stated in the HTML file that it "Can not create models used by Sphinx-II.
 * Results:

1: Research the issue with creating models used by Sphinx-II.
 * Plan:

None at this time.
 * Concerns:

7FEB18
Reviewed logs.
 * Task:

Reviewed logs.
 * Results:

Research the areas that need to be addressed as listed in the Model Group Tasks for the upcoming rough draft proposal.
 * Plan:

None at this time.
 * Concerns:

10FEB18
Utilizing Create LM [] and Run Decode Trained Data [] with appropriate modifications recommended by other students in this iteration, I was able to successfully make it through Decode but not through Scoring as I received an error that seems to have to do with correct memory usage due to a Segmentation fault when trying to compare the train file that has been decoded with a reference file.
 * Task:

Segmentation fault (core dumped) occurred when trying to run sclite 2.3 TK Version 1.3 "Begin alignment of Ref File: '001_train.trans' and Hyp File: 'hyp.trans' which does show in command line "SYSTEM SUMMARY PERCENTAGES by SPEAKER" so it appears to start to run (even though I didn't have it outputting to file on this second attempt as I wanted to see what it would output to command line without trying anything differently OTHER THAN not outputting to a file) BEFORE the Segmentation fault (core dump) occurs. When reviewing the decode.log file, "FATAL_ERROR: "mdef.c", line 680: No mdef-file" appears on line 195 so I need to determine why that is appearing (besides the obvious that an mdef file is expected and isn't there).
 * Results:

Research the scoring issue(s?) and find a way ahead. Tomorrow meeting my Modeling Group teammates to hammer out the rough draft proposal.
 * Plan:

That there are not enough hours in the week to accomplish everything in a timely manner.
 * Concerns:

11FEB18
Based on another student's experience, the 0303/001/001 directory structure I was using was incorrect. I deleted both 001's and started from scratch in order to run a 5 hour train and create LM within 0303/001 alongside all of the other files and folders that were created like /etc and the like.
 * Task:

This turned out to be successful as I was able to run a 5 hour train, decode it and score it successfully.
 * Results:

Tomorrow devise a Plan of Action and Milestones for the Modeling Group moving forward.
 * Plan:

Comprehending enough of what we are supposed to do to be successful.
 * Concerns:

12FEB18
Finished typing up my notes for procedures and commands to run a successful Train, Decode and Scoring. Shared them with my Modeling Group partners and placed them in Speech:Exps 0303 001 along with the results from the Scoring Log. I informed all Capstone Team Members via Discord that I had done this, recommended that those who had run a successful Train, Decode and Scoring all the way through do the same. I also invited everyone to view my procedures and commands, use them, and provide feedback on usability - meaning, does what I have there make sense to everyone.
 * Task:

Positive feedback so far. No input for changes as of this posting.
 * Results:

Invited Modeling Group team partner Brian to let me know if he needed any assistance with what I provided him to run a Train, Decode and Scoring. Awaiting any response. Tomorrow morning will work with Modeling Group team partner Hannah to run all the way through. I also need to develop a Group Modeling Team POA&M for EVERYTHING that needs to be done. We have to track what is being done, current status, and what needs to be started better than we have been.
 * Plan:

The later I work at night, the more tired I become, the greater the chance I will make mistakes.
 * Concerns:

13FEB18
Assisted Hannah (5hr) and Danielle (30hr) with running a Train, Decoding and Scoring.
 * Task:

Received feedback on my Train, Decode and Scoring instructions that I made available in my Experiment Log. Changed single parentheses style so that the code could be copied and pasted correctly without generating a format error. Hannah and Danielle were both able to complete a Train, Decoding and Scoring to generate the expected files.
 * Results:

Enhance the Modeling Group proposal as stated in the Modeling Group Log under Tasks.
 * Plan:

Not enough hours in the day.
 * Concerns:

14FEB18
Run a 30hr Train, Decode and Scoring as sub-experiment 020.
 * Task:

Successfully ran 30hr Train, Decode and Scoring as sub-experiment 020 over 3 hours this morning. Placed the results in the appropriate sub-experiment log.
 * Results:

Probably will use the results of sub-experiment 020 as the comparison to another 30hr Train, Decode and Scoring with Linear Discriminant Analysis (LDA) once I figure out how to incorporate LDA into the process.
 * Plan:

Same as yesterday.
 * Concerns:

15FEB18
Attempting to run a test on Unseen Data.
 * Task:

1. addExp.pl -s as you normally would
 * Results:

2. Run a training experiment as you normally would using the desired amount of hours

3. Build the language model as you normally would.

4. run "makeTest.pl -t switchboard/30hr 0303/020 0303/023"

5. Following appears: LM dir is ready

cp: overwrite './bin/inc_comp'?

6. Should that appear? Others appeared, several dozen different ones to be overwritten.

7. Clearly not doing this in the right place. I think.

8. After saying no to every overwrite question, it stated the following:

Decode is ready to be executed.

AM pointed to 0303/020

LM generated from switchboard/30hr/test/trans/train.trans

Note: Generate feats with genFeats.pl -d

then execute

run_decode.pl 0303/020 0303/023 1000

9. Looking at https://foss.unh.edu/projects/index.php/Speech:Run_Train_Setup_Script,

it appears that I should create a test directory within the experiment folder that I

just created the train and LM in.

10. Doing that created the same results in steps 5 and 8 so I said no to all the overwrite questions. 11. Will now attempt to set this up with an additional sub-experiment under 0303.

12. In 0303, created 024 and attempting to run "makeTest.pl -t switchboard/30hr 0303/023 0303/024"

13. Ok, this may have worked as I received the following:

LM dir is ready.

Decode is ready to be executed.

AM pointed to 0303/023.

LM generated from switchboard/30hr/test/trans/train.trans

Note: Generate feats with

genFeats.pl -d

then execute

run_decode.pl 0303/023 0303/024 1000

14. Runs through and completes.

15. cd etc

16. nohup run_decode.pl 0303/023 &

17. Returned [1] 10432

18. Transform the docode.log file to hyp.trans file by running "parseDecode.pl decode.log hyp.trans".

19. Returned "rm: cannot remove 'hyp.trans': No such file or directory", which is expected.

20. Ran "sclite -r 024_train.trans -h hyp.trans -i swb >> scoring.log".

21. Returned error '024_train.trans does not exist. Which I knew from ls'g the folder.

22. Am I in the wrong /etc folder?

NEED TO COMPARE WHAT IS IN OTHER etc FOLDER AND THIS ONE!
 * Plan:

Spending way too much time trying to figure out what has already been done but documented horribly.
 * Concerns:

16FEB18
Entering this on Sunday night, 18 February, trying to remember what I worked on regarding the Project Proposal. I just can't remember at this point.
 * Task:

I don't remember. This why I need to update this (or the text file I keep open for Capstone) each and every day.
 * Results:

Meeting with Hannah and Brian tomorrow to continue working on our Model Group Project Proposal. Still need to figure out what is going wrong with running an Unseen Data Experiment.
 * Plan:

Not enough time to do all the things expected. It seems we are on pace with previous iterations based on viewing their logs. I don't know.
 * Concerns:

17FEB18
Entering this on Sunday night, 18 February. Worked with Hannah and Brian on the Model Group Project Proposal together today for 3 hours. Worked at home on Project Proposal for 3 hours afterwards.
 * Task:

Revised Overview; researched Torque to little progress; started paragraphs on Parallelization, Documentation. At home, completed Parallelization, Timeline, Configuring Sphinx, Documentation, Timeline.
 * Results:

Tomorrow will work on why I cannot make the Unseen Data experiment work - specifically determine if I am using the right procedures regarding directory structure and which directory to be in when trying to run decode and scoring after running the train and language model.
 * Plan:

Same as yesterday.
 * Concerns:

18FEB18
Working in parallel with Dan Beitel, determine correct directory structure and move forward with a 5hr train (instead of the 30hr train used a few days ago) in creating a decode and score with Unseen Data.
 * Task:

Using the successful procedures from the 5hr experiment on Unseen Data 0303/028 and 0303/029, decided to go back to the successful 30 hour Train in 0303/023 and Decode and Score that in 0303/024.

We determined I was on the correct path the other day but missed a step or two:
 * Results:

Picking up from step 22 on the 15th of February:

23. Attempting again in 0303/028 with a 5hr train. Train and LM created without issue. 24. Created 0303/029 to run "makeTest.pl   within it.

25. So ran "makeTest.pl -t switchboard/5hr 0303/028 0303/029". 26. Ok, this may have worked as I received the following (again):

LM dir is ready.

Decode is ready to be executed.

AM pointed to 0303/023.

LM generated from switchboard/30hr/test/trans/train.trans

Note: Generate feats with

genFeats.pl -d

then execute

run_decode.pl 0303/028 0303/029 1000

27. Run "genFeats.pl -d"

28. Move to /etc folder in /029.

29. The instructions here https://foss.unh.edu/projects/index.php/Speech:Run_Decode_Unseen_Data from in bold

"At this point, execute the following commands regardless of whether or not you executed makeTest.pl.", state to go into the etc directory (of the second folder, in this case 029) and execute the following:

"nohup run_decode.pl 0303/028 0303/029 1000 &"

30. Returned [1] 17470

31. Transform the decode.log file to hyp.trans file by running "parseDecode.pl decode.log hyp.trans"

31. Inside 028/etc folder, "cp 028_train.trans /mnt/main/Exp/0303/029/etc"

32. Move back to 029/etc folder

33. Run "sclite -r 028_train.trans -h hyp.trans -i swb >> scoring.log"

33. THIS RESULTED IN A CORE DUMP first time through.

34. Back to step 25 in this instruction. This worked the second time through!!

For 0303/023 to 0303/024, I was successful and the results are in the experiment log.

Tomorrow will work on the LDA portion and try to run on the 5hr train that I did today with Unseen Data.
 * Plan:

Trying to accomplish everything listed out by the Project Lead this past Tuesday. Not likely.
 * Concerns:

19 FEB 2018
Determine correct directory structure and procedures for a 30hr Train, Decode and Scoring with Unseen Data utilizing Linear Discriminant Analysis.
 * Task:

Close but no cigar. Attempted twice after determining exact procedures from wiki notes and prior iteration Model Group log.
 * Results:

Derived from https://foss.unh.edu/projects/index.php/Speech:Spring_2017_Modeling_Group and enhanced: by "vi /mnt/main/Exp/0303/031/etc". Navigate and change as noted lines 156 and 158 in the configuration file.
 * 1. Ensure server is using Python 2.6.6 (2.7)
 * 2. If not, run "export PATH=/usr/local/miniconda/bin:$PATH)"
 * 3. run "addExp.pl -s" on caesar
 * 4. ssh to a drone server
 * 5. Make the train with "makeTrain.pl switchboard 30hr/train"
 * 6. locate "sphinx_train.cfg" file in /mnt/main/Exp/0303/031/etc
 * 7. alter the sphinx_train.cfg file so that $CFG_LDA_MLLT = 'yes' and $CFG_LDA_DIMENSION = 32
 * NOTE: while in vi, press i to enter Insert mode. To leave Insert mode press ESC.
 * All of the following may be utilized and must be done outside of Insert mode =
 * Press x to delete the character under the cursor. Press :x to save. Press :q to leave the file.
 * Press :q! to leave the file WITHOUT saving. Use arrow keys to navigate through the text.


 * 8. After exiting, to confirm changes, while still in /etc, "cat sphinx_train.cfg"
 * 9. "cd .." to move up one directory from /etc
 * 10. "genFeats.pl -t"
 * 11. "nohup scripts_pl/RunAll.pl &"
 * 12. While in MODULE 5 Train LDA Transformation, Phase 3 Forward-Backward, received the following:
 * packet_write_wait: Connection to 132.177.189.63 port 22: Broken pipe.
 * Apparently lost ssh connection.
 * So MODULE 5 Phase 1: Cleaning up directories: accumulator...logs...qmanager...
 * Phase 2: Flat initialize

When this happens, the nohup command may still be running. ***TAKE NOTE OF THE FILES THAT ARE IN FOLDER AND ETC FOLDER BEFORE and AFTER RUNNING nohup command.*** After assumed run time, look for sub-directory#.html log to confirm it completed.

I had already wiped Files in subdirectory folder BEFORE nohup: bin / etc / logdir / model_parameters / scripts_pl / bwaccumdir / feat / model_architecture / python / wav Files in etc folder BEFORE nohup: 031.dic / 031_train.fileids / feat.params / sphinx_train.config / 031.filler / 031_train.trans / genPhones.csh / 031.phone / add.txt / sphinx_decode.cfg

BEGAN SECOND ATTEMPT AT nohup AT 10:56 am. Made it straight to MODULE 5 Phase 2 Flat initialize right away. Within a minute, moved to Phase 3 Forward-Backward. Looking at top through another terminal window, the PID 22897/22898 for my username keep incrementing for the time column for the command bw, but the terminal showing the MODULE: 05 stays at Phase 3: Forward-Backward Eventually the PID changed to 22924/22925, still with bw command, time+ keeps increasing. PID 23021/23022.

Received at 11:32 am: MODULE: 06 TRAIN MLLT transformation Phase 2: Flat initialize FATAL_ERROR: "main.c", line 98: Failed to read LDA matrix This step had 1 ERROR messages and 0 WARNING messages. Please check the log file for details. Something failed: (/mnt/main/Exp/0303/031/scripts_pl/06.mllt_train/slave_mllt.pl)

Having difficulty FINDING the "log file".

Will have to investigate which log the system is referring to and proceed forward from there. Same as always. Not enough hours in the day. Most things take much longer than they should which puts me farther behind.
 * Plan:
 * Concerns:

23 FEB 2018
Continue to determine correct directory structure and procedures for a 30hr Train, Decode and Scoring with Unseen Data utilizing Linear Discriminant Analysis.
 * Task:

Third attempt at 5hr train, decode and scoring on Unseen Data using LDA! by "vi /mnt/main/Exp/0303/031/etc/sphinx_train.cfg". Navigate and change as noted lines 156 and 158 in the configuration file. All of the following must be done outside of Insert mode = Press x to delete the character under the cursor. Press :x to save. Press :q to leave the file. Press :q! to leave the file WITHOUT saving. Use arrow keys to navigate through the text. For example, "makeTest.pl -t switchboard/5hr 0303/031 0303/032"
 * Results:
 * 3. run "addExp.pl -s" on caesar
 * 4. ssh to a drone server
 * 5. Make the train with "makeTrain.pl switchboard 30hr/train"
 * 6. locate "sphinx_train.cfg" file in /mnt/main/Exp/0303/031/etc
 * 7. alter the sphinx_train.cfg file so that $CFG_LDA_MLLT = 'yes' and $CFG_LDA_DIMENSION = 32
 * NOTE: while in vi, press i to enter Insert mode. To leave Insert mode press ESC.
 * 8. After exiting, to confirm changes, while still in /etc, "cat sphinx_train.cfg"
 * 9. cd .. to move up one directory from /etc
 * 10. genFeats.pl -t
 * 11. nohup scripts_pl/RunAll.pl &
 * 12. create new subexperiment in the wiki with "addExp.pl"
 * 13. run "makeTest.pl   inside new subexperiment. Use -t flag for test data.
 * 14. This should return:
 * LM dir is ready.
 * Decode is ready to be executed.
 * AM pointed to 0303/031
 * LM generated from switchboard/5hr/test/trans/train.trans
 * Note: Generate feats with
 * genFeats.pl -d
 * then execute
 * run_decode.pl 0303/031 0303/032 1000

Well, those files were not present. Here are the files that were present: 032_decode.fileids, 032.filler, feat.params, sphinx_train.cfg, 032.dic, 032.phone, run_decode.pl For example, "nohup run_decode_lda.pl 0303/031 0303/032 1000 &" You may see the ususal "rm: cannot remove 'hyp.trans': No such file or directory [1] + Done		run_decode_lda.pl 0303/031 0303/032 1000" For example, INSIDE the 0303/031/etc folder, "cp 031_train.trans /mnt/main/Exp/0303/032/etc" For example, "sclite -r 031_train.trans -h hyp.trans -i swb >> scoring.log"
 * 15. AT THIS POINT SPRING 2017 MODEL GROUP NOTES STATE: "cd into the etc folder, notice the files names "est. ". Change the names to "test. ".
 * 16. In 0303/032, run "genFeats.pl -d"
 * 17. cd into 032/etc
 * 18. run "nohup run_decode_lda.pl  &" NOTE THE "lda"!
 * 19. Returned "[1] 31656" (NOTE: first time through. 4th time returned "[1] 2677")
 * 20. run "parseDecode.pl decode.log hyp.trans"
 * 21. Copy over the /etc/_train.trans from the training experiment.
 * 22. move back to your test exp/etc folder, in my case 032/etc.
 * 23. run "sclite -r _train.trans -h hyp.trans -i swb >> scoring.log"

SECOND, THIRD, FOURTH TIME THROUGH RESULTED IN A CORE DUMP!! ARGH!!! Back to step 13 above for the third time tonight. Back to step 13 above for the fourth time tonight, but this time on server drone Asterix. CORE DUMPED AGAIN ON FIFTH & SIXTH ATTEMPTS, both on Asterix. CORE DUMPED AGAIN ON SEVENTH ATTEMPT, on Miraculix. CORE DUMPED AGAIN ON EIGHTH ATTEMPT, on Idefix.

Keep going until the core stops dumping. Had this happen before and it stopped happening without my doing anything differently. Strongly suspect the first six core dumps were due to genFeats.pl being run in the wrong location due to incorrect directions from previous semester (what a shock). However, the decode log for the seventh attempt on Miraculix stated the following: "/usr/local/bin/sphinx3_decode: error while loading shared libraries: libs3decoder.so.0: cannot open shared object file: No such file or directory". Decode log for the eighth attempt on Idefix stated: "FATAL_ERROR: "mdef.c", line 680: No mdef-file". Unable to run without the core dumping. Which should stop at some point. Right?
 * Plan:
 * Concerns:

24 FEB 2018
Continue to determine correct directory structure and procedures for a 30hr Train, Decode and Scoring with Unseen Data utilizing Linear Discriminant Analysis.
 * Task:


 * Results:
 * Created sub-experiment 0303/046 to run the Decode and Scoring with LDA on 0303/031 SECOND attempt; this time on server drone Obelix.
 * Created sub-experiment 0303/047 to run the Decode and Scoring with LDA on 0303/031 THIRD attempt; this time on server drone Miraculix.
 * Created sub-experiment 0303/050 to run the Decode and Scoring with LDA on 0303/031 FOURTH attempt; this time on server drone Asterix.
 * Created sub-experiment 0303/051 to run the Decode and Scoring with LDA on 0303/031 FIFTH attempt; this time on server drone Majestix.
 * Created sub-experiment 0303/052 to run the Decode and Scoring with LDA on 0303/031 SIXTH attempt; this time on server drone Idefix.

The plan with doing this on each server drone was to determine which error could be replicated after determining that there was a significant problem with running this correctly. ALL BUT ONE RESULTED IN CORE DUMP with different reasons in the decode log.
 * Plan:
 * 0303/047 on Miraculix resulted in CORE DUMP with "/usr/local/bin/sphinx3_decode: error while loading shared libraries: libs3decoder.so.0: cannot open shared object file: No such file or directory".
 * 0303/051 on Majestix resulted in an inability to execute final command "sclite -r ….." as it returns "sclite: Command not found."
 * 0303/046,050,052 on Obelix, Asterix, and Idefix, respectively, all resulted in CORE DUMP with the same error in the decode log: "FATAL_ERROR: "mdef.c", line 680: No mdef-file"

This FATAL_ERROR indicates to the Experiment Group and myself that there is a file missing (mdef) and that if Miraculix and Majestix were not having their own different issues, this FATAL_ERROR regarding the mdef-file would occur there as well.

Currently working with Experiment Group to determine why this is happening. See Dan Beitel's personal log for this date for details of his theory. Running the LDA without a core dump.
 * Concerns:

25 FEB 2018
Started Model Group's Second 300hr Train, Decode and Scoring (0303/053). This one is using genTrains.pl from 2016 and the all new and improved parseLMTrains.pl which Data Group was good enough to come up with. This was started at 09:18 am and is continuing as I view the "top" command on Obelix at 08:32 pm. The ssh pipe in another terminal window to Obelix broke many hours ago so I cannot view the MODULE process as it iterates.
 * Task:

So far, so good on "nohup scripts....." running a couple more hours I should think.
 * Results:

Create the Language Model first thing in the morning. That I will have nightmares about this turning into a core dump.
 * Plan:
 * Concerns:

26 FEB 2018
Created the Language Model, Decoded and Scored the 300hr TDS II (0303/053). Also commenced 300hr TDS III (0303/055) with a higher senone count (8000 instead of 1000 used in 0303/053).
 * Task:

0303/053: The Language Model creation, Decode and Scoring took less than 15 minutes. So the 12 hours that it took for the "nohup scripts..." while making the Train is definitely the winner for time consumption in creating a 300hr train. I am presuming the high Summary Average is due to the 1000 senone count that I used. However, there were two SPKR with absurdly high error rates as well, sw2007b and sw2013a. 7 of the 13 SPKRs had an error rate at or above 50%. Working with Data Group to determine why, especially sw2013a which had a 100% error rate.
 * Results:

0303/055 is in progress.

0303/055: Create the Language Model, Decode and Score this evening after class. UPDATE - after evening class: still running, most likely due to someone else running a smaller train on Obelix while /055 was running. Which I observed happen. That the higher senone count will not have an impact on the overly high error rate going from /053 to /055.
 * Plan:
 * Concerns:

27 FEB 2018
Created the Language Model, Decoded and Scored the 300hr TDS III (0303/055). Created subexperiments /063 on Miraculix and /064 on Idefix with 5hr TDS to determine individual drone server performance (pass/fail).
 * Task:

0303/055: FAILED with a CORE DUMP due to "FATAL_ERROR: "mdef.c", line 680: No mdef-file". This is the fifth time I have seen this error. I have started a log off the wiki to keep track of what errors have occurred in which experiment on which server. (i.e., 050 Asterix "FATAL....").
 * Results:

Idefix (0303/064) COMPLETED a 5hr TDS. Miraculix (0303/063) FAILED with CORE DUMP due to "/usr/local/bin/sphinx3_decode: error while loading shared libraries: libs3decoder.so.0: cannot open shared object file: No such file or directory".

Commenced a 300hr Train & LM on Unseen Data on Obelix (0304/001).

Determine which drone servers have been used to successfully complete at least one TDS. So far, Obelix (/053) and Asterix (/049) are the only ones I have on official record (cited in a wiki entry). Perform 5hr TDS on the other two drone servers which are available for testing, Miraculix and Idefix, to determine TDS availability.
 * Plan:

The failure on Miraculix is the third one with that same Fatal Error. 0303/031 is a 5hr Unseen Data with LDA TRAIN ONLY without Decode and Scoring which ran successfully on Miraculix. Looking through the actual command line after running "./lm_create.pl trans_parsed" shows the following: "sphinx_lm _convert: error while loading shared libraries: libsphinxbase.so.1: cannot open shared object file: No such file or directory" This needs to be investigated and fixed. From Camden C Marble in Systems Group: "Miraculix seems to have been configured for last year's modeling group to do some special test. No idea what the test was. But miraculix uses it's own local tools instead or Caesars tools, thus; the success or even output of your Trains, LMs or Decodes could be different from the baseline, if they output at all."

We keep spinning our wheels running 300hr trains that end in a core dump due to a situational but non-persistent error.
 * Concerns:

28 FEB 2018
Use the 2017 Spring winning team's 300hr TDS on Unseen Data and try to match their WER %.
 * Task:

I think that has been accomplished. Used an 8000 senone count in the sphinx_train.cfg and for the decode and it ran to completion (0304/003).
 * Results:

Run another 300hr Unseen Data Decode and Scoring (0304/004) using the default genTrains.pl and the new parseLMTrains.pl that the Data Group came up with. Also run a simple 5hr Train with LDA on Asterix as we have not had one completed yet.
 * Plan:

None at this time.
 * Concerns:

3 MAR 2018
1. Using the 300hr Data and Language Model from 0304/001, create a New Language Model on 0304/004 by copying over the contents of /001 to /004, removing the LM directory, recreating the LM directory and its contents but with the parseLMTrans_2018.pl from the Data Group.
 * Task:

2. Taking the 300hr Data and New Language Model of 0304/004, Decode and Score in 0304/006.

Had to rename everything labeled "001" to "004" in applicable directories in /004 so the scripts in /006 would work on Unseen Data utilizing the steps shown here:
 * Results:

copy over the contents of the folder you are taking the data from to another directory/sub-experiment, delete the LM, then...

mkdir LM

cd LM

cp -i /mnt/main/corpus/switchboard/300hr/train/trans/train.trans trans_unedited

parseLMTrans_2018.pl trans_unedited trans_parsed

cp -i /mnt/main/scripts/user/lm_create.pl.

IF YOU LS THE FOLDER, SHOULD SEE: lm_create.pl, trans_parsed, trans_unedited

./lm_create.pl trans_parsed

IN CAESAR, DO A addExp.pl -s TO ADD THE DIRECTORY YOU WILL PERFORM THE DECODE AND SCORING IF YOU HAVEN'T ALREADY.

cd /mnt/main/Exp/your_main_exp#/your_sub_exp# WHERE YOU WILL DO THE DECODE AND SCORING NOT WHERE THE TRAIN & LM WAS DONE!!!

makeTest.pl -t switchboard/300hr 0304/004 0304/006

genFeats.pl -d

cd to /006/etc folder

nohup run_decode.pl 0304/004 0304/006 8000 &

parseDecode.pl decode.log hyp.trans

cd to /004/etc folder

cp 004_train.trans /mnt/main/Exp/0304/006/etc TO COPY THE TRAIN'S .trans FILE TO NEW DIRECTORY

cd to /006/etc folder

sclite -r 004_train.trans -h hyp.trans -i swb >> scoring.log

IN ORDER TO USE THE SAME DATA AS 0304/001 BUT A DIFFERENT LM, COPY THE CONTENTS OF THE DIRECTORY USING cp -R 001/* 004. THEN DELETE THE LM IN 004, RECREATE THE LM, CD INTO IT, AND FOLLOW THE STEPS DOWN FROM "cp -i.... trans_unedited" ONWARD. EXCEPT BEFORE "makeTest.pl....", MUST RENAME ANYTHING IN /004 USING THE mv COMMAND THAT IS 001 TO 004 OR THE SCRIPTS USED IN /006 FOR THE DECODE AND SCORING WON'T WORK. THIS INCLUDES etc, model_parameters, model_architecture, trees AND bwaccumdir DIRECTORIES!!!!!!


 * Plan:

Attempt LDA tomorrow as the process has yet to be completed.

Look at 0301/011 experiment by Greg Tinkham, Spring 2017.

The two experiments 0304/003 and 0304/005 using the 300hr Decode and Scoring of Unseen Data in 0304/001 had error rate averages of 33.3% and 24.5%, respectively using the historical parseLMTrans.pl while the 0304/006 using the same Data as /003 and /005 which originates in /001 returns an error rate average of 45.2%.
 * Concerns:

4 MAR 2018
1. Determine if the sphinx_train.cfg file from Greg Tinkham's 0301/011 is any different than the current one in 0304/001.
 * Task:

2. Using the sphinx_train.cfg parameters from 0301/011, Decode and Score the Unseen Data in 0304/001.
 * Copy contents of 0304/001 to 0304/007.
 * Rename everything 001 to 007.
 * Change the config parameters as stated below in Results 1.

3. As the LM already exists from 0304/001, create /008 to Decode and Score the Data.

1. Yes, it was:
 * Results:
 * Line 101, "$CFG_STATESPERHMM = 5;" instead of "3".
 * Line 102, "$CFG_SKIPSTATE = 'yes';" instead of "no".
 * Line 107, "$CFG_FINAL_NUM_DENSITIES = 32;" instead of "8". AMMENDED 5 MAR 2018: Determimed that 0301/006 which is original data, this value was 16!! So 32 was used in 0301/011 only!!
 * Line 120, "$CFG_N_TIED_STATES = 8000;" instead of "1000".
 * Line 156, "$CFG_LDA_MLLT = 'yes';" instead of "no".
 * Line 158, "$CFG_LDA_DIMENSION = 32;" instead of "29".

LDA was used in this experiment.

2. Changes made IAW Task 2.

3. COMPLETED in 0304/008. So my first attempt with LDA on a 300hr Train was successfull and resulted in a 33.3% average error rate with the parameters used in the sphinx_train.cfg file of 0301/011 which had a 28.4% average error rate.

Going to run another 300hr TDS on Unseen Data with LDA but using new data just for this experiment and also use the configuration parameters of 0301/011. Started the 300hr Train and LM in 0304/009. Will continue into tomorrow with Decode and Scoring in 0304/010.
 * Plan:

None at this time. Will consider using new parseLMTrans_2018.pl with either /007 and/or /009.
 * Concerns:

5 MAR 2018
Decided to leave the Data in 0304/009 alone with 0301/001's sphinx_train.cfg parameters and WITHOUT a Language Model so it can be copied to /010 for use with the default Language Model for that data BUILT in /010 and Decoded and Scored in /011. (This prevents the necessity of removing the LM from /009 for other LM configuration uses). Also, will add /012 for data from /009 to be used with LM created there in /012 using parseLMTrans_2018.pl that the Data Group developed. This will be Decoded and Scored in /013.
 * Task:


 * Results:
 * First result was not promising. After creating the LM in /010, attempted Decode and Scoring in /011. Twice this resulted in a CORE DUMP with fatal error of missing mdef file which I have seen multiple times before. I looked back at the model_parameters and model_architecture in /009 to determine the issue and found the following:

/009 model_parameters contained:
 * 009.ci_lda
 * 009.ci_lda_flatinitial
 * 009.ci_mllt_flatinitial

/009 model_architecture contained:
 * 009.ci.mdef
 * 009.phonelist
 * 009.topology

It appears that the "missing mdef file" issue is related to model_architecture because here is what is in those two model folders (parameters & architecture) in a successful TDS on Unseen Data using LDA:

/0301/011 model_parameters contained:
 * 011.ci_lda
 * 011.ci_lda_flatinitial
 * 011.ci_mllt
 * 011.ci_mllt_flatinitial
 * 011.lda
 * 011.mllt
 * 011.mllt_cd_cont_8000
 * 011.mllt_cd_cont_8000_1
 * 011.mllt_cd_cont_8000_16
 * 011.mllt_cd_cont_8000_2
 * 011.mllt_cd_cont_8000_4
 * 011.mllt_cd_cont_8000_8
 * 011.mllt_cd_cont_initial
 * 011.mllt_cd_cont_untied
 * 011.mllt_ci_cont
 * 011.mllt_ci_cont_flatinitial

/0301/011 model_architecture contained:
 * 011.ci.mdef
 * 011.alltriphones.mdef
 * 011.mllt.8000.mdef
 * 011.phonelist
 * 011.topology
 * 011.tree_questions
 * 011.untied.mdef

Now, the /0301/011/model_architecture/011.mllt.8000.mdef is labeled differently in /0304/001/model_architecture as 011.8000.mdef. I do not know if this makes any difference but doesn't seem to be the source of the fatal error of missing the mdef file when running the sclite command.

However, the model folders for 0304/007 are different as well, and that resulted in a successful TDS in /008. The 8000 in place of ci in 007.8000.mdef didn't seem to have an impact. Recall that the files below are copied from 0304/001 and renamed.

/0304/007 model_parameters contained:
 * 007.ci_cont
 * 007.ci_cont_flatinitial
 * 007.cd_cont_untied
 * 007.cd_cont_initial
 * 007.cd_cont_8000
 * 007.cd_cont_8000_1
 * 007.cd_cont_8000_2
 * 007.cd_cont_8000_4
 * 007.cd_cont_8000_8

0304/007 model_architecture contained:
 * 007.8000.mdef
 * 007.alltriphones.mdef
 * 007.ci.mdef
 * 007.phonelist
 * 007.topology
 * 007.tree_questions
 * 007.untied.mdef

The folder structure for model_parameters and model_architecture is created in the first command run, makeTrain.pl, however those folders are not populated until the nohup scripts_pl/RunAll.pl & command is run. Since that takes over 12 hours for a 300hr train, this creates a time management issue.
 * Plan:

I have created 0304/014 to recreate /009. Once the sphinx_train.cfg file is created, I will change the parameters to match 0301/011 (now the data it was based on, 0301/006 with a Final Density count of 16, not 32 which was used in 0301/011) and proceed. Tomorrow morning I will verify the contents of the model_parameters and model_architecture folders and go from there by creating the LM in /014 instead of using another sub-experiment to do that.

That the damn model folders will not populate as I believe they should. This work over the last few days was predicated on 0301/011. Just determined that the data it is based on, 0301/006 used a density count of 16 instead of 32 in line 107 of the sphinx_train.cfg file. This means the value was changed to 32 from 16 during the process used in 0301/011. Also, it may not be possible to keep the LM separate from the Train Data or I am doing something wrong in trying to do so.
 * Concerns:

6 MAR 2018
Check on completion of Train in 0304/014. Take note of the contents of the model_architecture and model_parameters folders and display folder item names in results below. Create default Language Model in /014. Take note of any file name additions in model_architecture and model_parameters folders. Decode and Score with Final Density changed from 16 to 32 in sphinx_train.cfg file in /015. Determine if same results from /009 and /010 exist with model folders contents. If so, troubleshoot further. Check Greg Tinkham's personal logs for about the time of 12 April 2017.
 * Task:

FOLLOW THE NUMBERED ENTRIES BELOW FOR STEPS TAKEN IN RESULTS AND PLAN!

1. The completed train in 0304/014 yielded the following files in the following folders:
 * Results:

/0304/014/model_parameters:
 * 014.ci_lda
 * 014.ci_lda_flatinitial
 * 014.ci_mllt_flatinitial

/0304/014/model_arch:
 * 014.ci.mdef
 * 014.phonelist
 * 014.topology

3. There were no additional files added to the above directories after creating the Language Model.

5. The Decode and Scoring of 0304/014 in 0304/015 did result in a CORE DUMP as I suspected it would due to "FATAL_ERROR: "mdef.c", line 680: No mdef-file" in the decode log.

2. Create the Language Model and check for any additions to those folders.
 * Plan:

4. Run the Decode and Scoring, but I suspect I already know the result.

6. Have to determine WHY the directory structure in subsequent experiments is different from 0304/001 (which was copied into /007).

7. Decided to run another 300hr Train and LM with 0301/011 configuration for the sphinx train but WITHOUT LDA to see what happens. I am just happy to be here.
 * Concerns:

8 MAR 2018
Pulling back from the plan of Tuesday, 6 March 2018 which was established before class that day. It was determined over the following 24 hours that we have been doing the decode completely wrong. Even though I have watched the decode process run without error in the terminal window, then return a command prompt, this does not mean that the decode process is complete. This would explain why I saw the decode process on a 300hr Data Train continue AFTER I completed a Score on Asterix the previous weekend - twice.
 * Task:

Now I have to change the procedures based on the commands provided through Professor Jonas, Rose and Dan Beitel over the last 48 hours. This will be accomplished using Data Train and LM without LDA in 0304/016 and Decoded and Scored in 0304/021.

Yet another "FATAL_ERROR: "mdef.c", line 680: No mdef-file". Working with Brian Barnes, determined that line 7 in the decode file states "-hmm /mnt/main/Exp/0304/016/model_parameters/016.cd_cont_8000" will not be found, nor will the "mdef" file that is being sought within it, as the model_parameters sub-directory files in /016 are all listed with _1000 at the end of them and therefore the mdef file could not be found. Assuming _1000 is the senone count, they should read 8000 instead of 1000. There also should be in sequence of the number of final densities (i.e. _1, _2, _4, _8, _16, _32) appended at the end of each of those files. In /016 they only go to _8 when the number of final densities in the sphinx train config file is 32. Not understanding why this is happening.
 * Results:

Running a 30hr Train to determine if the contents of the model_parameters folder will be what we suspect they should be. That the plan doesn't work requiring further troubleshooting that is a step backwards in time management.
 * Plan:
 * Concerns:

9 MAR 2018
My test 30hr train using the sphinx_train.cfg parameters from 0301/011 minus LDA was successful. The model_parameters folder shows .cd_cont files with _8000, _8000_1, _8000_2, _8000_4, _8000_8, _8000_16, _8000_32 as I was expecting to see, but had not in previous attempts. Now I can move forward with the language model, then decode and score in a different directory.
 * Task:

Created the Language Model then ran the decode in another directory while running "tail -f decode.log" in another terminal window to watch to completion. The model_parameters folder in the Decode and Scoring sub-experiment populated with the folders I was hoping to see: .cd_cont_8000, _8000_1, _8000_2, _8000_4, _8000_8, _8000_16, _8000_32, _initial, _untied, _flatinitial and just cd_cont. And .cd_cont_8000 contains the mdef file that has not been seen previously causing the errors resulting in so many CORE DUMPs.
 * Results:

The decode on the 30hr Trained Data took 10 hours and 20 minutes. Performing a "grep FWDVIT decode.log | wc" returned "3992  58498   367203". So the word count for the decode log was 3,992. This is in line with what Rose and I researched for a number of the 30hr trains which have already take place. Now I can proceed forward with transforming the decode.log file to hyp.trans.

Decode and Scoring on that 30hr Train using the Rebels Team sphinx_train.cfg parameters minus LDA completed with a Sum/Avg WC of 3992 and a WER of 15.8%, which being below 20% was expected on a 30hr Train.

Tomorrow, use the 300hr Train Data from 0301/011 (Rebels Team 28.4% WER result which used the data in 0301/006) and Decode and Score to try to replicate results. That it won't work, which would be another huge waste of time as the Decode alone on a 300hr Train should take over 100 hours to complete.
 * Plan:
 * Concerns:

10 MAR 2018
Run sub-experiment 0304/019. This is a replication of last year's Rebels Team 0301/011 Decode and Scoring on 0301/006 Data which uses LDA and different sphinx_train.cfg parameters than the default parameters. The 0301/006 Data is copied into 0304/017 and the LM was created in 0304/019 with default parameters.
 * Task:

Considering the Decode process (finally correctly performed) on a 30hr took 10 hours and 20 minutes, I expect this Decode on a 300hr to take over 100 hours. But that is just conjecture.
 * Results:

Once 0304/019 is completed successfully, utilize 0304/017 in another Decode and Scoring sub-experiment with the parseLMTrans_2018 that this year's Data Group has come up with. I am just happy to be here.
 * Plan:
 * Concerns:

11 MAR 2018
The 300hr Decode and Scoring that I started yesterday completed during the night. I concluded this by seeing that the sphinx3_decode process was no longer running on Asterix and that when I perform a "grep FWDVIT decode.log | wc" in the 0304/019/etc folder, it returns 4034, which is what I expect for a 300hr experiment. However, when I run the final step, "sclite -r...", it errors out with something I have not seen before and shown in Results below...
 * Task:

Determine cause of "Error: Not enough Reference Files loaded
 * Results:

Missing:

(sw2001a-ms-98-a-0048)" and on and on and on with more reference files.

According the the following, https://foss.unh.edu/projects/index.php/Speech:Run_Decode_Unseen_Data, at the bottom it states to perform a "% uniq hyp.trans >> hyp.trans.uniq" to remove all the redundant lines in the hyp.trans file in the etc directory of the decode sub-experiment. This yielded the following: "hyp.trans.uniq: Too many arguments." The wiki referenced above does not address this, but states to restart the SClite while using the newly created hyp.trans.uniq file. This resulted in a CORE DUMP (of course). The wiki referenced above states that if you get the same error again, repeat the same process EXCEPT doing so with the _train.trans file and to specify that file when running the SClite process again. Once again, this yielded the following: "017_train.trans.uniq: Too many arguments.". Continuing on with the SClite process again, as the wiki states to do, I left the hyp.trans.uniq in the command line and this also resulted in a CORE DUMP.

I reached out to Brian Barnes to see if there is something he can think of. He has determined that the "wav" directory in ANY of the Unseen Data Decode and Scorings we have performed has been empty each and every time. This is preventing a proper Decode and Scoring from happening and would add to the list of reasons why the SClite would not run as referenced in the Speech:Run Decode Unseen Data wiki page.
 * Plan:

Talking with the Experiment Group, it would appear that the "makeTest.pl" script does indeed create the "wav" directory, but it does not copy anything to it as it does "etc" or "model_parameters" from the train experiment. This will need to be addressed, either by altering the "makeTest.pl" script or by adding another manual copying step to the process prior to decoding in the test sub-experiment.

I am just happy to be here.
 * Concerns:

12 MAR 2018
See if 0304/023 Decode completed. If so, continue with parsing the decode. Also, review the directory contents of previous iterations' 300hr Decodes and compare them to ours to see what is different. Why is model_architecture not needed in the Decode & Scoring sub-experiment? Or is it? Should the wav directory contents be copied over from the original Train experiment?
 * Task:

0304/023 Decode FAILED.
 * Results:

The answer to the last question is probably not. Copying over the wav directory contents had no effect on the sclite process as I once again received "Error: Not enough Reference files loaded

Missing: " and then a LOT of .sph files.

But this begs the question - why are they present in the 0301/011 wav directory? Were they manually copied in there or was there a different version of the makeTest.pl script that placed them in there? Did their presence in the wav file make any difference when SClite was run by the previous semester? Because it made no difference here unless I am missing another step/procedure/command.

Run a simple 5hr Data Train, perform an Unseen Decode and Scoring, no LDA, default Language model, default sphinx_train.cfg.
 * Plan:


 * Concerns:

13 MAR 2018
The 5hr Train Data, Unseen Decode and Scoring completed (0304/026, /027) with a 4172 word count (expected) and a WER of 18.0% using default Language Model (created in the Decode and Scoring in /027), the sphinx_train.cfg from 0301/006 but without LDA and a 1000 senone count for a 5hr train instead of 8000 for a 300hr.
 * Task:

Will now perform a 30hr Train Data, then Unseen Decode and Scoring using default Language Model and sphinx_train.cfg from 0301/006 but without LDA and a 5000 senone count for a 30hr train instead of 8000 for a 300 hr.

Creating the 30hr Train Data in 0304/028 with the sphinx_train.cfg parameters from 0301/006 less LDA and a 5000 senone count instead of 8000 resulted in the expected sub-directories in model_parameters and the expected files in model_architecture. I decided to create the LM in the Decode and Scoring (/029) so that the Data Train in /028 can be used with dev.trans and eval.trans in other Decode and Scoring sub-experiments.
 * Results:

Tomorrow I will see if the Decode in 0304/029 completed then Score it.
 * Plan:


 * Concerns:

14 MAR 2018
The 30hr Train Data, Unseen Decode and Scoring completed (0304/028, /029) with a 3992 word count (expected) and a WER of 18.8% using default Language Model (created in the Decode and Scoring in /029), the sphinx_train.cfg from 0301/006 but without LDA and a 5000 senone count for a 30hr train instead of 8000 for a 300hr.
 * Task:

Will now perform a 300hr Train Data, then Unseen Decode and Scoring using default Language Model and sphinx_train.cfg from 0301/006 but without LDA and 8000 senone count.

Creating the 300hr Train Data in 0304/030 with the sphinx_train.cfg parameters from 0301/006 less LDA and a 8000 senone count.
 * Results:

Tomorrow I will see if the Decode in 0304/030 completed then Score it.
 * Plan:


 * Concerns:

15 MAR 2018
I was going to Decode and Score 0304/030 but Professor Jonas took a look at our sub-experiments in 0304 and determined that the data was still decoded as seen and not as unseen. He determined this by running a "% diff 027_decode.fileids ../../026/etc/026_train.fileids" and it returned nothing. This indicates that the same data was used to both decode and train making it seen data instead of unseen. Had it returned different file ids then it would have included some unseen data and had all of the file ids been different, then it would be completely unseen data.
 * Task:

This means that every single so called "Unseen" Experiment I have run in 0304 is actually "Seen".
 * Results:

So I have to go back and change the title and description of every single one of those in the wiki.
 * Plan:


 * Concerns:

19 MAR 2018
Run another Decode and Scoring of 0304/026 5hr Train Data with what Brian and I believe are the correct steps and commands. The decode process will run through the night.
 * Task:

Decoded overnight successfully (the process ran without error, anyways). See log of 20 March 2018 for more.
 * Results:

So I have to go back and change the title and description of every single one of those in the wiki.
 * Plan:


 * Concerns:

20 MAR 2018
Change each 0304 sub-experiment that is labeled as "Unseen" to "Seen" in the wiki. Change the Speech:Run Decode Unseen Data wiki to conform to Professor Jonas's changes in his email of 1:48 pm 16 March 2018. Prepare to score then score 0304/032.
 * Task:

Changed each 0304 sub-experiment that is labeled as "Unseen to "Seen" in the wiki. I also placed "(ORIGINALLY BELIEVED TO BE UNSEEN DECODE)" at the end of "Purpose" in each applicable sub-experiment. Changed the Speech:Run Decode Unseen Data wiki to conform to Professor Jonas's changes in his email of 1:48 pm 16 March 2018. Prepared to score then scored 0304/032 which resulted in missing file error when running the sclite process. Worked with Brian to determine what was missing command wise - genTrans.pl and running sclite with the wrong file to be scored against the hypothesis file. Once we fixed that, the sclite seems to run fine but returned an atrociously bad 62% word error rate for a 5hr corpus.
 * Results:


 * Plan:

Re-creating 300hr baseline from 2017 this yields 28.4% WER on test-on-train (Steve) just copy (recursively and run 0301/011) this yields 41.3% WER on unseen (Brian/Hannah) need to find which one it was may just be 0301/011 with test-on-dev if not then need to figure out which one For these, be sure to coordinate with Data Group since the are fixing parseNLTrans.pl           you do not want to use the fixed version for these two experiments since all you are trying to do is recreate them. Steve, you can just cp -r yours and change the paths and re-run it, no need to use any scripts from /mnt/main/scripts/user Brian/Hanna, if you cannot find which experiment yielded 41.3% then also cp -r (a second time into a separate experiment) and then you'll need to use tools to create test-on-dev...here you need to be wary which parseNLTrans.pl script you are using.

Once these are done then recreate my experiments for 300hr test-on-train experiment 0305/012: keep [], - for LM, trans, dict (Steve) this should be a rerun of Steve's experiment above but with fixes to parseLMTrans.pl that the Data Group is working on. experiment 0305/011: strip [], keep - for LM, trans, dict (Hannah) experiment 0305/013: strip [], - for LM, trans, dict (Brian)


 * Concerns:

24 MAR 2018
Rename all files with "011" representing 0301/011 to "033" representing 0304/033 in 0304/033 to recreate 2017 Rebels Team experiment. Rerun experiment.
 * Task:

I renamed all of the files and attempted a "makeTrain.pl -t switchboard 300hr/test". It asked if I wanted to overwrite etc/033.filler and I told it no. It also asked if I wanted to overwrite etc/./genPhones.csh and I said no. I changed the sphinx_train.cfg file to match 0301/011's. Since this was a train (or attempted retrain) using LDA, I obtained permission from Team Guardians to use Miraculix. I ran "nohup scripts_pl/RunAll.pl" which may be wrong, but since the sphinx_train.cfg file had been altered by me, I believe I had to run this perl script in order to utilize the changes to the config file. The train seemed to run to completion (seemed because it didn't error out). Since this was to be a retrain and recreation, I used the Language Model (LM) that was in the /011 folder without changing it. I then proceeded to decode as everything between creating the language model and decoding was done from copying the original /011. This immediately errored out with "usr/local/bin/sphinx3_decode: error while loading shared libraries: libs3decoder.so.0: cannot open shared object file: No such file or directory". To test why I received this error, I tried "makeTest.pl -d..." but attempting to decode again received the same error.
 * Results:

Per Professor Jonas's email, I can conduct a train and try to replicate the Rebels Team 28.4% WER that way, so this is the direction I will go in.
 * Plan:


 * Concerns:

25 MAR 2018
Run a new train or identify a train modified by Rebels Team sphinx_train.cfg file parameters including LDA, use Rebels Team LM, decode and score.
 * Task:

I had already copied Rebels Team Train Data 0301/006 to 0304/017. The 0301/006 is the basis for Rebels Team 0301/011 with a 28.4% WER. So I checked the sphinx_train.cfg and found the expected altered parameters that lead to the 28.4% WER. I created 0304/036 to run the decode in and copied the LM from 0301/011 to there as well for two reasons: 1. I am trying to keep as much to the original experiment as possible and 2. Rose is telling me via Discord that current parseLMTrans.pl has been modified this semester so that one would not be conducive to replicating any results from before that perl script was altered. Then I ran "makeTest.pl -d...", then genFeats, then ran the decode when I received the same error as yesterday: "usr/local/bin/sphinx3_decode: error while loading shared libraries: libs3decoder.so.0: cannot open shared object file: No such file or directory".
 * Results:

At this point I decided to see if I could avoid the error by creating the LM in /017 which didn't work. When I tried running "lm_create.pl" I received the following error: "sphinx_lm_convert: error while loading shared libraries: libsphinxbase.so.1: cannot open shared object file: No such file or directory".

As it turns out in consulting with Camden in Systems Group this turned out to be a Miraculix shared library issue. I went back into /017, deleted the LM, went into /036, deleted the LM there and copied over the LM from 0301/011 and ran the decode on Asterix.

See if the decode process completes then score it.
 * Plan:


 * Concerns:

26 MAR 2018
See if the decode in 0304/036 has completed. If so, score.
 * Task:

Completed with a WER of 37.6% which is nowhere near Rebels Team 28.4%. The file directory structure of 0301/011 is baffling to me since it in no way matches what is produced whenever I run a Decode and Scoring with what I think are the correct instructions.
 * Results:

It just dawned on me at 10:36 am today that I did this wrong by using -d instead of what I should have used, -t (for test on train which is what Professor Jonas stated to do in his email instructions. Damn. Damn. Damn. Starting over again! with makeTest.pl -t instead of -d.
 * Plan:


 * Concerns:

27 MAR 2018
Determine way forward for replicating last year's 28.4% WER.
 * Task:

Determined with the professor that trying to determine all of the file differences between 0301/006 and 0301/011 would not be necessary because that team's logs (or the individual who ran that particular experiment log) states that a final density of 16 was used during the train in /006. Then the final density was increased in /011 to 32. The file structure of model_parameters reflects this and would only be possible if the train from /006 was retrained in /011.
 * Results:

Determine results of using -t instead of -d in recreating the results of last year's 28.4% WER. Also, retrain the train that is copied over from 0301/011 into 0304.
 * Plan:

That we will not be able to replicate the 28.4% WER, but Professor Jonas stated in class today that at some point if we cannot then we will just have to cut our losses and move forward.
 * Concerns:

31 MAR 2018
Determine which drone servers can run LDA.
 * Task:

Observe the directory structure/file differences between trained data without LDA and trained data with LDA.
 * Results:

A train experiment without LDA should have a model_parameter structure that looks something like this: (### represents the subexperiment number)


 * ### .cd_cont_1000 (where 1000 is the senone count stated in the sphinx_train.cfg file when the train is run)


 * ### .cd_cont_1000_1 (where _1 is the final density count factored by 2 to the final density count)


 * ### .cd_cont_1000_2


 * ### .cd_cont_1000_4


 * ### .cd_cont_1000_8


 * ### .cd_cont_1000_16


 * ### .cd_cont_1000_32


 * ### .cd_cont_initial


 * ### .cd_cont_untied


 * ### .ci_cont


 * ### .ci_cont_flatinitial

A train experiment with LDA should have a model_parameter structure that looks something like this: (### represents the subexperiment number)


 * ### .ci_lda


 * ### .ci_lda_flatinitial


 * ### .ci_mllt


 * ### .ci_mllt_flatinitial


 * ### .lda


 * ### .mllt


 * ### .mllt_cd_cont_8000


 * ### .mllt_cd_cont_8000_1


 * ### .mllt_cd_cont_8000_2


 * ### .mllt_cd_cont_8000_4


 * ### .mllt_cd_cont_8000_8


 * ### .mllt_cd_cont_8000_16


 * ### .mllt_cd_cont_8000_32


 * ### .mllt_cd_cont_initial


 * ### .mllt_cd_cont_untied


 * ### .mllt_ci_cont


 * ### .mllt_ci_cont_flatinitial

Run a train on a 5hr corpus on each drone server to determine which ones it will work on and which will not. Then inform Systems Group to determine what the differences are between the servers.
 * Plan:

None at this time.
 * Concerns:

1 APR 2018
Determine which drone servers can run LDA.
 * Task:

Conducted the train process after activating LDA via the sphinx_train.cfg file on each server then determined the directory structure within model_parameters to see if their are "mllt" labeled files or not.
 * Results:

Performed on Asterix: during "nohup run_scripts/RunAll.pl &", MODULE: 06 Train MLLT transformation, Phase 2: Flat initialize, "FATAL_ERROR: "main.c", line 98: Failed to read LDA matrix....Something failed: (/mnt/main/Exp/0304/100/scripts_pl/06.mllt_train/slave_mllt.pl)"

Performed on Obelix: during "nohup run_scripts/RunAll.pl &", MODULE: 06 Train MLLT transformation, Phase 2: Flat initialize, "FATAL_ERROR: "main.c", line 98: Failed to read LDA matrix....Something failed: (/mnt/main/Exp/0304/100/scripts_pl/06.mllt_train/slave_mllt.pl)"

Performed on Traubadix (clone of Miraculix): train with LDA runs, but since it is a clone of Miraculix, it cannot run or create a Language Model due to shared library issue: when running "lm_create.pl trans_parsed", the following error occurs - "sphinx_lm_convert: error while loading shared libraries: libsphinxbase.so.1: cannot open shared object file: No such file or directory"

Also cannot run decode due to "/usr/local/bin/sphinx3_decode: error while loading shared libraries: libs3decoder.so.0: cannot open shared object file: No such file or directory". This occurs with either run_decode.pl or run_decode_lda.pl.

Inform team members of which ones worked and which ones did not.
 * Plan:

I am just happy to be here.
 * Concerns:

2 APR 2018

 * Task:


 * Results:


 * Plan:


 * Concerns:

Stomach flu - unable to do schoolwork.

3 APR 2018

 * Task:


 * Results:


 * Plan:


 * Concerns:

Stomach flu - unable to do schoolwork.

5 APR 2018

 * Task:


 * Results:


 * Plan:


 * Concerns:

Stomach flu - unable to do schoolwork.

7 APR 2018

 * Task:


 * Results:


 * Plan:


 * Concerns:

Stomach flu - unable to do schoolwork.

9 APR 2018

 * Task:


 * Results:


 * Plan:


 * Concerns:

Stomach flu - unable to do schoolwork.

10 APR 2018
Going to move forward with no trade of team members between Guardians and Avengers. I am told the 28.4% WER from 0301/011 was replicated by Brian and Hannah. Need to determine the next experiment to recreate.
 * Task:


 * Results:

Determine the next experiment to recreate.
 * Plan:


 * Concerns:

12 APR 2018
Pulled into Professor Jonas's office tonight after leaving Networking Technologies class. Spent 6 hours with him determining that the 28.4% WER from 0301/011 was NOT replicated as LDA was not fully utilized due to Python not working as it is supposed to. Professor Jonas spent hours determining which python versions were running where and also determined that even if LDA is switched from 'no' to 'yes' in the sphinx_train.cfg file, that the train process WILL CONTINUE when Python does not have the necessary libraries (scipy and numpy) WITHOUT any error causing the train to stop. Professor Jonas was able to eventually copy Python 2.7 as a hotfix to Caesar and hopefully this will allow us to run LDA on any drone server.
 * Task:

5hr Train that Professor Jonas was running was creating the correct subdirectories in model_parameters (with mllt in the subdirectory names where applicable) as of 1:15 am.
 * Results:

Verify LDA availability on drone servers tomorrow.
 * Plan:


 * Concerns:

13 APR 2018
Determine LDA use on drone servers Asterix, Obelix and Traubadix.
 * Task:

Asterix: during "nohup run_scripts/RunAll.pl &", MODULE: 06 Train MLLT transformation, Phase 2: Flat initialize, "FATAL_ERROR: "main.c", line 98: Failed to read LDA matrix....Something failed: (/mnt/main/Exp/0310/001/scripts_pl/06.mllt_train/slave_mllt.pl)"
 * Results:

Apparently this is due to Asterix being a 32 bit OS system as python2.7 would not run, claiming wrong architecture per Professor Jonas in 0304/042.

Obelix: during "nohup run_scripts/RunAll.pl &", MODULE: 06 Train MLLT transformation, Phase 2: Flat initialize, "FATAL_ERROR: "main.c", line 98: Failed to read LDA matrix....Something failed: (/mnt/main/Exp/0310/002/scripts_pl/06.mllt_train/slave_mllt.pl)"

Apparently this is due to Obelix being a 32 bit OS system as python2.7 would not run, claiming wrong architecture per Professor Jonas in 0304/042.

Traubadix: Train ran. However, LDA not fully utilized in train process since the following were created in model_parameters:

003.cd_cont_1000 003.cd_cont_1000_1 003.cd_cont_1000_2 003.cd_cont_1000_4 003.cd_cont_1000_8

which should all read 003.mllt_cd_cont_1000, etc.

A few LDA subdirectories were created:

003.ci_lda 003.ci_lda_flatinitial 003.ci_mllt 003.ci_mllt_flatinitial 003.lda (as a file)

So overall another failure.


 * Plan:

HUGE email from Professor Jonas addressed the drone server LDA issues, mostly with Systems Group having specific tasking on this matter. With the change to the decode process, that is my current focus to make that work and disseminate how to do so in order to move forward. Otherwise, I cannot recreate the 28.4% WER from 0301/011 on Caesar as I have been directed.

Wasting so much time spinning our wheels on IT stuff that should have been addressed over two months ago.
 * Concerns:

14 APR 2018
Work the -lda and -ldadim flag issue in the decode process.
 * Task:

Now that we are manually performing a decode with individual command line entries in lieu of the decode.pl script, we can manually add the -lda and -ldadim flags in order to decode with LDA. However, whenever we tried it, it would error and all attempts at trying to identify the path, if there is a required path, failed:
 * Results:

/usr/local/bin/sphinx3_decode \
 * -lda have tried the following here:
 * /mnt/main/Exp/0310/004/model_parameters/004.lda
 * /mnt/main/Exp/0310/004/model_parameters/004.ci_lda
 * /mnt/main/Exp/0310/004/model_parameters/004.ci_lda_flatinitial
 * /mnt/main/Exp/0310/004/model_parameters/004.mllt
 * /mnt/main/Exp/0310/004/model_parameters/004.ci_mllt
 * /mnt/main/Exp/0310/004/model_parameters/004.ci_mllt_flatinitial
 * /mnt/main/Exp/0310/004/model_parameters/004.mllt_cd_cont_initial
 * /mnt/main/Exp/0310/004/model_parameters/004.mllt_cd_cont_untied
 * /mnt/main/Exp/0310/004/model_parameters/004.mllt_cd_cont
 * /mnt/main/Exp/0310/004/model_parameters/004.mllt_cd_cont_flatinitial
 * /mnt/main/Exp/0310/004/model_parameters/004.mllt_cd_cont_1000

ERRORS OUT WITH THE FOLLOWING= "SYSTEM_ERROR: "lda.c", line 76: fopen(, rb) failed; No such file or directory" follwed by "FATAL_ERROR: "kbcore.c", line 268: LDA initialization failed."

DOING A "cd /" THEN "find . -name "lda.c"" SHOWED lda.c IN A NUMBER OF feat SUBDIRECTORIES FOUND UNDER root. EXAMINING 3 OF THOSE lda.c FILES SHOWS THAT WITHIN AN IF STATEMENT AT LINE 75 IT IS TRYING TO OPEN AN ldafile, "rb" AND THAT IF IT ISN'T FOUND, IT NEEDS TO FAIL. \
 * -ldadim \
 * -hmm /mnt/main/Exp/0310/004/model_parameters/004.mllt_cd_cont_1000 \
 * -lm /mnt/main/Exp/0310/004/LM/tmp.arpa \
 * -dict /mnt/main/Exp/0310/004/etc/004.dic \
 * -fdict /mnt/main/Exp/0310/004/etc/004.filler \
 * -ctl /mnt/main/Exp/0310/004/etc/004_decode.fileids \
 * -cepdir /mnt/main/Exp/0310/004/feat \
 * -cepext .mfc >& decode.log &

Since we were not able to make -lda work, we didn't attempt much with -ldadim.

We were directed by Professor Jonas via email to research CMU Sphinx page decode process for guidance on this issue.
 * Plan:


 * Concerns:

16 APR 2018
Attempting to recreate the Rebels Team results of 28.4% WER on Seen Decode and Scoring of a 300hr train.
 * Task:

Two points of approach: A. Recreate the experiment by using copied contents of 0301/011 in 0304/033 and retrain. B. Recreate the experiment by using copied contents of 0301/006 in 0304/017 and retrain.

A. 0304/033 is a copy of 0301/011, which is the Retrained Data with Seen Decode and Scoring that obtained a 28.4% WER. I am attempting to retrain here, which I realize may be a mistake (on my third attempt with RunAll.pl), in order to recreate the 28.4% WER. I think this may be a mistake because this would be a retrain of retrained data. But we will see.
 * Results:

Commands used to rerun 0301/011 in 0304/033:

makeTrain.pl -t switchboard 300hr/test DOING THIS FOUND THAT EACH .wav SYMBOLIC LINK ALREADY EXISTED AND COMPLETION PERCENTAGE WAS APPROXIMATELY 1% PER MINUTE

THIS ASKED IF I WANTED TO OVERWRITE etc/033.filler, I SAID NO. ALSO ASKED IF I WANTED TO OVERWRITE etc/./genPhones.csh, I SAID NO. COMPLETED PREPARATION. I CHANGED THE sphinx_train.cfg FILE TO MATCH 0301/006'S. !!!WHEN DECODING, CHANGE TO 32 FOR line 107!!!

TRIED RUNNING genFeats.pl -t DOING THIS FOUND THAT CREATING THE SYMBOLIC LINK FOR EACH feat/....mfc THAT EACH FILE EXISTS.

RAN nohup scripts_pl/RunAll.pl & WHICH MAY BE WRONG, BUT SINCE THE sphinx_train.cfg FILE HAD TO BE ALTERED, I BELIEVE I HAD TO RUN THIS TO UTILIZE IT.

on second & third attempts, received "Phase 7: TRANSCRIPT - Checking that all the phones in the transcript are in the phonelist, and all phones in the phonelist appear at least once Something failed: (/mnt/main/Exp/0304/033/scripts_pl/00.verify/verify_all.pl)"

So, since I think this is the wrong approach anyway, I will focus on approach B.

B. 0304/017 is a copy of 0301/006, which is the Trained Data used for 0301/011. Then everything labeled /011 has been renamed to /017 except for the contents of model_architecture and model_parameters. I have copied over the LM from 0301/011 as that is where it was created. I think it is possible that last year's Rebels Team copied the contents of 0301/006 into 0301/011 and retrained the data in there where they also created the LM. This is the approach I am taking here. I am leaving the contents of model_architecture and model_parameters with 006 at the beginning where appropriate. I want to see the "scripts_pl/ RunAll.pl &" command create the new subdirectory contents of those two subdirectories with 017 at the beginning (in addition to one more .mllt_cd_cont_8000 with _32 as should occur with what I note in the next paragraph).

I have changed the $CFG_FINAL_NUM_DENSITIES = 16; to 32 in the etc/sphinx_train.cfg file. In /006, during the original train, the number used was 16. It appears (and Greg Tinkham noted in some log) that the number was changed to 32. Question is, was that for the decode process or the retraining? Well, I am attempting it for the retraining, so it will be 32 along with their other changes

PERFORMED ON DRONE SERVER TRAUBADIX.

Since makeTrain.pl modifies the sphinx_train.cfg, then all of the settings that were set when /006 was trained would be reset to default, so this script will not be used.

I am going to alter the sphinx_train final density number from 16 to 32 and do RunAll. (Before doing this I deleted the contents of the model_architecture and model_parameters sub- directories as those are created by the RunAll perl script. I want to see what is created rather than what may or not be replaced.)

When I try running "scripts_pl/RunAll.pl &" I receive "Can't open perl script "/mnt/main/Exp/ 0301/006/scripts_pl/00.verify/verify_all.pl": No such file or directory Something failed: (/mnt/main/Exp/0301/006/scripts_pl/00.verify/verify_all.pl)" This makes sense in regards to location, because /0301.. resides in /Exp/sp17, so I have to fix this. Vi'd into the /mnt/main/Exp/0304/017/scripts_pl/00.verify and I do not readily see where the original location is called for.

Went into sphinx_train.cfg and saw that on line 7, the $CFG_BASE_DIR = "/mnt/main/Exp/0301/006"; which makes sense because when this directory was created, it wasn't under Exp/sp17 yet - it was moved there after the semester was over! So, I went in and manually changed it to /0304/017. Changed line 6 from "006" to "017" as well as that is the $CFG_DB_NAME.

This fixed the error above and is now running.

Continue with decode since LM was copied over from 0301/011 once the RunAll.pl is complete.
 * Plan:

That the RunAll.pl errors out while I am asleep.
 * Concerns:

17 APR 2018
Continue with recreating 0301/011 based on 0301/006 Data Train in 0304/017.
 * Task:

"RunAll.pl" ran through the night (about 10 hours) but it failed to create two items: one in model_architecture - 017.mllt.8000.mdef and one in model_parameters - 017.mllt. I do not know why it failed to do this.
 * Results:

Since this didn't work on Traubadix, I will try to do this again with a fresh copy of 0301/006 in 0304 and run on Caesar whenever it becomes available.
 * Plan:

From today's Capstone class: !!!!!!!!!! NO MORE LDA TRAINS UNTIL FURTHER NOTICE !!!!!!!!!! No more LDA experiments without a bunch of 30 hour LDA's to find the best number, otherwise the LDA experiments are useless. May need to run a bunch of test sets with small half hour or 1 hour test sets on switchboard. Don't just take the first, sample evenly across.

Can continue with LDA decoding to determine how to use the -lda and -ldadim flags.

We are no longer looking to recreate the 28.4% WER experiment from last year (0301/006 Train Data then Retrained in 0301/011).

Run a 300hr non-LDA baseline experiment with new genTrans.pl script on new drone server Automatix. (See experiment 0310/019)


 * Concerns:

18 APR 2018
Decode Seen experiment 0310/019 which is a 300hr Data Train with 8000 senone count and final density of 8 utilizing new genTrans.pl script performed as first experiment conducted on new drone server Automatix.
 * Task:

I checked the model_parameters subdirectory to insure that all of the requisite subdirectories within it were present. I confirmed by inspecting 0304/001 (which also is a 300hr Data Train with final density of 8) that there should be a 019.cd_cont_8000_8 within /019's model_parameter's subdirectory, but there is not. I looked at the end of the 019.html file and there is no indication of any error that I can see.
 * Results:

I will have to run "scripts_pl/RunAll.pl" again.
 * Plan:

I can either rm -rf the contents of model_architecture and model_parameters or I can just allow them to be overwritten. I will allow them to just be overwritten. Of course I could just run a completely new experiment and I will do so if this errors out. I want to see what overwriting does, see if it causes any errors.

That it won't overwrite without erroring out OR that 019.cd_cont_8000_8 still isn't created without any visible reason as to why.
 * Concerns:

AMMENDED: 019.cd_cont_8000_8 was created during second run of "scripts_pl/RunAll.pl". Awaiting process to complete to move on to LM and then decoding overnight.

19 APR 2018
Check completion of Data Train, then create LM, then perform Seen Decode and Score on 0310/019, a 300hr non-LDA Data Train with 8000 senone count and new genTrans.pl
 * Task:

Data Train appears to have completed. Viewing the contents of model_parameters indicates that two subdirectories were redone/created based on their respective time stamps:
 * Results:


 * 019.ci_cont_flatinitial...23:10 17 Apr 18
 * 019.cd_cont_untied........03:02 18 Apr 18
 * 019.cd_cont_initial.......04:05 18 Apr 18
 * 019.cd_cont_8000..........04:23 18 Apr 18
 * 019.cd_cont_8000_1........05:22 18 Apr 18
 * 019.cd_cont_8000_2........07:13 18 Apr 18
 * 019.cd_cont_8000_4........09:53 18 Apr 18

(after rerunning "RunAll.pl"...)
 * 019.ci_cont...............12:57 18 Apr 18
 * 019.cd_cont_8000_8........13:35 18 Apr 18

Created LM on Automatix.

Will Decode and Score on Automatix when Tri is done running his experiment.
 * Plan:


 * Concerns:

20 APR 2018
Check completion of Tri's Data Train on Automatix, then perform Seen Decode and Score of 0310/019 on Automatix, a 300hr non-LDA Data Train with 8000 senone count and new genTrans.pl.
 * Task:

ALSO, talked to Josh Young who is investigating the -lda flag path for the decode process. He has determined that there are four files that the /python/sphinx/lda.py python code is running through and he suspects that based on what is called - *end comment* - that this code is not the correct path/file for using the -lda flag for decoding with LDA. He will continue to investigate further and let us know. ADDENDUM: The -lda flag during decode is looking for the LDA Matrix. Now we have to determine it's location to be called in there when the decode process is run with -lda.

Commenced Seen Decode at 4 pm. Expect to run past midnight.
 * Results:

Confirm Decode ran completely in the morning, then score.
 * Plan:

That the Decode will not run correctly.
 * Concerns:

21 APR 2018
Check completion of Seen Decoding of 0310/019 on Automatix which ran overnight. If completed, Score.
 * Task:

Completed and scored with a WER of 40.6%.
 * Results:

Determine next experiment to run in coordination with Data Group.
 * Plan:


 * Concerns:

23 APR 2018
Experiment 0310/019, a 300hr non-LDA Train with Seen Decode and Scoring, yielded a WER of 40.6% with the new genTrans.pl script developed by the Data Group. I decided to see what improvement, if any, there would be if the changes to the sphinx_train.cfg that the Rebels Team came up with last year (minus LDA) were applied to a 300hr Train. Those changes are as follows:
 * Task:

In the sphinx_train.cfg file, change the following:


 * Under the "Four (4) stream features for Sphinx II";
 * "CFG_STATESPERHMM = 3;" changed the 3 to a 5 (if you "vi" the cfg file, this is line 101)
 * "CFG_SKIPSTATE = 'no';" changed the no to a yes (line 102)


 * Under "Single stream features - Sphinx 3";
 * "CFG_FINAL_NUM_DENSITIES = 8;" changed the 8 to a 32 (line 107)

This Train is run in 0310/036 on drone server Traubadix.

To be determined as this Train is running beyond today.
 * Results:

Once the Train completes, Decode and Score.
 * Plan:

None at this time.
 * Concerns:

24 APR 2018
Determined that the Capstone Final Report is due Monday, 14 May with input from all groups and that a good basis for a draft is the 2015 final report. Peer evaluations will be conducted on 8 May. Will evaluate all fellow Avengers Teammates and my Modeling Group teammates and some others as well. The Avengers Team report rough draft will be submitted on Tuesday, 1 May with the Avengers Team Final report due for submission (printed, stapled, in triplicate for competition evaluation and grading) on Tuesday, May 8th. N/A.
 * Task:
 * Results:

Avengers Team will have to meet on Discord over the next week to provide input on the draft report due on 1 May.
 * Plan:

The content for the Modeling Group will be a challenge as the results from last year's group have been determined to be incorrect due to two different issues discovered with the use of LDA - that multiple .5 or 1 hour corpus test on trains would need to be conducted (anywhere from two dozen to thirty) in order to determine the best result WITHOUT repeating the data from one of those corpa to another due to the randomness of LDA; and that last year's group did not use the "-lda" flag in the decode process (nor did they use the "-ldadim" flag either).
 * Concerns:

25 APR 2018
Conduct a 30hr LDA Train with Seen Decode and Scoring using the "-lda" and "-ldadim" flags in the decode process. Attempted in experiment 0310/066 with LDA_MLLT configuration in sphinx_train.cfg switched from no to yes and the LDA_DIMENSION switched from 29 to 32. Used the "-lda" and "-ldadim" flags during the decode process but the "-ldadim" flag errored out during decode: "-ldadim: Command not found." That is highly annoying.
 * Task:
 * Results:

Fly to Aruba and sit on a beach drinking rum.
 * Plan:

That we won't figure out how these two flags work properly in the decode process before this semester ends.
 * Concerns:

26 APR 2018
The 300hr Train for 0310/036 has finally completed. This will be a Seen Decode and will run into tomorrow. To be determined.
 * Task:
 * Results:

Check correctness of decode at completion and score accordingly.
 * Plan:

That the decode fails and another day is shot.
 * Concerns:

27 APR 2018
A: Complete 0310/036 by confirming successful Seen Decode completion and Score.
 * Task:

B: Started reviewing log entries for fellow Avengers Teammates to determine comments for justification for grades for peer evaluations to be conducted during the last class on Tuesday, 8 May. A: 0310/036 Seen Decode ran successfully and the Scoring shows a WER of 33.4% which is a 7.2% improvement over the 300hr Train Seen Decode in 0310/019 which used the standard sphinx_train.cfg parameters and new genTrans.pl while 0310/036 300hr Train Seen Decode used the altered sphinx_train.cfg parameters described in that experiment. It would seem that these altered parameters should always be used in a 300hr train and that once correct LDA use is determined, another comparison should be done between the two on Seen Decode before proceeding to Unseen experiments with dev (-d) followed by eval (-e).
 * Results:

B: I have at least one good thing to say regarding each teammate. Naturally, some take the reins of leadership or research more than others, but everyone has contributed something of quality.

A: Further evaluation of the altered sphinx_train.cfg with correct LDA use will have to be conducted by the next iteration as we are simply out of time.
 * Plan:

B: To prepare a "passdown" of important information, including the best instructions possible of how to conduct an experiment (both Seen and Unseen Decodes, with and without LDA as we best understand it at the time) to four future students whom I know will be in this class next spring.

Figuring out how to correctly utilize LDA in both the Train and Decode processes before the semester is over.
 * Concerns:

30 APR 2018
Worked with Avengers Teammates via Discord on "Results of Capstone Team Baseline Modeling Competition Spring 2018" rough draft report for Team Avengers. Covered the Goal, Description, Results and Summary.
 * Task:
 * Results:

To be proof read tomorrow, printed and submitted at class start.
 * Plan:

I am just happy to be here.
 * Concerns:

May 1, 2018
One last course tasking is to compare what is yielded by using LDA in a single 5hr train, the to do a seen decode in 3 ways:
 * Task:
 * using run_decode.pl
 * using run_decode_lda.pl
 * using manual input of decoding parameters

To be determined.
 * Results:

Danielle, Dan Beitel and myself will meet tomorrow to conduct a single 5hr LDA Train and then copy as necessary into other subexperiments for the applicable decoding process. None regarding this.
 * Plan:
 * Concerns:

May 2, 2018
Run one single 5hr LDA Train. Copy as necessary for 3 separate decode attempts using 3 different means of decoding.
 * Task:

To be determined.
 * Results:

Create a new upper level experiment directory - 0311 LDA Validation Experiments.
 * Plan:

Create 001 LDA 5hr Train Only BUT with randomness eliminated IAW instructions in the experiment Results.

Create 002; copy Train from 001; create Language Model (LM); decode manually.

Create 003; copy Train from 001; copy LM from 002; decode with run_decode.pl

Create 004; copy Train from 001; copy LM from 002; decode with run_decode_lda.pl

That none of the decodes will work.
 * Concerns:

May 5, 2018
Meet with fellow Modeling Group members Hannah Yudkin and Brian Barnes to draft the Modeling Group input to the Capstone final report that is due in less than two weeks.
 * Task:


 * Results:


 * Plan:


 * Concerns:

May 6, 2018

 * Task:


 * Results:


 * Plan:


 * Concerns: