Speech:Spring 2018 Jaden Henry Log


 * Home
 * Semesters
 * Spring 2018
 * Proposal
 * Report
 * Information - General Project Information
 * Experiments - List of speech experiments

Week Ending February 5th, 2018
1/30 - After being assigned to the Experiments group, I started thinking about what improvements could be made to scripts created by previous semesters' students. Once I got home, I logged into Caesar and browsed to the folder containing scripts. I then did a rudimentary comparison of scripts within the folder and scripts documented within the wiki to make sure nothing was missing. My main goals for tonight were to get into the scripts folder on Caesar/explore other folders and to start familiarizing myself with how these scripts work. I'm eager to start learning Pearl and to tweak some of these scripts to be more user-friendly.
 * Task:

2/1 - Today Danielle, Lamia, Steve, Dan B, and I met in the afternoon at school to go over what we've done so far as well as answer each others' questions about navigating Caesar and running experiments. During this meeting, I created my experiment folder within /mnt/main/Exp/0303 and then a sub-experiment folder. I also installed FileZilla and tested opening a connection to Caesar with it for if I need to copy files to my own desktop.

2/3 - Today I reviewed other students logs and went through more of the documentation on the wiki relevant to experiments and scripts.

2/4 - Today I logged into Caesar to run my first experiment, using the Experiment Setup wiki page as a guide.

2/4 - While trying to run my first experiment, I got a weird error that said "Something failed: (/mnt/main/Exp/0303/015/001/scripts_pl/00.verify/verify_all.pl" after running "nohup scripts_pl/RunAll.pl &". I decided to keep going with the experiment setup just in case it was not a real error; a hiccup in the script rather than something actually wrong. After continuing a few steps without error, I got to the decode step and received the error "Segmentation fault (core dumped)" after running sclite -r 001_train.trans -h hyp.trans -i swb >> scoring.log. There were no fixes listed in the SCLite common errors section though I did try the script at the bottom to remove redundant files.
 * Results:

1/30- As for editing scripts that everyone else in the class depends on, I think it will be important to make a copy of the script file, rename it somehow, perform testing, and after various other members of the class have tried it, replace the original script file. I pitched this idea to Dan and he stressed the importance of learning from previous semesters.
 * Plan:

2/1- I plan to run my first experiment tomorrow night after going over the Experiment walk-through again. For my first time, I intend to leave the default options as is suggested. However, I think it is vital to edit those in the future as a member of the Experiments group.

2/3 - During our next in-class meeting when we are hopefully all present, I would like to setup time to meet outside of class. I've seen that other groups have decided to meet either weekly or bi-weekly.

2/4 - I plan to try troubleshooting my experiment setup process more tomorrow and discuss it with my group members. 1/30 - My only concern at this time is that one group member was not present and hasn't been in communication with us since class ended. Dan sent an email with information regarding the group and the procedure for if one is to arrive after 2 pm.
 * Concerns:

2/1 - After looking over some of the scripts' code, I can't easily make sense of what is going on unlike some other scripting languages such as PowerShell. I plan to invest time in an online Pearl course so I can navigate my way around the various scripts and edit them in the near future.

2/3 - My biggest concern at the moment is coordinating with everyone else to get our project proposal ready on time. I think it would be most efficient to setup a shared Google Doc, create an outline, and have people add content/refine as we get closer to the deadline. I think it will also be helpful to track who does and does not contribute to the writing process. Another concern I have is whether or not we'll be able to find a consistent time that works for all of us in which we can meet outside of class.

2/4 - The fact that I got an error that wasn't addressed on the wiki while running the decode of my experiment is frustrating but it appears that all my files are in the correct place; there aren't any huge file dumps in any of the main directories.

Week Ending February 12, 2018
2/6 - Log into a drone and run a train.
 * Task:

2/7 - Create shared document to work on our project proposal and share with the class.

2/8 - Start drafting our group's proposal, which the class decided to have due on Sunday before 6 pm. I also read other students' logs to see where everyone was at with running their first experiment(s).

2/9 - Continued working on our group's project proposal.

2/12 - Reviewed other students' logs and tried running an experiment from the beginning again.

2/6 - I was able to run the train successfully after deleting the original sub-experiment directory I created and starting fresh. I regrettably chose 30 hours to train on rather than 5, which contributed to how slow the servers were.
 * Results:

2/7 - I created the project proposal document and pasted a link to it in the general Discord channel so everyone could see. Dan B tested accessing the document for me to confirm that other people could also get in.

I also got a more concrete path as far as editing scripts, organizing directories, and making adjustments to reflect these changes on the wiki during our in-class meeting. I knew we were tasked with addressing some of the issues with the addExp script but clarification on other smaller projects was helpful as well.

2/9 - Today I found out that I'd had the experiment directory structure wrong for every experiment I've attempted thus far. As a few other students have done, I had been making another sub-directory within my experiment folder and had been running trains from within there. This meant that scripts weren't able to access the necessary files and was the source of my weird errors. Luckily, Dan B helped me figure out that we have to ignore the documentation on the wiki that lead us to believe we needed an extra folder.

I currently have a 30 hour train running and will finish the experiment if it completes in time tonight.

2/7 - The plan once the project proposal is complete within the shared Word doc is to paste it into the wiki and re-format it.
 * Plan:

2/8 - I plan to continue working on our project proposal tomorrow night and will seek feedback from my group members when I am satisfied with what I've written. I would also like to continue with my experiment tomorrow, hopefully completing it before the weekend.

2/12 - Edit the Experiment Setup documentation on the wiki to actually reflect what the directory structure for an experiment is supposed to be in the coming days.

2/6 - We still haven't heard much from our third group member.
 * Concerns:

2/7 - I'm somewhat concerned about the project proposal being due in a few days and hope that each team will fill out their designated section according to the template and the expectations set forth by previous semesters.

Week Ending February 19, 2018
2/13 - Today it was imperative that I fully complete my first experiment. Up until today, I had been starting fresh each time I received an error on the final script run during the decoding process. Fortunately, Dan B was able to help me with the decode and find what I was doing wrong before. This means that everyone in our group has successfully run an experiment so we can focus our efforts on our real tasks.
 * Task:

We also started looking at the addExp script as a group. We also looked at resources where we could learn Pearl the quickest to bridge the gaps in our understanding of what each script is doing.

The last item today was updating our group's wiki page to reflect what we are currently working on as a group.

2/15 - I helped revise our group's section of the project proposal that we had given a due date of Saturday.

2/16 - I reviewed other students' logs and continued working on our revised proposal.

2/19 - I reviewed other students logs and continued with a Perl online course. On the previous day, I worked on the revised project proposal some more.

2/13 - My first experiment has been run correctly and I updated my first experiment's page with the results.
 * Results:

2/19 - The project proposal was completed yesterday and the group's task list was updated.

2/13 - Devote time tonight to studying Pearl and work on revising our group's section of the proposal after receiving feedback from Professor Jonas.
 * Plan:

2/15 - After the group collectively completes the revised project proposal, I will dive deeper into understanding Perl and create a root experiment that my group will use to debug the addExp script. This will serve two functions: test the usability of the -r switch of addExp and create a directory that will serve as a sandbox. Our group knows going into this that debugging the script will involve a lot of creating and then immediately deleting wiki page entries and hyperlinks to said entries on the root experiment's page.

One thing I would like to test in my own copy of addExp would be what happens when you remove the line that creates the temp variable, which is not referenced anywhere else in the file. Dan B noted that he didn't experience any odd behavior when testing the sub-experiment creation through addExp but it will be helpful to have multiple tests.

2/19 - I plan to continue with the Perl course and help debug addExp. The source of the error has been narrowed down to within a few lines of code (we believe) but no clear indication as to why that code is creating the problem. It is important to get this aspect fixed before the group attempts to restore the missing features - auto-increment and adding the student's name to the wiki entry. However, if this takes too long, it may be a good idea to work on createExp in parallel.

Lastly, after reworking the project proposal, the group has a more in-depth guide for the rest of the semester

2/13 - None at this time now that we have a full group.
 * Concerns:

2/15 - Still no concerns, I am confident in our group's abilities.

2/19 - No clear cause of the issues within addExp, will require a deeper understanding of Perl and researching online forums.

Week Ending February 26, 2013
2/20 - Today I continued with the Perl course and started looking up our addExp error on Stack Overflow. I searched using the following words "Perl unable to find file" and mostly came up empty handed. It was suggested that we try an if else statement that checked for the file but I don't believe our issue is stemming from an inability to find the actual file. The directory the error is pointing to appears to be random so I'm wondering where that is even being introduced.
 * Task:

2/22 - I continued with the online Perl course and logged into Caesar to get the version of Perl running on it, which was v5.10.1. It appears the most current version of Perl is 5.26 so it is not incredibly outdated. The group has ruled out the version of Perl negatively impacting the addExp script because both the version on Caesar and the current version are within 5.X. I had learned in another class that any changes that would impact previously compiled code would be bumped up to 6.X. Additionally, I found out from my online Perl course that the fact that the command perl -v returns the expected output, that means that there isn't anything wrong with the installation on Caesar. This rules out a couple things in relation to the addExp script but means our group will have to do more research on online forums such as Stack Overflow.

2/24 - I completed the online Perl course "Learn Perl in 7 Days" from UDemy. I thought the lecturer explained things very thoroughly and would definitely use this site as a resource again in the future. The most advanced topics the course covered dealt with file operations such as copying contents of one file to a new one or adding to an existing file. I feel much more confident looking over the contents of the scripts now that I understand the syntax of the language and differences between Perl and languages I've already learned. For example, Perl subroutines don't require you to specify arguments when you call them like they would in Java or VB, two languages I've taken in school. As I've finished the course fairly late tonight, I plan to do more in-depth debugging and testing of addExp tomorrow.

2/26 - The task for tonight was continuing to debug addExp.

2/24 - The Perl course is complete and I'm ready to keep tackling addExp. Not only do I have a better understanding of what is going on in the script, I also know how to better phrase my questions for when the need arises to consult online forums in order to solve an issue. When looking over addExp before completing the course, I was able to make assumptions such as $ creates a variable and @ creates an array, but I didn't know that $ referred to a scalar variable. Another helpful thing I've picked up from the course is a Perl IDE called Padre, which the lecturer suggested.
 * Results:

2/26 - 	While debugging addExp, I noticed that within the method createSubExperiment, there doesn't even exist a broken version of a way to automatically increment the sub-experiment number. In other words, it appears that the description on the page is not outdated. Instead, we hypothesized, a previous group copied the description from a script that just created the root experiment and never bothered to read through to make sure it was accurate.

After this setback, I did fix the issue that was throwing an error as soon as users went to execute the script. After looking at the version of Perl last night and looking at various other causes, I decided to try what I saw as the simplest or "dumbest" solution: I found all instances of the diamond operator in the code and replaced it with . The group had figured out that the diamond operator was interchangeable with  but I just wanted to be sure after reading through sources online that <> was typically only used in a while loop. I also commented out line 81, which had the temp variable as well as the diamond operator. After Dan and I tested, we confirmed that this solution worked. One hiccup we noticed was that you can't specify a file name in the sub experiment's name or you get a token error. We are unsure whether or not this needs to be fixed or is even preventable. I recently learned that in Perl, you can use a variable within a string and it will be evaluated as a variable rather than text. I believe a similar thing is happening when you use a file extension but will verify tomorrow during our in-class meeting.

After testing the fix, I then logged in as root, opened the original addExp file within /mnt/main/scripts/user, deleted all the code, and pasted in the code from my addExp-JH script that was working. I'm not sure if this is the proper form but will also ask about it in class tomorrow. Because the original version was updated, students will notice that they no longer get the error when they each run it. Dan and Arias both have addExp-FIRST + LAST INITIAL copies of the script in the same directory because we don't have a code repository available. This helps limit the possibility of code being overwritten and helps cut down confusion. Lastly, I have not updated the code for addExp on the wiki yet because I am continuing to work on the username functionality and will hopefully have more updates to it.

UPDATE: Tonight I added the ability to prefix the sub-experiment link text with the username of whoever is running it. This is only available in my copy of the script (addExp-JH) and will confirm with Jonas if this is the functionality he would like. If not, we could supplement this with first name and last initial or something similar.

2/20 - I'm continuing to comment code within my local copy of addExp so I can better understand everything. In the near future, I would like to upload my own edited version to the server and run a few tests such as removing the line in question completely or adding a chomp command directly after it. This script, compared to createExp, has less potential to be disruptive to other students when debugging because its main function is to edit the wiki. Our group has reached out to the Systems group about a testing environment and have yet to hear back just yet.
 * Plan:

2/20 - I plan to finish the online Perl course tomorrow and begin testing potential fixes for addExp, while cleaning up bogus experiment entries I create on the wiki while debugging.

2/24 - I plan to keep working on addExp tomorrow as well as on Monday. After making adjustments, I will push my copy of addExp, which I've renamed addExp-JH, back up to Caesar and test.

2/26 - The group has divided the remaining task of adding missing functionality to addExp. I will work on prefixing the experiment name with the user's AD username and both Arias and Dan will work on adding the auto-increment feature to the script. It is expected that adding auto-increment will take longer than prefixing the student's username so I plan to jump onboard with that task after finishing my piece. I also plan on improving the language within the prompts with running addExp. 2/20 - I'm concerned about being able to solve the addExp error. Hopefully with a combination of continuing to learn Perl as well as continuing to look at Stack Overflow and other forums, the group can make a discovery.
 * Concerns:

2/22 - No further concerns at this time.

2/26 - We're pretty sure that adding auto-increment to addExp will require taking in an XML path, similar to the way it is done in the createRootExperiment method. This seems to be a difficult task and we're unsure how long it will take. May require asking Professor Jonas for help.

Week Ending March 5, 2013
2/27 - Today, after the group received approval from Professor Jonas, I added the functionality to addExp involving adding the username to the sub-experiment entry on the wiki to my own copy of addExp.
 * Task:

3/2 - I pushed the code that would append the username to the author line of the wiki page created by addExp. Tonight I am also working on adding auto-increment to addExp.

3/3 - Write up a quick breakdown of the auto-increment functionality within addExp to share with my group members, test addExp using a new root experiment, update main addExp file if it works, and start debugging createExp.

3/4 - I reviewed logs and attempted to troubleshoot an error Steve was getting when he tried to run addExp. It appears that somehow the entry wasn't actually creating a page and that threw off the auto-increment. I then updated the addExp page on the wiki to include the most recent code.

2/27 - My version of addExp will now append (UserID: jah2009) next to Author: on the sub-experiment's dedicated wiki page, replacing my ID with the ID of the author. The version we proposed to professor Jonas added the username to just before the link but this way conveys the same information once you click on a specific sub-experiment and keeps the list cleaner.
 * Results:

3/2 - I used code from the createWiki_sub_experiment script that used to properly auto-increment the given sub-experiment. I haven't been able to get it to properly go beyond 001 for some reason and it's pretty frustrating but I am making headway in understanding how the script actually connects to the wiki and posts content. One way I've been debugging is to comment out the actual code that posts to the wiki and instead replacing it with a print statement that shows what the number of the sub-experiment WOULD be should it be posted. This saves having to clean up entries.

UPDATE 3/2 - I was able to get the auto-increment working by using the old code. One of the annoyances of dealing with the createWiki_Sub_Experiment script was that there weren't any methods. So instead of copying the contents of a method and not having to worry about variables not existing, I had to go into the code and find exactly what I wanted. Luckily, all the code that actually posted to the wiki was the same except for the name of a variable. Essentially, I pasted about 20 lines of code, made sure everything matched up as far as brackets and if/else statements go, and then changed an existing variable, numStr, to be subExperiment. I haven't updated the main addExp file yet because I'd just like to test at a time that isn't so late to make sure it's all set.

3/3 - The auto-increment still worked when I created a root experiment. When the root experiment is created, it automatically prompts the user for details for the first sub-experiment. If addExp.pl -s is run subsequently, then it auto-increments properly. One thing I did note was the somewhat lengthy process of creating the root experiment would easily be thwarted if the script was terminated early, which is what happened to Rose last week. I'm trying to think of ways to prevent this or at least stop it cleanly if the script is closed prematurely. Professor Jonas suggested only doing the actual posting/updating of the wiki at the very end but I'm not sure how easily we'll be able to re-order everything in the script's current state. I believe the most pressing issue to address is the fact that createExp is broken but I do think the team should return to that problem after we are done with that task.

I got the following errors when trying to run createExp.pl -a

3/3 - I ran addExp in our own directory and used the same name that Steve had used when the script didn’t work. I was unable to replicate the issue and can't think of anything that would be causing that behavior. I also tried adding a trailing space in the sub-experiment name and that didn't break it either.

2/27 - Now that I've ensured it works, I will publish this change by editing the original addExp file tomorrow morning when there is a lull in activity and the potential for someone else to need root access is slimmer. Once that is complete, I will keep testing solutions/researching online to add auto-increment into createSubExperiment within addExp. Our group is fully focused on this task and hope to complete it within the next 1-2 weeks. I will simultaneously rework the user prompts within addExp as mentioned in our project proposal. While working on this, I will try to recall the confusion I experienced when running the script for the first time before the class officially started. The biggest source of confusion, I believe, was the naming convention for sub-experiments. After completing those items, we can move to debugging createExp. If we don't make any headway with the auto-increment, which I find unlikely, then we will start working on createExp at the same time to maximize the group's results for the semester.
 * Plan:

3/2 - I will continue trying to get the auto-increment function working tomorrow. If I can't make any progress using the old code, I will start writing my own from scratch and see if my results improve.

UPDATE 3/2 - After figuring out auto-increment at the last second tonight, I will update the main addExp file tomorrow if further testing goes well and will start working on createExp with the rest of my team.

3/3 - I plan to look deeper into the errors that come up at specific lines within createExp. The documentation doesn't state what the -a switch does so hopefully this would run all steps in the experiment creation process. The errors I received from running the script are more promising than I thought they would be, as I can go into the code and see exactly what it's referring to.

3/3 - I will continue to investigate the issue with addExp and then will keep working on createExp. 2/27 - No major concerns at this time other than the fact that documentation surrounding how to turn the wiki page with all the sub-experiments in any given root experiment into xml and read all of it seems to be non-existent. We will try to mimic the functionality in createRootExperiment to get the auto-increment working in createSubExperiment.
 * Concerns:

3/2 - None other than feeling like this script is trying to consume my soul but it won't win!

3/3 - The sporadic issue that Steve was seeing is concerning to me because I as well as Dan B were unable to break the script in an effort to start troubleshooting it.

Week Ending March 12, 2013
3/7 - Today I assessed the copyExp script by reading the in-depth description on the wiki as well as going through the code. I chose to look at this script tonight instead of createExp because the class has been experiencing issues with the decode step of experiments. At present, the group is unsure how to run every step of the experiment process sequentially without starting one step in the process before the previous one finishes. I've also reviewed other students' logs for more information on the decode situation that the class has been updating each other on through email and Discord.
 * Task:

3/9 - I attempted to debug createExp by going into my own copy of the script, commenting out code that would actually run other scripts, and instead adding print statements that would just tell me the value of variables to be used in a path. The advantage (if this had worked) would've been that I wouldn't be using nearly as much CPU and it would also take less time to get an answer as to whether or not the directory variable was correct in each method.

3/11 - I continued working on createExp. Today I tried removing the global $dir variable in favor of local variables within each method. This, of course, messed up every path so the rest of the time was spent adding print statements and running the script to check what paths were being used.

3/12 - Remove my copy of createExp from /mnt/main/scripts/user and move it to a new scripts directory within my user folder. I then continued working on createExp by testing each step of the experiments creation process and seeing if the correct directories were used for each command. I attempted the make my version of the script more readable (for me as well as future users) by using variable names such as $rootExpDir and $subExpDir to describe the current working directory. This way, I don't have to retrace what's going on or add comments to the code. I also implemented the fix for a variable followed by an underscore present in a filename. Arias reached out last night after reading my log and told me to use the escape character after the variable. Looking back, this seems obvious, so that definitely slipped past me after I'd been working on the script for a while.

3/9 - Unfortunately, my debugging method returned a bunch of errors dealing with how I was calling variables that couldn't necessarily be printed, or at least that was how I interpreted the message. One example being the following line:
 * Results:

After receiving the error, I put that idea on the back burner and started looking at the code within the debug method. I was skeptical of the line

and wondered if that could account for why the directory variable wasn't actually going up one and not reporting it if it didn't. However, after looking on an online forum, I found the exact same fix posted. I then looked elsewhere to see if I could find other approaches to test but didn't see anything that was very promising.

3/11 - I was able to get the train and LM methods to use the correct paths and therefore build the correct files. The issue I'm having now is passing a command that includes a variable followed by an underscore. If I keep the two together, it is evaluated as a single variable. If I break them up into 2 concatenated strings, it for some reason only reads the second half of the string. I will need to do more research on how to do this correctly.

3/12 - I ran a successful train using the -t switch from createExp. In my opinion, the script is easier to understand in places where I've supplemented a well-named local variable instead of a global variable named $dir.

3/7 - copyExp is a much shorter script that is less complex than createExp. Professor Jonas expressed more desire to ensure that this script is working correctly so I plan to devote my time to testing and fixing this script over the next week. It is possible that part of the code in copyExp that sets the source/destination paths will aid us in the createExp script. Overall, the knowledge we've gained recently may indicate that createExp will not be feasible this semester in the way that we had hoped.
 * Plan:

3/9 - Tomorrow I want to try writing my own version of createExp while still maintaining a few elements from the current version. I'd like to try multiple directory variables rather than a global directory variable as well as perhaps prompting for more user input instead of relying on splitting paths using the / character as a delimiter. Both of these ideas have been proposed by my groupmates who have more experience with the script but I'll give it a shot myself nonetheless.

3/11 - I plan to finish setting up the correct paths in createExp using local variables. Once that is complete, much more testing is required.

3/12 - I will continue working on createExp. More specifically, I will revist the makeLM & decode methods. I would also like to be able to generate some sort of output in createExp while the script

Is running. I added a line tonight that indicates the script is running right before the user sees that a command is running. Otherwise, the state of the process at that moment is ambiguous as genFeats.pl says something to the effect of "You're all set, now run nohup!". This is confusing because the user may think they now have to manually run this unless they get output stating otherwise. 3/7 - The group is concerned about our ability to complete createExp.
 * Concerns:

3/12 - No further concerns at this time.

Week Ending March 26, 2013
3/20 - Today the Avengers group met after class and discussed our game plan and tasks for the upcoming week. We decided we will meet after class every week because it seems to work for everyone on the team and keeps information fresh in our minds.
 * Task:

3/22 - Tonight I tried fixing the addExp bugs that would make link creation on the wiki unsuccessful if the user accidentally added a leading space or used the '&' symbol anywhere in their sub-experiment title.

3/23 - Today I've been trying to run an experiment on unseen data using the 5hr corpus. Once I know I can successfully run the experiment on the smaller subset of data, I will move to larger ones so I can accurately gauge the WER.

3/26 - After attempting to run another experiment on unseen data and receiving the same error yesterday, I went back to working on createExp today. The specific error I received on two separate occasions was that  was missing. Steve was able to do some more testing and found that one of the commands was being run in the wrong directory, which meant that file wasn't in the right place. After this happened, I realized that I need to be more vigilant as a member of the Experiment group and rely less on any written instructions given to me. Furthermore, I need to contribute more to the actual testing and writing of instructions for the Avengers team. After some time debugging createExp and making a tiny bit of progress, I dove into copyExp and briefly revisited addExp as well.

3/20 - I shared with the team what I'm currently working on and we all discussed how to contribute to the baseline. We setup additional Discord channels within our server to better keep track of important information.
 * Results:

3/22 - I was able to fix the one bug in addExp regarding a leading space in the sub-experiment title by adding an extra line with a trim method call with the parameter of the string holding the typed sub-experiment name (indicated by the user). I tried a few fixes for the other bug but was still finding that the link was being incorrectly created. For some reason, it was only keeping the first word from the string, forgetting the rest, and failing to create a proper link because the double brackets were left off at the end of the line. My approach, as Professor Jonas suggested, was to find any occurrence of the & symbol and replace it with the + symbol. I looked on various online forums and saw that the fix should be easy and would not require making an additional string. However, I did not find this to be true. I even tried using escape characters before both symbols and was still getting the same odd behavior of it only keeping the first complete word.

3/23 - I accidentally forgot to run a command in the experiment creation process, which I realized when I went to score and it wasn't finding a specific directory. Using the history command and then going back through the instructions, I realized I missed the makeTest command. I decided to delete my directories and start fresh due to past experiences on experiments using previously seen data. In the future I will pay more attention to what script I'm running and why and not try to multi-task as this is eating up extra CPU utilization.

3/26 - I was able to get copyExp working tonight by adding a simple line to the very beginning of the script. Prior to this, the script was stuck in a loop no matter where you ran it or what parameters were used. I was able to fix it by adding a single line that I noticed at the very beginning of a script I knew was compiling fine, createExp. The line was. However, I'm unsure of whether or not adding any commented/useless code to the very first line would've also fixed it. addExp also compiles and does not have that line at the beginning of the script. The source of the error appears to be the  line being the first line of the script now that I reflect. I still wasn't able to make much headway on createExp and the remaining fix for addExp dealing with the need to replace '&' with '+'.

3/20 - Tomorrow night I will continue working on copyExp so I can finish as soon as possible to assist Dan with debugging copyExp, which will be vital in the upcoming weeks.
 * Plan:

3/22 - I'm going to put the last addExp bug on the back burner and return to createExp so I can finish it up quickly and then move on to copyExp, which seems to be the most important task at the moment. I had hoped to finish up the addExp bug in one go tonight but I feel that it's easy to tell the group not to include & in the title for now. The most cumbersome of the bugs, in my opinion, was the leading space because that is more likely to accidentally happen. I do have an expected finish date for createExp before I move on to helping Dan with copyExp. If I don't finish createExp by next Tuesday, I will halt work on it for now, re-familiarize myself with copyExp, and work with Dan. 3/23 - If the decode does not finish before I go to bed tonight, I will finish that in the near future and then see how much I can get done within createExp. After that (by Tuesday), regardless of whether or not my portion is complete, I will move to copyExp.

3/26 - I plan to continue working on addExp and when that is finished, return to createExp. With the remaining half of my time this coming week, I will try again to run an experiment on unseen data.

3/20 - Both teams are unsure what their third drone will be until we can gather more information of the status of LDA.
 * Concerns:

3/26 - No concerns at this time, I believe that my group and team are making good progress.

Week Ending April 2, 2013
3/28 - My task for this evening was another attempt at a train on unseen data.
 * Task:

3/29 - I completed the decode and scoring for the experiment on unseen data I started on the day prior on the 5hr corpus. I also worked on addExp more to see if I could finally get the character replacement using regex to work.

4/1 - I reviewed other students' logs.

4/2 - I began an experiment using the 30hr corpus on unseen data and got to finish the train before going to bed. I also made adjustments to createExp that correspond with the feedback our group received from Professor Jonas. It was our understanding that createExp would setup and populate all folders THEN run the given experiment based on the parameters/input given to it. We were told today not to worry about actually running the experiment and to take out the actual  and   commands. This actually simplifies the script and allows us to make improvements. To begin, I went back into the makeTrain and makeLM methods of createExp and simplified each command to run as though it was already in the proper directory. This meant removing the local directory variable in each method because the assumption is that the user will run the script within the folder they want to have the experiment be in. I then removed the makeDecode method and its method call within the makeAll method. This method originally called makeTrain, makeLm, and makeDecode. I thought this method would still be handy if the user wanted to quickly create both of the remaining items so I kept it. As a note to the Avengers group, the finished version of this script will also handle up to step 16 (aside from the call to addExp) of the instructions we've been using to run experiments on untrained data.

3/28 - Unfortunately, I wasn't able to fully complete the experiment because of the time it took for the decode. The current time for just a 5hr corpus is upwards of an hour so I will have to finish it tomorrow. Recent experiments on unseen data had gone well until scoring so I hope there are no surprises tomorrow and I'm able to collect my results.
 * Results:

3/29 - Results of the experiment are in 0308/022. Tonight I was still unable to fix the character replacement in addExp. After a few more tests which were combinations of including and not including the escape character before '&' and '+' in the regex line, I tried to see if I could just replace the letter e with z. Unfortunately, this didn't work either so I continued searching online Perl forums to see if someone had asked a similar question in the past AND to get someone else's working code to see if the issue was even me incorrectly typing the regular expression. I was getting frustrated and thought it was unlikely that I was incorrectly typing this or using the wrong syntax because it seems pretty straightforward. is all I needed to do. Luckily, I did find someone who asked about replacing all periods in a sentence with an underscore. Seeing that they also wanted to replace a special character with a different one, and one of them needing an escape character before it, I thought this would be a perfect test. The user also displayed their output so in my mind, this code should definitely work when I paste it into addExp and use the character to be replaced in the experiment title. To my surprise, this didn't work either. I can now confirm that there is something strange going on with this script and will reach out to Professor Jonas about it.

4/1 - Caught up on what other students are accomplishing.

4/2 - I was unable to test the simplified version of createExp tonight because I was already using one drone to run my 30hr experiment and didn't want to steal any more from other group members the night before class. The script shouldn't have any issues because it has been simplified as I detailed in the task portion of tonight's log.

3/28 - I will finish the experiment I started today, hopefully start another one on a different corpus, and then go back to working on addExp. The Experiment group may seek help from Professor Jonas if we can't get regex to work within our replacement of '&' with '+'.
 * Plan:

3/29 - The next time I login to the system, I will run a 30hr train on unseen data and hopefully work more on addExp. I will email Professor Jonas in the near future to explain my regex issue to see if he has any ideas.

4/2 - I will complete my 30hr experiment by creating the LM and then decoding tomorrow night. I will also test and modify createExp accordingly then update it on the wiki. I've decided that I will talk to Professor Jonas tomorrow after class instead of emailing so I can show him the exact behavior within the addExp script regarding regular expressions not being executed. 3/28 - No concerns.
 * Concerns:

3/29 - The strange behavior of regular expressions in addExp is concerning to me.

Week Ending April 9, 2013
4/3 - The Avengers group met today after class to discuss our progress and delve deeper into our game plan for the rest of the semester. I then tested my solution for createExp that I came up with last night by running the script with the -t switch and then the -l to setup the directories for a train and then creating the Language Model.
 * Task:

4/5 - Today I met with members of the Avengers group to discuss strategy for the second time in two days. Tonight I updated the main addExp file to be the version that automatically trims the sub-experiment string to get rid of the issue with broken links if the user adds a leading space. I retried testing the simplified version of createExp because I wasn't able to run the full experiment on 5hr seen data correctly. I will again try running the createExp command twice, one with the -t switch and one with the -l switch because I want to make troubleshooting the process easier. If I were to use -a to run the train and then create the LM directly after, there is more room for error. This test will make sure I didn't make any silly mistakes the last time, which seems likely because the script hinges on the user being in the correct directory but only has a few commands per method. I had also run two different experiments very close to each other which may have lead to some mistakes.

4/8 - I reviewed other students' logs.

4/9 - I tried to create a new root experiment for the Avenger's team seeing as the Guardians have already created one and populated it with a few sub-experiments of their own. However, I received the error  This is very strange to me seeing as I tested the same code using -s and never got anything like that. Conversely, it worked perfectly to protect against users accidentally adding a leading space before their sub-experiment name which would break the script and therefore break the link generated on the experiment's page. Seeing this new output, I removed the code in the addExp script and re-tested.

As a potential fix for the issue I mentioned in a previous log involving directories in addExp and moving from makeTrain to makeLM, I moved the method call getDirectory to be inside of makeTrain instead of at the beginning of the script outside of a method. This means the user will be prompted for the directory when they run makeTrain but the script will assume they're in the correct directory to run makeLm and will just execute all the code in that method. I will add some text to the makeLM method that communicates this much to the user if my test of the new code works.

After beginning to run this code on a 5hr corpus and fixing a few mistakes, I also decided to CD the user into the sub-experiment they specified after the last command within makeTrain is run to make the user experience more smooth.

4/3 - I was able to successfully get the train setup through createExp and was then able to run  from within the directory easily and without confusion. The script then built the directory to contain the Language Model, changed directories to be inside the newly created one, copied the necessary files, then created & parsed the LM.
 * Results:

4/5 - I've found one pain point when trying to run createExp with -t followed immediately by -l. Currently, createExp seeks to be in the root experiment folder because it asks prompts for the sub-experiment directory and either cd's into it or creates a new one. However, you are in the sub-experiment directory after running the train. If the user immediately calls createExp -l, it will say it can't find the sub-experiment folder and create a duplicate folder within the one that it's intended to be. I would like to rework createExp to be more user friendly because the action of forgetting to go back to the root experiment directory and deleting excess files that the script creates isn't saving any time compared to manually running each command. I think the gap in understanding as well as the ability to implement it according to Professor Jonas' wishes stems from the previous year's Experiments group attempting to run every aspect of the experiment from a single script.

4/9 - After removing the line including the trim command from addExp, it worked perfectly. This has me wondering if a previous group implemented this functionality to protect against this behavior but found that the same thing happened. I will tell Prof. Jonas of this behavior tomorrow in class during our status meeting and see if he has any ideas as to why it would only exhibit this behavior of not knowing the command with only one of the flags.

On one hand, my edit to createExp involving moving the getDirectory call around worked. On the other hand, the CD into the sub-experiment directory did not. I've been under the impression so far that changing directories in Perl correctly is nearly impossible and always a huge pain so should be avoided when possible.

makeTrain ran smoothly but makeLM seems to be hanging on  after. This means that the script will attempt to change to the sub-experiment directory that was passed to it when the script was first executed and if that fails, end and tell the user the action couldn't be completed.

4/23 - After adding the code that would alert me if the script was unable to change directories, I got this message:. For me, this was actually a step in the right direction because I now know that the script is at least trying to change directories and not doing something weird instead.
 * Results:

For my next attempt at getting this aspect of the script to work, I commented out the code that would actually change directories and instead added a line to print what the subExp variable contained. Contrary to what I would've expected seeing as I get the message that the directory doesn't exist, it contained exactly what I wanted it to: the sub experiment directory. In this particular instance, it was 002.

Next, I created a local variable that would hold the parent working directory or the root experiment directory in hopes that adding it before the sub-experiment directory in the chdir call would yield different results. The new call would look like. This still didn't seem to work but I think I understand my mistake. I've been testing by running the script and then when the script says to call the runAll script, I've been doing it outside of createExp. It seems that chdir is only relative to what is going on inside the script and Perl then resets to the directory it started in once it finishes. Therefore, I need to go back to the drawing board because the functionality I'm trying to implement in order to make createExp more user friendly is not going to work given that Professor Jonas doesn't want the script to actually run the train. Now I see why the script was written the way it was before and am cursing myself for being stubborn for so long.

After some more testing and troubleshooting, I drafted some new instructions on how to correctly use my version of the script once it is complete:


 * 1) SSH into a drone
 * 2) Run createExp -t from root experiment directory
 * 3) After genFeats completes, cd into sub-experiment directory and run "nohup scripts_pl/RunAll.pl &"
 * 4) Once that completes, run createExp -l (you must be in your sub-experiment directory if returning to this script at another time)
 * 5) Run createExp -d . Once that finishes, pick up in the experiment directions at "nohup run_decode.pl"

Unfortunately, the script said "Something failed" during training so more testing is required.

4/17 - I will meet with Dan R later this week to continue working on our piece. Meanwhile, I will keep working on createExp.
 * Plan:

4/22- Tomorrow I will work on createExp.

4/23 - Once I get the revised script fully working, I will upload it to the wiki and update the description to indicate to the user that despite the name of the script (and the intention of its creators last year), it does not fully create an experiment for them. Instead, I will stress that the script merely sets up an experiment on trained/untrained data. I would also like to gather feedback on the current implementation of my createExp script.

4/17 - No big concerns at the moment, I think the team is doing a really good job so far.
 * Concerns:

Week Ending April 30, 2013
4/24 - I worked Dan, Lamia, and Isaac primarily during and after class to work on Avengers team stuff.
 * Task:

4/26 - Today I did some research in regard to a few variables in the sphinx config file that will lengthen the decode process to ideally yield better results. The Avengers team also discussed strategy as the end of the semester draws closer.

4/29 - I worked on the Avengers team's final report draft with Lamia and Faruk.

4/30 - I updated our master spreadsheet with information on the experiments I've run. I also continued working on the final report draft. Additionally, the team had a meeting over Discord to troubleshoot aspects of our solution.


 * Results:

4/30 - Before next week, I will go through the scripts directory on the server/the scripts wiki page to make sure everything is accurate.
 * Plan:
 * Concerns:

Week Ending May 7, 2013
5/1 - The group met during class time to go over our plan for the rest of the semester and ask Professor Jonas a question concerning an error we have been receiving recently when running special experiments.
 * Task:

5/3 - I further refined my version of createExp and attempted to run a whole 5hr experiment. I also worked on the Experiment group's portion of the final report. Will update this entry later if it is successful.

5/3 - createExp worked! I updated the main version on Caesar as well as the wiki version. I also changed the description and added instructions on running it.
 * Results:

5/4 - I continued working on Experiment group section of the final report.

5/3 - If createExp works for the 5hr experiment, I will edit the main version and edit the wiki entry.
 * Plan:
 * Concerns: