Speech:Spring 2016 Kevin Soucey Log


 * Home
 * Semesters
 * Spring 2016
 * Proposal
 * Report
 * Information - General Project Information
 * Experiments - List of speech experiments

Week Ending February 9, 2016
today I logged in to check logs! today I logged in to check logs!

Today I went through Caesar in depth to see what types of experiments would be run. I looked at results as well in order to try and understand what they translate to. I did not run any test due to not being able to log in with my UNH credentials so I did this through the root user. I read through text files using nano to open and read them. My goals for the next week are to successfully log into my actual account and run a test train just to get a feel for what it is like and how long it can take. Also I will want to brainstorm with my team on how to proceed with a project proposal for the experiment group and what we will be working towards throughout the semester.

Week Ending February 16, 2016
Today i checked the train i had run in class on Wednesday. I created the language Model: Made a folder called LM. mkdir LM
 * Task:

Copied over the transcripts used from the corpus directory Prepared the transcript and executed the script that would build the language model. Prepared the transcript: % /mnt/main/corpus/switchboard/dist/transcripts/ICSI_Transcriptions/trans/icsi/ParseTranscript.perl trans_unedited trans_parsed

Copyied the script that creates the language model. % cp -i /mnt/main/scripts/user/lm_create.pl.

Executed the script: % ./lm_create.pl trans_parsed

I went to run the decoder but ran into error when trying to run the nohup run_decode_v2 as well as just run_decode.

I created the proper directory DECODE and copied the run_decode but when i went to actually run i would get an error unknown command. This stopped me for the night as i emailed my team asking for assistance but will continue tomorrow on the full decode process.

-               SYSTEM SUMMARY PERCENTAGES by SPEAKER

,-.     |                            hyp.trans                            | |-|     | SPKR    | # Snt # Wrd | Corr    Sub    Del    Ins    Err  S.Err | |-+-+-|     | sw2001b |   18    163 | 79.8   16.6    3.7   39.9   60.1  100.0 | |-+-+-|     | sw2001a |   14    101 | 82.2   15.8    2.0   50.5   68.3  100.0 | |-+-+-|     | sw2005a |   39    701 | 81.9   13.1    5.0   13.8   32.0   94.9 | |-+-+-|     | sw2005b |   67    613 | 64.9   24.3   10.8   29.2   64.3  100.0 | |-+-+-|     | sw2006b |   29    618 | 72.5   18.4    9.1    7.3   34.8  100.0 | |-+-+-|     | sw2006a |   33    455 | 85.9   11.0    3.1   15.4   29.5  100.0 | |-+-+-|     | sw2007a |   61    614 | 76.2   16.0    7.8   17.3   41.0   91.8 | |-+-+-|     | sw2007b |   60    861 | 81.3   14.6    4.1    9.4   28.1   90.0 |

|=================================================================|     | Sum/Avg | 1000  12903 | 74.7   18.4    6.9   17.6   42.9   96.6 | |=================================================================|     |  Mean   | 35.7  460.8 | 75.8   17.9    6.2   20.8   45.0   96.8 | | S.D.   | 16.3  229.0 |  7.6    5.7    2.8   11.3   13.5    4.1 | | Median | 33.5  459.5 | 76.5   16.7    5.9   17.3   41.4   98.6 | `-'

Today I was able to complete my full train and decode the data! The above is a snippet of the output. I was able to get in contact with some team members and discuss the errors I ran into yesterday and was able to resolve them which was great. I was able to create the hypothesis transcript with /mnt/main/scripts/user/parseDecode.pl decode.log ../etc/hyp.trans I was able to parse the data using sclite -r _train.trans -h hyp.trans -i swb >> scoring.log which created the file needed to analyze the results. Within etc. finally I used nano to go through the results and see how my train went.

Signed into view logs Signed in to view logs
 * One thing I found in the wiki is at project notes --> run a decode --> unseen data a script run_decode_v2 doesn't exist. we should remove possibly?
 * Results:


 * Plan:


 * Concerns:

Week Ending February 23, 2016
Signed in to view logs; Signed in to view logs;

Met with team today over skype. we discussed each persons role and went over how to complete it asking questions to one another. Started on the addExp script

pseudo code: Logic First step is to ask if  exists already; if no Execute createWiki_Experiment and createWiki_Subexperiment If yes Run createWiki_Subexperiment

What I have done so far with no tests yet.

Print "Have you created an experiment yet: y or n?"; $answer = ; # Holds user input

if ($answer == 'n' || $answer == 'N'){ system(”/mydocuments/UNHM/Spring2016/InternshipClass/createWikiExperiment.pl”); # call perl script createWikiExperiment.pl then system("/mydocuments/UNHM/Spring2016/InternshipClass/createWiki_Sub_Experiment.pl"); # call perl script createWiki_Sub_Experiment.pl }; else if ($answer == 'y' || $answer == 'Y'){												# If Experiment is already created system("/mydocuments/UNHM/Spring2016/InternshipClass/createWiki_Sub_Experiment"); #call perl script createWiki_Sub_Experiment.pl };

Goal by end of week is to have a test done to see how code runs with final version and then make error changes from there.

I actually was able to test what I have done. I made minor changes to syntax errors so I could execute as well as remove characters that were not allowed

Ive been researching and think I will try including the file paths of each createwiki and createwiki_sub as variables within addExp.pl This is my goal for next week as well as finding the proper way to execute these separate files.

Lastly looking long term it would be nice to add rules so someone cant break the code or input wrong information. Also it would be cool to streamline these two perl files into one without needing to ask for the experiment number if the user responds 'n' to the input request within addExp.pl

I believe these are attainable but will require extensive work


 * Task:


 * Results:


 * Plan:


 * Concerns:

Week Ending March 1, 2016
Met as a group and was told to change the way we approach the creation of addExp.pl.

-- I will be creating the perl script addExp.pl. This will streamline the future creations of Experiment links within the Wiki of this project as well as the creation of the sub experiments on the Wiki. The purpose of this script is so future users need only type one command on the command line to either create a whole new experiment as well as the first sub experiment within the Wiki as well as for creating any sub experiment on the Wiki. This script will utilize createWiki_Experiment.pl and createWiki_Sub_Experiment. I will create a whole new script which includes the content of the above two scripts I will wrap them into two functions. I will call those functions within an if statement dependent on user input; whether an experiment has been created or not. --
 * Proposal info

This will most likely be very close to the final code with minor tweaks and additions for simplicity as well as prevent user error.

print 'Have you created an Experiment already?'; $answer = <>;

if ($answer = 'y' or $answer = 'Y'){ createWiki_Sub_Exp; } else{ createWiki_Exp; createWiki_Sub_Exp; }

Met as a group today to discuss where we are at where we need to go and how we will succeed Signed into check logs
 * Task:


 * Results:


 * Plan:


 * Concerns:

Week Ending March 8, 2016
signed into view logs


 * Task:


 * Results:

sub ifNoCreateWiki_Sub_Exp { require LWP::UserAgent; require HTTP::Request; require HTTP::Cookies;

my $cj = HTTP::Cookies->new; $cj->clear;
 * 1) create cookie object

my $request;
 * 1) create request variable

my $ua = LWP::UserAgent->new; $ua->timeout(10); $ua->env_proxy; $ua->cookie_jar($cj);
 * 1) create user agent object
 * 1) give the user agent its cookie object

my $info = ''; my $lines = ''; my $token_start =''; my $token_end = ''; my $token = ''; my $substring =''; my $session_start = ''; my $session_end = ''; my $session ='';
 * 1) variable setup

print "please enter your username (prefix domain/ if not WILDCAT)->"; $cred = <>; chomp $cred;
 * 1) get the user's username and password

if (index($cred, "/") != -1) { ($domain, $user) = split('/', $cred); $domain = uc $domain; } else { $user = $cred; $domain = 'WILDCAT'; # domain for user login default to student WILDCAT }

print "Credentials: ". $domain. "/" . $user. "\n";

print "please enter your password->"; system('/bin/stty', '-echo'); # Disable echoing $pass = <>; system('/bin/stty', 'echo');  # Turn it back on chomp $pass; print "\n";

my $response = $ua->post('https://foss.unh.edu/projects/api.php?action=login&lgdomain='.$domain.'&lgname='. $user .'&lgpassword='. $pass .'&format=json');
 * 1) initiate login

if($response->is_success) {	#Get the token used to authenticate Login $info = $response->content; $token_start = (index $info, "token\":") + 8;	$substring = substr $info, $token_start, (length $info);	$token_end = (index $info, "\",\"");	$token = substr $info, $token_start, $token_end;	#END	#GET session id used to create a cookie	$session_start = (index $info, "sessionid\":\"") + 12;	$substring = substr $info, $session_start, (length $info);	$session_end = (length $substring) - 3; #the session info ends three characters from the end	$session = substr $info, $session_start, $session_end;	#END 	#Create the login Cookie	$cj->clear;	$cj->set_cookie(0, "oitw_wiki_session", $session, "/", "foss.unh.edu", 80, "path_spec", 0, 180, "discard");	#create the http request and add the cookie to the header to authenticate	$url = 'https://foss.unh.edu/projects/api.php?action=login&lgdomain='.$domain.'&lgname='. $user .'&lgpassword='. $pass .'&lgtoken='.$token.'&format=json';	$request = HTTP::Request->new(POST => $url); $cj->add_cookie_header($request); $response = $ua->request($request); #end #user is now logged in	#get the edit token (the edit token is needed to make changes) $response = $ua->post('https://foss.unh.edu/projects/api.php?action=tokens&format=json'); $info = $response->content; $edit_start = (index $info, "\":\"") + 3; $substring = substr $info, $edit_start, (length $info); $edit_end = (index $info, "\+"); $edit_token = substr $info, $edit_start, ( (length $substring) - 6); $edit_token = $edit_token."%2B%5C"; #get which experiment to make a sub-experiment in	$experimentToEdit = $ifNoHolder; chomp $experimentToEdit; #get the sub-experiment number $response = $ua->post('https://foss.unh.edu/projects/api.php?action=query&titles=Speech:Exps_'.$experimentToEdit.'&prop=links&pllimit=400&format=xml'); my $lines = ''; $done = false;#this is used to determine if we found the next valid sub-experiment number if($response->is_success) {		$info = $response->content; $info = "". $info;

if ( !($info =~ /page pageid=/) ) {			print "Experiment ". $experimentToEdit. " does not exist\n"; exit -1; }

$number = 0; while( ($info =~ /(Exps $experimentToEdit \d\d\d)/g) && ($done ne true) )#gets all the experiment numbers until we have found the next valid sub-experiment number {			#print "\n\n\$1: ".$1."\n\n"; $number = substr($1, -3, 3); }

#if there are no sub-experiments, set the next experiment to 001 if($number < 1) {			$number = 1; }		#else increase the number of the current sub-experiment by one to get the next sub-experiment number else {			$number = $number + 1; }	}	else {		die $response->status_line; }

$numStr = substr("000" . $number, -3, 3); # prefix with 0's

#get experiment name print "what is your sub-experiment's name?->"; $experiment = <>; chomp $experiment; $experiment =~ s/\s/%20/g; #make the edit to the experiment page $text = "\n*".$numStr." ".$experiment."\n"; $response = $ua->post('https://foss.unh.edu/projects/api.php?action=edit&format=json&title=Speech:Exps_'.$experimentToEdit.'&appendtext='.$text.'&token='.$edit_token); if (index($response->content, "error") != -1) {               print "ERROR: (" . $response->content . ")\n"; exit -1; }

#get experiment info print "please enter the author's name->"; $author = <>; chomp $author; $author =~ s/\s/%20/g;#replaces any whitespaces with html friendly %20 that preserve the whitespace since html will not print "Please enter a brief description of your sub-experiment->"; $description = <>; $description =~ s/\s/%20/g;#replace whitespaces ($sec,$min,$hour,$mday, $mon, $year,$wday,$yday,$isdat) = localtime; #get the date and time $year = $year + 1900;#convert from years since 1900 to current year $mon++;#months listed 0-11 add 1 to make listed 1-12 $date = $mon.'-'.$mday.'-'.$year;#formate the date #create the corresponding experiment page $text = "===%20Description%20===\nAuthor:\n".$author."\n\nDate:\n".$date."\n\nPurpose:\n".$description."\n\nDetails:\n\nResults:"; $response = $ua->post('https://foss.unh.edu/projects/api.php?action=edit&format=json&title=Speech:Exps_'.$experimentToEdit.' '.$numStr.'&appendtext='.$text.'&token='.$edit_token);

if (index($response->content, "error") != -1) {               print "ERROR: (" . $response->content . ")\n"; exit -1; }	print "\nYour sub-experiment number is: ". $numStr. "\nPlease go to Exps on Caesar and make a directory for this sub-experiment\n";

} else {	die $response->status_line; }

}	Add arguments to command line This would be the changes I made in version 4 to allow for arguments to be taken in from the command line $ARGV[0]; indicates weather an experiment has been created for example: "$ perl addExp.pl y" $ARGV[1]; indicates the experiment number one is using to automate creating the subexperiment for example:"$ perl addExp.pl y 0287" This will automate how the experiments and their sub experiments get created on the wiki with arguments from the command line
 * Plan:

if ($ARGV[0] = 'y' or $ARGV[0] = 'Y')
 * 1) print 'Have you created an Experiment already?';
 * 2) $answer = <>;
 * 3) if ($answer = 'y' or $answer = 'Y'){

createWiki_Sub_Exp; } else{ createWiki_Exp; ifNoCreateWiki_Sub_Exp; }

#print "please enter the main experiment number (Ex: 0268)->"; #Allows for the arguement taken from the command line $experimentToEdit = $ARGV[1]; chomp $experimentToEdit;
 * 1) get which experiment to make a sub-experiment in
 * Concerns:

Week Ending March 22, 2016
Over the break I was able to contact morgan who created the scripts createwikiExperiment and sub experiment. I asked him a few questions about them and what he did as well as asked about the HTTPS connection to the wiki. He explained to me that unfortunately adding HTTPS throws errors which will require more work around then simply adding HTTPS to all of the html calls within the scripts. When trying to add the command line arguments to these scripts I kept running into error "Bad token". this is because of the addition to HTTPS. I thought the scripts ran with https but I noticed not all the html calls were switched to https which is why it was successful. Going forward I believe the only fix to this would be to make the site secured. I will need to go through the scripts later in the week to make the corrections and finalize the command line argument addition to the code. Signed into view logs
 * Task:
 * Results:

Signed into view logs Can't open y: No such file or directory at finalAddExp.pl line 234. Can't open 0282: No such file or directory at finalAddExp.pl line 234.
 * Plan:
 * Concerns:

These are the errors I kept running into when trying to take in command line arguments. Things I tried.

my ($isOneMade, $arg2Placeholder) = @ARGV; This was for possibly storing the arguments in an array to pass into the script. It seemed as if this worked but the one thing I feel is happening is that its looking for a file with that

if ($isOneMade = 'y' or $isOneMade= 'Y'){

createWiki_Sub_Exp; } else{ createWiki_Exp; ifNoCreateWiki_Sub_Exp; }

This is the logic for how to run this script. It should take the command line argument and if $ARGV[0] is null it would just run as if there is a no if there is the y like there should be it would execute createWiki_Sub_Exp. for some reason I feel as if its not reading the y properly.

$experimentToEdit = $arg2Placeholder;

This holds the experiment number that is already created in order to add the sub experiment number.

./2FinalAddExp.pl: Permission denied. [kcq37@caesar ~]$
 * [kcq37@caesar ~]$ ./2FinalAddExp.pl y 0282
 * Lets go time to talk to systems more posted in a bit.

Week Ending March 29, 2016
Today was a meeting day. I now need to verify why the transition of foss over to HTTPS broke the code. Ive tried HTTPS::Request $ua = LWP::UserAgent->new(ssl_opts => { verify_hostname => 1 });

I e-mailed Jonas about getting the contact info of the person in charge of transitioning foss over to HTTPS in order to gain knowledge of how this happened. Finally I did some work with taking in arguments from the command line as I wait on the contact info. these are the changes.

Logic my ($isOneMade, $arg2Placeholder) = @ARGV;

if ($isOneMade = 'y' or $isOneMade= 'Y'){

createWiki_Sub_Exp($arg2Placeholder); } else{ $ifNoHolder = 	createWiki_Exp; ifNoCreateWiki_Sub_Exp($ifNoHolder); }

New variables and passing variables from one sub to the next

Added return $number; to pass the experiment number within sub createWiki_Exp;

$experimentToEdit = $_[1]; This is within createWiki_Sub_Exp and is the placeholder for taking in the argument at position 1 from the command line

$experimentToEdit = $_[0]; This is within sub ifNoCreateWiki_Sub_Exp { and is for passing the new experiment number that was created within createWiki_Exp so it doesn not need to ask the user for imput it just grabs the number and creates a sub experiment.

signed in to view logs
 * Task:

Signed into view logs
 * Results:


 * Plan:

What im going to need to do is basically re write the entire scrip to match the proper format for and HTTPS request. I will need to re do the login, how I can get the edit token, re do how we create an experiment and sub experiment to verify that is correct. I know it is an issue with the URL’s because I am able to login; it appears the cookie is coming in correct but the token is completely messed up and it is either the URL needs to be reworked so that the token is in proper format or because it is not saving the proper user so when going to edit, the user has no privilege to do so. Ive been doing a lot of research and have come to the conclusion that unless I can work on this for a couple weeks there is no way it will be completed this semester.

[kcq37@caesar ~]$ perl https.pl Have you created an Experiment already?y please enter your username (prefix domain/ if not WILDCAT)->kcq37 Credentials: WILDCAT/kcq37 please enter your password-> ï»¿{"warnings":{"tokens":{"*":"Action 'edit' is not allowed for the current user"}},"tokens":[]}[kcq37@caesar ~]$


 * Concerns:

Week Ending April 5, 2016
Worked on making a world class baseline for the team... sent an email on my thoughts... will keep doing more

Signed in to view logs addExp.pl v3: consolidated the login function to just be interpreted literally rather than having it baked into each sub-routine. Based on Matt tested both of the flags again and they are functioning as you will see when you go to the experiments page. /mnt/main/scripts/user/History/addExp/4/ and replacing cur/
 * Task:
 * Results:

Signed in to view logs
 * Plan:
 * Concerns:

Week Ending April 12, 2016
Signed into view logs
 * Task:


 * Results:

Signed into view logs
 * Plan:


 * Concerns:

Week Ending April 19, 2016
signed into view logs
 * Task:

Today I debugged addExp for a few hours and traced it back to the cookie and its creation. I emailed Matt with the files and I tried to move them to the proper directories and got an error which was logged. I plan on meeting with matt and tom over hangouts of some sort tomorrow to discuss what needs to happen with this. I am concerned the change to https was more than it seems.
 * Results:

Working with Tom and Matt we were able to find the port number needed to change to port 443 for all instances within addExp.pl due to the change over to https. When I went to go debug my code I got an error message "500 Cant connect to foss.unh.edu" which led me to believe it couldn't connect to the local server. I ran an older version of my code that had print statements for debugging that couldn't even run so something is not working properly externally to the code itself. Will email Matt with my findings.
 * Plan:
 * Concerns:

Week Ending April 26, 2016
Signed into view logs.
 * Task:


 * Results:


 * Plan:


 * Concerns:

Week Ending May 3, 2016
Sign into view logs sign ionto view logs
 * Task:
 * Results:


 * Plan:

Isn't the report with the results also supposed to be on the wiki? I think we just modify the wiki, and grab a section from our own teams? There is one (class-wide) on the wiki, and one that would normally be handed in for the competition judgement portion. I’m not sure how the lack of a competition changes things for us though. I honestly don’t know exactly. What we’re doing is the right place to start, but we should probably get a better idea what Mike (Jonas) expects of us on Wednesday.
 * Concerns:

Week Ending May 10, 2016
Signed into view logs
 * Task:


 * Results:

Signed into view logs Wrote up the portion of our results for what the experiment grtoup did
 * Plan:
 * Concerns: