Speech:Spring 2012 Modeling Group

From Openitware
Jump to: navigation, search


Groups


Group Members


SUCCESS

THE MODELING GROUP HAS RUN THE TRAIN!!!!

Group Logs

Experiment #: 0001

Server: Automatix

NOTE: Run chmod -R g+w 0001 after running a script that creates directories/files/etc., anything new.


  • We will be using the 0001 experiment from Group 1 of the April 24 group.
  • Changed the make_feats.pl variable for directory back to wav
  • Jon copied over correct feats files and wav files as we forgot to grab those
  • running make_feats.pl - successful
  • Ran RunAll.pl received this error:

MODULE: 00 verify training files O.S. is case sensitive ("A" != "a"). Phones will be treated as case sensitive.

   Phase 1: DICT - Checking to see if the dict and filler dict agrees with the phonelist file.

WARNING: The phonelist (/mnt/main/Exp/0001/etc/0001.phone) does not define the phone SIL (required!)

   Found 130614 words using 43 phones

WARNING: This phone (SIL) occurs in the dictionary (/mnt/main/Exp/0001/etc/0001.dic), but not in the phonelist (/mnt/main/Exp/0001/etc/0001.phone)

   Phase 2: DICT - Checking to make sure there are not duplicate entries in the dictionary

Can not open listoffiles (/mnt/main/Exp/0001/etc/0001_train.fileids) at /mnt/main/Exp/0001/scripts_pl/00.verify/verify_all.pl line 203. Something failed: (/mnt/main/Exp/0001/scripts_pl/00.verify/verify_all.pl)

    • Basically what this error is telling us that the paths for all of the executable files being called by RunAll script are wrong so we're going to have to change them all individually
      • It appears that the executable files are using the $0 variable so we're going to need to change that to the correct path (hopefully this is the solution)


  • (Ted) Hey guys, if there's anything that I can help with could you please let me know? I'm not too sure what I should do. You can email me at edpulkow1@gmail.com or leave a message under this note.
  • (Jonathan)Ted, we need to go through the Perl scripts that the run all script is calling to find out what directories in those scripts are looking for, if we can figure that out and fix them we can get the training done. If you have a questions or need to bounce off any ideas, email me at jwz42@wildcats.unh.edu or leave a message under this note. I will be working on the scripts all day Sunday, so I will be on this page.
 use lib catdir(dirname($0), updir(), 'lib');
  • (Jonathan)The $0 in the verify_all.pl needs to be changed to /mnt/main/Exp/0001/script_pl/00.verify - but the read-only is set on the script. I tried to run it with root. Still did not work anyone with Perl knowledge know how to turn of the ready only feature on the script. I also tried to copy the perl script and then make changes. It says I can make the file. Any ideas?
  • (Aaron) Jonathan, is it possible to copy the code of the script to a new one with the same name, and change the info in the new script?
  • (Ted) Using WinSCP, I managed to be able to edit the verify_all.pl script. I did this by copying the file from caesar (automatix) then using notepad within Windows to change '$0' to '/mnt/main/Exp/0001/scripts_pl/00.verify'. After modifying the script I copied it back to automatix, overwriting the old file.
  • Ted, not sure why you changed it to that directory, did it work for you? By the way, you can use vi editor to edit the scripts. just do vi scriptname.pl to enter the editor.
  • I was unable to edit the script in vi because I couldn't get around the read-only problem. Jonathon told me that the $0 in the line mentioned just above needed to be changed to that directory. Sorry it was supposed to be '/mnt/main/Exp/0001/scripts_pl/00.verify' and not '/mnt/main/Exp/0001/script_pl/00.verify'. I'm unsure if this correction actually works because RunAll.pl is getting a compilation error around line 48 and I don't know how to fix it.
  • (Jonathan) If you look up what $0 is in Perl, it says it is looking for the local directory of the script. But, because we moved the scripts we have to give a new path to that variable. Can you post the error message on here, so we can document it?
  • I used the following command and received the following error:
   automatix 0001/scripts_pl> perl RunAll.pl
   Configuration (e.g. etc/sphinx_train.cfg) not defined
   Compilation failed in require at RunAll.pl line 48.
   BEGIN failed--compilation aborted at RunAll.pl line 48.
  • (Jonathan) It is looking for the sphinx_train.cfg file in the etc directory at root level. We need to find line 48 and change the path to /mnt/main/Exp/0001/etc. The 48th Line in RunAll.pl is use SphinxTrain::Config; We need to find where one of the scripts is putting the path into SphixTrain and change it. It might be in RunAll.pl, but I do not know.

caesar:/mnt/main/Exp/0001 # ./scripts_pl/RunAll.pl MODULE: 00 verify training files O.S. is case sensitive ("A" != "a"). Phones will be treated as case sensitive.

   Phase 1: DICT - Checking to see if the dict and filler dict agrees with the phonelist file.

WARNING: The phonelist (/mnt/main/Exp/0001/etc/0001.phone) does not define the phone SIL (required!)

       Found 130614 words using 43 phones

WARNING: This phone (SIL) occurs in the dictionary (/mnt/main/Exp/0001/etc/0001.dic), but not in the phonelist (/mnt/main/Exp/0001/etc/0001.phone)

   Phase 2: DICT - Checking to make sure there are not duplicate entries in the dictionary

Can not open listoffiles (/mnt/main/Exp/0001/etc/0001_train.fileids) at /mnt/main/Exp/0001/scripts_pl/00.verify/verify_all.pl line 203. Something failed: (/mnt/main/Exp/0001/scripts_pl/00.verify/verify_all.pl)

  • 0001_trains.pl was named wrong

./scripts_pl/RunAll.pl MODULE: 00 verify training files O.S. is case sensitive ("A" != "a"). Phones will be treated as case sensitive.

   Phase 1: DICT - Checking to see if the dict and filler dict agrees with the phonelist file.
WARNING: The phonelist (/mnt/main/Exp/0001/etc/0001.phone) does not define the phone SIL (required!)
       Found 130614 words using 43 phones
WARNING: This phone (SIL) occurs in the dictionary (/mnt/main/Exp/0001/etc/0001.dic), but not in the phonelist (/mnt/main/Exp/0001/etc/0001.phone)
   Phase 2: DICT - Checking to make sure there are not duplicate entries in the dictionary
   Phase 3: CTL - Check general format; utterance length (must be positive); files exist
   Phase 4: CTL - Checking number of lines in the transcript should match lines in control file
   Phase 5: CTL - Determine amount of training data, see if n_tied_states seems reasonable.
       Total Hours Training: 0.206707478632478
       This is a small amount of data, no comment at this time
   Phase 6: TRANSCRIPT - Checking that all the words in the transcript are in the dictionary
       Words in dictionary: 130611
       Words in filler dictionary: 3
   Phase 7: TRANSCRIPT - Checking that all the phones in the transcript are in the phonelist, and all phones in the phonelist appear at least once

Something failed: (/mnt/main/Exp/0001/scripts_pl/00.verify/verify_all.pl)

  • (Ted) I tried entering the phone SIL into 0001.phone to solve the current issue, but I then came up with another issue:
    • (Jonathan) Ted when you said you enter the phone SIL what do you mean?
    • (Ted) I just typed it into the phone list 0001.phone.
MODULE: 00 verify training files
O.S. is case sensitive ("A" != "a").
Phones will be treated as case sensitive.
   Phase 1: DICT - Checking to see if the dict and filler dict agrees with the phonelist file.
       Found 130614 words using 43 phones
   Phase 2: DICT - Checking to make sure there are not duplicate entries in the dictionary
   Phase 3: CTL - Check general format; utterance length (must be positive); files exist
   Phase 4: CTL - Checking number of lines in the transcript should match lines in control file
   Phase 5: CTL - Determine amount of training data, see if n_tied_states seems reasonable.
       Total Hours Training: 0.206707478632478
       This is a small amount of data, no comment at this time
   Phase 6: TRANSCRIPT - Checking that all the words in the transcript are in the dictionary
       Words in dictionary: 130611
       Words in filler dictionary: 3
   Phase 7: TRANSCRIPT - Checking that all the phones in the transcript are in the phonelist, and all phones in the phonelist appear at least once
Bareword found where operator expected at /mnt/main/Exp/0001/scripts_pl/01.vector_quantize/slave.VQ.pl line 52, near "/mnt/main"
       (Missing operator before ain?)
syntax error at /mnt/main/Exp/0001/scripts_pl/01.vector_quantize/slave.VQ.pl line 52, near "/mnt/main"
Execution of /mnt/main/Exp/0001/scripts_pl/01.vector_quantize/slave.VQ.pl aborted due to compilation errors.
Something failed: (/mnt/main/Exp/0001/scripts_pl/01.vector_quantize/slave.VQ.pl)
  • I am now getting an error on line 52 of /mnt/main/Exp/0001/scripts_pl/01.vector_quantize/slave.VQ.pl. I'm not sure how to fix this. The line in question is listed below:
use lib File::Spec->catdir(dirname(/mnt/main/Exp/0001/scripts_pl/01.vector_quantize), File::Spec->updir(), 'lib');
  • After fixing Teds error. All of the scripts that RunALL.pl calls had to have their dirname() changed form the full path of /mnt/main/Exp/0001/scripts_pl/Script Directory/script to $0 Once this was changed the RunAll.pl script went through all the modules and the HTML shows Passed. Now we need to run the DECODE.
  • (Ted) If we've run the train, what else is there to do?
  • We need to run the decode.
    • Copied lm_create.pl to LM
      • cp -i /mnt/main/scripts/user/lm_create.pl . Where did you run this command?
    • When trying to run lm_create.pl 0001_lm.trans &> lm_create.log - receive error: Invalid null command
    • I ran the script without piping it to a log file and this is what I got
      • Generating a word Frequency file . . .

sh: /mnt/main/home/sp12/bpa22/speechtools/CMU-Cam_Toolkit_v2/bin/text2wfreq: No such file or directory

Generating Vocab file . . .

sh: /mnt/main/home/sp12/bpa22/speechtools/CMU-Cam_Toolkit_v2/bin/wfreq2vocab: No such file or directory

Generating ID 3 Gram file . . .

sh: /mnt/main/home/sp12/bpa22/speechtools/CMU-Cam_Toolkit_v2/bin/text2idngram: No such file or directory

Generating Language Model ( arpa Format )

sh: /mnt/main/home/sp12/bpa22/speechtools/CMU-Cam_Toolkit_v2/bin/idngram2lm: No such file or directory

Generating Language Model ( binary Format )

sh: /mnt/main/home/sp12/bpa22/speechtools/CMU-Cam_Toolkit_v2/bin/idngram2lm: No such file or directory

Generating the CMU binary form (DMP)

sphinx_lm_convert: error while loading shared libraries: libsphinxbase.so.1: cannot open shared object file: No such file or directory

  • So as you can see it's trying to go to my directory instead of root so we need to change the script to not try to go in my username
    • Changed the lm_create.pl script
      • Changed variable $folder from "~/speechtools/CMU-Cam_Toolkit_v2/bin/"; to "/mnt/main/root/tools/CMU-Cam_Toolkit_v2/bin/";
    • Running script again without piping to a log file goes almost all the way through, new error is:
      • sphinx_lm_convert: error while loading shared libraries: libsphinxbase.so.1: cannot open shared object file: No such file or directory
    • Had to comment out last two prints in lm_create.pl
    • Had to change premissions in /mnt/main/local/bin/ to allow access to 790
    • Still get error: /usr/local/bin/sphinx3_decode: error while loading shared libraries: libs3decode

r.so.0: cannot open shared object file: No such file or directory

  • Run ./run_decode.pl 0001
    • When I look at the log I see this error: /usr/local/bin/sphinx3_decode: error while loading shared libraries: libs3decode

r.so.0: cannot open shared object file: No such file or directory **Ted can you please move the file you moved to fix it in the LM directory and get it to fix this error in the DECODE directory.

    • How did you get that error? When I ran run_decode.pl from /mnt/main/Exp/0001/DECODE from automatix, I received this error:
  sh: ./decode.log: Permission denied
    • When I ran run_decode.pl from /mnt/main/Exp/0001/DECODE from caesar, it recorded something different than your error. To prevent the log that was recorded when you ran the script, I renamed it to decode_old.log. My error was recorded in the new decode.log when I ran it from caesar.
    • Also the only file that I remember fixing was the lm_create.pl script which has already been updated in the /mnt/main/Exp/0001/LM directory.
    • The error I see is FATAL_ERROR: "mdef.c", line 680: No mdef-file at the bottom. Looks like it pretty much ran all the way. I'm going to look into to see if that .c file is there


Modifications

  • Changed script
  • 01.vector_quantize - slave.VQ.pl
    • Changed the $0 in use lib File::Spec->catdir(dirname(/mnt/main/Exp/0001/scripts_pl/01.vector_quantize)
  • 02.falign_ci_hmm - slave_convg.pl
    • Changed the $0 in use lib File::Spec->catdir(dirname(/mnt/main/Exp/0001/scripts_pl/02.falign_ci_hmm)
  • 03.force_align - slave_align.pl
    • Changed to $0 in the lib File::Spec->catdir(dirname(/mnt/main/Exp/0001/scripts_pl/03.force_align)
  • 04.vtln_align - slave_align.pl
    • Changed to $0 in the lib File::Spec->catdir(dirname(/mnt/main/Exp/0001/scripts_pl/04.vtln_align)
  • 05.lda_train - slave_lda.pl
    • Changed to $0 in the lib File::Spec->catdir(dirname(/mnt/main/Exp/0001/scripts_pl/05.lda_train)
  • Had to change all the scripts back.