Speech:Spring 2013 Scott Adie Log


 * Home
 * Semesters
 * Spring 2013
 * Proposal
 * Report

Week Ending February 5th, 2013
Gain access to STEM to configure the live version of SpEAK as well as fix the database Work on implementing the ability to have a login system in SpEAK Work on implementing the ability to add and edit users Work on user-based permissions Rewrite the SQL script to better support the site
 * Task:

February 6, 2013 Fixed issues with logins and user experience. Also many issues that occurred when other users committed files.
 * Results:

Code Changes:


 * https://code.google.com/p/speak/source/detail?r=158
 * https://code.google.com/p/speak/source/detail?r=159
 * https://code.google.com/p/speak/source/detail?r=160

February 4, 2013

Got logins working. Permissions are not being checked for yet.

Code Changes:


 * https://code.google.com/p/speak/source/detail?r=143

February 3, 2013

Worked on implementing user logins and sessions. Very difficult so far. No code committed as it would make it impossible for others to work.

February 2, 2013

Created dbUsers.php to handle the users table in the database. Also worked on admin.php to allow the creation of users. Passwords for users are salted and hashed before being stored in the database. The salt is a 64 character string generated by mcrypt using URANDOM. The password is then set to an sha1(plaintext_password + salt), where + means concat

Code Changes:


 * https://code.google.com/p/speak/source/detail?r=142

January 31, 2013

Rewrote the SQL script which now only creates two tables. Users and Experiments. The other tables were not required and did not fit the needs of the site correctly. After modifying the script, many features of the site did not work (Note: Most of the site did not work from the start, due to the mysql server being non-existant) so I modified many files to fix these issues. Some areas sitll do not work with the new database structure but adapting it will not be an issue. Code changes:
 * https://code.google.com/p/speak/source/detail?r=141
 * https://code.google.com/p/speak/source/detail?r=140
 * https://code.google.com/p/speak/source/detail?r=139

Learn about how to implement a simple yet secure login system for SpEAK. We already have a users table, so it is simply managing the sessions correctly and safely. Before getting a login system working fully, having a function system in place to create users is required, so that will come after fixing the SQL. Timeline will be like: SQL -> User Management -> Login -> Permissions
 * Plan:

I've never implemented a login system in PHP before, I've usually piggy-backed on other systems to handle it for me. I want to make sure this system is secure but simple enough for users to follow and not be affected adversely.
 * Concerns:

Week Ending February 12, 2013

 * Task:
 * User permissions
 * User management
 * More experiment fields


 * Results:

February 6, 2013

Helped Marc and Nick get better equiped with everything. Worked on the edit page with Marc.

February 8, 2013

Read logs

February 10, 2013

Added more fields for experiments on the add page. Not functional in the DB yet.

February 12, 2013

Added the database fields for the new fields in the add page. Also read logs Tackle each task one by one. They are all mostly in their own subsystems so conflicts should not result. Handling user permissions on a per page basis, wondering if 5 fields in the database is enough. Getting speak to run properly on stem, it's been causing a lot of issues due to mysql problems.
 * Plan:
 * Concerns:

Week Ending February 19, 2013

 * Task:

Continue working on our four week plan, which includes getting permissions to work on a per-user basis, as well as getting all the fields in the new add page to work.


 * Results:

2/13/13

Showed nick how to get google reader setup with the wiki to make it really easy to stay up to date on the logs. Not directly related to the project but took place during project time.

2/15/13

I read logs today, You can see the process for setting up google reader on Nick Regan's wiki page.

2/17/13

Work began on fixing up the database controllers to handle the new SQL fields, There are many functions which require updating, so I began working on that. All previous connections to the database relied on receiving the older columns, this needs to be changed so that the new columns are respected.

2/19/13

Finished up getting the new fields to function in the site. Everything its back to as it was before adding the fields, expect it's better now.


 * Adding experiments works
 * Viewing experiments works
 * Searching... No idea

THE CODE:

https://code.google.com/p/speak/source/detail?r=172

I did not have to update the database anymore, it looks like those days are behind us. I much prefer the way our current structure is set up, and the only thing we should ever have to touch the database for in the future is adding fields that the experiments require. We want to look into having the ability to attach files (sph files, wav files, log files), but that can take a lot of HDD space, and we are not sure if we want to delve into that just yet.

I looked into using LDAP with php, there are native functions, and there are lots of resources online for getting it to work, however, I'm not sure what the schools policy would be on allowing this, but its okay, we were only meant to investigate the issue, not implement. It seems possible, but for the time we have at the moment, not suggested.


 * Plan:

I will be working on getting the new fields in the add page to work correctly this week. This will involve editing the database again (which will require everyone to remake their databases), but after this there should not be any more modifications to the database.

Meet with Marc over Skype or Google Hangouts to solve issues or questions he has.


 * Concerns:

If we keep making changes to the database it could get very annoying for everyone to maintain their site. The plan is to only do it when required, OR make incremental scripts so users on one version of the database can run scripts to catch them up without losing their data.

Week Ending February 26, 2013

 * Task:
 * Get a simple non-install version of svn put on the stem server
 * Get speak running on stem ready for use
 * Document entire source code thus far, for help with future semesters.
 * Fix some errors noticed in presentation last week

2/20/2013
 * Results:

Installing subversion was not as simple as I thought it would be. Finding an actual zip containing svn binaries was easy though, a stack overflow question contained a lot of information.

StackOverflow Link

Initially this was okay, but I didn't want to have to manually call the executable every time by its path. I do not have root to the server, so I did research and came to find that there is a way to modify the path my editing the path variable via command line. it was simply PATH = C:\new\path\to\add;%PATH%. HOWEVER! This only does it for that command session. I spent more time researching, and came to find the command setx, which allows you to set the path without needing admin privs. doing svn update is now possible on stem!

2/23/2013

Read logs

2/25/2013

Read Logs

2/26/2013

I went through all our source files which contain PHP, and documented them. This involved an entire code review and I was able to delete some unneeded things that were still in the source. This should allow future semesters to have a much clearer understanding of the codebase, meaning they can start working on it in less time than it would have taken without this documentation.

I was able to fix the issue that popped up in class where you could not search by user. This did not work previously because the SQL command was still referencing the old database structure. This is now fixed.


 * Plan:

Nothing other than tackle each issue one-by-one
 * Concerns:

Getting SVN to run in the path without admin access.

Week Ending March 5, 2013

 * Task:


 * Work on implementing user-based permissions
 * Work on implementing the edit user page


 * Results:

March 01, 2012

Read logs.

March 03, 2012

Worked on implementing user based permissions. We have a variable that can be accessed on each page to tell if the user has a permission now. This needs to be brought into if statements for validation, which would deny the user if they don't have the correct privileges.

March 04, 2012

Read more logs

March 05, 2012

Implemented some permission things.


 * All permissions are now working, you can only access what you have permission for
 * Worked on the edit user page, At the moment if you try and edit a user, all their current permissions will be blown away. This is a bug and will get fixed. Editing email, last name, and first name work. Only admins or people with the Modify Users permission can modify users other than themselves (as well as grant more permissions). This means a user cannot grant themselves more permissions, an admin or an authorized person must.

code changes:

https://code.google.com/p/speak/source/detail?r=189


 * Plan:

I would like to implement the two tasks above, which will bring us closer to a secure site that users are treated with their proper permissions
 * Concerns:

Managing the permissions, it could get difficult depending on what sort of page we are at, since the permission check needs to be done on almost all pages

Week Ending March 12, 2013

 * Task:


 * Results:


 * Plan:


 * Concerns:

Week Ending March 26, 2013

 * Task:


 * Get a virtual machine setup to be able to run trains
 * Google Hangout w/ group Sunday at 7PM to go over the current state and how to run everything.


 * Results:


 * So far I've been just looking at the old documentation trying to find out how to install a virtual machine for this. I hate to say it but the wiki is absolutely terrible for searching. It took me over 30 minutes just to find setup instructions (Which will probably not even work) that were made last year. I thought there were more up to date instructions from groups this semester, but I am unable to find them. The wiki may have a good structure, but it's disorganized (in it's current state at least). Finding specific information from logs is near impossible. The search function does not search the bodies of wiki pages, so there's that. The only way I've found to have a decent search is to go to google and do

site:foss.unh.edu


 * As expected, attempting to setup a VM did not work out so well. I Created a VM with Fedora 18 on it and it will not even isntall. It simply shows a desktop background when loading the VM and just sits there. So I gave up on a VM. Then i decided to try and get it running on my laptop since I run a flavor of Linux on there. I followed the previous semesters guide to the T, but it did not work. I researched and it seems the guide is completely wrong about the location of where the folders should be. However, after even attempting to manually fix them, it still would not work so I have temporarilly given up on this. I am running Linux Mint for anyone wondering. The guide fails to explain that you need to separately download and compile sphinxbase, which has the header files and includes not found in sphinx3.0.8 package. I may end up giving up on this totally and just sticking to using the schools servers for everything.

text2wfreq /mnt/main/local/bin/ INFO: fe_sigproc.c(771): Will not use double bandwidth in mel filter INFO: wave2feat.c(139): /mnt/main/Exp/0030/wav/sw2001B-ms98-a-0012.sph ERROR: "wave2feat.c", line 655: Cannot read /mnt/main/Exp/0030/wav/sw2001B-ms98-a-0012.sph FATAL_ERROR: "wave2feat.c", line 90: error converting files...exiting miraculix Exp/0030> ls wav/
 * Attemted to follow a full train setup on Google+ using [this guide] but lots of errors occured. First, I received the following:
 * I fixed by editing my path and adding
 * to it. This worked. We then moved on through the guide to the point of Generate Feats data, but the following error occured
 * Created experiment [0030]


 * Fixed issues above with some investigating. Sox is installed, but not in the path. It was installed to /mnt/main/local/bin . I symbolically linked the binaries in that directory to /usr/local/bin and the command was found, however, the shared libraries were not available. I had to edit /etc/ld.so.conf and add /mnt/main/local/lib/ to/ the list of shared directories and then run ldconfig to get them to be available. Sox now runs on Miraculix.


 * Created an ran Exp 0037, after my fixes above, got a new error during running the train. See exp for details.


 * Plan:


 * Try and get a train to run on a virtual machine. If it fails or the VM install fails, debug it until I cannot any further.


 * Concerns:


 * From peoples logs, it seems the instructions for getting everything set up on a VM does not work, I fear this will take up a lot of time getting it to work.

Week Ending April 2, 2013

 * Task:
 * Have all steps of the process completed by next class


 * Results:
 * Began working on experiment 0059, still in the process as I was interupted during my initial experiment by something exterior. I've had to add several words manually to the dictionary so far which is an issue, I believe there is a way to add items to the master dictionary so I will do this eventually to help others and myself in the future.


 * Gave up on experiment 0059 due to errors with the dictionary. Started experiment 0060 and the experiment seemed to finish. I'm not sure if it was successful or not but the process completed. Output will be posted to an experiment page soon.


 * Ran the language model and decode on the 0060 experiment. The language model ran fine without issue, but the decode resulted in some issues with multiple repeated lines. Thankfully, there was information in tutorial on how to fix each individual error.


 * Read the glorious logs.


 * Plan:

Finish the language model and decode a mini train
 * Concerns:

Issues regarding problems which could occur during running the following steps. This is the first time i'm doing it.

Week Ending April 9, 2013
Run a train on the first five hours
 * Task:


 * Results:
 * Been reading them logs like there is no tomorrow
 * Worked on setting up a train for the first five hours, LOTS of words and things are missing from the dictionary and phonelist, it is going to be quite the challenge to get this working


 * Plan:
 * Work on getting a train
 * Concerns:
 * Missing words in the dictionary

Week Ending April 16, 2013
Work on a regular expression for genTrans which will properly filter out unwanted information
 * Task:


 * Results:

4/10/2013

$string =~ s/regex/replacement/g; [\[<].*?[\]>][ ]? $string =~ s/[\[<].*?[\]>][ ]?//g;

http://codepad.org/fBQpOnqV

4/12/2013

Read logs

Use previous knowledge of regular expressions to properly filter out unwanted information.
 * Plan:

Matt mentioned that not everything between brackets should be filtered out, and that certain regular expressions were already present in the genTrans3.pl file which he uploaded.
 * Concerns:

Week Ending April 23, 2013
Get the group train to run with a significantly better score than the pervious attempts
 * Task:

4/19/13
 * Results:

Read logs

4/21/13

Added words to the dictionary document that tyler made on Google docs. Went through the list looking up every line in the CMU dictionary site. After that I worked on partial words, where I would take the line, and figure out words that had that sequence of characters, look up that word in the dictionary, and then take that specific part of speech out of the pronunciation to give the partial word a mapping.

4/22/13

Added significantly more words to tylers dictionary. This took up all of my time dedicated for today on this class.

Communicate with the team for instructions on the progress of this train. The expectations for this week were not really given, so we assumed to attempt a better score on the first 5hr
 * Plan:
 * Concerns:

Week Ending April 30, 2013
Work on the final paper
 * Task:


 * Results:

4/24/2013

Created the initial report on Google docs

4/26/2013

Read logs

4/27/2013

Made adjustments to the final report draft

4/29/2013

Read logs


 * Plan:

@ Task
 * Concerns:

Getting everyone to collaborate properly on the document

Week Ending May 7, 2013
Finish up the final report
 * Task:


 * Results:

5/1/2013

Discussed the report with the group and the process we will take to finish it

5/3/2013

Read logs

5/5/2013

Worked on the report and made few adjustments

5/6/2013

Read logs


 * Plan:

See task...
 * Concerns:

Having the report in a finished state in the given time