Speech:Spring 2018 Systems Group


 * Home
 * Semesters
 * Spring 2018
 * Proposal
 * Report
 * Information - General Project Information
 * Experiments - List of speech experiments

Groups

 * [Systems Group]
 * Experiment Group
 * Data Group
 * Modeling Group
 * Software Group

Group Member Logs

 * Yashna Giri
 * Camden Marble
 * Christopher Nobrega
 * Daniel Rubin

System Configuration
Known configuration as of 03/13/2018 [Outdated -- ignore]
 * https://foss.unh.edu/projects/images/f/f0/ServerStatus_03132018.pdf

Software Installation Procedure

 * Before you begin
 * Do not install software willy-nilly. The purpose of the Systems Group is to support the endeavors of the Speech Project as a whole, as such, any software/tool that is installed must have the least amount of impact to system performance with the most return value (bang for buck). You should do a decent amount of research on the software/tool and then gain approval from Professor Jonas to utilize said software. Once you have this approval, below is how you install it.


 * Step 1
 * You need to ensure that the Server you are installing on does not 'infect' the other servers with the software you are adding. This is so that if the installation breaks something, you haven't destroyed the entire system. We will be using Rome as the server in the following example.


 * Disconnect Rome from Caesar and the Drones.
 * Rome is connected to Caesar by a Mount and a Symbolic Link. You can confirm the Mount configuration by doing user@rome ~]$ more /etc/fstab within you should see caesar:/mnt/main       /mnt/main               nfs     defaults        0 0 . To disconnect this, you can simply do  umount -a and then confirm it has been removed by first doing  cd /  to get to the top directory, then  cd /mnt  within, you should no longer see that /mnt/main is mounted.
 * Next, you need to ensure that the Symbolic Link is removed. The symbolic Link (aka Softlink) will be located on Rome at /usr/local this points back to /mnt/main/local to 'reproduce' the files at the /mnt/main/local location. You should google this to understand it better if you don't currently. -- When on Rome, do cd /usr then do ls -l (that's LS - L). If there is a symbolic link already active, it will read as root root   15 Mar  3  2016 local -> /mnt/main/local if there is no link it will simply read as root root  4096 Apr 15  2017 local.  In order to unlink this, simply type rm nameoffolder while in the appropriate directory, or use the full path.


 * Step 2
 * You need to disconnect Rome from the local network and ensure it is only connected to the outside world. This is best done by physically disconnecting the Ethernet cables on the back. There is one Ethernet Cable that plugs into the switch rack on the right via a currently Orange Ethernet Cable. This is the outside world connection. Ask Jonas about this as we're not really suppose to use that except for temporary circumstances such as installations.


 * Disconnect the local internet connection so as to isolate Rome from the System.


 * Step 3
 * You need to capture a snapshot of the current state of the installation locations for Linux. There are roughly 15 of them. If i don't write one later (and I'll add it below if I do) you will probably want to script this to keep your sanity.


 * Capture a snapshot of Rome, Pre-installation.
 * First you will need a destination to add/save your snapshots, you will see mine at /root/Documents/snapshot_ddmmyyyy and /root/Documents/snapshot_ddmmyyyy_softwarethatwasinstalled . Next, you will need to run the following command from within your snapshot directory to save the output.  ls -al /file/path > file-path.txt  where /file/path would be something like /usr/local/bin and saved as a text file with the same name, replacing the / with a -. The list of directories is below (in their txt format)

Note: There does not appear to be a /usr/local/lib64 folder as I believe these are 32 bit systems. Not sure at this point.

bin.txt lib.txt sbin.txt sys.txt usr-bin.txt usr-libexec.txt usr-lib.txt usr-local-bin.txt usr-local-lib.txt usr-sbin.txt usr-share-libgnomekbd.txt usr-share-libgweather.txt usr-share-libhangul.txt usr-share-librarian.txt usr-share-libthai.txt usr-share-libtool.txt usr-share-libwacom.txt usr.txt var-lib.txt var-local.txt


 * Step 4
 * Here is where you begin your actual install. If you cannot complete it by the standard yum install nameofsoftware you may need to create a repo. This would be done at /etc/yum.repos.d. This will change with whatever software you are installing, however; below is what was used for Nginx:


 * Install your Software:
 * At the directory /etc/yum.repos.d add the repo for the software you need to install, I.E.: nginx.repo then use vi or my preferred nano editor to modify the Nginx.repo.

[cat nginx.repo] [nginx] name=nginx repo baseurl='http://nginx.org/packages/rhel/6/$basearch/' gpgcheck=0 enabled=1


 * Here, within the baseurl, the rhel/6, refers to the operating system and version number. That is Red Hat Enterprise Linux - Version 6. Once the repo has been created, re-run  yum install nginx and you should be good.


 * Step 5
 * Capture post-install snapshot:
 * Repeat Step 3 in a different directory so that you have snapshot_preinstall and snapshot_postinstall. This allows you to understand what files have been added/modified in the installation directories. -- If you do not intend on moving the installation up to Caesar so that all Drones can access the tool, then you do not need to complete a diff and run it by Jonas. --- If you do intend to move the installation to Caesar, you must run diff /root/Documents/snapshot_pre /root/Documents/snapshot_post and review with Jonas. Then likely perform a hotfix. I did not need to do this, so I do not have these instructions for you.


 * Step 6
 * Assuming you did not go with the hotfix method and only need the software installed on Rome, you are good to bring Rome in from the cold.


 * Reconnect Rome to the System:
 * First, re-connect the Ethernet cables you had removed previously. Disconnect the orange outside-line that you are only supposed to use temporarily. Then, on Rome, run the command mount -a to re-mount Caesar. You are done.


 * Note: You will not want to reconnect the symbolic link at /usr/local as this will override the installation you just did. If for whatever reason you needed to do this, you will first want to move the local folder mv local local.old then make the new symbolic link  ln -sf /mnt/main/local /usr (NOTE, This is LN not LS. Similar but different). The vice versa is unlink local then put the old directory back mv local.old local.

Hope that's helpful.

Firewall Procedure: Open a port

 * Before your begin:
 * Watch a youtube video on Iptables to understand the nat and filter tables. Lookup SELinux.


 * Step 1


 * Determine where things are and how to run them
 * For the purposes of this procedure, I will use Rome as the example, however this procedure should be applicable to all the servers. Lets first identify where the configuration files live. -- Navigate to  cd /etc/sysconfig here, you will see the confusion which is two firewalls. If you were to do an 'ls' in the 'sysconfig' directory, you will see a number of 'iptables' or 'ip6tables' which you would think is all you need. Unfortunately, there will also be a 'system-config-firewall' file in here also. Thankfully, easily editable but lots of fun when you don't know its there. Additionally, while there is no symbolic link back to Caesar you will need to run iptables as:  sudo /sbin/iptables [command].

Rome /etc/sysconfig]# ls acpid        httpd                        kernel           pluto          smartmontools atd          i18n                         keyboard         prelink        snmpd auditd       init                         modules          quota_nld      snmptrapd authconfig   ip6tables                    netconsole       raid-check     sshd autofs       ip6tables-config             network          readahead      sysstat cbq          ip6tables.old                networking       readonly-root  sysstat.ioconf clock        iptables                     network-scripts  rhn            system-config-firewall console      iptables-config              nfs              rngd           system-config-firewall.old cpuspeed     iptables_LoadCurrent_config  nginx            rsyslog        system-config-users crond        iptables.old                 nginx-debug      samba          udev firstboot    iptables.save                nspluginwrapper  sandbox        wpa_supplicant grub         irqbalance                   ntpd             saslauthd      xinetd htcacheclean kdump                        ntpdate          selinux


 * Step 2
 * let us start by making a copy of the CurrentConfig of iptables, which we will then edit. cp iptables iptables_032018Config. You will see mine above with the poorly named 'iptables_LoadCurrent_config'. Next, lets edit this file, I would recommend using the text editor nano with the switchs '-S -c such that it reads:  nano -Sc iptables_032018Config . This will appear as:

iptables_LoadCurrent_config
 * 1) Generated by iptables-save v1.4.7 on Sat Mar  3 17:29:35 2018
 * filter
 * INPUT ACCEPT [0:0]
 * FORWARD ACCEPT [0:0]
 * OUTPUT ACCEPT [590:112218]

-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT -A INPUT -p icmp -j ACCEPT -A INPUT -i lo -j ACCEPT -A INPUT -p tcp -m state --state NEW -m tcp --dport 22 -j ACCEPT -A INPUT -p tcp -m state --state NEW -m tcp --dport 8022 -j ACCEPT -A INPUT -p tcp -m state --state NEW -m tcp --dport 8020 -j ACCEPT -A INPUT -j REJECT --reject-with icmp-host-prohibited -A FORWARD -j REJECT --reject-with icmp-host-prohibited COMMIT
 * 1) Completed on Sat Mar  3 17:29:35 2018
 * 2) Generated by iptables-save v1.4.7 on Sat Mar  3 17:29:35 2018
 * nat
 * PREROUTING ACCEPT [12:720]
 * POSTROUTING ACCEPT [3:320]
 * OUTPUT ACCEPT [3:320]

-A PREROUTING -p tcp --dport 8022 -j REDIRECT --to-ports 8020 COMMIT
 * 1) Completed on Sat Mar  3 17:29:35 2018


 * Here, we will mostly be working with the *filter section. Where you can then infer that the system is accepting input via tcp to destination port 22, 8022 and 8020 with a 'jump' of accept (could also deny). You should look up commands or youtube videos on iptables to understand this better.
 * The *nat section, is currently redirecting any traffic that enters port 8022 to port 8020. This is specifically configured for the above Nginx software install. Presuming that I am successful and you have an SSH Load Balance while reading this, please don't touch that redirect.
 * Now, Add your own port by copying the preceding line and adjusting the number: -A INPUT -p tcp -m state --state NEW -m tcp --dport ## -j ACCEPT where ## is your port.

Note: ports below 1024 are privileged and may throw a hissy fit


 * Step 3
 * Next, let us update the other firewall. While in /etc/sysconfig run the following nano -Sc system-config-firewall. Editing should be self explanatory:

system-config-firewall
 * 1) Configuration file for system-config-firewall

--enabled --port=##:tcp --port=8022:tcp --port=8020:tcp --port=22:tcp


 * Step 4
 * Now load your config file, into iptables.  sudo /sbin/iptables-restore < iptables_032018Config  - You're config file should now be active. You can confirm by running  sudo /sbin/iptables -nL and sudo /sbin/iptables -t nat -nL.

Note: The Capital 'L' in -nL is important.
 * Additionally, you should run netstat -tapln and see them there. Once you are confident in your configuration, run sudo service iptables save then run sudo service iptables restart and then recheck your config sudo/sbin/iptables -nL. If your config is still there, you're good.


 * Step 5
 * Log out of Rome and then either from Caesar or another drone, run nmap rome -Pn this will checks the common 1,000 ports. the -Pn switch is there to basically say 'I don't care if you think rome is offline, run the nmap command anyway'. This is because ICMP is blocked. Note you may have to add the port number if you are above the 1024 mark, such as: nmap rome -p8022 -Pn. This should now confirm for you that the port has been opened and you should be all set.


 * Step 6
 * You're Done! -- however, if you have issues with SELinux policies, check out my (Camden Marble's) log for March 3rd 2018.

Hope that's helpful,

Nginx: What is it and why is it here?

 * Nginx is a http host (in that it serves a web page like Apache) but also does Load Balancing (LB). Now, these Linux servers appear to come with a HTTP host preinstalled, so that is not the function we set out to use. Instead, Nginx was installed for its LB capabilities. Nginx has the ability to LB HTTP Requests, UDP and TCP.
 * The purpose for having installed Nginx was to Load Balance SSH Access to the Drone Servers on a Round Robin using Nginxs' TCP Stream module. Nginx is only installed on Rome and that should be the only place necessary. How this would work is from Caesar, you would ssh -p8022 rome where IPtables on Rome will route traffic from 8022 (SSH Listening) to 8020 (Nginx listening), then Nginx would pass that traffic down 'stream' to the backend drones on port 22, as seen below.

/etc/nginx [cck27@rome nginx]$ cat nginx.conf user nginx; worker_processes auto; pid /var/run/nginx.pid;

events { worker_connections 768; # multi_accept on; }

stream { upstream backend { server asterix:22; server obelix:22; ##server miraculix:22; server idefix:22; #server traubadix:22; #server majestix:22; #server automatix:22; #server methusalix:22; #server verleihnix:22; #server rome:22; SELF #server brutus:22; #server caesar:22; NEVER

}       server { listen 8020; proxy_pass backend; }

}


 * And this works! ... well, it works once. Then SSH complains in various forms. I believe this has to do with SSHs known_hosts or perhaps SSH Identities, in that the connection keeps changing from 'Rome'. If you fix this aspect then Nginx will be fully functional. You 'Should' also be able to create a new pool, i.e.: backend_two and open another port, say 8023 (ssh) and route to a new listening for Nginxs' second pool, say 8021. This has not been tested yet however, as I can't get it to round robin more than once before SSH complains.

Nagios Core: To be installed
Ref: https://www.nagios.com

For the tools we decided to install Nagios.

Nagios Core is a free and open source computer-software application that monitors systems, networks and infrastructure. Nagios offers monitoring and alerting services for servers, switches, applications and services. It alerts users when things go wrong and alerts them a second time when the problem has been resolved. It is written in C Programming Language.

Currently it provides: •	Monitoring of network services (SMTP, POP3, HTTP, NNTP, ICMP, SNMP, FTP, SSH) Monitoring of host resources (processor load, disk usage, system logs) on a majority of networking system. •	Monitoring of any hardware (like probes for temperature, alarms, etc.) which have the ability to send collected data via a network to specifically written plugins •	Monitoring via remotely run scripts via Nagios Remote Plugin Executor. •	 Remote monitoring supported through SSH or SSL encrypted tunnels. •	A simple plugin design that allows users to easily develop their own service checks depending on needs, by using their tools of choice (shell scripts, C++, Perl, Ruby, Python, PHP, C# . •	Available data graphing plugins •	Parallelized service checks •	Flat-text formatted configuration files (integrates with many config editors) •	The ability to define network host using 'parent' hosts, allowing the detection of and distinction between hosts that are down or unreachable •	Contact notifications when service or host problems occur and get resolved (via e-mail, pager, SMS )or any user-defined method through plugin system) •	The ability to define event handlers to be run during service or host events for proactive problem resolution •	Automatic log file rotation. •	Support for implementing redundant monitoring hosts •	Support for implementing performance data graphing •	Support for database backend. •	An web-interface for viewing current network status, notifications, problem history, log files, etc.

In details now: -Nagios provides monitoring of all mission-critical infrastructure components including applications, services, operating systems, network protocols, systems metrics, and network infrastructure. Hundreds of third-party addons provide for monitoring of virtually all in-house and external applications, services, and systems.

-Nagios Log Server greatly simplifies the process of searching your log data. It sets up alerts to notify you when potential threats arise, or simply query your log data to quickly audit any system. With the help of Nagios Log Server you get your entire log data in one location, with high availability and fail-over built right in. And there are no data limits.

Nagios Fusion offers our network a high degree of visibility and scalability, helping solve problems that come with multiple networks and geographical separation. It allows you to visualize multiple Nagios XI and Core servers in one location, network management becomes simplified by centralization.

Also, used for

OS monitoring - Operating System (OS) Monitoring with Nagios Nagios provides complete monitoring of desktop and server operating systems – including system metrics, service states, process states, performance counters, event logs, applications (IIS, Exchange, Apache, MySQL, etc), and services (Active Directory, DHCP, Sendmail, etc). Nagios supports monitoring of Windows, Linux, Unix, Solaris, AIX, HP-UX, and Mac OS/X operating systems. Benefits Implementing effective operating system monitoring with Nagios offers the following benefits: •	Increased server, services, and application availability •	Fast detection of network outages and protocol failures •	Fast detection of failed services, processes and batch jobs It monitors windows, linux, server monitoring, RHEL monitoring, CentOS monitoring, AIX monitoring.

Protocol Monitoring with Nagios Nagios provides complete monitoring of network protocols – including TCP/IP and UDP protocols. Implementing effective network protocol monitoring with Nagios offers increased server, services, and application availability, as well as fast detection of network outages and protocol failures. Benefits Implementing effective network protocol monitoring with Nagios offers the following benefits: •	Increased server, services, and application availability •	Fast detection of network outages and protocol failures -	SNMP monitoring, HTTP monitoring, SSH Monitoring, FTP monitoring, Protocol Monitoring and SMTP Monitoring.

Application Monitoring With Nagios includes (Web application monitoring, Application monitoring, Windows service monitoring, Tomcat monitoring, Java Monitoring, JMX Monitoring)

Nagios provides complete monitoring of applications and application state – including Windows applications, Linux applications, UNIX applications, and Web applications. Benefits Implementing effective application monitoring with Nagios offers the following benefits: •	Increased server, services, and application availability •	Fast detection of network outages and protocol failures •	Fast detection of failed services, processes and batch jobs

Database Monitoring with Nagios Nagios provides complete monitoring of database servers and databases – including availability, database and table sizes, cache ratios, and other key metrics. Benefits Implementing effective database monitoring with Nagios offers the following benefits: •	Increased application availability •	Increased database performance •	Fast detection of database outages, failures, and table corruption •	Predictive analysis of storage requirements and index performance

-	Log Monitoring and Management with Nagios includes (Log file, Syslog Monitoring, Windows Event Log Monitoring, Open Source Log Monitoring, Application Monitoring and Security Monitoring. )

Nagios provides complete monitoring and log management of application logs, log files, event logs, service logs, and system logs on Windows servers, Linux servers, and Unix servers. Nagios is capable of monitoring system logs, application logs, log files, and syslog data, and alerting you when a log pattern is detected. Benefits Implementing effective log monitoring with Nagios offers the following benefits: •	Increased security •	Increased awareness of network infrastructure problems •	Increased server, services, and application availability •	Fast detection of network outages and protocol failures •	Fast detection of failed processes, services, cron jobs, and batch jobs •	Audit compliance and regulatory compliance

Bandwidth Monitoring with Nagios Nagios provides complete bandwidth monitoring of switches and Routers via SNMP. Switches and routers can be monitored via SNMP v1, 2c, or 3 and deliver bandwidth utilization for both inbound and outbound traffic. Thousands of different network devices are enabled by default for this type of monitoring. Nagios XI makes this process even easier by allowing you to run the switch or router monitoring wizard and setup to monitor bandwidth on the device can be done in just minutes. Benefits Implementing effective monitoring of bandwidth with Nagios offers the following benefits: •	Easily find over utilized ports •	Discover possible network abusers •	Ability to track per-port bandwidth utilization and errors •	Fast detection of network outages and protocol failures

2018 Backup Solution

 * Lutetia is the backup server -- renamed from 'capstonebackup'

Helpful link: http://rsnapshot.org/rsnapshot/docs/docbook/rest.html

See the following general topology before reading: https://foss.unh.edu/projects/images/f/f2/Overview.pdf


 * Lets Begin
 * Rome is serving as the go-between to backup Caesar to lutetia. This is done by having Caesar and Lutetia mounted to Rome at /mnt/main and /mnt/backup_solution respectively.


 * Step 1
 * For the first back up, do a clean copy cp of the directory to back it up.

nohup cp -i -v -ar /mnt/main/Exp/0309/ /mnt/backup_solution/rsyncTest/


 * Step 2
 * Do an archival rsync which will determine the difference between the initial copy and the current state of the directory. Running this as a nohup and including the backupconfirmation.txt is advisable as follows:

nohup rsync -av /mnt/main/Exp/0309/ /mnt/backup_solution/rsyncTest/ && rsync -av /mnt/main/backup_solution/backupconfirmation.txt /mnt/backup_solution/rsyncTest/ &


 * Note
 * You should see a backupconfirmation.txt this is a cronjob on Caesar that adds the date to the text file once an hours. Then writes-over itself once a week

[root@rome backup_solution]# pwd /mnt/main/backup_solution [root@rome backup_solution]# cat backupconfirmation.txt Sun Apr 8 15:00:25 EDT 2018 Sun Apr 8 15:00:29 EDT 2018 Sun Apr 8 16:00:01 EDT 2018

Known Issue(s)

 * Slow
 * The Copy and Rsync processes are very slow, this could potentially be sped-up by running multiple rsyncs at once, such as


 * Rsync directory A
 * Rsync directory B
 * Rsync directory C
 * Changing the CPU affinity and renice priority had no effect on speed.

Possible Solution Create a parallelization script in Perl to stack rsyncs atop each other.


 * Space \ Storage
 * If you tail -20 nohup.txt in the Rome:/mnt/backup_solution/rsyncTest/nohup.txt you will see the following errors:

rsync: recv_generator: mkdir "/mnt/backup_solution/rsyncTest/40/qmanager" failed: No space left on device (28)
 * Skipping any contents from this failed directory ***
 * If you run df -h on Rome you will see that /mnt/backup_solution says it is taking up 84% of the partition. Given that Lutetia has a 4TB hard drive, the resolution for this should be to simply extend the size of the partition to utilize the full 4TBs.

The Following Rsnapshot Was Discontinued as it does a rolling backup and not a iterative backup
Helpful Link: http://rsnapshot.org/rsnapshot/docs/docbook/rest.html

See the following general topology before reading: https://foss.unh.edu/projects/images/f/f2/Overview.pdf


 * Lets Begin
 * Rome is serving as the go-between to backup Caesar to the capstonebackup. This is done by having Caesar and capstonebackup mounted to Rome at /mnt/main and /mnt/backup_solution respectively. Following this, rsnapshot is used to copy files from Mount A to Mount B.

/etc/rsnapshot.conf excerpts:
 * 1) SNAPSHOT ROOT DIRECTORY #
 * 1) SNAPSHOT ROOT DIRECTORY #

snapshot_root  /mnt/backup_solution --
 * 1) All snapshots will be stored under this root directory.
 * 1)           BACKUP INTERVALS            #
 * 2) Must be unique and in ascending order #
 * 3) i.e. hourly, daily, weekly, etc.      #
 * 1) i.e. hourly, daily, weekly, etc.      #

interval       weekly  4 --
 * 1) interval      hourly  1
 * 2) interval      daily   7
 * 1) interval      monthly 1
 * 1) BACKUP POINTS / SCRIPTS ###
 * 1) BACKUP POINTS / SCRIPTS ###

[...] [...]
 * 1) LOCALHOST
 * 1) EXAMPLE.COM

backup /mnt/main/backup_solution/backupconfirmation.txt        localhost/caesar_backup
 * 1) CAPSTONE BACKUP SOLUTION


 * After final testing, the backup will include /mnt/main/* to backup the rest of Caesar.


 * NOTE
 * The rsnapshot weekly command is run every Sunday at 3 AM. This can be seen or adjusted by reviewing the cronjob on Rome via crontab -e while logged in as root.


 * 1) For details see man 4 crontabs

0 3 * * 1 root rsnapshot weekly
 * 1) Example of job definition:
 * 2) . minute (0 - 59)
 * | .- hour (0 - 23)
 * | |  .-- day of month (1 - 31)
 * | |  |  .--- month (1 - 12) OR jan,feb,mar,apr ...
 * | |  |  |  . day of week (0 - 6) (Sunday=0 or 7) OR sun,mon,tue,wed,thu,fri,sat
 * 1) *  *  *  *  * user-name command to be executed
 * 1) *  *  *  *  * user-name command to be executed
 * 1) M H DOM MOY DOW COMMAND


 * You should then see weekly.0 in the backup_solution [/mnt/backup_solution]

[root@rome backup_solution]# ls weekly.0
 * Inside you should see

Caesar_backup /mnt/backup_solution/weekly.0/localhost/caesar_backup


 * Note
 * You should see a backupconfirmation.txt this is a cronjob on Caesar that adds the date to the text file once an hours. Then writes-over itself 24 hours after the backup is complete. If your backup shows this 24 hour difference between when the backup was run and the file was written over, it is likely that there may be something wrong with when the backup is kicking off.

[root@rome backup_solution]# pwd /mnt/main/backup_solution [root@rome backup_solution]# cat backupconfirmation.txt Sun Apr 8 15:00:25 EDT 2018 Sun Apr 8 15:00:29 EDT 2018 Sun Apr 8 16:00:01 EDT 2018

Other Procedure to be Determined
Words