American Whitewater has two servers and a webspace. The first server runs the website and contains a MySQL database that houses the content management side of the AW site. The second server runs a postarrest database and is optimized for that database as well as a geospatial information systems software. The two servers interact with each other via SSH tunneling. The second server also has the ability to act as a fully functioning backup of the primary server in the case of catastrophe. The American Whitewater webspace is used as a content repository for photo backups, provides hosting for AW-affiliated projects, and serves to handle American Whitewater FTP space for staff.

The servers run CentOS 5.x and are updated monthly. The Web server runs Lighttpd, mysqld, memcached, andh PHP running fastCGI. The database server runs postgresSQL, geoserver on tomcat, and Apache. Firewall is provided both by configuring the internal software firewall and by using the external Cisco firewall. The only serviceable ports on the production Web server are the webserver ports and SSH. The database server only runs with SSH and a nonstandard web server port.

Lighttp hosts the main website and proxies to Apache on the database server for source code control. The main website uses a local connection to the MySQL server and uses an SSH tunnel to get to the database server's postgresql.

Both of American Whitewater's servers are backed up by reobak on a nightly basis to an offsite FTP server. All relevant files and configuration are backed up on a 10 to 14 day differential backup. Databases are dumped before the backup process moves the files offsite and are included in the 14 day differential with the exception of the age history database which is on a single nightly differential. Reoback configuration files are located in /etc/reoback and can be inspected by the system administrator for more details on how this process works. System backups are verified manually on a monthly basis as part of the routine maintenance task list.

• every day
  • review log watch and take any necessary corrective action
  • review the security bulletins for AW-integrated software (CentOS, SquirrelCart, Phorum, PHP 5.3, Lighttpd, Apache 2.1, Exim)
  • review performance data and take corrective action as necessary
• every month
  • test the security patch configuration on the virtual machines
  • compare the virtual machine patch list and software repository lists to the production servers
  • do a supervised software repository update on the production servers
  • verify backups are being produced in being placed in the right spaces
  • do a manual root kit and virus check and fix any problems reported
  • sample the administrator log, the bug log, the performance log, and the Web server error log for potential problems
  • clear out the temporary directory and make sure that the logging directory is healthy
• every quarter
  • look at disk space allocation and make sure that the system is growing as expected, make adjustments as necessary
  • clear up the mail and old logs including performance logs and bug logs which are typically in nonstandard places
  • review the backup coverage and expand or prune as necessary
  • review major software revision upgrades and evaluate each in turn
  • produce a summary of actions taken over the prior quarter and anticipate new hardware or software configuration for the next quarter
  • perform software benchmarks on the website looking for optimizations
  • perform a filesystem check
  • perform a database diagnostic

• Site Account (Website)
  • DB Maintence(./updateclubs.php) - runs through the jobs table every 2 minutes and runs a series of MySQL queries from the jobs table and the MySQL database. the main thing that this updates is the global expirations and marks users as members or nonmembers
  • Fetchmail - takes a nightly update from our membership database and pulls it into the membership table on the MySQL database (/home/site/app/code/jobs/php/update_memberlist.php)
  • PWUpdate (./updatetracusers.php), updates website passwords for users that are staff and board to access trac and myadmin
• Root Account (Website and Production)
  • REOBack - run a nightly backup (/etc/reoback/backup.sh)
• Site Account (Database)
  • Gaugecycle - restarts the main daemon process called gaugedaemon.php. This process is responsible for querying the gauge sites for updates, correlating stream gauges on the website, producing gauge notification e-mails,and performing regular database maintenance on the gauge database.

Check /etc/lighttpd/lighthttpd.conf for our application specific configuration including lots of misc. force-types and rewrite rules used by the old wh2o code to get pretty URLs. Quite a bit of lock-down information is also included there.