DOC

MIT OPENCOURSEWAREMIRROR SITE PROGRAM

DOCUMENTATION

Last Updated November 24, 2008

If you have any questions about this document,

please contact Yvonne Ng, OCW External Outreach

Manager, at:

[email protected] or +1-617-253-4719

TABLE OF CONTENTS

ABOUT THIS DOCUMENT....................................................................................................................................................................2

OCW MIRROR SITE PROGRAM OVERVIEW..........................................................................................................................................3

Background and Goals...................................................................................................................................................................3

Responsibilities of Participating Institution.....................................................................................................................................3

Responsibilities of OCW..................................................................................................................................................................3

MIRROR SITE LAUNCH.......................................................................................................................................................................4

OCW MIRROR SITE CONTENTS...........................................................................................................................................................5

Organization of external hard drive................................................................................................................................................5

Terms of Use.................................................................................................................................................................................. 6

MIRROR SITE CONFIGURATION..........................................................................................................................................................7

Mirror Site Technical Requirements................................................................................................................................................7

Basic Instructions........................................................................................................................................................................... 9

Advanced Instructions....................................................................................................................................................................9

Technical Requirements for User Environment............................................................................................................................16

Customization of Mirror Site.........................................................................................................................................................16

MIRROR SITE USAGE TRACKING.......................................................................................................................................................19

Overview...................................................................................................................................................................................... 19

Monthly Reports........................................................................................................................................................................... 20

AWStats Introduction....................................................................................................................................................................20

AWStats Requirements.................................................................................................................................................................21

MIRROR SITE UPDATES....................................................................................................................................................................21

Overview...................................................................................................................................................................................... 21

OCW Rsync Program....................................................................................................................................................................22

Rsync Server Connection.............................................................................................................................................................23

Recommended Rsync Configuration.............................................................................................................................................24

Rsync Commands.........................................................................................................................................................................24

Rsync Logging..............................................................................................................................................................................26

Rsync Troubleshooting.................................................................................................................................................................27

APPENDIX........................................................................................................................................................................................ 30

Search Functionality.....................................................................................................................................................................30

Feedback Functionality.................................................................................................................................................................37

AWStats Installation with Apache on Windows.............................................................................................................................40

AWStats Installation with IIS on Windows.....................................................................................................................................43

AWStats Installation on Unix-like Operating Systems...................................................................................................................44

Rsync Installation.........................................................................................................................................................................47

ABOUT THIS DOCUMENT

This document provides extensive documentation about the OCW Mirror Site program, hosted by MIT

OpenCourseWare (OCW), found at http://ocw.mit.edu. The first two sections of this document, “OCW Mirror

Site Program Overview” and “Mirror Site Launch,” contain important details about institutional participation

and responsibilities. The remaining bulk of the document provides extensive documentation about how to

use the OCW Mirror Site to install and maintain a mirror site, or local copy, of OCW materials.

The HTML version of this file can be viewed in a Web browser, and the PDF version of this file can be viewed

in a PDF reader. If you do not have access to a PDF reader, see the “Technical Requirements for User

Environment” section in this document for suggestions on how to obtain a PDF reader.

Questions related to this document, or the OCW Mirror Site program, can be referred to Yvonne Ng, OCW

External Outreach Manager, at [email protected] or +1-617-253-4719.

OCW Mirror Site ReadMe Page 2 September 2008

mailto:[email protected]

http://ocw.mit.edu/

OCW MIRROR SITE PROGRAM OVERVIEW

Background and Goals

A fundamental part of OCW’s mission is to extend the reach and impact of OCW materials throughout the

world. Much of OCW’s efforts have focused on Africa, where OCW materials are largely underutilized due to

limited Internet connectivity. OCW has installed a small number of mirror sites, or local copies of OCW

content, at African educational institutions and our evaluation data show that these mirror sites are having

a significant positive impact.

The OCW Mirror Site program intends to build off of the success of the mirror site program and substantially

scale up the effort by distributing OCW Mirror Site packages to universities throughout the developing

world.

Responsibilities of Participating Institution

The participating institution has agreed to the following as part of the OCW Mirror Site program:

Host and maintain a mirror site that contains a complete, updated, and accurate copy of the OCW site and

is publicly accessible on campus;

Provide to OCW a monthly report of mirror site usage (which can be collected with the usage tracking

software provided on the hard drive);

Promote the mirror site to at least one major local or national media outlet;

Provide a staff member to serve as the central point of contact for OCW inquiries; and

Upon request, contribute to a case study or other activities for the OCW Mirror Site program.

Please refer to the next section on “Mirror Site Launch” for additional details on how to carry out these

responsibilities.

Responsibilities of OCW

OCW has agreed to the following as part of the OCW Mirror Site program:

Provide the participating institution with a free OCW Mirror Site package;

Acknowledge the participating institution on the OCW Web site;

Grant authenticated secure shell (SSH) access using digital keys to our mirror site synchronization server,

for easy and scheduled content updates; and

Provide an OCW staff member who will serve as the central point of contact for technical assistance and

other inquiries.


MIRROR SITE LAUNCH

Once the mirror site has been installed and is publicly accessible on your network, please make sure to

complete the following steps:

Notify Yvonne Ng, OCW External Outreach Manager, at [email protected] or +1-617-253-4719 and provide

the following information:

o Mirror site URL (if the URL is not accessible from outside the network, please indicate this);

o AWStats statistics URL (if the URL is not accessible from outside the network, provide a contact

person who can manually email the statistics on a monthly basis);

o Plans for promoting the mirror site; and

o Central point of contact for the mirror site.

Implement strategies for promoting the mirror site.

[NOTE: Use of MIT’s name must not be used in ways that suggest or imply the endorsement of other

organizations, their products, or their services. The use of the Institute's name, seal, and photographs

in the advertising and other promotional material and activities of outside organizations is prohibited

when such use is likely to be understood as an endorsement, even if such an endorsement is not the

intention of the person or organization seeking to use MIT's name. All proposals, therefore, for the use

of MIT's name or other identification in advertising, sales literature and videos, and commercial

publicity must be submitted to the Director of the News Office. Please contact Yvonne Ng

([email protected]) to facilitate your promotion.] Please feel free to use the materials provided on the

external hard drive in the PromotionMaterials folder for these promotional activities, but coordinate

your efforts with Yvonne Ng.

Once your mirror site is launched, OCW will include your institution’s name on a page dedicated to the OCW

Mirror Site program on our main site, at http://ocw.mit.edu. In addition, we will include you on any relevant

communications to our supported mirror sites. Your central point of contact at OCW will be Yvonne Ng, OCW

External Outreach Manager, who can be reached at [email protected] or +1-617-253-4719.



http://ocw.mit.edu/


OCW MIRROR SITE CONTENTS

Organization of external hard drive

The organization of this OCW Mirror Site is described below:

NR (13.0 GB)- This folder contains half of the main content from the OCW site, including PDF files for

all the course section pages (course home, calendar, syllabus, lecture notes, assignments, exams,

etc.)

OCWExternal (205 GB)- This folder contains all of the enhanced content from the OCW site,

including video, audio, applets, and course zip packages.

OcwWeb (960 MB)- This folder contains the other half of the main content from the OCW site,

including HTML files for all the course section pages (course home, calendar, syllabus, lecture notes,

assignments, exams, etc.)

Promotion Materials (5.58 MB)- This folder contains helpful information and templates about OCW

and the OCW Mirror Site Program, which institutions can use to announce and promote their mirror

site.

Tools (163 MB)- This folder contains optional software tools and documentation that can be used,

but are not required, to enhance the mirror site. Subfolders include:

o Installer

o PDFReader

o Rsync

o Search

o UsageTracking

o WebTemplate

index.htm- This file is the home page for the OCW Web site. If you open this file from the drive

directly, you will be able to browse the OCW site locally off the drive.

MIT OpenCourseWareLegal Notices.htm or pdf – This file contains the full legal license that

governs the use of the OCW materials.

MITOCWWelcomeLetter.htm or pdf - This file contains a welcome message from OCW’s Executive

Director.

ReadMe.htm, pdf, or doc – This file contains extensive information about the OCW Mirror Site

program, including configuration of the mirror site using the OCW Mirror Site contents.


Terms of Use

OCW materials

With the exception of the content within the Tools directory and subdirectories, all materials on the OCW

Mirror Site are licensed by the Massachusetts Institute of Technology under a Creative Commons License.

For more details, please refer to the HTML and/or PDF versions of the files MITOCWFullLicense and

MITOCWterms-of-use, located on the root drive of this external hard drive.

The underlying premise and purpose of OCW is to make course materials used in MIT courses freely and

openly available to others for non-commercial educational purposes. Through OCW, MIT grants the right to

anyone to use the materials, either "as is," or in a modified form. There is no restriction on how a user can

modify the materials for the user's purpose. Materials may be edited, translated, combined with someone

else's materials, reformatted, or changed in any other way. However, there are three requirements that an

OCW user must meet to use the materials:

Non-commercial: Use of OCW materials is open to all except for profit-making entities who charge a

fee for access to educational materials.

Attribution: Any and all use or reuse of the material, including use of derivative works (new

materials that incorporate or draw on the original materials), must be attributed to MIT and, if a

faculty member's name is associated with the material, to that person as well.

Share alike (aka "copyleft"): Any publication or distribution of original or derivative works, including

production of electronic or printed class materials or placement of materials on a Web site, must

offer the works freely and openly to others under the same terms that OCW first made the works

available to the user.

Other materials

The content within the Tools directory and subdirectories are not affiliated with, or endorsed by, OCW and

are licensed under different terms. All licenses are provided in the same folders as the installation files for

each tool. Please read through the licenses carefully to make sure that your use complies with the terms of

use provided in the licenses.

Please note that these tools are optional and should be used at your own discretion. OCW is not able to

provide any support for these tools, and is including them as part of the external hard drive as a

convenience to mirror sites.


MIRROR SITE CONFIGURATION

Mirror Site Technical Requirements

Software

The information provided below includes requirements for mirror sites running either Windows or

Linux/Unix-type operating systems, although other operating systems can be used as well.

Windows Linux/Unix-type system

IIS or Apache Apache

AWStats 6.8 (included in Tools folder on external

hard drive)

AWStats 6.8 (included in Tools folder on external

hard drive)

cwRsync (included in Tools folder on external hard

drive)

rsync (native to operating system)

In addition, Java Run-time environment (JRE) 1.4x or newer is recommended. If your system does not

already have this installed, you may download it at: http://www.java.com/en/download/manual.jsp.

Disk space

There are two options for hosting OCW content on the mirror site: (1) Host the entire site, including video,

audio files, and course zip packages, or (2) Host the bulk content of the site, including most of the course

materials minus the video and audio files.

The size of the OCW content is broken up in the following ways:

Bulk content of site (NR, OcwWeb folders): 14 GB

Video, audio, and other enhanced content (OcwExternal folder): 205 GB

Entire site (NR, OcwExternal, OcwWeb folders): 219 GB

Please note that the OCW site will not grow considerably, as only new courses and updates will be

published, while older versions of courses are unpublished and archived to D-Space, an online repository.

Hosting video and audio

If you decide to host the entire site, you will need to decide how you’d like to host the RealMedia streaming

video and audio files on the site. There are three options for hosting this content: (1) Host them as

downloadable files, (2) Host them as HTTP streaming files through your Web server, or (3) Host them as

streaming files through a streaming server.

Downloadable files


http://www.java.com/en/download/manual.jsp

Hosting them as downloadable files is the simplest option and requires no changes to the pages or your

server setup. However, in order for your users to access these files, they will need to have sufficient disk

space on their local environments to download the files completely. In addition, depending on the local

network speed, it may take some time to download the files.

HTTP streaming files

You can also host the video and audio files as HTTP streaming files through your existing Web server.

However, if you anticipate heavy usage of the mirror site, you may want to consider the streaming server

option, since HTTP streaming will incur a heaver load on your Web server as more users access the

streaming files.

To set up HTTP streaming, you will need to follow this process for every streaming audio and video file on

the site:

Go to the list of courses with complete sets of audio and video at:

OcwWeb/Global/OCWHelp/avocw.htm

Starting with the first course on the list, go to the section of the course that links to the

audio/video.

Identify the first audio/video file that you’d like to stream.

Open a text editor (such as Notepad) and enter the URL of the file.

Example: OcwExternal/3/3.091/f04/video/ocw-3.091-lec01-56k.rm

Save the file with the same filename, but with the extension .ram, instead of .rm. This .ram file is

known as a metafile - a file which contains data about another file.

Example: OcwExternal/3/3.091/f04/video/ocw-3.091-lec01-56k.ram

Open the HTML code for that section of the course in a text editor (such as Notepad).

Example: OcwWeb/Materials-Science-and-Engineering/3-091Fall-2004/LectureNotes/index.htm

In the HTML code, find the link for the .rm file and change it to link to the .ram file:

Example: <a href="OcwExternal/3/3.091/f04/video/ocw-3.091-lec01-56k.ram">RAM - 56K</a>

Follow this process for all audio/video files in the course.

Repeat this entire process for all other courses listed at: OcwWeb/Global/OCWHelp/avocw.htm.

Streaming files

If you anticipate significant usage of your mirror site and would like to host the audio and video files as

streaming, you can set up a streaming server using RealMedia's free Helix server or an open-source

streaming server. Additional information on the Helix server can be found at: https://helix-

server.helixcommunity.org/.

Basic Instructions

A Web server is strongly recommended for running the mirror site. It is possible to host the mirror site

directly off the external hard drive, without a Web server, but it will have limited performance and other

features, such as content updates, usage tracking, and search functionality, will not be possible to


https://helix-server.helixcommunity.org/

https://helix-server.helixcommunity.org/

implement. If you decide to run the mirror site off the external hard drive directly, make sure that the

location of all content is unchanged and remains located at the top level of the drive, so that all the links

may work properly. Please note that if your users view the mirror site using the Firefox browser, they may

have issues viewing the content if it is served from the external hard drive, because Firefox assumes that

the content is located on the same drive as the browser. You may be able to remedy this situation by having

users modify their settings in the Firefox browser.

For mirror sites that plan to host the content on a Web server, the external hard drive contents are

organized such that a simple copy and paste operation will be sufficient for transferring the contents onto

the server. The NR, OCWExternal, and OcwWeb folders, and index.htm file, all found at the root directory of

the external hard drive, should be copied over to the desired location on the mirror site server. As long as

these 3 folders and the index.htm file all reside at the same directory level, and no contents within the

folders are moved, the mirror site will work in the local environment without any further modification.

If you do not plan on hosting audio, video, and course zip packages (see section on Disk Space below), you

do not need to copy the OCWExternal folder to your mirror site server. Please note that links to these files

will be broken if you do not include the contents of this folder on the mirror site. If you decide to host these

files at a later date, you can always copy over the OCWExternal folder from the external hard drive onto

your mirror site server.

Advanced Instructions (see installation videos for these instructions on the hard drive)

Single Step Installer

In an effort to make installation as simple as possible we have provided installer scripts that will install

Apache and AWStats, and configure them for your mirror site. For both Windows and Linux you must have

administrator/root access to complete the installation.

Each installer takes 3 arguments: Server Name: IP address (ex. 18.72.0.3) or hostname (ex. http://ocw.mit.edu) of server. (Note: A

hostname is preferable to an IP address.)

Server Administrator: Email address of the server administrator. When Apache experiences an error, this address will be displayed so that clients can’t report the problem to the server administrator.

OCW Content Location: Path to the OCW content. (Note: This path must be to a directory that contains the subdirectories OcwWeb, OCWExternal, and NR.)

Linux (OCW Mirror Site Installation for Linux.wmv video)

The Linux installer is a shell script that will compile Apache, copy the AWStats files to /usr/local, and

configure both applications to work with one another. Please download the Perl application from

http://www.activestate.com/Products/activeperl/index.mhtml and install the program onto your system.

To run the script, do the following:1. Open a terminal and cd to the location of the installer. (The installer is located by default at <path-to-

ocw-drive>/Tools/Installer/Linux/.)

2. Run linux_install.sh as root:

$ sudo ./linux_install.sh <path-to-ocw-content> <server-name> <server-admin-email>


http://www.activestate.com/Products/activeperl/index.mhtml

http://ocw.mit.edu/

Ex. sudo ./linux_install.sh /mnt/ocw ocw.myschool.edu [email protected]

Notes: There is a single space between each command line argument.

Some distributions use “su” rather than “sudo” to run commands with root access.

The path to the content should NOT include a trailing slash.

When the script runs you will see the verbose output of the Apache installation and status messages from

the installer. After it is complete the script will provide you with two URLs – one pointing to your OCW mirror

site, and the other to your AWStats page.

Windows (see OCW Mirror Site Installation for Windows.wmv video)

For Windows we have created an executable file with a graphical user interface. Here is what you need to

do:1. Run setup.exe. (The installer is located by default at <path-to-ocw-drive>/Tools/Installer/Windows/.)

2. Provide the server name, admin’s email, and content path when requested.

3. Click Install.

In addition to installing the three applications mentioned above, the Windows installer will create scheduled

task, OCW AWStats Update, that will update the AWStats daily. Once the installer completes it will

automatically open two web pages for you. The first points to your OCW mirror site, and the second to your

AWStats page.

Manual Installation

Manual installation of the applications requires installing the three applications separately and modifying

Apache’s httpd.conf file to serve OCW content. This method should be used if you have an existing web

site/server configuration and would like to add a directory or virtual host for OCW content.

Note: ALWAYS make a backup of any configuration files BEFORE editing them. This method of installation is

only recommended for those who have experience modifying Apache configuration files.

Here is an overview of the process:1. Use individual installers Apache/IIS and Perl.

Integrate Perl with IIS, if using IIS.

2. Copy AWStats files to a local directory on your filesystem.

3. Configure Apache/IIS to serve the OcwWeb, OCWExternal, and NR directories from the root of the site.

4. Configure Apache/IIS to serve the awstats, awstatsicons, awstatscss, and awstatsclasses directories.

5. Create and edit awstats.ocw.conf to configure AWStats for your OCW installation.

Apache

Most Linux distributions include Apache. To determine the default location of Apache for your distribution,

refer to http://wiki.apache.org/httpd/DistrosDefaultLayout. If you cannot locate Apache on your system, or if

you are running Windows, source code and an MSI can be found in

<path-to-ocw-drive>/Tools/Installer/Apache/. If you are running a firewall on your server, you should open

TCP port 80 to allow client connections.

Compile/Install


http://wiki.apache.org/httpd/DistrosDefaultLayout


Windows users should simply run the MSI file to install Apache. Linux users will need to do the following to

compile and install Apache:

1. Copy httpd-2.2.8.tar.gz to a location in your local file system.

2. cd to the location in step 1 and run the following commands from a terminal:

tar zxf httpd-2.2.8.tar.gzcd httpd-2.2.8./configuremakemake install

These commands will install Apache in /usr/local/apache2.3. Set Apache to run at system boot.

a. cd back to the directory containing httpd-2.2.8.tar.gz.

b. Copy httpd script from this directory to /etc/init.d with the following command:

cp httpd /etc/init.d/httpdcd /etc/init.d

c. Finish the setup:

Debian/Ubuntu users run the following:

update-rc.d httpd defaults

Other distributions run:/sbin/chkconfig --add httpd/sbin/chkconfig --level 2345 httpd on

4. Start Apache with the command /usr/local/apache2/bin/apachectl –k start.

5. Test the installation by navigating to http://localhost in a web browser. You should see a default Apache welcome message: It works!

Configure Apache for OCW

In order for your server to serve OCW content it needs to know where to find the content. (Note: We highly

recommend copying the OCW content from the drive provided to an internal drive on your local server so

that you’ll have a backup in case the external drive is damaged or misplaced.) Apache gets this information

from its httpd.conf file. The configuration process is the same for both Windows and Linux.

OCW-only Server

This procedure will describe how to setup Apache to only serve OCW content (ie. http://yourdomain.edu will

point to the OCW homepage).

1. Make a backup of httpd.conf.

2. Open httpd.conf in a text editor.

3. Find the line that begins with DocumentRoot and replace the path in quotes with the path to your OCW content.


http://yourdomain.edu/

http://localhost/

4. Below DocumentRoot should be a Directory tag followed by a path in quotes (the same path we replaced earlier). Replace this path again with the path to your OCW content. Again, make sure you keep the quotes.

5. Below the closing </Directory> add the following lines, replacing <CONTENT-ROOT> with the path that was used earlier:

<Directory "<CONTENT-ROOT>\OcwWeb"> Options Indexes FollowSymLinks AllowOverride None Order allow,deny Allow from all</Directory>

<Directory "<CONTENT-ROOT>\OCWExternal"> Options Indexes FollowSymLinks AllowOverride None Order allow,deny Allow from all</Directory>

<Directory "<CONTENT-ROOT>\NR"> Options Indexes FollowSymLinks AllowOverride None Order allow,deny Allow from all</Directory>

6. Search for the line containing the text DirectoryIndex index.html and append index.htm to it. (Note: There should be a space between index.html and index.htm.)

7. Configure Apache to use the combined log format:

a. Locate the line containing LogFormat "%h %l %u %t \"%r\" %>s %b" common and comment it out by placing a # at the beginning of the line.

b. Comment the line CustomLog "logs/access.log" common.

c. Uncomment the line CustomLog "logs/access.log" combined.

8. Save the httpd.conf file and restart Apache from the command line:

Windows

net stop apache2

net start apache 2

Linux

(Note: Replace /usr/local/apache2 with the appropriate path to your installation.)

/usr/local/apache2/bin/apachectl -k restart

9. Navigate to http://yourdomain.edu in your browser once more and you should see the OCW homepage.

OCW as a Directory

This procedure sets up OCW to be served as a directory off the server root (ie. http://yourdomain.edu/ocw).




http://yourdomain.edu/ocw



4. Copy and paste the lines below at the end of httpd.conf, replacing <CONTENT-ROOT> with the path to your OCW content:

# OCW Content DirectoriesAlias /ocw "<CONTENT-ROOT>/"Alias /OcwWeb "<CONTENT-ROOT>/OcwWeb/"Alias /OCWExternal "<CONTENT-ROOT>/OCWExternal/"Alias /NR "<CONTENT-ROOT>/NR/"

<Directory "<CONTENT-ROOT>/"> Options None AllowOverride None Order allow,deny Allow from all</Directory>





6. Save the httpd.conf file and restart Apache.

7. Navigate to http://yourdomain.edu/ocw in your browser once more and you should see the OCW homepage.

OCW as a Subdomain

This procedure will setup OCW to be served from a subdomain (ie. http://ocw.yourdomain.edu). Before

starting this procedure you must set a DNS record so that ocw.yourdomain.edu points to your server’s IP

address. Without a DNS change, the following procedure will not work.




4. Search for the line containing the text ServerName declaration (ie. ServerName yourdomain.edu:80) and uncomment it, if necessary, by removing the # at the beginning of the line.

5. Copy and paste the lines below at the end of httpd.conf, replacing <CONTENT-ROOT> with the path to your OCW content and <YOUR-DOMAIN> with the hostname of your server (Note: <YOUR-DOMAIN> should be the same as the hostname declared on the ServerName line from step 4.):

# OCW Virtual HostNameVirtualHost *

<VirtualHost *> ServerName localhost ServerAlias <YOUR-DOMAIN></VirtualHost>

<VirtualHost *> ServerName ocw.localhost


http://ocw.yourdomain.edu/


ServerAlias ocw.<YOUR-DOMAIN> DocumentRoot "<CONTENT-ROOT>" <Directory "<CONTENT-ROOT> "> Options Indexes FollowSymLinks AllowOverride None Order allow,deny Allow from all </Directory>

ErrorLog "logs/ocw-error.log" CustomLog "logs/ocw-access.log" combined</VirtualHost>





7. Save the httpd.conf file and restart Apache.

8. Navigate to http://ocw.yourdomain.edu in your browser once more and you should see the OCW homepage.

IIS

The instructions below were developed for IIS 6 on Windows Server 2003. While it may be possible to adapt

these instructions to IIS 6 on Windows XP or IIS 7 on Windows Vista/Server 2008, this has not been tested

and is not recommended unless you have experience with IIS setup and administration.

1. If you intend to use AWStats you should first install ActiveState Perl using the instructions below.

2. Set index.htm as a default file:

a. Right-click on the Web Sites directory > Properties.

b. Click the Documents tab.

c. Click the Add button, type index.htm, and click OK twice to save the changes.

OCW-only Server

This procedure will describe how to setup IIS to only serve OCW content (ie. http://yourdomain.edu will point

to the OCW homepage).

1. Open the IIS Manager (Start > Control Panel > Administrative Tools > Internet Information Services (IIS) Manager).

2. Expand the folder labeled “Web Sites” where you should find the Default Web Site. Stop this site, if it is running, by right-clicking on it and selecting “Stop”.

3. Rather than overwriting the default web site, create a new site:

a. Right-click the Web Sites directory > New > Web Site.

b. Set the description to OCW and leave the IP/TCP settings unchanged.



http://ocw.yourdomain.edu/

c. At the Path field enter the path to the root of your OCW content (This is the drive/folder that contains the subdirectories NR, OCWExternal, and OcwWeb.).

d. Set the permissions for the site to “Read.”

4. Set index.htm as a default file:

a. Right-click on the Web Sites directory > Properties.

b. Click the Documents tab.

c. Click the Add button, type index.htm, and click OK twice to save the changes.

5. Navigate to http://yourdomain.edu in your browser once more and you should see the OCW homepage.

OCW as a Directory

This procedure sets up OCW to be served as a directory off the server root (i.e. http://yourdomain.edu/ocw).

1. Open the IIS Manager (Start > Control Panel > Administrative Tools > Internet Information Services (IIS) Manager).

2. Create a new virtual directory:

a. Right-click on your existing site > New > Virtual Directory.

b. Use the alias OCW.

c. At the Path field enter the path to the root of your OCW content (This is the drive/folder that contains the subdirectories NR, OCWExternal, and OcwWeb.).

d. Set the permissions for the site to “Read.”

3. Navigate to http://yourdomain.edu/ocw in your browser once more and you should see the OCW homepage.

Technical Requirements for User Environment

The information provided below includes recommended software for users accessing the mirror site either

via Windows or Linux/Unix-type operating systems, although other operating systems are possible as well.

Windows Linux/ Unix-type system

Web Browser

One of the following is recommended:

Internet Explorer 6.0+

Firefox 1.0+

Mozilla 1.4+

Web Browser


Firefox 1.0+

Mozilla 1.4+





Windows Linux/ Unix-type system

Media Player


RealOne Player software (streaming or

downloadable video)

QuickTime Player (downloadable video)

Windows Media Player (downloadable

video)

Media Player


RealOne Player software (streaming or

downloadable video)

QuickTime Player (downloadable video)

PDF reader

The bulk of our site, including lecture notes, assignments, exams, etc., is comprised of PDF files and will

require a PDF reader to be viewed. If your mirror site users do not have access to a PDF reader, one possible

option to consider is the free tool Ghostscript/Ghostview (http://www.cs.wisc.edu/~ghost/gsview/index.htm).

Ghostscript is an interpreter for PDF files, and Ghostview provides a graphical interface for Ghostscript. The

installers and documentation are provided on the external hard drive at: Tools/PDFReader. In addition to

Ghostscript/Ghostview, there are a number of other free software programs that can be used as well (you

can search for other PDF readers at http://www.sourceforge.net).

Customization of Mirror Site

This section describes the steps that can be taken to add a limited degree of customization and branding to

your mirror site. There are three possible changes that can be made which are discussed in detail below:

(1) adding your institution’s logo to the mirror site’s home page, (2) adding text about your institution

and/or the mirror site on the home page, and/or (3) adding your institution’s logo to the page header, which

appears on every page of the site except for the home page.

Note: There are sample pages which follow the instructions below on the external hard drive at:

Tools/WebTemplate. These are sample pages for reference purposes only and should not replace the

actual Web pages for the mirror site. Please follow the instructions and make the modifications to the actual

pages listed below.

Adding your institution’s logo to the home page

A logo may be displayed to the right of the MIT OpenCourseWare logo on the mirror site’s home page (see

the “University of Smithville” example below).


http://www.sourceforge.net/

http://www.cs.wisc.edu/~ghost/gsview/index.htm

Here’s how to add a logo: The maximum dimensions of your logo are 275 pixels wide by 36 pixels high. Exceeding this size

limitation in either dimension will cause unwanted changes to the page layout. If you wish to make your image a transparent GIF that will match the existing beige background color,

the hexadecimal value is #edecdf. Your image should be placed in the /OcwWeb/images/ directory on your mirror drive when it is ready. In order to place your image, you will need to add one line of html code to four different copies of the

home page. These files must be edited individually; do not edit one and save it to all four locations. The files can be found on your mirror drive at:

/index.htm /OcwWeb/index.htm /OcwWeb/web/index.htm /OcwWeb/web/home/home/index.htm

Within each file, search for the text: div class=”logo”

On the following line you will see the link and image that represents the MIT OpenCourseWare logo:

<h1><a href="/OcwWeb/web/home/home/index.htm"><img src="/OcwWeb/images/logo-ocw-

home_new.gif" alt="MIT OpenCourseWare" width="289" height="36" /></a></h1>

You will be adding your image (linked if you choose) between the last two html tags on that line (between </a> and </h1>).

The line you add should be as follows:

<a href="[your Web site address]"><img src="/OcwWeb/images/[your image filename]"

style="padding-left: 15px;" /></a>


In this example, [your Web site address] should be replaced with the complete URL that you want people

sent to when they click on your image. If you do not want your logo to link to your site, you may remove

the portion before “<img” as well as the “</a>” at the end. [your image filename] should be replaced

with the name of the logo you made for this site (like “our-logo.gif”).

As an example, this is what the completed line would look like, after having added your logo next to the MIT OpenCourseWare logo:

<h1><a href="/OcwWeb/web/home/home/index.htm"><img src="/OcwWeb/images/logo-ocw-

home_new.gif" alt="MIT OpenCourseWare" width="289" height="36" /></a><a

href="http://www.smithville.edu"><img src="/OcwWeb/images/our-logo.gif" style="padding-

left: 15px;" /></a></h1>

Adding text to the home page

Text about your organization and/or mirror site may be added to replace the area of the home page

initially labeled as “Featured Course.” See the example below.

Here’s how to add the text: Using the same four index.htm files referenced in the previous portion of this document, search for

the text “FEATURED COURSE”. You can replace this text with whatever you would like the heading for your text to be. It will remain red and bold as long as you do not change any of the surrounding HTML.

Below that, you will see 6 lines of html that begins with “<div” and ends with </div>. Leave those two lines intact, but you may delete everything in between those two lines and replace it with your own text and images.

This area is limited to 320px wide, but will extend vertically to suit as much or as little content as you choose to add.


Impact of customization on site updates

If you do any customization of your mirror site, please keep in mind that these changes could be overwritten

when you update the mirror site via our rsync process.

If you make the logo and/or text changes to the home page, please make sure to save a copy of the

customized home page in a separate location so that you can replace the updated home page after


synchronization. If you make the logo change to the page header throughout the site, all you need to do is

maintain a copy of the logo file in a separate location and replace the file after synchronization. The correct

logo will be shown if this process is followed.

Please read through the “Mirror Site Updates” section of this document carefully for more details on how to

reinstate your mirror site’s customized elements after updates.

MIRROR SITE USAGE TRACKING

Overview

After installing the mirror site, the most important task is to configure the mirror site to collect usage

statistics. One possible option to consider is the open-source tool AWStats (http://awstats.sourceforge.net/),

and detailed instructions on installing the tool are provided below. The installers and documentation are also

provided on the external hard drive at: Tools/UsageTracking. In addition to AWStats, there are a number

of other free software programs that can be used as well (you can search for other Web server log tools at

http://www.sourceforge.net).

Monthly Reports

As part of the monthly reporting requirement for mirror sites, we ask that you provide the number of visits

to your mirror site. If you are able to automatically generate the statistics to a Web page (which is

configurable by AWStats- see instructions below), please send the URL to Yvonne Ng, OCW External

Outreach Manager, at [email protected]. This will be sufficient to meet the monthly reporting requirement.

If you are unable to automatically generate the statistics to a Web page, please identify a central point of

contact who we can email on a monthly basis to collect the data. Please provide this contact information to

Yvonne at [email protected].

AWStats Introduction

AWStats is a free tool that generates advanced Web server statistics, graphically. This log analyzer works as

a perl CGI or from a command line and shows you all possible access information contained in your Web

server log, in a few graphical Web pages. It is able to process large log files, often and quickly. It can

analyze log files from all major server tools like Apache log files (NCSA combined/XLF/ELF log format or

common/CLF log format) and IIS (W3C log format) log files.

AWStats reports the following statistics:

- Number of visits, and number of unique visitors

- Visit duration and last visits

- Authenticated users, and last authenticated visits

- Days of week and rush hours (pages, hits, KB for each hour and day of week)




http://www.sourceforge.net/

http://awstats.sourceforge.net/

- Domains/countries of hosts visitors

- Hosts list, last visits and unresolved IP addresses list

- Most viewed entry and exit pages

- Files type

- OS used by visitors

- Browsers used by visitors

- Search engines, keyphrases and keywords used to find your site

- HTTP errors

- And more...

AWStats supports the following features:

- It can analyze many log formats: NCSA combined log files (XLF/ELF), common (CLF), IIS log files (W3C),

WebStar native log files.

- It works from a command line and from a browser as a CGI.

- Updates of statistics can be made from a Web browser and not only from your scheduler.

- It can process log files of unlimited size, and it supports split log files (load balancing system).

- Supports 'not correctly sorted' log files even for entry and exit pages.

- Allows reverse DNS lookup before or during analysis and support DNS cache files.

- Country detection from IP location (geoip) or domain name.

- It supports several languages.

- No need of rare perl libraries. All basic perl interpreters can make AWStats working.

- Dynamic reports as CGI output.

- Static reports in one or framed HTML/XHTML pages, experimental PDF export.

- Look and colors can match your site design, can use CSS.

- Help and tooltips on HTML reported pages.

- Easy to use. There is just one configuration file to edit.

AWStats Requirements

AWStats runs on Unix, Linux and Windows operating systems, requiring only a basic Perl installation. (No

specialized Perl modules are needed). Perl is almost always present on Unix-like operating system platforms,

but usually needs to be installed on Windows. The Perl installer should be downloaded from

http://www.activestate.com/Products/activeperl/index.mhtml and installed onto your system. Please place it

in your Tools folder.

To use AWStats, the following specific requirements need to be met:

- Your server must log Web access in a log file you can read.

- You must be able to run perl scripts (.pl files) from command line and/or as CGI (Perl 5.00503 or higher

required to run AWStats 6.0 or higher).

Detailed AWStats documentation is included on the external hard drive at: Tools/ UsageTracking/Unix-

Linux/ awstats_docs.pdf or Tools/ UsageTracking/Windows/awstats_docs.pdf.



MIRROR SITE UPDATES

Overview

OCW has a bi-annual publication schedule, which involves a major publication of courses in April and

October of every year. After each publication is complete, mirror sites should update their content to

include the new and updated courses published, and remove the old courses unpublished, from the main

OCW site. There are two methods for updating a mirror site’s content: (1) for those mirror sites who have no

or very limited Internet connectivity, they can send the hard drive back to OCW for a new version of the

entire site to be loaded onto the drive and shipped back, or (2) for those mirror sites that have some type of

Internet connectivity, we recommend updating content via the Internet using an rsync process.

This section contains instructions on how to update your mirror site via the rsync process. Rsync is an open

source utility that provides fast incremental file transfers. The rsync remote-update protocol allows one to

efficiently transfer just the differences between two sets of files even across “slow” (i.e. low bandwidth, high

latency) network connections. This makes rsync ideally suited to synchronizing content when both

endpoints contain different versions of the same files.

Rsync is run in daemon mode on the OCW mirror server, and partner mirror sites usually connect to this

process using a client-side rsync command-line utility. The data transfer between partner site and OCW

occurs over a secure SSH connection.

For more technical overview on rsync technology, see http://samba.anu.edu.au/ftp/rsync/rsync.html.

OCW Rsync Program

OCW maintains a dedicated rsync server (the “mirror server” located at ocw-mirror.mit.edu) which can be

used by partner sites to obtain updated versions of OCW content. The OCW mirror server contains all OCW

material up to the most recent publishing cycle. OCW partner sites can use the freely-available rsync

command-line utility to connect to the mirror server and download content updates.

Mirror sites can rsync with the OCW mirror server on a schedule that is aligned with the publishing

schedule, typically two months after each major publication, to allow time for the content to stabilize on the

main OCW site. At these times, OCW makes its mirror server available and mirror sites can “pull” content

for a period of time.

Key components

The following summarizes the key components of the rsync program:

Rsync is currently available only to registered OCW mirror sites. Access is secured through a private

SSH key, which is provided by the OCW External Outreach Manager.

Content to be synced includes two directories from the OCW Web site: OcwWeb_for_Intranets (or

OcwWeb) and NR.


http://samba.anu.edu.au/ftp/rsync/rsync.html

Audio, video, and other enhanced OCW content is also available to be synced. This content is available

in a third directory called OCWExternal.

If you plan to locally host the audio and video content, make sure to sync with the

OcwWeb_for_Intranets directory, instead of the OcwWeb directory. The OcwWeb directory contains

HTML files with the original OCW links to the audio and video content and would require an Internet

connection to access.

Content will be current as of the most recent major publishing cycle.

Any files that have been deleted from the OCW Web site (e.g. courses that have been retired and/or

updated) will be deleted from your mirror site directories after rsync.

If you have customized HTML files on your mirror site, these files could get overwritten during

rsync, unless you configure your server not to sync these files. In this case you will need to maintain

rsync content updates in a directory that is distinct from your actual Web site DocumentRoot directory.

This is especially important if you decide to customize your mirror site (see “Customization of Mirror

Site” section of this document).

Prerequisites

The following are some of the prerequisites for a mirror site to participate in rsync:

At least 30 GB of disk space to accommodate basic OCW HTML and PDF content and the site’s

anticipated growth over the next 1.5 years, when the site will reach a steady-state size. (At this

point, the size will not grow considerably, as only new courses and updates will be published, while

older versions of courses are unpublished.) As of 9/1/2008, this content (in /OcwWeb_for_Intranets

and /NR) was about 14 GB.

An additional 270 GB of disk space if a mirror site plans to accommodate enhanced OCW content

(audio, video, zip packages, etc) over the next 1.5 years. As of 9/1/2008, this content totaled

about 205 GB.

Rsync 2.6.4 or newer (provided on the external hard drive).

OpenSSH 1.2.32 or newer (provided on the external hard drive).

An SSH private key (provided via email from the OCW External Outreach Manager).

Mirror sites using current versions of the Unix or Linux operating system may already have workable

versions of rsync and ssh installed. If not, then these can be easily be built from source or downloaded as

part of a binary distribution. Please see the appropriate section below for installation instructions.

Mirror sites using a Windows server platform will need to follow the instructions below and use the cwRsync

installer package, which installs both the needed rsync and ssh components.

Rsync Server Connection

In order to connect to the OCW mirror server at ocw-mirror.mit.edu, please follow these steps:


Extract the SSH private key (which is provided in a password-protected .ZIP file, emailed from the OCW

External Outreach Manager.)

Place the SSH private key into the appropriate ssh directory (the path depends on your operating

system and configuration; for Windows, the path may be “ C:\Program Files\cwRsync\bin\ssh\” and for

Unix-like systems, you should locate the administrator’s .ssh/ directory.

The private key is password protected, and this password must be entered each time when you use the

SSH key to connect to ocw-mirror.mit.edu. The password will be communicated to you by OCW.

Now test rsync and ssh by obtaining a directory listing from the mirror server using a command like:

$ rsync -arvz -e "ssh -i .ssh/partner4" [email protected]::

- Please replace the “partner4” text above with the actual name of your key file.

- For Windows users the “.ssh” directory in the command above should be “ssh”. The “.” character is

not always supported in Windows folder names.

You now should be able to mirror all OCW Web content by using commands described in the

subsequent sections.

Recommended Rsync Configuration

Please make sure that your local rsync directories being synchronized with the OCW mirror server are

NOT the same as your Web site DocumentRoot directory. Instead, it is recommended that partner sites

maintain separate locations in the file system to hold rsync updates. This is especially recommended for

the OcwWeb/ and NR/ directories. Although this will require a manual reconciliation between the rsync

updates and the live mirror content, this configuration ensures that the content on the live mirror site is not

left in an inconsistent state, if the rsync operation is not completed successfully.

In addition, please note the following important considerations:

Due to its large size, it may not be possible for mirror sites to host two copies of the OCWExternal/

directory, and may thus choose to rsync directly with their live OCWExternal directory.

If you are hosting local copies of audio and video files, you will be synching from the

OcwWeb_for_Intranets directory on the mirror server. Please make sure that you construct the rsync

commands such that your local path points to the OcwWeb directory (not OcwWeb_for_Intranets), since

all of the relative links are written to point to OcwWeb/. For sample commands, please refer to the

following section.

Rsync Commands

General Format

The rsync command to be used typically has the following format:

rsync <command line options> –e “ssh –i <location of the private key file>/<name of the key file>”

<remote user name>@<remote host name>::<remote path> <local path> 1> <rsync log file path>

2><rsync error log file path>



Command line options: -arvz --stats

Location of the private key file: SSH directory under the home directory of the administrator i.e.

~/.ssh

Name of the key file: identityfile

Remote User Name: ocwpartner

Remote Host Name: ocw-mirror

Remote path: OcwWeb_for_Intranets, NR, or OcwExternal (OcwWeb is also available)

Local Path: Path on the local machine where the content needs to be copied (OcwWeb should be used

here for content synched from OcwWeb_for_Intranets/)

Rsync log file Path: Path on the local machine where the rsync will log it’s output

Rsync error log file Path: Path on the local machine where the rsync will log the output for the

exceptions

Apart from the remote host name, remote server name and remote path, all other parameters can be

modified as needed.

Sample Commands

Once the server has been setup with the rsync functionality and SSH keys as described in the previous

sections, you can start receiving updates to your local OCW Web content using the following commands:

rsync -arvz –-stats –-delete -e "ssh -i ~/.ssh/identityfile" [email protected]::

OcwWeb_for_Intranets OcwWeb 1> rsync.log 2> error.log

rsync -arvz –-stats –-delete -e "ssh -i ~/.ssh/identityfile" [email protected]::NR NR 1>

rsync.log 2> error.log

rsync -arvz –-stats –-delete -e "ssh -i ~/.ssh/identityfile" [email protected]::OcwExternal

OcwExternal 1> rsync.log 2> error.log

Please note: For Windows users the “.ssh” directory in the command above should be “ssh”

The “.” character is not always supported in Windows folder names.

Please also keep the following in mind when using these rsync commands:

• These commands should be adequate for most applications. Other rsync command options can be

used, as described in the additional documentation on the external hard drive.

• These commands will use the SSH key file (provided by OCW) to connect to the OCW mirror server,

and will begin mirroring OCW content directories (OcwWeb_for_Intranets, NR, OcwExternal) to

corresponding directories on your Web server.

• These commands have been tested on a BASH shell. Other shells may require different syntax.

• The target directories will be created within the rsync directories on your server if they do not exist.

• Make sure that the sync’d up directories are then (manually) copied to the Web accessible directories.

• Please verify that any updated files moved to your server's DocumentRoot directory have permissions

that are appropriate for your environment.


• Make sure that the rsync commands are being run after logging in as root. Executing rsync under a

"root-like" account helps sidestep two additional problems that could occur otherwise:

o After downloading the OcwWeb_for_Intranets content, moving the content to the live Web

directory and a possible restart of the Web servers may require root-like privileges.

o In order to Web-publish the content that was just downloaded, you may need to change

permissions on a large number of files. An account with explicit root privileges is guaranteed

to be able to do this.

• Again, it is strongly recommended that your local (rsync) directories being synchronized with the OCW

mirror server are NOT the same as your Web site DocumentRoot directory. Instead, it is recommended

that partner sites maintain separate locations in the filesystem to hold rsync updates. This ensures

that, if the rsync operation is not completed successfully, the content on the live Web site is not left in

an inconsistent state.

Rsync Logging

Rsync logs provide useful information for mirror site partners. For example, the logs detail which files have

been added or deleted during an rsync run.

As mentioned earlier, when obtaining course content from the OCW mirror server, the mirror site

administrators will use a command-line similar to:

$ rsync -arvz –-stats –-delete -e "ssh -i ~/.ssh/identityfile" [email protected]::

OcwWeb_for_Intranets OcwWeb 1> rsync.log 2> error.log

The corresponding file transfer log, called rsync.log in the above example, can be accessed to view

additional details on the content transfer process. Important summary information, such as the number of

files transferred and the total transferred file size, is contained at the end of this log file and can be used to

confirm whether or not the rsync process was successful. A sample of this summary found in the log file can

be seen below.

Number of files: 195

Number of files transferred: 179

Total file size: 1052231 bytes

Total transferred file size: 1052231 bytes

Literal data: 1052231 bytes

Matched data: 0 bytes

File list size: 3985

Total bytes sent: 3668

Total bytes received: 257140

sent 3668 bytes received 257140 bytes 57957.33 bytes/sec

total size is 1052231 speedup is 4.03

When the administrator downloads the same directory again several months later, after OCW removes

obsolete files and sub-directories, the summary information at the end of the log may look something like:


Number of files: 18

Number of files transferred: 0

Total file size: 166809 bytes

Total transferred file size: 0 bytes

Literal data: 0 bytes

Matched data: 0 bytes

File list size: 533

Total bytes sent: 88

Total bytes received: 609

sent 88 bytes received 609 bytes 278.80 bytes/sec

total size is 166809 speedup is 239.32

These examples show that both new file/directory creations and deletions are clearly shown in the rsync

transfer log.

Rsync Troubleshooting

Any problems with rsync operation can be reported by sending an e-mail to [email protected].

1. The rsync connection is getting denied by the OCW Mirror Server.

Reason: If the rsync connection is getting refused by the OCW Mirror Server, it

may mean that the internal rsync on the OCW Mirror Server is currently taking

place. This operation blocks any external SSH access on the OCW directories so

that the content being synced by the partner mirror sites is not in an inconsistent

state i.e. half updated and half out-of-date. This operation takes around half an

hour to complete.

Solution: Retry rsync commands after half an hour. If the connection keeps getting

refused, please refer to the “How to report issues” section to report the matter.

2. The given rsync command does not work and throws errors.

If the output from rsync command is being logged, the following error messages will appear in the log

file rather than on the console:

Error message: rsync: command not found

Reason: rsync command path is not in the PATH.

Solution:

Option 1

1. Find out the path where rsync was installed. (generally /usr/local/bin)



2. Add the rsync path in the PATH parameter so that rsync can be run from anywhere.

Option 2

1. Run the rsync command with the full path:

/usr/local/bin/rsync -arvz –-stats –-delete -e "ssh -i ~/.ssh/identityfile" ocwpartner@ocw-

mirror.mit.edu::OcwWeb_for_Intranets OcwWeb

Error message: rsync: Failed to exec ssh: No such file or directory

Reason: SSH path is not set

Solution:

Option 1

Add the SSH path in the PATH parameter so that SSH can be run from anywhere. You can get the

SSH path by running the following command:

which ssh

Option 2

Run the SSH command with the full path:

/usr/local/bin/rsync -arvz –-stats –-delete -e "/usr/local/bin/ssh -i ~/.ssh/identityfile"

[email protected]::OcwWeb_for_Intranets OcwWeb

Warning: Identity file ~/.ssh/identityfile does not exist.

Reason: Identityfile has not been placed in the correct directory

Solution:

1. Make sure you are logged in as the user which has to run the rsync command.

2. Go to .ssh directory.

3. Make sure Identity file exists there.

4. Now run the command again.

Error message: host/servname not known

Reason: Incorrect spelling for the remote server

Solution: Make sure that the remote host name is ocw-mirror.

Issue: Password is not accepted

Reason: Incorrect spelling for the remote user name

Solution: Make sure that the remote user name is ocwpartner.

Error message: link_stat failed: No such file or directory


Reason: Incorrect remote path

Solution: Make sure that the remote path is either one of the following

/var/https/jakarta-tomcat-4.0.6-LE-jdk14-prod/Webapps/OcwWeb_for_Intranets

/var/https/jakarta-tomcat-4.0.6-LE-jdk14-prod/Webapps/NR

/var/https/jakarta-tomcat-4.0.6-LE-jdk14-prod/Webapps/OcwExternal

Error message: failed to set times on <directory name>

Reason: Directory does not have the correct permissions set

Solution: Make sure that the directory to which the content is being copied has write permissions

for group and other.

3. Incomplete rsync operation

To determine if the rsync operation was not completed, check the following:

If the command is being run from the command line in the foreground, there will be an error

thrown by rsync which will appear on the console like “some files could not be transferred.”

One can also use the following command options with rsync in order to make it robust. These

command line options will help in catching any abrupt abnormal termination of the rsync

command. (Detailed description of each of these options can be found at:

http://hpux.connect.org.uk/hppd/hpux/Networking/Admin/rsync-2.6.4/man.html)

o --checksum option

Forces the sender to checksum all files using a 128-bit MD4 checksum before

transfer. Checksum is then explicitly checked on the receiver

Any files of the same name which already exist and have the same checksum and

size on the receiver are not transferred.

This option can be quite slow.

Help in reducing the number of transfers if rsync has to be resumed again.

o –partial

By default, rsync will delete any partially transferred file if the transfer is interrupted.

This option is used to keep partially transferred files so that a subsequent transfer of

the rest of the file is much faster.

o --partial-dir=DIR

To specify a DIR that will be used to hold the partial data instead of writing it out to

the destination file.

On the next transfer, rsync will use a file found in this dir as data to speed up the

resumption of the transfer and then deletes it after it has served its purpose.

o –-progress

Print information showing the progress of the transfer. Implies --verbose if it wasn't

already specified.


When the file is transferring, the data looks like this:

782448 63% 110.64kB/s 0:00:04

Current file size

Percentage of the transfer that is complete

Current calculated file-completion rate

Estimated time remaining in this transfer

After a file is complete, the data looks like this:

1238099 100% 146.38kB/s 0:00:08 (5, 57.1% of 396)

Final file size

Percentage of the transfer that is complete

Final transfer rate for the file

Amount of elapsed time it took to transfer the file

Total-transfer summary in parentheses.

How many files have been updated

What percent of the total number of files has been scanned

oOnce it has been determined that the previous rsync operation ended abnormally, it can be

resumed by invoking the rsync commands again.

APPENDIX

Search Functionality

The search functionality will not work on your mirror site by default, because the search on the OCW site

relies on code hosted on our servers and is not localizable. This section offers potential solutions for mirror

sites that wish to provide some form of site-wide content search to their users.

There are two general ways we recommend to make the OCW mirror content searchable, due to their

relative simplicity and ease of use. When implementing a search feature you have to first decide which of

the following statements applies to your site:

1) You DON'T have a reliable or high-bandwidth Internet connection AND you use a Web server to publish

the OCW content in your local area network (LAN) only.

2) Your site uses OS file sharing mechanisms (Windows File-sharing, Samba or NSF) to publish content on

your local area network (LAN) only.

3) Your site is well connected to the Internet AND uses a Web server to publish the OCW content to locations

outside your local area network (LAN).


Option 1: Regain Desktop search

Overview

If you fall into one of the first two categories, you can implement a content search feature using an

opensource software product called Regain Desktop Search (http://regain.sourceforge.net/), which is

installed on the site's Web server.

Regain is a Java search engine based on the Jakarta Lucene project. It provides indexing and searching files

for plenty of formats (currently HTML, XML, Excel, Powerpoint, Word, PDF and RTF). The Desktop Version

allows users to search both content stored on a server's local hard disk drive or published on a Web site.

The Desktop Version runs on both Windows and Linux machines, and is available in the form of a simple-to-

use installer package. This package will install all the necessary components, Java and a small dedicated

Webserver. The installer files for Windows and Linux/Unix operating systems are provided on the external

hard drive at: Tools/Search.

Please Note:

If you are running other Unix-like operating system (Solaris, *BSD, etc) on your server, you will

need to obtain the Regain Server Version, which runs as a Web application within Tomcat (or other

Java servlet container such as Resin or Jetty). Thus the Server version requires you to install Java

AND Tomcat as appropriate for your server.

Regain software is intended to completely replace the Search and Advanced Search fields that are

visible on your mirror site. Those fields will not work. Instead for search your users will visit

separate unique URLs (see below).

Desktop versus Server Versions

(Documentation here adapted from http://regain.murfman.de/wiki/en/index.php/Main_Page)

Both Regain Desktop and Regain Server include the crawler, which is needed to create the search index,

and the search mask, which is needed for searching on a search index. Also the configuration of both

variants is done in the same way: XML files CrawlerConfiguration.xml and SearchConfiguration.xml, but the

desktop search offers a Web interface that writes the most important settings to these two files.

The Desktop Search also comes with a small program that manages these two parts. On Windows, this

program integrates itself in the task bar and automatically starts the crawler, so the search index is updated

regularly or whenever the configuration has changed. Furthermore Regain Desktop provides a Web server,

which is needed by the search mask in order to work.

For the Server Search, the crawler is an independent program, which the administrator has to call by hand

(or automated) in order to update the search index. Furthermore the administrator has to run the search

mask in a servlet engine. The Web server is not delivered with Regain.

Technically speaking, the Desktop Search is a stand-alone application, an application that needs -- aside

from Java -- no more programs to work. The Server Search is split in the crawler on the one hand, which is a

stand-alone console application and the search mask on the other hand, which is a .war archive that has to


http://regain.murfman.de/wiki/en/index.php/Main_Page

http://regain.sourceforge.net/

be integrated in a Java servlet engine (like Tomcat).

Installation and Configuration of Regain Desktop

1. Use the appropriate installer for your operating system from the external hard drive at: Tools/Search.

For Windows you may use the Windows-installer for installation, under other platforms you have to unpack

the zip file. Follow the step-by-step instructions.

2. Regain must be started if you want to search for something. On Windows you can recognize if Regain is

started by the regain-symbol (a blue "r") in the task bar beside the clock.

3. The Windows-installer sets regain Regain to start automatically (even after rebooting). At the first start

your browser will come up and show the welcome page. If you have downloaded the zip file for Linux then

you can start Regain by entering the following commands in the console:

cd [Regain-Directory]

java -jar regain.jar

Please note the following:

Regain Desktop ships with its own built-in Webserver. You will need to make sure that this

Webserver application is not bound to the same TCP port as your primary Webserver!

Regain Desktop also requires Java to be installed on your server. Even a small run-time version of

Java will work.

Option 2: Google search

Mirror sites falling into the third category, which have a reliable Internet connection, can implement a

content search feature by leveraging the free indexing service provided by the Google search engine

company.

This involves two things:

Invite Google to index your Website by registering here: http://www.google.com/addurl/?

continue=/addurl

Allow visitors to search the content on your site using a modified version of the Google search

script below hosted on your server. (Script courtesy of Dr. Dilvan de Abreu Moreira, University of

São Paulo, Brazil)

------- Sample JSP script ------------------------------------------------------------------------

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">

<html lang="en">

<head>

<title>MIT OpenCourseWare | Advanced Search</title>

<meta name="Author" content="Dilvan de Abreu Moreira">

<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">


http://www.google.com/addurl/?continue=/addurl

http://www.google.com/addurl/?continue=/addurl

<meta http-equiv="Pragma" content="no-cache">

<link rel="stylesheet" href="../style/core.css" type="text/css">

</head>

<body >

<div align="left">



<table class="header" width="100%" cellpadding="0" cellspacing="0" border="0" summary="Header Table">

<tr>

<td valign="top" width="100%" height="10"><img src="../images/spacer.gif" alt="" width="1" height="1"

border="0"></td>

</tr>

<tr>

<td width="100%">

<table width="100%" cellpadding="0" cellspacing="0" border="0" summary="Navigation and Logo">

<tr>

<td width="10">

<img src="../images/spacer.gif" alt="" width="1" height="1" border="0">

</td>



<td align="left" valign="top">

<a href="../index.htm"><img src="../images/logo-ocw-home.gif" alt="MIT OpenCourseWare" width="180"

height="21" border="0"></a>

</td>



<td align="right" valign="top">



<table cellpadding="0" cellspacing="0" border="0" summary="Top Site Navigation">

<tr>


<a href="../index.htm"><img src="../images/ocw-home.gif" alt="OCW Home" width="61" height="12"

border="0"></a>

</td>


<a href="../Global/all-courses.htm"><img src="../images/course-list.gif" alt=

"Course List" width="79" height="12" border="0"></a>

</td>


<a href= "../Global/AboutOCW/about-ocw.htm"><img src="../images/about-ocw.gif" alt= "About OCW"

width="74" height="12" border="0"></a>

</td>


<a href="../Global/OCWHelp/help.htm"><img src="../images/help.gif" alt="Help with OCW" width="36"


</td>



<a href="../jsp/feedback.jsp?Referer="><img src="../images/feedback.gif" alt="Feedback" width="61"


</td>

</tr>

</table>

</td>



<td width="10"><img src="../images/spacer.gif" alt="" width="1" height="1" border="0"></td>

</tr>

</table>

</td>

</tr>

<tr>

<td width="780" height="10" valign="top"><img src="../images/spacer.gif" alt="" width="1" height="1"

border="0"></td>

</tr>



<tr class="redbar">


border="0"></td>

</tr>



</table>



<table width="200" cellpadding="0"cellspacing="0" border="0" summary="Left-Hand Column">

<tr>

<td class="leftnav" width="10%" height="480" align="left" valign="top">

<table cellpadding="0"cellspacing="0" border="0" summary="Search Form" width="200">

<tr>

<td><img height="0" alt=""

src="/OcwWeb/images/spacer.gif" width="200" align="top" border="0"></td>

</tr>

</table>

</td>



<td width="10" align="left" valign="top"><img src="../images/spacer.gif" alt="" width="10" height="1"

border="0"></td>



<td width="100%" align="left" valign="top"><table width="560" cellpadding="0" cellspacing="0" border="0"

summary="Main Content Header">


<tr>


border="0"></td>

</tr>



<tr>

<td valign="top"><span class="breadcrumb">» <a href="../index.htm">OCW

Home</a></span> <p class="pagetitle">Advanced Search</p></td>

</tr>

<tr>

<td height="10"><img src="../images/spacer.gif" alt="" width="10" height="1" border="0"></td>

</tr>

<tr>

<td><img src="../images/dottedline_560.gif" alt=""></td>

</tr>





</table>

<p class="caption">Select search options below, then click the Google Search button, or press Enter.</p>

<p>Only Google search is available for this mirror site. You can search the original OCW site at MIT using <a

href="http://ocw.mit.edu/OcwWeb/jsp/AdvancedSearch.jsp">Advanced Search</a>.</p>



<FORM method=GET action="http://www.google.com/search">

<input type=hidden name=ie value=UTF-8>

<input type=hidden name=oe value=UTF-8>

<TABLE bgcolor="#FFFFFF"><tr>

<td>

<INPUT TYPE=text name=q size=31 maxlength=255 value="">

<INPUT type=submit name=btnG VALUE="Google Search">

<font size=-1>

<input type=hidden name=domains value="java.icmc.usp.br"><br>

<input type=radio name=sitesearch value="java.icmc.usp.br"> java.icmc.usp.br

<input type=radio name=sitesearch value="java.icmc.usp.br/OcwWeb" checked> java.icmc.usp.br/OcwWeb

<input type=radio name=sitesearch value="java.icmc.usp.br/NR/rdonlyres"> java.icmc.usp.br (OCW pdfs) <br>

</font>

</td></tr></TABLE>

</FORM>



<p>Google search is subject to Google indexing. It may take some time until Google indexes the entire mirror. In the mean

time, search may be incomplete or even return nothing.</p>


<p class="forty"><img src="../images/spacer.gif" alt="" width="1" height="1" border="0"></p></td>

<td width="10" align="left" valign="top"><img src="../images/spacer.gif" alt="" width="1" height="1"

border="0"></td>

</tr>

<tr>

<td class="leftnav"><a target="_new" href="http://Web.mit.edu/"><img src="../images/logo-mit-new-on-cccc99.gif"

alt="MIT Home" width="38" height="21" class="tenten" border="0" ></a></td>

<td class="white" colspan="3"><img src="../images/spacer.gif" alt="" width="1" height="1" border="0"></td>

</tr>





<tr>

<td colspan="4" height="10" class="footer"><img src="../images/spacer.gif" alt="" width="1"

height="1" border="0"></td>

</tr>

<tr>

<td width="100%" valign="top" class="footer" colspan="4">

<div class="ten">

<span class="footercopy"><img src="../images/footernav.jpg" border="0"

usemap="#map" width="435" height="13" alt=""><br>

Your use of the MIT OpenCourseWare site and course materials is subject to

the

conditions and terms of use in our Legal Notices section.</span>

</div>

</td>

</tr>

<tr>


<td colspan="4" height="10" class="footer"><img src="../images/spacer.gif" alt="" width="1"

height="1" border="0"></td>

</tr>

</table>

<map name="map">

<area shape="rect" coords="4,2,292,18" href="http://Web.mit.edu" target="_blank" alt="Massachusetts Institute of

Technology">

<area shape="rect" coords="301,2,381,11" href="/OcwWeb/Global/terms-of-use.htm" alt="Terms of Use">

<area shape="rect" coords="390,1,429,12" href="/OcwWeb/Global/privacy.htm" alt="Privacy">

</map>



</table>

</div>

</body>

</html>

---------------- End of Example Script ---------------------------------------------------------------------------

Please note the following:

The example script shown above will need to be modified by your system administrator to use URL

parameters appropriate for you organization.

The script is a JSP script and as such needs to be run within a Java application server like Tomcat,

Resin, Jetty or others. Thus you will need to install Java and Tomcat as appropriate for your server.

If you do not have Java available in your environment, OCW can provide you with a Python version

of this script to be run as a CGI script.

The script above is intended to completely replace the Search and Advanced Search fields that are

visible on the OCW Web pages made available on your OCW hard disk drive. Those fields will

not work. Instead, for search your users will visit a separate unique URL.

Feedback Functionality

The feedback functionality from the OCW main site may not be transferable to your mirror site if you don’t

have the Tomcat Java application server or a reliable Internet connection. This section offers potential

solutions for mirror sites that wish to provide some option of feedback to their users.

The main OCW site gathers feedback information from visitors using a Java-based Web form (feedback.jsp),

which emails visitor comments to the OCW administrators. The link to that form can be found in the upper-

right hand corner of the page header throughout the site, as well as in the gray information box of every

course home page.

To offer feedback functionality as part of your mirror site, you can use a simplified Web form,

“feedback.jsp”, or a Perl based CGI script, “feedback.cgi”, both provided below.


For those mirror sites who have Java installed, the feedback.jsp code below serves as a drop-in replacement

for the non-functioning feedback.jsp script currently found on the external hard drive at:

OcwWeb/jsp/feedback.jsp. (You can rename the old feedback.jsp file and substitute the new script shown

below.)

For those mirror sites who don’t have Java installed, the perl script “feedback.cgi”, shown below, processes

comments submitted via the Web form and appends it to a text file in your server's hard disk drive, which

the site administrator can periodically review and share with OCW. This alternative mechanism should work

for any mirror site, especially the ones that don't use email internally or have Java.

Option 1: Using a simplified Web form

The feedback.jsp script is provided below. Please note that the feedback.jsp page is not really a Java Server

Page, because it contains no real Java code. So in order for this file to work as a direct replacement for the

original feedback.jsp file, you will need to adjust your Web server's configuration to have files with the *.JSP

extension treated as simple HTML documents. In Apache that can be done by adding the “AddType text/html .jsp” directive to the httpd.conf file and restarting the server. Similar options are available

in other Web server software. In addition, this script can be freely modified by your site administrator to

meet your organization's specific needs.

---------------------replacement feedback.jsp script --------------------------------------------------------------<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN""http://www.w3.org/TR/html4/loose.dtd"><html><head><title>Ocw Mirror Site Feedback Form</title><meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1"></head>

<body><form method="post" action="/cgi-bin/feedback.cgi"> <h2>We welcome your feedback! </h2> <p>With over 1100 courses now available this site is fulfilling the promise of OpenCourseWare. Feedback from visitors will help us to understand how to improve your OCW Web site . </p> <p>Please fill out the feedback form below.</p>

<p> <input type="hidden" name="submitaddress" value="administrator@localhost"> <br> Your name:<br> <input type="text" name="name"> <br> Your email:<br>

<input type="text" name="email">

<br></p><p>Your comment:<br> <textarea name="comment" cols="100" rows="20" wrap="OFF"></textarea> <br> <input name="submit" type="submit"> </p></form>

</body></html>


---------------------end replacement feedback.jsp script ----------------------------------------------------------

Option 2: Using a Perl based CGI script

The feedback.cgi script is provided below. In order to work, the feedback.cgi file needs to be placed in your

Webserver's cgi-bin directory. In addition, you must make certain that a working Perl environment is

available on your server and that CGI support is enabled in your Webserver. The Perl installer should be

downloaded from http://www.activestate.com/Products/activeperl/index.mhtml and installed onto your

system. Please place it in your Tools folder. In addition, this script can be freely modified by your site

administrator to meet your organization's specific needs.

-----------------feedback.cgi-----------------------------------------------------------------------------------------

#!/usr/bin/perl

# The following accepts the data from the form and splits it into its component parts

if ($ENV{'REQUEST_METHOD'} eq 'POST') {

read(STDIN, $buffer, $ENV{'CONTENT_LENGTH'});

@pairs = split(/&/, $buffer);

foreach $pair (@pairs) {

($name, $value) = split(/=/, $pair);

$value =~ tr/+/ /;

$value =~ s/%([a-fA-F0-9][a-fA-F0-9])/pack("C", hex($1))/eg;

$FORM{$name} = $value;

}

# Then append user feedback to a file on the filesystem

open(MYOUTFILE, ">>/var/www/log/feedback.log"); #open for write, append

print MYOUTFILE "To: $FORM{submitaddress}\n";

print MYOUTFILE "From: $FORM{name}, reader\n";

print MYOUTFILE "Reply-to: $FORM{email}\n";

print MYOUTFILE "Subject: Feedback from $FORM{name} \n\n";

print MYOUTFILE "$FORM{name} wrote:\n\n";

print MYOUTFILE "Comment: $FORM{comment}\n\n";

print MYOUTFILE "=====================================\n";

close (MYOUTFILE);

&thank_you; #method call

}



#The code then goes on to generate the thank-you page

sub thank_you {

print "Content-type: text/html\n\n";

print <<EndStart;

<html>

<head>

<title>Thank You</title>

</head>

<body bgcolor="#ffffff" text="#000000">

<h1>Thank You</h1>

<p>Your feedback has been received. Thanks for sending it.</p>

<hr>

EndStart

print "<p>You wrote:</p>\n";

print "<blockquote><em>$FORM{comment}</em></blockquote>\n\n";

print <<EndHTML;

</body>

</html>

EndHTML

exit(0);

}

------------------end feedback.cgi-----------------------------------------------------------------------------------

AWStats Installation with Apache on Windows

Install Perl, if necessary

The Perl installer should be downloaded from http://www.activestate.com/Products/activeperl/index.mhtml

and installed onto your system. Please place it in your Tools folder.



Install AWStats

An installer is included on the external hard drive at: Tools/UsageTracking/Windows/.

You may also download the latest Windows EXE installer at:


To install, double-click on the installer file -> This will launch a wizard based installer Click “Next”, “I Agree” and “Install” in sequence, when prompted A Windows Command prompt will open (see screenshot below) which guides user through the rest of

the configuration process. Answer the prompts appropriately for your environment.

In our example below, the Web site name is “www.ocwpartner.org”; the instructions below are written using

that example name.


http://www.ocwpartner.org/


The subsequent steps (still within the same command window) are as follows:

-----> Need to create a new config file ?

Do you want me to build a new AWStats config/profile file (required if first install) [y/N] ? y

-----> Define config file name to create

What is the name of your Web site or profile analysis ?

Example: www.mysite.com

Example: demo

Your Web site, virtual server or profile name:

> www.ocwpartner.org (this is just an example. Use your own Website name here!)

-----> Create config file 'C:/Program Files/AWStats\wwwroot\cgi-bin\awstats.www. ocwpartner.org.conf'

Config file C:/Program Files/AWStats\wwwroot\cgi-bin\awstats.www.ocwpartner.org.conf created.

-----> Add update process inside a scheduler

Sorry, for Windows users, if you want to have statistics to be updated on a regular basis, you have to add

the update process in a scheduler task manually (See AWStats docs/index.html).

Press ENTER to continue...

A SIMPLE config file has been created: C:/Program Files/AWStats\wwwroot\cgi-bin\

awstats.www.ocwpartner.org.conf

You should have a look inside to check and change manually main parameters.

You can then manually update your statistics for 'www.ocwpartner.org' with command:

> perl awstats.pl -update -config=www.ocwpartner.org

You can also read your statistics for 'www.ocwpartner.org' with URL:

> http://www.ocwpartner.org/awstats/awstats.pl?config=www.ocwpartner.org

Press ENTER to finish...

Hitting ENTER here will close the command window, and will open up a AWStats documentation page in your browser. (The entire AWStats documentation has now been installed in C:\Program Files\AWStats\docs on your server's hard disk drive)

Switch windows back to the Wizard-based AWStats installer, and click on the “Close” button to quit the installer. This concludes the software installation.

Restart the Apache Web server Go to C:\Program Files\AWStats\wwwroot\cgi-bin on your Windows server. Locate the AWStats configuration file for your Web server. Its name should be something like:

“awstats.www.ocwpartner.org.conf” in our example (it usually follow a format like

“awstats.site.name.com.conf”).

Open this file with a text-editor like Notepad, included in Windows.

Within the file locate the option: AllowToUpdateStatsFromBrowser=0

Change that to: AllowToUpdateStatsFromBrowser=1


http://www.ocwpartner.org.conf/


Locate the line LogFile="C:/Program Files/Apache Group/Apache2/logs/access.log"

Make sure your the Web server access log file-path is accurate here. Also, note the direction of the

slashes in the path example above!

Save the AWStats configuration file, and exit the text-editor.

Open a command window on your server at C:\Program Files\AWStats\wwwroot\cgi-bin\

Run the following command to update your AWStats statistics for the first time:

> perl awstats.pl -update -config=www.ocwpartner.org (in our example)

You now can view (and update) your Website statistics at

http://www.ocwpartner.org/awstats/awstats.pl?config=www.ocwpartner.org (in our example)

Your AWStats installation on Windows should now be complete! However, you may choose to create a task in the Windows Scheduled Tasks which keeps the Website statistics updated automatically by running the perl command above nightly.

AWStats Installation with IIS on Windows

Install Perl if necessary

The Perl installer should be downloaded from http://www.activestate.com/Products/activeperl/index.mhtml

and installed onto your system. Please place it in your Tools folder.

Configure IIS logging

Configure IIS to create logs in the "Extended W3C log format" (You can still use your own custom log format

but setup is easier if you use the standard extended format). To do so, start the IIS management console

snap-in, select the appropriate web site and open its Properties. Choose "W3C Extended Log Format", then

Properties, then the Tab "Extended Properties" and uncheck everything under Extended Properties. Once

they are all cleared, check just the following fields:

date

time

c-ip

cs-username

cs-method

cs-uri-stem

cs-uri-query

sc-status

sc-bytes

cs-version

cs(User-Agent)

cs(Referer)

To be sure the log format change is effective, you must stop IIS, restart IIS and go to your homepage. This is

an example of the type of records you should find in the new log file:

2000-07-19 14:14:14 62.161.78.73 - GET / 200 1234 HTTP/1.1 Mozilla/4.0+(compatible;

+MSIE+5.01;+Windows+NT+5.0) http://www.from.com/from.htm



http://site.name.com/awstats/awstats.pl?config=site.name.com


Installing AWStats

The software installer for AWStats on Windows with IIS web server is the same as for the Apache install

documented above. Therefore you should follow the instructions above for the Apache install.

The differences between the IIS and the Apache configuration steps occur mostly after you have run the GUI

installer. The IIS-specific steps are described below:

* Step 1:

Copy the contents of the AWStats from C:\Program Files\AWStats\wwwroot\cgi-bin to your IIS server's cgi-bin

directory (this includes awstats.pl, awstats.model.conf, and the lang, lib and plugins sub-directories).

Move AWStats icon sub-directories and its content into a subdirectory of your wwwroot by your web For

example ...\wwwroot\icon\.

* Step 2:

• Create a configuration file by copying awstats.model.conf to a new file named

awstats.ocwpartner.org.conf (where "ocwpartner.org" is an example). The real value you should use here is

your domain or virtual host name. This new file must be saved in the same directory as awstats.pl (i.e. In the

cgi-bin directory).

• Open the AWStats configuration file for your web server with a text-editor included in Windows. In our

example open “awstats.www.ocwpartner.org.conf”

• Change the LogFile value to the full path of your web server log file. Make sure your the web server

access log file-path is accurate here. Also, note the direction of the slashes in the path example above!

• Change the LogType value to “W” for analyzing web log files.

• Change the LogFormat to 2 if you are using the "Extended W3C log format" as described above ; in

the case of a custom format, list the IIS fields being logged, for example:

LogFormat="date time c-ip cs-username cs-method cs-uri-stem cs-uri-query sc-status sc-bytes cs-version

cs(User-Agent) cs(Referer)"

• Change the DirIcons parameter to reflect relative path of icon directory.

• Set the SiteDomain parameter to the main domain name or the intranet web server name used to

reach the web site being analyzed (Example: www.ocwpartner.org).

• Within the file locate the option: AllowToUpdateStatsFromBrowser=0


• Save the AWStats configuration file, and exit the text-editor.

• You now can view (and update) your website statistics at

http://www.ocwpartner.org/awstats/awstats.pl?config=www.ocwpartner.org (in our example)

• Your AWStats installation on Windows and IIS should now be complete!

(However, you may choose to create a task in the Windows Scheduled Tasks which keeps the website

statistics updated automatically by running the perl command above nightly)

AWStats Installation on Unix-like Operating Systems

Install Perl, if necessary


Perl is often already installed on Unix-like operating systems, so you may not actually need to install

anything. Check with your local system administrator to see if Perl is available. Just in case it is not, the Perl

installer should be downloaded from http://www.activestate.com/Products/activeperl/index.mhtml and

installed onto your system. Please place it in your Tools folder.

You can also download the software from here: http://www.perl.com/download.csp

To install Perl from source you will need access to common Unix build tools on your server. These include

the “Gcc” compiler and the “Make” program. If these are available then do the following to install:

uncompress the Perl source code archive

> tar -zxf stable.tar.gz

move into the resulting source directory

> cd stable

build Perl

>./configure

>make

>make install

Install AWStats

An installer is included on the external hard drive at: Tools/UsageTracking/Unix-Linux.

You may also download the latest installer from the web at: http://awstats.sourceforge.net/

Move the awstats-6.8.tar.gz file located on the hard disk drive to a directory on your web server

Uncompress the awstats-6.8.tar.gz file by running:

>gzip -d awstats-6.8.tar.gz

>tar -xf awstats-6.8.tar

These steps result in a directory called "awstats-6.8/" on your server

Go into the "awstats-6.8/tools " directory within "awstats-6.8/"

Locate the awstats_configure.pl file. This is the awstats setup script for Unix-like operating systems.

Run the script by using the following command:

>perl awstats_configure.pl"

This script tries to determine your current log format from your Apache web server configuration file

"httpd.conf" (it asks for the path if not found). If you use a common log format for Apache, the script will

suggest changing it to the "NCSA combined/XLF/ELF" format

If you answer yes to the suggested change, the script will automatically modify your httpd.conf by

changing the following directive:

from

CustomLog /yourlogpath/yourlogfile common

to

CustomLog /yourlogpath/yourlogfile combined



http://www.perl.com/download.csp


(See the Apache manual for more information on what this directive means.)

The script will then add, if not already present, the following directives to your Apache configuration file

(note that the "/usr/local/awstats/wwwroot" path might differ according to your distribution or OS:

=======this section gets added========================================

#

# Directives to add to your Apache conf file to allow use of AWStats as a CGI.

# Note that path "/usr/local/awstats/" must reflect your AWStats Installation path.

#

Alias /awstatsclasses "/usr/local/awstats/wwwroot/classes/"

Alias /awstatscss "/usr/local/awstats/wwwroot/css/"

Alias /awstatsicons "/usr/local/awstats/wwwroot/icon/"

ScriptAlias /awstats/ "/usr/local/awstats/wwwroot/cgi bin/"

#

# This is to permit URL access to scripts/files in AWStats directory

#

<Directory "/usr/local/awstats/wwwroot">

Options None

AllowOverride None

Order allow,deny

Allow from all

</Directory>

================================================================

If the above changes were made, the script restarts Apache to apply the changes. To ensure the log

format change is effective take a look at your Apache access log file. The new log entries in the file

(after Apache was restarted) should now look something like this:

62.161.78.75 [dd/mmm/yyyy:hh:mm:ss +0000] "GET / HTTP/1.1" 200 1234

"http://www.from.com/from.html" "Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0)"

Finalizing the AWStats Installation

At this point AWStats is mostly set up. But just to be sure, please follow the steps below to verify or finalize

your site's specific configuration.

The script will ask you for a name for the configuration profile file. Enter an appropriate name such as

that of your web server or the virtual domain to be analyzed, i.e. www.ocwpartner.org (This is just an

example. Use your own Website name here!).

The script will create a new file called awstats.www.ocwpartner.org.conf. The new file location is:

/etc/awstats.

The script ends.

Locate the AWStats configuration file for your Web server. Its name should be something like:

“awstats.<http://www.ocwpartner.org.conf/>www.ocwpartner.org.conf” in our example.


Open this file with your preferred text-editor (Vi, EMACS, Nano, etc)

Within the file locate the option: AllowToUpdateStatsFromBrowser=0


Locate the line LogFile="\var\www\log\access.log"

Make sure that the path to your Web server access log file is accurate here.

Save the AWStats configuration file, and exit the text editor

Change directory to \usr\AWStats\wwwroot\cgi-bin\

Run the following command to update your AWStats statistics for the first time:

> perl awstats.pl -update -config=<http://www.ocwpartner.org/>www.ocwpartner.org (in our example)

You now can view (and update) your Website statistics at

<http://site.name.com/awstats/awstats.pl?config=site.name.com>http://www.ocwpartner.org/awstats/

awstats.pl?config=www.ocwpartner.org (in our example)

Your AWStats installation should now be complete!

You may choose to create a task in Crond to keep the Website statistics updated automatically by

running the perl command above nightly.

This can be done with a crontab entry like:

“50 23 * * * cd /usr/local/awstats/wwwroot/cgi-bin/;perl awstats.pl -config=www.ocwpartner.org -update

> /dev/null”

Rsync Installation

Unix-like operating systems

The systems administrator of a mirror site needs to follow the instructions in this section in order to install

the necessary rsync and SSH command-line utilities on their Unix/Linux server. It is assumed that the server

has a functioning build environment which includes working versions of “gcc” and “make” utilities. Please

note that OpenSSH may already be installed (which is quite likely), so the OpenSSH setup below may not be

necessary.

Rsync Installation

• Un-compress the rsync source code included on the external hard drive at: Tools/Rsync/Unix-Linux/

• Change into the resulting rsync-2.6.8 directory and execute the following commands there:

./configure

make

make install

OpenSSH installation

Prerequisites

You will need working installations of Zlib and OpenSSL.

Zlib 1.1.4 or 1.2.1.2 or greater (ealier 1.2.x versions have problems):

http://www.gzip.org/zlib/



OpenSSL 0.9.6 or greater:

http://www.openssl.org/

(OpenSSL 0.9.5a is partially supported, but some ciphers (SSH protocol 1

Blowfish) do not work correctly.)

OpenSSH can utilise Pluggable Authentication Modules (PAM) if your system

supports it. PAM is standard on Redhat and Debian Linux, Solaris and

HP-UX 11.

NB. If you operating system supports /dev/random, you should configure

OpenSSL to use it. OpenSSH relies on OpenSSL's direct support of

/dev/random. If you don't you will have to rely on ssh-rand-helper, which

is inferior to a good kernel-based solution.

PAM:

http://www.kernel.org/pub/linux/libs/pam/

Building / Installation

To install OpenSSH with default options:

Un-compress the OpenSSH source code included on the external hard drive at: Tools/Rsync/Unix-Linux (It

is also available for download here: ftp://ftp.openbsd.org/pub/OpenBSD/OpenSSH/portable/)

Change into the resulting OpenSSH directory and execute the following commands there:

./configure

make

make install

This will install the OpenSSH binaries in /usr/local/bin, configuration files

in /usr/local/etc, the server in /usr/local/sbin, etc. To specify a different

installation prefix, use the --prefix option to configure:

./configure --prefix=/opt

make

make install

Will install OpenSSH in /opt/{bin,etc,lib,sbin}. You can also override

specific paths, for example:

./configure --prefix=/opt --sysconfdir=/etc/ssh

make

make install


ftp://ftp.openbsd.org/pub/OpenBSD/OpenSSH/portable/



This will install the binaries in /opt/{bin,lib,sbin}, but will place the

configuration files in /etc/ssh.

If you are using Privilege Separation (which is enabled by default)

then you will also need to create the user, group and directory used by

sshd for privilege separation. See README.privsep for details.

If you are using PAM, you may need to manually install a PAM control

file as "/etc/pam.d/sshd" (or wherever your system prefers to keep

them). Note that the service name used to start PAM is __progname,

which is the basename of the path of your sshd (e.g., the service name

for /usr/sbin/osshd will be osshd). If you have renamed your sshd

executable, your PAM configuration may need to be modified.

A generic PAM configuration is included as "contrib/sshd.pam.generic",

you may need to edit it before using it on your system. If you are

using a recent version of Red Hat Linux, the config file in

contrib/redhat/sshd.pam should be more useful. Failure to install a

valid PAM file may result in an inability to use password

authentication. On HP-UX 11 and Solaris, the standard /etc/pam.conf

configuration will work with sshd (sshd will match the other service

name).

There are a few other options to the configure script:

--with-pam enables PAM support. If PAM support is compiled in, it must

also be enabled in sshd_config (refer to the UsePAM directive).

--with-prngd-socket=/some/file allows you to enable EGD or PRNGD

support and to specify a PRNGd socket. Use this if your Unix lacks

/dev/random and you don't want to use OpenSSH's builtin entropy

collection support.

--with-prngd-port=portnum allows you to enable EGD or PRNGD support

and to specify a EGD localhost TCP port. Use this if your Unix lacks

/dev/random and you don't want to use OpenSSH's builtin entropy

collection support.

--with-lastlog=FILE will specify the location of the lastlog file.

./configure searches a few locations for lastlog, but may not find

it if lastlog is installed in a different place.

--without-lastlog will disable lastlog support entirely.

--with-osfsia, --without-osfsia will enable or disable OSF1's Security


Integration Architecture. The default for OSF1 machines is enable.

--with-skey=PATH will enable S/Key one time password support. You will

need the S/Key libraries and header files installed for this to work.

--with-tcp-wrappers will enable TCP Wrappers (/etc/hosts.allow|deny)

support. You will need libwrap.a and tcpd.h installed.

--with-md5-passwords will enable the use of MD5 passwords. Enable this

if your operating system uses MD5 passwords and the system crypt() does

not support them directly (see the crypt(3/3c) man page). If enabled, the

resulting binary will support both MD5 and traditional crypt passwords.

--with-utmpx enables utmpx support. utmpx support is automatic for

some platforms.

--without-shadow disables shadow password support.

--with-ipaddr-display forces the use of a numeric IP address in the

$DISPLAY environment variable. Some broken systems need this.

--with-default-path=PATH allows you to specify a default $PATH for sessions

started by sshd. This replaces the standard path entirely.

--with-pid-dir=PATH specifies the directory in which the ssh.pid file is

created.

--with-xauth=PATH specifies the location of the xauth binary

--with-ssl-dir=DIR allows you to specify where your OpenSSL libraries

are installed.

--with-4in6 Check for IPv4 in IPv6 mapped addresses and convert them to

real (AF_INET) IPv4 addresses. Works around some quirks on Linux.

--with-opensc=DIR

--with-sectok=DIR allows for OpenSC or sectok smartcard libraries to

be used with OpenSSH. See 'README.smartcard' for more details.

If you need to pass special options to the compiler or linker, you

can specify these as environment variables before running ./configure.

For example:

CFLAGS="-O -m486" LDFLAGS="-s" LIBS="-lrubbish" LD="/usr/foo/ld" ./configure


Configuration

The runtime configuration files are installed by in ${prefix}/etc or

whatever you specified as your --sysconfdir (/usr/local/etc by default).

The default configuration should be instantly usable, though you should

review it to ensure that it matches your security requirements.

To generate a host key, run "make host-key". Alternately you can do so

manually using the following commands:

ssh-keygen -t rsa1 -f /etc/ssh/ssh_host_key -N ""

ssh-keygen -t rsa -f /etc/ssh/ssh_host_rsa_key -N ""

ssh-keygen -t dsa -f /etc/ssh/ssh_host_dsa_key -N ""

Replacing /etc/ssh with the correct path to the configuration directory.

(${prefix}/etc or whatever you specified with --sysconfdir during

configuration)

If you have configured OpenSSH with EGD support, ensure that EGD is

running and has collected some Entropy.

For more information on configuration, please refer to the manual pages

for sshd, ssh and ssh-agent.

(Optional) Send survey

$ make survey

[check the contents and make sure there's no sensitive information]

$ make send-survey

This will send configuration information for the currently configured

host to a survey address. This will help determine which configurations

are actually in use, and what valid combinations of configure options

exist. The raw data is available only to the OpenSSH developers, however

summary data may be published.

Problems

If you experience problems compiling, installing or running OpenSSH.

Please refer to the "reporting bugs" section of the webpage at

http://www.openssh.com/

$Id: INSTALL,v 1.70 2005/04/24 07:52:23 dtucker Exp $

See http://www.openssh.com/txt/release-4.3p2 for the release notes.

This is the port of OpenBSD's excellent OpenSSH[0] to Linux and other


http://www.openssh.com/txt/release-4.3p2


Unices.

OpenSSH is based on the last free version of Tatu Ylonen's sample

implementation with all patent-encumbered algorithms removed (to

external libraries), all known security bugs fixed, new features

reintroduced and many other clean-ups. OpenSSH has been created by

Aaron Campbell, Bob Beck, Markus Friedl, Niels Provos, Theo de Raadt,

and Dug Song. It has a homepage at http://www.openssh.com/

This port consists of the re-introduction of autoconf support, PAM

support, EGD[1]/PRNGD[2] support and replacements for OpenBSD library

functions that are (regrettably) absent from other unices. This port

has been best tested on AIX, Cygwin, HP-UX, Linux, MacOS/X,

NetBSD, OpenBSD, OpenServer, Solaris, Unicos, and UnixWare.

This version actively tracks changes in the OpenBSD CVS repository.

The PAM support is now more functional than the popular packages of

commercial ssh-1.2.x. It checks "account" and "session" modules for

all logins, not just when using password authentication.

OpenSSH depends on Zlib[3], OpenSSL[4] and optionally PAM[5].

There is now several mailing lists for this port of OpenSSH. Please

refer to http://www.openssh.com/list.html for details on how to join.

Please send bug reports and patches to the mailing list

[email protected]. The list is open to posting by

unsubscribed users.Code contribution are welcomed, but please follow the

OpenBSD style guidelines[6].

Please refer to the INSTALL document for information on how to install

OpenSSH on your system. There are a number of differences between this

port of OpenSSH and F-Secure SSH 1.x, please refer to the OpenSSH FAQ[7]

for details and general tips.

References

[0] http://www.openssh.com/faq.html

[1] http://www.lothar.com/tech/crypto/

[2] http://www.aet.tu-cottbus.de/personen/jaenicke/postfix_tls/prngd.html

[3] http://www.gzip.org/zlib/

[4] http://www.openssl.org/

[5] http://www.openpam.org


(PAM also is standard on Solaris and HP-UX 11)



http://www.openpam.org/



http://www.aet.tu-cottbus.de/personen/jaenicke/postfix_tls/prngd.html

http://www.lothar.com/tech/crypto/

http://www.openssh.com/faq.html

http://www.openssh.com/list.html


[6] http://www.openbsd.org/cgi-bin/man.cgi?query=style&sektion=9

[7] http://www.openssh.com/faq.html

Windows systems

The systems administrator of a mirror site needs to follow the instructions in this section in order to install

the necessary rsync and SSH command-line utilities on their Windows server, which can then be used to

access the OCW mirror server.

• Locate the cwRsync 2.0.7 installer provided on the external hard drive at: Tools/Rsync/Windows

• Double-click to start the cwRsync installation.

• At the “Choose Components” window the “Secure Channel Wrapper” box need not be checked.

• When asked for the installation directory choose: “C:\Program Files\cwRsync\”

• Click on “Next” until the installation is complete.


http://www.openssh.com/faq.html

http://www.openbsd.org/cgi-bin/man.cgi?query=style&sektion=9