BiblioTech Online Services Technology

Confidential Page 1 21/03/2001

BiblioTech

Technology & System Architecture

BY

BiblioTech Confidential

Prepared By Jonathan Lishawa

Business Development BiblioTech Ltd.

Unit 3, The Piper Centre, 50 Carnwath Rd London SW6 3EG

13th March 2001


1 BIBLIO TECH HISTORY................................................................................................... 5 2 BIBLIO TECH ADVANTAGE............................................................................................ 6 3 TECHNOLOGY OVERVIEW ............................................................................................ 8 4 NETWORK ARCHITECTURE.......................................................................................... 9 5 DATABASE ARCHITECTURE .......................................................................................13 6 SOFTWARE ARCHITECTURE .......................................................................................14 6.1 SOFTWARE BUILD ENVIRO NMENT .............................................................................14 7 SERVER ARCHITECTURE.............................................................................................17

8 STANDARD TRANSACTION FLOW ..............................................................................19 9 PRODUCTION..................................................................................................................20 10 SERVICE LEVEL AGREEMENT....................................................................................21


ACKNOWLEDGEMENTS

Contributing to this article were Richard Backer, Robin Buller, Tom Vivian, Daniel Berranger and Richard Jones.


OVERVIEW

This document describes the overall technology and system architecture behind the services BiblioTech provides. It’s intended to be used by BiblioTech’s system department, members of the development team, management and potential customers/partners who want to understand BiblioTech services in more detail.


1 BIBLIOTECH HISTORY BiblioTech is a leading Internet software development company providing sophisticated interactive online environments. Our Application Service Provider (ASP) model allows for open and collaborative environments, through which applications are receptive to communications with other applications either within the organisational boundaries or beyond. Building on this collaborative enterprise application environment, BiblioTech can offer fully managed solutions and services to organizations. The ASP model BiblioTech provides uses proprietary web-enabled software applications that have been developed and build solutions, that can, collectively and singly, fulfil all existing and future market requirements. Within any given iteration of BiblioTech’s ASP model, there exists the potential to offer a range of value-added services. These value-added services may already exist, be developed or outsourced as appropriate to serve the specific requirements and sensitivities of organisations and their membership.


2 BIBLIOTECH ADVANTAGE

BiblioTech empowers organizations worldwide by providing an online environment with associated web based applications and integrated communication services. BiblioTech enables organizations to immediately deploy a community-building, fully integrated communication service that is scaleable across hundreds of thousands of end-users. BiblioTech’s modular applications use a platform independent proprietary web based interface. These solutions are backed by over five years of development expertise and one of the most experienced technical teams in the industry. Since BiblioTech’s inception in 1996, it has been at the forefront of developing interactive online environments and associated applications. BiblioTech is an Internet ASP with two key business products, launched in 1997 and 1999 respectively:

Postmaster is a free multilingual web based email system which was judged ‘simply the best’ in a November 1999 survey of nearly 300 such services by internet.com, and

Schoolmaster is the only secure multinational multilingual online education portal. It has already registered in excess of 25% of UK senior schools, with 4,300 registered schools worldwide. It won the Intel-sponsored ‘Best Application for Secondary Schools’ at BETT 2000.

2.1.1 PROVEN EXPERIENCE, AND RELIABILITY

BiblioTech has been providing free web based email and other associated services to the general public since 1996, through Postmaster, while schools around the world rely on schoolmaster.net Service Portal every day to provide hundreds of thousands of students with a secure, robust, easy-to-use, 24-hour online education environment catering to their immediate communication needs. No other Web company can rival our long history of actively developing for and supporting the needs of organisations and online communities worldwide.

2.1.2 SERVICE INTEGRATION

BiblioTech’s offers multiple premium service options to organisations and their constituents. BiblioTech has successfully integrated a wide range of communication products directly into this environment, providing organisations with a secure cost effective solution to all their online communication needs. BiblioTech also has the technical capability to rapidly integrate organisation’s existing information systems into the environment, enabling them to enjoy the benefits that such service integration provides. With the open framework supplied by the enterprise environment, integrating third party products and services is also not problematic and is encouraged where advantageous or requested.

2.1.3 BENEFITS OF THE BIBLIOTECH SOLUTION

The solutions and community environment BiblioTech provides are distinguished by numerous technical and operating advantages that allow BiblioTech to offer


organisations a significant and sustainable lead in user service. This allows for applications to be developed or purchased, and integrated at less cost and on a shorter timeframe than most other off the shelf proprietary solutions. Exploiting our history of technical leadership, BiblioTech’s on going investment in software development enables us to offer better products and services for less, which drive new users to, and more revenue per user, on the sites we serve. BiblioTech is one of the only ASP’s with both the technology and the vision to transform the user interface into a comprehensive communication service centre for organisations. Schoolmaster.net, BiblioTech’s education service portal with over 4,000 schools and 500,000 registered students proves this.


3 TECHNOLOGY OVERVIEW

BiblioTech’s software and system architecture is designed to achieve superior levels of service quality based on scalability, reliability, and efficiency. BiblioTech maintains total redundancy within network and hardware architecture to prevent any single point of failure, avoiding most service interruptions. BiblioTech leverages the competency and 24x7x365 support of RedBus InterHouse its hosting partner, allowing BiblioTech to focus on developing best-of-breed application services and support, while providing highly available connectivity out to the Internet. PSINet and AboveNet networks provide BiblioTech services with reliable Internet connectivity through highly redundant OC-192 and OC-48 backbones ensuring efficient delivery of service to customers. Additionally, these partners are interconnected to every major IXP, NAP or MAE (i.e Sprint, UUNet, MCI, AT&T, AOL) and have access to multiple power grids and diesel generators on site in the event of a power failure. BiblioTech’s network topology ensures that data is transmitted through the shortest paths by leveraging connectivity partners extensive combination of public and private peering relationships. As a result, BiblioTech can scale its bandwidth on demand to meet the needs of our customers, while minimizing response times for your customers.


4 NETWORK ARCHITECTURE BiblioTech’s network is powered by, state of the art equipment. Internet connectivity is supplied through a 2Mb/second circuit. Client side servers are connected directly into Alteon ACEDirector layer 4 switches that communicates directly with the 3640 router. Cisco Catalyst 3500 swtiches and Cisco xxxx routers provides the platform on which the private network is based. This high end mission critical equipment ensure that client data travels at the fastest possible speed across the internal network and between servers.

4.1.1 INTERNAL NETWORK INFRASTRUCTURE

BiblioTech’s internal network is separated into two distinct layers. The first layer is the publically accessible network, 212.57.32.0/24 (front network). This hosts the servers that are directly accessed by customers. The second layer is the private network, 10.10.0.0/24 and hosts servers that customers do not require direct access to. These two networks are connected by CISCO xxxx routers implementing an access list that allows only a small defined set of services through thereby providing security to the servers on the back network. This in particular to prevent unauthorised access to the database servers. As well as this access list the fact that the back network is a private address space network makes it non routable across the Internet.


4.1.2 ROBUSTNESS

The network is designed to be almost completely redundant. Having detected the failure of a network segment or individual hardware component the network sub system redirects the flow of traffic through unaffected systems with no noticeable degradation in quality of service. In order to achieve this almost all the network systems are mirrored.


4.1.3 LOAD BALANCING

Incoming connections to BiblioTech servers are load balanced by layer 4, Alteon ACEdirector switches, this enables virtually unbounded server capacity. The ACEdirector employs two powerful RISC processors on each of its eight 10/100 Mbps ports and can switch Web sessions at speeds--up to 200,000 sessions per second.

4.1.4 EXTERNAL INFRASTRUCTURE

BiblioTech’s upstream connectivity providers PSINet and AboveNet have networks that are engineered for maximum performance and scalability. These networks provide reliable Internet connectivity through highly redundant OC-192 backbones and high-bandwidth capacity for efficient delivery of our services across the Internet. Additionally, they are interconnected to every major INX, or NAP (e.g., Sprint, UUNet, MCI, AT&T, AOL), be on multiple power grids and have diesel generators on site in the event of a power failure. Furthermore, they utilize a Border Gate Protocol (BGP) that can provide BiblioTech with the fastest reaction to a failed connection. BGP enables BiblioTech to setup a truly redundant network. When a failure on a leased line is detected, BGP will automatically close that line and re-route traffic to another connection point.

4.1.5 NETWORK MONITORING

BiblioTech’s dedicated technical staff identifies and resolves potential problems before there is an impact on service. BiblioTech monitors network services on a 24/7 basis. BiblioTech operators also monitor system performance around the clock to respond to potential service-disrupting situations before they occur. Such events include usage spikes, spam attacks, storage limits, application failure and other occurrences. All operators have instant access to a clear chain of command that can authorize additional resources to respond to service-affecting situations. Monitoring includes:

1. Web Servers using login procedure 2. Mail Server: availability of all mail ports. (POP3, IMAP4, SMTP, management

ports) 3. Disk Space 4. Mail Send/Receive: receiving and sending mail from mail servers 5. Data Base connectivity 6. DNS availability 7. Load Balancing: load balance functionality 8. Network Functionality (firewall, packetloss and latency) In addition to BiblioTech’s internal process monitoring tools, we have deployed Big Brother as our external monitoring tool. Our 24/7 monitoring is employed within our network, and from two external locations around the world. Monitors continually perform roundtrip response tests on all the servers in our system, checking incoming and outgoing mail, user login times and application performance for all services. If any problem is detected, BiblioTech technicians are immediately dispatched according to predefined procedures based on the severity of the event. Lastly, BiblioTech has taken the extra precaution to proactively monitor the health of our servers using both HPOV and TNG UniCenter. These tools monitor our mission-critical processes and system resources based on thresholds we set to allow


our service technicians to quickly and accurately diagnose problems before they occur, on a 24/7 basis. As a result of implementing this three-tiered, high-availability monitoring service, BiblioTech can offer organizations a superior Service Level Guarantee as well as online statistics. It should be noted that BiblioTech would make every effort to prevent user spamming from our email system. Anti-spam measures are already designed into the service, including a limit on the number of addresses in the address fields (To: cc: bcc) and limits on the number of users in a single group in the address group. At any time you can request that BiblioTech block a particular users’ email accounts. Likewise, BiblioTech provides organizations with an online administrative tool that will enable you to edit, modify and delete specific user accounts at their discretion.


5 DATABASE ARCHITECTURE BiblioTech’s provides a separate Informix Dynamic Server 2000 database for each service portal. These databases run on dedicated Sun Microsystem Enterprise 450 servers. Each Sun Microsystem Enterprise 450 has 4 x 400MHz UltraSPARC-II CPUs, 2GB RAM, a D1000 storage sub system, and runs the Solaris 2.7 operating system. BiblioTech’s database engine is provided by Informix Dynamic Server.2000, IDS 9.2. Informix Dynamic Server delivers an industry-proven transaction engine for mission critical applications while providing an upgrade path to the Internet. Capable of supporting thousands of concurrent users, Informix Dynamic Server delivers maximum reliability, availability, and scalability to power BiblioTech systems. The D1000 storage sub system offers performance and flexibility able to meet the needs of BiblioTech’s continually growing storage. Designed for high data integrity and availability, the D1000 arrays provide full RAID functionality. Database systems are remotely housed at Red Bus Interhouse.

5.1.1 ROBUSTNESS

IDS used in conjuncture with Sun Microsystem Solaris and Sun Microsystems servers is a tried and tested database platform with excellent reliability. To extend fault tolerance data is mirrored to guard against media or hardware failure in the storage system. There is no resilience in the event of hardware or software failure within the E450 itself; this was a decision made for cost reasons when the E450s were purchased. The Sun Enterprise 450's extensive array of reliability, high-availability, and serviceability features allow customers to deploy these systems in their most business-critical environments with complete confidence. Ensures minimum latency and maximum system resource utilization for sustained high performance under the heaviest workloads, through high-speed parallel data flows.

5.1.2 SCALABILITY

The IDS engine is highly scalable and can handle databases holding over a terabyte of data and scaling upto 64 CPU per server. The Sun Enterprise 450 servers provide scalable computing performance for databases with up to one million users.


6 SOFTWARE ARCHITECTURE

BiblioTech services are structured around well established open source technologies. All the public facing internet services are built with standard open source applications. The web service is a hybrid between the classic CGI architecture and the increasingly popular application server approach. A standard Apache web server manages all incoming HTTP requests. If a request is determined to be for static content (ie graphics, icons, static HTML), it is handled by the apache server directly, however, if a request for dynamic content is encountered it is passed on to an embedded Perl application server. Each service comprises a base framework providing user & group management and access controls. Individual applications are then built on top of this flexi ble base to provide facilities such as email, discussion groups, web publishing. Each application can be considered to have four parts:

1. Modules: perl libraries and classes containing functionality used by many scripts within the application

2. CGI scripts: make use of the shared modules and templates to build dynamic responses to incoming requests. To enable the provision of a multi-lingual service, all embedded text is passed through a translation engine prior to being sent to the user.

3. Templates: standard html pages containing placeholders, conditional expressions, and loops into which generated content is substituted. In addition to the original English templates, translations are maintained for each language used on the site.

4. Icons and graphics: static content files which are served directly to the user with no intervention from the application server. Icons do not require translation since they contain no text, however, graphics have to be maintained in all the provided languages.

In contrast to traditional CGI environments, the scripts are executed directly in the embedded application server's memory space. This eliminates the overhead of process forking, and allows compiled scripts to be cached in memory, significantly increasing the response time for subsequent requests. Each web server process also maintains a single persistent database connection, which is shared by all the CGI scripts. Again, this provides a large performance boost over traditional CGI environments where each script has to connect to the database afresh. The application server, operating in the web server's memory space, runs the appropriate CGI script and sends the dynamically generated output back to the user. Compiled perl CGI scripts are cached in memory when first used, significantly decreasing the overheads for subsequent requests to the same URL. In addition to caching of individual scripts, Since the Perl interpreter is embedded in the web server process itself,

6.1 SOFTWARE BUILD ENVIRONMENT

The current (15th March 2001) breakdown of programming languages in all Schoolmaster products is as follows: Perl 331,000 lines


C 23,000 lines (Chat server) PHP3 7,000 lines (Shoppingmaster)

CVS is used as the standard version control system. The CVS repository stores all code, web pages, images, documentation and more besides. CVS coordinates updates to individual files if two or more developers or production members edit the same file, their edits are merged together correctly. CVS also allows developers to go "back in time" to earlier versions of the code and track changes. CVS allows parallel development on the same source tree in different branches. Almost all of the software is built using the GNU autoconf/automake tools. This allows tarballs to be built easily from which Red Hat packages (RPMs) can be derived very quickly. In addition, unit tests, regression tests and documentation manual pages can be added to each application quite simply using autoconf/automake. The Rolling Build system periodically checks out all of the code from CVS, builds it all, runs all the available unit tests and regression tests, and builds RPMs. These RPMs are served to the company intranet from an FTP server and are automatically installed overnight onto the developer workstations. This cycle of building, testing, packaging and installing ensures that bugs are found very quickly and that developers always have the latest version of all code and documentation available on their workstations. Software is delivered to QA in the form of a collection of immutable binary RPM packages. If rejected by QA, fresh RPMs are built containing bug fixes. When QA accept a package, the identical package is installed on the live system.

6.1.1 ROBUSTNESS

The applications make use of a number of techniques for ensuring robustness when in use. First, and foremost, the execution of each script is encapsulated in an atomic database transaction. The transaction is initiated when execution of the script begins, and upon succesful completion all the changes will be committed to the database. In the unlikely event of error occurring during execution, an exception handle will rollback all the changes. This ensures that the state of the dataset is always consistent. Since a large portion of the dataset is created by the users when interacting with the applications, it is essential to validate their input before storing it in the database. To simply this process there are a number of modules which can, when given descriptions of the data expected (through regular expression), automatically check script parameters, and redisplay forms with the helpful error messages next to incorrect data. These modules also take care to correctly escape embedded HTML codes thus avoiding potential security vunerabilities.

6.1.2 SCALABILITY

There are several aspects of the software architecture which contribute to the scalability of the services. The programmatically elegant, but inefficient object-relational mapping classes are being progressively replaced by libraries of advanced SQL queries. These are capable of extracting large quantities of targetted data with the minimum number of database queries. The aggressive caching of frequently accessed items, including precompiled CGI scripts, shared libraries and classes, database query handles, and persistent database connections, has a large positive impact on reducing the length of time required to serve a single request. Since any


single machine has a limit on the number of concurrent connections it can handle, this has a direct influence of the number of requests which can be served per second. Finally, the embedding of an application server directly in the web server has produced a modular system, whose capacity can be increased simply by plugging in additional web servers.

6.1.3 CVS

CVS is used to keep track of collections of files in a shared directory called "The Repository". Each collection of files can be given a "module" name, which is used to "checkout" that collection. After checkout, files can be modified, "committed" back into the Repository and compared against earlier revisions. Collections of files can be "tagged" with a symbolic name for later retrieval. BiblioTech’s CVS saves its version-control information in RCS files stored in a directory hierarchy, called the Repository, which is separate from the development area. Files in the Repository are stored in a format dictated by the RCS commands CVS uses to do much of its real work. RCS files are standard byte-stream files with an internal format described by keywords stored in the files themselves.


7 SERVER ARCHITECTURE BiblioTech servers are built in house to a standard specification, with the exception of database systems. Each server is housed in 2U 18" Rack mountable case with 250W power supply and 6 cooling fans. Servers are powered by Intel Pentiu m III 650 MHz processors with 256MB of ECC RAM. Storage is supplied by 15Gb IBM IDE Disks and connected to the network through a 3Com 10/100 Ethernet card.

7.1.1 SERVER INFRASTRCUTURE

BiblioTech server architecture for Postmaster and Schoolmaster service portals. Postmaster 4 x web servers - pmweb5, 6,7, &8 2 x Mail delivery servers - pmmail2 & pmmail3 1 Pop server - pmpop2 Schoolmaster 4 x web servers - smweb5, 6,7, &8 2 x Mail delivery servers - smmail2 & smmail3 1 Pop server - smpop2 1 Static web server - epages2 1 Admin server- smadmin General 4 x SMTP front facing mail relays - smtp5, 6,7 & 8 4 x DNS servers, 2 front facing ns5 & ns6, 2 rear facing ns7 & ns8 2 x Static web servers - web3 & web4 Big Brother system monitoring servers - mon2 and mon3 System messaging logging - syslog2 Statistics server for http access, error and mail logs - stats2 CVS version control server - cvs2 Tape Backup Server - Amanda2 Build server - kickstart

7.1.2 SERVER CONFIGURATION

RedHat Linux 6.0 is installed on every server to a standard configuration, defined by a Kickstart server. The kernel version is 2.2.16 Security All non-essential services and daemons are turned off to ensure maximum security and performance. The front facing servers are running a firewall based on ipchains-1.3.8 with only the ports essential to its specific purpose open. Remote server administration access is only conducted through SSH, which itself is confined to the internal BiblioTech network.


Web Servers All web servers are running Apache 1.3.12 lis tening on port 80 Mail Servers SMTP traffic is handled by Sendmail-8.9.3 listening on port 25 Database Mail Delivery Servers Bibliotech's pm-deliver and sm-deliver servers handle mail delivery to the Schoolmaster and Postmaster databases respectively Pop Pickup Servers Bibliotech's POP pickup servers listen on ports 109 and 110 Name Servers Domain resolution is handled by Bind-8.2.3 listens on port 53 Network time is served using xntp3-5.93-12 on port 37 Inter-server transportation of bulk data e.g.: log and error files and DNS or CVS changes, is handled by rsync-2.3.1 on port 873.

7.1.3 ROBUSTNESS

The high build and individual component quality of each server ensures an inherently robust system designed to be reliable and run continuously with minimal System Administrator input. A system wide monitoring service is provided by Big Brother 1.4 to page the duty System Administrator immediately in the event of a problem. Each server is monitored for it's internal hardware integrity e.g.; CPU load, disk use, connectivity and system daemons; as well as the mission critical services it provides. Two Big Brother servers monitor each other to ensure 3rd level backup A separate server running Amanda-2.4.1 handles nightly backups onto tape of all servers. REDUNDANCY Redundancy is built into the system by having multiple servers performing specific tasks. Internal redundancy also exists within each server due to deliberate over specification of hardware. This ensures that servers run on very low load, peak demands are easily absorbed and server overloads are extremely rare.

7.1.4 SCALABILITY

The DNS system load balances the web and mail server groups in a round robin fashion. We find this more than adequate for an equally loaded system.

7.1.5 LOAD BALANCING

Our server model is highly scalable with the existing system able to handle in excess of 1 Million users. Should loads increase beyond this we would be able to quickly replicate any server and include it in the round robin list.


8 STANDARD TRANSACTION FLOW


9 PRODUCTION

Production is BiblioTech’s description for its graphic design and front end coding department. Production is responsible for the look and feel of all the services and their associated web pages. The production department uses a variety of applications to build and maintain both dynamic and static web sites. The primary development applications are Dreamweaver, Photoshop, Freehand and Trados. All work is co-ordinated with the technical department through a Concurrent Versions System - CVS. CVS is a "Source Control" or "Revision Control" tool designed to keep track of source changes made by the production and technical departments while working on the same files, allowing the departments to stay in synchronisation with each other. The production department use development servers and staging servers, which mimic the live systems, to test and preview all work during the development cycle. Once production is satisfied with work, it is then passed on to the QA department and tested against the business requirements and specifications articulated in the business case. Any last minute problems identified through QA are addressed before going live. Application List Dreamweaver - to design layouts, create templates for the sites CVS - version control and group development sofftware Photoshop - graphic design Freehand - graphic design Trados - translation management software Staging servers - development platforms for static sites that mimic the live servers. Static web sites - flat marketing and brochure sites Development servers - development platforms for dynamic sites that mimic the live servers. Dynamic web sites - portals and applications


10 SERVICE LEVEL AGREEMENT

10.1.1 UPTIME GUARANTEE

BiblioTech’s services will be available 99% or more of the scheduled uptime. Calculation of this average availability will not include scheduled downtime for maintenance. Scheduled and unscheduled down time will each be limited to 25 minutes per day and the client will receive 7 days’ notice of scheduled downtime by email or phone. Any network outages caused by a BiblioTech’s service provider (other than a substantial failure of the Internet generally across a wide geographic area) shall constitute down time. BiblioTech maintains a reliable and scalable network that can support unlimited numbers of end-users. BiblioTech is constantly improving its infrastructure and performs continual audits, improvements and upgrades for hardware, software and bandwidth resources to provide the highest performance available. In those instances where network problems arise BiblioTech performs immediate investigation and resolution. BiblioTech will work with its partners to solve any problems that may occur for organizations as a result of network connectivity. If a system outage should occur, BiblioTech will make reasonable efforts to place a post-explanatory page on the partner email site(s).

10.1.2 ROUNDTRIP GUARANTEE

BiblioTech guarantees an average of less than 3 seconds round-trip time (login page + logout), measured inside the network. Incoming Mail will be received by the end-user Inbox in less than 30 seconds on average and for 90% of the messages in less than 5 seconds, not including time of maintenance. Outgoing email will leave BiblioTech Network in less than 30 seconds and for 90% of the messages in less than 5 seconds, not including time of maintenance. (These averages measure only BiblioTech server response time, and not Network WAN transmission.)

10.1.3 SECURITY

In an effort to address the physical security concerns of our clients, BiblioTech have placed the bulk of its server environment behind dual firewalls. To address the users concerns with regards to Internet transmission, BiblioTech has developed a secure connectivity solution that incorporates SSL encryption of Web-based services. It should be noted however, that this feature is disabled by default due to the well-known performance implications of running SSL.

10.1.4 DISASTER RECOVERY

As more businesses turn to outsourcing the management of their mission-critical applications, the need for maximum availability and consistent uptime has grown substantially. Companies who do business online are quickly finding that any downtime at all, no matter how short a period, can cause drastic effects on their revenues, reputations and market status. Relying on the Internet, in other words, means finding a solution that provides more than redundant network access. Organizations need a partner that also have a reliable process in place for proactive problem prevention. In response to these needs, BiblioTech has developed a proven and complete Disaster Recovery Plan.


When a service disruption occurs, BiblioTech assigns highest priority to the timely restoration of services. After service is restored, organizations will typically have access only to new communications until data backup from online or offline sources are complete. If data is being restored from offline archives, this may take as long as five hours. Organizations will not have access to archived information during this time. In order to assure the readiness of BiblioTech operators to complete the offline restoration process, the company runs frequent drills to test, troubleshoot and improve restoration performance. Below is an outline of Bib lioTech’s Disaster Recovery plan and Service Restoration Procedures: 1. Web Server Failure – Because of the redundant design of our topology, the

service will not be affected. 2. Mail Server Failure - In case of mail server failure, we can restore service with

a functional mail server within one hour. Only users of the specified mail server will experience a service interruption for this time period. Recovery of mail data will be completed in less than five hours.

3. Data Base Failure - Operational procedures are in place to restore service within twenty minutes.

4. MDS Failure - Due to the BiblioTech design and implementation, this type of failure will not affect users service.

5. Local Director - Due to hot standby machines, this type of failure will not affect user service.

6. Firewall Failure: Fully redundant with ability to reroute traffic on the fly if required backed by an on-site, dedicated security team on a 24/7 basis.

7. Service Disruptions : Service disruptions can be classified by degree of severity. We have unique operational plans for disruptions where a) service is unavailable, but no data is compromised, and b) where user data is compromised.

8. Service Unavailability: If service is temporarily unavailable due to server problems, all incoming messages are stored offsite until server operation is restored. Messages stored offsite during the disruption may be unavailable for a period of several minutes to several hours after service is restored, but no data is lost.

9. Loss of Data: In the unlikely event that both primary and secondary online disk storage becomes corrupted, data is recovered from offline, archived media.

10. Disruption Recovery: BiblioTech technicians and automated monitoring systems continually assess service availability and response time. Any service disruption will be detected and automatically reported to system operators within seconds. System operators who are able to diagnose system failure modes and implement recovery plans are available 24 hours per day, on-site and through immediate pager notification. BiblioTech operators have 24/7 access to all relevant server locations, in addition to remote access to servers and other resources, speeding response time and simplifying recovery efforts. The company maintains an extensive inventory of critical spare parts at key locations to insure rapid response.

11. Failover: By design, BiblioTech maintains failover capability with remote service centers that can be deployed in case of service disruptions at a specific location. Service maintenance procedures are continuously tested under BiblioTech failover scenario planning. The BiblioTech systems are designed to automatically implement failover, and to mirror and duplicate all services at remote facilities during the failover event. Organizations may not have access to


all their data during a failover event until service is restored at the primary location.

12. Backup: BiblioTech clearly recognizes the value of organisations data and has taken all the necessary precautions to insure a “high-availability” solution in place. Data storage is of paramount importance to the BiblioTech service. All data storage systems are fault-tolerant, with redundant components for key sub-systems, including power supplies and disk drives. In an effort to minimize network performance issues, backups are performed on a separate network so that the backup traffic has no impact on service traffic. The automated backup sequence is programmed and executed based on the following schedule:

13. Database – Backed up every 6 hours and restored to another machine with the

same HW configuration as the Main IDS2000 machine for redundancy. 14. Mail Server - A Full image of the mailbox volume is taken once a week, and an

incremental backup is performed daily. 15. System - 4 DLT drives working as a tape Raid for high backup performance and

fault tolerance is backed up once a week. A set of these tapes consisting of the system backup is then taken off-site and stored in a secure location.

10.1.5 SERVICE MONITORING

A robust monitoring system and escalation procedure is crucial to minimizing downtime and assuring that problems are rectified immediately. BiblioTech’s dedicated technical staff is highly qualified and identifies and resolves potential problems before there is an impact on the end-user. BiblioTech monitors services on a 24/7 basis. BiblioTech operators also monitor system performance around the clock to respond to potential service-disrupting situations before they occur. Such events include usage spikes, spam attacks, storage limits, application failure and other occurrences. All operators have instant access to a clear chain of command that can authorize additional resources to respond to service-affecting situations. Monitoring includes:

1. Web Servers using login procedure 2. Mail Server: availability of all mail ports. (POP3, IMAP4, SMTP, management

ports) 3. Disk Space 4. Mail Send/Receive: receiving and sending mail from mail servers 5. Data Base connectivity 6. DNS availability 7. Load Balancing: load balance functionality 8. Network Functionality (firewall, packetloss and latency) In addition to BiblioTech’s internal process monitoring tools, we have deployed Big Brother as our external monitoring tool. Our 24/7 monitoring is employed within our network, and from two external locations around the world. Monitors continually perform roundtrip response tests on all the servers in our system, checking incoming and outgoing mail, user login times and application performance for all services. If any problem is detected, BiblioTech technicians are immediately dispatched according to predefined procedures based on the severity of the event. As a result of implementing this high-availability monitoring service, BiblioTech can offer organizations a superior Service Level Guarantee as well as online statistics. It


should be noted that BiblioTech would make every effort to prevent user spamming from our email system. Anti-spam measures are already designed into the service, including a limit on the number of addres ses in the address fields (To: cc: bcc) and limits on the number of users in a single group in the address group. At any time you can request that BiblioTech block a particular users’ email accounts. Likewise, BiblioTech provides organizations with an online administrative tool that will enable you to edit, modify and delete specific user accounts at their discretion.

Documents

BiblioTech Online Services Technology