View
5
Download
0
Category
Preview:
Citation preview
MIS for Blended MOOCs onIITBombayX
Submitted in partial fulfillment of the requirements
of the degree of
Master of Technology
by
Rajeev Kumar Gautam
(Roll no. 13305R007)
Supervisor:
Prof. Deepak B. Phatak
Department of Computer Science and Engineering
Indian Institute of Technology Bombay
2016
Contents
List of Figures iv
Abstract 1
1 Introduction 1
1.1 Classroom Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.1.1 Conventional Classroom . . . . . . . . . . . . . . . . . . . . . . . . 1
1.1.2 Flipped Classroom . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.1.3 MOOCs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.1.4 Blended MOOCs . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.2 Need for a comprehensive information system . . . . . . . . . . . . . . . . 3
1.3 Problem statement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
2 Literature Survey 4
2.1 Open source LMS/CMSs . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
2.2 LMS interoperability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
2.3 Moodle . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
2.3.1 Integration with Moodle . . . . . . . . . . . . . . . . . . . . . . . . 5
2.3.1.1 Data Integration . . . . . . . . . . . . . . . . . . . . . . . 5
2.4 Open edX . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
2.4.1 Open edX LMS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
2.4.2 Open edX CMS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
2.5 IITBombayX platform . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
2.6 Current education system in India and how blended model will help . . . . 7
2.7 Blended MOOCs from IIT Bombay . . . . . . . . . . . . . . . . . . . . . . 7
2.8 Assessing Effectiveness of MIS . . . . . . . . . . . . . . . . . . . . . . . . . 7
i
Contents CONTENTS
2.9 System evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
3 Functional Requirements of Proposed System 9
3.1 Main objective . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
3.2 Consolidation of proposed functionality . . . . . . . . . . . . . . . . . . . . 9
3.2.1 Admin’s Role . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
3.2.2 Instructor’s Role . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
3.3 Integration with Moodle . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
3.4 Typical use cases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
3.4.1 Admin Role . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
3.4.2 Teacher Role . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
4 System Design 14
4.1 System view and Data Flow Diagram . . . . . . . . . . . . . . . . . . . . . 14
4.1.1 System view . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
4.1.2 Data Flow Diagram . . . . . . . . . . . . . . . . . . . . . . . . . . 15
4.2 Modules and interfaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
4.2.1 Admin Interfaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
4.2.1.1 Department . . . . . . . . . . . . . . . . . . . . . . . . . . 16
4.2.1.2 Course . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
4.2.1.3 Register Session . . . . . . . . . . . . . . . . . . . . . . . . 17
4.2.1.4 Course Assignment in Current Session . . . . . . . . . . . 17
4.2.2 Teacher Interfaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
4.2.2.1 View . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
4.2.2.2 Grade policy . . . . . . . . . . . . . . . . . . . . . . . . . 18
4.3 User Interfaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
4.3.1 Admin UI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
4.3.2 Teacher UI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
4.4 Design and implementation . . . . . . . . . . . . . . . . . . . . . . . . . . 20
4.4.1 Database Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
4.4.2 Database Schema . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
4.4.3 Technology Used . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
4.4.4 MOOCs Assessment and Composite Mark sheet . . . . . . . . . . . 25
4.4.4.1 MOOC grade policy . . . . . . . . . . . . . . . . . . . . . 25
ii
List of Figures CONTENTS
4.4.4.2 All MOOC assessment . . . . . . . . . . . . . . . . . . . . 25
4.4.4.3 Composite mark sheet with Moodle . . . . . . . . . . . . . 25
4.4.4.4 Composite mark sheet without Moodle . . . . . . . . . . . 26
4.5 Flow Chart . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
4.6 Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
5 System Testing 30
5.1 Unit Testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
5.1.1 Teacher Module (Local Evaluation) . . . . . . . . . . . . . . . . . . 30
5.1.1.1 Create Grade Policy . . . . . . . . . . . . . . . . . . . . . 31
5.1.1.2 Upload Local Marks . . . . . . . . . . . . . . . . . . . . . 33
5.1.1.3 Update Local Marks . . . . . . . . . . . . . . . . . . . . . 33
5.1.2 Teacher Module (MOOCs Evaluation) . . . . . . . . . . . . . . . . 34
5.1.2.1 Enter or update Blended MOOCs weight . . . . . . . . . . 34
5.1.2.2 Upload Blended MOOCs Grade Policy . . . . . . . . . . . 34
5.1.2.3 Upload Blended MOOCs student’s marks . . . . . . . . . 35
5.1.2.4 Distribute Blended MOOCs Weightage . . . . . . . . . . . 35
5.1.3 Admin Module . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
5.1.3.1 Local System Set up . . . . . . . . . . . . . . . . . . . . . 36
5.2 Load Testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
5.2.1 Load Testing Tool Used . . . . . . . . . . . . . . . . . . . . . . . . 38
5.3 Installation Testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
5.4 Logging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
6 Future Work and Conclusion 40
6.1 Login- Module . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
6.2 Teacher- Module and Interface . . . . . . . . . . . . . . . . . . . . . . . . . 40
6.3 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
iii
List of Figures
3.1 Admin Role . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
3.2 Teacher Role . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
4.1 System View . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
4.2 Level 0 Data Flow Diagram . . . . . . . . . . . . . . . . . . . . . . . . . . 15
4.3 Level 1 Data Flow Diagram (Part-1) . . . . . . . . . . . . . . . . . . . . . 16
4.4 Level 1 Data Flow Diagram (Part-2) . . . . . . . . . . . . . . . . . . . . . 17
4.5 Admin Dashboard . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
4.6 Teacher Dashboard . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
4.7 ER diagram part-1.1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
4.8 ER diagram part-1.2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
4.9 MOOC grading policy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
4.10 All MOOC assessment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
4.11 Composite Mark Sheet with Moodle . . . . . . . . . . . . . . . . . . . . . . 26
4.12 Composite Mark Sheet without Moodle . . . . . . . . . . . . . . . . . . . . 27
4.13 Generate Composite Mark Sheet Flow Chart . . . . . . . . . . . . . . . . . 28
5.1 Load Testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
iv
Abstract
Multiple Institutes are partnering with IIT Bombay to offer blended MOOCs. Students
will study the online course on IITBombayX, and will also study the same course normally
in their Institute. Final grade will be based on the composite performance of students,
in the online assessment, and in the tests/exams at the Institute. This system will help
teachers of engineering colleges in India for assessment of blended MOOCs. The teachers
from various institutes participating in the blended MOOCs, will be able to generate
a composite mark sheet for the particular course by giving certain weightage to online
assessment along with their regular university assessment. For example, a certain portion
of online assessment, say 20% to 30% can get incorporated in final university exam grading.
This system can also be integrated with Moodle for generation of the grade report.
v
Chapter 1
Introduction
In the following section we will see that how teaching method change with time and
technology. From traditional classroom learning to blended learning we have developed a
new teaching style to smoothing the blended learning system came into existence.
1.1 Classroom Models
Classroom model can refer to an educational approach, or physical setting in which teach-
ing and learning occur. In the following section we will see different types of classroom
learning that evolve with time and technology, we will explain some of them further.
1.1.1 Conventional Classroom
Conventional Classroom is concerned with the teacher being the controller of the learning
environment. This provides face to face group interaction, permitting discussions, spon-
taneous questions, and synchronized attention of all students. This environment allows
the instructor to precisely determine the aims, content, organization, pace and direction
of a presentation. The instructor can arouse interest in a subject by giving some real life
examples. Today in engineering education, a course is typically taught in a semester long
duration, ending with a final examination. Scheme of marking/grading is predefined. En-
courages one-way communication, therefore, the lecturer must make a conscious effort to
become aware of student problems and student understanding of content without verbal
feedback.
1
Chapter 1. Introduction 2
1.1.2 Flipped Classroom
Flipped classroom is an instructional strategy and a type of blended learning that re-
verses the traditional educational arrangement. The flipped classroom intentionally shifts
instruction to a learner-centered model in which class time explores topics in greater depth
and creates meaningful learning opportunities, while educational technologies such as on-
line videos are used to deliver content outside of the classroom. Problem solving happens
at in lecture hour inside class room. One of the most prominent issues is the necessity for
students to have access to a computer and Internet in order to view the lectures. This is
particularly hard on students from low-income districts who already have limited access
to resources.
1.1.3 MOOCs
MOOCs stand for massive open online courses which aim for large scale interactive learn-
ing that is equivalent to classroom learning providing all the resources related to the
classroom learning that includes videos,slides handouts and quizzes [1]. The MOOCs are
a good way to reach a huge number of students and help them in learning online through
a variety of interactive resources. It is aimed at open access and unlimited participation.
Some example of MOOCs are Coursera, edX, IITBombayX, etc.
1.1.4 Blended MOOCs
Blended MOOCs takes advantage of all the learning techniques mentioned above [2]. The
Blended MOOCs can be used in engineering level institutions in India, where the course
running in the institute will take some part of content and assessment of a parallel running
course on MOOCs. The teacher in an institute may ask student to view video lecture
on a specific topic in a syllabus and the institute will consider the marks of assessment
conducted by MOOCs, the teacher can also conduct the assessment for that particular
content cover on MOOCs.
2
Chapter 1. Introduction 3
1.2 Need for a comprehensive information system
Multiple Institutes are partnering with IIT Bombay to offer blended MOOCs. Students
will study the online course on IITBombayX, and will also study the same course normally
in their Institute. Final grade will be based on the composite performance of students, in
the online assessment, and in the tests/exams at the Institute.
Design and implement MIS to support teachers in participating institute:
• To maintain information of students, and synchronize it with IITBombayX.
• To define parameter and weightage of all assessment, and calculate final grades.
• To interface with Moodle, if used locally, to import and export relevant data.
1.3 Problem statement
Implementation of an automated system which keeps a record of students marks and
generate the grades of students and also generate the composite mark sheet of blended
MOOCs and of institute.
The goal of this thesis is to allow the participating institute faculty to access the
online assessment marks of their students, and to factor the portion /percentage of marks
in their regular university grades. Further, the system will allow the instructor to generate
composite mark sheet, based on uploaded local assessments and online MOOC assessment.
3
Chapter 2
Literature Survey
2.1 Open source LMS/CMSs
The improvement in information and communication technologies and in the use of inter-
net brought lots of opportunities in field of education. The conventional classroom has
taken many form with the help of these technologies like flipped classroom, MOOCs, etc.
To support these new methodology, Learning Management Systems (LMSs) came in to
existence. We will discuss few of open source LMS like Moodle, ATutor, etc.
These LMS provide creation of content in different ways like audio, video, pdf, csv,
etc. They also have multiple language support for the different group of people in world,
that facility extend the reach of LMS on the globe. These LMS support different kinds of
online exams for the evaluation of students, and also provide a space for student interaction
called chat tool in which they can interact one-to-one or by forming groups.
2.2 LMS interoperability
LMSs are built on extensible frameworks where it can modify according to suitability of
users [3]. LMS is the backbone of e-learning.
• First generation At this time LMS design platform concentrated on the delivery
and interoperability of content designed for a specific purpose, such as a particu-
lar course. At that time, a range of standards emerged one of the standards was
interoperability at content level.
4
Chapter 2. Literature Survey 5
• Second generation During this generation the architecture of LMS design became
modular, where it can share the content and also share with other tool or LMS.
• Next generation In coming generation, the service oriented LMS will be built.
Which will provide integration with other service tools, and various kinds of envi-
ronments to work with. The next generation will not be monolithic, it will provide
many solutions in one product.
2.3 Moodle
Moodle is a learning and course management system for online learning. The name Moodle
stands for Modular Object-Oriented Dynamic Learning Environment. Moodle is written
in PHP and runs on any OS like Unix, Windows, etc. It supports database like MySQL
and PostgreSQL, DB2, Microsoft SQL, etc. Moodle also supports Interoperability with
other web-based systems [4].
2.3.1 Integration with Moodle
Many institutes use Moodle internally to store academic records of students. Goal is to
integrate Moodle with our system, such that data import/export can be done through it.
For example, the student’s marks are stored in Moodle database, then system will import
marks of students from Moodle and generate a composite mark sheet of local assessment
and online MOOC assessment. In this way teachers won’t have to fill the marks again on
our system manually or create new CSV file for update. Moodle integration will provide
flexibility of data exchange.
2.3.1.1 Data Integration
Moodle and our system will share the common data format. Data will be given in form
of CSV file containing grade policy of course, marks of students, etc. Marks obtained by
students on MOOC can be downloaded by teacher from IITBpmbayX in form of CSV file.
Then, teacher will be able to upload these marks using our system to generate composite
mark sheet [5].
5
Chapter 2. Literature Survey 6
2.4 Open edX
According to definition in open edx documentation- ”The Open edX platform is a free–
and open source–course management system (CMS) that was originally developed by edX.
The Open edX platform is used all over the world to host Massive Open Online Courses
(MOOCs) as well as smaller classes and training modules” [6]
2.4.1 Open edX LMS
Open edx LMS is a place where learners can view course contents like video, quizzes,
slides, handouts, etc. On the other hand instructor can view the progress of students. In
this part students in interacts with the system and view the content, quizzes, etc. It also
has a discussion forum where students discuss on some topic.
2.4.2 Open edX CMS
”This is course authoring tool and studio authoring system. This is the system that an
instructor uses to build an online course. EdX studio is a CMS. A Django application
that uses MongoDB for content and MySQL/SQLite for other contents [6].”
2.5 IITBombayX platform
IITBombayX is built on the top of open edX to provide blended MOOCs, Where other
engineering colleges can participate in the blended MOOCs offered on the platform.The
feature that will content by well known professor online discussion groups, regular as-
sessment of learning as a learner progresses through a course, and online laboratories [7].
IITBombayX has been created for learners and institutions that that do not have quality
resource like teacher, environment, etc. The goal of of IITBombayX to provide quality
education, advance teaching and learning and expand access to education in engineering
institute in India [8]. IITBombayX will offer MOOCs through online medium.
6
Chapter 2. Literature Survey 7
2.6 Current education system in India and how blended
model will help
In recent years, the number of engineering colleges in India is increasing with very high
rate. But, the quality of education is missing due to some constraints like a fixed syl-
labus of course by the university. Due to a fixed syllabus, teacher is bound to teach
predefined topics and conduct predefined set of internal exams. Another reason may be
lack of academic exposure to the students due to not having a quality teacher or may
be technological hindrance. The quality of exams conducted by the university is also not
efficient, the majority of students mug up and write the exam and get marks without
understanding concepts.
2.7 Blended MOOCs from IIT Bombay
Providing the quality education in engineering colleges in India, IIT Bombay proposed a
blended MOOCs model where student will be part of their regular university course as well
as the blended MOOC. Practical sessions, exams, etc. will be conducted under supervision
of local teacher. Students will also complete the same course offered on IITBombayX [2].
The blended MOOCs also provides quizzes, programming assignments, etc. Teachers in
institutes will have freedom to include marks of these online quizzes, assignments, etc. in
their final grade sheet by giving some weightage [9].
2.8 Assessing Effectiveness of MIS
One of the main goals of the MIS is to develop information systems that will increase
efficiency to achieve this, MIS eases the process of any tedious task, and development
takes hierarchical approach to determine effectiveness of system.
• Level 0: This level deals with initial phase of development, for example complains
in database design, documentation quality, programming standards, etc. [10]
• Level 1: This level deals with resource allocation to project, for example time
difference, budget difference, etc.
7
Chapter 2. Literature Survey 8
• Level 2: This level deals with production capability of human resource assigns to
MIS development.
2.9 System evaluation
In blended model, teachers will have to combine marks of students for both class room
and MOOC. Currently they use Excel sheets to do this, which is not an efficient method.
To ease this process, an efficient system is needed which can generate composite mark
sheet by taking grades from both modes in some predefined format. This system should
run locally. In addition to that, integration with Moodle is also required. So, teachers
who already have local marks on Moodle, can directly enter them on new system. Our
goal is to build a system to cater above needs.
8
Chapter 3
Functional Requirements of
Proposed System
Functional requirement will describe a particular behaviour of the function of the system
when certain conditions are met. Functional requirements may be calculations, data
manipulation and processing and other specific functionality that defines what a system
is supposed to achieve.
3.1 Main objective
The main objective of this system to smoothing the process of maintaining the marks of
students and generate a composite mark sheet. This system can be used with blended
leaning or local storage of marks. Given the course marks of students, the system will
generate an automated grade sheet based on the course policy. Along with this, it has
the option of integrating Blended MOOCs marks based on instructor’s choice.
3.2 Consolidation of proposed functionality
In the system we have two kind of key role who will use this system. Responsibility of
these two role given below. one role will maintain the system and others will use it.
3.2.1 Admin’s Role
• Admin will be created at the time of system installation.
9
Chapter 3. Functional Requirements of Proposed System 10
• There will be only one Admin credential.
• It can add various department in system.
• Admin can deactivate/reactivate department.
• It can add courses in the department.
• It can also deactivate/reactivate courses.
• Admin will do mapping of various courses to various instructors at the beginning of
each semester.
• Admin can add/remove instructor.
3.2.2 Instructor’s Role
• Instructor Registration: The instructor need to register to use the system. The
registration link is provided on the system’s login page.
• Instructor can view all the courses registered against his/her name in current semester
as well as past semesters.
• Instructor will insert course grading policy of his/her registered course at the be-
ginning of current semester.
• System will import marks of students from LMS like-Moodle or it can browse marks
in the form of a CSV file from the local machine for specific locally conducted
assignments.
• Upload Blended MOOCs Grade Policy: The instructor will download blended MOOCs
grade policy in CSV format from the blended MOOCs website and will upload this
grade policy for the corresponding course.
• Upload Blended MOOCs Mark Sheet: The instructor will download blended MOOCs
mark sheet in CSV format from the blended MOOCs website and will upload this
grade policy for the corresponding course.
• Instructor has choice to incorporate Blended MOOCs weightage for final composite
mark sheet.
10
Chapter 3. Functional Requirements of Proposed System 11
• Upload Institute Mark Sheet: The Instructor can upload the maximum marks for
assignment and marks obtained by students in the CSV file.
• Update Institute Mark Sheet: In case of any discrepancy, instructor can update the
marks of students.
• Instructor can give priority to some assignment conducted on Blended MOOCs.
• Generate Final Mark Sheet: If the instructor chooses to include blended MOOCs
marks, then the final mark sheet will be a composite of local institute marks and
blended MOOCs marks otherwise final mark sheet will consist only of institute
marks.
3.3 Integration with Moodle
Till now we have seen that the system will store local and blended marks of students
in systems own database and finally produce composite mark sheet. It might pos-
sible that any institute store local marks of students in any Learning Management
System (LMS) like Moodle. For now I will consider that an institute store local
marks of students in Moodle. If this is the case then the system will able to store
blended MOOCs marks in the system’s local database and retrieve students local
marks from Moodle database and finally generate composite mark sheet.
For example, I will consider an institute use open source LMS like Moodle this sys-
tem will integrate with Moodle and will get the student marks from Moodle and gen-
erates grade sheet. I have installed and configure Moodle. Moodle database store, in
MySQL database. At the time of configuration I have created 4 courses, 4 teachers
and 15 students. The teacher is assigned some courses to teach in current semester.
I have examined database schema of Moodle where it stores marks of students.
Moodle store total marks of students in relation named mdl course completions.
Accessing Moodle marks of students from the system and generate a final mark
sheet.
11
Chapter 3. Functional Requirements of Proposed System 12
3.4 Typical use cases
In the system mainly we have two types of use cases teacher and admin where both of
them have its own access and responsibility. Each of role activities is shown in following
diagrams.
3.4.1 Admin Role
The admin will be responsible for maintaining the system within the institute after instal-
lation. Figure 3.1 shows the responsibility of the admin. For example, admin will register
the session and also assign semester course for teacher.
Figure 3.1: Admin Role
3.4.2 Teacher Role
The teacher will maintain local marks of students in the system and download MOOC
marks in the form of CSV from IITBombayX and generate a composite mark sheet of
12
Chapter 3. Functional Requirements of Proposed System 13
local marks and MOOC marks.
Figure 3.2: Teacher Role
13
Chapter 4
System Design
4.1 System view and Data Flow Diagram
In figure 4.1 we can see that system will stand alone runs in institute and will import/ex-
port data if necessary from Moodle. The user of this system may be part of of iitbombayX
where they can import MOOCs grading criteria along with marks of students, who have
registered for MOOCs course. Later teachers can use both local and MOOCs mark sheets
to generate composite mark sheet.
4.1.1 System view
System will be installed locally with in the institute and teacher will use it for recording
and generating marks of students as shown in figure 4.1.
14
Chapter 4. System Design 15
Figure 4.1: System View
4.1.2 Data Flow Diagram
Teacher has a facility to upload local assessment and generate compositr mark sheet
comprises of local marks as well as online MOOC marks.
Figure 4.2: Level 0 Data Flow Diagram
4.2 Modules and interfaces
We have designed module and interfaces that interact with the external environment.
Here we have designed interfaces for teacher and admin such that they can easily perform
the required operation.
15
Chapter 4. System Design 16
Figure 4.3: Level 1 Data Flow Diagram (Part-1)
4.2.1 Admin Interfaces
The admin interfaces deals with all the activity performed by admin. These interfaces
include setting up the local environment, creating department, course, etc.
4.2.1.1 Department
• Add Department: Add department in the system.
• Deactivate Department: Deactivate department in the system, if necessary.
• Reactivate Department: Reactivate the deactivated departments in the system, if
necessary.
• View Department: View current department in the system.
16
Chapter 4. System Design 17
Figure 4.4: Level 1 Data Flow Diagram (Part-2)
4.2.1.2 Course
• Add Courses: Add courses in department in the system.
• Deactivate Course: Deactivate course in the system, if necessary.
• Reactivate Course: Reactivate the deactivated course in the system, if necessary.
• View Course: View all the course or department wise in the system.
4.2.1.3 Register Session
• Register current session at beginning of each semester in the system.
• Can view all the registered session in the system.
4.2.1.4 Course Assignment in Current Session
• Do the mapping of course to Instructor at the beginning of each semester.
17
Chapter 4. System Design 18
4.2.2 Teacher Interfaces
The teacher interfaces deals with all the activity performed by teacher, for example,
view assign course, upload the marks of student, update the marks of students, generate
composite mark sheet.
4.2.2.1 View
• Can view all the assigned courses in current semester.
• Can view all the assigned courses.
4.2.2.2 Grade policy
• Create: Create grade policy for each assigned course at beginning of each semester.
• View: Can view the created grade policy.
4.3 User Interfaces
In the we have mainly two types of dashboard one is for teacher other for admin. Through
these interfaces teacher and admin perform some kind of operation that ease the process
of setting up a system in an institute and store marks of students in the system and finally
generate a composite mark sheet for some course.
4.3.1 Admin UI
After login the admin will see the dashboard where admin can perform some kind of
operation like manage department, course, etc.,
4.3.2 Teacher UI
The teacher in his/her dashboard will be able to see assigned courses for the current
semester and also all the previous courses, if any. As shown in the Figure 4.5, for mark
sheet generation module, teacher will be able to see create and view grade policy, give
weightage to blended MOOC, etc.
18
Chapter 4. System Design 19
Figure 4.5: Admin Dashboard
Figure 4.6: Teacher Dashboard
19
Chapter 4. System Design 20
4.4 Design and implementation
While developing the system I have taken help from the Django documentation[11] with
MySQL as back end database and develop front page using technology HTML, CSS, etc.
4.4.1 Database Design
We use Django authentication tables to authenticate user. We are storing grade policy
for the current semester in grade policy table. To store blended MOOCs grade policy, we
have created a separate table, that table is used to populate all the evaluation of blended
MOOCs. We are also storing blended MOOCs weightage to be included. To store blended
MOOCs marks of students, we have created a different table that will be used to generate
final mark sheet.
Figure 4.7: ER diagram part-1.1
20
Chapter 4. System Design 21
Figure 4.8: ER diagram part-1.2
4.4.2 Database Schema
We have created number of relation that is used to store data.
• All deprtment (CharField department name, CharField department abb, Boolean-
Field is active): This relation is used to store all the registered department in the
system. The department name column is primary key and also have unique values
in the department abbreviation column.
• All course (CharField course code, CharField course name, ForeignKey dp-
name, BooleanField is active): This relation is used to store all the registered
course, in the corresponding department in the system. The course code column is
primary key and also have unique values in the course name column. The dpname
column represents course associate to the corresponding department.
• Academic session (CharField year, IntegerField semester, BooleanField is active):
21
Chapter 4. System Design 22
This relation is used to store current academic semester. In current running semester
is active value will be true for others will be false.
• Moodle (BooleanField is active): Boolean field in this relation indicates that
Moodle is used to store local marks or not.
• Course mapping (CharField mdl cid, ForeignKey cid , BooleanField is active):
This relation is used to store mappings of Moodle course id and course id in the
system. The mdl cid column represents course id from Moodle database and did
represent course id from the system database. The mdl cid and side column is a
unique pair in the database.
• Userlogin (OneToOneField user, IntegerField usertypeid, BooleanField sta-
tus, DateTimeField last login, IntegerField nooflogins): This relation is used to
store details of the user at login time. The user column field defines which user, user-
typeid define which type of user, status define user is active or not, DateTimeField
define at what time, nooflogins define how many times user login.
• Personinformation (EmailField email, IntegerField titleid, CharField first-
name, CharField lastname, IntegerField designation, CharField gender, For-
eignKey departmentid, CharField experience, CharField telephone, DateField
createdondate, BooleanField is active): This relation is used to store personal in-
formation of the user, at the time of user registration. details of the user at login
time. User fields defines which user, usertypeid define which type of user, status
define user is active or not, DateTimeField define at what time, nooflogins define
how many times user login. The email id column is the primary key for this relation.
• Courseallotment (ForeignKey personid, ForeignKey courseid, ForeignKey sessionid ,
BooleanField is active): This relation is used to store mappings of teacher to some
course in a current academic semester. Pair of teacher id and course id will be unique.
Once the semester will over mapping will be deactivated.
• Evaluation table (CharField eval name, CharField eval no, FloatField max marks,
ForeignKey courseallotid, BooleanField is active): This relation is used to store
details of local evaluations. The eval name column store name of evaluation for ex-
ample: quiz, mid sem, etc. The eval no column store number if multiple evaluations
22
Chapter 4. System Design 23
occur of the same type. The max marks field store maximum marks for particular
evaluation. The courseallotid store object of Courseallotment relation.
• Evaluation marks (CharField roll no, FloatField marks, ForeignKey evalid,
BooleanField is active): This relation is used to store local marks of students in
a particular course. The roll no column store roll number of students. The marks
column store marks obtain by students in some evaluation. The evalid store object
of Evaluation table.
• Grade policy (CharField gp id, CharField abbreviation, CharField assign-
ment, IntegerField total, IntegerField mandatory): This relation is used to
store details of local grade policy for some course. The gp id column store store
unique grade policy id, which is generated by a combination of the course code,
current academic session and semester number. The abbreviation column is used to
store abbreviation for evaluation. assignment column is used to store the name of
evaluation. The total column is used to store the total number of evaluations. The
mandatory column is used to store mandatory evaluation out of the total evaluation.
• CourseGrade (ForeignKey gp id, CharField course code, ForeignKey per-
sonid, ForeignKey sessionid): This relation is used to store grade policy created
by teacher for some course.
• Bm max marks (CharField course name, CharField all eval, ForeignKey
personid, ForeignKey sessionid, IntegerField max marks): This relation is
used to store details of blended MOOCs evaluation conducted on IITBombayX.
max marks column store maximum marks for that particular evaluation. all eval
store name of evaluation.
• Bm student marks (CharField roll no, CharField course name, ForeignKey
personid, ForeignKey sessionid, TextField marks): This relation is used to store
MOOCs marks of students for some course.
• Bm weight ( CharField course name, ForeignKey personid, ForeignKey
sessionid, IntegerField bmoocs weight ): This relation is used to store MOOCs
weightage some course. MOOCs weightage can be 0 to 100%.
23
Chapter 4. System Design 24
• Bm grade policy ( CharField course name, ForeignKey personid, ForeignKey
sessionid, IntegerField min count, FloatField weight, CharField examtype,
IntegerField drop count, CharField short label): This relation is used to store
MOOC’s grade policy. The min count column defines the minimum number of eval-
uations of the same type. The eighth column defines the total weight for evaluations.
The drop count is used to to store drop able evaluations. The short label for exam
type.
• Bm policy weight (CharField course name, ForeignKey personid, ForeignKey
sessionid, TextField pweight): This relation is used to store MOOCs weight distri-
bution among different evaluations. The pweight column store a vector of distributed
value.
Other then above relational schema I have also used some of the relation from Moodle
database. For example mdl user enrolments, mdl enrol, mdl course, mdl course completions,
etc.
4.4.3 Technology Used
• HTML: It is the main mark-up language for displaying web pages and other infor-
mation that can be displayed in a web browser.
• CSS: Cascading Style Sheets (CSS) is a style sheet language used for describing the
presentation of a document written in a markup language.
• JavaScript: It is a prototype-based scripting language that is dynamic, weakly typed
and has first-class functions and is mainly used for client side validation etc.
• AJAX: AJAX is the art of exchanging data with a server, and updating parts of a
web page - without reloading the whole page.
• MySQL: It is the world’s most used open source relational database management
that runs as a server providing multi-user access to a number of databases.
• Django/Python: Django is a free and open source web application framework, writ-
ten in Python, which follows the model-view-controller (MVC) architectural pattern.
24
Chapter 4. System Design 25
• Bootstrap: Bootstrap is the most popular HTML, CSS and JavaScript framework
for developing responsive, mobile-first web sites.
• The Operating System used primarily in this project is Ubuntu 14.04.
4.4.4 MOOCs Assessment and Composite Mark sheet
4.4.4.1 MOOC grade policy
Figuere 4.8 shows the grading policy for a MOOC. It is a CSV file in which the first column
is the number of evaluations of an assignment, and the second column is weightage of an
assignment, and the third column is name of an exam, and the forth column contains the
number of droppable evaluations of an assignment, and fifth column contains short label
for a respective assignment.
Figure 4.9: MOOC grading policy
4.4.4.2 All MOOC assessment
Figuere 4.9 shows the MOOC marks of all the students, who are registered for MOOC
course with IITBombayX and regular university course. The first line in the Figure shows
the maximum marks for the evaluation, second line contains the list of MOOC evaluations,
and from third line onwards contain marks obtained by each student in corresponding
evaluation.
4.4.4.3 Composite mark sheet with Moodle
Figuere 4.12 shows the composite mark sheet of local marks and MOOC marks of all the
students, who are registered for blended MOOC course with IITBombayX along with their
regular university course. Local marks of students are stored in Moodle database. Figure
25
Chapter 4. System Design 26
Figure 4.10: All MOOC assessment
shows the blended MOOC weight in this case it is 25(shown in figure). It also shows
weight distribution and maximum marks for each of the blended MOOC evaluation.
Figure 4.11: Composite Mark Sheet with Moodle
4.4.4.4 Composite mark sheet without Moodle
Figuere 4.13 shows the composite mark sheet of local marks and MOOC marks of all
the students, who are registered for blended MOOC course with IITBombayX along with
26
Chapter 4. System Design 27
their regular university course. Local marks of students are stored in system database.
Figure 4.12: Composite Mark Sheet without Moodle
4.5 Flow Chart
Teacher will login in system and upload grade policy and blended MOOC assessments in
system. Teacher will select weightage of MOOC and also select MOOC evaluations to be
included in composite mark sheet.
27
Chapter 4. System Design 28
Figure 4.13: Generate Composite Mark Sheet Flow Chart28
Chapter 4. System Design 29
4.6 Algorithm
I have designed a following algorithm to a generate composite mark sheet of MOOC
evaluations and local evaluations. I assume that teacher has downloaded MOOCs grading
policy and mark sheet of students for the corresponding course.
Step 1: Start
Step 2: Teacher login into the system.
Step 3: Teacher selects weightage of MOOC
Step 4: If weightage > 0
Upload MOOC grade policy
Generate template for some course
Fill template with MOOCs marks of students
Upload filled template with marks of students
Select MOOC evaluations to be included in final mark sheet
Step 5: If Moodle used for local storage
Read MOOC assessments from system
Read local assessments from Moodle database
Output a composite mark sheet in a form of CSV file
Step 6: Else
Read MOOC assessments from system
Read local assessments from system database
Output a composite mark sheet in a form of CSV file
Step 7: Stop
29
Chapter 5
System Testing
System testing of software testing is conducted on a complete, integrated system to evalu-
ate the system’s acceptance with its defined requirements. System testing falls within the
scope of black-box testing. Unit testing is also performed to evaluate each independent
module.
5.1 Unit Testing
A unit is the smallest testable part of the software. A unit could be an entire module, but
it is more often an independent function or procedure. Unit Testing is a level of software
testing where individual units/ components of a software are tested. The goal is to ratify
that each unit of the software performs as designed. It normally has one or a few inputs
and usually a single output. In procedural programming a unit may be an individual
program, function, procedure, etc. Unit Testing is performed by using the White Box
Testing method.
5.1.1 Teacher Module (Local Evaluation)
The teacher has to perform some kind of operation to store local marks of students. To
effectively run whole process, we need to test that function or module producing valid
output on valid input data. So, in following section we have created some test cases that
validate function or module’s effectiveness.
30
Chapter 5. System Testing 31
5.1.1.1 Create Grade Policy
This module creates grade policy for a course in current semester that later use to compute
total marks on the basis of evaluation’s weightage criteria defined in grade policy and
finally generate composite mark sheet. The following value in the table shows valid and
invalid values for creating grading policy.
31
Chapter 5. System Testing 32
Step Test Steps Test Data Expected Result Status
1Enter value of abbr
columnEX, QZ Successfully created Pass
2Enter value of abbr
column2, E1
*ERROR-Abbreviation
column should contain
ONLY uppercase alphabets
of Maximum length 4
Fail
3Enter value of assign-
ment column
Mid Sem, End
SemSuccessfully created Pass
4Enter value of assign-
ment columnMid-Sem, End1 Do not use hyphen (-) Fail
5Enter value of No. of
assignment column1 to 99 Successfully created Pass
6Enter value of No. of
assignment column-1, 0, 101, 500
*ERROR- No.of Assign-
ments field value should be
greater then ZERO and less
then 100.
Fail
7Enter value of No. of
Best Score column
<= No. of
assignment col-
umn
Successfully created Pass
8Enter value of No. of
Best Score column
>No. of assign-
ment column
*ERROR- No.of Assign-
ments field value can not be
less then No. of Best Score
field value.
Fail
9Enter value of No. of
assignment column-1, 0, 101, 500, kl
*ERROR- No. of Best
Score column value should
be only positive INTEGER
greater then zero.
Fail
10Enter value of Weight
column
Weight column
Sum = 100Successfully created Pass
11Enter value of Weight
column
Weight column
Sum != 100
*ERROR-: Weight column
SUM should be exactly 100.Fail
32
Chapter 5. System Testing 33
5.1.1.2 Upload Local Marks
To upload marks of students we need to verify that maximum marks for any evaluation
should be greater/equal than the marks obtain by the students. Maximum marks and
marks obtain should be greater than zero.
Step Test Steps Test Data Expected Result Status
1 Upload File Correct fileYou have successfully up-
loaded marksPass
2 Upload File Not .CSV file*ERROR-: Uploaded file is
not .CSV file.Fail
3 Upload FileFile already Up-
loaded
*ERROR-: Marks for this
evaluation is already Up-
loaded !!!
Fail
4 Upload File
Max marks <
marks obtain <
0
*ERROR-: Uploaded file
contain maximum marks
less the obtain marks or ob-
tain marks less then zero at
given line no
Fail
5.1.1.3 Update Local Marks
In case of any discrepancy in the marks of students due to any human, teacher can update
marks of students we need to verify that maximum marks for any evaluation should be
greater/equal than the marks obtain by the students. Maximum marks and marks obtain
should be greater than zero.
Step Test Steps Test Data Expected Result Status
1 Update student marks 10,5You have successfully up-
date the marksPass
2 Update student marks -5, > Max marks
*ERROR-: Update marks
is greater then Max marks
or less then zero for evalua-
tion.
Fail
33
Chapter 5. System Testing 34
5.1.2 Teacher Module (MOOCs Evaluation)
The teacher has to perform some kind of operation to store MOOCs marks of students.
To effectively run whole process, we need to test that function or module producing valid
output on valid input data. So, in following section we have created some test cases that
validate function or module’s effectiveness.
5.1.2.1 Enter or update Blended MOOCs weight
To test the weightage of MOOCs to be included to generate final composite mark sheet,
weightage of MOOCs should be entered from 0 to 100 both included. Other values should
not be entered.
Step Test Steps Test Data Expected Result Status
1
Enter or update
Blended MOOCs
marks
0 to 100
You have successfully se-
lected or update weight of
Blended MOOC
Pass
2
Enter or update
Blended MOOCs
marks
other values
*ERROR-: Number range
should be between 0 and
100 both inclusive.
Fail
5.1.2.2 Upload Blended MOOCs Grade Policy
Blended MOOCs grade policy should be a .CSV that has been downloaded from bmw.iitbombayX.
This file has some specific format previously shown in figure 4.10.
Step Test Steps Test Data Expected Result Status
1Upload Blended
MOOCs grade policyValid .CSV file
You have successfully sub-
mitted blended MOOCs
grading policy for given
course.
Pass
2Upload Blended
MOOCs grade policyNot .CSV file
*ERROR-: Uploaded file is
not .CSV file.Fail
34
Chapter 5. System Testing 35
5.1.2.3 Upload Blended MOOCs student’s marks
To upload Blended MOOCs marks of students, teacher have to fill the marks downloaded
marks from bmw.iitbombayX in a template, which can be generated from the system.
The upload file should be.CSV format.
Step Test Steps Test Data Expected Result Status
1Upload Blended
MOOCs mark sheetValid .CSV file
You have successfully sub-
mitted blended MOOCs
student’s marks for given
course.
Pass
2Upload Blended
MOOCs mark sheetNot .CSV file
*ERROR-: Uploaded file is
not .CSV file.Fail
3Upload Blended
MOOCs mark sheetf4,th
*ERROR-: Students marks
can not be other then Num-
ber and NA for given Roll
number.
Fail
3Upload Blended
MOOCs mark sheet<0
*ERROR-: Students marks
can not be less then zero for
given Roll number.
Fail
4Upload Blended
MOOCs mark sheet< Max marks
*ERROR-: Students marks
can not be more then max
marks for given Roll num-
ber.
Fail
5.1.2.4 Distribute Blended MOOCs Weightage
To distribute Blended MOOCs weightage in preferred evaluations. The evaluation sum
equals to selected weightage. Negative and empty values are not allowed.
35
Chapter 5. System Testing 36
Step Test Steps Test Data Expected Result Status
1Distribute Blended
MOOCs Weightage
Distribute
weight sum =
submit weight
You have successfully
submitted weight of your
blended MOOCs grading
policy.
Pass
2Distribute Blended
MOOCs WeightageEmpty value
**ERROR-: Enter valid
number greater then zero
!!!!!
Fail
3Distribute Blended
MOOCs Weightage
Distribute
weight sum !=
submit weight
*ERROR-: Your dis-
tributed weight sum should
be equal to given some. It
is now some value
Fail
5.1.3 Admin Module
At the time of system installation admin has to create department, course, register aca-
demic session, etc. To smoothly run whole process, we need to test that function or mod-
ule producing valid output or enter into a database on valid input data. So, in following
section we have created some test cases that validate function or module’s effectiveness.
5.1.3.1 Local System Set up
To set up a local system in an institution admin has to create some department as well
as a course for corresponding department. admin also maps the teacher to some course
in the current semester.
• Create Department: Department name should start from alphabet other values
will be invalid values. Department abbreviation should also follow above criteria.
36
Chapter 5. System Testing 37
Step Test Steps Test Data Expected Result Status
1Create Department
Name
Computer
Science and
Engineering,
Mechanical
Engineering
You have successfully added
some department.Pass
2Create Department
Name2cse
*ERROR-: Department
name should start from
character only
Fail
3Create Department
AbbreviationCSE, ME
You have successfully added
some department.Pass
4Create Department
Abbreviation2cse, −ME
*ERROR-: Department ab-
breviation should start from
character only
Fail
• Create Course: Course name should start from alphabet other values will be
invalid values. Course code should also follow above criteria. Duplicate course can
not be registered.
Step Test Steps Test Data Expected Result Status
1 Create Course Name
Data Structure,
Thermodynam-
ics
You have successfully added
some course.Pass
2 Create Course Name −data structure
*ERROR-: Course name
should start from character
only
Fail
3 Create Course Code CS101, ME209You have successfully added
some course code.Pass
4 Create Course Name already register*ERROR-: Course name al-
ready registerFail
5 Create Course Code already register*ERROR-: Course code al-
ready registerFail
37
Chapter 5. System Testing 38
• Semester Course Allocation: Course allocation for teacher in current semester
will be unique. Duplicate entry can not be registered.
Step Test Steps Test Data Expected Result Status
1 Allocate Courseunique course,
teacher pair
You have successfully allo-
cate course to the teacher.Pass
2 Allocate Courseunique course,
teacher pair
*ERROR- Pair of selected
course and teacher for this
semester is already allo-
cated.
Fail
5.2 Load Testing
”Load testing is the process of putting demand on a software system or computing device
and measuring its response. Load testing is performed to determine a system’s behaviour
under both normal and anticipated peak load conditions.”
I have done load testing of the system using an open source load testing tool Locust. I
have applied load on the system from 0 to 1500 users, the response time shown in the
graph. I have tested all the interfaces of a system that interacts concurrently with the
database. As shown in the graph, as number of user increases response time also increases.
5.2.1 Load Testing Tool Used
Locust [12]:- An open source load testing tool
Locust is an easy-to-use, distributed, user load testing tool. Intended for load testing
web systems and figuring out how many concurrent users a system can handle. Locust
is completely event based, and therefore it’s possible to support thousands of concurrent
users on a single machine.
5.3 Installation Testing
Installation testing is examines that the software system is successfully installed and it
is working as anticipated after installation. I have done installation testing on a testing
38
Chapter 5. System Testing 39
0
100
200
300
400
500
600
700
800
0 200 400 600 800 1000 1200 1400
Response T
ime(m
s)
Number of Users
MedianAverage
Min
Figure 5.1: Load Testing
machine where I have installed all the required software package, database, dump file into
a database, etc. To smoothly install the system I have created Makefile.
5.4 Logging
Logging module is also created where all the error and exception log generated by the
system will be recorded. This will help to debug the problem if there any problem occurs
in the system. To implement logging facility I have used Django’s logger module.
39
Chapter 6
Future Work and Conclusion
6.1 Login- Module
• Send confirmation link when instructor register first time.
• A module for forgot password will be integrated with a valid register email id.
6.2 Teacher- Module and Interface
• Re-use Grade Policy if Instructor wants to continue with any one of previously
generated grade policy, then the instructor can forward one of defined grade policy.
• Grade Generation Once the final mark sheet is generated, instructor will enter a
range of marks for specific grades and will get a final grade sheet.
6.3 Conclusion
The blended MOOCs requires the composition of institute marks and MOOCs marks.
So, to make this process easy and efficient for the blended MOOCs partner teachers. We
have built a system which stores marks of institute evaluations and takes MOOCs marks
in the form of CSV file and generate a composite mark sheet. The teachers, who are not
the part of blended MOOCs can also use this system to keep records of students marks.
The system will install and runs locally in engineering colleges in India.
40
Chapter 6. Future Work and Conclusion 41
Blended MOOCs require the composite nature of in-classroom learning and online
participation. To this effect, both evaluation i.e. classroom as well as online evaluation,
are necessary for the complete assessment of students. Regular assessments in different
forms (online quizes, in-classs exams) make the teacher aware of the current subject
understanding of students in a class.
41
References
[1] Massive open online course, . URL https://en.wikipedia.org/wiki/Massive_
open_online_course. Retrieved on 2015-9-20.
[2] Deepak B Phatak. Adopting moocs for quality engineering education in india. In
Proceedings of the International Conference on Transformations in Engineering Ed-
ucation, pages 11–23. Springer, 2015.
[3] Declan Dagger, Alexander O’Connor, Seamus Lawless, Eddie Walsh, and Vincent P
Wade. Service-oriented e-learning platforms: From monolithic systems to flexible
services. Internet Computing, IEEE, 11(3):28–35, 2007.
[4] Moodle, . URL https://en.wikipedia.org/wiki/Moodle. Retrieved on 2015-9-15.
[5] Ricardo Queiros, Lino Oliveira, Jose Paulo Leal, and Fernando Moreira. Integration
of eportfolios in learning management systems. In Computational Science and Its
Applications-ICCSA 2011, pages 500–510. Springer, 2011.
[6] About open edx, . URL https://open.edx.org/about-open-edx. Retrieved on
2015-10-8.
[7] Definition iitbombatx. URL https://www.iitbombayx.in/. Retrieved on 2015-10-7.
[8] About iitbombatx, . URL https://www.iitbombayx.in/about. Retrieved on 2015-
10-7.
[9] Consultation conclave on blended moocs. URL http://www.it.iitb.ac.in/frg/
wiki/images/9/9a/Minutes_Consultation_Conclave_on_Blended_MOOCs.pdf.
Retrieved on 2015-9-23.
42
Bibliography 43
[10] Scott Hamilton and Norman L Chervany. Evaluating information system
effectiveness-part i: Comparing evaluation approaches. MIS quarterly, pages 55–69,
1981.
[11] Django documentation. URL http://media.readthedocs.org/pdf/django/1.8.
x/django.pdf. Retrieved on 2015-8-30.
[12] Locust documentation. URL http://locust.io/. Retrieved on 2016-5-20.
43
Recommended