Mobile Computing and Mobile Computing and
Technology Assessment: Technology Assessment:
A Case Study with A Case Study with
ResultsResultsStephen G. Landry, Ph.D. Chief Information Officer
Heather Stewart, Ph.D. Assistant Vice President, Finance and Technology
Copyright Stephen G. Landry and Heather Stewart, 2003
Seton Hall UniversityWho Are We?
• We Are A Mid Sized, Private, Catholic Affiliated University in Central New Jersey
• Carnegie Classification: Research / Doctoral II• Main Campus in South Orange, NJ
– 15 miles from New York City• 10,000 Students
– 4,400 Full Time Undergraduates (50% Residential)
• 350 FT Faculty (450 FTE Faculty)• FY’01 Annual Operating Budget Approx. $150 million
– FY’00 G&E for South Orange Campus Approx. $115 million
Pace Of Change at Seton Hall University
• Before 1995:
– Inadequate Computer Labs
– Inadequate Local Networks/E-mail
– Lack of Integration of Technology in Teaching
– Disorganized Support/Allocation of Resources
Pace Of Change (Cont.)
Now:– Seton Hall University is doing IT “right”
• 2000 EDUCAUSE Award (Honorable Mention) for Systemic Progress in T&L with Technology
• 1999 EDUCAUSE Award for Campus Networking Excellence
• In Top 50 of Yahoo! Internet Life Survey of “Most Wired” Universities
• Growing National Reputation of our Mobile Computing Initiative and other Teaching, Learning, and Technology Initiatives
• Alliance with IBM Corporation
• Site Visits by National/International Universities
• Presentations at National Conferences/Workshop
A Vehicle for Change:University Strategic Planning
• Seton Hall University has accomplished heightened recognition among top-tier Catholic Universities nationally
• The strategic goals include a critical focus on the intellectual, personal and spiritual development of all students
• Students experience a rigorous, value-centered and technologically enhanced environment
Advancing Strategic Goals with Technology
• To advance the strategic goals, a strong focus was placed on communication, teaching and learning as well as support services for students using technology
• It was recommended that a strategic agenda include an Information Technology Strategic Planning process to build an IT Long-Range Plan
Seton Hall UniversityHow Did This Happen?
• Began With Development of IT Plan in 1995– Part of the University’s Strategic Planning Process– Cross Functional Team
• Included Faculty and Administrators• Given Half-Time Release to Work on IT Plan• Intensive Immersion in IT Issues, Trends, Directions
– Executive Sponsorship/Commitment to Implement• Chief Academic Officer/Chief Financial Officer
– Involvement of User Community• Focus Groups/Surveys• Involvement of Faculty and Administrator Constituency Groups
(e.g., Academic Computing Advisory Committee, Administrative Computing User Groups, etc.)
– Technology Planning Consultant from IBM Education
Seton Hall University’sStrategic Technology Plan
• Our Technology Strategy:
– Seton Hall University will develop and implement a• learner-centered, • network-centered, • distributed (mobile) learning environment
– The core of our distributed learning environment will be a robust set of digital information resources (e.g., a “digital library”) in support of teaching, learning, scholarship, and institutional transformation
SHU’s Strategic IT Plan
• Implications of the IT Plan:Focus on student experience– Students Want / Need:
• Ubiquitous Access to Technology• University-wide Hardware and Software Standards• University-wide Network• Central IT Support Services• Greater Curricular Integration of Technology • More Efficient Business Processes / Greater Access to
Institutional Information and Services / Bringing Together Academic and Administrative Computing
Implementing the Strategy
• In order to implement the strategy, the University undertook a number of teaching, learning, and technology initiatives:– Mobile (Ubiquitous) Computing Program– SetonWorldWide (Online Professional Programs)– Web-enabling Enrollment Services– Reorganization of Computing Services
• Created New User Services Organization, Including a Teaching, Learning, and Technology Center
– Major Technology Infrastructure Upgrade• Gigabit Ethernet Campus Backbone; Server Consolidation;
Campus-wide Email System; 802.11b Wireless; Internet Upgrade; etc.
Mobile Computing atSeton Hall University
• Seton Hall University’s Mobile Computing Program is an innovative academic program involving three components:
– Access: The University licenses the use of a laptop computer to all undergraduates as part of their tuition and fees
– Curricular Integration: The University provides support and incentives to faculty to use technology in innovative ways to enhance teaching and learning
– Network and Support Services: The University provides the infrastructure and support services that enable the effective use of technology in teaching and learning
Mobile Computing (cont.)• Current Model: IBM ThinkPad R-series computer
– 1.2 MHz PIII; 14” TFT Screen; 256 Mb RAM; 30 Gb HDD; CD-RW; Built-in 802.11b wireless networking; 10/100 Ethernet; 56Kb Modem; LiON Battery
– Computer is replaced every two years
• Current Technology Fee: $700 per semester for students in the program (“non mobile” students, e.g., many graduate students, pay a technology fee of $200 per semester)
• Bundled software includes: MS Windows ME, MS Office 2000, SPSS, Maple V, various utilities
• Bundled services include:– Technology Help Desk, PC Repairs, Loaner Computers– Network Services, including wireless network access from most
academic and public spaces
Mobile Computing (cont.)
• Program Phase-In
– 1995 – 1997: Pilot Programs
• Focus on Teaching and Learning
• Based on cohorts with common academic experiences (e.g., Business Honors Program in 1995, expanded to all 1st Year Business, Biology, and Honors students in Fall 1997)
• Began Teaching, Learning, and Technology Center and internal grant program to provide incentives to departments willing to integrate technology into large enrollment core courses
Mobile Computing (cont.)• Program Phase-In (cont.)
– 1998 – 1999: Full Roll Out of Program
• Program is Mandatory for ALL Incoming FT Undergraduates
• Focus on Infrastructure
• Major Network / Server Upgrades– Lotus Notes Email / Lotus LearningSpace Course
Management System– MS FrontPage Enabled Web Servers for Faculty and
Students– ATM Backbone / Campus Internet Upgrade /
Server Consolidation / Y2K Preparation
Mobile Computing (cont.)• Program Phase-In (cont.)
– 2000 – Present: Enterprise Perspective
• Focus on Enterprise Systems
– Implementation of Web services for students (e.g., online registration, course audit, grades, etc.)
– Automation of Mobile Computing processes (e.g., asset management, account generation, etc.)
– Implementation of BlackBoard course management / portal system; 24/7 support services through Eduprise
– Campus-wide wireless networking / Campus network upgraded to gigabit Ethernet backbone
Mobile Computing (cont.)
• Major investment on the part of the University– Central IT Budget (incl. laptop leases /student fees)
for FY’02 approx. $15 million or 13% SO Campus G&E(10% of the total University budget)• Almost equally divided between personnel, operating, and Mobile
Computing Program (technology fee) budgets• Most costs are operationalized / very little capital funding
• Seton Hall University is a different place because of the University’s strategic initiatives (enrollments up, SAT’s up, new programs launched, etc.)
• Mobile Computing Assessment Project indicates the program has had positive impact on learning / student recruitment
• Value of Seton Hall Results - DATA
• Underlying, Theory and Process
• Technology Assessment in Larger Institutional Context
Charge of theMobile Assessment Team
• Assess the impact of mobile computing on fall 1998 entering freshman
• This includes student satisfaction with the learning environment and student learning outcomes
Full Report and Resources
Available On-Line at:http://tltc.shu.edu/initiatives/assessment/index.html
Key Researchers and Co-Authors
Eric Fountain, Ed.D.
John Collins, Ed.D.
Janet Easterling
Circles of Assessment
Institution
External
Faculty
Students
When Assessment Occurs
Before
Determine Needs
During
Make Changes in Process
After
Evaluate Experience
Examples of Assessment Topics
Faculty Students Institution
• Technical Skills
• Instructional Goals
• Technical Skills
• Enrollment Decisions
• Community Engagement
• Resources
• Support
• Time Distribution
• Access
• General Satisfaction
• Bottlenecks
• Network Stability
• Training Needs
• Tool Fits Goal
• Info. Access• Communication• Technical Skills
• Delivery Change
• Evaluate Tech. Options
D
A
B
Types of Assessment
• Focus Groups• Interviews• Journals• Observations• Portfolios• Performances
• Standardized Tests• Surveys• Tracking Systems• Longitudinal Records• Document Reviews• Skills Application
Demonstrations
Narrowing the Question
Fantasy:a + b = c
a = teaching, b = technology, c = learning
Reality:a + b + c + d + e + f…
discipline, teaching style, technology proficiency, fit of application to task, desired learning objectives...
Faculty Journals (1997)
• “Cognitive, behavioral and affective sides of transition”
• “Technology has become a natural part of my teaching”
• “Be prepared to punt”
• “A year ago I couldn’t imagine using all this stuff: now I can.”
• “Technology is only, alas, a dumb machine that is no more creative, inventive, or productive than the human using it”
Technology Functions
• Edit• Collaborate• Create• Compose• Automate• Visualize• Simulate
• Communicate• Store• Structure Practice• Retrieve
• Contact between student and professor
• Cooperation among students
• Active learning• Prompt feedback• Time on Task• High Expectations• Respect diversity in
talents and learning
Seven Principles(Chickering and Gamson, 1989)
Nature of Learning(AAHE, NASPA, 1998)
• Strongly affected by educational climate
• Rapid, rich, frequent feedback
• Much is informal and incidental
• Requires transfer of specific knowledge
• Ability of learner to monitor own learning
• Making Connections
• Compelling Situation
• Active search for meaning by learner
• Developmental, cumulative process
• Individual learners are social beings
Evaluation of General Satisfaction, Technology Use, and Student
Perceptions of the Impact of Mobile Computing on the Learning Environment
at Seton Hall UniversitySpring 1998 – Fall 2000
Gender
R1a. Does a student’s gender have an effect on their perceptions regarding the satisfaction of the SHU Mobile Computing Program?
R2a. Does a student’s gender have an effect on their perceptions regarding the use of computers in the SHU Mobile Computing Program?
R3a. Does a student’s gender have an effect on their perceptions regarding the desired outcomes of the SHU Mobile Computing Program?
Independent Variables
Housing Status
R1b. Does a student’s housing status have an effect on their perceptions regarding satisfaction of the SHU Mobile Computing Program?
R2b. Does a student’s housing status have an effect on their perceptions regarding the use of computers in the SHU Mobile Computing Program?
R3b. Does a student’s housing status have an effect on their perceptions regarding the desired outcomes of the SHU Mobile Computing Program?
Independent Variables
Ethnicity
R1c. Does a student’s ethnic affiliation have an effect on their perceptions regarding satisfaction of the SHU Mobile Computing Program?
R2c. Does a student’s ethnic affiliation have an effect on their perceptions regarding the use of computers in the SHU Mobile Computing Program?
R3c. Does a student’s ethnic affiliation have an effect on their perceptions regarding the desired outcomes of the SHU Mobile Computing Program?
Independent Variables
• Structural
• Use of Mobile vs. Non-Mobile Construct
• Neutral Category
• Administrative
• Sampling
• Change from paper to online.
Limitations
RESULTS
Term Target Popn: #FR
Survey Responses#FR (FR Resp.Rate)
TargetPopn:#Non-FR/U(returning students)
Survey Responses#Non-FR/U (Non-FR/U Resp. Rate)
Target Popn:
Comb. Total
Survey Responses(Comb. Resp.Rate)
Fall 1998 1226 324 (26%) 365(20+20+325)
90 (25%) 1591 414 (26%)
Spring 1999
1226 216 (18%) 365(20+20+325)
124 (34%) 1591 340 (27%)
Fall 1999 1070 311 (29%) 1591(365+1226)
453 (28%) 2661 764 (29%)
Spring 2000 1070 179 (17%) 1591(365+1226)
18 (01%) 2661 197 ( 7%)
Fall 2000 1125 235 (21%) 2661(1591+1070)
476 (18%) 3786 711 (19%)
Totals – Fall Terms Only
All 5 Terms
3421 870 (25%) 4617 1019 (22%) 12,290 2,426 (20%)
Response Rates
Gender Ethnicity Housing Status
Fall Terms Only FR Only N=870
46%
Male
35% Non-White or other/(missing)
39%
Commuters
SHU Freshmen(Fall ’98,’99,’00)
48%
Male
52% Non-White or other/(missing)
37%
Commuters
Response Demographics
46.5
21.9
45.7
18.2
39.1
17
41.4
14.1
40.5
12.7
0
10
20
30
40
50
60
70
Per
cen
tag
e
Satisfactionw / laptop
Satisfactionw / SHU
Value ofVisiting Help
Desk
Satisfactionw / MobileCourses
Satisfactionw / Non-Mobile
Courses
Satisfaction Items - "TOP 5"
Student Satisfaction - Items with highest proportions satisfied
Very HighSatisfaction
High Satisfaction
68.463.9
56.1 55.553.2
Top Five Responses – Satisfaction68.4 63.9 56.1 55.5 53.2
40.3
20.7
34.8
24
29.2
25.9
25.9
17.7
14.3
19.9
0
10
20
30
40
50
60
70
Pe
rce
nta
ge
Drill &
Pra
ctice
Compu
ter-b
ased
train
ing
Prese
ntat
ions
Comm
unicat
ing (e
, etc.
)
Resea
rchi
ng (I
nter
net,
etc.
)
Technology Use Items - "TOP 5"
Technology Use - Items w ith highest proportions specifying 6 or more hours per week
6-10 hours per week
More than 10 hoursper week
61.0 58.855.1
43.6
34.2
Top Five Responses – Technology Use
54.5
23.7
50.7
23.4
53.6
19.2
52.6
19.5
54.4
17.7
0
10
20
30
40
50
60
70
80
Per
cen
tag
e
Contact w /Professors
Accessmaterialsrelated tocourse
Learntechnology
skills
Get promptfeedback
fromProfessors
Present w orkin manyw ays
Satisfaction Items - "TOP 5"
Student Satisfaction - Items with highest proportions satisfied
Very Positive
Positive
78.274.1 72.8 72.1 72.1
Top Five Responses – Outcomes
Examining the Research Questions
Gender
R1a. Does a student’s gender have an effect on their perceptions regarding the satisfaction of the SHU Mobile Computing Program?
R2a. Does a student’s gender have an effect on their perceptions regarding the use of computers in the SHU Mobile Computing Program?
R3a. Does a student’s gender have an effect on their perceptions regarding the desired outcomes of the SHU Mobile Computing Program?
Independent Samples Test - Gender
9.471 .002
-2.276 558.193 .023
.452 .501 -3.076 2186 .002
.190 .663 .760 1976 .447
Equal variancesassumedEqual variancesnot assumedEqual variancesassumedEqual variancesnot assumedEqual variancesassumedEqual variancesnot assumed
Sum of SatisfactionVariables
Sum of Use Variables
Sum of OutcomesVariables
F Sig.
Levene's Test forEquality of Variances
t df Sig. (2-tailed)
t-test for Equality of Means
Gender
Housing Status
R1b. Does a student’s housing status have an effect on their perceptions regarding satisfaction of the SHU Mobile Computing Program?
R2b. Does a student’s housing status have an effect on their perceptions regarding the use of computers in the SHU Mobile Computing Program?
R3b. Does a student’s housing status have an effect on their perceptions regarding the desired outcomes of the SHU Mobile Computing Program?
Housing Status
Independent Samples Test - Housing Status
1.136 .287 -.420 592 .675
1.596 .207 -7.200 2175 .000
.681 .409 -.649 1969 .516
Equal variancesassumed
Equal variancesassumed
Equal variancesassumed
Sum of SatisfactionVariables
Sum of Use Variables
Sum of OutcomesVariables
F Sig.
Levene's Test forEquality of Variances
t df Sig. (2-tailed)
t-test for Equality of Means
Ethnicity
R1c. Does a student’s ethnic affiliation have an effect on their perceptions regarding satisfaction of the SHU Mobile Computing Program?
R2c. Does a student’s ethnic affiliation have an effect on their perceptions regarding the use of computers in the SHU Mobile Computing Program?
R3c. Does a student’s ethnic affiliation have an effect on their perceptions regarding the desired outcomes of the SHU Mobile Computing Program?
EthnicityANOVA - Racial/Ethnic Identification
129.872 4 32.468 .964 .426
19694.6 585 33.666
19824.5 589
449.155 4 112.289 1.541 .188
157751 2165 72.864
158200 2169
3072.575 4 768.144 3.332 .010
452080 1961 230.535
455153 1965
Between Groups
Within Groups
Total
Between Groups
Within Groups
Total
Between Groups
Within Groups
Total
Sum of SatisfactionVariables
Sum of Use Variables
Sum of OutcomesVariables
Sum ofSquares df Mean Square F Sig.
Changes in MC 2002 Results
• 81% of the respondents said they had used email to communicate on class-related information on a daily or weekly basis.
• 68% of the respondents used the internet for research on a daily or weekly basis.
• 56% submitted work electronically.
Changes in MC 2002, cont.• 70% had contact with their professors often or very
often. An additional 20% reported occasional contact.• 68% reported that it helped them take control of their
own learning.• 70% used the internet to access class materials online.• While 66% of students felt their laptop use did not
detract from their ability to pay attention in class, 45% felt that laptop use was distracting other students.
Conclusions
•Attracted to Seton Hall University by the availability of technology at the University and the infusion of technology in the curriculum, as suggested by student reported positive influences on the decision to attend the University,
• Satisfied with the Mobile Computing Program as well as with the laptop and support services once enrolled at the University,
• Making good use of the technology available to all students, at least in terms of certain types of technology use, and.
• Perceiving a substantive positive impact of Seton Hall University’s Mobile Computing Program on the learning environment.
The typical Seton Hall University student, whether representing all respondents, or particular groups of students (groups identified by gender, race/ethnicity, and residence status), is generally:
Technology Assessment Projects
Recurring Program Assessment
University Assessment, Planning and Budget
Technology Assessment
Initiatives
Programs for Coordinating Assessment
Major University Assessment,
Budgeting and Planning
Mobile Computing Outcomes Assessment Team
Sesquicentennial Strategic Plan
Tablet PC Pilot TLTR Subcommittee
IT 2
Large Course Redesign
Middle States Association
Questions?