TECHNICAL APPENDIX

TECHNICAL APPENDIX

METHODOLOGY SUMMARY FOR CPM RESEARCH

Missions International PO Box 681299 – Franklin, TN 37068 – www.missions.com

© 2004 Missions International

Human Resourcing

• Design of CPM assessment instrument and analysis support

• Hired Statistician– Ph.D. from Vanderbilt in Psychology– VP of Methodology for a Market Research Firm– Over 15 years corporate research experience

with Fortune 500 companies

• Previous analytic work for MI on (SG)2


OBJECTIVES

• Discover elements (DNA) that drive conversion and planting growth– Measure conversions and planting growth– Measure comprehensive set of potential

“drivers” (DNA)– Build predictive models to uncover which

elements from comprehensive set are most “important” in driving conversions/planting

– Substantive CPM interpretations across predictive models


Assessment Design

• Qualitative content inputs from key expert leaders in Church Planting Movement - Watson, Sergeant, Sanchez, etc.

• What are “DNA” elements of church planting movement?

• Developed initial battery of items to capture that domain

• Subjected to review by experts• Iterated to list of 31 items designed to capture

domain broadly• All elements measured on five-point frequency-

of-occurrence scale: “Very Often” to “Almost Never”


Other Variables Measured

• Also included several descriptive / demographic measures– age, gender, education, size of community,

leader/member, formal training, how long ago church started, how many members attend, etc.

• Also included “Dependent” Variables– Self-report on frequency of starting new churches

• On same “Very Often” to “Almost Never” scale– Raw numeric dependent variables:

• How many converted through your church?• How many converted through your church have

been baptized?• How many churches branched off from your

church?


Samples• Sample sizes by contributing countries:

– (n=542)– (n=507)– (n=858)– (n=61)

• One sample showed extremely limited variability (interpreted as most elements from expert inputs fully in operation in this movement, thus “topping out” the scale – i.e., all aspects tend to be happening very often in this movement).

• Analysis conducted on other countries where more variability existed - pooling of other nations (starting n=1110)


More About Dependent Variables

• Self-report of Church planting activity– “Our church starts new churches…”

– “Very Often” – “Often” – “Sometimes” – “Rarely”– “Almost Never”

• Coded as quasi-interval 1-5 ratings


Numeric Dependent Variables

• Number of conversions, baptisms, new churches started

• All influenced by (a) length of existence– 24 conversions from 2 year old church, vs. 24

conversions from 2 month old church– 1 conversion per month vs. 12 per month

• All influenced by (b) number of members– 50 conversions from church of 10, vs. 50 conversions

from church of 25– 5 conversion per member vs. 2 per member

• Must control for length of existence and number of members to make raw numbers sensible and comparable


“Per Member Per Month”- pmpm

• Implemented per member per month approach• 100 conversions from 10 member church in existence for

1 month =10 conversions per member per month

• 200 conversions from 20 members in existence 5 months =

2 conversion per member per month• Raw numbers would have made 200 look “better” than

100, when clearly in this scenario 200 was less productive relative conversion growth than 100

• Thus created metric – per member per month (pmpm) – to take care of this issue

• Approach taken with number of conversions, number of baptisms, number of churches planted


Further Treatment of Dependent Variables - indexing

• Baptism often associated with conversion, thus reported numbers of baptisms highly correlated with numbers of conversions

• Created a single index of conversions and baptisms

• This index is a simple average of the two quantities, expressed in pmpm units


Further Treatment of Numeric Dependent Variables- normalization

• Distributions of variables in original and pmpm forms “positively” / “right” skewed

• Multivariate procedure - assumptions call for transformations to normality

• Cube root transformation chosen


Further Treatment of Dependent Variables - outliers

• Examined distributions on numeric pmpm variables in terms of “standardized” / “z-scores”

• This procedure helps identify “outliers” – cases that are quite different from most other cases in the sample

• A typical cut-off would be z-scores greater than 3.0 in absolute value (+/- 3.0)

• After inspection, allowing retention of more original sample, the cut-off for this study was set at +/- 3.5

• Typically less than 1% of cases eliminated


Final Variables For Predictive Models after all Treatments

• Dependent Variables– Index of conversions and baptisms, pmpm

(cube-root transformed)– Number of churches planted, pmpm (cube-

root transformed)– Self-reported frequency of church starts

• Independent Variables– 31 items capturing domain of “DNA of CPM”


Model Building - regression

• Many available approaches to predictive model building

• Typical statistical tool is multiple regression• Predict a dependent variable from a set of

independent variables• Find a weight for each independent variable

such that the weighted combination of independent variable values produces an output that is “as close as possible” to the actual/observed dependent variable values


Model Building - Regression

• Weights combine independent variables to produce an output – a predicted dv value

• (w1*iv1 + w2*iv2 + w3*iv3…) = output / predicted dv

• Output/predicted dv as close as possible to observed dependent variable scores

• Technically, squared errors between predicted dv and observed dv are minimized, hence the name “least squares” regression

• When standardized, regression weights can be interpreted as “relative impact” – how much unique impact each iv has on the dv


Model Building Procedure

• There are different approaches to regression depending on the research objectives

• Many options for variable selection in building a predictive multiple regression equation– Backward entry– Forward entry– Stepwise entry– Simultaneous/ forced entry

• A hybrid “procedure” was developed here to arrive at the final models


Hybrid Model Building Procedure

• Not logical to accept a variable as a significant driver if it shows no meaningful bivariate correlation

• Thus, screened independent variable set to include only starting variables with meaningful correlations

• Organized meaningfully correlated independent variables into “tiers”, e.g.,– .4s and above– .3s– .2s– .1s


Hybrid Model Building Procedure

• Used forward entry within each tier, starting with strongest correlation tier first

• Forward entry added variables one at a time based on biggest improvement in R-square

• Forward entry thus identified subset of significant predictors in strongest correlational tier

• That subset became starting basis for consideration of next tier


Hybrid Model Building Procedure• Forward entry within next tier added additional significant

predictors• Significant predictors from first two tiers became starting

basis, in moving to the next correlational tier• Tested that next tier for any additional variables adding

predictive power• Procedure continued until all tiers were exhausted• (note – not all models had all four tiers of correlational

strengths. If only .3s, .2s, .1s existed, general procedure was still same, progressing from highest strength tier to lowest strength tier)

• After no additional variables added significant predictive power, tried to re-enter any previously eliminated variables

• When no additional variables add anything, final predictive set has been discovered


Hybrid Model Building Procedure• Traditional regression diagnostics checked at each step

along the way (e.g., outliers, collinearity, normality of errors of prediction, quality of prediction as evidenced in adjusted R-squared, checks for sign reversal/ suppressor variables)

• Note 1 – for treatment of “missing” data, large total sample size allowed use of “listwise” procedure – modeling used only those individual respondents with complete data on all variables in any given model

• Note 2 – a few negative regression coefficients were accepted as valid. This was an acceptable finding only when:– Original bivariate correlation was negative, thus indicating not a

sign-reversal / suppressor problem– Interpretation of negative association was plausible and logical

given understanding of domain of study


Overall Model Summaries

Dependent Variable R R SquareAdjusted R Square F

df Reg

df Res signif

Self-Report Planting 0.614 0.377 0.372 70.877 9 1052 0.000Conversion/Baptism (cube root) 0.625 0.391 0.385 66.130 9 928 0.000Actual Planting (cube root) 0.467 0.218 0.213 45.064 6 971 0.000


Self-Report Planting Drivers

= negative association

Item Beta t signifOur church prays passionately at our meetings… 0.21 6.27 0.000A new convert starting a new church immediately after conversion is something that happens… 0.19 6.89 0.000When I share the Gospel with others I try to invite their relatives and friends… 0.18 6.03 0.000In our church we see God perform miracles… 0.09 3.07 0.002

Our church looks for places that have not heard the Gospel, then we do our best to win converts and plant churches there. We intentionally do that… 0.09 2.72 0.007I pray passionately… 0.08 2.39 0.017In our church we assist new converts when they start new churches… 0.07 2.15 0.032When someone from our church starts a new church, we watch over them just long enough for them to get established. Then we leave them to function on their own as soon as possible. That is the way we do it… 0.06 2.02 0.044

We intentionally spread the Gospel to other people groups outside of our community. We do that … -0.24 -7.25 0.000


Conversion/ Baptism Drivers


Item Beta t signifEncouraging new believers to start new churches immediately, is something we do … 0.14 4.10 0.000Think about members of your church who are not your relatives. How often is it the case that they were already your friends before you both became believers… 0.12 4.09 0.000In our church we model for new converts how to start new churches… 0.10 3.40 0.001A new convert starting a new church immediately after conversion is something that happens… 0.09 3.27 0.001

We intentionally spread the Gospel to other people groups outside of our community. We do that … 0.09 2.46 0.014In our church, based on what we are learning from God's word and each other, we make changes in the way we act … 0.07 2.16 0.031It is more important to learn from the Word of God than from any human teacher. We emphasize that principle in our church… 0.06 2.03 0.042Our church receives financial support from outside sources … -0.21 -6.04 0.000People of all backgrounds, classes, occupations, ages, and genders mix together in the churches of our community. That happens… -0.34 -9.27 0.000


Numeric Planting Drivers


Item Beta t signif

When I share the Gospel with others I try to invite their relatives and friends… 0.19 5.45 0.000A new convert starting a new church immediately after conversion is something that happens… 0.18 5.89 0.000From the time I became a Christian until now, I have been sharing the Gospel with unbelievers… 0.12 3.38 0.001My church holds me accountable to put into practice what we are learning. They do that… 0.08 2.61 0.009

We intentionally spread the Gospel to other people groups outside of our community. We do that … -0.15 -4.28 0.000People of all backgrounds, classes, occupations, ages, and genders mix together in the churches of our community. That happens… -0.15 -5.00 0.000


Interpretative Note • Substantive interpretation is a synthesis across

predictive models, also made in light of original qualitative research with experts

• This synthesis draws out key areas of focus found to be predictive of Church Planting success

• If generalizability holds beyond the measured samples, based on the data collected for this study, deeper implementation of positively related DNA elements (and avoidance of negatively related ones) should lead to more conversion growth and baptisms, and increased levels of church planting activity

Documents

TECHNICAL APPENDIX