Design-based research

How much should educational research techniques mirror natural science research techniques?

What about the research approaches of engineering? Prototyping.

Technology = Application of knowledge

Technology = Creation of knowledge

Technology = Transforming and intervening in the natural world to produce desired results

See handout

Selection skills

Pay close attention to:
1. Data being measured (nominal "categorical", ordinal "ranked", interval "scaled")
2. Number and kind of groups involved
3. Groups are compared or related

SPSS practice

(2) Estimate population mean based on sample mean when sigma is not known (t):

&

(3) Test for statistically significant difference between means (do the intervals overlap?):

SPSS, Analyze, Descriptive Statistics, Explore, Statistics, Confidence Interval
Dependent List = Scaled data variable
Factor List = Grouping variable

Graphing (SPSS, Graphs, Legacy Dialogs, Error Bar) Variable = Scaled data variable, X axis = Grouping variable


Test for statistically significant difference between means (does interval contain zero?) for (4) between groups:

SPSS, Analyze, Compare Means, Independent Samples T-test (Does the interval cross zero? If crosses zero, then no significant difference.)


Test for statistically significant difference between means (does interval contain zero?) for (5) repeated measures:

SPSS, Analyze, Compare Means, Paired Samples T-test (Does the interval cross zero? If crosses zero, then no significant difference.)

Confidence Intervals

The larger the interval range, the more the confidence percentage. People usually set the confidence percentage first and then find the range.

Confidence interval is plus or minus the z score of alpha/2 multiplied by standard error (sigma/square root of n). You can substitute t for z and sigma for s, if you do not know sigma.

"95% confident that the interval includes the parameter." Avoid the "parameter falls between the interval."

Comparing Means With Confidence Intervals

Do the intervals overlap? If yes, no statistical difference. That's it. Replaces Ho testing for between groups and repeated measures.

Confidence Interval for proportions/percentages/pearson r is not tested in this course.

Confidence Interval applications:
1. Estimate population mean based on sample mean when sigma is known (z)
2. Estimate population mean based on sample mean when sigma is not known (t)
3. Test for statistically significant difference between means (do the intervals overlap?)
4. and 5. Test for statistically significant difference between means (does interval contain zero?) for (4) between groups and (5) repeated measures

Chapter 14 Confidence Intervals

Pg (440-447 only)

Confidence intervals look like 78% (+ - 3%). The interval is 75% to 81%.

Non-parametric tests

Chi-square test of independence (two variables or groupings). The differences between groups are tested. You test the Ho and either reject or fail to reject Ho. No association between groups is called independence.





(SPSS, Analyze, Descriptive Statistics, Cross-tab, Statistics, Chi-square)








Chi-square test of goodness of fit (one variable or group). The differences between the group and your expectations (chance) are tested. You test the same way as the test of independence. Expectations of Ho are equal proporation across all choices (chance).



(SPSS, Analyze, Nonparametric Tests, Chi-square)







Chi-square test of significance of a proportion (compare 2 frequencies only like "yes" and "no"). Small case of goodness of fit test.

Chi-square homework set in blackboard.

Non-parametric tests (Chapter 13)

Chapter 13 (Pages 407-416, 431-432 only)

Parametric techniques when we have scaled data (interval or ratio) such as r, z, t, F.

Non-parametric techniques when we have non-scaled data (ranked or categorical) such as Chi-square or phi coefficient.

Chi-square test of independence (conditional probability):
This is the test of two variables (groups) that shows they are independent of each other.

Chi-square test of goodness-of-fit:
Compare proportions.

ANOVA and exam 2

Repeated measures output. If you get significance then do Post Hoc, which is a paired samples t-test with corrected p-value for multiple t-test. Bonforoni correction: 0.05 divided by the number of comparisons.






Between Groups:





Important output for Between Groups ANOVA. It follows the BG WG formulas.



Study guide in blackboard for exam 2.



Studying in this exam (7 kinds):

1-way ANOVA (2 kinds) - (1-way = 1 independent variable)

1. Between Groups (SPSS: Analyze, Compare Means, One-way ANOVA)

2. Repeated measures (SPSS: Analyze, General Linear Model, Repeated Measures)

T-test (3 kinds)

1. Single sample (SPSS: Analyze, Compare Means, One sample t-test)

2. Between Groups (SPSS: Analyze, Compare Means, Independent samples t-test)

3. Repeated measures (SPSS: Analyze, Compare Means, Paired-samples t-test)

Pearson r

Simple linear regression

ANOVA - Chapter 12

Don't read pg 368-373.

Things beyond this class:

1. Factorial ANOVA designs
2. Dunns procedure

Announced that even though the course schedule has the 1st draft of full report due, the draft is now due Apr 5.

ANOVA accounts for the variation within groups (WG) and between groups (BG).

Optimal case is low variation within groups (WG) and large variation between groups (BG) because it is said that you have controlled extraneous variables and are measuring what you want to measure.

Total Variance (TV) = BG + WG

F statistic = BG/WG

A significant F statistic does not tell you which means are significantly different from which means. It only tells you that at least one mean is significantly different from another.

T-test and ANOVA

T-tests are designed for smaller samples.

Three ways to compare means of two groups (sample vs. population, sample vs. sample, pre vs. post)

pg 308-320 is not on the test. Scan it, but don't do z-test practice problems.

pg 339 #3 question (make sure you understand how to answer this.)

Data enter one column for age and one column for status (married or bachelor).



ANOVA

Chapter 12

T-test only accomodates comparing 2 means. ANOVA is used to compare more than 2 means. Multiple t-tests inflates the alpha level, so we use ANOVA, since a extreme sample would show up multiple times in the pairings. Sample 1 vs. Sample 2, Sample 1 vs. Sample 3, etc. If sample 1 is an outlier, each pairing might give you a type 1 error.

ANOVA is the workhorse of statistics...any research question you have, many would conform it to how to run an ANOVA. That is a little backwards, but it has been like that in the past.

We will study between-groups and repeated-measure ANOVAs.

ANOVA (analysis of variance)

When to use ANOVA:
1. More than two means compared
2. The group means should vary widely from grand mean (mean of means)
3. The groups' raw data does not vary widely. (Close clustering of scores)

Total variance = Variance + error or Between-groups variance + within-groups variance

Between-groups variance is the good stuff. (treatment + error)
Within-groups variance is the bad stuff (variance due to error only since each member of the group is exposed to the same treatment).

F statistic = BG/WG or (treatment + error)/error

F = 1 (means treatment had no effect)

Chapter 11

Analyze data from experiments, quasi-experiments, and comparing groups

T-test (used to compare groups) - replaces z-test

T-test of the means of groups. Use the t-test when the population standard deviation is unknown (sigma).

Sample compared to population:
T = sample mean - population mean divided by standard error

df = degrees of freedom, typically n-1

Sample compared to sample:
T = sample1 mean - sample2 mean divided by standard error

df = n1 +n2 - 2

Pre-post comparison: (repeat measures format)

Page 335 for formula

Chapter 10

Chapter 10 – Designing Experiments

Causality
What is the cause of phenomena?
3 Conditions of internal validity:
1. X comes before Y (antecedence)
2. X and Y are in the same space and time (contiguity)
3. Z is explained away (necessary connection)

Experiments
Prediction and control…independent variable (we manipulate), dependent variable (we measure).
Pre-test -> Treatment -> Post-test
Pre-test -> Control -> Post-test
*Randomly assigning participants to treatment and control groups. This is not random sampling a population. Both groups are equivalent.
External validity: generalization from lab to real world.
Internal validity goes up (control goes up) then external validity goes down (generalizability to real life)

Between groups and repeat measure are the basic experimental design.
Time related effects are the huge critic against repeated measure cause. (pp 279-285).

Experiments and quasi-experiments

ROXOA
R=Randomization
O=Observation
X=Treatment
A=Analyze data

Experiments:
ROXOA
RO OA

R XOA
R OA

Threats to validity: R

Selection (messed up the random assignment)
Mortality (participants drop out of groups unequally) If most of the females drop out in one group, it throws off the experiments. Called mortality because it references the use of rats and rats dying at unequal rates.

Threats to validity: O

Testing: the testing sensitizes participants to treatment
Instrument: measurement devices are messed up)

Threats to validity: X

Experimenter bias (treat groups unequally)
Experiment diffusion (something contaminates the control group)

Threats to validity: O - O (multiple observations)

History (something affects participants outside of the experiement like TV show, breaking news, etc.)
Maturation (something within the participants changes)

Threats to validity: A

Statistical regression (extreme scores are not repeatable, tend to move toward the mean in second observations)
Statistical conclusion (improper or faulty analysis)


Quasi-experiments

Do not include randomization, so groups are not equivalent.

1. Nonequivalent control group: OXOA compared with O_OA
2. Simple interrupted time-series: OOOOOXOOOOOA (no random and no group compare)
3. Combined (Time series with nonequivalent control group): OOOXOOOA compared with OOO_OOOA.

SPSS

SPSS definitions:

p-value = Significance (SPSS)

pearson r = Pearson Correlation (bivariate SPSS)

Results section

"A significant correlation was observed between stress3 and coursgrad, r=-0.83, p=0.000."

Discussion section

"I interpret the correlation...." Correlation is for prediction, not causal. Avoid the impression that readers may have that you view it as causal.

r squared is the percentage of variance in y that is systematically varying with x. R-squared is the name of the game. The higher the better with the least number of independent variables.

y-intercept is the number in the constant row under Unstandardized Coeffificent B column (y = slope * x + y-intercept) (y=bx + a) See image above.

slope is the number under the constant row in the same area. (b**)

**If you standardize b, you get r.

The Ed Psych Data Directions in Blackboard is what we need to know.

Regression

In a regression line, for each x score you get a normal distribution of y scores with the mean on the regression line. See page 222.

Regression lines should not go beyond the original data points.

Error is regression is the difference between the average predicted Y and the actual Y.
(Y-Ybar). Observed minus expected. This is called standard error of the estimate, but it is the standard deviation using the Y and expected Y along (running) the regression line. Also known as coefficient of non-determination. (1-r squared).

Opposite of the error is the coefficient of determination (variability accounted for).

How much variance is error and how much is accounted for by a correlation with another variable?

Total variance = variance accounted for + error.

Variance accounted for is r squared (pearson r squared)

The connection between correlation and regression is r = a standardized slope of the regression line. r = b (s sub x / s sub y)

Correlation and Regression

Covariance is x and y in the same formula as variance is x squared.

Table B3 determines whether or not the observed pearson r is a "rare event" unlikely to have occured by chance.

A large sample size usually makes r values significant.

The formula and calculation for comparing r's will not be required on test.

Regression

Uses the classic equation for a line: y = mx + b, but the letters are different in stats. It is y=bx+a where b is the slope and a is the y intercept.

Slope = rise over run or y1-y2 divided by x1-x2

Prediction comes from graphing the line and predicting x, y coordinates on the line.

Data that can be described as a line is known as perfectly linear relationship.

Best-fitting line is known as the regression line.

Method of least-square creates the regression line (or best-fitting line): It is the line that minimizes the overall distance between the regression line and the data points.

Calculation is not required for test.

Correlation

Correlation: Indicates the relationship between 2 or more variables like smoking and lung cancer. Strength of relationship and direction of relationship (positive or negative).

Scale data - Pearson Product-Moment Correlation (aka Pearson Correlation)
Ordinal data - Spearman Rank-Order Correlation
Nominal data - Phi Coefficient

Pearson Correlation: Range is from 1 to -1. Closer to 1 or -1 the stronger the relationship. At 0, no linear relationship whatsoever. Scatter-graph that looks like a line is a strong relationship.

Correlation coefficient = r, r is a standard index from 1 to -1.

Important caveats about Pearson r:

1. Not all important or interesting relationships are linear. (Yerkes-Dodson Law)

2. Watch out for spurious correlations (counterfeit correlation)

A. Restricted range (see handout) - full range shows relationship where restricted range shows counterfeit correlation.

B. Combined groups: combining groups may off-set or wipe out a correlation that exists when the groups are not combined. Breaking out groups by demographics or gender or something helps avoid this problem.

C. Outliers: outliers through off calculations. Why is there an outlier? You have to explain the outliers.

Correlation does not equal causation, it equals a degree of covarying.

Correlaiton does not tell us:
x -> y
y -> x
z -> x and y
coincedence

Pearson r formula is covariance divided by total variability

SPSS introduction

Data, Select cases - Look at a sub-set of the data

Transform, Recode into different variable - Change the data like gender as 1 and 2 could be changed to 3 and 4, or grades like everything above C is 1 and below is 2.

Transform, Compute variable - take several variables and calculate a new variable.

Analyze is where SPSS is powerful.

• Descriptive statistics
• ANOVA
• T-test
• General linear model
• Correlate
• Regression
• Nonparametric tests
• Scale

Hypothesis testing - probability

Alpha level (:z-critical) aka cut-off: the likelihood of obtaining a type 1 error (errorenously rejecting Ho or random sampling error), traditional level is 0.05.

Directional: words like below or above or more or less are used.

Non-directional: words like difference or change or impact are used.

P score (:z-observed) aka observed score: Alpha is set by you like 0.05. P score is the probability of the observed score from your sample given that H0 (null hypothesis) is true.

"If your P score is less probable than alpha, you have a score to reject H0 (null hypothesis)".

Decision Errors

Type 1 error is false positive (errorneously rejecting H0) - the likelihood is alpha.
Type 2 error is false negative (errorneously failing to reject H0) - the likelihood is called beta. (Beta is not taught in this class)

As alpha decreases beta increases and visa versa.

Power: Probability that the test will lead to a reject H0 when H0 is actually false. You rejected H0 when you should reject H0.

Telescope example...type 2 error is a telescope that doesn't have enough power to see the asteriod that exists. If it has enough power, then you correctly reject H0.

Hypothesis testing - Standard error of the means

"If you criticize something, you are obligated to know it better than those that espouse it."

Raw data -> Summarized, Organized, Simplified (Descriptive statistics: s, x-bar, s-squared) -> Sample to population inferences (Inferential statistics: p, z, t, F, q)

Hypothesis testing

1. Simple random sampling: used for statistical inference, where populations are inaccessible, and are often more accurate. All units in the population have an equal chance of being selected.

2. Proportional Stratified Random Sample: sample maps exactly onto the population in terms of proportions of sub-groups (e.g. population has 10% seniors and sample has 10% seniors)

3. "Errors" in sampling (sampling error and non-sampling error) must be dealt with. Samples and population don't match-up. Non-sampling errors include question text and framing that creates confusion. Other things like cultural issues can cause non-sampling error.

Sampling Distributions

How do you detect how much error (sampling error) is in the sample? Use a standard-deviation-like calculation (spread of scores with respect to the mean).

By selecting multiple samples and calculating the means of those samples, then using the means in the place of raw scores and calculate the standard deviation of the means-like statsitic called the standard error of the means (s-sub-x-bar).

Standard error of the means = sample standard deviation divided by the square root of the number of observations in the sample. (s / sqrt n)

or theoretical (sigma-sub-x-bar = sigma / sqrt N)

A sampling (sample of means) distribution is normally distributed when it is drawn from a normally distributed population or the size of the samples is reasonably large (at least 30).