Difference between revisions of "Guidelines"

From StatWiki
Jump to: navigation, search
(Order of Operations)
(Order of Operations)
Line 74: Line 74:
### Test significance of moderation via chi-square difference test
### Test significance of moderation via chi-square difference test
# Report findings in a concise table
# Report findings in a concise table
##      Ensure global and local tests are met
##      Include post-hoc power analyses for unsupported direct effects hypotheses
# Write paper
# Write paper
## See guidelines below
## See guidelines below

Revision as of 10:40, 22 April 2016

On this wiki page I share my 10 Steps to building a good quantitative model, as well as some general guidelines for structuring a quantitative model building/testing paper. These are just off the top of my head and do not come from any sort of published work. However, I have found them useful and hope you do as well.

Example Analysis

I've created an example of some quantitative analyses. The most useful part of this example is probably the wording. It is often difficult to figure out how to word your findings, or to figure out how much space to use on findings, or which measures to report and how to report them. This offers just one example of how you might do it.

Ten Steps

Ten Steps for Formulating a Decent Quantitative Model

  1. Identify and define your dependent variables. These should be the outcome(s) of the phenomenon you are interested in better understanding. They should be the effected thing(s) in your research questions.
  2. Figure out why explaining and predicting these DVs is important.
    1. Why should we care?
    2. For whom will it make a difference?
    3. What can we possibly contribute to knowledge that is not already known?
    4. If these are all answerable and suggest continuing the study, then go to #3, otherwise, go to #1 and try different DVs.
  3. Form one or two research questions around explaining and predicting these DVs.
    1. Scoping your research questions may also require you to identify your population.
  4. Is there some existing theory that would help explore these research questions?
    1. If so, then how can we adopt it for specifically exploring these research questions?
    2. Does that theory also suggest other variables we are not considering?
  5. What do you think (and what has research said) impacts the DVs we have chosen?
    1. These become IVs.
  6. What is it about these IVs that is causing the effect on the DVs?
    1. These become Mediators.
  7. Do these relationships depend on other factors, such as age, gender, race, religion, industry, organization size and performance, etc.?
    1. These become Moderators
  8. What variables could potentially explain and predict the DVs, but are not directly related to our interests?
    1. These become control variables. These are often some of those moderators like age and gender, or variables in extant literature.
  9. Identify your population.
    1. Do you have access to this population?
    2. Why is this population appropriate to sample in order to answer the research questions?
  10. Based on all of the above, but particularly #4, develop an initial conceptual model involving the IVs, DVs, Mediators, Moderators, and Controls.
    1. If tested, how will this model contribute to research (make us think differently) and practice (make us act differently)?

Order of Operations

Some general guidelines for the order to conduct each procedure

  1. Develop a good theoretical model
    1. See the Ten Steps above
    2. Develop hypotheses to represent your model
  2. Case Screening
    1. Missing data in rows
    2. Unengaged responses
    3. Outliers (on continuous variables)
  3. Variable Screening
    1. Missing data in columns
    2. Skewness & Kurtosis
  4. Exploratory Factor Analysis
    1. Iterate until you arrive at a clean pattern matrix
    2. Adequacy
    3. Convergent validity
    4. Discriminant validity
    5. Reliability
  5. Confirmatory Factor Analysis
    1. Obtain a roughly decent model quickly (cursory model fit, validity)
    2. Do configural and metric invariance tests (if using grouping variable in causal model)
    3. Validity and Reliability check
    4. Common method bias (marker if possible, CLF either way)
    5. Final measurement model fit
    6. Optionally, impute factor scores
  6. Structural Models
    1. Multivariate Assumptions
      1. Outliers and Inflentials
      2. Multicollinearity
    2. Include control variables in all of the following analyses
    3. Mediation
      1. Test indirect effects using bootstrapping
      2. If you have multiple indirect paths from same IV to same DV, use AxB estimand
    4. Interactions
      1. Optionally standardize constituent variables
      2. Compute new product terms
      3. Plot significant interactions
    5. Multigroup Comparisons
      1. Create multiple models
      2. Assign them the proper group data
      3. Test significance of moderation via chi-square difference test
  7. Report findings in a concise table
    1. Ensure global and local tests are met
    2. Include post-hoc power analyses for unsupported direct effects hypotheses
  8. Write paper
    1. See guidelines below

Structuring a Quantitative Paper

Standard outline for quantitative model building/testing paper

  • Title (something catchy and accurate)
  • Abstract (concise – 150-250 words – to explain paper): roughly one sentence each:
    • What is the problem?
    • Why does it matter?
    • How do you address the problem?
    • What did you find?
    • How does this change practice (what people in business do), and how does it change research (existing or future)?
  • Keywords (4-10 keywords that capture the contents of the study)
  • Introduction (2-4 pages)
    • What is the problem and why does it matter? And what have others done to try to address this problem, and why have their efforts been insufficient (i.e., what is the gap in the literature)? (1-2 paragraphs)
    • What is your DV(s) and what is the context you are studying it in? Also briefly define the DV(s). (1-2 paragraphs)
    • One sentence about sample (e.g., "377 undergraduate university students using Excel").
    • How does studying this DV(s) in this context adequately address the problem? (1-2 paragraphs)
    • What existing theory/theories do you leverage, if any, to pursue this study, and why are these appropriate? (1-2 paragraphs)
    • Briefly discuss the primary contributions of this study in general terms without discussing exact findings (i.e., no p-values here).
    • How is the rest of the paper organized? (1 paragraph)
  • Lit review (1-3 pages)
    • Fully define your dependent variable(s) and summarize how it has been studied in existing literature within your broader context (like Information systems, or, Organizations, etc.).
    • If you are basing your model on an existing theory/model, use this next space to explain that theory (1 page) and then explain how you have adapted that theory to your study.
    • If you are not basing your model on an existing theory/model, then use this next space to explain how existing literature in your field has tried to predict your DV(s) or tried to understand related research questions.
    • (Optionally) Explain what other constructs you suspect will help predict your DV(s) and why. Inclusion of a construct should have good logical/theoretical and/or literature support. For example, “we are including construct xyz because the theory we are basing our model on includes xyz.” Or, “we are including construct xyz because the following logic (abc) constrains us to include this variable lest we be careless”. Try to do this without repeating everything you are just going to say in the theory section anyway.
    • (Optionally) Briefly discuss control variables and why they are being included.
  • Theory section (take what space you need, but try to be parsimonious)
    • Briefly summarize your conceptual model and show it with the Hypotheses labeled (if possible).
    • Begin supporting H1 then state H1 formally. Support should include strong causal logic and literature.
    • H2, H3, etc. If you have sub-hypotheses, list them as H1a, H1b, H2a, H2b, etc.
  • Method (keep it as brief as possible)
    • Explanation of study design (e.g., pretest, pilot, and online survey about software usage)
    • Explanation of sample (some descriptive statistics, like demographics, sample size, computer experience, etc.), don`t forget to discuss response rate (number of responses as a percentage of number of people invited to do the study).
    • Mention that IRB exempt status was granted and protocols were followed if applicable.
    • Method for testing hypotheses (e.g., structural equation modeling in AMOS). If you conducted multi-group comparisons, mediation, and/or interaction, explain how you kept them all straight and how you went about analyzing them. For example, if you did mediation, what approach did you take (hopefully bootstrapping)? Were there multiple models tested, or did you keep all the variables in for all analyses? If you did interaction, did you add that in afterward, or was it in from the beginning?
  • Analysis (1-3 pages)
    • Data Screening
    • EFA (report pattern matrix and Cronbach`s alphas in appendix) – mention if items were dropped.
    • CFA (just mention that you did it and bring up any issues you found) – mention any items dropped during CFA. Report model fit for the final measurement model. Supporting material can be placed in the Appendices if necessary.
    • Mention CMB approach and results and actions taken if any (e.g., if you found CMB and had to keep the CLF).
    • Report the correlation matrix, CR and AVE (you can include MSV and ASV if you want), and briefly discuss any issues with validity and reliability – if any.
    • Report whether you used the full latent SEM, or if you imputed factor scores for a path model.
    • Report the final structural model(s) (include R-squares and betas) and the model fit for the model(s).
  • Findings (1-2 pages)
    • Report the results for each hypothesis (supported or not, with evidence).
    • Point out any unsupported or counter-evidence (significant in opposite direction) hypotheses.
    • Provide a table that concisely summarizes your findings.
  • Discussion (2-5 pages)
    • Summarize briefly the study and its intent and findings, focusing mainly on the research question(s) (one paragraph).
    • What insights did we gain from the study that we could not have gained without doing the study?
    • How do these insights change the way practitioners do their work?
    • How do these insights shed light on existing literature and shape future research in this area?
    • What limitations is our study subject to (e.g., surveying students, just survey rather than experiment, statistical limitations like CMB etc.)?
    • What are some opportunities for future research based on the insights of this study?
  • Conclusion (1-2 paragraphs)
    • Summarize the insights gained from this study and how they address existing gaps or problems.
    • Explain the primary contribution of the study.
    • Express your vision for moving forward or how you hope this work will affect the world.
  • References (Please use a reference manager like EndNote)
  • Appendices (Any additional information, like the instrument and measurement model stuff that is necessary for validating or understanding or clarifying content in the main body text.)