To some extent the difference in the results has to do with the difference in the aov tests and the lsmeans tests.  For example, if you had a more limited model (effect ~ treatment + treatment:time ), the lsmeans would come out the same.  But the F test for the interaction effect in the aov would come out significant, because the sum of squares is partitioned differently.

I think in any case, if you say, "I am going to conduct an aov in this way, and if the p-value is >= 0.05, I will consider it non-significant.", then that's really what you have to do.  Normally, you wouldn't conduct a post-hoc test (lsmeans) after a non-significant result anyway.

Thinking about reality, with a p-value of 0.06 for the interaction effect, it would be difficult to conclude that there was or wasn't an interaction, because the p-value is close to 0.05.  Looking at a plot of the data, it's not clear if there is an interaction effect or not:  The pattern from week 1 to week 2 to week 3 for each treatment is basically the same, though not exactly.

As a final note, with the repeated measures nature of the data, is there a better model to use?

I'll just add as an observation, that if you fit your model with maximum likelihood methods (gls in nlme or glm with "gaussian" family), the p-values for the effects are lower.

@Salvatore S. Mangiafico

Sir,

the dataset contains the effect of water samples collected from three stations (treatment- with 3 replicates in each station) for a duration of 3 weeks.

1) Would it be valid to use a repeated measures ANOVA with this data?.

2) Can we use "gls" in nlme or glm for continuous response?

Would it be valid to use something like " m1

1.

Yes, you could use a repeated measures design.  The advantage to this is that it can account for the effect of the station.  For example, if one station was always high and another station always was low. 

The simplest approach is to add Station as a random blocking variable in the model.  This would be a mixed-effects model.  This can be done using lme4 or nlme. ( http://rcompanion.org/handbook/I_07.html ).

A somewhat different approach is here:

http://rcompanion.org/handbook/I_09.html

If you go with a repeated measures design, I recommend using lme4 or nlme rather than the older anova-based methods (that can be done with aov by specifying the error, or using the car package).

The disadvantage here is that your readers or reviewers may be more familiar with anova than mixed-effects models.

2. Yes, both gls and glm fit models by maximum likelihood estimation methods.  gls assumes continuous dependent variable.  glm with a "gaussian" family assumes a continuous dependent variable, with the usual normality assumption I think.  MLE methods have fewer assumptions about the data than the ordinary least squares (OLS) approach of fitting models (anova, lm).  In my opinion, since we have easy access to powerful computers, we should probably be using MLE by default.  The following link has some good explanations by Jochen.

https://www.researchgate.net/post/Does_the_estimation_method_in_ANOVA_regression_influence_the_F-ratios

Again, the disadvantage is that readers and reviewers may be less familiar with ML and REML fitting than OLS.

Yes, that is the correct gls model.

3.  Yes, you could use a White-adjusted anova.

4. Of course, you should _not_ be trying all these methods and choosing the method that gives you the result you want.  Really, any of the approaches would be appropriate.  Since you you do have a repeated measures design, you should use something appropriate for this. 

@Salvatore S. Mangiafico

Sir,

I want to include a third factor-test duration (2 parallel tests for a duration of 1 day and 2 day performed on the samples taken on each week) so that there are three factors - time (week 1,2 and 3), test duration (day1 and day2- sub samples of each week), treatments (station 1, 2 and 3).

I already know that "treatment (sites)" within-subject factor.

1) However, I find it difficult to identify which among other two ("time" and "test-duration") is a within-subject factor. Are they both fixed factors? Can you give me a suggestion regarding this?

2) I also noticed that all the tutorials on lme and gls use datasets with responses without decimals (like count data). Is this a requirement for the lme and gls functions so that I should round off the decimals?

1) I think both Time and Test-duration are probably fixed effects.  You want to know the actual effect of Week1 vs. Week 2.  Likewise, you want to know the actual effect of a 1-day sample and a 2-day sample.  On the other hand, Station is can be a random effect:  no one cares if Station 1 reads lower than Station 2, just that you took the difference in to account statistically.

2) No, both lme and gls would normally be used with continuous data.  Count data wouldn't normally be fit with gls or lme.

0) I'm not sure how to specify the model with test-duration added.  Maybe the effect would be nested in Time?

@Salvatore S. Mangiafico

Sir,

Actually, I also want to study the difference between stations as well (which would be the main focal point), because some stations might be polluted. I wanted to see if the response (effect) changes with regard to sites, season, test- duration, and their interaction. Can I still use the models you have suggested?.

Yes, you can still use the fitting methods I suggested.  I suspect you will end up wanting to use a mixed model (packages nlme or lme4).

The only difficultly is specifying the model that will accurately model the effects in the way that you want  (that is, what is most appropriate for the design of the experiment).

Some questions are: 1) Should an effect be fixed or random.  2) Should the effects be crossed or nested?

The way I usually think of fixed and mixed effect is this (and I could be wrong).  If you are interested in the effect of specific levels of the variable, it's a fixed effect.  For example, applied treatments are fixed effects.  I want to know the effect of this treatment vs. this treatment.  Random effects are taken into account statistically in the model, but I don't really care what the effect of different levels are.  For example, in agriculture, this field and that field may have different crop yields, but no one else in the world cares about the crop yield in this specific field.  That's not the research question. The only important thing is to take account that different fields might be different.

@Salvatore S. Mangiafico

Sir, 

To make the things comprehensible, I have attached an image depicting the exp design. Please have a look at it.

Does the model given below make sense if I follow the design I have given here?

##(sites= treatment; week = time; testduration = to be included in the dataset)

## There are 3 replicates from each site

m1

I'm not sure how to use the weights=varIdent option.

It does sound like you want to treat all factors as fixed.

I'm not sure if you want all factors crossed if some should be nested.

Sorry, I'm not too helpful at this point.

@Salvatore S. Mangiafico · 

Sir, 

Thank you for sharing your valuable time . I think I need to learn more in this aspect. I will post it here when I find a solution.

Hi, @Syamkumar,

I agree with @Salvatore - you ought to use repeated measures aov.

In my opinion the 3 stations and the 2 times (2 and 4 days) are 2 fixed factors. as you defined and controlled them. I´m not familiar with r, though, and cannot help you with the code/packages.

kind regards

Irene

@Irene Zweimüller

Madam,

Actually, there is a third factor- season, that (which I wanted to include later and not shown in the original dataset) probably you might not have noticed. Please go through the last image I have uploaded.

Hi, @Syamkumar,

Hmmm.. well - this is sort of a philosophical question. Do you follow Heraklit? If yes, then your samples are no repeated measurements and season is your third factor.

Without knowing more about your data, it is possible to have a nonsignificant overall test but show significance in main effects or simple main effects.

Check power and effect sizes.

Marcie Zinn