Design Experiments - the new way of studying educational environments?

Recently I read an ariticle by Tabak & Baumgartner: "The teacher as a partner: Exploring participant structures, symmetry, and identity work on scaffolding." I was excited to read about it, and hear this new approach to teaching: teacher as a partner. It was published in a very reputable publication (Cognition and Instruction) so I knew the results of the study must have been significant and relevant. In the methods section, the authors describe the huge set-up for the experiment. There were three schools involved all around the United States. A computer software was developed for the students to learn first hand about natural selection in the wild, and more specifically the "structure-function reasoning" (i.e. how a structure on an animal/plant determines or constrains funtion). Students were placed in groups of three around a computer looking at the first hand data of finches on the galapagos islands. Researchers listened in, observed for 8 whole weeks of this activity. Very cool.

And then when I reached the results section of the article, I was stunned. The authors only recapped converstaions of one group in one of the schools. In fact, they said that this teacher (and this situation) was the only one that took on the role of a partner-teacher, instead of a mentor or monitor. The first conversation didn't reveal any "partner" teachering. The second episode had a very small amount of symmetry involved. And only in the third episode was the teacher really acting like a "partner-teacher". (Note: A partner teacher is a teacher that acts like a student engaging in the inquiry process as much as the students, acting like a partner, not a superior entity.) In the first and second converstaion there was no evidence that the students really understood the "structure-function reasoning". However, only two of the three members were taking part in the talking. In the third conversation a few weeks down the road, a third participant decided to participate, and it seemed he had the "structure-function reasoning" mastered. According to the authors, this was due to the teacher taking on a role of "Partner".

I couldn't believe what I was reading! Given the authors had three schools with many teachers and many students, they chose to focus on this one group, this one conversation? And then they concluded (although at the end of the article, they said that they are only specualting, not concluding) that this partner-teacher structure has everything to do with the third student mastering scientific reasoning. Actually there is no evidence that this third student didn't actually have this reasoning in place before the activity began!

Here is what I didn't understand:

How can this huge set-up (three schools, eight weeks of study, special computer software created) lead to such an insignificant find?
***As a teacher I have these kinds of relevations all the time, yet I don't get them published in an education journal.
How can this seem to be relevant enough for a reputable publication to publish it?
Why didn't the authors give more examples, more data supporting their claim?

Then I learned about DESIGN EXPERIMENTS. This was a design experiment! What a relevation.

According to Collins, Joseph and Bielaczyc :

The term “design experiments” was introduced in 1992, in papers by Ann Brown (1992) and Allan
Collins (1992). More recently the term design research has been applied to this kind of research.
Design experiments were developed as a way to carry out formative research to test and refine
educational designs based on principles derived from prior research.

I understand it to be a grand design of teaching something specificin an innovative way in a classroom or a school or even in many schools. Then the design is reanalysed on an ongoing basis to see if it's working and whether things need to be fixed or improved upon. The procedure, although set up from the start, is flexible and adjustable in the implementation of the design. Design experiments are "messy". They usually don't have set independent variables, dependent variables, and controlled variables like in a scientific experiment. Instead, the independent variables are characterized initially, there is a slew of dependent variables and there is no way one can talk about controlling anything. Another important factor in design experiments is that the researcher is not just an observer, but more like a co-participant, possibly acting as a teacher in some instances.

The design experiment is like a nightmare for a traditional scientist (or someone that thinks like one). There is no control, no isolation in what affects what. One cannot say that "changing this variable influenced the change in that variable". I am getting the feeling, that hypothesis testing is not at all the purpose of design experiments. I think that design experiments are made to propose theories, get new insites into learning. They were not created as a substitute to traditional experimentation. I think they were a response to the challenge of all educational research:

"How do we see learning happening in the REAL setting of a classroom?"
"How can we see students learn in context, without making it artificial?"

There are way too many factors in a classroom setting to be able to account for them in a standard lab/experimental design. I see it as the butterfly effect (or Chaos Theory). There needed to be a different methodology of studying classroom activities and students learning in a classroom - and design experiments is what came forth to tackle this challenge.

I am starting to come around to this methodology. I see that there is a need for it. But as a traditionalist scientist, I am very skeptical of the results. I see it as being a way to get insights into how kids learn in a real situation. I see it as a way of researchers to have similar experiences as teachers, so they understand what it means to teach a student, how this really works. What teachers do on a regular day might not be easy to see / observe, so this is an official way to do that. But I don't think that these insights can be "generalized". I don't think we should see them as "theories" or "laws". They might be suggestions, ideas, but not true recipies for teaching / learning (not that there ever can be recipies for teaching).

"Design experiments" in educational research is to the "learning sciences" as "engineering" is to "physics". I know that pure physics research can seem very useless, and that in engineering there is real application, real construction. But without the "physics" the engineers would be nowhere. So in the same way, even though design experiments are very "real" and very practical to the classroom environment, I don't think there is a place for them unless they use tested principles or theories, ones that can be generalized, ones that are tested in isolation, in a laboratory setting. Therefore, I think design experiments complement the analytical sciences.

Finally, I still have reservations to this new learning science methodology. Here are some questions / reservations:

When doing a Design Experiment, how do we select data to present? Since we don't relay all the data, only about 5%, can't we just "lie" to prove our point?
Without only trying to make a case for "our theory", how do we choose what to report back to the scientific community?
If we can change the procedure mid way, how is it that we know we aren't trying to manipulate the experiment just to support our own theory?
A theory might be false, yet we try and try again, until we have a small amount of evidence for it, with our constant manipulation of our procedure. How is that reasonable? Won't it lead to supporting a theory that is wrong?
Sometimes the theories / ideas are not generalizable or repeatable entirely. What's the point of reporting them then?

I see Design Research as a way of getting a glimpse into the black box of learning in context. But that's it. I think teachers get a glimpse of this kind every single day of their teaching career. However, they don't have authority to report it, and possibly don't know how or what to report. I might not even have a real understanding of Design Experimentation - all I know is that it's a mind shifter, not real science, and definitely the wave of the future when it comes to education research.