Introduction

It’s very common, when debating subjects of a scientific nature, for people to quote a study as “proof” of their point or opinion. If they have a more sophisticated understanding of the science, they may even quote several studies. The conversation usually ends with the person who referenced the studies walking away smugly believing that he or she has proven his or her point and the debate is over.  While it’s great that people are referencing scientific studies, they often do so incorrectly and inappropriately. Unfortunately, I’m not just referring to scientific lay people, but often to actual scientist, doctors, and engineers. The very people who should know better.

The reality is that not all studies are created equal. Some studies provide a great deal of insight, while others provide little of value. Designing a good study usually requires a great deal of practical knowledge about the subject,  allowing the researchers to avoid collecting data that has no real meaning or practical use.  Unfortunately, it is almost a cliché, that many of those doing research often lack the practical experience needed to ask the relevant questions needed to avoid these problems.

How many studies have been conducted on forefoot vs heel-striking? There have been many, and the data have been very ambiguous. Why? Because footstrike is one variable among many in regard to running technique, and it’s not  very meaningful without a great deal more context. Studies on stride-length are another example.  Again, a single variable that is not very meaningful by itself.

Generally, these studies add little to what is already known, and yet they keep coming, with many people quoting these studies believing that they offer valuable insight. The unfortunate result has been that the often heated discussions about running technique have generally not progressed much beyond arguments over foot-strike and stride-length lacking any other context, and missing the larger and more important concepts.

Common Mistakes in the Interpretation of Studies

When lay people (and sometimes actual scientists) attempt to interpret studies, they often make certain assumptions that they shouldn’t.  Here are some important things to keep in mind.

• All studies are not created equal. Studies vary wildly in quality, and no single study is perfect. Before quoting a study it is important to understand the strengths a weakness of its design,  and the questions those strengths and weakness raise.  Many studies are deeply flawed,  and those flaws often undermine the author’s conclusions.  Because of this, no study should be taken simply at face value.  Unfortunately more often than not people do no more than read the conclusion, assuming that everything else about the study was in order.
• Studies rarely, if ever, “prove” anything.  Outside of mathematics, “proof” is actually quite rare.  All studies must be evaluated in the context of all other related studies, and the value of the data must be weighed based on the quality of the study’s design and methodology.  Simply selecting studies that support one’s opinion, or “cherry picking” can be a very misleading. Also, if there are only a limited number of relevant studies, one should not draw hard conclusions.  It is quite common that further research will expose flaws in earlier research.
• Another issue is that many people often assume that the current scientific consensus is definitive.  Science is a process, and that process is slow and error prone.  One should always consider the possibility that new research can completely shift the context of preceding research.

I could go on; the design and interpretation of scientific studies are subjects that could fill many books, but the basic point is that designing meaningful and relevant studies is not a straightforward or obvious process.

Limitations with Reductionist Methodology

In school everyone learns the basics of how science is performed. It goes something like this; researchers are supposed to strictly control all the variables, leaving all but one constant. Methodically and diligently checking every possible combination of the variables in order to fully understand how they interact. Unfortunately, in practice this is not practical or even possible, particularly in the biological and social sciences, and more specifically on studies involving human subjects. This reality has important implications when interpreting the meaning of scientific research.

What I described above is called reductionism.  This approach assumes that by studying the individual variables, it is possible to understand how they all work together.  Often, though we don’t even know what all of the significant variables are.  This opens up the concept of ‘Confounding Variables’.  If a variable is unknown, then it is unlikely to be controlled, and the resulting data are likely to be ambiguous.  There are statistical methods to help analyze and make sense of such data, despite the presence of confounding variables. However, statistical studies can rarely be used to uncover causal relationships. That is to say, they do not determine cause and effect directly.  Usually they simply lead researchers to likely possibilities for further research.

Researchers lacking much practical knowledge of a subject have no choice but to use the reductionist approach. Basically they are forced to thrash around hoping to uncover meaningful data. In new areas of study, there may not be any other choice. However after data has accumulated, and the pieces of the puzzle are starting fall into place, there are alternative methods that often lead to deeper and more meaning insights.

 

Analysis of the running stride within the Pose Method concept.

Alignment of Variables in Systems

Reductionist methodology has an important place as the most fundamental form of scientific investigation to be sure. However, in practice, it is slow, cumbersome and impractical. Firstly, it is almost never possible to control every relevant variable within a study, because people have so many differences between them, such as genetics, diets, dimensions, life experiences and so forth. It is simply not possible, or even ethical, to attempt to control for everything. Secondly, even if it was possible to control for all of the important variables, it is not always possible to identify them all. The result is that researchers are forced to use statistical tools and methods, which, as I already stated, generally don’t uncover the underlying mechanisms of cause and effect.

So what happens if clear differences in results can only be identified when the variables align in specific ways? This is common in many types of systems, where  ideal efficiency or effectiveness is only achieved when all of the variables are “just right”.  In theory, it is possible to uncover this “alignment of variables” using a purely reductionist methodology, but in practice it’s neither practical nor likely. For the sake of expediency researchers are forced to make educated guesses in order to eliminate as many combinations of variables as possible.  However, to do this they must have some underlying concept or model of the system, and how the variables relate to each other.  The branch of study called “Systems Science” (also referred to as Cybernetics) addresses this.  Systems Science recognizes that often the relationships between the variables can be as important, or even more important, than variations in specific variables. This method of analysis based on relationships is referred to a “synthesis”. The reductionist approach ultimately must use synthesis to place the variables in some kind of meaningful context. If the context is too limited, the bigger picture is missed and the results can be misleading. This is also one reason why so many studies seem to contradict each other. They lack proper context when analyzed.

By evaluating a runner as a system, interestingly, it places many arguments about running and running technique in a context that can help explain some perplexing observations.  One key element of a system is resiliency.  Systems must often continue working well, or at least well enough, despite less than ideal conditions. The reason for this is the need for adaptability. If a system is too specialized, and only works well under ideal conditions,  then it is of limited value. This would explain why so many runners are able to run well despite having less than ideal technique.  However,  it is important to understand that although they may be running well, they are not achieving their full potential.

Another way to frame this discussion is as “Conceptual Science” versus “Descriptive Science”.  The concept of conceptual science is akin to systems science, and the concept of descriptive science is more similar to the reductionist approach. However one chooses to frame the discussion, these concepts are not mutually exclusive. They are interdependent methods when studying complex subjects with many relevant variables. One method is more concerned with the specific variables, and the other is more concerned with the relationships between them. The problem with a lot of research is that rather than studying the relationships between the variables, many researchers only focus on specific variables failing to advance our understanding of the subject significantly.

Credits

I would like to credit Ivan Rivera Bours whose blog runninginsystems.com introduced me to the idea of applying systems science to running. I would also like to thank Ivan for his invaluable assistance and feedback in the writing of this article.