Reflections on User Research Smoke & Mirrors article

The five-part article on "User Research Smoke & Mirror" debates on the usefulness of scientific research to lead the design process.

In the first part of the article the author argues that the scientific approach to design process is idealistic and can hurt the practice of design. Since individuals are subjective, putting quantitative results to the subjective studies and misinterpreting these results will mislead the design changes. A good expert with deep design experience and good instincts could have rather avoided the design errors by him/herself.

I agree with the author to an extent. On one hand, I do agree that it is not 100% possible to put assign a number to user's emotion. All scientific research use preset questions that do not allow flexibility in user's feedback. If you interview 10 people, you will get 10 different answers with variations in intensity. However, scientific research constrains what the respondents can choose within a specified scale. On the other hand, I do think that scientific research, while not a good tool as a foundation of design process, could still be helpful to give general trend how a user feel about the design. Since expert designers are also individual with subjective thoughts, sometimes they could be so engrossed in their designs that they think they are perfect. The importance lies in where the designer has to draw a line analytically when to follow the research findings or when not to.

The second part of the article mainly talks about the usefulness of eye tracker to help with website designs. I do agree with the author that eye tracker only gives information of 'where' the user is looking at but not 'what' and 'why'. While pure scientific measure such as this eye tracker might not be of much help to the design process, a combination of the scientific measurements and qualitative measurement such as "think aloud" technique will be a powerful tool to have insights into user's reactions to the design.

The third part of the article reveals the industrial truth surrouding the research. The author explains why we need to conduct research methodologies for websites even though the flaws of a website is so distinct towards an expert. I would agree that it is helpful to use research data as a tool to justify the non-user stakeholders or big bosses of the design proposal because they would prefer to see a concrete figure rather than listening to the designer's explanation. However, I also feel that it is a total waste of resources just to cook up a figure for the justification.

The fourth part talks about how the professional researchers tend to make up the environments for the users to be in. Therefore, their date become more artificial than actual. However, their services are still good not from the research methodologies but from the expertise and experience that those professionals themselves. If this is the case, then why do we need the research after all? In this part, I think the examples that the author showed are too extreme to one end. I believe there are good research that are done in the controlled environments as well.

The last part does state an need for non-scientific research to guide the design process. Again this is very subjective. Just as I mention above, the designers have to be careful of whether the feedback should be incorporated into the design process or not, from their careful analysis of the feedback and situations themselves.

My overall reflection on the article is that some parts are well-argued, such as "design as politics", explaining why there is a need to conduct the quantitative research. However, a thing to be careful about is that not every single feedback from the qualitative research should be incorporated into the design process.