Theme 6: Qualitative and case study research
I selected the paper Robustness Testing of Embedded Software Systems: An Industrial Interview Study by Syed Muhammad Ali Shah, Daniel Sundmark, Birgitta Lindström and Sten F. Andler. It is published in the Journal IEEE Access and has an impact factor 1.270.
The objective of the study was to understand the state of the practice of robustness testing of embedded software systems, and upon the collected data build empirical knowledge.
The designed an exploratory multiple case study with an exploratory research method to answer the research question.
The state-of-the-art knowledge was surveyed to understand the existing knowledge of robustness testing. They identified key areas and interview questions were developed by considering the main aspects of robustness testing found in scientific literature. The interview questions were developed in iterative fashion, exploiting the perceptions, opinions, experiences and beliefs of all the co-authors of the paper. One pilot interview was conducted. Some questions were open-ended to give an opportunity to ask follow-up questions and thereby encouraging exploratory discussions.
They selected interviewees from industrial contacts and from profiles of experts of robustness testing on LinkedIn. Twelve male and one female. They interviewees had experience of testing robustness testing of embedded software systems.
One thing I learnt is that it requires a lot of preparatory work regarding the interview questions.
The benefits of using qualitative methods are typically that they more flexible, they allow greater spontaneity and adaptation of the interaction between the researcher and the study participant. The questions are “open-ended” and not necessarily worded in exactly the same way with each participant. Participants have the opportunity to respond more elaborately and in greater detail.
Instruments use more flexible, iterative style of eliciting and categorizing responses to questions. Use semi-structured methods such as in-depth interviews, focus groups, and participant observation. Seek to explore phenomena.
Limitations in qualitative research, is that only a sample of a population is selected for any given study.
The study’s research objectives and the characteristics of the study population determine which and how many people to select.
It was a rather small sample group, but the research aim to investigate an area of expertise, therefore it must be harder to collect valid candidates.
A threat related to construct validity is that the interview is answered by guessing what the researcher has in mind rather than answering the question. To reduce this threat the open-ended questions, asked the interviewee explicitly to give their answers in terms of describing examples from their fields and made sure to not intervene during their answers.
A case study is a quantitative, qualitative, or combined method which is used to provide a deeper understanding of a certain phenomena or entity. According to “Building Theories from Case Study Research” it’s a research strategy from which data can be analyzed in order for theories to be created.
I choose the The Design Space of Bug Fixes and How Developers Navigate It by Emerson Murphy-Hill, Thomas Zimmermann, Christian Bird, and Nachiappan Nagappan. I is published in the Journal IEEE Transactions on Software Engineering and has an impact factor of 1,516.
The case study in my selected article was rather well executed since it followed the format of Eisenhardt’s table 1 quite accordingly. In the getting started phase, the problem which is to be investigated is defined through an explanation and a presentation of related work. They provide us with a motivation of the study. As for the selecting cases phase, the authors have chosen to go with different methods.
The paper examines alternative fixes to bugs and present an empirical study of how engineers make design choices about how to fix bugs. The case study is based on qualitative interviews with 40 engineers working on a variety of products, data from six bug triage meetings, and a survey filled out by 326 Microsoft engineers and 37 developers from other companies. The authors presents a number of factors, many of them non-technical, that influence how bugs are fixed and discuss implications for research and practice, including how to make bug prediction and localization more accurate.
The four research methods that was used was opportunistic interviews, firehouse interviews, triage meeting observations, a survey, and a replication of that survey. They triangulate the answers to improve their accuracy.
An important limitation is that of generalizability beyond the population that they studied (external validity), which they also stated in the paper.
