Executive Summary *

1. Introduction *

2. Definition of Empirical Research *

3. Impediments to Empirical Research in Software Engineering *

4. Lessons from Observation of Successful Organizational Structures *

5. Keys to Success in Collaborative Research *

6. How Academics can Improve their Chances of Success in Empirical Work *

7. Fundable Projects *

8. How People in Industry can Encourage Empirical Research *

9. Keys to Encouraging Fruitful Interaction at Technical Meetings *

10. How Funding Agencies can Encourage Empirical Work *

11. Suggested Changes in Infrastructure *

12. Technology Transfer *

13. Action Items *

14. Suggested Reading *

15. Bibliography *

APPENDIX A: Workshop Attendees *

APPENDIX B: A Priori List of Questions for Discussion *

APPENDIX C: Issues and Positions Submitted by Participants Prior to Workshop *

This is a report of a workshop held in June 1998 to discuss the issue of empirical research in software engineering. The Universities of Virginia and Maryland organized the workshop, and it was funded by the National Science Foundation (NSF). The workshop was attended by representatives from academia, funding agencies, and industrial software development organizations.

Empirical research in software engineering is the observation of some aspect of software development in an experimental sense. The observation might be of an existing activity employing accepted techniques or of the application of new techniques. Clearly, this type of research is best performed on real development projects with professional developers although much of value has been learned by doing experiments with student developers. To ensure adequate input from professional developers, several representatives from industry attended the workshop who both understand research and have experience with development.

Empirical research is important to the software engineering field because the results of such research both help to characterize the technical problems with which the field is concerned and evaluate new techniques in a relevant context. In the view of the workshop organizers, insufficient empirical research in software engineering is being conducted despite the need and commendable efforts by funding agencies such as the NSF. The reason for holding this workshop was to review the situation and seek ways of enabling more empirical research.

The conclusions of the workshop are quite detailed. Many of the conclusions are suggestions to researchers of ways to make empirical research more successful. There are also ideas for industrial organizations and funding agencies. Beyond the detailed conclusions, several general and important observations came from the workshop discussions. Specifically:

1. There is a significant need for empirical research in software engineering. Many important issues faced by the community can only be addressed by experimentation.
2. Experimental work is complex and expensive to perform.
3. Although there are exceptions, in many cases industry does not perceive a significant benefit from working with academic researchers in joint activities of an experimental nature.

Of these three observations, the third is the most serious and is at the heart of the issue. The representatives from industry at the workshop expressed doubt that the return to industry from an investment in joint research with academics would be worth the cost. To achieve success in this area, it is essential that this state of affairs be changed. Industrial developers should view the work of academics as valuable, and something to be promoted and sought after.

Many changes will have to be made if empirical research in software engineering is to be more extensive, more significant, and more successful. All three parties that are involved (industry, academic researchers, and funding agencies) will need to make changes, and they will have to coordinate these changes so as to ensure that all the concerns of the stakeholders are addressed.

1. Introduction

The software industry is facing significant problems in software development that can be addressed and resolved, at least in part, by careful research. However, there seems to be a disconnect between the industrial and research communities. Practitioners are, in many cases, not especially interested in the work that is being done by academic researchers. Even the most successful research projects in universities are not always embraced by industry. Many academic researchers in software engineering think that more extensive empirical research--research in which the problems faced by developers were studied carefully by academic researchers and new techniques evaluated in production circumstances--would be valuable. In practice, opportunities for such research are severely limited.

This is the report on a workshop that was organized to discuss the issue of empirical research in software engineering. The workshop, funded by the National Science Foundation and organized by the University of Virginia and the University of Maryland, took place at the Marriott Hotel in Greenbelt, Maryland, June 29-30, 1998. The workshop participants are listed in Appendix A.

The goal of the workshop was to look for new and innovative ways to achieve more extensive empirical research in software engineering. Ideas leading to research approaches which include more industrial participation and artifacts were sought. A list of specific questions to be addressed by the workshop was made available before the workshop via the World Wide Web. Appendix B lists these questions. Attendees were asked to submit prior to the workshop a set of up to ten questions or comments relating to the workshop theme. The questions and comments that were submitted appear in Appendix C.

The first day of the workshop was devoted to focused discussions of the workshop issues. Participants separated into two smaller discussion groups and then came together to present and discuss the ideas generated by the discussion groups. The second day was used to synthesize the ideas that had been discussed and to develop a list of action items.

Beyond the detailed conclusions, several general and important observations came from the workshop discussions. Specifically:

1. There is a significant need for empirical research in software engineering. Many important issues faced by the community can only be addressed by experimentation.
2. Experimental work is complex and expensive to perform.
3. Although there are exceptions, in many cases industry does not perceive a significant benefit from working with academic researchers in joint activities of an experimental nature.

Of these three observations, the third is the most serious and is at the heart of the issue. The representatives from industry at the workshop expressed doubt that the return to industry from an investment in joint research with academics would be worth the cost. Perhaps this observation was known by some and suspected by others, but it documents a very serious state of affairs. If practitioners do not see an overwhelming benefit from working with the research community, then the value of ongoing research must be questioned since it is by the current output that research is judged.

To achieve success in this area, it is essential that this state of affairs be changed. Industrial developers should view the work of academics as valuable, and something to be promoted and sought after.

The remainder of this report details the ideas that were generated and discussed at the workshop. The organizers of the workshop are extremely grateful to the participants for their contributions and to the National Science Foundation for its support. The ideas set forth in this report are those of the participants. Any errors in reporting those ideas in this report are entirely the responsibility of the authors of the report.

1. Definition of Empirical Research

Empirical research can be defined as analysis based on the observation of actual practice for the purpose of discovering the unknown or testing a hypothesis. It involves an investigator gathering data and performing analysis to determine the meaning of the data, and encompasses the following research strategies:

• The experiment, in which the researcher has control over some of the conditions in which the study takes place and manipulates independent factors to elicit dependent factors responses. The purpose of the experiment may be to test a hypothesis, illustrate a known law, or attempt to discover an unknown effect or law.
• The observation of a single data point, including anecdotal and case studies that investigate real-life phenomena in the context of a current theory.
• A demonstration of technology.

An analysis of the state of the art in empirical software engineering entitled "The Role of Experimentation in Software Engineering: Past, Current, and Future" was presented by V.R. Basili in a keynote address at the 18^th International Conference on Software Engineering, Berlin, Germany, 1996 (see section 14).

1. Impediments to Empirical Research in Software Engineering

Industry and academia are not monolithic entities. There are different problems and motivations, so impediments vary from one situation to another. Also, some impediments may separate developers from researchers within the same organization as well as from external researchers.

The following is a list of impediments that were identified as factors that inhibit empirical research in software engineering:

• Proprietary materials.

• Time scales

• The emphasis in industry is on short-term results, but research seldom gives rise to short-term benefits; it is a strategic rather than a tactical investment.
• The emphasis in industry is on time to market, leading to chronic schedule and budget crunches in development organizations. Research impedes progress and is somewhat invasive, tending to increase rather than decrease time to market, at least in the short term.

• Perceived research benefits

• The benefits of proposed research often are not quantifiable and may not even be known in advance.
• A spectrum of people in industry must be convinced that a project is worthwhile before it is undertaken.
• Developers and researchers have different views of what constitutes results. Industry’s big concern is to find ways to make engineers more productive. They are not interested in publication, collaboration for its own sake, or promotion of the common good.
• Industry appears to have lost the idea of research as a pipeline. Funding is pulled from the beginning of the pipeline, and the effects are not noticed for awhile.

• Reward structure

• Problem nature

• Developers are smart people, so the easy problems have already been solved.
• Many of industry’s real problems are not entirely technical; they have a large management component, making them less attractive to many researchers.

• Student involvement

• Culture

• Industrial developers sometimes have an internal bias against academic research.
• The prevalent attitudes of computer scientists in academia may inhibit effective collaboration with developers.

1. Lessons from Observation of Successful Organizational Structures

Two examples of successful long-term organizations for collaborative research in software engineering were considered:

• GSFC/UMd/CSC Software Engineering Laboratory

• NRL SCR/A7

The following organizational attributes appear to have been important to the success of these collaborative efforts:

• The development organization should recognize the need for research support.
• The research model should span from basic to applied.
• The reward structure should include publication, application, and technology transfer.
• The organization infrastructure and commitment to the problem are both important, although to some extent either of these can compensate for a failure of the other.
• Because results are slow and early efforts usually result only in understanding the current process and identifying problems:

• Key researchers should have tenure or other job security.
• All involved must be patient.
• Those funding the research should take a long-term view.

Observations from the study of these organizations also led to insights about the keys to success in collaborative work. These insights are incorporated in the next section.

1. Keys to Success in Collaborative Research

A number of factors were identified that appear to contribute to the success of collaborative research, based on observations of successful collaborations. These include:

• The development of working relationships between researchers and developers:

• It is important to recognize that people develop working relationships, not organizations.
• A pairwise paradigm, one research group working with one development group, seems to work best.
• Researchers must be an integral part of the development team.

• The project must have management support to survive crises; otherwise funding disappears.
• There are several attributes of the project that are important to success.

• There should be a clearly defined objective at the outset.
• The potential benefits of the project should be defined and clearly communicated.
• There should be a clear vision of the course of the research.
• It is important to have intermediate products. This helps to bridge the gap in development/research time scales and gives an opportunity for assessment of progress.

• During the course of the project, some principles should be applied to keep a focus on distant objectives.

• Keep a history of all project decisions.
• Don’t change the problem when an approach doesn’t work; change the approach.
• There should be constant self-criticism and course correction.

• There are some important characteristics that should apply to personnel involved in the project.

• Quality and continuity are vital, so some professional researchers with a long-term commitment must be involved. Students can make only a limited contribution because of the extended timeframe.
• All parties must be committed and motivated.
• All parties must be willing to compromise

1. How Academics can Improve their Chances of Success in Empirical Work

There are number of things that academics who aspire to do good empirical work in software engineering can do to increase their probability of success. Advice for these academics includes:

• Cultivate a long-term relationship with an industrial partner.

• Take every opportunity to meet and talk with people from industry. Go to meetings and listen.
• Learn about the industry (or industries) in which you are interested.
• Find and get to know the "technical gatekeepers" who can make investment decisions and control enough resources to start a collaborative endeavor. The engineers in the organization know who they are. They may be the internal software production researchers, if they exist.
• Remember that you are working with the people in an organization, not just the technology.
• Never dismiss the problems identified by the developers as "non-problems" because this will undermine the research relationship as well as possibly keeping you from recognizing a research opportunity. Successfully addressing a problem that is trivial from a research standpoint may enhance your credibility with the developers. Regard this as an investment in the relationship. (Actual example: The developers’ compilations were taking too long. Researchers helped them to modify their "make" files, earning their gratitude and respect.)
• Don’t perturb project activities to the point of delaying the project.

• Study real problems rather than developing solutions and then searching for problems to which they apply.

• Study the industry and use this knowledge as a take-off point for finding and defining research problems.
• Be open to problems as perceived by the developers.

• Be a salesperson for your project.

• Present "best case" expectations for benefits; emphasize the potential of the proposed project.
• Understand what industry wants, which is to make engineers more productive within about a 2-year time frame. Publications, support for research, and promotion of the common good are desirable side-effects, but are not sufficient motivation for doing the research.
• Tailor your arguments to the audience; you will need support at all levels within the development organization

• It may be easier to make a case for a new source of revenue rather than for a reduction in costs.

• Be open to a wider variety of empirical research opportunities. Software engineering research is not limited to process improvement.

• Consider focusing on a technological approach to opportunity exploration vs. a methodological approach to cost reduction; product innovation is important to industry.
• Appreciate the management as well as the technological aspects of software engineering problems. Research is needed to improve project management skills.
• More work is needed in the "product line" rather than point development area.
• Research is needed on how to help management make better product development decisions, including the ideal time to release a product.
• Work is needed on geographically distributed software development.

• It may help to institutionalize a research problem-solving environment because the organizational structure can keep the research efforts moving forward even as the research problem and vision changes.
• Solicit from industry through their trade publications and help them to understand what "research issues" are.
• Try to get invited to review on-going projects and to attend project post-mortem discussions.
• Timing is important; it may be worthwhile to try again if an initial effort is unsuccessful.
• Find a mentor who has been successful in working with industry.

Academics can also contribute to improving the climate for empirical research by developing and exporting an empirical attitude.

• Introduce empirical methods into the curriculum to cultivate an empirical attitude in students.
• Encourage empiricism in reviewing papers for publication.

• Insist on more empirical content.
• Be open to replication and negative results.

• Educate colleagues about the importance of empirical work in software engineering.

1. Fundable Projects

There are a number of common characteristics of projects that are deemed worthy of funding by industry. These include:

• The research should address a significant problem.
• The researcher must have credibility with the development organization.
• The external researcher’s expertise should complement that of the internal research organization.

1. How People in Industry can Encourage Empirical Research

Successful collaborative research efforts are likely to be beneficial to industry. There are some ways in which people in industry can encourage research within their organizations and enhance their likelihood of forming a successful research partnership with academic researchers.

• Learn how to talk to academics. It may help to talk to people who have recently left the university setting.
• Establish long-term relationships with researchers.
• Calculate the return on investment for a research program on the program as a whole, not on a project-by-project basis.
• Develop a reward structure to encourage cooperation with academics and reward the introduction of new technology.
• Characterize the problem space and communicate needs to researchers.
• Build a reflective, empirical analysis mentality in the organization. Don’t forbid failures. Admit to them and publish them so that you and others can learn from them.
• Talk about your problems with academics.
• Find out what researchers are doing.
• Invite researchers to review current activities and to participate in project post-mortems.
• Don’t give up just because some attempts at relating to academics are unsuccessful.

1. Keys to Encouraging Fruitful Interaction at Technical Meetings

The planners of technical meetings can structure meetings in ways that encourage fruitful interaction between people from industry and people from academia. Helpful strategies include:

• Allow plenty of time for dialog rather than squeezing in as many formal technical presentations as possible.
• Time for socializing should be planned between and after, rather than before, technical presentations.
• Small rather than large meetings are more conducive to successful interaction.

1. How Funding Agencies can Encourage Empirical Work

Government funding agencies can play a significant role in encouraging empirical work in software engineering. The following are suggestions for new funding initiatives that could contribute to the encouragement of empirical research in software engineering:

• Fund students working at industrial sites.
• Develop centralized mechanisms to connect industry with students in internship programs.
• Fund retrospective studies of empirical software engineering organizational successes and failures.
• Support leaders of failed projects in publishing their memoirs.
• Fund development of a systematic collection of case studies with artifacts.
• Fund a workshop with R&D directors from other disciplines to find out if they have the same gulf between researchers and practitioners as exists in software engineering, and if not, how they have bridged the gulf.
• Fund a workshop with GAO auditors, Inspector General’s Office personnel, and others who see a lot of failures and disasters to interact with software engineering researchers.
• It might be useful to fund "bakeoffs," in which a problem is posed and proposed solutions are published.
• Encourage case studies of successful technology transfer.
• Fund replications of experiments and studies and families of experiments.
• Provide funding to support improvements in the infrastructure supporting empirical work. (See next section for suggestions for specific infrastructure changes.)

There are also some adjustments to funding practices that could affect the success of funded empirical projects:

• Provide the continuity and longevity of funding needed for empirical studies.
• It might be helpful to allow extra funding in grants to bring in other researchers. This may allow new researchers to leverage off existing industry/academic relationships.

1. Suggested Changes in Infrastructure

One possible reason for the dearth of empirical work in software engineering is the lack of an infrastructure to support and encourage this work. The following are potential infrastructure improvements:

• There is a need for a forum to serve as an advocacy, educational, and promotional group for empirical research in software engineering.
• It might be useful to form an organization to serve as a broker between development companies whose projects are appropriate research vehicles (a scarce resource) and software engineering researchers.
• Research centers that facilitate rather than control research are needed to:

• Provide focus for external researchers
• Coordinate researchers and developers
• Facilitate regular meetings, data and artifact repositories, etc.

• It might be useful to make project data for real projects available on the World Wide Web. There is a question as to whether this would be useful, because a large component of empirical work is the collection of the appropriate data and usually it is necessary to have involvement in a project to understand the context of the data. Such data might be useful if someone "asks the inspired question" and can use already available data to answer it.

1. Technology Transfer

Although technology transfer was not the central issue of the workshop, it is closely related to the problem of encouraging empirical research. The historical lack of success in technology transfer is an impediment to research, because no one wants to invest resources in developing technologies that will never be used. On the other hand, good empirical research is likely to enhance technology transfer by providing evidence of the effectiveness of new technologies and by highlighting repairable problems with proposed technologies.

The following are barriers to successful technology transfer:

• No one wants to be the first to adopt a new technology:

• There are enormous constraints on development organizations, and the cost of adopting new technology is high. Industry sees risk in the adoption of new techniques and wants to reduce the risk by seeing these techniques work in an industrial environment.
• There are legitimate non-technological issues that affect business decisions to use prevalent technologies despite the availability of a superior technology. For example, there will be better and continuing support for a popular technology.

• There may be problems within the development organization:

• Software developers may have a bias against and resentment of academic research and its practitioners
• The development organization may not be ready for the new technology.
• Bureaucratization and stagnation within the development organization can present problems. Developers may lack interest in the real goal of improving the process, and begin to regard incorporating new technology as a "box to check off."

• There may be problems with the technology itself:

• The new technology may not actually have anything to offer to developers.
• New technology is often superficially easy but fundamentally hard. The syntax is easy, but the fundamental difficulties of getting it right remain.

There are some keys to achieving successful technology transfer:

• New technology needs a champion within the adopting organization.
• Tool support for the new technology is critical.
• The possibility of incremental adoption may significantly reduce the risk for the development organization.
• Successful transfer often requires full contact between researcher and developer.
• Technology transfer sometimes requires ugly compromises.
• Persistence pays off.

1. Action Items

A number of action items were suggested during the workshop and/or identified at the end of the workshop. These include the following:

• Develop a report defining what constitutes empirical research in software engineering and how it should be performed, evaluated, and reviewed. Select and document the body of excellent existing work in the field to serve as an example.
• Size and calibrate the problem with case studies and retrospectives; ask people in other disciplines why they do not seem to have the same dearth of empirical work.
• Develop texts, workshops, and other training in empirical methods with the goal of changing the culture in computer science academic programs.
• Lever off of the Motorola post-mortem meeting. Organize a similar meeting at Lucent, but smaller and with more time for informal discussion; possibly hold a NSF-funded workshop on empirical research in tandem with this meeting.
• Do a case study of researchers who have obtained industry funding to find out how the person-to-person relationships that are the key to success have been developed.
• Collect histories of successful/unsuccessful research organizations to determine what works and what does not.
• Put together a "starter kit" for the new researcher who wants to get started doing collaborative work.

1. Suggested Reading

The following are reading suggestions from workshop participants:

1. Simon, Herbert, Models of My Life, MIT Press, 1996.
2. National Research Council, Academic Careers for Experimental Computer Scientists and Engineers, Committee on Academic Careers for Experimental Computer Scientists, ISBN 0-309-04931-8; 1994.
3. National Research Council, Scaling Up: A Research Agenda for Software Engineering, Computer Science and Technology Board, ISBN 0-309-04131-7; 1989.
4. National Research Council, Statistical Software Engineering, Panel on Statistical Methods in Software Engineering, ISBN 0-309-05344-7; 1996.
5. Kuhn, Thomas The Structure of Scientific Revolutions, University of Chicago Press, Third Edition, 1996.
6. Allen, Thomas, Managing the Flow of Technology: Technology Transfer and the Dissemination of Technological Information Within the R and D Organization, MIT Press, 1977.

1. Bibliography

This is a brief bibliography of papers that address a variety of topics in empirical software engineering. This bibliography is included to provide the reader with some starting points for reviewing the literature in the field. Many papers listed here come from the journal "Empirical Software Engineering". This journal leads the field in presenting results in this area.

1. Avritzer, A.; Weyuker, E. J.; Monitoring smoothly degrading systems for increased dependability. In: Empirical Software Engineering. Vol.2, no.1; 1997; p.59-77.
2. Basili, V. R.; Green, S.; Laitenberger, O.; Shull, F.; Sorumgard, S.; Zelkowitz, M. V.; The empirical investigation of perspective-based reading. In: Empirical Software Engineering. vol.1, no.2; 1996; p.133-64.
3. Bertolino, A.; Marre, M.; Automatic generation of path covers based on the control flow analysis of computer programs. In: IEEE Transactions on Software Engineering. vol.20, no.12; Dec. 1994; p.885-99.
4. Bowdidge, R. W.; Griswold, W. G.; How software engineering tools organize programmer behavior during the task of data encapsulation. In: Empirical Software Engineering. Vol.2, no.3; 1997; p.221-67.
5. Briand, L. C.; Bunse, C.; Daly, J. W.; Differding, C.; An experimental comparison of the maintainability of object-oriented and structured design documents. In: Empirical Software Engineering. Vol.2, no.3; 1997; p.291-312.
6. Briand,L.C.; Daly, J.W.; Wust, J. A unified framework for cohesion measurement in object-oriented systems. In: Empirical Software Engineering. Vol.3, no.1, 1998, p.65-117.
7. Brooks, A.; Meta analysis-a silver bullet-for meta-analysts. In: Empirical Software Engineering. Vol.2, no.4; 1997; p.333-8.
8. Brown, A. W.; Carney, D. J.; Clements, P. C.; Meyers, B. C.; Smith, D. B.; Weiderman, N. H; Wood, W. G.; Assessing the quality of large, software-intensive systems: a case study. In: Software Engineering - ESEC Ô95. 5th European Software Engineering Conference. Proceedings. Springer-Verlag, Berlin, Germany; 1995; p.384-404.
9. Bruegge, B.; Dutoit, A.H.; Communication metrics for software development. In: Proceedings of the 1997 International Conference on Software Engineering, ICSE 97. ACM, New York, NY, USA, 1997, p.271-81.
10. Daly, J.; Brooks, A.; Miller, J.; Roper, M.; Wood, M.; Evaluating inheritance depth on the maintainability of object-oriented software. In: Empirical Software Engineering. vol.1, no.2; 1996; p.109-32.
11. El Emam, K.; Hoeltje, D.; Qualitative analysis of a requirements change process. In: Empirical Software Engineering. vol.2, no.2; 1997; p.143-52.
12. El Emam, K.; Madhavji, N. H.; An instrument for measuring the success of the requirements engineering process in information systems development. In: Empirical Software Engineering. vol.1, no.3; 1996; p.201-40.
13. EuroSTAR '94, the Second European Conference on Software Testing, Analysis and Review. Software Quality Eng, Jacksonville, FL, USA; 1994; 716 pp.
14. Fusaro, P.; Lanubile, F.; Visaggio, G.; A replicated experiment to assess requirements inspection techniques. In: Empirical Software Engineering. Vol.2, no.1; 1997; p.39-57.
15. Graham, D.; Software testing paradoxes, panaceas and engineering. In: Software Quality and Business Opportunities. Fifth European Conference on Software Quality Conference Proceedings. Irish Quality Assoc, Dublin, Ireland; 1996; p.239-45.
16. Jankowski, D.; Computer-aided systems engineering methodology support and its effect on the output of structured analysis. In: Empirical Software Engineering. Vol.2, no.1; 1997; p.11-38.
17. Johnson, P.M.; Tjahjono, D.; Does every inspection really need a meeting? In: Empirical Software Engineering. Vol.3, no.1; 1998; p.9-35.
18. Kiper, J.D.; Auernheimer, B.; Ames, C.K.; Visual depiction of decision statements: what is best for programmers and non-programmers? In: Empirical Software Engineering. Vol.2, no.4; 1997; p.361-79.
19. Koike,H.; Hui-Chu-Chu. How does 3-D visualization work in software engineering?: empirical study of a 3-D version/module visualization system. In: Proceedings of the 1998 International Conference on Software Engineering: Forging New Links. IEEE Comput. Soc, Los Alamitos, CA, USA, 1998, p.516-19.
20. Lott, C. M.; A controlled experiment to evaluate on-line process guidance. In: Empirical Software Engineering. Vol.2, no.3; 1997; p.269-89.
21. Millerl, J.; Daly, J.; Wood, M.; Roper, M.; Brooks, A.; Statistical power and its subcomponents-missing and misunderstood concepts in empirical software engineering research. In: Information and Software Technology. vol.39, no.4; April 1997; p.285-95.
22. Miller-J; Wood-M; Roper, M. Further experiences with scenarios and checklists [software inspection]. In: Empirical Software Engineering. Vol.3, no.1; 1998; p.37-64
23. Morasca, S.; Applying QIP/GQM in a maintenance project. In: Empirical Software Engineering. Vol.2, no.2; 1997; p.163-6.
24. Munson, J. C.; Hall, G. A.; Estimating test effectiveness with dynamic complexity measurement. In: Empirical Software Engineering. vol.1, no.3; 1996; p.279-305.
25. Ohlsson, N.; Eriksson, A. C.; Helander, M.; Early risk-management by identification of fault-prone modules. In: Empirical Software Engineering. Vol.2, no.2; 1997; p.166-73.
26. Pitschinetz, R.; Wegener, J.; TESSY-management of software tests. In: Experience with the Management of Software Projects 1995 (MSPS'95). A Proceedings Volume from the 5th IFAC/IFIP/GI/GMA Workshop. Pergamon, Oxford, UK; 1996; p.11-16.
27. Rosenblum, D.S.; Weyuker, E. J.; Lessons learned from a regression testing case study. In: Empirical Software Engineering. Vol.2, no.2; 1997; p.188-91.
28. Rothermel, G; Harrold, M. J.; Experience with regression test selection. In: Empirical Software Engineering. Vol.2, no.2; 1997; p.178-88.
29. Schneidewind, N.; NASA shuttle software maintenance evolution. In: Empirical Software Engineering. Vol.2, no.2; 1997; p.192-6.
30. Seaman, C. B.; Basili, V. R.; The study of software maintenance organizations and processes. In: Empirical Software Engineering. Vol.2, no.2; 1997; p.197-201.
31. Silverman, B.G.; Mehzer, T.; A study of strategies for computerized critiquing of programmers. In: Empirical Software Engineering. Vol.2, no.4; 1997; p.339-59.
32. Sova, D. W.; Smidts, C.; Increasing testing productivity and software quality: a comparison of software testing methodologies within NASA. In: Empirical Software Engineering. vol.1, no.2; 1996; p.165-88.
33. Tryggeseth, E.; Report from an experiment: impact of documentation on maintenance. In: Empirical Software Engineering. Vol.2, no.2; 1997; p.201-7.
34. Von Mayrhauser, A.; Vans, A. M.; On increasing our knowledge of large-scale software comprehension. In: Empirical Software Engineering. Vol.2, no.2; 1997; p.159-63.
35. Zuse, H.; Comments to the paper: Briand, Eman, Morasca: on the application of measurement theory in software engineering. In: Empirical Software Engineering. Vol.2, no.3; 1997; p.313-16.

William W. Agresti

Frank A. Anger

Victor R. Basili

Susan S. Brilliant

Betty H. C. Cheng

Richard A. DeMillo

Michael Deutsch

Michael Evangelist

Stuart R. Faulk

Warren Harrison

Mary Jean Harrold

Charles C. Howell

John Knight

Nancy Leveson

Stephen P. Miller

Linda Ott

Colin Potts

Kevin Sullivan

Lawrence G. Votta

Allan Willey

Marvin Zelkowitz

Specific questions to be addressed include (but are not limited to):

SUE BRILLIANT

• One thing that might lead to greater research cooperation between academics and industry would be greater contact between these two groups in general. If a researcher already has a working relationship with someone working in industry, he/she is more likely to know about a project with research potential, and is also more likely to be trusted in a cooperative research effort. So the question that arises is, how can closer ties between academics and industry people be forged?
• A practical issue for academics is that their work be viewed as valuable by their peers who evaluate their performance for tenure and promotion purposes. Often theoretical work is viewed as more valid and valuable than empirical work. What can be done to enlighten our peers about the importance of empirical work?

BETTY CHENG

• What are specific characteristics of empirical research? (Should have specific criteria)
• How do we facilitate technology exchange between industry and academia?
• How do you get academic administrators to recognize the value of empirical research?
• How do you obtain funding to build software tools needed to demonstrate "proof of concept"?
• How can we improve tool dissemination, both research and commercial tools? (Related issue: how can we encourage industrial organizations to provide tools at affordable costs, or better yet, free?)
• How can we set up useful benchmarks (and get people to use them) for evaluating different types of software technologies? (Rather than each researcher having their own set of toy, contrived examples to illustrate their specific technique)
• What are good forums for publishing empirical research results (in a timely manner)? (Similar question for tool descriptions and utilities.)

WARREN HARRISON

• How important *is* sharing data/knowledge with each other? Assuming it *is* important, how can we use the World Wide Web to ...

• share objective, "formatted" data?
• share anecdotal, "unformatted" knowledge?
• share "meta knowledge"?

• By training software engineers the same way physicians are trained (i.e., the "university hospital" model) would academic researchers get better access to data (or perhaps access to better data)? What would be necessary to do this?

MARY JEAN HARROLD

• For studies concerned with software, can we establish sets of benchmark programs for various applications. Such benchmarks exist in other fields? For example, the Transaction Processing Performance Council (TPC), an industry consortium, has defined a set of standard database benchmarks, and the Standard Performance Evaluation Corporation (SPEC), a non-profit corporation, has established a set of standard benchmarks that can be used to measure computer performance.
• Could subjects in repositories be made available in a common organization and format so that researchers could easily and quickly use these artifacts in their experiments? It has been my experience that it takes a considerable amount of time to prepare each new subject for experimentation -- a common organization and format for the artifacts could reduce this time.
• Can guidelines be established, or experience of others be recorded, to help researchers prepare for successful experimentation in an industrial setting? After such a collaboration has been arranged, there is considerable risk that it will not be successful. How can we improve our chances for success in these endeavors?
• How can we make experimental work more attractive to academics? There are still to many academics that neither appreciate the time and effort that experimentation involves nor value the results as research.

CHUCK HOWELL

• Opportunities for academic participation in DOD "Red Teams", GAO/IG software project audits, etc.? ("I'm from UVA and I'm here to help" or, "You thought Industry had an adversarial relationship with academics before?"). Red Teams do provide a good way to learn the real problems and issues of software developments in a short period of time.
• Near term opportunities for traction seem better for diagnostics and tangible tools than process enhancements (MRI and treadmill test vs. "wellness guidelines"), even though the long-term ROI is clearly better for building systems correctly than diagnosing where things went wrong. True? Bogus?
• Demand pull vs. technology push: what are key technology needs in Industry?
• Why aren't more developments using currently available technology/techniques that seem to have high ROI (e.g., inspections) - whatever barriers there are to adoption for current "mainstream" technology will be at least that high for more leading edge ones, no?
• Demand pull vs. technology push II, The Sequel: Often the model for empirical study seems to be to confirm/refute a specific hypothesis about the efficacy of a new technique or technology, rather than gathering a sense of what the key open problems are in Industry.
• Is there a role for a "trusted intermediary" to "sanitize" collected artifacts, process instrumentation, etc. before passing on to researchers? The organization (Government agency?) could also act as a "clearinghouse" providing analysis results back to the Industry participant.

STEVE MILLER

• One of the largest obstacles to industrial use of new methods is the lack of commercially available tools and training from established vendors. What will be the role of commercial tool vendors in this effort? What steps can be taken to accelerate the introduction of useful discoveries into commercial tools?
• Methods and tools that do not integrate well with other approaches is another important obstacle to industrial acceptance. For example, traceability tools often work with only a few other tools. What infrastructure can be put in place to ensure that methods and tools developed in this project integrate smoothly?
• One of the more difficult aspects of working with industrial examples is that the problem domain evolves continuously. Once suitable examples are found, how will this be simulated so that researchers can assess the true usefulness of their approach?
• Methods developed by researchers often fail to take into account the importance of cost, schedule, and time to market, with the result that seemingly promising techniques are not accepted by industry. What can be done simulate these challenges while recognizing that methods and tools may only be cost-effective after they've matured?
• Different software domains pose very different challenges. For example, the problems of safety-critical embedded systems are very different than those of information management systems. Assuming finite resources, what domains should be tackled first?
• Most companies actually make the same product over and over again. Yet most methods assume a single product and clean sheet of paper, with the result that much industrial development is ad hoc. Should this project focus on methods for product families? If so, how.
• Certification issues can greatly affect a software development process. To what extent should certification issues be considered in the demonstration projects? Should representatives from the FAA, NRC, FDA, be involved?
• Industry developers respond much better to good examples from their domain rather than those from other domains, e.g., developers of flight control software tend to have only limited interest in methods demonstrated on avionics displays. Should many smaller examples be supported, or only one or two large examples. If the latter, which domains would have the broadest appeal?

LARRY VOTA

Position Statement for Empirical Workshop in Software Engineering

The starting point for many of my ideas for the roles of empirical work and the relationship of Engineering and Science are my personal experiences (as a physicist - 10 years, software developer - 12 years and software engineering researcher - 8 years), Thomas Kuhn's writings on the history of Science (The Structure of Scientific Revolutions[1]), and Thomas Allen's writings (Managing the Flow of Technology[2]) on the roles of engineers and scientist.

My personal experiences illustrate that to develop effective collaborations among researchers (industrial/industrial and industrial/university) and between researchers and engineers there must be common goals, an effective distribution of labor, and positive benefits and equivalent risks and costs for all participants. If any of these areas are missing or out of balance the resulting collaboration has a high probability of failure.

The writings of Thomas Kuhn illustrate the importance of the role of empirical work to any scientific discipline. Although many of his ideas have been recently challenged and several of his ideas replaced, the essence of the intrinsic symbiotic relationship between experimental and theoretical branches of a scientific discipline remain intact.

Finally, from an outsider’s view, the methods, techniques, and tools employed by scientist and engineers appear to be the same -- the reality is that they are used to achieve different goals. Thomas Allen in his book illustrates these differences and offers a simple model to discuss and appreciate the differences between scientist and engineers. His work has led to a simple model of how the three groups (researchers, state of the art engineers and everyday engineers) interact. As the size of these groups grows they become organizations with their own unique cultures which Tom discusses in the last half of his book.

The following observations highlight some of the differences between university research, industrial research, forward looking work, and development and are sources of many failed collaborations. They are interesting points to discuss and think about.

• As you move across the spectrum from university research to industrial research to development, there is a significant difference in expectations about time to complete work from years and months for research to days and weeks for development.
• Research and development have two very different definitions of results. Researchers usually consider papers or talks their "results". Development usually considers products and patents their "results".
• If you examine the population of workers - in a university research group the population split between apprentices and professionals is significantly different than for a development group.
• The perspective of the problem is different between research and development. Research has a tendency to align along academic boundaries; whereas, development boundaries align along needs of customers in a market place. These boundaries are not the same.

References

[1] Thomas Kuhn. "The Structure of Scientific Revolutions", University of Chicago Press, Second Edition, 1962.

[2] Thomas Allen. "Managing the Flow of Technology", MIT Press, 1977.

MARV ZELKOWITZ

• Why does industry adopt techniques that have not proven to be valuable in the laboratory and why are new research results so readily published without confirming experimentation? Or stated another way, why do the research community and industrial community not think it important to talk to one another?
• Is there a difference between experimental software engineering and empirical software engineering? Both terms are used, and I think there is an important difference between the two.
• The perception is that other engineering disciplines "do it right" whereas software engineers just muddle through and eventually get it done. Is this perception realistic? How well does empirical research work in CE, ME, or other engineering fields?