This Technical Brief describes objectives, organization, and methods that could be used for conducting peer reviews in public archeology projects. Each of these elements is illustrated with examples drawn from peer reviews conducted by the Department of the Interior's Departmental Consulting Archeologist between 1981 and 1990. The purpose of this Technical Brief is to provide guidance to government agencies and other archeological resources management programs on use of the peer review process as one tool to improve the effectiveness of their activities. It may be an especially important tool to help resolve conflicts which may arise due to the need for interagency cooperation, the complexity of the archeological work, or professional disagreements. This Technical Brief demonstrates how peer reviews can address these typical problems and explains ways the process can be implemented.
The review of research by peers has been one hallmark of science throughout its history. Peer reviews are the conscience of science; they provide a necessary check on its practitioners and an explicit evaluation of their efforts. Other professions have recognized the importance of peer reviews for applied sciences as well. For example, the American Society of Civil Engineers and American Consulting Engineers Council (ACEC) have developed a program that offers peer review services for engineering projects and programs (ACEC 1990 a,b,c,d).
The Departmental Consulting Archeologist (DCA) and his staff in the Archeological Assistance Program are authorized by the Secretary of the Interior to provide leadership and coordination for Federal and other types of public archeology. The Secretary is assigned these roles by several statutes including the Antiquities Act, Historic Sites Act of 1935, National Historic Preservation Act, and Archaeological Resources Protection Act. The peer review of selected Federal archeological projects is one among a number of DCA programs that provide leadership and coordination (Canouts 1992; Canouts and McManamon 1991; Knoll 1991; Knudson and McManamon 1992; McManamon, Smith, and Waldbauer 1990).
This Technical Brief describes the purposes and the goals of the Federal archeology peer review process as conducted by the DCA between 1981 and 1990. The government agencies and the professionals who have been in these reviews recognize that peer reviews can provide an important contribution to project design and management, thereby advancing American archeology. This brief provides an example of the peer review process in the Federal archeological arena, as opposed to the academic domain. It also supplies information about the planning, design, and implementation of peer reviews. Four appendices with examples of documents illustrating efforts at planning and implementing peer reviews are provided.
Purposes and Goals of the Federal Peer Review Process
In an academic research framework, as opposed to typical public archeology projects, the principal investigator develops a research program to investigate problems that are of theoretical, methodological, and/or substantive interest. The researcher develops a research design, identifies and obtains permission to investigate the sites that fit the project's scientific needs, and raises the necessary funds by obtaining a grant. The researcher plans and conducts the work and publishes the results. Review of the scientific merits of the proposed work by his or her peers occurs during the grant application process, in the review of the results by the granting organization, as part of the publication process, and in the reception of the product by the scientific community.
On the other hand, archeology conducted to comply with Federal historic and archeological preservation laws must follow sound archeological practice as well as the regulations, procedures, and guidelines mandated by and developed from those laws. Archeology conducted by these authorities is to benefit the American people generally through the preservation of important archeological resources and information. The first objective of research in public archeology is to identify, evaluate, and record significant archeological resources. The scientific information is used for a variety of practical purposes, including education and creating an appreciation of the nation's heritage among its citizens. The concomitant benefits that individual archeologists may derive from these endeavors with regard to professional standing and career advancement are secondary.
Early in the historic preservation process, the Federal agency identifies archeological properties within the boundaries of an undertaking. This is accomplished through literature reviews, records checks, and field surveys. The significance of the sites, especially for the important archeological and historical data they contain, is determined based upon eligibility for the National Register of Historic Places. Significance commonly is assessed in reference to State comprehensive historic preservation plans or historic contexts used by the State Historic Preservation Officer (SHPO).
If important data will be lost because of the undertaking, the Federal agency must consider this potential and consult with the SHPO to develop ways to mitigate the loss. The agency also may have to get the comments of the Advisory Council on Historic Preservation (ACHP) before moving forward with the undertaking.
Several parameters have been defined for Federal archeological projects that constrain the scientific freedom to develop research topics. The principal difference between an exclusively academic research design and one developed for a public project is that the former develops from scientific curiosity while in the latter scientific research needs are conditioned by the potential data content of the archeological site being impacted by the public undertaking.
These constraints on the opportunity for an individual researcher to develop theoretical, methodological, or substantive aspects of the research design do not mean that Federal projects are devoid of such concerns. On the contrary, the process of developing a research design and a strategy to implement it, completing the work, and reporting are the same general steps of any scientific research project. Also, the development of the archeological project within Federal historic preservation is a consultative process in the hands of Federal archeologists and their counterparts in the SHPO office. Project plans and archeological results also may be reviewed by archeologists and historic preservation professionals of the ACHP. Frequently suggestions made during these consultations result in design improvements. As a result of this coordination, the project has a firm scientific foundation.
Before the passage of the National Historic Preservation Act of 1966 (NHPA), rescue or salvage archeology was handled without input from the States or the ACHP. Archeological rescue work was usually controlled by the time and money available rather than by the development of thoughtful and explicit research designs. The National Park Service (NPS), operating with its own professional staff through the River Basin Survey program of the Smithsonian Institution or with academic contractors, conducted the archeological salvage program of the nation. Private energy developers and State highway departments contracted directly with academic institutions to complete salvage archeology in their construction projects. Competition among academic institutions for government, private utility company, or highway department contracts for archeological work did not exist during this period. Normally the NPS, the utility company, or highway department approached an academic institution and asked the institution to carry out whatever archeology was needed within the time and funding available.
Today Federal agency archeologists or contractors from the public and private sectors conduct Federal archeological projects. Competition among archeological contractors is the norm unless the work is done by agency personnel. It does not matter who does the project for the Federal peer review process to work.
The perception has existed that Federal archeological project research design is less open to evaluation and criticism than an academic research project funded by a granting agency. This is hardly the case. As mentioned, evaluations occur as part of memoranda of agreement, National Register of Historic Places nominations, determinations of effect, reviews of scopes of work for data recovery plans, and requests for proposals that form the bases for contracts. Depending on the specifics of the project, these evaluations are performed by archeologists, other cultural resources specialists, and perhaps officials of various agencies. These reviews represent a broad range of scientific and legal compliance concerns. Frequently the evaluations improve the agency proposal to mitigate the loss of important information, and stipulations for formal peer reviews are often specifically included in memoranda of agreement or agency contracts. This is all done before a project is announced for bidding. As part of the proposal, the winning bidder also prepares a research design or work plan. This provides refinements for conducting the research. It also defines the protocols that will serve as project guides.
The primary goal of the peer review process developed by the DCA is to evaluate the conduct of Federal archeological projects and assess the competence and efficiency of projects relative to archeological practice and legal compliance. An academic peer review rates a project on its scientific merits alone; reviewers examine the theoretical basis for the research, the data needed, and how they contribute to the research goals. They evaluate the methodology used to get needed data and the analytical methodology. They determine if the research was conducted in an appropriate manner. In Federal archeology, the peer review examines additional topics as well as these. Is the agency following the stipulations of the compliance documents agreed to under the Section 106 process of NHPA? Have data fulfilled the needs of the research design? Have unexpected data or research domains come to light that need additional investigation? Does the project need to be changed or redesigned to take these findings into consideration?
During the 1980s, Federal archeology project peer reviews were undertaken when controversy stirred about an undertaking or the archeology associated with it. Some reviews were conducted because of major disagreement between the agency and the contractor about the work. Allegations or questions may have come from the archeological profession, the media, special interest groups, or members of Congress. More recently agencies have requested reviews by the DCA to investigate whether projects were adequate. One review was conducted for an agency that wished to improve a project.
In public archeology the use of the peer review process also serves other purposes:
Philosophy of Peer Reviews for Public Projects
In designing the peer review process the scientific value of an archeological project, as determined by the NHPA Section 106 consultation, is accepted and used. Furthermore, the research goals and priorities and the stated applications of archeological methodology also are accepted and used. The peer review evaluates the project using the following criteria:
This approach accepts previous decisions made by the various consulting parties to mitigate the loss of important archeological information caused by a Federal undertaking. It also provides an occasion to assess the project to determine if any changes are desirable. This approach steers clear of second-guessing earlier decisions or trying to identify all of the approaches for conducting archeological investigations. The manager of the peer review and the agency archeologist establish a project review agenda and develop a list of suggested review topics. These aid the peer review panel in examining archeology defined by the NHPA Section 106 process.
Since 1981, the DCA has conducted six peer reviews (Table 1). The experience from conducting these reviews is the basis for this Technical Brief. These projects were selected for a variety of reasons. The Dolores Reservoir Project was reviewed because of problems between the contractor and the agency about the scope of the investigation and project management. The Central Arizona Project was selected because the Bureau of Reclamation wanted to confirm that the scope and focus of the archeological project were appropriate. The Jackson Lake Dam Project was evaluated to demonstrate to the members of the Wyoming Congressional delegation that the work was adequate and satisfied State interests. The Fish and Wildlife Service sought a peer review to address Native American concerns and to assure that historic preservation requirements were being met in the Stillwater Wildlife Management Area Project. The U.S. Army Corps of Engineers wanted to confirm that its historic preservation responsibilities were being fulfilled in a manner sensitive to Native American religious concerns in the Libby Dam Project. The Soil Conservation Service wanted an evaluation of the Alkali Creek Project in light of SHPO and professional community concerns.
Organizing and Conducting Peer Reviews
A peer review of a project is an activity that must be developed harmoniously as a partnership between the peer review manager and the agency. For all but the Alkali Creek peer review, the DCA served as peer review manager. In the case of Alkali Creek, a senior NPS archeologist well acquainted with the project area was designated by the DCA as peer review manager. Normally the agency representative has been the archeologist responsible for the administration and conduct of the agency's archeological program, although another agency official may serve in this role. The need for frequent and clear communication between the peer review manager and the agency archeologist or other official in charge of the project is mandatory. Good communication is an absolute necessity for a successful peer review. No matter who serves in them, these roles are essential components of the peer review process.
All aspects of the peer review are developed through close coordination between the peer review manager and the agency. Both parties must agree on the scope of the review. Frequent and open communication is necessary in all aspects of the project. The agency archeologist needs information about the progress of the planning effort by the peer review manager to develop agency plans. The peer review manager also discusses the peer review with the SHPO and ACHP to gather information that should be supplied to the team. Frequent communication bolsters the commitment made by both parties to assure that the necessary actions are completed in a timely manner. In undertaking a peer review, one of the peer review manager's goals is to create a partnership with the agency. The creation of such a partnership is the principal ingredient that has made the process, as conducted by the DCA, successful.
Once a project is selected, the following tasks, at a minimum, must be undertaken to conduct the peer review:
The following sections of this Technical Brief will provide information about each of these tasks. The examples cited are drawn from peer reviews conducted in the past by the DCA.
the Scope of the Peer Review
of the Peer Review and Funding
Honoraria or consulting fees have not been paid to members of the DCA peer review panels. Payment of honoraria to non-Federal members of the team is legal, but it is illegal to pay a Federal team member. It seemed inappropriate to compensate some team members but not others. Furthermore, there was concern that if honoraria were paid the peer review team might not feel that it could be critical and candid in its assessment. There was also a potential for the review to appear prejudiced.
The costs of transportation and per diem of the peer review members must be considered. The purchase of airline tickets may take a large portion of the funds available. Other costs include: reproduction and postage to provide project information to the peer review team; on-site equipment, materials, and work space for the team; administrative support; and final report preparation and distribution. Many of these costs can be covered in-kind by the agency or the peer review manager.
and Selection of the Peer Review Team
For later reviews, an archeologist from a public agency, but one different from the one under review, was added to the team. This served the additional goal of increasing the interaction among government archeologists and their academic colleagues. This was a way to improve the credibility of the government archeologists with their academic counterparts. We have found that a balanced peer review team consists of one public archeologist from an agency other than the one responsible for the project, an academic expert in the archeology of the project area or region, and an academic expert from outside the research area.
To identify potential review team members the peer review manager and the agency archeologist independently compile three lists of candidates for: (1) the regional academic expert, (2) the outside academic expert, and (3) the public agency archeologist. They consolidate their lists and decide which candidates and alternates to invite.
Care is to be given in selecting members of the panel. Beyond a candidate's scientific credentials and availability, other elements are considered, for example:
Once the peer review manager and agency archeologist agree on the candidates and the approximate dates for the review, the peer review manager telephones the candidates to determine their willingness and availability. Acceptance of the invitation is followed by a formal letter from the peer review manager providing information about the project, the schedule, and other details.
Documentation for the Peer Review Team
The package of material compiled for the orientation of the peer review team will vary from project to project. Typically it will consist of survey and evaluation reports and NHPA Section 106 documentation, such as the Memorandum of Agreement or Determination of No Adverse Impact, scope of work or request for proposal, and research design. Interim or annual progress project reports, if available, also are provided. Additionally, correspondence and internal documents may be included. (A list of the kinds of materials supplied for the Central Arizona Project and Stillwater National Wildlife Management Area peer reviews is provided in Appendix B.)
the Peer Review
A second consideration in scheduling the peer review is for a time that is acceptable to potential team members. It is useful to identify a 3- to 4-week period for the peer review approximately 3 months in advance of the on-site visit. Within such a time frame it is usually possible to fit the schedules of all personnel involved. Situations that can affect the scheduling of the review must be considered. These include college and university calendars or special events like conventions or athletic contests in the project area. These situations can affect travel arrangements, as lodging may be difficult to find in such circumstances. The schedules and commitments of the individuals representing the various organizations who will be interviewed by the peer review team must be considered. The on-site visit must be scheduled so that those who will travel from outside the project area, the SHPO or appropriate staff, ACHP personnel, and other interested parties, can participate.
The peer review manager, in consultation with the agency archeologist, develops an agenda for the peer review team's on-site visit. A detailed agenda assures that the on-site visit is carefully structured. Each interview focuses on a specific task or topic. Each interview session is allotted a specific amount of time. Sessions usually do not exceed two hours without a break. Maintaining strict adherence to the agenda assures that the peer review team stays focused on the task and adheres to the schedule.
During the first session of the on-site visit the peer review manager, the agency archeologist, and the peer review team discuss the agenda in detail and modify it as necessary. Implementing modifications in the agenda may require complex changes in the interview schedule. It is important to determine the final agenda as early as possible.
The peer review manager and the agency archeologist provide the necessary administrative support to the peer review team. This assures that an efficient review occurs. Meeting facilities, telephones, transportation, and lodging and dining facilities must be available. Secretarial assistance, equipment, and supplies may be needed. The object of taking care of these necessities is to allow the team to devote its energies to the peer review. Personal computers have been very useful. In providing these kinds of administrative support, the peer review manager and agency function as aides to the team.
The first part of the agenda is devoted to briefings by the peer review manager and agency personnel. The peer review manager describes and then emphasizes the specific scope of work of the peer review (see Appendix C for examples). The peer review team can examine any aspects of the project related to the archeological work. The plan for the rest of the on-site visit will depend on the schedules of individuals, the weather, field conditions, and other variables. During the week-long peer reviews conducted by the DCA, personnel interviews and field visits normally were concluded by the end of the third day so the team could review its findings and draft its report. Typically the draft report was completed by the evening of the fourth day. Exit interviews with the DCA, agency personnel, and other interested parties were conducted on the fifth day.
The peer review team normally interviews agency personnel such as the project engineer, environmental staff, and the archeologist. Other historic preservation specialists, the contracting officer, or contracting officer's technical representative may be interviewed. The team meets with the contractor or principal investigator and the principal investigator's senior staff, such as field directors, crew chiefs, data control manager, and laboratory director. The team also meets with the interdisciplinary consultants, such as geologists, geomorphologists, pedologists, paleobotanists, paleozoologists, and physical anthropologists. The team usually will visit the site or sites currently under investigation and the laboratory facilities of the contractor to get a feel for the day-to-day operation and management of the project. In some instances the peer reviewers will meet with some members of the professional community and Indian Tribes or their representatives. The team also may interview interested members of the public and Congressional staffers.
Each peer review will vary regarding the parties interviewed because the focus of the peer review will be different in each project. In the peer reviews conducted by the DCA, it was standard procedure to provide the SHPO and ACHP with a chance to meet with the peer review team. Sometimes this was done by telephone interviews. This consideration is based on the regulatory roles of these agencies in historic preservation. Furthermore, the professional staff of the SHPO are especially knowledgeable about their State's archeological resources and can provide important perspectives to the team.
The scope of the peer review and the specific circumstances will determine whether or not the agency archeologist or official attends all of the interviews conducted by the team. In cases where problems exist between the agency and the contractor, SHPO, ACHP, or other parties, it has been found prudent for the team to meet both with and without the agency personnel. The absence of agency personnel may allow for open and candid discussion that otherwise would be difficult.
The on-site visit is a period of intense work by the peer review team. Every effort is made to keep the team on schedule and focused on its task. The peer review manager and agency personnel provide all reasonable support requested to assure that the team's tasks are completed by the scheduled time.
Review Draft Report
The importance of coordination and careful attention to details in production of the final draft report cannot be overestimated by the peer review manager. Some cautionary examples are cited from the peer reviews conducted by the DCA to demonstrate the importance of organized, considered activities and then continuing to learn from experience.
The peer review team for the Dolores Project at McPhee Reservoir spent an intensive week interviewing various parties concerned with the project. After the on-site visit the team members returned to their regular employment and prepared the draft final report. As a long-distance effort, completion of the draft was delayed considerably. This delay was caused by factors such as lack of communication between the members of the team, the time taken to circulate drafts and comments sequentially through the mail, and other commitments or priorities of the panel members. The Dolores Project peer review team was not provided with a proposed report outline, so team members had to develop an outline and report format, which proved to be very time-consuming. Based on this experience the DCA determined it would be helpful in the future to provide guidance and suggested topics to cover. This practice has expedited the preparation of draft peer review reports.
The Central Arizona Project peer review team decided that it would complete a draft report by the end of the on-site visit. Despite certain misgivings, this proposal was encouraged and the draft report was delivered as promised. The DCA edited the draft report for grammar and worked with the format, but made no changes in content. A revised draft was mailed simultaneously to the authors for revision. Marked and revised copy was returned to the DCA and changes made. The final report was produced and distributed to the Bureau of Reclamation in a short period along with the DCA's own recommendations. Other interested parties were sent copies as part of this distribution. The Central Arizona Project peer review demonstrated the clear value of requiring the team to complete a draft report by the end of the on-site visit.
Appendix D provides examples of report outlines presented to the peer review teams for the Central Arizona and Carson Desert projects.
The peer review manager transmits the report and recommendations for action to the agency for consideration. The peer review manager may wish to include his or her own recommendations, which address the findings and recommendations of the peer review team and may be mandated according to the authorities by which the peer review manager undertook this task. While the peer review manager normally endorses most of the recommendations made by the peer review team, occasionally it will be determined that the recommendations do not really improve the project when compared to costs by the agency to implement them. Irrespective of whether the peer review manager supports or rejects the recommendations of the team, a clear explanation is always provided.
Peer reviews for public archeology projects have been and will continue to be important tools for agencies to use to improve both research and preservation activities. They assist agencies in meeting the objectives for which public archeology projects are undertaken. In addition, they can help improve public awareness of the value of contributions to knowledge about the nation's cultural past that are central to such projects. When peer reviews are conducted in this comprehensive way they have had two important long-term results. First, the agency gains a better perspective on the impacts of sound archeological resources management and therefore can develop effective means to improve its programmatic efforts. Second, the benefits of public archeology research are clarified for the academic disciplines and therefore can be incorporated appropriately into the most current theoretical and interpretive developments for explaining the archeological record.
Implicit also in the purpose of this Technical Brief to provide guidance to governmental agencies and other archeological resources management programs on the peer review process is encouragement for those agencies to undertake their own. Such peer reviews may be full-scale versions, similar to ones completed by the DCA, or they may be smaller, to fit the needs of more localized projects. In either case, the results can lead to cost savings and increased support for archeology among members of the public and within agencies.