Published by the DOI Departmental Consulting Archeologist/NPS Archeology Program, National Park Service, Washington, DC, September, 1993.

This Technical Brief describes objectives, organization, and methods that could be used for conducting peer reviews in public archeology projects. Each of these elements is illustrated with examples drawn from peer reviews conducted by the Department of the Interior's Departmental Consulting Archeologist between 1981 and 1990. The purpose of this Technical Brief is to provide guidance to government agencies and other archeological resources management programs on use of the peer review process as one tool to improve the effectiveness of their activities. It may be an especially important tool to help resolve conflicts which may arise due to the need for interagency cooperation, the complexity of the archeological work, or professional disagreements. This Technical Brief demonstrates how peer reviews can address these typical problems and explains ways the process can be implemented.

Introduction

The review of research by peers has been one hallmark of science throughout its history. Peer reviews are the conscience of science; they provide a necessary check on its practitioners and an explicit evaluation of their efforts. Other professions have recognized the importance of peer reviews for applied sciences as well. For example, the American Society of Civil Engineers and American Consulting Engineers Council (ACEC) have developed a program that offers peer review services for engineering projects and programs (ACEC 1990 a,b,c,d).

The Departmental Consulting Archeologist (DCA) and his staff in the Archeological Assistance Program are authorized by the Secretary of the Interior to provide leadership and coordination for Federal and other types of public archeology. The Secretary is assigned these roles by several statutes including the Antiquities Act, Historic Sites Act of 1935, National Historic Preservation Act, and Archaeological Resources Protection Act. The peer review of selected Federal archeological projects is one among a number of DCA programs that provide leadership and coordination (Canouts 1992; Canouts and McManamon 1991; Knoll 1991; Knudson and McManamon 1992; McManamon, Smith, and Waldbauer 1990).

This Technical Brief describes the purposes and the goals of the Federal archeology peer review process as conducted by the DCA between 1981 and 1990. The government agencies and the professionals who have been in these reviews recognize that peer reviews can provide an important contribution to project design and management, thereby advancing American archeology. This brief provides an example of the peer review process in the Federal archeological arena, as opposed to the academic domain. It also supplies information about the planning, design, and implementation of peer reviews. Four appendices with examples of documents illustrating efforts at planning and implementing peer reviews are provided.

Purposes and Goals of the Federal Peer Review Process

In an academic research framework, as opposed to typical public archeology projects, the principal investigator develops a research program to investigate problems that are of theoretical, methodological, and/or substantive interest. The researcher develops a research design, identifies and obtains permission to investigate the sites that fit the project's scientific needs, and raises the necessary funds by obtaining a grant. The researcher plans and conducts the work and publishes the results. Review of the scientific merits of the proposed work by his or her peers occurs during the grant application process, in the review of the results by the granting organization, as part of the publication process, and in the reception of the product by the scientific community.

On the other hand, archeology conducted to comply with Federal historic and archeological preservation laws must follow sound archeological practice as well as the regulations, procedures, and guidelines mandated by and developed from those laws. Archeology conducted by these authorities is to benefit the American people generally through the preservation of important archeological resources and information. The first objective of research in public archeology is to identify, evaluate, and record significant archeological resources. The scientific information is used for a variety of practical purposes, including education and creating an appreciation of the nation's heritage among its citizens. The concomitant benefits that individual archeologists may derive from these endeavors with regard to professional standing and career advancement are secondary.

Early in the historic preservation process, the Federal agency identifies archeological properties within the boundaries of an undertaking. This is accomplished through literature reviews, records checks, and field surveys. The significance of the sites, especially for the important archeological and historical data they contain, is determined based upon eligibility for the National Register of Historic Places. Significance commonly is assessed in reference to State comprehensive historic preservation plans or historic contexts used by the State Historic Preservation Officer (SHPO).

If important data will be lost because of the undertaking, the Federal agency must consider this potential and consult with the SHPO to develop ways to mitigate the loss. The agency also may have to get the comments of the Advisory Council on Historic Preservation (ACHP) before moving forward with the undertaking.

Several parameters have been defined for Federal archeological projects that constrain the scientific freedom to develop research topics. The principal difference between an exclusively academic research design and one developed for a public project is that the former develops from scientific curiosity while in the latter scientific research needs are conditioned by the potential data content of the archeological site being impacted by the public undertaking.

These constraints on the opportunity for an individual researcher to develop theoretical, methodological, or substantive aspects of the research design do not mean that Federal projects are devoid of such concerns. On the contrary, the process of developing a research design and a strategy to implement it, completing the work, and reporting are the same general steps of any scientific research project. Also, the development of the archeological project within Federal historic preservation is a consultative process in the hands of Federal archeologists and their counterparts in the SHPO office. Project plans and archeological results also may be reviewed by archeologists and historic preservation professionals of the ACHP. Frequently suggestions made during these consultations result in design improvements. As a result of this coordination, the project has a firm scientific foundation.

Figure 1. Archeologists at site of Hohokam pioneer period pithouse. Photo courtesy Bureau of Reclamation/Tom Lincoln

Before the passage of the National Historic Preservation Act of 1966 (NHPA), rescue or salvage archeology was handled without input from the States or the ACHP. Archeological rescue work was usually controlled by the time and money available rather than by the development of thoughtful and explicit research designs. The National Park Service (NPS), operating with its own professional staff through the River Basin Survey program of the Smithsonian Institution or with academic contractors, conducted the archeological salvage program of the nation. Private energy developers and State highway departments contracted directly with academic institutions to complete salvage archeology in their construction projects. Competition among academic institutions for government, private utility company, or highway department contracts for archeological work did not exist during this period. Normally the NPS, the utility company, or highway department approached an academic institution and asked the institution to carry out whatever archeology was needed within the time and funding available.

Today Federal agency archeologists or contractors from the public and private sectors conduct Federal archeological projects. Competition among archeological contractors is the norm unless the work is done by agency personnel. It does not matter who does the project for the Federal peer review process to work.

The perception has existed that Federal archeological project research design is less open to evaluation and criticism than an academic research project funded by a granting agency. This is hardly the case. As mentioned, evaluations occur as part of memoranda of agreement, National Register of Historic Places nominations, determinations of effect, reviews of scopes of work for data recovery plans, and requests for proposals that form the bases for contracts. Depending on the specifics of the project, these evaluations are performed by archeologists, other cultural resources specialists, and perhaps officials of various agencies. These reviews represent a broad range of scientific and legal compliance concerns. Frequently the evaluations improve the agency proposal to mitigate the loss of important information, and stipulations for formal peer reviews are often specifically included in memoranda of agreement or agency contracts. This is all done before a project is announced for bidding. As part of the proposal, the winning bidder also prepares a research design or work plan. This provides refinements for conducting the research. It also defines the protocols that will serve as project guides.

The primary goal of the peer review process developed by the DCA is to evaluate the conduct of Federal archeological projects and assess the competence and efficiency of projects relative to archeological practice and legal compliance. An academic peer review rates a project on its scientific merits alone; reviewers examine the theoretical basis for the research, the data needed, and how they contribute to the research goals. They evaluate the methodology used to get needed data and the analytical methodology. They determine if the research was conducted in an appropriate manner. In Federal archeology, the peer review examines additional topics as well as these. Is the agency following the stipulations of the compliance documents agreed to under the Section 106 process of NHPA? Have data fulfilled the needs of the research design? Have unexpected data or research domains come to light that need additional investigation? Does the project need to be changed or redesigned to take these findings into consideration?

During the 1980s, Federal archeology project peer reviews were undertaken when controversy stirred about an undertaking or the archeology associated with it. Some reviews were conducted because of major disagreement between the agency and the contractor about the work. Allegations or questions may have come from the archeological profession, the media, special interest groups, or members of Congress. More recently agencies have requested reviews by the DCA to investigate whether projects were adequate. One review was conducted for an agency that wished to improve a project.

In public archeology the use of the peer review process also serves other purposes:

Philosophy of Peer Reviews for Public Projects

In designing the peer review process the scientific value of an archeological project, as determined by the NHPA Section 106 consultation, is accepted and used. Furthermore, the research goals and priorities and the stated applications of archeological methodology also are accepted and used. The peer review evaluates the project using the following criteria:

This approach accepts previous decisions made by the various consulting parties to mitigate the loss of important archeological information caused by a Federal undertaking. It also provides an occasion to assess the project to determine if any changes are desirable. This approach steers clear of second-guessing earlier decisions or trying to identify all of the approaches for conducting archeological investigations. The manager of the peer review and the agency archeologist establish a project review agenda and develop a list of suggested review topics. These aid the peer review panel in examining archeology defined by the NHPA Section 106 process.

Project Selection

Since 1981, the DCA has conducted six peer reviews (Table 1). The experience from conducting these reviews is the basis for this Technical Brief. These projects were selected for a variety of reasons. The Dolores Reservoir Project was reviewed because of problems between the contractor and the agency about the scope of the investigation and project management. The Central Arizona Project was selected because the Bureau of Reclamation wanted to confirm that the scope and focus of the archeological project were appropriate. The Jackson Lake Dam Project was evaluated to demonstrate to the members of the Wyoming Congressional delegation that the work was adequate and satisfied State interests. The Fish and Wildlife Service sought a peer review to address Native American concerns and to assure that historic preservation requirements were being met in the Stillwater Wildlife Management Area Project. The U.S. Army Corps of Engineers wanted to confirm that its historic preservation responsibilities were being fulfilled in a manner sensitive to Native American religious concerns in the Libby Dam Project. The Soil Conservation Service wanted an evaluation of the Alkali Creek Project in light of SHPO and professional community concerns.

Organizing and Conducting Peer Reviews

A peer review of a project is an activity that must be developed harmoniously as a partnership between the peer review manager and the agency. For all but the Alkali Creek peer review, the DCA served as peer review manager. In the case of Alkali Creek, a senior NPS archeologist well acquainted with the project area was designated by the DCA as peer review manager. Normally the agency representative has been the archeologist responsible for the administration and conduct of the agency's archeological program, although another agency official may serve in this role. The need for frequent and clear communication between the peer review manager and the agency archeologist or other official in charge of the project is mandatory. Good communication is an absolute necessity for a successful peer review. No matter who serves in them, these roles are essential components of the peer review process.

All aspects of the peer review are developed through close coordination between the peer review manager and the agency. Both parties must agree on the scope of the review. Frequent and open communication is necessary in all aspects of the project. The agency archeologist needs information about the progress of the planning effort by the peer review manager to develop agency plans. The peer review manager also discusses the peer review with the SHPO and ACHP to gather information that should be supplied to the team. Frequent communication bolsters the commitment made by both parties to assure that the necessary actions are completed in a timely manner. In undertaking a peer review, one of the peer review manager's goals is to create a partnership with the agency. The creation of such a partnership is the principal ingredient that has made the process, as conducted by the DCA, successful.

Once a project is selected, the following tasks, at a minimum, must be undertaken to conduct the peer review:

The following sections of this Technical Brief will provide information about each of these tasks. The examples cited are drawn from peer reviews conducted in the past by the DCA.

1. Defining the Scope of the Peer Review
The reasons for selecting a project for peer review normally defines the scope of the review. If charges of inadequacy have been made, the peer review will focus on the conduct of the archeological investigations under the agreements and documents covering the project. If the peer review is a response to problems between the agency and the contractor, the peer review will focus on these matters. The topics identified for review for the Stillwater Wildlife Management Area are provided as examples in Appendix A; those for the Jackson Lake Dam project were similar. In any case, the scope of the peer review should focus on identifying solutions to problems and not foster inappropriate or adversarial investigative roles.

2. Cost of the Peer Review and Funding
Arrangements must be made to fund the various costs for peer reviews and establish the budget parameters within which the peer review manager must operate. The costs of peer reviews conducted by the DCA, for instance, have been covered by various means, including the DCA's annual operating budget, shared expenses of the DCA and the agency, and full funding by the agency. When an agency shares or covers the costs, the funds typically come from the project budget. Cost sharing between the DCA and the agency has been the most common arrangement.

Honoraria or consulting fees have not been paid to members of the DCA peer review panels. Payment of honoraria to non-Federal members of the team is legal, but it is illegal to pay a Federal team member. It seemed inappropriate to compensate some team members but not others. Furthermore, there was concern that if honoraria were paid the peer review team might not feel that it could be critical and candid in its assessment. There was also a potential for the review to appear prejudiced.

The costs of transportation and per diem of the peer review members must be considered. The purchase of airline tickets may take a large portion of the funds available. Other costs include: reproduction and postage to provide project information to the peer review team; on-site equipment, materials, and work space for the team; administrative support; and final report preparation and distribution. Many of these costs can be covered in-kind by the agency or the peer review manager.

3. Composition and Selection of the Peer Review Team
The first peer review team assembled by the DCA consisted of three nationally recognized archeologists from academic positions. Two were experts in the archeology of the project region. One was an expert from outside the area. This composition was modified for later peer reviews. It was learned that academic archeologists were unfamiliar with the NHPA Section 106 process and unaware of many of the legal requirements of Federal historic preservation. They had little or no knowledge of Federal procurement and contracting regulations and, therefore, they found it difficult to readily identify efficient, effective solutions to problems related to these topics.

For later reviews, an archeologist from a public agency, but one different from the one under review, was added to the team. This served the additional goal of increasing the interaction among government archeologists and their academic colleagues. This was a way to improve the credibility of the government archeologists with their academic counterparts. We have found that a balanced peer review team consists of one public archeologist from an agency other than the one responsible for the project, an academic expert in the archeology of the project area or region, and an academic expert from outside the research area.

To identify potential review team members the peer review manager and the agency archeologist independently compile three lists of candidates for: (1) the regional academic expert, (2) the outside academic expert, and (3) the public agency archeologist. They consolidate their lists and decide which candidates and alternates to invite.

Care is to be given in selecting members of the panel. Beyond a candidate's scientific credentials and availability, other elements are considered, for example:

1. Does the candidate have a reputation for completing work in a reasonable time?
2. Does the candidate have a previous relationship or any other consideration with the agency or the project that could be construed as a conflict of interest?
3. Was the candidate an unsuccessful bidder for the contract?
4. Does the candidate work well with others?
5. While these elements may seem straightforward, ignoring them could make completing an acceptable peer review difficult or affect its timeliness.

Once the peer review manager and agency archeologist agree on the candidates and the approximate dates for the review, the peer review manager telephones the candidates to determine their willingness and availability. Acceptance of the invitation is followed by a formal letter from the peer review manager providing information about the project, the schedule, and other details.

4. Compiling Documentation for the Peer Review Team
By the time of the onsite visit the peer review team should have a thorough knowledge of the project. The agency responsible for the project compiles and furnishes the panel with the documents that describe the undertaking, the archeological resources, their significance, the research questions, and background information about the local archeology. Acquisition of this necessary knowledge by the peer review team is facilitated by providing all the documentation to team members at least three weeks before the on-site visit. If it is provided with less time before the on-site visit the peer review team will not have time to review this basic information.

The package of material compiled for the orientation of the peer review team will vary from project to project. Typically it will consist of survey and evaluation reports and NHPA Section 106 documentation, such as the Memorandum of Agreement or Determination of No Adverse Impact, scope of work or request for proposal, and research design. Interim or annual progress project reports, if available, also are provided. Additionally, correspondence and internal documents may be included. (A list of the kinds of materials supplied for the Central Arizona Project and Stillwater National Wildlife Management Area peer reviews is provided in Appendix B.)

Figure 2. Archeologists sharing information at site of Hohokam village. Photo courtesy Bureau of Reclamation/ Tom Lincoln

5. Scheduling the Peer Review
The ideal time for a peer review is at the midpoint of a project. Typically, field and laboratory work has proceeded to the stage where the review team can make a reasonable evaluation of the results. A peer review at the beginning of a project is premature because there will be few results to assess. If the project is near completion the important recommendations of the peer review team will be difficult to implement; usually time and money have become short at this stage.

A second consideration in scheduling the peer review is for a time that is acceptable to potential team members. It is useful to identify a 3- to 4-week period for the peer review approximately 3 months in advance of the on-site visit. Within such a time frame it is usually possible to fit the schedules of all personnel involved. Situations that can affect the scheduling of the review must be considered. These include college and university calendars or special events like conventions or athletic contests in the project area. These situations can affect travel arrangements, as lodging may be difficult to find in such circumstances. The schedules and commitments of the individuals representing the various organizations who will be interviewed by the peer review team must be considered. The on-site visit must be scheduled so that those who will travel from outside the project area, the SHPO or appropriate staff, ACHP personnel, and other interested parties, can participate.

The peer review manager, in consultation with the agency archeologist, develops an agenda for the peer review team's on-site visit. A detailed agenda assures that the on-site visit is carefully structured. Each interview focuses on a specific task or topic. Each interview session is allotted a specific amount of time. Sessions usually do not exceed two hours without a break. Maintaining strict adherence to the agenda assures that the peer review team stays focused on the task and adheres to the schedule.

During the first session of the on-site visit the peer review manager, the agency archeologist, and the peer review team discuss the agenda in detail and modify it as necessary. Implementing modifications in the agenda may require complex changes in the interview schedule. It is important to determine the final agenda as early as possible.

6. On-Site Visit
The peer review manager and the agency archeologist plan the on-site visit to use everyone's time efficiently. Usually the peer review manager, agency personnel, and peer review team arrive the day before the on-site visit begins.

The peer review manager and the agency archeologist provide the necessary administrative support to the peer review team. This assures that an efficient review occurs. Meeting facilities, telephones, transportation, and lodging and dining facilities must be available. Secretarial assistance, equipment, and supplies may be needed. The object of taking care of these necessities is to allow the team to devote its energies to the peer review. Personal computers have been very useful. In providing these kinds of administrative support, the peer review manager and agency function as aides to the team.

The first part of the agenda is devoted to briefings by the peer review manager and agency personnel. The peer review manager describes and then emphasizes the specific scope of work of the peer review (see Appendix C for examples). The peer review team can examine any aspects of the project related to the archeological work. The plan for the rest of the on-site visit will depend on the schedules of individuals, the weather, field conditions, and other variables. During the week-long peer reviews conducted by the DCA, personnel interviews and field visits normally were concluded by the end of the third day so the team could review its findings and draft its report. Typically the draft report was completed by the evening of the fourth day. Exit interviews with the DCA, agency personnel, and other interested parties were conducted on the fifth day.

The peer review team normally interviews agency personnel such as the project engineer, environmental staff, and the archeologist. Other historic preservation specialists, the contracting officer, or contracting officer's technical representative may be interviewed. The team meets with the contractor or principal investigator and the principal investigator's senior staff, such as field directors, crew chiefs, data control manager, and laboratory director. The team also meets with the interdisciplinary consultants, such as geologists, geomorphologists, pedologists, paleobotanists, paleozoologists, and physical anthropologists. The team usually will visit the site or sites currently under investigation and the laboratory facilities of the contractor to get a feel for the day-to-day operation and management of the project. In some instances the peer reviewers will meet with some members of the professional community and Indian Tribes or their representatives. The team also may interview interested members of the public and Congressional staffers.

Figure 3. Re-excavation conference with contractor and peer review team, Central Arizona project. Photo courtesy Bureau of Reclamation/ Dan McKeever

Each peer review will vary regarding the parties interviewed because the focus of the peer review will be different in each project. In the peer reviews conducted by the DCA, it was standard procedure to provide the SHPO and ACHP with a chance to meet with the peer review team. Sometimes this was done by telephone interviews. This consideration is based on the regulatory roles of these agencies in historic preservation. Furthermore, the professional staff of the SHPO are especially knowledgeable about their State's archeological resources and can provide important perspectives to the team.

The scope of the peer review and the specific circumstances will determine whether or not the agency archeologist or official attends all of the interviews conducted by the team. In cases where problems exist between the agency and the contractor, SHPO, ACHP, or other parties, it has been found prudent for the team to meet both with and without the agency personnel. The absence of agency personnel may allow for open and candid discussion that otherwise would be difficult.

The on-site visit is a period of intense work by the peer review team. Every effort is made to keep the team on schedule and focused on its task. The peer review manager and agency personnel provide all reasonable support requested to assure that the team's tasks are completed by the scheduled time.

7. Peer Review Draft Report
The peer review manager and the agency archeologist prepare a proposed outline for the report, making it clear to the peer review team that the topics and proposed outline are only suggestions. The content of the final report is determined by the peer review team. The purpose of the proposed report outline is to provide the team with guidance. The interviews provide specific and important information and views from a variety of perspectives. The peer review manager always assures the team in each review that the content and recommendations of the final report are entirely their responsibility. The team is welcome to change or reorganize the outline and format. Team members may add or drop topics as they think necessary or appropriate, though departures from the model usually concern format or organization.

The importance of coordination and careful attention to details in production of the final draft report cannot be overestimated by the peer review manager. Some cautionary examples are cited from the peer reviews conducted by the DCA to demonstrate the importance of organized, considered activities and then continuing to learn from experience.

The peer review team for the Dolores Project at McPhee Reservoir spent an intensive week interviewing various parties concerned with the project. After the on-site visit the team members returned to their regular employment and prepared the draft final report. As a long-distance effort, completion of the draft was delayed considerably. This delay was caused by factors such as lack of communication between the members of the team, the time taken to circulate drafts and comments sequentially through the mail, and other commitments or priorities of the panel members. The Dolores Project peer review team was not provided with a proposed report outline, so team members had to develop an outline and report format, which proved to be very time-consuming. Based on this experience the DCA determined it would be helpful in the future to provide guidance and suggested topics to cover. This practice has expedited the preparation of draft peer review reports.

The Central Arizona Project peer review team decided that it would complete a draft report by the end of the on-site visit. Despite certain misgivings, this proposal was encouraged and the draft report was delivered as promised. The DCA edited the draft report for grammar and worked with the format, but made no changes in content. A revised draft was mailed simultaneously to the authors for revision. Marked and revised copy was returned to the DCA and changes made. The final report was produced and distributed to the Bureau of Reclamation in a short period along with the DCA's own recommendations. Other interested parties were sent copies as part of this distribution. The Central Arizona Project peer review demonstrated the clear value of requiring the team to complete a draft report by the end of the on-site visit.

Appendix D provides examples of report outlines presented to the peer review teams for the Central Arizona and Carson Desert projects.

8. The Final Report
The production of the final report of the peer review can be a rather simple, but intense task. The draft report is edited for style and format and unclear language is revised. All changes in the manuscript are made using the "strikeout," "bold" or other options of the chosen word processing program. This allows the author to identify easily the changes that have been made in the revised text. This draft is mailed simultaneously to the authors for their corrections and approval. The peer review manager makes the changes required by the authors and produces the final report. Depending on the quality of the draft report, production of the final version requires some 16 to 24 person hours of effort.

The peer review manager transmits the report and recommendations for action to the agency for consideration. The peer review manager may wish to include his or her own recommendations, which address the findings and recommendations of the peer review team and may be mandated according to the authorities by which the peer review manager undertook this task. While the peer review manager normally endorses most of the recommendations made by the peer review team, occasionally it will be determined that the recommendations do not really improve the project when compared to costs by the agency to implement them. Irrespective of whether the peer review manager supports or rejects the recommendations of the team, a clear explanation is always provided.

Conclusion

Peer reviews for public archeology projects have been and will continue to be important tools for agencies to use to improve both research and preservation activities. They assist agencies in meeting the objectives for which public archeology projects are undertaken. In addition, they can help improve public awareness of the value of contributions to knowledge about the nation's cultural past that are central to such projects. When peer reviews are conducted in this comprehensive way they have had two important long-term results. First, the agency gains a better perspective on the impacts of sound archeological resources management and therefore can develop effective means to improve its programmatic efforts. Second, the benefits of public archeology research are clarified for the academic disciplines and therefore can be incorporated appropriately into the most current theoretical and interpretive developments for explaining the archeological record.

Implicit also in the purpose of this Technical Brief to provide guidance to governmental agencies and other archeological resources management programs on the peer review process is encouragement for those agencies to undertake their own. Such peer reviews may be full-scale versions, similar to ones completed by the DCA, or they may be smaller, to fit the needs of more localized projects. In either case, the results can lead to cost savings and increased support for archeology among members of the public and within agencies.

Return to Technical Briefs | Text | Reports and References | Appendices