UNDERSTANDING SUCCESS Measuring Effectiveness of Preservation and Security Programs

Institutions must find accurate and cost-effective ways to assess the effectiveness of their preservation and security programs and to balance access with protection. They need to determine how well controls are working and what needs to be done to correct deficiencies—and then to demonstrate to funders that the money being spent is achieving the stated goals. In large institutions, conducting a full inventory and then periodically measuring changes is often not practicable. Rather, institutions can use statistically valid selection and measuring techniques to estimate the status of the total population of items.

This paper describes different sampling methods and identifies sampling projects undertaken at the Library of Congress. It discusses designing and developing a statistically valid baseline and then using carefully controlled measurements to determine the status of controls in place. Armed with accurate measurements, the institution can make informed decisions about implementing remedies proportional to the risk.

Since its founding, the Library of Congress has been entrusted with the preservation of major works in history, literature, the arts, science, language, and a variety of emerging cultural works. In addition, the Copyright Office function adds hundreds of thousands of works in various formats to the collections each year. The Library staff must find new ways to store, catalog, secure, and preserve many of these works for future generations to come. At the same time, the Library must make its collections relatively open to public observation and use by scholars and private citizens alike. The challenge is to find the balance between these competing goals, namely preservation and protection versus public accessibility.

Beginning in the 1970s, staff observations and formal studies funded by the Library indicated that public access had resulted in missing materials; defacing of texts, manuscripts, posters, and pictures; and wearing out of materials because of use.

To prevent the first two areas of concern, the Library needed to tighten security and change its methods of serving materials to the public. Library management wanted to be sure that installation of security measures would be done in proportion to the risk involved—but there had to be a balance. Overreaction was to be avoided. For example, it was not feasible to simply close access to most areas. Security experts were consulted, with the result that the security staff was augmented and professionals in the field were hired.

During the 1990s, the Library implemented many innovative security features, several of which involved preventative measures incorporating location and access controls. Other features involved observational and search measures. However, none of these security features pose a major threat to public access.

Installing security and control features can be very costly The Congress wanted to know whether or not the new security measures were working and, if so, whether or not they were worth the cost. Because much of the cost of security is ongoing, such concerns continue to affect the budget annually.

Measurement of improvement in security and reduction in risk usually involves establishing a baseline and then following up with multiple periods of specific attribute measurements. Some security-minded managers believe that before security features are installed, a baseline wall-to-wall inventory should be taken, followed by another complete inventory in a future period. Making complete inventories for the Library of Congress or for sections within it, however, would be extremely costly and time-consuming. By the time such an inventory was completed, major changes could have taken place, rendering the information from the inventory useless or outdated. The large expenditure of funds would have been wasted.

For almost a century now, business managers in public and private institutions alike have been using quality assurance and control studies involving statistical sampling to save both time and money. When the universe of items to be studied is very large, the institution can use a probability sample of reasonably small size to estimate the condition of that larger universe. With probability selection, we can place reliability bounds on the estimate and provide as precise an estimate as is needed by both management and oversight groups—in the case of the Library, these groups include the General Accounting Office, the Library's Inspector General, and congressional staff. There is an extremely large and varied literature involving the theory and application of statistical sampling to such problems. Both government agencies and universities use these methods. In the Washington, D.C., area, there is no lack of qualified experts in the field.

Sampling with known probabilities of selection allows extrapolation of results to the broader population of items or to locations from which the sample was drawn. Types of sampling plans include:

(1) simple random (equal probability),
(2) stratified random (equal probabilities within subgroups),
(3) cluster or multistage (hierarchical sets of probabilities),
(4) probability proportional to size,
(5) cohort group sampling over time, and
(6) paneled sampling with "births and deaths" over time.

These and other techniques can be tailored to provide efficient and effective measurements with high confidence and precise estimates, but at a much lower cost than a complete inventory of a population. Sampling for measurement of change (because of security or control measures, for example) must also start with a baseline. However, this baseline can be the result of a carefully designed statistical sample. Subsequent samples can then be done over time to measure change within the bounds of statistical significance.

The Library has designed more than a half-dozen sampling plans for its various entities. A few of these plans have been funded, and the results of the testing have been helpful to management in understanding the status of the collections at any given point in time. We were not able to start these samples with a natural baseline at the point where security measures were installed, however, so it is difficult to know the total value of these measurements. If sampling is repeated at regular intervals, management will be able to measure the improvement or deterioration of the collections and evaluate the results in terms either of security control matters or of condition based on normal wear, tear, and deterioration through time.

Because of the lack of a meaningful baseline at the Library in any of its divisions, we had to choose a place to start, with the goal of measuring change through time. We applied sampling techniques to the Prints and Photographs Division and to the National Library Service for the Blind and Physically Handicapped inventory of reading machines for the blind. We were able to study Library holdings for existence and completeness as well as quality. With a baseline, we will be able to measure both quantitative and qualitative benefits of installed controls and security measures. These could then be compared to quantitative and qualitative costs of the installed controls in order to provide the net benefit of the installation and operation of such controls. The key to success in measuring the net benefit is the continuation of sampling and statistical analysis through time.

Although it is true that measurement and testing alone will not fix the problem (actions must be taken and money must be spent to do that), nevertheless such analysis techniques will permit management and oversight authorities to discover and highlight the extent of a problem and the extent of improvement achieved because of actions taken by management.