Newspaper digitization projects involve more than scanning pages. Decisions about source material, image quality, OCR, metadata, file formats, preservation, online access, and vendor responsibilities can all affect the quality, cost, and long-term value of the finished collection.
Planning these elements together from the outset can prevent expensive rework and help ensure the digitized newspapers are searchable, manageable, and useful to researchers.
These 10 practical tips draw on the Veridian team’s experience supporting large and complex newspaper digitization projects.
1
Plan the complete project from the outset
Consider the entire project and all its steps from the outset, and talk to all your proposed vendors/suppliers early in the planning process.
2
Decide how the collection will be preserved and accessed
Sometimes the decisions you make about the platform the finished collection will reside on may affect the type/format of digital objects you need to produce. If possible make decisions about hosting platforms and long term preservation prior to starting on the scanning/digitization process.
3
Determine which work to complete internally
Consider which parts of the project you’re able to do yourself and which you might outsource. For example, if the project is not too large you might consider doing the scanning work in-house, if you have suitable equipment. For larger projects though it might make sense to outsource all the scanning and data preparation work.
4
Assess whether existing workflows suit newspapers
Carefully consider how the project fits with (and differs from) your other digitization projects. Is the platform and/or workflow you’ve used for previous digitization projects suitable for newspapers? Are there better alternatives?
5
Seek advice from experienced newspaper digitization teams
Get advice from those who have worked on large newspaper digitization projects before. Newspaper projects have unique characteristics and are often more complex than other types of digitization projects. You are likely to get good advice from other institutions that have gone through the process. Following a similar approach/workflow to that you’ve developed for other types of projects may not always be the best option.
6
Choose appropriate metadata and file formats
Consider using METS/ALTO standard format o represent the structure, metadata, layout, and OCR text associated with digitized newspapers. These widely used, documented standards can support interoperability, data exchange, search, and future reuse. See Why choose METS/ALTO? for a full explanation of why this is so important.
7
Evaluate whether to scan microfilm or original newspapers
Evaluate whether to digitize from microfilm or original newspapers. Microfilm may be easier to transport and can reduce handling risks for fragile originals. In some projects, it may also provide a more economical production route.
Original newspapers may offer better image quality, particularly when the available microfilm is incomplete, poorly exposed, or created from damaged source material. The best choice depends on the condition and completeness of both sources, the required output quality, the available equipment, and the project budget.
8
Test image quality before beginning OCR
When different teams are responsible for scanning and OCR, involve the OCR provider before large-scale scanning begins. Supply representative sample images so the provider can test whether resolution, contrast, compression, cropping, and other image characteristics support accurate text recognition.
Image processing intended to improve visual appearance can sometimes remove or distort details needed for OCR. Testing the full workflow early helps identify these issues before they affect thousands of pages.
9
Assess the value of article segmentation
Consider the costs and benefits of article segmentation. Many modern newspaper digitization projects now do this, to allow individual newspaper articles to be identified on the page. Article segmentation is generally considered to provide a nicer user experience, but there is additional cost. Typically you might expect it to cost $0.20 – $0.50 per page to digitize newspapers without article segmentation, or $0.70 – $1.00 with it.
10
Run a pilot before full-scale production
Consider running a pilot project with a small number of newspaper pages, prior to making any final decisions. Many scanning, OCR, and hosting platform vendors (including us at Veridian!) are willing to process a small number of samples and put them online for evaluation, for little or no cost. Doing so allows you to ensure the entire process works as expected before you commit to it.