Veridian uses Cloudflare to help protect digital collections from scraping, cyberattacks, and other online threats while improving site speed and reliability.
Libraries, archives and historical societies work hard to make unique collections available online. As these collections grow in visibility and use, they can attract unwanted activity, including:
Automated bots attempting to harvest large volumes of content
Distributed denial-of-service attacks (explained below) intended to overwhelm servers
Attempts to exploit website or application vulnerabilities
Without appropriate protection, these activities can slow down the site, increase infrastructure costs and, in serious cases, make a collection temporarily unavailable. They can also lead to misuse of copyrighted material.
For many of the collections we host and support, Cloudflare forms part of a wider security and performance strategy. It is one of several tools we use to help protect collection infrastructure while maintaining reliable access for legitimate users.
Cloudflare is a web security, performance and content-delivery platform. It operates between a website and its visitors, where it can inspect incoming requests, filter suspicious traffic and deliver cached content from servers closer to users.
For digital collections, this additional layer can help reduce malicious traffic, improve resilience and make frequently accessed content faster to load.
Cloudflare helps filter suspicious traffic, reduce cyber threats and manage harmful bots, while improving site speed, reliability and server performance. Below, we explore these benefits in more detail.
Cloudflare continuously analyzes traffic across its global network to identify suspicious behavior and known malicious sources. It uses this threat intelligence to automatically filter known harmful traffic before it reaches your collection. This includes blocking requests from IP addresses, networks, and automated tools associated with cyberattacks, scraping, and other abusive activity.
Our team uses these capabilities, where Cloudflare is deployed, to monitor traffic patterns, investigate unusual activity, and configure custom security rules. When we identify new scraping techniques or malicious sources, we can quickly update these rules.
Where enabled, Cloudflare stores copies (or caches) of frequently accessed content on servers around the world through its Content Delivery Network. This helps collections load faster and respond more quickly for users wherever they are located. By serving much of this content directly, Cloudflare also reduces the load on the collection’s servers, helping lower bandwidth usage and associated hosting costs.
Cloudflare protects against common attacks such as Distributed Denial of Service (DDoS) attacks, which attempt to overwhelm a site with excessive traffic; SQL injection attempts designed to manipulate a website’s database; brute-force attacks intended to crack passwords; and other widely recognized vulnerabilities, including those listed in the OWASP Top 10.
Some bots are helpful, such as search engines that index content for discovery. Others are designed to copy large amounts of material, overload servers, or misuse website resources. Cloudflare uses machine learning and behavioral analysis to distinguish legitimate users from malicious bots. It also offers innovative tools to deter scrapers, including the ability to serve realistic AI-generated content to suspected bots trapping them in a virtual labyrinth.
Once again, our team actively monitors bot activity and, where Cloudflare is deployed, updates security rules when we identify new scraping techniques.
Related reading: Bot Wars 4.0
Where Cloudflare is used, it allows us to create custom security rules that help control how visitors access the collections. These rules can automatically slow down, challenge, or block suspicious traffic when needed, giving us an additional layer of protection against scraping, abuse, and other malicious activity.
Custom security rules offer fine-grained control over who is detected, what actions to take when they are, and can even be used as means to whitelist certain actions or users if they need heightened access to the collection (for example automated monitoring or reporting tools like SiteImprove).
When we identify a new scraping technique or malicious source, we can update our Cloudflare rules so that other collections using the same protection can benefit from the same improvements.
This shared intelligence means each client benefits from lessons learned across our entire network of collections.