https://unsplash.com/photos/y6HpQzW87Vc
Learn about the state-of-the-art technologies Imperva uses to detect bots
2014 was the first year bots outnumbered human users online. That number has only increased since then, and is virtually guaranteed to continue.
In previous articles, we’ve talked about how Imperva differentiates good bots from bad bots, and what kinds of strategies are effective against various kinds of bad bots. But all of these processes rely on a single, all-important first step – distinguishing between bots and legitimate users.
Bots are simply software applications that run scripts on the Internet. Simple bots are easy to detect: Most human users cannot type more than 80 words per minute, or navigate dozens of web pages per second.
But the malicious bots developed by today’s well-funded cyber criminal enterprises are anything but simple. Detecting automated behavior requires some of the tech world’s most sophisticated technologies alongside clever engineering.
For example, there are companies out there that offer wide proxy network services that can defeat simple bot mitigation strategies. There are APIs designed to solve CAPTCHAs. Cybercriminals know this, and are increasingly adopting these tools in their illicit workflows.
How To Detect Advanced Persistent Bots (APBs)
APB is the technical term for the complex bots behind web scraping schemes, credential stuffing scams, and denial of service attacks. These bots are programmed to act like human users online, and can be difficult to detect using old methods.
For example, it used to be the case that a security engineer could identify automated behavior by scanning the IP addresses sending traffic to a website. If a single IP address sent 1000 requests to a website, it would block that device.
An modern APB could bypass this IP-oriented security tactic by making one request each, using 1000 different IP addresses to do it. Catching automated behavior requires a more sophisticated approach.
There are several techniques available today that can detect bot behavior far better than simple IP logging. According to the OWASP Automated Threats Handbook, some examples include:
- Fingerprinting. Websites and applications can qualify incoming requests by a range of factors that might give away automated behavior. Verifying user agent strings, HTTP request formats, HTTP header inconsistencies, and device-specific content significantly reduces the risk of being targeted by bots.
- Requirements. Investing in security risk assessment tooling can help developers assess the effects of the security countermeasures they need to implement. They can then use that information to define additional development and deployment requirements.
- Reputation. Automated systems can verify incoming requests using a range of factors to create a user-specific reputation score. This can include the user’s web browser fingerprint, device fingerprint, geolocation, and method of entry onto the secured web server, among many other options. Bots may have disreputable origins or other inconsistencies that will give them away on a reputation-based verification. Advanced attackers are increasingly coming from residential and mobile networks, as well.
- Monitoring. Websites and applications can invest in automated systems that monitor anomalous behavior, function sequencing, and technical errors in a way that can help bot detection strategies. The moderation of user-generated content also falls into this category.
- Instrumentation. Built-in application-wide instrumentation can respond to automated attacks once identified, closing the gap between the suspicion of automated behavior and the action of blocking it. The instrument-based approach relies on constant real-time attack detection and a variable set of responses, from increased monitoring to user lock-outs, CAPTCHA verification, and more.
Imperva uses a variety of these measures and empowers them with machine learning technology. The data sets generated by fingerprinting and reputation monitoring tend to be large and diverse, making them perfect for machine learning-oriented analytics.
Imperva uses state-of-the-art machine learning tools and biometric validation to analyze user behavior and identify bots. Our high-definition fingerprinting tools analyze more than 200 device attributes and assign a unique reputation score to each user. This approach ensures that our bot detection and mitigation strategies remain up-to-date and effective against the latest threats.
The Anatomy of the Bad Bot Industry
Taking the big data approach offers security vendors like Imperva the ability to correlate important data points and understand the dynamic bot landscape of today. Bots, like their creators, share a great deal of characteristics that sufficiently advanced analysis can uncover.
Every year, we comb through this data and release our findings to the greater security community. Some of the key findings from our Bad Bot Report 2020: Bad Bots Strike Back
include:
- In 2019, bad bot traffic rose to its highest ever percentage of 24.1 percent of all traffic.
- 37.2 percent of all internet traffic wasn’t human. Human traffic increased by 1.1 percent to 62.8 percent of all traffic.
- Advanced persistent bots (APBs) continue to plague websites and often avoid detection. APBs cycle through random IP addresses, enter through anonymous proxies, change their identities, and mimic human behavior
- Bad bots continue to follow the trends in browser popularity, impersonating the Chrome browser 55.4 percent of the time. The use of data centers reduced again in 2019 with 70 percent of bad bot traffic emanating from them—down from 73.6 percent in 2018.
- With most bad bot traffic emanating from data centers, the U.S. remains the “bad bot superpower” with 45.9 percent of bad bot traffic coming from the country. For the third year in a row, the most blocked attacks originate in Russia (21.1 percent). Bots deployed from Amazon reduced significantly to 11.6 percent.
Bot operators and cybersecurity vendors like Imperva are locked in a constant arms race against one another. Investing in the latest technologies and continually improving our approach is critical to the success of our bot detection and mitigation initiatives. Our bot mitigation strategies will continue to improve as we gather more data and hone our response to one of the web’s most dynamic threats.
Learn More with Imperva Community
The Imperva Community is a great place to learn more about how to use Imperva cybersecurity technologies like On-Prem WAF, Cloud WAF and more to establish efficient, secure processes for enterprise networks. Rely on the expertise of Imperva partners, customers and technical experts.
Related Content: