This document describes how to use ObservePoint Audits and Journeys to scan parts of your website that are protected by CAPTCHA.
What is CAPTCHA?
CAPTCHA stands for Completely Automated Public Turing test to tell Computers and Humans Apart. It is a security mechanism used by websites to determine if each user is a human or a bot, so that:
Websites can exclude bot traffic from their metrics
Websites can prevent bots from committing unscrupulous acts like fraud, denial of service, unauthorized data scraping, etc.
CAPTCHA technology uses proprietary heuristics such as observing user behavior (like mouse movements) or challenges that are easy for humans but difficult for machines. Even though the effectiveness of these challenges is decreasing thanks to AI and human CAPTCHA farms and CAPTCHA has been criticized for excluding people with disabilities, CAPTCHA remains a popular technique for preventing bot traffic. CAPTCHA vendors, on the other hand, are engaged in a digital race to stay ahead of people who want to circumvent CAPTCHA tests.
Google’s reCAPTCHA is by far the most popular CAPTCHA vendor, with some reports showing reCAPTCHA has a staggering 99% market share. Some ObservePoint customers also use hCaptcha.
Google reCAPTCHA is available in two versions:
This document will focus on using ObservePoint with reCAPTCHA and hCaptcha.
How does reCAPTCHA integrate with my website?
You and your development team need to know how reCAPTCHA integrates with your website to know how to bypass it for ObservePoint.
Important point: All reCAPTCHA tests have a front-end and back-end component. They work together every time reCAPTCHA runs. Your development team may need to make changes to your website's front-end code and back-end code to allow automated tests like ObservePoint to bypass the CAPTCHA challenge.
Every reCAPTCHA implementation needs two keys:
Site key: Used by your front-end JavaScript code to load the reCAPTCHA code from Google. Google tracks your usage with this key. The reCAPTCHA code generates an encrypted “token” that has enough information for Google’s backend servers to determine if the user is a bot
Secret key: Used by your back-end code to send the encrypted “token” from the front-end to Google’s servers.
The following sequence diagrams show how this works for reCAPTCHA v2 and v3:
reCAPTCHA v2 sequence of events: (source)
reCAPTCHA v3 sequence of events: (source)
How do I use ObservePoint to scan parts of my site that require reCAPTCHA?
ObservePoint does not try to hide the fact that we are an automated website scanner, and ObservePoint only scans sites that customers own. Consequently, reCAPTCHA reliably and correctly classifies ObservePoint as a bot.
You have multiple options for how to bypass reCAPTCHA for ObservePoint automated tests. You and your development team will need to choose what works best for your business needs. Every option requires your development team to make changes to their code, unless they have already built a bypass system into their reCAPTCHA implementation. Google does not provide an off-the-shelf mechanism for bypassing reCAPTCHA for certain IP addresses or certain users.
Your development team may want to review the sequence diagrams above to determine how best to integrate the reCAPTCHA bypass for ObservePoint or other automated testing platforms.
Option 1: Bypass reCAPTCHA (Back-end Code Only)
This option is to modify your back-end code to choose to bypass the reCAPTCHA check for certain users or certain IP addresses (or any other heuristic you choose). ObservePoint uses static IP addresses to visit your site, which your back-end code can check and decide not to validate the reCAPTCHA token.
In this option, your front-end code remains unchanged. Google’s reCAPTCHA code runs normally in the browser, but when the browser submits requests to the back-end from ObservePoint’s IP addresses or for certain users, the back-end skips the reCAPTCHA API check.
Pros:
Simpler to implement than option 2, because front-end code changes are not required
Cost is lower due to not actually sending reCAPTCHA validations to Google’s API for ObservePoint traffic
Cons:
(applies to reCAPTCHA v2 only) The reCAPTCHA UI will not appear for ObservePoint traffic, so ObservePoint will validate a slightly different experience from your normal users
Less secure, since your code will have logic that allows some interactions to bypass reCAPTCHA. This leaves rooms for inadvertent bugs in the future where regular users do not get checked via reCAPTCHA.
Option 2: reCAPTCHA Test Keys (Front-end and Back-end Code)
Google has special site and secret keys that you can use on your front-end and back-end which disable bot detection while still presenting the same user interface to the user.
To implement these testing site keys and secret keys on your site, your development team will need to conditionally load these testing keys based on conditions that work for your business. ObservePoint suggests your team consider these options:
By IP Address. ObservePoint uses a set of static IP addresses, which your front-end and/or back-end code can detect to decide to use a test key instead of your normal production key.
By User. If your system knows the user who is logged in before presenting the reCAPTCHA, you could choose to use a test key in that scenario.
How to use a reCAPTCHA test key?
Using a test key with reCAPTCHA v2:
The same special test keys are used by all reCAPTCHA v2 users: (source)
V2 test site key: 6LeIxAcTAAAAAJcZVRqyHh71UMIEGNQ_MXjiZKhI
V2 test secret key: 6LeIxAcTAAAAAGG-vFI1TnRWxMZNFuojJ4WifJWe
When you use the test keys, the reCAPTCHA v2 UI looks like this:
Using a test key with reCAPTCHA v3:
With v3, you must generate your own site key and secret key for testing in the Google developer console. Find the team that manages your reCAPTCHA Enterprise account in Google Cloud, and give them these instructions:
Click “Create Key”
Give the key a name like “ObservePoint test key”
Choose platform type: “Website”
Click the checkbox: “This is a testing key”
Enter score: 1.0 (this ensures that reCAPTCHA will classify every user of this key as a human, not a bot)
See screenshot below
Save the key
Option 2 Summary:
Pros:
ObservePoint experiences the exact same visuals as regular users, so you get a higher confidence validation
You get metrics in Google’s console showing how often your test keys are used
More secure, since every interaction must go through reCAPTCHA. No exceptions for testing systems. This reduces the risk that regular users will accidentally bypass reCAPTCHA due to an inadvertent bug in your code.
Cons:
More complex and more effort than option 1: Front-end and back-end code changes are required.
How do I use ObservePoint to scan parts of my site that require hCaptcha?
hCaptcha offers test keys that follow the same pattern as reCAPTCHA v2. You have the same options for hCaptcha as reCAPTCHA.
You can follow the instructions above for reCAPTCHA v2, but use the test keys below, depending on the type of hCaptcha account you have:
Test Key Set: Publisher or Pro Account
hCaptcha Site Key: 10000000-ffff-ffff-ffff-000000000001
hCaptcha Secret Key: 0x0000000000000000000000000000000000000000
Test Key Set: Enterprise Account
hCaptcha Site Key: 20000000-ffff-ffff-ffff-000000000002
hCaptcha Secret Key: 0x0000000000000000000000000000000000000000
If you need more info, see the hCaptcha Instructions for automated testing.