All Collections
REST API
API Recipe: Extract All Cookies from All Web Pages
API Recipe: Extract All Cookies from All Web Pages
Luiza Gircoveanu avatar
Written by Luiza Gircoveanu
Updated over a week ago

Overview

Given an Audit Run, you can extract all the console logs collected on each page.

Note: More information about each web page is available from the ObservePoint API, but this recipe only covers collected console logs on each page. Using this recipe as a starting point, you can expand your integration to include other information.

Step 1: Get your audit ID and run ID

You’ll need an Audit ID and Run ID to get started. There are multiple ways to do this, depending on your needs. The ObservePoint API is flexible, so you can use the approach that works best for you.

Here are three ways to get your Audit ID and Run ID:

  • From a webhook: If you have a webhook configured, the webhook payload will include Audit ID and run ID. See the Webhooks section above for setting up webhooks.

  • Manually (good/suitable one-time testing): You can find the Audit ID in the ObservePoint application under "Data Sources". Click on the Audit you want, note the Audit ID and run ID in the address bar.

Step 2: Download all the pages from the Audit Run:

Make an authenticated POST request to this URL with an empty request payload:

In your code, start with page=0, and make multiple requests, incrementing the page number for each request, until you have downloaded all the web pages for this run (see the “Pagination” section above for more instructions).

Each response will look like the following. Note that this example Audit Run scanned 544 web pages, and we requested a page size of 100, so there are 6 total pages to request.

{ "metadata": { "pagination": { "totalCount": 544, "totalPageCount": 6, "pageSize": 100, "currentPageSize": 100, "currentPageNumber": 0 } }, "pages": [ { "pageId": "77ebb089815a3b8d82813ebaf6320730", "dataCollectionUuid": "77ebb089815a3b8d82813ebaf6320730", "pageUrl": "http://example.com/", "pageTitle": "Example Home Page", "pageLoadTime": 665, "pageStatusCode": 200, "initialPageStatusCode": 200, "finalPageStatusCode": 200, "redirectCount": 0, "size": 34363 }, { "pageId": "139cf2a71e4ecb1967b7a5b47770e66a", "dataCollectionUuid": "139cf2a71e4ecb1967b7a5b47770e66a", "pageUrl": "http://example.com/path", "pageTitle": "Example Web Page", "pageLoadTime": 1621, "pageStatusCode": 200, "initialPageStatusCode": 200, "finalPageStatusCode": 200, "redirectCount": 0, "size": 956436 }, ... ] }

In the next step, you will use the pageId field from each of the web page records you fetch above.

Step 3: For each web page, fetch its cookies

From step 2, you have a list of page IDs (example: 139cf2a71e4ecb1967b7a5b47770e66a).

The next step is to query the API for the cookies which ObservePoint captured on each page.

For each page, make an authenticated GET request to this URL:

Each response will look like this. Note that getInsights was enabled in the above call and returns with aggregated data about the cookies within that page, namely the number of unique cookie names and number of unique cookie domains.

{ "insights": { "noOfUniqueCookies": 34, "noOfUniqueDomains": 2 }, "cookies": [ { "name": "MID", "value": "139cf2a71e4ecb1967b7a5b47770e66a", "domain": ".example.org", "path": "/", "httpOnly": false, "secure": false, "expires": "2023-01-01T00:00:00.000Z", "partyType": "1st_party" }, ...

This API endpoint returns these fields in its payload:

Field Name

Field Description

noOfUniqueCookies

Number of unique cookies set on that page

noOfUniqueDomains

Number of unique domains for all cookies set on page

name

Name of the cookie set

value

Value set within that cookie

domain

Domain set for cookie

path

Path set for cookie

httpOnly

Cookie field defining if a client side script can access the cookie

secure

Cookie field defining if cookie can only be sent through secure channels

expires

Expiration of cookie

partyType

Defines if the cookie a 1st party, 3rd party, or 3rd party (owned) cookie, based on the page domain and domain set within the cookie78


Conclusion

With all the web pages and cookies downloaded for this Audit Run, you can store them in a database and report/visualize them as you like.

Did this answer your question?