Proxy

The proxy sits between an agent and the target website. The connection is intercepted, and useful information is extracted.

The service collects two main pieces of information: JavaScript from the website, and API endpoints.

flowchart LR 
 Agent --> Proxy 
 Proxy --> JavaScript 
 Proxy --> Endpoints
 JavaScript --> Website
 Endpoints --> Website

JavaScript Service

We pull out JavaScript URL links from intercepted requests. We then beautify these files, extract other lazy-loaded chunks, pull out useful matches for the LLM, and place the files in permanent storage.

We partner with Jxscout to provide advanced matching, beautification, and chunk extraction. To access the advanced features, a valid license is required.

Endpoints Service

As the agent works, we intercept API requests that it makes. We then create a map of the website's API, extracting parameters observed in the requests, and providing a sample request/response pair for future work.

This means that the work of one agent will power the future research. We infer paths with embedded parameters, and filter them out to provide an accurate model of the website's structure.