Automate Supply Chain Market Research with Custom AI

Most supply chain teams start with manual data collection in Google Sheets. An analyst spends Monday morning copying prices for 100 SKUs from five competitor sites into a spreadsheet. By Tuesday, a competitor runs a flash sale, rendering the entire dataset obsolete. The process is slow, error-prone, and the data is always stale.

Teams then try off-the-shelf web scraping tools. These point-and-click tools work for simple, static HTML sites but break when a site uses a modern JavaScript framework to load product data. The scraper often fails silently, and the team gets an empty report without knowing why. These tools also cannot handle login-protected supplier portals or complex conditional logic.

For example, a business needs to check a supplier's inventory and only place a purchase order if the price is below a certain threshold and stock is above 500 units. A generic scraper cannot perform this multi-step logic. The team is forced back to manual checks for their most critical, business-driving workflows.

Syntora's engagement would begin by thoroughly mapping the target websites, which commonly include competitor sites and supplier portals. We would use Python with httpx to analyze the network requests each site makes, determining if direct access to internal APIs is feasible. This approach is generally faster and more reliable than parsing HTML. For sites that do not expose APIs, or require complex interactions, Playwright would be implemented to control a headless browser.

The core data processing pipeline would then be designed as a FastAPI service, intended for deployment on AWS Lambda. This service would orchestrate the data collection, capable of processing hundreds of products in parallel. As raw HTML or JSON data is retrieved, it would be passed to the Claude API with a structured prompt. The Claude API, when configured with appropriate prompts, demonstrates capabilities for extracting required fields (product name, SKU, price, stock level, shipping estimate) with high accuracy, even from inconsistent website layouts. We have experience building document processing pipelines using Claude API for financial documents, and the same pattern applies to structuring data from web sources.

The structured, time-stamped data would be stored in a Supabase Postgres database. This design creates a historical record of all price and stock movements for every tracked product. A separate Python function would be configured to run after each data refresh, comparing the latest data to the previous day's snapshot. If predefined conditions are met, such as a component's price dropping or a supplier's stock falling below a specified threshold, an alert could be sent to a designated Slack channel.

The deployed system would be scheduled with AWS EventBridge, running automatically at a set time each day. All application logs would be written as structured JSON using structlog, facilitating failure diagnosis. The client receives the complete source code in their private GitHub repository, allowing for full ownership and future modifications.