01
Scope call
(15-min)
Sites, regions, fields, frequency, and where data lands. You receive: feasibility assessment + 48-hour trial plan.
What we discuss:
Sites to monitor and data fields you need
Update schedule (daily, weekly, 3x/week, or custom)
Delivery destination (CSV/Excel, S3, BigQuery, Sheets, Email, API)
Acceptance criteria (data freshness, completeness targets)
What you receive:
Scope document with proposed schema, feasibility per site, and 48-hour trial plan
02
Trial run
(48-72 hours)
We collect sample data from 2-3 sites. You review: format, fields, quality.
How we handle different sites:
Adaptive collection methods for simple and complex sites
Proxy routing to reduce blocks
Scroll handling, click actions, and multi-page navigation
Change detection system flags layout shifts before they cause issues
When sites change:
We detect changes (usually within 15-20 minutes), adapt collection methods, and keep your data flowing. You never know it happened.
Legal & evidence:
Public pages only. No logins, no paywalls. Audit trail saved (URL, timestamp, screenshot). See Trust & Security
03
Setup
(2-3 days)
We build collection infrastructure, QA rules, and delivery pipeline. Typical accuracy: 95%+ after tuning.
4-layer quality control:
Layer 1: Automated validation Adaptive collection methods for simple and complex sites
Layer 2: Business rules Brand lists, category rules, MAP thresholds, price change alerts (e.g., flag if price jumps >50%)
Layer 3: Human QA Spot-checks for anomalies and edge cases. Review borderline scenarios before delivery.
Layer 4: Evidence & audit Every row includes: URL, UTC timestamp, screenshot (where required, e.g., MAP violations)
What we track:
Data completeness, how recent data is, anomaly resolution time. Targets set per project.
04
First production run
Clean files delivered to your system (S3/BigQuery/Sheets/Email). You confirm meets requirements.
File formats:
CSV, Excel (.xlsx), JSON, or custom format
Delivery destinations:
Files: Email, SFTP
Cloud storage: S3, Google Cloud Storage, Azure Blob
Data warehouses: BigQuery, Snowflake, Redshift
Spreadsheets: Google Sheets, Excel Online
API: Push to your endpoint (where required)
Schema stability:
Field names stay consistent. We notify you 3+ business days before any changes.
No dashboard to learn unless you want one.
05
Ongoing delivery
Daily monitoring, alerts, reviews, and named response tiers. Files arrive on schedule.
Run summaries & alerts (email/Slack):
Completion confirmations
Anomaly flags
Site change detections
Regular reviews:
Weekly or monthly (your choice) to discuss coverage, accuracy, insights, and improvements
Response times:
Routine questions: 4 hours
Data issues: 2 hours
Critical (delivery blocked): 1 hour
Typical targets (finalized per project):
Data delivered by agreed time (e.g., Mon 7am your local time)
95%+ data completeness after stabilization (week 2-3)
Same-day anomaly response
SLAs available on request
