The Problem
Last month I was shopping for a new mechanical keyboard. I spent 3 hours opening 8 different tabs — Amazon, AliExpress, JD, Taobao, Best Buy, eBay, Newegg, Walmart — comparing the exact same model across every store. The price difference was 40%.
I thought: why isn't there a single place where I can search once and see prices from ALL stores?
What I Built
ShopAI — AI-powered price comparison across every major online store. Type what you're looking for, and ShopAI:
- Scrapes prices from 6+ major e-commerce platforms simultaneously
- Uses AI to intelligently match products across different stores (same keyboard, different listings with different descriptions)
- Shows you the cheapest option, the fastest shipping, and the best overall value
- Works for any product — electronics, books, fashion, home goods
The Tech Stack
- Next.js + React — Frontend with server-side rendering for SEO
- AI Product Matching — Custom model for cross-store product deduplication (this was by far the hardest part)
- Web Scraping Pipeline — Rotating proxies + rate limiting to handle multi-store queries
- Edge Functions — Price queries resolve in under 2 seconds
Hardest Problems Solved
1. Product Matching is AI-Hard
The same keyboard on Amazon is "Keychron K2 Wireless Mechanical Keyboard (Hot-Swappable)" — on AliExpress it's "Keychron K2 75% BT5.1 RGB Swappable" and on JD it's just "Keychron K2机械键盘". Same product, three completely different descriptions. I had to build a custom matching algorithm using embeddings + fuzzy string matching.
2. Anti-Bot Protection
Every store has different anti-scraping measures. The pipeline handles Cloudflare challenges, CAPTCHAs, rate limiting patterns, and dynamic content loading differently per store.
3. Speed vs Completeness
Users expect results in 2 seconds, but scraping 6+ stores takes time. I built a streaming architecture where results appear as they come in, with the best matches surfacing first.
Results So Far
- Launched 2 weeks ago
- 500+ products searched daily
- Average price savings per search: 22%
- Users from 20+ countries (mainly US, China, India)
What's Next
- Adding more stores (Costco, Target, ASOS, Zalando)
- Price history tracking and drop alerts
- Mobile app with barcode scanning
- Browser extension for one-click price comparison
Lessons for Solo Makers
- Ship fast, iterate faster — First version was just 2 stores and a basic text search. I shipped it in 3 days.
- AI is the differentiator — Without the product matching AI, this would just be another comparison site. The AI makes it feel magical.
- Pricing transparency is a superpower — Users love knowing they're actually getting the best deal. The trust builds quickly.
Thanks for reading! I'd love feedback from the dev community — especially on the scraping architecture and the AI matching. What would you do differently?
Try it out: shopai.fun
Follow my solo maker journey: @yjj061014