Jobs Tracker

Technical reference & replication guide

How this dashboard is built: every job-board URL, the deduplication logic, the anti-bot tooling, every bug fixed and what to watch for. Snapshot 2026-06-02 — 35 tickers, 62 distinct (ticker, ats, board) leaves, ~39,000 open reqs/day.

Part 1 — Per-ticker board inventory

1.1 Greenhouse-hosted boards (24)

Public, unauthenticated. Listing JSON at https://boards-api.greenhouse.io/v1/boards/<slug>/jobs; per-job detail at …/jobs/<job_id>.

1.2 Workday-hosted (6)

POST to https://<tenant>.<subdomain>.myworkdayjobs.com/wday/cxs/<tenant>/<site>/jobs with body {"appliedFacets":{}, "limit":20, "offset":0, "searchText":""}. The subdomain (wd1/wd3/wd5/wd108) is the Workday cluster. Pagination caps at 10000.

EXPE's Workday is the single consolidated portal for the entire Expedia Group (Hotels.com, Vrbo, Travelocity, Hotwire, Orbitz, Ebookers, CheapTickets, CarRentals, Wotif, Expedia Cruises). No per-brand careers subdomain exists.

1.3 Lever-hosted (2)

Public API https://api.lever.co/v0/postings/<company>?mode=json — returns descriptionPlain (clean text), description (HTML), createdAt (Unix ms), categories.{team, commitment, location}.

BMBL was Lever-hosted (bumbleinc) from ~May 2026 until ~Sep 2026, when the Lever board went empty and team.bumble.com switched to Ashby — see §1.4.

1.4 Ashby-hosted (2)

GraphQL: https://jobs.ashbyhq.com/api/non-user-graphql?op=ApiJobBoardWithTeams, body {"operationName":"ApiJobBoardWithTeams","variables":{"organizationHostedJobsPageName":"kayak"}}.

1.5 Rippling-hosted (1, post-migration)

OPEN migrated Greenhouse → Rippling on 2026-05-18. Old Greenhouse board now 404s. Clean public API:

1.6 Phenom-hosted (2)

Phenom is a third-party career-site platform. Its paginated API caps at ~500 with shuffle-induced duplicates — the bypass is the SEO sitemap (see §4.6). Each detail page embeds JSON-LD JobPosting.

1.7 Custom / first-party scrapers

AMZN — Amazon corp (amazon_jobs) — 4 passes, 8+ board values

• Pass 1 (corp ~21k): https://www.amazon.jobs/en/search.json — sharded by 60 business_category buckets (endpoint hard-caps hits=10000 globally). Saves board='amazon', department=<category>. The department drives the AWS-only / Fulfillment-only virtual pills.
• Pass 2 (hourly): https://hiring.amazon.com/app/api/jobs/ (US), hiring.amazon.ca (CA). Spikes to 1000s at peak season.
• Pass 3 (DSP drivers): https://dspjobhub.com/vacancies-sitemap.xml — NOT Amazon employees (independent contractors) but a proxy for delivery demand.
• Pass 4 (salary sampling): 10% random detail fetch https://www.amazon.jobs/en/jobs/<id>.

AMZN — Whole Foods / One Medical (subsidiaries)

• Whole Foods: https://careers.wholefoodsmarket.com/api/v1/posts — ~2,761 retail/store. Detail enrichment intentionally skipped (too expensive nightly).
• One Medical: https://www.onemedical.com/about/careers/jobs/ — ~283 healthcare/clinical, JSON-LD detail.

BKNG — Booking / Priceline

• Booking: https://careers.booking.com/api/jobs?count=100&offset=0
• Priceline: sitemap https://careers.priceline.com/job-sitemap.xml; custom <p>/<li> extractor inside <article> (Priceline dropped JSON-LD May 2026).

DASH — Deliveroo / Wolt

• Deliveroo: sitemap https://careers.deliveroo.co.uk/role-sitemap.xml (fetched via Bright Data — CF-challenged from datacenter IP); custom <p> extractor (JSON-LD now only @graph WebPage/BreadcrumbList).
• Wolt: https://wolt.com/en/jobs?country=ALL — DoorDash's EMEA+APAC+LATAM markets, jobs in initial HTML.

GOOGL — Google Careers / Waymo

• Google: https://www.google.com/about/careers/applications/jobs/results/?page=<N> — server-rendered AF_initDataCallback({key:'ds:1', data:[...]}); 3-layer Oxylabs fallback (direct → rotating → sticky). Embedded totalHits in last array element.
• Waymo: sitemap https://careers.withwaymo.com/sitemap.xml + JSON-LD detail. CloudFront WAF rejects Python TLS → curl_cffi impersonate=chrome120 + Bright Data fallback.

META — Meta Careers (most expensive scraper)

• Listing GraphQL POST https://www.metacareers.com/api/graphql/ with doc_id=29615178951461218 (rotates ~quarterly). Requires Accept-Encoding: br (brotli) or returns 400.
• Detail https://www.metacareers.com/profile/job_details/<jid> — JSON-LD JobPosting (description) + public_compensation regex (salary). Both off one fetch.

Other single-board custom scrapers

• MTCH→Tinder: https://www.lifeattinder.com/sitemap.xml + JSON-LD detail.
• SPOT: WordPress API https://api.lifeatspotify.com/wp-json/animal/v1/job/search (no description) + __NEXT_DATA__ detail at props.pageProps.job.content.descriptionHtml.
• UBER: jobs.uber.com/api/jobs/search/?page=N&pagesize=100 — first-party JSON (title/teams/locations/salary/DisplayDate/description HTML; pagesize honoured to 100 → 6 calls) fetched via Bright Data Web Unlocker because Cloudflare challenges direct + Oxylabs; /en/jobs/sitemap.xml is unchallenged and supplies the authoritative req_id set. The old www.uber.com/api/loadSearchJobsResults RPC was removed ~Aug 2026. Corp only (drivers not captured).
• UMG: 3 regional Workday tenants — umusicus.wd1, umusicuk.wd1, umusicapac.wd1.
• SEAT: https://vividseats.applytojob.com/apply (Greenhouse-style).
• W→Wayfair: BD Scraping Browser only (PerimeterX); backing XHR https://www.wayfair.com/a/careers/careers/job_search_data has descriptions inline.
• WBTN→Webtoon: English subs via Lever API api.lever.co/v0/postings/wattpad/<uuid>; Korean subs (NAVER, STUDIO_LICO) via RSC self.__next_f.push([1, "…"]) payload (pick the push with the most HTML tags).

Part 2 — Tools used

Typical nightly cost: <$0.10 Bright Data + ~$5/mo Oxylabs. Worst case if multiple CDNs degrade: ~$2.50/night (capped by BD_MAX_CALLS_PER_RUN=1500).

Part 3 — Deduplication

Layer 1 — per-scrape in-memory (storage._dedup)

Key is (ticker, req_id); first occurrence wins. Collapses the same req appearing under multiple board values (Amazon category shards, TTWO sibling boards, Rippling cross-listed UUIDs).

def _dedup(postings):
    seen = set(); out = []
    for p in postings:
        key = (p.ticker, p.req_id)
        if key in seen: continue
        seen.add(key); out.append(p)
    return out

Layer 2 — per-day in-DB write scope (storage.record_run)

DELETE is scoped to (run_date, ticker, ats, board IN <boards-in-result>) — so multiple yaml entries sharing (ticker, ats) but different boards (CVNA carvana+adesa, BKNG agoda+opentable, AMZN's 8 boards) don't wipe each other.

Layer 3 — virtual leaves (computed, NOT re-scraped)

AMZN's "AWS only" and "Fulfillment only" pills are post-hoc department filters on the same Amazon rows — NOT separate scrapes:

• AWS only = department='aws' (~7,196)
• Fulfillment only = department IN (fulfillment-and-operations, fulfillment-ops, fulfillment-ops-team, transportation-and-logistics, transportation-shipping-logistics)

Each virtual leaf carries virtual:True so the "All" total excludes them — else AMZN would double-count by ~11,600. (This was the 37,825-vs-26,163 bug.)

Anomaly-check scope

Keyed by yaml disambiguator (board/tenant/company/org), not (ticker, ats) — avoids false "95% drop" when one of CVNA's two greenhouse boards reports its smaller count alone. Also now consults the runs table + re-runs after extended retries so transient Workday flake (ETSY/SNAP/UMG/Z) doesn't false-alarm.

Part 4 — Errors fixed (root causes)

4.1 GOOGL — JSON-unescape corruption

Descriptions stored with literal

garbage. Google's AF_initDataCallback serializes HTML with JSON unicode escapes; normalize_description's tag regex never saw a real <. fix _js_unescape() before normalize, then strip leading/trailing JSON-array wrappers. watch spot-check last 80 chars of a GOOGL desc — should be real text, not "],2],.

4.2 GOOGL — taxonomy gap (45% → 25% unclassified)

Title parser missed orgs in trailing parentheticals, comma-less titles, and orgs not in KNOWN_ORGS. fix strip (English, Japanese) suffixes, expand KNOWN_ORGS to 60+ orgs, mine "About the team" paragraph as fallback, preserve acronym casing (AI not Ai). watch top depts should be Google Cloud / Data Center / GCS / YouTube / DeepMind.

4.3 META — description never captured

Listing GraphQL returns only 5 fields. fix parse JSON-LD JobPosting off the Pass-2 detail HTML we already fetch for comp (no extra requests). watch doc_id rotates ~quarterly; 0/N enriched in logs → re-capture HAR.

4.4 Waymo — TLS fingerprint rejected

CloudFront WAF profiles the TLS handshake (JA3/JA4), not just headers → 202/405. fix curl_cffi.Session(impersonate="chrome120") as the direct client (free, ~95%); Bright Data Web Unlocker sequential fallback for stragglers. Went 9/410 → 410/410.

4.5 Wayfair — PerimeterX placeholders

Returned 390 Role-#N stubs. curl_cffi AND BD Web Unlocker both fail. fix BD Scraping Browser (real Chrome + residential IP); the backing XHR has descriptions inline. 0 → 377 real jobs at $0.24/mo.

4.6 EBAY — Phenom pagination cap (498 → 576)

Phenom's ?from=N&s=1 shuffles results so duplicates fill page slots; ~78 jobs unreachable. fix use the SEO sitemap (sitemap1.xml 500 + sitemap2.xml 77 = 576). Lightsail fetches sitemaps directly; only the search API was IP-blocked.

4.7 Deliveroo — JSON-LD dropped (0 → 100%)

JobPosting JSON-LD replaced by @graph with only WebPage/BreadcrumbList; helper silently returned blanks. Plus CF challenged the sitemap from datacenter IP. fix sitemap via BD, custom <p> extractor, concurrency 8→4 (CF burst limit).

4.8 AMZN — recruiter boilerplate dominating excerpts

Concatenated description + description_short both led with "Transform the future…" marketing, repeated across 100x-reposted reqs. fix restructure to basic_qualifications → responsibilities (marker-split) → preferred_qualifications; smart-excerpt at export time anchors on "Key job responsibilities" markers.

4.9 Anomaly false-positives (ETSY/SNAP/UMG/Z)

Anomaly check fired before extended retries recovered transient Workday flake. fix consult the runs table (ok-run = not missing) and re-run after extended retries.

4.10 Dashboard — AMZN double-count

All-total summed virtual leaves (AWS + Fulfillment) on top of the Amazon rows they filter → 37,825 vs real 26,163. fix exclude virtual:True leaves from the sum.

4.11 UTC midnight crossing — date-stamp bug

A regen after 00:00 UTC creates a scrape_exports/<tomorrow>/ dir; web server routes / there showing partial data. watch when backfilling near midnight UTC (19:00-20:00 ET), remove the rogue dir + restamp rows.

4.12 Phenom — proxy env-var quote bug

OXYLABS_USER="..." with literal quotes → %22 in proxy URL → httpx InvalidURL. fix proxy.py strips surrounding quotes universally. watch don't quote env-var values.

Part 5 — What to watch for

Source-side changes that break silently

• META doc_id rotation: comp pass: 0/N enriched → re-capture HAR.
• Workday tenant migrations: 404 → check <company>.workdayjobs.com for the new redirect.
• ATS migrations: Greenhouse 404 → check the careers-page Apply button host (OPEN went Greenhouse→Rippling, BMBL Workday→Lever→Ashby, U Greenhouse→Workday, UBER www.uber.com RPC→jobs.uber.com).
• Phenom sitemap restructure: parser follows <sitemapindex>, but if the root URL moves, update yaml.
• TLS fingerprint escalation: bump curl_cffi profile (chrome120 → chrome125+) every ~12 months.

If you replicate this

Minimum viable (~95% of dollar value, zero external cost): Greenhouse + Workday + Lever scrapers cover 32 of 35 tickers (~3 files, ~600 lines). Add Amazon corp (biggest single source) + Google Careers (regex on AF_initDataCallback). One $40/mo Lightsail box, SQLite, Cloudflare DNS, Let's Encrypt.

Don't repeat these mistakes:

• Don't use playwright + Oxylabs proxy= — Chromium hits ERR_TUNNEL_CONNECTION_FAILED. Use httpx proxy=.
• Don't scope the storage DELETE to (run_date, ticker, ats) when entries share that but differ on board.
• Don't sum "All" totals over computed/virtual leaves.
• Don't run the anomaly check before extended retries complete.
• Don't try Wayfair without BD Scraping Browser — only thing that beats modern PerimeterX.
• Don't run parallel agents on the same source files without disjoint scopes (race conditions).

Full version with every URL and code snippet lives in REPLICATION_GUIDE.md in the project repo. ↑ back to top