Build or Skip

mturk replacement labeling marketplace

We said SKIP on August 27, 2026. Not settled — due August 27, 2027.

Read this with the caveat. We tested the engine that produced this verdict against 292 launches whose outcomes we already knew, and could not show it predicted which survived. Some of its data sources were also dead at the time of scoring. The verdict stays up, dated and unedited, because a record you can quietly revise is not a record — but it is worth less than it looked when it was written.

Demand for labeling is commercially proven by five paying competitors, but there is zero organic signal in social or search to bootstrap from, and the product form is the worst possible one for a solo founder: a two-sided marketplace with cold-start dynamics, payout rails, and fraud control against funded incumbents. The identified gaps are real but capital- and ops-heavy. SKIP as a marketplace; the only defensible version is a narrow managed-service or tooling wedge.

Was there demand

5out of 10

Commercial demand is proven by five established players charging money for crowd labeling, but every discovery channel outside competitors is empty — zero HN discussions, no keyword pull, and the Reddit hits are about Amazon grocery stores, not labeling work.

Could a builder win it

3out of 10

This is a two-sided marketplace with a hard cold-start problem: no requester pays for a worker pool that doesn't exist, and no worker shows up for tasks that don't exist, while incumbents are funded/enterprise operations (Appen, Lionbridge) rather than beatable indie tools. The listed gaps are real but each requires payments infrastructure, fraud/quality control, and global payout rails — heavy engineering for a solo founder with no supply side.

The case against this verdict

The strongest case for building: the vertical-specialization gap (e.g. medical imaging or code-review labeling with certified workers) can be started as a services business, not a marketplace — hand-recruit 20 vetted labelers, sell managed labeling to a handful of AI startups at high margins, and only productize once flow is steady. That path sidesteps the cold-start problem entirely and AI-training data spend is genuinely growing, so a solo operator could reach real revenue before ever building a two-sided platform.

Who was already there

  • ProlificFocused on research/academic studies; higher payment rates reduce profit margins; smaller task volume; stricter qualification requirements limit worker pool
  • AppenHigh barrier to entry for requesters; expensive pricing; slower onboarding; enterprise-focused limits SMB access; less transparent pricing model
  • ClickworkerOlder UI/UX; limited transparency; payment delays reported; worker support gaps; smaller requester base than competitors
  • Connect AI (CloudResearch)Research-focused; smaller worker pool; higher costs; limited task types outside academic research
  • Lionbridge AIProject-based hiring; less flexible for casual workers; higher barrier to entry; limited self-service options

Other verdicts

What is worth more than this page. The register records what became of 6,266 real launches. No engine has to be right for that to be true.