Apify scraping to content: using a scraper as a content source in 2026

Published 2026-05-19 · Updated 2026-10-23

Apify is a hosted scraping platform: it runs headless-browser Actors against any public web property on a schedule and emits structured JSON. Most teams reach for it to build lead lists. The under-used application is as a content input source — point a scraper at public places where your industry argues and whose terms allow it (Hacker News, a niche forum, a competitor's blog feed) and you have a steady stream of raw material that a generation pipeline can turn into timely commentary.

The reason this matters is timing: commentary on a live discussion only works if you move fast enough to catch the attention wave. The bottleneck was never the writing — it was noticing the thread, reading it, and getting a take out before the moment passed. A scraper closes that gap. The honest caveat, which this guide does not bury: scraped data is unverified by construction, so the workflow that ingests it has to be built around human verification, not bolted onto it. Kompozy has no built-in Apify connector. Its automatic input is the RSS source, and Apify can serve a dataset as an RSS feed, which is how the two connect. Of everything you can route in that way, scraped data demands the most discipline. Pairs with our webhook-pipelines spoke for routing business events in the same way and gmail-to-content for the inbox source.

Where does Apify sit in a content pipeline?

Apify is the data-acquisition layer, not a content tool. It produces one thing: structured JSON rows representing whatever you scraped. Everything that makes those rows safe to publish from — the voice, the fact-checking, the attribution, the platform shaping — happens downstream in the content engine, not in Apify. Treating the scraper as "the automation" is the most common way this goes wrong; the scraper is the easy part and the verification is the load-bearing part.

The full path, in order: an Apify Actor runs on a schedule and scrapes a public source into structured items. Apify's dataset items API can return those items as an RSS feed, and Kompozy's RSS source polls that feed, so each new item lands as raw_content. Nothing marks it as scraped. Kompozy treats it like any other feed item, so the suspicion has to come from you. Claude reads each item and transforms it against your Persona Brief, producing a commentary draft rather than a summary. That draft is written under the same prompt rules as any source, including the instruction not to invent figures, but nothing checks it against the scraped item afterward, and scraped numbers and quotes are exactly the kind of claim that gets invented or mis-attributed. The draft then waits for review, or is approved automatically if Auto-Publish Generated is on, before the scheduler and publish. The scraper feeds the front of a pipeline that was already built to be careful; it does not get a shortcut around any of the careful parts.

This is the mental model to hold for the rest of this guide: Apify is a source, the same way RSS or your inbox is a source. It is the riskiest source you can route in because the data is the least trustworthy, which is why the verification discipline is heavier here than anywhere else in the cluster. See autopilot-explained for how a source like this rides the broader autopilot loop, and fact-anchor-gate for what Kompozy does and does not check on numbers.

What is worth scraping for content?

Scrape the public places where genuine discussion in your field happens; not every scrapable source produces content worth shipping. The sources that work share a shape: they are public, they are where genuine discussion happens in your field, and they surface signal (what is being argued about right now) rather than just data (a list of facts). The ones that consistently earn their compute cost:

• Industry subreddits, through the front door only. Reddit's User Agreement prohibits scraping without Reddit's prior written consent, so do not point a scraper at it. Use the subreddit's RSS feed through the RSS source, or Reddit's official API under its terms. The thread is the signal; your take is the content.
• Hacker News for B2B SaaS and dev-tooling topics. The comment threads are where the real positions get staked out. A post that extends or disputes a strong HN comment outperforms a post that merely reports the link.
• Niche industry forums — BiggerPockets for real-estate investing, Indie Hackers for bootstrapped founders, specialized Discord-to-RSS bridges for tighter communities. The narrower the forum, the higher the signal per scraped row.
• Competitor blog publish events. A scraper that watches a competitor's feed and fires when a new post lands gives you a window to publish your own differentiated angle while the topic is warm.
• Trend feeds — Google Trends exports, a spike in a tracked keyword, a rising search query. A detected spike is a prompt to generate commentary on why it is spiking, which is timely by construction.

The common thread is that you are scraping for awareness of a live conversation, then adding a position to it. The moment the goal shifts from "what should I have an opinion about" to "give me text I can repost," the workflow has crossed from content intelligence into republication, which is the line covered below.

What should you never scrape?

Never scrape anything behind authentication, such as LinkedIn, gated newsletters, or paywalled publications. These are compliance and reputation boundaries, and crossing them is how a content-automation setup turns into a legal problem. Treat this list as hard rules, not guidelines:

• Anything behind authentication. LinkedIn, gated newsletters, paywalled publications. LinkedIn's User Agreement bans using software to scrape or copy the service, and gated publications generally restrict it too. The data is not worth the exposure.
• Content scraped for verbatim republication. Pulling a blog post and reposting it as your own is plagiarism regardless of how the pipeline dresses it up. The pipeline exists to add commentary, not to launder copying.
• Personal user data. Even from public profiles, scraping names, contact details, or other PII pulls you into GDPR and CCPA territory. Content scraping should operate on discussions and ideas, not on people.
• Sources whose robots.txt or terms explicitly forbid scraping, which includes Reddit unless you have its written consent. Respecting the stated boundary is both the compliant move and the defensible one — "it was technically reachable" is not a defense.

How do you wire Apify into a content pipeline?

Kompozy has no Apify connector and no inbound webhook, so the handoff runs through a feed: Apify serves the scraped dataset as RSS, and Kompozy polls it. The steps:

1. Pick or build an Apify Actor for your target source — an HN scraper, a forum scraper, a generic site-content scraper pointed at a competitor feed. Apify's store has maintained Actors for the common cases, so you rarely write one from scratch.
2. Configure the Actor input: which URL, how many items, what minimum comment count or recency threshold counts as "trending." Filtering at the scrape step is cheaper than filtering after — you pay compute on rows you keep.
3. Schedule the Actor. Every few hours suits trend-watching; once a day is plenty for competitor-feed watching. The schedule is a direct cost lever, covered in the cost section below.
4. Get the dataset's RSS URL from Apify's Get dataset items API with format=rss, and use a dataset URL that stays the same between scheduled runs (Apify's docs cover how your Actor stores results). Open it in a browser and check that every item has a title, a description, and a link or GUID. Kompozy drafts from the title and description and uses the GUID or link to skip items it has already seen.
5. Add that URL as an RSS source in Kompozy, pick the formats it may produce, and set its max items per poll. Kompozy polls it every 15 minutes by default. It has no relevance filter of its own, so drop off-topic rows in the Actor input before they cost a generation.

Each new item becomes raw_content, and Kompozy generates a draft against the Persona Brief. Kompozy does not add attribution by itself, so write the credit rule into your Persona Brief (for example, open with the source: "Trending on Hacker News today: [thread title]. Here is where I land...") and check it in review. That attribution is not decoration; it is the difference between commentary and theft.

Why is a human fact check mandatory for scraped data?

Scraped data has no trust floor at all, which is why a human fact check is mandatory. Every other input source has some implicit trust floor: your own uploads are yours, an email you forwarded you chose to amplify, an RSS feed is a publisher's own structured output. A forum comment claiming, say, "73% of wholesalers fail in year one" is a stranger's assertion with no citation, scraped into your pipeline as if it were fact. If your commentary post repeats that number, you have published a fabricated statistic under your brand, sourced from a forum comment you never verified.

Kompozy does not run an automated fact check, so both dangers here land on your review. The first is model hallucination: the model inventing a stat that was never in the scraped item. The prompt rules tell it not to, which lowers the odds, but nothing compares the draft with the source afterward. The second is scraped-source contamination: a claim that is faithfully present in the scraped row but is itself unverified, like the forum comment's made-up 73%. No automated check could catch that one anyway, because the number does trace to the source; the source just was not telling the truth. That is why scraped content always needs a human spot-check on numeric claims, more so than a newsletter from a known publisher.

In practice this means keeping Auto-Publish Generated off in any workspace fed by a scraper, and treating every statistic in a draft as a flag for the reviewer rather than a green light. The discipline is straightforward: check that the draft did not invent a claim, and check that you are not amplifying someone else's invented claim. Skipping either one is how scraped content earns its bad reputation. More in fact-anchor-gate.

Where is the line between commentary and republication?

The legal and ethical line is clean even if the temptation to blur it is strong: scraping for awareness and adding your own substantive take is fair use and aligns with the terms of service of the platforms worth scraping; scraping for republication is plagiarism and, depending on the source, copyright infringement. Write the rules into your Persona Brief and your review checklist rather than leaving them to memory, because memory fails the day someone is in a hurry.

• Every output should credit the source: the forum and author, the competitor and post. Kompozy does not add credit on its own, so make it a Persona Brief rule and check it in review.
• Every output should add original commentary, not a summary. Tell the model in your Persona Brief to extend, dispute or reframe the source and never restate it. A summary with attribution is still a derivative work; a take is yours.
• Compare each draft with the source text before you approve it. Kompozy has no similarity check, so a reviewer is the only backstop against the model paraphrasing instead of commenting.
• Keep the scraped item open beside the draft during review, so attribution and originality get checked rather than trusted blind. This is also where contaminated stats get caught.

Commenting on a public discussion, with attribution and your own take, is the pattern to aim for, but check each platform's terms on how you collect the data first: Reddit, for one, bars scraping without written consent. The single line you cannot cross is shipping content that is verbatim, or near-verbatim, from a scraped source and presenting it as original. Kompozy has no similarity check, so that line holds only if a reviewer compares drafts with the source, even on a deadline. For the broader framing of why automated commentary still needs a human judgment layer, see content-repurposing.

How much does Apify cost for content scraping?

Apify bills on a usage model rather than a flat seat price, which is the right model for this workload but means the cost is a function of how aggressively you scrape, not how much you publish. Apify's plans include a monthly amount of prepaid platform usage, and usage beyond that is billed as you go, so the bill tracks the work your Actor runs actually consume. A forum hot-threads scrape pulling a few dozen rows a few times a day is inexpensive; a broad crawl across many large sites every hour gets expensive in a hurry. The cost lever you control is the schedule and the scope, not the destination.

Because the bill scales with scraping volume and the content engine charges separately per generation, the two costs are independent and worth budgeting separately. The trap is scheduling an Actor far more frequently than your publishing cadence justifies — scraping a forum every fifteen minutes when you publish commentary twice a day means paying compute on rows you will never use. Match the scrape frequency to how fast the source actually moves and how fast you actually ship.

Timing is the whole value here. A scraper shrinks the gap between "I noticed this thread yesterday" and "my take is live," and a take published a week late is just ordinary evergreen content. We have no measured engagement lift to quote, so judge it on your own numbers.

The scraper wiring is a one-time job. The recurring work is tuning the relevance filter, so you only generate on rows worth commenting on, and spot-checking every number that survives, so you do not amplify someone else's fabrication.

When should you not use a scraper as a source?

Scraping is the right source for a specific job — timely commentary on public industry discussion — and the wrong source for several others. Knowing the boundary keeps you from building a fragile, compliance-exposed pipeline to do something a simpler source does better:

• If your content is evergreen rather than reactive, a scraper adds risk for no benefit. Evergreen content has no timing wave to catch, so the scraped-data verification burden buys you nothing. Use your own uploads or a Persona Brief topic pool instead.
• If your best source is a publisher you trust, use RSS or the inbox, not a scraper. A newsletter from a known author carries a trust floor a scrape never will, so it needs less verification overhead. Reserve scraping for sources where no trust floor exists.
• If you cannot commit to the human spot-check on numerics, do not scrape. Kompozy does not check drafts against the source, and nothing can catch contamination; a scraping pipeline with no human pass on scraped numbers will eventually amplify someone's made-up statistic. No spot-check, no scraping.
• If the only sources worth scraping are gated or paywalled, the answer is to engage natively and link out, not to ingest. The compliance exposure of scraping behind a wall is never worth the content.

When is a scraper a good content source?

When the source permits it and you treat the output as what it is: a fast feed of unverified public discussion that earns its keep on timing. Point an Apify Actor at public places where your industry argues and whose terms allow scraping, expose the results as an RSS feed for a pipeline built around a Persona Brief and a human review step, ship commentary that extends the conversation rather than copying it, and spot-check every number because nothing automated can vouch for a stranger's claim. Do those things and scraping is one of the strongest reactive-content engines available. Skip the verification layer and it is the fastest way to publish a fabricated statistic under your own name. Start with pricing to size the content-engine tier, and read webhook-pipelines for routing business events in through a feed the same way.

What does this look like in Kompozy?

An infographic photo post generated in the Kompozy workspace, the kind of commentary asset a scraped discussion can feed once its numbers are checked.

Who is this for, and who should skip it?

Best for:

• B2B and niche brands that publish timely commentary on public industry discussion
• Teams that will spot-check every scraped statistic before it ships
• Operators who watch competitor blog feeds for topics

Not ideal for:

• Evergreen content with no timing advantage; use your own uploads or a topic pool
• Anyone wanting Reddit or LinkedIn data without permission
• Teams that cannot commit to human review of scraped numbers

Frequently asked questions

How do we know this?

We read Reddit's and LinkedIn's user agreements, Apify's pricing page and Apify's dataset items API docs on 2026-10-23 and rewrote the page to match them. The same day we checked Kompozy's code: it has no Apify connector or inbound webhook, so a scraper reaches it through its RSS source. Unsourced engagement, timing and similarity figures were removed.

Sources

1. Reddit — User Agreement (scraping without consent prohibited) (read 2026-10-23)
2. LinkedIn — User Agreement (scraping prohibited) (read 2026-10-23)
3. Apify — Pricing (read 2026-10-23)
4. Apify API — Get dataset items (format=rss) (read 2026-10-23)

Which other content automation guides should you read next?

• Webhook content pipelines: triggering generation from external events — Kompozy has no inbound webhook. Send business events through Zapier into an RSS feed it polls, and send finished blog posts out as a signed webhook.
• RSS-to-social automation: the complete setup for podcast and blog feeds — How to wire RSS feeds (podcast hosts, Substack, Ghost, WordPress) directly into a content generation pipeline that fans out to 9 platforms automatically.
• Gmail-to-content automation: turning newsletters and emails into social posts — Forward newsletters and threads to a workspace ingest address with an allowed-senders list. Kompozy still shows inbound delivery as pending provider setup.
• Content automation in 2026: definition, mechanics, and what it is not — The clean definition of content automation, what separates it from scheduling, and the 4 layers every real automation pipeline needs.
• The 7 ways content automation breaks (and how to detect each) — Drift, hallucinations, platform deprecations, OAuth expirations, rate-limit blocks, queue overflow, voice degradation — the 7 failure modes every automation operator must monitor.

Which related Kompozy topics cover this next?

• Autonomous Content Creation — How Kompozy autopilot really works: one workspace switch that approves and schedules generated posts, the checks that run before they publish, and a 14-day ramp for turning it on.
• AI Content Repurposing — The complete methodology for turning one source into 25-35 pieces of native-format content across every platform — without producing AI slop.

← Back to Content Automation overview · Get started →