How to Build an AI Competitor Monitoring Agent in 2026

September 6, 2026 · AI Automation Tutorials, Competitor Research, Solopreneur Systems

Competitor monitoring is one of those tasks every solopreneur knows they should do, but almost nobody does consistently.

You check a competitor’s website once. Maybe you skim their pricing page. Maybe you follow them on X. Then three months later you realize they launched a new offer, changed positioning, added a lead magnet, started running ads, or quietly moved into your niche.

The fix is not another $199/month competitive intelligence platform. For most indie hackers and lean operators, you need a simple AI agent that watches a small list of competitors, detects meaningful changes, summarizes them, and sends you a short report.

This tutorial shows you how to build that system with Node.js, scheduled scripts, web scraping, and an LLM API. The goal is not to collect infinite data. The goal is to surface useful changes before they become surprises.

What the Competitor Monitoring Agent Will Do

We are going to build a lightweight AI agent that:

• Tracks a list of competitor URLs
• Fetches their homepage, pricing page, blog, or changelog
• Extracts readable text from each page
• Compares the latest version against the previous version
• Uses AI to summarize meaningful changes
• Sends a daily or weekly report
• Stores snapshots locally so you can review history later

This is enough for most solopreneurs. You do not need enterprise-grade monitoring unless you are tracking hundreds of companies. If you are tracking 5-25 competitors, a script like this can save hours every month.

Step 1: Decide What You Actually Need to Monitor

Do not monitor everything. That creates noise. Pick the pages that reveal business strategy.

Good pages to track:

• Homepage: positioning, headline, target customer, major launches
• Pricing page: plan changes, discounts, packaging, annual pricing
• Blog: SEO strategy, new content angles, keyword focus
• Changelog: product velocity and new features
• Docs: technical positioning and supported integrations
• Landing pages: paid campaign angles and new offers

For example, if you sell AI automation templates, you might track competitors’ pricing pages, Gumroad profiles, feature pages, and new blog posts. If you are building a content site like TheOpsDesk.ai, you might track article topics, lead magnets, and affiliate positioning.

Step 2: Create a Competitor List

Create a file called competitors.json:

{
  "competitors": [
    {
      "name": "Example Competitor",
      "urls": [
        "https://example.com/",
        "https://example.com/pricing",
        "https://example.com/blog"
      ]
    },
    {
      "name": "Another Tool",
      "urls": [
        "https://anothertool.com/",
        "https://anothertool.com/changelog"
      ]
    }
  ]
}

Start with 5-10 competitors. Add more only after the reports are useful. A common mistake is trying to track 100 companies before the workflow is tuned.

Step 3: Set Up the Node.js Project

Create a new folder:

mkdir competitor-agent
cd competitor-agent
npm init -y
npm install axios cheerio turndown dotenv openai

Create this folder structure:

competitor-agent/
  competitors.json
  .env
  monitor.js
  snapshots/
  reports/

Add your API key to .env:

OPENAI_API_KEY=your_api_key_here

You can swap OpenAI for Anthropic, Gemini, Ollama, or another model provider. The architecture stays the same.

Step 4: Fetch and Clean Web Pages

Create monitor.js and start with the page fetcher:

import fs from "fs";
import path from "path";
import axios from "axios";
import * as cheerio from "cheerio";
import TurndownService from "turndown";
import OpenAI from "openai";
import dotenv from "dotenv";

dotenv.config();

const openai = new OpenAI({ apiKey: process.env.OPENAI_API_KEY });
const turndown = new TurndownService();

const SNAPSHOT_DIR = "./snapshots";
const REPORT_DIR = "./reports";

fs.mkdirSync(SNAPSHOT_DIR, { recursive: true });
fs.mkdirSync(REPORT_DIR, { recursive: true });

function safeFileName(input) {
  return input.replace(/[^a-z0-9]/gi, "_").toLowerCase();
}

async function fetchPage(url) {
  const response = await axios.get(url, {
    timeout: 20000,
    headers: {
      "User-Agent": "Mozilla/5.0 competitor-monitoring-agent"
    }
  });

  const $ = cheerio.load(response.data);

  $("script, style, noscript, iframe, svg").remove();
  $("nav, footer, header").remove();

  const title = $("title").text().trim();
  const bodyText = turndown.turndown($("body").html() || "");

  return {
    url,
    title,
    content: bodyText.replace(/\n{3,}/g, "\n\n").trim()
  };
}

This strips obvious junk and converts the page to Markdown-like text. It will not work perfectly for every site, but it is good enough for a lean monitoring agent.

Step 5: Save and Compare Snapshots

Add snapshot functions:

function getSnapshotPath(competitorName, url) {
  return path.join(
    SNAPSHOT_DIR,
    `${safeFileName(competitorName)}_${safeFileName(url)}.txt`
  );
}

function loadPreviousSnapshot(competitorName, url) {
  const filePath = getSnapshotPath(competitorName, url);
  if (!fs.existsSync(filePath)) return null;
  return fs.readFileSync(filePath, "utf8");
}

function saveSnapshot(competitorName, url, content) {
  const filePath = getSnapshotPath(competitorName, url);
  fs.writeFileSync(filePath, content, "utf8");
}

This keeps one latest snapshot per tracked URL. If you want full history, add timestamps to the filenames and store a separate latest pointer. For most small operators, latest-versus-previous is enough.

Step 6: Ask AI to Summarize Meaningful Changes

Now add the AI comparison function:

async function summarizeChanges({ competitorName, url, oldContent, newContent }) {
  if (!oldContent) {
    return `Initial snapshot created for ${competitorName}: ${url}`;
  }

  const prompt = `
You are a practical competitor intelligence analyst for a solopreneur.
Compare the OLD and NEW page content below.

Focus only on meaningful business changes:
- positioning changes
- pricing or offer changes
- new features
- new products
- new lead magnets
- SEO/content strategy shifts
- customer segment changes
- urgency for the founder

Ignore tiny copy edits, cookie banners, navigation noise, and formatting changes.

Return:
1. Change summary
2. Why it matters
3. Suggested action
4. Urgency: low, medium, or high

Competitor: ${competitorName}
URL: ${url}

OLD CONTENT:
${oldContent.slice(0, 12000)}

NEW CONTENT:
${newContent.slice(0, 12000)}
`;

  const completion = await openai.chat.completions.create({
    model: "gpt-4o-mini",
    messages: [{ role: "user", content: prompt }],
    temperature: 0.2
  });

  return completion.choices[0].message.content.trim();
}

The prompt matters. You are not asking the model to summarize a webpage. You are asking it to decide whether the change has business relevance.

Step 7: Run the Monitor

Add the main function:

async function run() {
  const config = JSON.parse(fs.readFileSync("./competitors.json", "utf8"));
  const reportItems = [];

  for (const competitor of config.competitors) {
    for (const url of competitor.urls) {
      console.log(`Checking ${competitor.name}: ${url}`);

      try {
        const page = await fetchPage(url);
        const previous = loadPreviousSnapshot(competitor.name, url);
        const current = `# ${page.title}\n\nURL: ${url}\n\n${page.content}`;

        const summary = await summarizeChanges({
          competitorName: competitor.name,
          url,
          oldContent: previous,
          newContent: current
        });

        reportItems.push({
          competitor: competitor.name,
          url,
          summary
        });

        saveSnapshot(competitor.name, url, current);
      } catch (error) {
        reportItems.push({
          competitor: competitor.name,
          url,
          summary: `Error checking page: ${error.message}`
        });
      }
    }
  }

  const date = new Date().toISOString().slice(0, 10);
  const report = formatReport(date, reportItems);
  fs.writeFileSync(path.join(REPORT_DIR, `${date}.md`), report, "utf8");
  console.log(report);
}

function formatReport(date, items) {
  return `# Competitor Monitoring Report — ${date}\n\n` +
    items.map(item => `## ${item.competitor}\nURL: ${item.url}\n\n${item.summary}\n`).join("\n---\n\n");
}

run();

Run it:

node monitor.js

The first run creates baseline snapshots. The second run starts producing useful comparisons.

Step 8: Schedule the Agent

For a daily report at 8 AM, use cron:

crontab -e

Add:

0 8 * * * cd /path/to/competitor-agent && /usr/local/bin/node monitor.js >> monitor.log 2>&1

For most solopreneurs, daily is enough. If you are in a fast-moving niche, run it every 6 hours. If you are tracking blogs and pricing pages only, weekly may be better.

Step 9: Send Reports to Yourself

The simplest version writes Markdown files to reports/. If you want delivery, add email, Slack, Discord, Telegram, or a private dashboard.

For example, a Discord webhook sender:

async function sendDiscordReport(report) {
  if (!process.env.DISCORD_WEBHOOK_URL) return;

  await axios.post(process.env.DISCORD_WEBHOOK_URL, {
    content: report.slice(0, 1900)
  });
}

Then call it after creating the report:

await sendDiscordReport(report);

Add this to .env:

DISCORD_WEBHOOK_URL=https://discord.com/api/webhooks/...

If the report is longer than Discord allows, either split it into chunks or send only high-priority changes.

Step 10: Reduce Noise With a Relevance Filter

Once the agent is running, your next job is reducing false positives. Competitor sites change constantly because of timestamps, testimonials, rotating banners, tracking scripts, and CMS widgets.

Add a minimum relevance rule to the AI prompt:

If there are no meaningful business changes, return exactly:
NO_MEANINGFUL_CHANGE

Then filter results:

if (summary !== "NO_MEANINGFUL_CHANGE") {
  reportItems.push({ competitor: competitor.name, url, summary });
}

This one change makes the system much more useful. A competitor monitoring agent should interrupt you only when something matters.

What to Monitor Beyond Websites

Web pages are the easiest starting point, but you can expand the agent later.

Do not add every source at once. Add one new source only when the existing report is consistently useful.

How This Helps a Lean Business

A good competitor monitoring agent is not about copying. It is about staying oriented.

It helps you answer:

• Are competitors changing their pricing?
• Are they moving upmarket or downmarket?
• What features are becoming table stakes?
• Which topics are they publishing around?
• Are they launching templates, courses, communities, or service offers?
• Are they repositioning toward a niche you care about?

For a solopreneur, this is leverage. You can spend 20 minutes reviewing a clean weekly report instead of randomly checking competitor sites when you remember.

If you sell digital products, templates, or automation systems, you can also use the report to identify product gaps. For example, if three competitors are writing about AI operations dashboards but none offer a practical implementation kit, that might become your next Gumroad product. You can browse related operator-focused products at opsdesk0.gumroad.com for examples of how small digital offers can support a broader content and automation business.

Practical Guardrails

Keep this system clean and ethical:

• Respect robots.txt and website terms where applicable
• Do not hammer sites with aggressive polling
• Do not scrape private or gated data
• Do not collect personal data unless you have a legitimate reason
• Use reasonable request intervals
• Cache results instead of refetching constantly

A daily or weekly check of public pages is usually enough. You are building a business radar, not a surveillance machine.

Recommended Upgrade Path

Start simple, then improve only where needed:

• Version 1: Track public webpages and write Markdown reports
• Version 2: Add email or Discord delivery
• Version 3: Add relevance scoring and urgency filtering
• Version 4: Track RSS feeds, changelogs, and YouTube channels
• Version 5: Store changes in SQLite and build a small dashboard

The best version is the one you actually read. A boring daily report with three useful bullets beats a complex dashboard nobody opens.

Final Takeaway

An AI competitor monitoring agent gives you consistent market awareness without turning competitor research into another manual chore.

The setup is simple: list competitors, fetch pages, store snapshots, compare changes with AI, and send yourself a short report. Once it runs on a schedule, you get a lightweight intelligence system that supports content planning, product strategy, pricing decisions, and positioning.

For lean operators, that is the point of AI automation: not replacing judgment, but making sure your judgment has better inputs.