Table of Contents

In the evolving world of AI orchestration, reactive chatbots are giving way to proactive agents. These agents don’t just wait for a prompt; they actively search, extract, analyze data, and trigger workflows autonomously. Today, we’re going to build a fully autonomous, proactive web-scraping agent using Python, the powerful Firecrawl API, and enterprise-grade Azure OpenAI. By the end of this tutorial, you will have a system that automatically monitors a competitor’s pricing page, extracts the data into a structured JSON format, compares it against historical state stored in Azure Cosmos DB, and alerts you via a webhook if anything changes.

Why Combine Firecrawl, Python, and Azure OpenAI?

Traditional web scraping tools like BeautifulSoup, Puppeteer, or Selenium are notoriously brittle. They break easily when websites change their DOM layout, implement anti-bot protection, or rely heavily on dynamic client-side JavaScript rendering. Firecrawl solves this massive headache by acting as a high-level extraction API. It effortlessly converts any URL—even complex Single Page Applications (SPAs)—into clean, LLM-ready markdown. It handles proxies, JS rendering, and rate limits out of the box, ensuring your data pipelines don’t fail silently in production.

Pairing this robust scraping capability with Azure OpenAI provides the enterprise security and scalability required for serious production workflows. Unlike consumer-grade API tiers which can suffer from throttling and data privacy concerns, Azure’s infrastructure ensures guaranteed throughput, strict data privacy compliance (your data isn’t used to train base models), and seamless integration with the rest of the Azure ecosystem. Python serves as the perfect glue language, offering native SDKs for both Firecrawl and Azure.

Before we start, ensure you have your Firecrawl API key and your Azure OpenAI Endpoint, API Key, and Deployment Name ready. You can grab a free API key directly from the Firecrawl dashboard to test this out.

Step 1: Setting Up the Python Environment

First, we need to install the necessary dependencies using pip. We’ll need the Firecrawl SDK for web extraction, the OpenAI library (which natively supports Azure endpoints), the Azure Cosmos DB SDK for state management, and `python-dotenv` for securely managing our environment variables.

pip install firecrawl-py openai azure-cosmos python-dotenv requests

Create a .env file in your project root to securely store your credentials. This prevents you from accidentally hardcoding sensitive keys in your source code.

FIRECRAWL_API_KEY=your_firecrawl_key
AZURE_OPENAI_API_KEY=your_azure_openai_key
AZURE_OPENAI_ENDPOINT=https://your-resource-name.openai.azure.com/
AZURE_OPENAI_DEPLOYMENT_NAME=gpt-4o
COSMOS_DB_ENDPOINT=https://your-cosmos-db.documents.azure.com:443/
COSMOS_DB_KEY=your_cosmos_key
SLACK_WEBHOOK_URL=https://hooks.slack.com/services/...

Step 2: Building the Extraction Logic with Firecrawl

We’ll start by initializing Firecrawl to scrape a target website. In this example, we’ll imagine we’re building a competitive intelligence agent that monitors a competitor’s pricing page for updates. The beauty of Firecrawl is that it returns perfectly formatted Markdown. Large Language Models (LLMs) digest Markdown significantly better than raw, nested HTML tags because Markdown strips away the noise (scripts, styles, nested divs) while preserving the semantic structure (headings, lists, tables).

import os
from firecrawl import FirecrawlApp
from dotenv import load_dotenv

load_dotenv()

# Initialize Firecrawl
firecrawl = FirecrawlApp(api_key=os.getenv("FIRECRAWL_API_KEY"))

def scrape_target_url(url):
    print(f"Scraping target URL: {url}...")
    try:
        scrape_result = firecrawl.scrape_url(
            url, 
            params={'formats': ['markdown']}
        )
        return scrape_result.get('markdown', '')
    except Exception as e:
        print(f"Error scraping URL: {e}")
        return None

# Example target
competitor_url = "https://example.com/pricing"
markdown_content = scrape_target_url(competitor_url)

Step 3: Structuring Unstructured Data with Azure OpenAI

Now that we have clean markdown, we feed it into an Azure OpenAI model (like GPT-4o) to extract structured insights. We’ll prompt the model to act as a competitive intelligence agent and analyze the pricing tiers. To ensure our pipeline is robust, we must force the LLM to output strict JSON. This allows us to easily diff the data later.

import json
from openai import AzureOpenAI

# Initialize Azure OpenAI Client
client = AzureOpenAI(
    api_key=os.getenv("AZURE_OPENAI_API_KEY"),
    api_version="2024-02-01",
    azure_endpoint=os.getenv("AZURE_OPENAI_ENDPOINT")
)

def analyze_content(markdown_text):
    system_prompt = """
    You are a competitive intelligence agent. Extract the pricing tiers and features from the following markdown.
    Return a JSON object with a single key 'tiers' containing a list of objects.
    Each object should have 'name', 'price', and 'key_features' (list of strings).
    """
    
    response = client.chat.completions.create(
        model=os.getenv("AZURE_OPENAI_DEPLOYMENT_NAME"),
        messages=[
            {"role": "system", "content": system_prompt},
            {"role": "user", "content": markdown_text}
        ],
        response_format={"type": "json_object"},
        temperature=0.1 # Low temperature for deterministic extraction
    )
    
    raw_json = response.choices[0].message.content
    return json.loads(raw_json)

if markdown_content:
    current_pricing_data = analyze_content(markdown_content)
    print("Successfully extracted structured pricing data.")
Pro Tip: Using `response_format={“type”: “json_object”}` alongside a low temperature setting (e.g., 0.1) forces the model to output strict, predictable JSON. This completely eliminates the need to write complex Regex parsers to clean up the LLM’s output.

Step 4: Managing State with Azure Cosmos DB

A truly proactive agent needs memory. If you want to detect when a competitor changes their pricing, your agent needs to know what the pricing was yesterday. By saving the JSON output from Step 3 into Azure Cosmos DB, you can query historical data on every run.

Cosmos DB’s native JSON support (via the NoSQL API) makes it the perfect companion for this workflow. On each execution, the agent pulls the latest JSON document, compares it to the fresh extraction, and if a diff is detected, flags it for review before overwriting the old state.

from azure.cosmos import CosmosClient, PartitionKey, exceptions

# Initialize Cosmos Client
cosmos_client = CosmosClient(
    os.getenv("COSMOS_DB_ENDPOINT"), 
    os.getenv("COSMOS_DB_KEY")
)
database = cosmos_client.get_database_client("IntelligenceDB")
container = database.get_container_client("CompetitorPricing")

def get_previous_state(competitor_id):
    try:
        return container.read_item(item=competitor_id, partition_key=competitor_id)
    except exceptions.CosmosResourceNotFoundError:
        return None

def update_state(competitor_id, new_data):
    document = {
        "id": competitor_id,
        "partitionKey": competitor_id,
        "pricing_data": new_data
    }
    container.upsert_item(document)

previous_data = get_previous_state("example-corp")

Step 5: Alerting on Changes (Webhooks)

If the agent detects a change between the Cosmos DB document and the newly scraped data, it should proactively notify your team. We can achieve this by comparing the JSON objects and firing an HTTP POST request to a Slack Webhook (or Microsoft Teams channel).

import requests

def send_slack_alert(message):
    webhook_url = os.getenv("SLACK_WEBHOOK_URL")
    if not webhook_url:
        return
    
    payload = {"text": message}
    requests.post(webhook_url, json=payload)

# Diffing Logic
if previous_data:
    if previous_data["pricing_data"] != current_pricing_data:
        alert_msg = f"🚨 **Pricing Change Detected!**\nExample Corp updated their pricing. Old vs New data differs."
        print(alert_msg)
        send_slack_alert(alert_msg)
        # Update the database with the new data
        update_state("example-corp", current_pricing_data)
    else:
        print("No pricing changes detected.")
else:
    print("Initial run. Saving base state.")
    update_state("example-corp", current_pricing_data)

Step 6: Making it Fully Autonomous with Azure Functions

To turn this script into a fully autonomous agent, we need to remove the human trigger. The best way to do this in the Azure ecosystem is to wrap our Python logic in an Azure Function with a Time Trigger (also known as a cron job).

Azure Functions allow you to run snippets of code on a schedule without provisioning or managing virtual machines. You simply define a function.json file or use the V2 Python programming model decorators to set a cron expression.

import azure.functions as func
import logging

app = func.FunctionApp()

# Run at 8:00 AM every day
@app.schedule(schedule="0 0 8 * * *", arg_name="myTimer", run_on_startup=False, use_monitor=False)
def proactive_pricing_agent(myTimer: func.TimerRequest) -> None:
    if myTimer.past_due:
        logging.info('The timer is past due!')

    logging.info('Proactive Pricing Agent started execution.')
    
    # 1. Scrape with Firecrawl
    # 2. Analyze with Azure OpenAI
    # 3. Compare with Cosmos DB
    # 4. Alert if changed
    
    logging.info('Proactive Pricing Agent finished execution.')

By deploying this code to Azure, your agent will wake up every day at 8:00 AM, scrape the target site, parse the markdown, query Cosmos DB, and send a Slack message if your competitor changed their pricing overnight. You have successfully built a system that works for you in the background. The days of manually checking competitor websites are officially over.

Conclusion

Building proactive agents requires shifting your mindset from single-turn chat completion to autonomous workflow execution. By combining Firecrawl’s resilient data extraction, Azure OpenAI’s intelligence, Cosmos DB’s state management, and Azure Functions’ scheduling capabilities, you can automate incredibly complex business processes. Start small with competitive intelligence, and eventually scale your agents to handle everything from lead generation to automated customer support auditing.

Categorized in:

Azure