Civic Tech and Open Source Data Platforms: Building CommunityOne

At 11 PM on a rainy Tuesday, I was doing what too many concerned residents do: clicking through a county website trying to find out why a zoning variance hearing had been moved. The notice was on page three of a "News" section that hadn't been redesigned since 2009, the PDF link was broken, and the only mention of the new date was buried in a scanned agenda from a meeting I couldn't attend. That frustration — not some grand vision about open government — is what got me building DoxDock.

DoxDock is a civic data platform that scrapes, normalizes, and serves public documents from local government websites. It watches a handful of city and county sources, pulls down agendas, meeting minutes, and ordinances, cleans them into a consistent schema, and exposes them through a FastAPI-backed API with Supabase as the storage layer. The project is open source, and it exists because civic data is almost always public and almost never accessible.

The rest of this post is the hard part: the engineering. Twenty thousand scraped records, four failed schema migrations, one deservedly angry email from a county IT department, and a deep respect for how brittle municipal websites can be.

What DoxDock Actually Is

The core loop is simple: fetch, parse, store, serve. A background job runs on a schedule, hits a list of source URLs, extracts document metadata and full text, and writes normalized rows to a Postgres database. A FastAPI app reads from that database and exposes JSON endpoints for search, filtering, and jurisdiction browsing.

The "docking" in DoxDock means the platform is where documents land and are anchored to a stable schema. A resident doesn't care whether the original agenda lives on a WordPress site or an old ASP.NET app. They care about the date, the title, the jurisdiction, and the text. DoxDock exists to produce exactly those four things, reliably.

What DoxDock is not: a PDF viewer, an archive of raw HTML, or a replacement for the official source. Every record keeps its original URL and a content hash so we can link back and re-scrape when the source changes. The source of truth is always the government website; DoxDock is an index, not an oracle.

The Scraper: Asking Politely for Public Data

The scraping layer is a set of async Python jobs. I chose httpx over requests because the whole pipeline is event-loop driven, and I chose selectolax over BeautifulSoup after profiling it against a corpus of ~2,000 cached council pages. Selectolax's C parser made extraction roughly 4x faster, which matters when you're re-scraping the same sources weekly.

import hashlib
from datetime import datetime, timezone

import httpx
from selectolax.parser import HTMLParser

SOURCE = "city-council-agenda"
BASE_URL = "https://example.gov/council/agendas"

async def scrape_agendas(client: httpx.AsyncClient, db):
    page = 1
    while page is not None:
        resp = await client.get(f"{BASE_URL}?page={page}")
        resp.raise_for_status()
        parser = HTMLParser(resp.text)
        for link in parser.css("a.agenda-link"):
            record = {
                "source": SOURCE,
                "url": link.attributes["href"],
                "title": link.text().strip(),
                "fetched_at": datetime.now(timezone.utc).isoformat(),
            }
            record["content_hash"] = hashlib.sha256(record["url"].encode()).hexdigest()
            await db.execute(
                """INSERT INTO raw_documents (content_hash, source, url, title, fetched_at)
                   VALUES ($1, $2, $3, $4, $5)
                   ON CONFLICT (content_hash) DO NOTHING""",
                record["content_hash"],
                SOURCE,
                record["url"],
                record["title"],
                record["fetched_at"],
            )
        next_link = parser.css_first("link[rel=next]")
        page = int(next_link.attributes["data-page"]) if next_link else None

The ON CONFLICT DO NOTHING is deliberate. Re-scraping a page should not overwrite the original fetched_at or re-insert a duplicate. The content hash is the idempotency key for the whole pipeline.

One lesson from the first month: never scrape from a scraper function. The scraper's job is to fetch and stash raw HTML. Every parsing decision I made inside the scraper early on came back as a bug when a site changed its markup. Fetch in the scraper, parse in the pipeline.

Here is the comparison table I wish I had before I started:

| Approach | Speed (pages/min) | CSS selector support | Renders JS | Best fit | |---|---|---|---|---| | requests + BeautifulSoup | ~150 | Yes | No | Static HTML, quick prototypes | | httpx + selectolax | ~300 | Yes | No | Async pipelines, this project | | Scrapy | ~200 | Yes | No | Large parallel crawling jobs | | Playwright | ~30 | Yes | Yes | Sites that render content client-side |

The Pipeline: From HTML to a Clean Schema

Raw HTML is useless to an API consumer. The pipeline stage transforms a raw document into a typed record: title, source, jurisdiction, doc type, published date, and full text. This is where most of the real work lives because every government website expresses dates and titles differently.

"Tuesday, May 14th, 2024 6:30 PM" and "05/14/2024" and "May 14 2024" all need to land in the same published_date column. I wrote one parser for each source, then a second parser that runs only when the first one fails. The two-stage approach brought our date parse rate from 88% to 99.4%, which still means roughly 120 records a year get quarantined for manual review. That is acceptable, because silently dropping a wrong date is worse than flagging it.

from html import unescape

from models import DocRecord
from validators import parse_civic_date, infer_doc_type
from text_utils import collapse_whitespace

def normalize_document(raw: dict) -> DocRecord:
    title = collapse_whitespace(unescape(raw["title"]))
    date = parse_civic_date(raw.get("published_date") or raw.get("meeting_date"))
    if date is None:
        raise QuarantineError(f"unparseable date: {raw['url']}")
    return DocRecord(
        content_hash=raw["content_hash"],
        source=raw["source"],
        jurisdiction=raw.get("jurisdiction", "unassigned"),
        doc_type=infer_doc_type(title, raw["url"]),
        title=title,
        published_date=date,
        full_text=collapse_whitespace(extract_text(raw["html"])),
    )

The quarantine pattern is the single best decision in this codebase. Instead of crashing the pipeline or silently writing junk, a record that fails validation goes to a quarantined_documents table. That table is my earliest warning system for source site changes — when a city redesigns its page and all its dates suddenly fail, the quarantine count spikes and I know before users do.

FastAPI: The Right Amount of Framework

I have built enough Django projects to know I did not want one here. DoxDock has one public-facing job: read JSON from a database and return it as JSON. FastAPI gives me that with async support, Pydantic validation out of the box, and a free OpenAPI spec that documents itself.

from fastapi import FastAPI, Query
from supabase import create_client

app = FastAPI(title="DoxDock API", version="0.4.0")
sb = create_client(SB_URL, SB_ANON_KEY)

@app.get("/documents")
async def list_documents(
    jurisdiction: str | None = None,
    doc_type: str | None = None,
    q: str | None = Query(default=None, max_length=200),
    date_from: str | None = None,
    date_to: str | None = None,
    limit: int = Query(default=25, ge=1, le=100),
    offset: int = Query(default=0, ge=0),
):
    query = sb.table("documents").select("*")
    if jurisdiction:
        query = query.eq("jurisdiction", jurisdiction)
    if doc_type:
        query = query.eq("doc_type", doc_type)
    if date_from:
        query = query.gte("published_date", date_from)
    if date_to:
        query = query.lte("published_date", date_to)
    if q:
        query = query.ilike("full_text", f"%{q}%")
    result = query.offset(offset).limit(limit).execute()
    return {"items": result.data}

That endpoint has been stable for six months. I did not need Django's admin, ORM migrations, or auth framework. I needed typed routes, dependency injection for the Supabase client, and automatic validation errors. FastAPI delivered all three without ceremony.

The tradeoff is that FastAPI's ecosystem is thinner than Django's. There is no mature admin panel, and I had to build my own small dashboard for quarantine review. That was the right trade — the API is the product, not the admin.

Supabase: Postgres Without the Ops Headache

I chose Supabase for an unglamorous reason: I wanted real Postgres with as little server management as possible. I was tired of running pg_dump on a tiny VPS and praying that the automatic backups were working. Supabase gives me a hosted Postgres instance, row-level security policies, and a connection string I can rotate from the dashboard.

The data model is boring, deliberately. A documents table, a raw_documents table, a quarantined_documents table, and a sources table. No event sourcing, no graph traversal, no vector search. Civic data queries are simple by nature: filter by date range, filter by jurisdiction, filter by keyword.

Row-level security was the feature that convinced me. The public API uses an anonymous key, and the database itself enforces that anonymous users can only read documents, never insert or update. Even if the API layer got compromised, the database would reject writes.

alter table documents enable row level security;

create policy "public_read_documents"
  on documents
  for select
  to anon
  using (true);

create policy "service_role_full_access"
  on documents
  for all
  to service_role
  using (true);

The real comparison I made before committing:

| Option | Managed Postgres | RLS built-in | Auth | Monthly cost (small) | Ops burden | |---|---|---|---|---|---| | Supabase | Yes | Yes | Yes | $25 | Minimal | | Railway Postgres | Yes | No | No | $15 | Moderate | | VPS + self-hosted Postgres | No | No | No | ~$10 | High | | PlanetScale (MySQL) | Yes | No | No | $0 hobby tier | Minimal but MySQL, not Postgres |

Supabase won because the RLS policies and the hosted Postgres were exactly the two features I did not want to build or operate myself.

Validation: Where Bad Records Go to Die

Validation lives at the boundary between the pipeline and the database. Every record must pass a Pydantic model before it can be inserted into documents. The model enforces types: dates must be real dates, titles must be non-empty, URLs must parse, and doc_type must be one of a known set.

The first batch of scraped records taught me the cost of being permissive. I started with a schema where published_date was nullable and full_text could be empty. Twelve percent of my first 5,000 records had a missing date or blank text. The API returned them fine, but anyone searching for "meeting minutes March" got a wall of unhelpful nulls. I tightened the model, backfilled the bad rows, and the error rate dropped to under 1.5%.

The strongest lesson: validation failures are not bugs to suppress, they are signals. I initially caught a QuarantineError and logged it with logger.warning. After a city website silently changed its date format from "MM/DD/YYYY" to "Month DD, YYYY", the quarantine count told me about the redesign four days before any human reported a problem. That signal is the reason the quarantine table exists.

From that redesign I also learned to separate the raw document from the parsed document. The raw_documents table keeps the original HTML, so when a parsing rule changes, I can re-run the pipeline on the last 90 days of raw data without re-scraping the source.

Should I have included a schema version in each document? Yes. Not having one is the thing I'd fix first if I started over.

Shaping the API Around Real Questions

Generic CRUD endpoints are easy. Useful civic data endpoints require knowing what people actually ask. I spent two weeks reading local forum threads, Nextdoor posts, and the questions residents emailed to their city council. Across three jurisdictions, the same patterns emerged: people want to know what happened at a meeting, when a document was published, and what a document says about a keyword.

From that research, the API is shaped around questions, not resources:

• /documents?jurisdiction=maplewood&doc_type=minutes&date_from=2024-01-01 — "What happened in Maplewood since January?"
• /documents?q=zoning — "Find everything mentioning zoning."
• /jurisdictions — "What sources are covered?"

The most-used endpoint is a search across full_text using ilike, which is not fast on huge tables, but it is honest and simple at 40,000 rows. I added a trigram index later to keep it responsive, and that was the right time to add it — after the query had proven itself, not before.

One query that surprised me: "How many times did the city council go into closed session in 2024?" It is a keyword search that residents genuinely run to audit transparency. A thin layer over the database satisfied that audit in a way that a pile of PDFs never could.

Throttling, Backoff, and the County IT Email

I received an email from a county IT administrator in month two. The subject line was polite, the body was not. They had seen our crawler making requests from a single IP every three to five seconds for thirty minutes straight, and they wanted to know who we were and why we were hammering their server over a public records request they had already fulfilled in 2017.

The issue was honest negligence: I had written a loop that fetched a paginated list and followed each document link, without any rate limiting between the document fetches. The fix was a shared rate limiter in the scraper that allowed at most one request per three seconds per source domain.

import asyncio
import time

class DomainThrottle:
    def __init__(self, per_domain: dict[str, float]):
        self.limits = per_domain
        self._last = {}

    async def acquire(self, domain: str):
        interval = self.limits.get(domain, 3.0)
        now = time.monotonic()
        last = self._last.get(domain, 0.0)
        wait = last + interval - now
        if wait > 0:
            await asyncio.sleep(wait)
        self._last[domain] = time.monotonic()

throttle = DomainThrottle({"example.gov": 3.0, "maplewood.gov": 2.5})

I also added exponential backoff on 429 and 503 responses, starting at five seconds and doubling to a maximum of two minutes. The county IT administrator and I are on good terms now. I send them the scraper's user agent, I keep the rate under their threshold, and they stopped blocking my IP. The civic tech community runs on relationships like that.

What I'd Do Differently Next Time

First, I would add a schema version to the documents table from day one. Migrations are unavoidable; versioned rows make the migration path obvious. The four failed migrations I hit were all cases where a migration ran halfway and left rows in an inconsistent state. A version column would have made the rollout safer.

Second, I would extract metadata at scrape time, not at parse time. The scraper already fetches the HTML; capturing the response headers and the last-modified date at that moment is nearly free. It solved a real problem for us later: conditional GET requests using If-Modified-Since cut our re-scrape bandwidth by roughly 70% for sources that send proper headers.

Third, I would treat the quarantine queue as a first-class part of the dashboard. It is currently a manual SQL query. That is fine for me, but if DoxDock grows past a handful of contributors, quarantine review needs a UI and a clear workflow.

Finally, I would start with a smaller jurisdiction. I learned the hard way that scraping a large county site with dozens of departments multiplies every inconsistency by ten. One city, one source, one solid end-to-end pipeline — then add the next city.

Conclusion

The most important lesson from building DoxDock is that civic data infrastructure is not a performance problem; it is a trust and reliability problem. No one needs a million rows served in 15 milliseconds. They need the right row, with the right date, from the right jurisdiction, every single time. That changes engineering priorities: validation matters more than vectorization, quarantine matters more than caching, and a well-worded email to a county IT administrator matters more than any load balancer.

DoxDock is also a reminder that scraping public data is a relationship with the institution, not an extraction exercise. The city and county sites I scrape run on aging servers and part-time staff. Treating them with respect — rate limits, identifying headers, conditional requests — is both the ethical choice and the practical one. The project's most reliable data source is the one whose webmaster I called on the phone.

The open source repository is modest. It is not a framework, it is not a SaaS, it is a working pipeline for one specific, frustrating problem: public documents that are technically online but effectively inaccessible. If you have ever spent an hour hunting for a meeting agenda, you know the feeling DoxDock is built to remove.

The code is on GitHub, and I keep a list of open issues tagged "good first scrape" for anyone who wants to add their own city. View the project on GitHub.