What this is
A re-presentation of New York City's public Open Data catalog — the same datasets you'd find at data.cityofnewyork.us, organized for browsing, with a search engine that's more forgiving than the official portal's. Every link points back to the City's authoritative version of the dataset.
Source
The catalog is fetched from the public Socrata Discovery API:
https://api.us.socrata.com/api/catalog/v1?domains=data.cityofnewyork.us
This is the same API the City uses to power its own portal search. Every record includes the dataset's name, agency-written description, the City's category assignment, tags, last-updated timestamp, and view counts. We pull the full catalog in pages of 100.
Categories
Categories on the home page are mostly the City's own assignments (domain_category in the Socrata API). Where the City didn't categorize a dataset, it appears under "Uncategorized."
One deliberate departure from the City's labeling: the City's "City Government" bucket holds 904 datasets, which is unhelpful — every dataset on the portal is technically city government data. We split that one bucket into five more useful buckets using a deterministic, transparent rule:
- Maps & Geography — datasets tagged GIS, map, aerial, ortho, boundary, district, planimetric, or imagery; or whose name contains those terms.
- Finance & Budget — published by the Department of Finance, Office of Management and Budget, Comptroller, Independent Budget Office, or otherwise about budget, spending, revenue, taxes, or audits.
- Procurement & Contracts — published by Mayor's Office of Contract Services or Department of Citywide Administrative Services, or otherwise about contracts, procurement, vendors, RFPs, or purchasing.
- Elections & Ethics — published by the Campaign Finance Board, Conflicts of Interest Board, or Board of Elections, or about elections, voters, campaigns, ethics, or lobbying.
- Government Operations — everything else from the original City Government bucket (Mayor's Office of Operations, Records and Information Services, etc.).
A small number of Parks Department datasets that the City filed under City Government are moved to Recreation. The exact rule is in refine_government() in build/fetch_catalog.py. All other City categories are preserved as published.
Plain-language summaries
Each card shows a one-sentence summary. That summary is the first sentence of the agency-written description, with HTML markup, entity codes, and stray whitespace removed and the length capped at 240 characters. We do not paraphrase or rewrite agency language at this stage. Future iterations may add a model-generated plain-English layer, which would be clearly labeled as such on the methodology page when it ships.
Freshness pills
The colored pill on each card is computed from the dataset's data_updated_at timestamp:
| Pill | Meaning |
|---|---|
| Fresh (green) | Updated within the last 30 days |
| Recent (yellow) | Updated within the last year |
| Stale (red) | Older than a year, or no update timestamp |
Note: "updated" reflects the dataset's data refresh, not metadata edits. Some agencies update on schedules (daily, monthly, quarterly); a "stale" pill doesn't always mean the dataset is abandoned, only that it hasn't refreshed in a year.
Declared cadence and the "Overdue" flag
Every dataset on the portal carries an agency-declared Update Frequency (Daily, Weekly, Monthly, Quarterly, Annually, "As needed," "Historical data" and so on), exposed by the API as Update_Update-Frequency. We store it (shown in small type after the freshness pill) and compare it with the actual last update. A dataset is flagged Overdue when it is well past its own promise:
| Declared cadence | Expected interval | Flagged overdue after |
|---|---|---|
| Daily | 1 day | 5 days (3× + 2, to forgive weekends and holidays) |
| Weekly | 7 days | 23 days |
| Every 2 weeks | 14 days | 28 days |
| Monthly | 30 days | 52 days (1.5× + 7) |
| Quarterly | 91 days | 144 days |
| Every 6 months | 182 days | 280 days |
| Annually | 365 days | 555 days |
| Every 2 years | 730 days | 1,102 days |
| As needed, Historical data, Other, To be determined, blank | Never flagged — the agency made no promise to measure against | |
The thresholds are ours and deliberately generous; the point is to surface datasets that are clearly behind what their publisher says, not to nag about a day's slippage. Filter with the "Overdue" pill in the sidebar or updated:overdue; freq:daily etc. filters by declared cadence alone. Roughly one dataset in seven is overdue by this test at any given time, most of them "Monthly" or "Annually" datasets that quietly stopped.
Agency normalization
Agency names on the City portal are inconsistent — the Department of Finance appears as both "Department of Finance" and "Department of Finance (DOF)", and curly apostrophes vs. straight create more duplicates ("Mayor's" vs. "Mayor's"). For the agency filter, we group records whose agency name matches after stripping a trailing parenthesized acronym, normalizing apostrophes, and lowercasing. The display name is the most common original spelling for that group. The exact rule is in normalize_agency() in build/fetch_catalog.py.
Search
Free-text search runs entirely in your browser using Fuse.js across dataset names, summaries, tags, and publishing agencies. It's a fuzzy match — typos and partial matches will still return results — and weights names highest. There's no logging, tracking, or server-side query handling.
The search box also supports a small set of operators, parsed before the fuzzy match runs:
"motor vehicle"— quoted phraseagency:nypd,cat:"public safety",tag:permits,type:map— field filters (substring match against the canonical value)updated:30d/updated:1y/updated:old/updated:overdue— freshness shortcutcol:bbl— datasets with a column whose name contains the term (the API'scolumns_namelist; about 80% of items carry one — maps, files and external links usually don't). Column names are also fuzzy-searched at low weight, so a plain query like precinct will surface datasets with a precinct field even when the word isn't in the title.freq:monthly— declared update frequency-tag:historical— leading minus negates a field filter
Table/map pairs
The portal often lists the same data twice — once as a table and once as a map, "filtered view" or external link with the same name (occasionally with a "(Map)" suffix). When both halves of a pair match your search, we show one card (the table, if there is one) with an "Also as map" link, and the counter says how many datasets that leaves. If only one half matches — you filtered to maps, say — it shows on its own. About 170 pairs are merged this way; two plain tables sharing a name are never merged.
When sorting by "Recently updated," datasets refreshed on the same day are ordered with described, human-named datasets first and bare GIS layers (all-caps machine names with no description) last, so the top of the list isn't a wall of CONDO_AREA.
Preview drawer
"Preview" on any card opens a panel without leaving the page: agency, category, declared cadence, freshness, publish date, views and downloads; the full column list (click a column to search for every dataset with a field like it); the first five rows and total row count, fetched live from the City's SODA API (resource/<id>.json?$limit=5 and $select=count(*)); any recent changes to the listing from the changelog; and up to five similar datasets, scored by shared tags, same agency and same category. Buttons copy the JSON API endpoint, the CSV download link, and a link to the preview itself (#d=<id>). Sample rows are only attempted for tabular items; keyless API calls are sometimes throttled, in which case the panel says so.
Favorites
Click the heart on any dataset card to save it to your favorites. Favorites are stored in your browser's localStorage — there's no account, no login, and no data sent to any server. The downside: favorites don't sync across devices or browsers. If you clear your browser data, they're gone. The "My favorite datasets" section appears at the top of the page whenever you have any.
"The week in city data" cards
The five small cards in the right rail show numbers pulled directly from City datasets, queried via Socrata's SoQL. The current set: 311 complaint volume + most-common type, motor-vehicle crashes + persons injured, DOB construction permits issued, restaurant inspections completed + grade-A share, and total datasets refreshed. Each card links to its source dataset on data.cityofnewyork.us. The exact queries are in build/weekly_stats.py; output lands in data/weekly_stats.json.
Windows end at the feed, not the clock. Every feed lags a little (311's by about a day and a half), and the newest day is usually still loading. So each card's "week" is the seven complete days ending the day before the newest row in that feed, and the twelve trend buckets are aligned the same way. Before this change the current week was systematically undercounted, which showed up as a phantom "−11% vs prior week" every day.
Stalled feeds are flagged, not zeroed. If a feed's newest row is more than 10 days old, the card says "Stalled" and gives the last date instead of printing a misleading zero. (The NYPD collisions feed stopped on June 11, 2026; the agency's own description now says the automated update is being fixed.) If a query fails on rebuild, that card is skipped; if every query fails, the previous file is kept rather than publishing an empty one.
"In the news" rail section
Once per rebuild, we fetch recent NYC headlines from Google News's public RSS feed — falling back to Bing News RSS when Google returns nothing, which it reliably does from GitHub's build servers — and pair each headline with the most relevant dataset on data.cityofnewyork.us. The pairing is editorial, not a clever NLP guess — we maintain a curated topic dictionary in build/news_match.py that maps news keywords (e.g. "shooting," "eviction," "subway delay") to specific datasets. Headlines that don't match a curated topic are skipped. The output goes to data/news_matches.json. Headlines link out to the original news article; dataset names link to data.cityofnewyork.us. Add or refine topics by editing the TOPICS list in the script.
"What's fresh" strip
Two horizontal carousels at the top of the page surface what's actually moving in the catalog: brand new this month (datasets whose createdAt is in the last 30 days, sorted by creation date) and updated this week (datasets whose data_updated_at is in the last 7 days, sorted by view count). Both lists are precomputed during the weekly build, so no client-side scanning is needed.
Journalist picks
About thirty datasets carry a gold star — these are an explicitly editorial selection of the datasets New York City newsrooms most often rely on (NYPD complaint data, 311, motor-vehicle collisions, ACRIS deeds, restaurant inspections, evictions, school quality, payroll, campaign finance, lobbying, jail population, and a few others). Each pick has a hand-written "why journalists use it" note and a "gotcha" callout. The list is in data/journalist_picks.json and is curated, not algorithmic — pull requests welcome. This layer is editorial; everything else on the page is mechanical.
"What changed" — the catalog changelog
The public request log
build/helpdesk.py pulls the NYC Open Data Help Desk: Public Dataset Requests file (63us-eqtq), the City's own log of dataset nominations submitted through the Contact Us form at nyc.gov/opendata, and writes data/helpdesk.json. It appears in the right rail and in full at requests.html.
- Status is the City's, not ours. Each row carries its own label — Closed or Pending, On Time or Past Due. Every legislated due date in the file is exactly 60 days after submission, so "past due" means the City missed its own 60-day window. We report the label rather than recomputing it; on the day a request flips, the two can disagree by one row.
- Waiting time is measured from the day the request was filed, not from the due date, and is shown only for requests still open.
- Dataset links are textual matches. Where a request or an answer names a four-by-four id that exists in today's catalog, the log links to it. A match means the id was mentioned — not that the request is what caused the dataset to be published.
- Requests are quoted as filed, including typos and the occasional joke. Nothing is paraphrased, and nothing is dropped: the file is published in full.
The three breakdowns, and why they are ours and not the City's
The City files no subject, no request type and no outcome — only an agency, a date and a status. The three charts on requests.html are keyword rules run over the City's own text in build/helpdesk.py. They are a reading of the log, not a field in it, and they are wrong at the edges. The rules:
- Subject is multi-label: a request about school bus routes counts under both Education and Transportation, so the bars sum to more than 2,963, and 597 requests match no subject at all and appear in none of them.
- Kind of ask is single-label, first rule wins, and the specific complaints are tested before the general nomination — a broken link, a stale dataset, a request for older data, a request for more fields, a request for help finding something, then an explicit ask to publish. Anything left is labelled "a straight ask for data," which is what 2,197 of them are.
- Kind of answer is single-label, first rule wins, in this order: no reply yet; a reply containing a data.cityofnewyork.us link; not a City dataset; withheld as not public; the agency says it holds no such data; a promise to publish; softer language pointing at data that exists; a redirect to FOIL, 311 or another portal. The portal link is tested early on purpose, so that "we do not have all of it, but here is what we do have" counts as pointing at data rather than as a refusal. Replies matching nothing are labelled "some other reply" — 624 of them.
Every bar filters the log beneath it, so any classification can be checked against the requests it claims to describe. If a bucket looks wrong, the requests inside it are one click away.
The City doesn't announce when it adds, removes, renames or rewrites a dataset. Because this tool has fetched the full catalog every day since April 30, 2026, and commits each snapshot to git, we can diff consecutive snapshots and log the differences (build/changelog.py, output in data/changelog.json, a full-page log at changes.html, and an RSS feed at feeds/changes.xml). Event kinds:
| Kind | Detected when |
|---|---|
| New dataset | An id appears that has never been seen in any snapshot |
| Removed | An id is missing from two consecutive snapshots and the portal's own api/views/<id> no longer returns 200 |
| Renamed | The name changed |
| Description changed | The description text changed; we store a window of text around the first differing character, so a one-word edit deep in a long description is visible |
| Agency changed | The attribution changed, after folding spelling variants (acronyms in parentheses, curly quotes, "&"/"and") so our own normalization never counts |
| Columns changed | Column names were added or removed (tracked from August 16, 2026, when we began storing columns) |
Caution built in: the Discovery API sometimes drops an item for a day and returns it the next, so a single-day absence is only "pending"; a partial fetch (fewer than 90% of the previous count) aborts the run rather than logging hundreds of false removals; and events on the first snapshot day reflect a rebuild of this tool's own text-cleaning rather than City edits. Description edits are where the news usually is — an agency noting that a feed has stopped, that fields with names or addresses were pulled, or that a data version was bumped.
Per-category feeds
For every category we publish a JSON and an RSS 2.0 feed of the most recently updated datasets, plus a master "newest datasets" feed. Feeds live under feeds/ and are regenerated on every weekly rebuild by build/generate_feeds.py.
Refresh cadence
The catalog is rebuilt daily at about 5:15 a.m. Eastern by a GitHub Actions job (fetch → changelog → feeds → weekly stats → news match → commit). The "Catalog refreshed" date in the search bar reflects the last successful fetch. Datasets new to the City portal appear in the explorer at the next rebuild. Each build step refuses to overwrite good data with bad: the fetch aborts if it returns fewer than 2,500 datasets, the changelog aborts on a partial fetch, weekly stats keep the previous file if every query fails, and the news matcher exits non-zero rather than writing an empty file.
Limitations and what's not here
- This explorer is read-only — it doesn't host the data itself. Every "open dataset" link goes to the City portal, where you can preview, query, and download.
- The City's category assignments are imperfect. A dataset about street trees might land under "Environment" or "City Government" depending on which agency uploaded it.
- Page-view and download counts come from the Socrata API and reflect activity on the City portal only.
- We do not include datasets from other Socrata domains (e.g., Health Department's separate portal at
nychealth.cityofnewyork.us). That would require a separate fetch and is on the backlog. - Agency attribution is taken from the API's
attributionfield. It's whatever the publishing agency wrote, and styling varies (e.g., "NYPD" vs. "Police Department"). Items with no attribution are grouped as "No agency listed." - The "Overdue" flag measures a dataset against its own declared cadence. An agency that declares "As needed" can never be overdue here, however long it goes quiet.
- Column names come from the catalog API's snapshot, which can lag the live schema slightly; the preview drawer's sample rows are live.
- The changelog only goes back to April 30, 2026, and cannot see changes the City made before then or changes to the data rows themselves — only to how each dataset is listed.
Code and rebuild
The fetch script is build/fetch_catalog.py in the repository. To rebuild locally: run python3 build/fetch_catalog.py, which writes data/catalog.json (full archive) and data/catalog.min.json (the file the front end actually loads). Then build/changelog.py (or --backfill to rebuild the log from git history), build/generate_feeds.py, build/weekly_stats.py and build/news_match.py.
Contact
Questions, bugs, or category gripes: file a GitHub issue on the project repository.