Can You Trust the Story? Thinking Critically with AI and NYC Open Data
NYC Open Data Week – March 26, 2026
VIDEO | AUDIO | RECAP EN / ES / FR | INFO | INDEX
Speakers: Dr. Cecilia Dones - Founder and Chief Data Officer, 3 Standard Deviations
Dr. Cecilia Dones led an interactive workshop examining how New Yorkers interpret data, how civic data is created through lived experience, and how generative AI systems can misrepresent or overextend conclusions drawn from public datasets. Using NYC 311 complaint data as a case study, she explored how critical thinking skills can help residents evaluate claims generated by dashboards, reports, and AI systems.
Data Literacy Through Everyday Experience
Dones opened the session by framing the discussion around trust, authenticity, and technology-mediated communication. She explained that her research background focuses on understanding whether information encountered through digital systems can actually be trusted. Rather than presenting a technical workshop, she emphasized that the session would focus on practical reasoning and lived experience rather than coding or advanced statistics.
Participants discussed what they had recently noticed in their neighborhoods, mentioning changing weather, flowers, scaffolding, traffic, noise, litter, and trash. Dones then shifted the conversation toward annoyance and reporting behavior, asking attendees whether they actually reported these issues to 311. Most participants acknowledged that although they notice many problems, they often do not formally report them.
From this, Dones introduced one of the workshop’s foundational ideas: not everything becomes data. Data, she argued, is ultimately a human choice about what gets measured, recorded, categorized, and stored. Many aspects of urban life never enter official datasets at all because people choose not to report them, do not know how to report them, or decide the issue is not serious enough.
Thresholds, Reporting, and Selection Bias
The discussion explored how different people have different “thresholds” for deciding when an issue deserves formal reporting. Participants debated whether loud music, trash, or lack of heat should always trigger complaints, eventually concluding that context matters heavily. Noise at noon differs from noise at 3 a.m.; no heat matters differently in winter than in summer.
Dones used these examples to explain that civic datasets are shaped by human tolerance, awareness, and behavior. What gets reported reflects not only the existence of a problem but also the point at which people feel compelled to act. She connected this idea to selection bias, emphasizing that datasets reflect social behavior as much as objective conditions.
She also highlighted awareness as a factor influencing reporting patterns. Some people may not know that 311 exists or may not trust that reporting will produce results. Others may solve problems informally or simply tolerate recurring issues. Missing data therefore also carries meaning because absence from a dataset does not necessarily indicate absence of a problem.
Understanding NYC 311 Data
Dones walked participants through the NYC 311 portal, describing it as a non-emergency system for reporting neighborhood issues such as noise, trash, heat, blocked driveways, and illegal parking. She explained that once complaints are submitted, they are categorized and routed to appropriate city agencies for action.
She stressed the importance of classification systems in civic data. Categories such as “noise” may contain very different phenomena, including construction, traffic, parties, barking dogs, or other disturbances. The way governments define and group these categories shapes how policymakers interpret urban problems and allocate resources. Oversimplified classification schemes can obscure important distinctions and potentially lead to ineffective policy responses.
Drawing on NYC Open Data, Dones reviewed some of the city’s most common 311 complaints, including residential noise, heat and hot water problems, illegal parking, blocked driveways, and other everyday quality-of-life concerns. She emphasized that these datasets are dynamic and continuously changing, noting that 311 information is refreshed daily and reflects evolving social conditions.
Temporal Context and Changing Data
A major theme of the workshop was that data is not static. Dones argued that many people mistakenly assume that official numbers in reports or dashboards represent timeless truth, when in reality they are snapshots shaped by changing circumstances.
To illustrate this, she asked participants how a five-day heat wave might affect complaint patterns. Attendees suggested increases in reports related to cooling centers, power outages, garbage smells, rats, pools, and health issues. Others noted that people might generally become less tolerant and report more issues simply because they were uncomfortable.
Dones connected this to temporal bias — the mistaken assumption that a dataset reflects permanent conditions rather than specific moments in time. She cited a 2019 study of 311 complaints about homelessness, where a sharp increase in reports initially appeared alarming. Researchers later determined that the increase reflected greater public awareness due to a new city program, rather than an actual rise in homelessness itself.
This example demonstrated how shifts in awareness, outreach, or reporting mechanisms can alter datasets without necessarily reflecting underlying social change. Context, therefore, becomes essential for interpreting numbers responsibly.
Generative AI and Misleading Confidence
The second half of the workshop focused on how generative AI systems interact with civic data. Dones narrowed the discussion specifically to large language models (LLMs) such as Claude and ChatGPT, explaining that these systems generate probabilistic responses based on patterns in data rather than direct understanding or reasoning.
She demonstrated this by querying Claude about the most common 311 complaints in New York City. Although the system correctly identified noise as a major issue, it relied on 2024 information rather than current data, despite the fact that 311 datasets update daily. This revealed one limitation of LLMs: their outputs may appear current and authoritative while actually relying on stale information.
Dones then drilled down into increasingly specific geographic questions about Bushwick and her former ZIP code. Claude initially admitted limitations and recommended consulting direct sources such as the NYC 311 monitoring tool and NYC Open Data portal. She viewed this transparency positively because the model acknowledged uncertainty and directed users toward primary sources.
ChatGPT, by contrast, attempted to provide more direct answers even when the underlying data was weak or unavailable. Although its responses sounded plausible and aligned with general assumptions about neighborhood noise, the actual ZIP-code-level data from the 311 monitoring tool showed that heat and hot water complaints were more prominent than noise complaints during the examined period.
This mismatch illustrated a core danger of generative AI systems: they can produce coherent and persuasive narratives that overgeneralize beyond what the underlying data actually supports. Dones emphasized that users must critically evaluate which parts of an AI-generated answer are grounded in real evidence and which parts reflect extrapolation or probabilistic guesswork.
Missing Data and Invisible Stories
The workshop also examined problems that never appear in official data at all. Participants mentioned issues such as blocked bike lanes, film crews, triple parking, unloading trucks, and temporary disruptions that residents often tolerate without reporting.
Dones argued that missing data is itself meaningful. The absence of reports may indicate resignation, lack of awareness, distrust in institutions, or acceptance of recurring problems rather than the nonexistence of those problems.
She revisited earlier research on 311 reporting patterns, noting that increased reporting by city inspectors had significantly altered datasets because inspectors became a new source of complaints within the system. This demonstrated that datasets are shaped not only by social conditions but also by institutional processes and changes in reporting infrastructure.
Building Critical Data Skills
Dones concluded by encouraging participants to develop habits of skepticism and contextual inquiry whenever encountering data, dashboards, reports, or AI-generated analysis. Before trusting a narrative, she recommended asking several questions:
Where did the data come from?
Why did it show up in this way?
What is missing from the dataset?
What was happening at the time?
Does the conclusion go beyond what the data actually supports?
She summarized five major lessons from the workshop:
Not everything becomes data.
Data reflects when people reach their limit.
Reporting patterns depend heavily on context.
AI-generated answers can begin from real data but still overextend conclusions.
Missing information is itself meaningful and deserves interpretation.
Dones closed by stressing that critical thinking about data will become increasingly important as AI systems and algorithmic tools become more integrated into everyday life. She encouraged participants to continue exploring NYC Open Data, experiment with civic datasets, and develop stronger habits of inquiry rather than accepting official-looking numbers or AI outputs at face value.
RESOURCES
Can You Trust the Story? — Dr. Cecilia Dones’s NYC Open Data Week workshop page
NYC311 — the city’s portal for reporting non-emergency issues
311 Service Requests dataset — the daily-updated source on NYC Open Data used in the talk
NYC311 Monitoring Tool — NY State Comptroller dashboard for neighborhood and ZIP-code complaint breakdowns
NotebookLM — Google’s source-grounded summarization tool Dones cited for working with text
Claude — one of the two LLMs Dones queried about 311 complaints in the live demo
ChatGPT — the second LLM compared in the demo on neighborhood complaint data
3 Standard Deviations — Dr. Cecilia Dones’s data and AI consultancy