What Most People Get Wrong About Data Collection & Analysis
Data collection and analysis mistakes cost clinics more than they realize. Every day, behavior analysts make small errors in how they gather, record, and interpret data. These mistakes ripple outward—leading to wrong clinical decisions, wasted intervention time, and sometimes real harm to the learners we serve.
This guide is for practicing BCBAs, clinic directors, supervisors, and anyone who wants better data systems. We’ll walk through the most common mistakes clinicians make. For each one, you’ll get a plain-language explanation, a quick fix, and practical tools you can use this week.
The goal isn’t perfection. It’s sustainable improvement that protects learner dignity and supports good clinical decisions.
Clear Definitions: What We Mean by Data Collection, Measurement, IOA, Validity, and Reliability
Before we dive into mistakes, let’s make sure we share the same vocabulary. When teams use words differently, confusion spreads fast.
Data collection is how you record behavior or skill occurrences—everything from clicking a tally counter to filling out a session log. The method matters because it shapes what you can learn from the numbers later.
Measurement types describe the dimension of behavior you’re capturing. Event recording counts how many times something happens. Duration recording tracks how long it lasts. Interval recording samples behavior during set time windows. Rating scales capture intensity or quality judgments. Each type answers a different clinical question.
IOA (interobserver agreement) checks whether two people watching the same session record the same thing. Single-observer data can drift without anyone noticing. When two people agree, we trust the data more. When they disagree, we know something needs fixing.
Validity asks whether you’re measuring what you think you’re measuring. If your definition of “aggression” captures behaviors that are actually playful, your data isn’t valid for decisions about aggression.
Reliability asks whether your measure is consistent over time and across observers. A reliable measure gives similar results under similar conditions. You need both validity and reliability for good clinical decisions—a measure can be reliable without being valid.
Quick Glossary Box
Data collection: Recording behavior occurrences. Measurement types: Event (count), duration (time), interval (sample), rating (judgment). IOA: Agreement between two observers on the same session. Validity: Measuring what you intend to measure. Reliability: Getting consistent results over time.
If any of these terms feel new, we’ll show examples throughout. You can also explore our full pillar on data collection and analysis for deeper guidance.
Why Accurate Data Matters in Clinical Practice
Bad data leads to bad decisions. When your numbers are off, you might continue an intervention that isn’t working. You might stop one that is. You might change approaches based on random noise rather than real trends.
This isn’t just about efficiency. Collecting only the data you need, in ways that respect learner dignity, is part of ethical practice. When we over-collect or collect carelessly, we risk invading privacy or missing what actually matters.
Accurate data also saves time. Teams that invest in good systems spend less time chasing false leads and more time making real progress. They build trust with families by showing clear, honest information.
Worth posting somewhere visible: Collect to inform, not to impress. Data serves clinical decisions. People make those decisions. Keep human oversight central.
For more on ethical data practices, see our ethics and privacy guidance.
Top Mistakes (Overview): What to Look For
Here’s a quick overview of the ten mistakes we’ll cover. Scan this list and jump to the ones that match your current challenges.
- Vague or missing operational definitions
- Inconsistent measurement procedures
- Low or missing IOA checks
- Poor treatment-integrity measurement
- Data entry errors and dirty datasets
- Wrong measurement system for the clinical question
- Over-aggregation or misleading graphs
- Inadequate staff training and supervision
- Sampling or observation bias
- Misreading trends and failing to link data to decisions
Each section explains what the mistake looks like, why it matters, and one immediate fix you can start using.
Mistake 1: Vague or Missing Operational Definitions
What it looks like: Two staff members both record “noncompliance,” but one counts any hesitation while the other only counts flat refusals. Their data can’t be compared because they mean different things.
Why it matters: Without a shared, concrete definition, your data becomes noise. You can’t track change accurately, train new staff, or make confident decisions.
Immediate fix: Write a short definition that includes examples and non-examples. Test it by having two staff use it independently for a few minutes. If they disagree often, revise until they agree.
A simple template: Start with the behavior name. Add one sentence describing what it looks like. List two examples that count. List two that don’t. That’s enough to get started.
For deeper guidance, see our guide on measurement selection.
Mistake 2: Inconsistent Measurement Procedures
What it looks like: One therapist runs 30-minute sessions; another runs 45. One records during structured tasks; another during free play. The data looks different, but you can’t tell if behavior changed or procedures did.
Why it matters: Inconsistent procedures create noise that hides real change. You end up chasing patterns that are actually artifacts of how you collected data.
Immediate fix: Standardize a short checklist every observer fills out: who observed, session start and end time, context, and any interruptions. This takes seconds and makes your data interpretable.
For support training staff, check our staff training resources.
Mistake 3: Low or Missing IOA (Interobserver Agreement)
What it looks like: Only one person ever observes sessions. No one checks whether observers see the same things. Drift happens silently.
Why it matters: Without IOA checks, you can’t trust that your data reflects reality. One person’s interpretation can shift gradually. Decisions made on drifted data are risky.
Immediate fix: Schedule short, regular IOA checks. Even ten minutes per week can catch drift early. Calculate agreement using a simple method, and share results with staff so everyone stays calibrated.
A common approach: Total Count IOA divides the smaller count by the larger and multiplies by 100. For interval data, divide agreements by total intervals. Many teams aim for at least 80 percent.
A scheduling script: “Can you pair with me for ten minutes this week to check agreement?” That one question prevents many problems.
For full guidance and a calculator template, see our IOA guide.
Mistake 4: Poor Treatment-Integrity Measurement
What it looks like: You assume the intervention happened as planned, but no one measured whether the steps were followed. When outcomes are poor, you can’t tell if the intervention failed or was never delivered correctly.
Why it matters: Unknown fidelity undermines every conclusion. You might abandon a good intervention because it was poorly implemented, or stick with a bad one assuming it was done right.
Immediate fix: Pick three critical steps of your procedure. Create a simple checklist a supervisor can complete in real time: done, partially done, or not done—plus a line for notes.
This doesn’t need to be complicated. Even a sticky note with three checkboxes improves your ability to interpret outcomes.
For detailed approaches, explore our resources on treatment integrity.
Mistake 5: Data Entry Errors and Dirty Datasets
What it looks like: Missing values appear randomly. Someone typed “100” instead of “10.” Duplicate records exist. Dates are inconsistent.
Why it matters: Analysis built on dirty data gives wrong answers. You might see trends that are actually typos, or miss real change because errors swamp the signal.
Immediate fix: Run a short daily or weekly check for obvious problems: missing fields, improbably high or low values, duplicates. When you find something odd, confirm with the original observer before correcting.
Prevention matters too. Use required fields at entry. Use date formats that can’t be mistyped easily. Designate one short block each week to clean flagged items.
For templates and checklists, see our templates section.
Mistake 6: Wrong Measurement System for the Clinical Question
What it looks like: You’re counting how often a behavior happens when you really need to know how long it lasts. Or you’re measuring duration when frequency matters more.
Why it matters: The wrong metric hides real change. You might think nothing is improving when the dimension you care about is actually getting better.
Immediate fix: Before collecting data, ask: “What decision will this data drive?” If you need to know duration, choose duration. If you need frequency, choose event counts. If you can’t observe continuously, interval recording may work best.
When in doubt, pilot two approaches for a week and see which gives more useful information.
For a visual guide, use our measurement selection flowchart.
Mistake 7: Over-Aggregation or Misleading Graphs
What it looks like: Weekly averages look stable, but session-level data would reveal large day-to-day swings. A sudden drop gets buried in the average.
Why it matters: Over-aggregation hides patterns that matter. You might miss a sudden change that needs attention or celebrate stability that doesn’t exist.
Immediate fix: Show raw session-level data whenever possible. Add a trend line if helpful. When you must aggregate, do so transparently and note what the aggregation hides.
A good caption states what’s plotted, session length, and any context changes during the window.
For graph templates and tutorials, see our graphing resources.
Mistake 8: Inadequate Staff Training and Supervision
What it looks like: Staff learn data collection informally. No checklist, no observation, no feedback. Small mistakes multiply and get passed to new hires.
Why it matters: Training gaps create downstream errors in every dataset. Turnover makes this worse—each new person inherits bad habits unless you interrupt the cycle.
Immediate fix: Create short initial training focused on one or two skills at a time. Follow up with weekly ten-minute checks on a single skill. Rotate what you check: definitions, IOA, fidelity, data entry.
Sustainable training uses short, focused practice with immediate feedback. You don’t need elaborate curricula—you need consistent attention.
For micro-plan templates, visit our staff training section.
Mistake 9: Sampling or Observation Bias
What it looks like: Observations only happen when the most experienced staff member is present. Sessions are always during the easiest time of day. The data reflects best-case scenarios, not typical behavior.
Why it matters: Biased samples don’t represent reality. Decisions based on best-case data will fail under typical conditions.
Immediate fix: Specify observation windows in advance. Rotate observers so no single person dominates the data. Pick session times from a schedule rather than convenience.
A simple rotation plan assigns observers to different days each week and randomizes session times from a short list. This takes planning but dramatically improves data quality.
Ensure observations respect learner privacy and assent. For more on privacy and consent, see our ethics and privacy guidance.
Mistake 10: Misreading Trends and Not Linking Data to Decisions
What it looks like: Someone changes the intervention after one bad session. Or a real trend goes unnoticed for weeks because no one has clear criteria for when to act.
Why it matters: Hasty decisions can harm progress. Delayed decisions waste time. Both problems come from missing decision rules.
Immediate fix: Set clear decision rules before you start collecting data. Decide in advance what level of change triggers a review. Write these rules down and attach them to graphs.
A simple example: “If behavior increases for three sessions in a row, schedule a team discussion.” This prevents both overreaction and underreaction.
For decision-rule templates, see our decision rules guide.
Concrete Examples: Short Clinic and Research Vignettes
Seeing mistakes in context makes them easier to recognize.
Clinic example: A team tracked “noncompliance” with inconsistent definitions. Some staff counted any delay; others counted only verbal refusals. The data showed wild variability. After writing a concrete definition with examples and non-examples, agreement rose to 85 percent. The data became usable.
Research example: A study measured “on-task behavior” using event counts. But the behavior was sustained attention, which varies more in duration than frequency. Switching to duration recording revealed a clear improvement trend that event counts had hidden.
Both follow the same pattern: problem, single fix, better result.
For more templates, explore our full resources.
Quick Checklist and Downloadable Templates
Here’s what you need to get started. A one-page clinic checklist covers definitions, IOA, fidelity, data cleaning, and decision rules.
Templates include:
- Operational definition template
- IOA worksheet
- Fidelity checklist
- Session log
- Graph template
Print the checklist and review it with your team. Use it for one week. Revisit as a group and adjust based on what you learned.
Find all templates in our templates section.
Visual Aids: Before/After Graphs, IOA Example Table, and a Simple Flowchart
Good visuals make data problems obvious.
An IOA example table shows two observers’ counts across five intervals, then calculates agreement using different methods. Seeing the math laid out helps staff understand what they’re doing and why.
A before/after graph pair shows the same data plotted as session-level points and as weekly averages. The session-level view reveals a sudden drop the weekly average smooths away.
A decision-rule flowchart walks through simple questions: Did behavior change for three sessions? If yes, review with supervisor. If no, continue monitoring.
For editable templates, see our graphing tutorial.
Sources and Further Reading: How to Evaluate Credibility
Not all sources are equally trustworthy. Prefer government, peer-reviewed, and professional association resources.
A quick credibility check: Who wrote this? What are their qualifications? When was it published? Has it been peer-reviewed? Sources from .gov domains or indexed in academic databases tend to be more reliable.
This guide doesn’t provide legal or medical advice. Consult your local policy and organizational requirements.
For a curated reading list, see our pillar resources on data collection and analysis.
Use Data Ethically: Privacy, Consent, and Human Oversight
Ethical data collection starts with dignity. Collect only what you need. Ask whether each data point serves a clinical purpose.
Privacy requires following local laws and organizational policies. Store data securely. Avoid protected health information on personal devices unless your organization has approved specific security measures.
Consent and assent mean explaining data collection simply to families and, when possible, to learners. Document consent. Respect refusals.
Human oversight means never letting automated summaries replace human judgment. Review data before acting. Verify that what the system shows matches what happened.
A short ethical checklist: Is this data necessary? Who will see it? How long will it be stored? Who reviews it before decisions are made?
For full guidance, see our ethics and privacy resources.
What to Do Next: A 4-Step Action Plan for Busy Clinicians
You don’t have to fix everything at once.
Step one: Pick one measurement problem from the list—the one that bothers you most or appears most often.
Step two: Use the checklist and one template to make a single change.
Step three: Run a one-week test. Collect data using the new approach. Run at least one IOA or fidelity check.
Step four: Review results with a supervisor or colleague. Decide whether to keep the change, adjust it, or move on.
This micro-pilot approach builds sustainable habits. Small wins accumulate.
For a downloadable one-week plan, visit our staff training resources.
Frequently Asked Questions
How do I know which mistake applies to my project?
Compare your practice to the numbered list. Ask: Are definitions shared and concrete? Do observers agree when checked? Is fidelity measured? If two or more answers are no, start with definitions or IOA—they have the largest ripple effects.
What’s a simple way to start checking IOA?
A second observer watches and compares notes. Schedule short checks during real sessions. Use a simple worksheet to record both observers’ counts and calculate agreement. Share results so everyone knows where they stand.
Can I fix data-entry errors without stopping clinical work?
Yes. Use a short weekly checklist to spot obvious problems. Set validation rules at entry. Designate one short block each week to clean flagged items.
How should I choose the right measurement type?
Tie the measure to the decision you need to make. Duration for how long; event counts for how often. When you can’t observe continuously, consider interval recording. Pilot for a week to see if it answers your question.
What basic privacy steps should my clinic follow?
Collect only what you need. Limit access. Follow your policy and local laws. Explain data use to families and document consent. Consult your compliance team for specifics.
Where can I find templates to use right away?
The Quick Checklist section links to everything: operational definition template, IOA worksheet, fidelity checklist, session log, graph template.
How often should I review data procedures?
Monthly or quarterly works for most clinics. Use weekly checks when rolling out new procedures. Adjust based on turnover and learner needs.
Conclusion
Good data collection isn’t about perfection. It’s about building systems that support good decisions while respecting the people we serve. Every mistake we covered is fixable.
Start by picking one problem. Use one template. Run a short pilot. Review with someone you trust. Repeat.
Data serves clinical decisions. Clinical decisions serve learners. When you improve your data systems, you improve outcomes, build trust with families, reduce wasted effort, and make your work more sustainable.
If you take one thing from this guide: small, consistent improvements compound. You don’t need a complete overhaul. You need a willingness to notice problems and address them one at a time.
Download the clinic checklist and template bundle to get started. Try the one-week micro-pilot. See what happens when your data actually tells you what you need to know.
