Design and Evaluate Assessments of Relevant Skill Strengths and Areas of Need
If you’ve ever felt stuck wondering whether a client truly needs help with a particular skill, whether the goal you’ve written is actually the right one, or whether the progress you’re tracking is meaningful—you’re facing an assessment design question. Assessment is how we answer these questions with confidence. It’s the compass that guides what goals we choose, how we measure progress, and whether our work is making a genuine difference.
This article is for BCBAs, clinic directors, senior therapists, and clinicians who want to design and evaluate assessments that are scientifically sound, ethically grounded, and genuinely useful. We’ll walk through how to build an assessment plan that doesn’t just collect data, but answers the questions that matter most to your clients and their families.
What Assessment Design and Evaluation Really Means
Assessment design and evaluation is the process of creating a way to measure a skill and then judging whether that method works. It sounds straightforward, but the details matter enormously.
On one side, you’re designing: deciding what to measure, how to measure it, when to measure it, and who will do the measuring. On the other side, you’re evaluating: checking whether your method is reliable (gives consistent results), valid (measures what you intend), and sensitive enough to catch meaningful change over time.
Think of it this way. A referral comes in: “This student doesn’t speak much.” That’s a starting point, but it’s too vague to guide an assessment. Before you measure anything, you need to clarify what “not speaking much” really means.
Are we talking about the number of words spoken per hour? The variety of different words used? The ability to ask for things unprompted? How often the student initiates conversation versus responds to questions? Each of these is a different skill with different measurement needs.
Once you define what you’re measuring, you pick your tools and procedures. Are you observing in the classroom, at lunch, and at home? Counting behaviors live or reviewing video? Will one teacher collect data, or will you train multiple staff? Are you collecting one observation or multiple across different days? All of these choices are part of your assessment procedure.
The Core Building Blocks of an Assessment Plan
A solid assessment plan has several key components that work together.
The referral question is where everything starts. This is the explicit question driving the evaluation. “What specific communication skills does this student need to participate in classroom discussions?” is far more useful than “Why doesn’t this student speak much?”
A good referral question is clear, answerable with data, and connected to something that matters in the client’s real life. Sometimes the referral question is stated plainly; sometimes you have to uncover what’s really being asked through conversation.
Target skills and operational definitions come next. You name the specific skill—for example, “spontaneous requesting”—and define exactly what that looks like in observable, measurable terms.
“The student uses a word, phrase, or AAC device to ask for a desired item, activity, or help without an adult prompt” is an operational definition. It’s concrete enough that two different people watching the same interaction would agree on whether it happened.
Measurement methods are your tools and procedures. Direct observation in natural routines is one choice. Caregiver report is another. Curriculum-based checklists, standardized tests, video coding—each has strengths and limitations. Your choice depends on your referral question, the setting, and the decision you need to make.
Context and data collection details specify who will collect data, where and when, how often, and for how long. “I’ll watch for 20 minutes in the morning classroom transition, three times per week for the next four weeks, and count requests” is much clearer than “monitor communication.”
Clarity prevents missed days and ensures whoever is collecting data knows exactly what to do.
Reliability and validity checks verify that your measurement method is sound. This might include calculating interobserver agreement—having two staff watch the same session and score it independently. If two observers give very different scores, that’s a red flag. You need to clarify definitions or provide more training before using the data to make decisions.
Why Assessment Matters: The Real-World Stakes
Assessment drives everything that comes after it.
If you measure the wrong skill, you’ll pursue the wrong goal. If your measurement is unreliable, you might think a client is progressing when they’re not—or vice versa—leading you to keep using an ineffective strategy when you should change course. If you ignore context, your assessment might show progress that doesn’t translate to real life.
The stakes are highest when decisions affect a client’s eligibility for services, the goals in their treatment plan, or the choice between interventions. Getting the assessment right protects clients from wasted time chasing meaningless goals, ensures resources go where they’re needed, and upholds the client’s right to understand what they’re being evaluated for and why.
There’s also an ethical thread here. Assessments can feel intrusive or stigmatizing. Designing thoughtfully—choosing relevant and respectful measures, explaining why you’re measuring what you’re measuring, and using results to guide real supports rather than just documenting deficits—honors client dignity and autonomy.
Screening, Baseline, Progress Monitoring: Different Purposes, Different Designs
Assessments serve different purposes at different points in a client’s journey, and your design choices change accordingly.
Screening is a quick check to flag who might need further evaluation. It’s brief, done with many people, and doesn’t try to measure everything. A reading fluency screen might take 5 minutes per student and identify who is at risk; those students then get a more detailed assessment.
Baseline measurement happens before intervention starts. It’s your snapshot of where the client is right now. You’ll measure the same skill the same way again later, so baseline data lets you see how much progress was made. Baseline usually includes a few observations to account for natural variability.
Progress monitoring is ongoing measurement during intervention. It’s frequent—often weekly—and focused on the specific skill being taught. It answers the question, “Is this intervention working?” If data show improvement, you continue. If not, you change something.
Each purpose drives different design choices. Screening needs to be quick and easy for many people. Baseline needs to be stable and representative. Progress monitoring needs to be frequent enough to guide instruction but not so burdensome that it doesn’t get done.
Tools Versus Procedures: A Critical Distinction
A measurement tool is the device itself—a test, a rating scale, a checklist, a video recording you’ll code. A measurement procedure is how you use that tool: the instructions, the setting, the timing, the training required.
Two clinicians could use the exact same tool—say, a 5-point social skills rating scale—but use very different procedures. One might have the teacher fill it out once at year’s end based on overall impression. Another might have the teacher observe specific behaviors during a structured lunch activity, code them live, and repeat weekly.
The tool is the same; the procedure is different. And the results will likely differ too.
This matters because standardized doesn’t always mean better. A standardized tool used carelessly can give unreliable data. A simple homemade checklist used carefully, with clear definitions and trained observers, can give reliable data that actually answers your question.
Measurement Quality: Validity, Reliability, and Sensitivity
Three concepts anchor how you judge whether your assessment method works.
Validity asks, “Are you measuring what you intend to measure?” A test of reading decoding is valid for decoding; it’s not valid for comprehension. An observation of play skills at a clinic table might not be valid for understanding how the child plays at recess with peers.
Reliability asks, “Do you get similar results when you measure again under similar conditions?” This includes test-retest reliability (does the same person score similarly on different days?) and interobserver reliability (do two people watching the same event score it the same way?).
Reliability is critical when decisions hinge on the data. If a child’s communication score jumps from 5 to 45 across two observations with no change in actual behavior, your measurement isn’t reliable.
Sensitivity to change asks, “Can your measure detect meaningful improvement or decline over time?” Some measures are so broad that small changes get missed. Others are so specific they catch change but miss the bigger picture. The right balance means your measure can detect whether the intervention is working.
Interobserver agreement (IOA) is the practical way to check reliability. Two independent observers measure the same client using the same definitions, and you calculate how often they agree. Eighty percent agreement or higher is a common benchmark; consistently below that signals time to clarify definitions, retrain, or redesign.
Assessment Purposes in Practice: Referral Questions Shape Everything
Different referral questions lead to different assessment designs.
A BCBA is asked to assess a 7-year-old’s communication after a caregiver reports few spontaneous requests. She doesn’t just start observing. First, she clarifies: What kinds of communication are we concerned about? What does the child currently do to communicate?
Once the referral question is refined to “Does the child make spontaneous requests for desired items, and if not, what communication modality might be most motivating to teach first?”, she can design an assessment that samples requests across settings, people, and motivators. She measures both frequency and form. She collects data across home, school, and therapy to see if the pattern is consistent.
In another scenario, a team doing transition planning needs to assess adaptive living skills for a student moving toward semi-independence. They’re not looking for a global score; they need to identify specific, teachable skills.
They choose a curriculum-based checklist breaking down meal preparation, laundry, money management, and safety. They train the parent to observe and record weekly. They also do a situational assessment—watching the student attempt a simple meal and noting what steps he does independently, what needs a verbal cue, and what requires hand-over-hand help.
In both cases, the assessment design is tailored to the referral question.
Building Your Data Collection Plan
Once you know what you’re measuring and why, you need a practical plan for who will collect data, where, when, and how often.
A complete data collection plan names specific people and specifies their training needs. Are they already skilled, or do they need a walk-through? Will you check their accuracy periodically?
It defines the setting: classroom during math, cafeteria at lunch, home during dinner prep. It specifies timing: Mondays, Wednesdays, and Fridays at 10 a.m., or every evening at 6 p.m. It sets frequency: weekly probes before a meeting, or continuous data during every session.
A vague plan—”monitor progress” or “collect data on communication”—leads to missed days, inconsistent measurement, and data you can’t rely on. A specific plan, written down and shared, ensures consistency.
Evaluating Your Measurement: Building in Quality Checks
Good assessment design includes planned checks to ensure your measurement is working.
At minimum, calculate interobserver agreement during setup. When you first train someone on a new measure, have them score alongside you and compare. If agreement is low, clarify definitions and train again. Once solid, sample IOA periodically to ensure consistency hasn’t drifted.
Beyond IOA, consider whether your results make sense. If scores are identical every day for weeks, that might signal a floor effect or that observers aren’t carefully distinguishing between performance levels. If results wildly fluctuate, consider whether context variables are changing: time of day, who’s present, hunger level.
Also ask: Is the client actually progressing, or is measurement just happening? If weeks of data show no trend and no improvement, that’s information you need to act on. Assessment exists to guide decisions, not just create a graph.
Contextual Variables, Prompts, and Fair Measurement
The context in which you measure matters enormously. A child might show strong communication in a quiet one-on-one session but struggle in a noisy classroom. Both are true; they’re just different contexts.
Note the context of every measurement: Who was present? What time of day? What was the task? Were preferred items available? Was the task easy or hard? Were prompts used?
Over time, context notes reveal patterns. Maybe the skill only emerges with certain people, motivators, or settings. That’s valuable information about what the client has learned and what still needs work.
If a goal is for the client to use a skill in the community, don’t measure only in the clinic. If independence is the goal, don’t always provide prompts. Your measurement context should match the intended real-world context.
Cultural and Linguistic Responsiveness in Assessment
Assessments can unintentionally overlook or misinterpret clients from different cultural or linguistic backgrounds.
If English isn’t the client’s first language, translated versions of standardized tools might not capture nuance or be culturally relevant. A rating scale developed on one population might not reflect how a client’s family understands the behavior you’re measuring.
Ask: Does this measure make sense for this client and family? Are adaptations needed? If using an interpreter, have they been trained on your specific terms and definitions?
Dynamic assessment—where you observe, provide a learning opportunity, and observe again—can be more informative than a single static test for clients from different backgrounds. It shows what they can do with support, not just alone. Gathering information from multiple sources also ensures you’re seeing the full picture.
Common Assessment Mistakes to Avoid
One frequent misstep is designing an assessment without first clarifying the referral question. You end up measuring things that don’t answer the actual question.
Another is relying on a single observation or observer. One moment in time, one person’s perspective—these are inherently unreliable. You need multiple samples across time and, ideally, multiple observers.
A third mistake is confusing measurement tools with intervention strategies. A test tells you what someone can do; it doesn’t teach them anything.
Ignoring context is another pitfall. A client might look very different in the clinic than at school or home. Measuring only in one place gives you an incomplete picture.
Finally, don’t assume a standardized, published instrument is automatically better than a carefully designed informal measure. Standardized tools have value for certain purposes, but if one doesn’t fit your referral question, using it just because it’s published wastes time.
When Assessment Disagreements Arise
Sometimes assessment data and caregiver perception don’t match. A parent says their child has made huge progress, but your observation shows minimal change. Or vice versa.
These disagreements often aren’t about right and wrong; they reflect different measurement methods and contexts. The parent watches the child across many situations; you might observe 30 minutes once a week. The parent sees the child tired or distracted; you might see them well-rested and motivated.
Rather than dismissing one perspective, investigate. Ask the parent specifically what they’re seeing. Review your measurement context. Are you measuring in the relevant, real-world setting? Is your measurement sensitive enough to pick up the change?
Often the solution is to gather additional data in the context that matters most, involve the caregiver in measurement, and interpret results together. You and the family should agree on what you’re measuring, why, and how you’ll use the results.
Ethical Foundations: Dignity, Consent, and Appropriate Boundaries
Assessment can feel invasive. You’re watching someone’s behavior, noting what they can’t do, documenting challenges. That’s legitimate work, but it demands respect.
Get informed consent before beginning, especially if the assessment involves detailed observation or sensitive topics. Explain what you’re assessing, why, how you’ll use the data, and how long it will take. If someone is uncomfortable, take that seriously.
Respect the client’s right to understand what you’re doing. A brief explanation—not just to the parent but to the client themselves—shows respect and often increases cooperation.
Keep assessment private and secure. Don’t share detailed observations with people who don’t need to know. Use only the information you genuinely need. And once assessment is done, use the results to actually improve the client’s situation. Assessment without follow-up action can feel like judgment without purpose.
Bringing It All Together: Your Assessment Plan in Action
An effective assessment plan connects all these pieces: a clear referral question, defined target skills, a measurement method matched to the question, a concrete data collection schedule, planned reliability checks, and a commitment to using results to guide decisions.
Before you start measuring, write it down. What are you measuring? Why? How? Who will do it? When and where? How will you know your measurement is reliable? What will you do with the results?
This plan becomes your roadmap. It keeps everyone on the same page. It ensures measurements are consistent and comparable. It protects against measurement drift. And it forces you to think through your assessment before you start.
Key Takeaways
Start assessments with a clear, operational referral question that reflects something genuinely important in the client’s life. Match your measurement method to that question. Plan for multiple observations across relevant contexts and, when the decision matters, multiple observers.
Check reliability before trusting data to guide high-stakes decisions. Design with dignity, get informed consent, and use results to actually improve the client’s life.
When disagreements arise between data and stakeholder observations, investigate rather than dismiss. And above all, assessment is only useful if it leads to action. Collect the data you need, interpret it carefully with the people who know and care about the client, and use it to make better decisions.
