How to get Good data for The Coolest Advising Study Ever

In the last post, I walked through how we built the survey. But a well-built instrument only gets you halfway. A survey about practice only works if people answer honestly, and that’s a lot to ask.

Peter Ewell has often talked about distinguishing the function of assessment between prove and improve. The former is for accountability or evaluation, where the goal is to demonstrate the impact of your efforts. The latter is typically internal, and focuses on identifying where results are not where they should be or could get better. The challenge, Ewell emphasized, is that these are two VERY different phenomena, because in one case you're trying to find where you're doing well, and in another you're trying to identify where you're not doing well (or at least as well as you could be).

Higher education assessment often contains a paradoxical mix of the two. Most accreditation agencies have moved to a continuous improvement mindset toward assessment. They don't need you to show that you're getting all your students to achieve a certain outcome, they just need you to show that you're making changes to get more students there tomorrow than you did yesterday. The paradox comes in that the positive outcome of this assessment process begins with acknowledging a flaw. As you can imagine, we as humans are not really good at expressing such vulnerability, especially in a high-stakes setting like accreditation.

I would argue that this humility - the ability to acknowledge when we could do more or better - is not only an integral part of any effective assessment, but of high-functioning programs, organizations, and people.

That's easy for me to say, but hard to instill. Nevertheless, that's the fundamental key to our data collection for this project.

Think about what we’re requesting. We want advisors to describe their own work, against a standard that, as I said last time, is aspirational. Most of them will fall short of some items. Under the wrong conditions, the safest answer is the one that makes you look good. And then the data are measuring social desirability, not practice.

So the question that mattered most this fall wasn’t “what should we ask?” It was “what would make it safe to answer truthfully?”

Make it development, not evaluation

The first decision was how the survey would be positioned. From the beginning, we framed it as professional development: a chance to reflect on practice and see what the field’s best thinking looks like, not a performance review.

This was not only conveyed early, but in each invitation, communication, reminder, and discussion. I could give you a bunch of tactical examples, but what's most important is that - if you're conducting any sort of project like this - you keep in mind that this developmental point is what a former boss of mine used to call a SOCO: a single, overriding communication objective. It's something that must be emphasized explicitly as often as possible and backed up in practice.

That framing only works if it’s true. Nobody gets a score. Nobody’s results go to a supervisor. The goal is to find practices worth sharing so advisors can learn from one another.

Be clear about support and ownership

The second point of emphasis was how to involve leadership. On one hand, we certainly wanted advising leadership on board. It was important to not only establish the value of these results intrinsically, but to ensure that (a) this wasn't stepping on any toes, and (b) there would be a meaningful use of results. Fortunately, our partner was incredibly on-board - without which, none of this would be possible. Once this engagement was in place, then we knew we'd have what we needed from a logistical side. For example, advising leadership was comfortable with essentially requiring a response to the survey for all staff. That's exceptional commitment.

However, I personally took ownership of the effort when introducing it to staff. This had some higher-level advantages... for one, it's an old rule of consulting that words mean more when spoken by someone from more than 50 miles away. So while advising leadership could say that the theory and survey efforts were important, that statement somehow carries more weight when coming from me. But perhaps the more important point gets back to helping any survey participant understand the why of a survey. I wanted it to be clear that this survey didn't originate because the leadership said advisors weren't doing a good job - it came from an effort to better understand what's happening in advising.

It's a fine line to walk, but the ultimate goal is to maximize the positive sides of internal and external drive in gathering data.

Say the quiet part out loud

Finally, we told people plainly that gaps are expected. The survey says so, the invitation says so, and the leaders said so. If every advisor already did everything SSIPP describes, we wouldn’t have much to learn.

What we’re seeing so far

So far, so good! We've received responses from roughly 70% of advisors, with the data reporting window closing tomorrow. Knowing how deadlines drive things like this, I'm hopeful we'll be at 80-90% by the time things are done.

Anyone who does survey research knows that validity is filtered by administration. Even with the best theoretical framework, item development, and psychometric methods, if you're not getting reliable and valid responses, your results will be meaningless. Here, we've emphasized function and transparency as two keys to get the best data possible.

P.s. If a survey like this sounds interesting, just reach out to us at connect@ISSAQ.net to see how we can collaborate!