A Love Letter to Data

When people talk about evidence-based decision making, they’re really talking about data. Data can support good decision making about policies, the allocation of resources, and provide a real-time look at how solutions are faring. But how do we make decisions on what to measure? Why does StriveTogether love data so much? And, does the data love us back? 

This episode features Sarah Rosen Wartell, president of the Urban Institute, and Ashwina Kirpalani-Vasanjee, vice president of insights and analytics at StriveTogether.

Learn more at StriveTogether.org.

Transcript

S3E4: A Love Letter to Data

Bridget  0:19  

Hi, I'm Bridget Jancarz, Chief of Staff at StriveTogether and your host for today's episode of Together for Change. 

This season, we're focusing on our North Star of economic mobility by diving deeper into how children and families are better off as a result of social impact work that treats the root causes of issues rather than just the symptoms. At StriveTogether, we do this by building civic infrastructure to change systems that put more kids on a path to economic mobility. 

So today is a special episode for those of you who love data. At StriveTogether, we talk a lot about evidence-based decision making. And what we're really talking about is data. Data can support good decision making about policies, how we allocate resources in communities, and provide a real time look at whether the things we're doing are actually working. Today, we're demystifying data and uncovering its hidden potential and social change. 

Joining me in this conversation is Ashwina Kirpalani-Vasanjee, Vice President of Insights and Analytics, and my colleague at StriveTogether. And Sarah Rosen Wartell, President at the Urban Institute. Let's get started. 

Sarah, I want to begin with you. We've never had a chance to meet before but I've heard a lot about you. I know a lot about your storied background. You've often shared your personal love and passion for data, and this stems from when you were working or serving at the White House. And today, you're now leading one of the premier nonprofit research organizations in the country. So this season, we're asking our guests to tell us a little bit about their why. How did you come to this work? And tell us why data driven analysis means so much to you.

Sarah  2:14  

Well, first of all, thank you. And it's a real pleasure to be with our friends and partners at StriveTogether, an organization that we at the Urban Institute deeply admire, because of its orientation and organization around helping drive collective decision making to outcomes for people. And that's what you guys do so well. And I'm a lawyer by training, right? I grew up in an urban area, I grew up in the Upper West Side of Manhattan and went to be a public policy analyst, then ended up in some wonderful positions in government in the Clinton White House at the end. And I was often a consumer of data on all kinds of issues, policy issues that I was wrestling with, or leading a decision-making process, and often had economists and analysts from different agencies with me. But the real moment when I had a kind of a-ha, that this was really a way of orienting, thinking about decision making, was during the mortgage crisis. 

I'm a houser, I do housing policy in my spare time these days. And you know, 2007, and eight, the economy was melting down, because we had made a whole set of really bad choices about what kinds of products private actors could make and how government regulated, or didn't, those decisions. And I have a lot of expertise in mortgage finance. I've worked in this space for 20 years. I worked with a lot of people who have had roles in government. And yet none of us really knew what was happening. And there were some small number of really impactful people at that moment. And they were the ones who brought us new insights from data. One of them, Laurie Goodman is someone I went on to work with, brought to the Urban Institute to lead our housing finance policy center. And I think I'll just, as one example, recently, when we had the pandemic, many of the solutions that were born of that data analysis during the mortgage crisis of 2008, nine, some of the solutions that were put in place, I think, avoided a huge number of people losing their homes this time. 

So I've seen firsthand in my own area of expertise, how using data can help you to understand a problem. We can use it to test solutions, we can use it to measure and guide us to see whether what we're trying to do works, we can use it to empower advocates who want to make a case for change, and we can use it to help decision makers accountable. So I've kind of become a true believer in the data space after all that.

Bridget  4:54  

Thank you. Data is definitely a Swiss Army Knife in our sector. So Ashwina, I'm gonna turn it to you. You're a fairly new addition to the StriveTogether team, but not at all a new addition to the StriveTogether Cradle to Career Network. You spent almost a decade with the Commit Partnership in Dallas, Dallas County, and doing statewide work and Texas. You've built a tremendous data initiative with AIM Commit. Why do you believe that data is so critical to our work to dramatically improve outcomes for kids and families?

Ashwina  5:30  

It is so great to be here with you and with Sarah today. This is my first podcast and I'm feeling pretty special and also a little nervous. Shout out to all my data leads out there. Data, in my mind is a powerful tool. And when we use it responsibly, it can be a catalyst for change, and a way to track progress. It's a tool that we use to start conversations to begin the process for changing behavior or changing resourcing. And it's really used to inform and to break mental models that may exist. So that's kind of how I really think about data. 

And I'll give you a recent example of a trip to Austin, to E3 Alliance, that we made, that I had a pleasure to be with them a few weeks ago. And the team there shared data with districts about outcomes for children disaggregated by race and ethnicity that showed disparities in higher level math completion for students who were high achievers in fifth grade. So what they did was looked at fifth grade outcomes, and followed students into eighth grade. And they found that children of color who were achieving at high levels in fifth grade, were in less rigorous math courses by eighth grade. And they were kind of trying to figure out ways to activate this data and how do you make change, right? They shared it with district leaders, and these district leaders then started acting on this data. They started looking at ways to change the system so that the outcomes can then also change, activating the data with parents and with teachers and really thinking about how to change this outcome for children of color. 

Data did not do the change, and did not cause the change, right? It was the thing that activated, that started the conversation that led district leaders to think about ways to improve the system and to change it. Data ignited leaders to act to have those conversations to make change and to track progress. And to me, that's really the power of data and what it can do.

Sarah  7:45  

I'm just nodding my head because I think Ashwina is right. None of us believe that data alone is the answer to the solution. That is the answer to all our problems. And even sometimes when you have a really compelling case and a clear path what to do. Sometimes it's not enough, there is the external environment. But there are always these teachable moments. And if you're kind of looking for them, and you bring data to the table, when you have the attention of the relevant folks, when they have a motivation or reason to change. It is data plus, that then leads us to having better outcomes.

Bridget  8:24  

And I think that's a story, we will want to tell our listeners today. To start us off. I have the privilege right now of being with two tremendous data leaders who know leaps and bounds more about the technical side of data and analysis than I ever will. I'll say I'm a data intermediate, not a novice, but I'm an intermediate, certainly not advanced. And I know for a lot of folks who may be either sitting in shoes similar to mine, or maybe even further removed from, quote unquote, data, it can seem technical, wonky. I'm wondering if you can both share, like what's a common misconception about data in this work?

Ashwina  9:09  

I think a common misconception is that the idea around this tremendous responsibility, that data leads have to keep this data centered on communities, and to make sure that descriptions of communities are asset based and the analysis that is conducted can have tremendous impact on what actions are taken. So there's like this great responsibility, this tremendous power, almost, that the insights at the focus of an analysis can actually lead to action and change in communities that are positive.

Sarah  9:46  

So I think I'm going to share something that is a different way of looking at the problem that Ashwina just described. If you ask about a misconception, in some ways, there is a perception that data is the facts. It's the truth. So if you look at a set of numbers, you now know the answer. 

But we all know that there's a lot that goes into what gets counted in the first place, what assumptions were built in to the society that you were then measuring. So if we, for example, have a society that has discrimination and inequity built into it, and then we try to use data to predict outcomes, we may be baking those inequities into our system. Unless we're intentional about it, we know that what gets measured gets counted. And that means that things that don't get counted, or what don't get measured, will often not be seen. So we know that you really have to ask hard questions about the data you have. 

I'll give an example here. In many cities use a data source they have like their 311 data that tells them where there are problems in their communities, to decide where to allocate resources to fix potholes, or street lamps or whatnot. Well, some parts of our population are much more likely have greater trust, are much more likely to report those problems. So if you don't understand what went in to the data that you're relying upon, you may make decisions that then bake that same disadvantage into society going forwards. 

Similarly, a lot of our data clump a lot of people together who may have very different experiences. So when you look at a set of data that says Black, Latinx, white, and AAPI populations, well, within that AAPI populations may be 45 different ethnicities. And then when we look at an average, you may be not in fact, understanding the very different lived experiences that a Hmong community than a Japanese American community might have. So the misconception I think, is that data somehow has all the answers data is the beginning of the question.

Bridget  12:07  

So let's talk a little bit about that. Data is often used, like you both said, to understand the current state of something. How young people are doing in school? What our community's health outcomes are? Something like the availability of affordable housing. But this data doesn't tell us how to improve outcomes. Ashwina, I like what you said data starts the conversation, it inspires action. Data in and of itself cannot do anything. People with data do things. So I'm curious how maybe you all would define data. Because I think the way we are talking about it probably is more of a reflection of quantitative data. So how do you all define data? And what does it look like to use data, to understand the what, and to inform the why?

Sarah  13:01  

So we often talk about evidence at the Urban Institute, and data is a source of evidence, but lived experience is a source of evidence. And in the worlds that were coming today, where we have natural language processing, we're able to actually scrub huge written words and turn that into data, data is the realities that exist in all of the different means by which we can collect it. So ethnographic research, where people might look at a set of numbers and say, I have a hypothesis from that, then I'm going to go into a community and spend time and listen and understand their experiences, and then draw examples from that experience to a larger population and see if it translates or if it does not, because sometimes our biases or other things mean it won't. 

Data is numbers, if you will, zeros and ones and in a computer system, are just one of the ways in which we as human beings can process information. And the interpreter is as important as the numbers itself. What questions we ask of the data, who collects it, who interprets it? How is it communicated back to the people whose lives are represented in the data? Those are all questions we have to be asking ourselves.

Ashwina  14:25  

Yeah, and as I look at kind of everything around us, we're surrounded by data. Anything can be turned into data like Sarah had mentioned. And I feel like when we start looking at numbers, we always have to make sure to tell the qualitative story along with the numbers, right? And the example that I have is that the Commit Partnership, we followed a woman named Mavis and kind of made a whole video on it as well, and followed her story as to why maybe she, her children may not be ready for kindergarten, and all of the obstacles and the limitations or her journey, what kind of impact her journey was having on her children, right? So we see numbers like, you know, 50% of children are ready for kindergarten. And then putting that alongside a narrative and bringing power and humanity to the numbers is so incredibly important. So, for me, not only are the numbers data, but everything around it as well, is data that's used to inform policies and actions and next steps.

Sarah  15:43  

Ashwina used a word before that was a really important concept. She talked about asset-based understanding. So much of work that's been done over the years to look at populations that have faced challenges comes in the form of describing a disparity. And if you describe the difference between how different populations are faring as though somehow it was naturally occurring, you end up with this kind of deficits model in your mind where you're saying you you create a norm, usually the privilege group, and then you are describing the deficit against the norm. But much of the way that we are trying to teach our colleagues and I'm learning to think about questions is what's the larger set of systems and structures that created those disadvantages? And if you always look at any research question, not just as though a number exists as a natural phenomenon, but the origins of it, you're then going to interpret that data to solution in a very different way.

Bridget  16:50  

You know, years ago, we were working with our friends in San Antonio, and we were looking at kindergarten readiness, early grade literacy, and how chronic absenteeism played a role in that. And as they started to unpack their data and look at different things, one of the things they noticed was that asthma and the prevalence of asthma attacks in students was a huge barrier to kindergarten readiness and attendance for kindergarteners. So the way they approached this was not to just do a campaign, get your kids to school, here's why it's so important, but to work with doctors to help families to understand how to prevent asthma attacks, before they even happened. And there's myriad other stories like that within the network. I'm wondering if either of you have one of these kinds of a-ha stories, where unpacking the data showed that, you know, it wasn't just the expected mental models or some of the narratives or tropes that we hear so often perpetuated in the media or by other groups, is there a surprise, you've seen that actually, once folks were able to uncover it really led to some kind of impact?

Ashwina  18:07  

Yeah, the same story that I kind of shared before with that E3 Alliance data around fifth grade to eighth grade progression for children of color. I actually shared that with a group of middle schoolers and high schoolers in a small session. And what I was pleasantly and really excited about seeing in that room was not so much the data because the data is really sad. But how the children and the students in that room, it became kind of a call to action for them. 

When they saw the data, they were very surprised and very angry, as well, because the system was failing them, right? The children in that room were were children of color. And what I was amazed by was, their very next step was we have to create a math camp. So that this doesn't happen, right? So that it doesn't happen to our younger brothers and sisters, or our, you know, our younger friends. And what clicked for me in that moment was how do we get more of these datasets or this important, like, substantial information into the hands of people who are the subject of that information? Who can then act upon it in ways that are trusted within their own communities, right? And so when I saw that happen, there was just something in me that clicked that very much understood the power of family, communities and students.

Sarah  19:33  

So I love that story, because it illustrates this challenge of information asymmetry, where in many cases, elite systems managers have access to data, researchers have access to data. And it's data about the lives of people who may never see the data or have a chance to be informed by it. And we have a wonderful tale that really was an eye opening for me. 

One of our research projects was working in a public housing project. With teens, they collected a bunch of information about patterns of behavior at different times of the month. And when they looked at the data, it made no sense. Why was it that in the third week of the month that the teens behavior was one thing and then the first week of the month, the teens behavior was something else? And it had to do with food insecurity. And the teens took one look at that data and they understood what was different between the first and the third weeks of the month, it had to do with the timing of when the family got a replenishment of we used to call food stamps at the time this research was done. 

So I think it's really important that we think about the fact that data reflects someone's life, and that if we are going to extract knowledge, we need to share that knowledge. If we're going to design a research, we're trying to train our researchers on learning alongside them in community engaged research methods that ask you, how are the people whose lives are represented in the data available, thought about at every stage of the process? How are we designing the research question? How are we collecting the information? How are we interpreting the information? 

How are they involved in thinking about the privacy issues that are implicated by their data. And as the technology becomes more and more complicated as them who are most at risk, when we think about the privacy issues at stake. And yet, we want to be able to combine data from different sources together to gain new insights, especially to be able to disaggregate data and understand different experiences. So it's really about putting those people who the data reflects in a true partnership role as you are using the data and trying to drive decision making.

Bridget  21:58  

You know, Sarah, this brings up a really interesting point that I know many of us are working on, which is democratizing data, making data transparent, putting it in the hands of folks who need it. And there's a lot of emerging technology, right? There's more data systems and infrastructure, but we are very much living in the time of AI, and ChatGPT. How do you, and I'm not asking you both to be fortune tellers, but what's the potential of something like that, you know, if we were to have this conversation, 10 years from now, what do we think that could look like? What are the benefits of it? And you know, if there's anything we should be scared of, or watch out for, what might that be?

Ashwina  22:44  

So I had a chance to recently attend ASU+GSV in San Diego. Walking into that conference, I knew ChatGPT would be a significant portion of what we talked about. And there was a moment that inspired me, based on the opportunities that are available. And that is when Salman Khan, Sal Khan from Khan Academy, started talking about how they were using ChatGPT and AI to support their students. And what he talked about was allowing AI to serve as an on demand tutor for children, for kids, who were studying various things and had questions and they would ask this tutor. The AI would then aggregate where students may be struggling for the teacher, and provide also progress reports to parents. He also talked about how much, and this is the part that really excited me, he also talked about the opportunity that teachers have in this moment to use AI so that they can spend more time supporting students in the way they need to be supported, versus getting out the content and making sure the content lands and focusing heavily on the content. 

That got me wondering about the opportunity for student advancement and support in the ways that we today are starting to use ChatGPT for ourselves. Hey, ChatGPT, can you write out an email for me to send off to someone or, you know, whatever the purpose is, and if we're able to actually use some of these technologies and tools to help us work faster and better. I imagine that that same tool can be used for teachers as well, to really help support classroom movement and have them spend time more on things like social, emotional development versus answering a question that can easily be answered by an AI tool.

Sarah  24:55  

So you really asked two questions there and I want to try to dissect them little bit. So the first one was really about data transparency and availability. And we and I'm sure Strive has as well been trying to build tools that other people, including citizens in a community, but also, educators who work in a school system or city officials who are trying to make decisions about allocating resources can use that one person develops a tool, and then anyone can kind of run their own data through it and help them to make decisions. So we have something we call a spatial equity data tool, where you can take a data set and look at any data set that's geocoded, we can look at whether or not an asset is being distributed equitably, because we know about the economic and racial and ethnic distribution of people by geography, those kinds of tools. 

You can imagine a city looking at their budget and where they're spending money on their infrastructure projects, suddenly has insight about equity and how that information can help them maybe make a different choice for which project gets prioritized. So technology, generally AI and other, is really enabling us to distribute tools. And we have a challenge, which is that people are overwhelmed. They don't know what to use. And they also don't all have the fluency, right? We're trying to make them simpler and easier and more accessible at the same time. The second question you asked is really about AI in particular, and sort of what are you excited about it? What are you scared about? And it's really related to the conversation we had before. If data is good, and by good, I mean, not only accurate and complete, but also we understand its origins, we don't take for granted that, for example, it's fully inclusive, you know, we don't take for granted that it doesn't represent some other inherent bias, then you build an algorithm based on it, you're much more likely to then make good decisions. 

But if that algorithm, if ChatGPT is giving you insights it found scrolling a universe that has some of society's worst challenges already baked into it, we could end up with decisions that rapidly exponentially increase the prevalence of those very problems, right? If you, in my world, in housing, if you use automated valuation models, you think great, we're going to avoid the bias that human appraisers have. But if the data you using on those valuation models still has a set of biases in them, you may actually create a problem in a different place. So at the end of the day, human beings are going to always need to scrutinize the inputs. And we should take advantage of these tools to accelerate how quickly we can do in mind and draw insight from a whole lot of information. But at the same time, we're going to have to find a way to make sure that we're not relying on information to make a set of decisions, where we don't fully understand all that it brings forward both the advantages and disadvantages in it.

Bridget  28:21  

So there's a lot on the horizon with data. There's also just a lot we've talked about today around how data drives really effective action, if we've got the people with the right lenses to be able to do that. The last question or theme I want to bring up, you know, in the social sector, there's definitely, I think, a deficit mindset around the ability to use data. 

We often talk about data deserts, or data, talent deserts, where we don't have the resources, we can't find the talent who can do this type of data work. But we know it's so critically important. Both of you have shared such extreme examples where it's necessary. So for our listeners and friends in the field with us, whether they're practitioners, whether they're in our network, whether they may actually be able to fund this kind of data work, what sort of the call to action or the lesson you'd give folks on why we have to continue to keep data front and center and social change?

Sarah  29:26  

Well, I'll start by saying that who's at the table really counts, right? It matters who you have. So it's not just that data fluency matters, but data inclusion matters. It is really important that we are mindful and I say this, with the full understanding that we face a decision around affirmative action in higher education that could create some challenges to our creativity in thinking about this, but I don't think this is just it's a very important remedial question for our society that has, in so many cases created barrier after barrier. 

But it's actually a quality question too, because the insights that you're going to get are going to be different than the questions you're going to ask are going to be different, the understanding of the data that you're going to have is going to be different if you have a broader set of people at the table, bringing a broader set of experiences. So, you know, my call to action is really about finding strategic ways. And I loved Ashwina's story about bringing data for kids, and how they were mobilized and activated by that, right, you know, you can think about in your schools, how children can use data to get something that's important to them. And then that's the first step in thinking about how they imagine themselves in society, I have a daughter who's interested in vets, she loves animals, right? The other day, she came home, and she was liking her programming class in high school. And I thought, you know, she says, yeah, but I don't need to do that, because I'm gonna go work with animals. I said, look, you know, work with animals is gonna be work in computer science and data in the future. And I think that the idea of giving children an opportunity, and being very mindful about inclusion in our pipelines to science roles, data analyst roles, is research roles really important.

Ashwina  31:25  

I think Sarah said it earlier and very well, right, whatever we measure is where we look, right? If we're not measuring something, we're not going to look at it. And so as a data lead here, at StriveTogether, I'm always looking for more and more data to pull insights out of. And my call to action would be, how do we have datasets that can tell a fuller picture that can be technical, like data interoperability, so connecting High School datasets to college data sets to then workforce, with some of the SLDS systems that the federal government is starting to put some money behind her and has put a lot of money into as well. 

Those systems will be incredibly important to build out in ways that allow for cross years many years of analysis. So we really understand how to help communities how to help students achieve economic mobility. So what I would say is as supportive as we can be of some of those data systems, would change the way we are able to analyze data and the insights that we are able to produce.

Bridget  32:40  

So my big takeaway, and Sarah, I loved the way you said this, data is all of the realities that exist, full stop. And we can't put more kids on a pathway to economic mobility without data. So I want to thank you, Sarah. Thank you, Ashwina, for this incredible conversation today. Thank you for listening. You can stay connected with us by visiting strivetogether.org to get the latest information for our monthly newsletter and new blogs. You'll also find transcripts and our Together for Change podcast series, case studies and more.