How the data sausage gets made

I like to research things. Back in high school, I loved going to the library and would look up the footnotes to see what other books or journals were connected to something I was reading and then I'd shift over to that resource to dive deeper. I still do this and it's led to an overflow of books in my home office.

To obtain my history degree for my B.A., I was tasked with doing a unique research project and writing a proper 40 page draft. I had chosen a dairy farmers strike in upstate New York during the 1930s. My adviser was kind of a pain in the ass and everything I did never seemed good enough. One time we had a meeting, he asked what secondary resources I was going to be citing. I went to the library, found literally every book I could think that may be useful, and dumped them all into a huge Dunkin' Donuts bag. He looked at it and then without referencing it asked me to talk about my primary resources.

I think a lot of how I operate can be traced back to moments like this, where I want to be able to have a mastery and understanding of a topic when I dive into it. Ultimately, even though I work for a technology company I don't consider myself very adept or smart at the technology side of things. But I like to think I have an understanding of how our sector ticks at its core.

A lot of that is gained through researching publications that the sector produces. I remember a few months ago when T. Clay Buck downloaded every report he could find. I found that an interesting project because I imagine there was a lot of fluff and BS to wade through in order to find some gems. It isn't to say that all vendor reports are bad - quite the opposite, I find myself citing the Generational Giving report by Blackbaud a fair amount. When done right, they can be utterly trustworthy and a great foundation of learning! But when done wrong, they can send the wrong message and point people toward implementing things that are not in their best interest.

I wanted to share a bit on how Neon One does their data analysis since we're going to start putting out more reports given the breadth and depth of our reach. We power over 35,000 organizations and actively provide data to several agnostic industry sources (Fundraising Effectiveness Project and the GivingTuesday Data Collaborative to name a few).

Data Sources

The vast majority of our ability to quickly pull a report comes from the foundation that NeonCRM has built. The work that was done around FEP has been instrumental in our ability to quickly and easily pull data reports across all our clients. These pulls never take any identifying information about individual donors and are run through a secure server. I'll get more into the mechanics shortly but the investments that were made a few years ago to align with the FEP monthly data push have gone a long way toward helping make analysis and research a part of our company's culture.

Neon One also has three other "Business Units," which are the entities that were brought along with NeonCRM to form our company. While each company works with data, not every team has the same approach or standard when pulling data. I learned this when doing the GivingTuesday analysis. Luckily, we have invested time and energy into defining common data elements across each platform and that makes pulling a report on donations versus events versus something else much easier. We have also invested time in the legal side of things as well, centralizing our Terms of Service across the entire company that respects our client's trust in knowing research will be secure, anonymous, and with easy opt out. We have also now embedded this approach into our partnership agreements as well, allowing for research projects around third party data entering into our ecosystem.

Process

When we want to do a data pull, I either reach out to the data contact I have at Rallybound, Civicore, or Arts People or if I need to do something with NeonCRM, I can submit a JIRA ticket that then goes to our development team for review and engagement. We have created special tags for these types of projects and given that this takes away from time that could be spent building out our software, I try not to do this very often.

When asking for a data request, the following items are important to identify clearly:

• The date range
• The types of organizations we're looking to analyze (e.g. all clients or a subset of client information)
• The type of data within the client's database we want to query against
• The type of data that we want as an output
• Restrictions or things we want suppressed (e.g. filtering out data that may lead to double counting a number)

The most comprehensive pull is the one that I just did for December giving that built on the success of the GivingTuesday data. While we can get a pull of each individual client's system, this would be a pretty large file so we focused on aggregate totals for a wide variety of criteria. This takes a fair amount of time and effort, so getting the request correct the first time is really important. We try to avoid secondary pulls as well.

Usage

We are pretty proud of the work that our clients do, but sometimes the data doesn't paint a pretty picture or fit into the narrative that marketing and sales wants it to be. That's why industry reports need to have more citations on how the data was pulled.

A great example of this is the recent State of Artificial Intelligence In The Nonprofit Sector report that I assisted our partners at pwrdby with. What I love about that report is that there is an entire section around how the data was acquired, what is the framework around it, and the sample size for reference.

That last part is also most likely the biggest issue we have in the sector - Brady Josephson rightfully called out vendors who put out GivingTuesday statistics without giving more context about how those numbers were obtained (yes, we were guilty too). While there are certain things that I just can't share publicly, I think that making sure that the reference points for context around reports are important.

There are three kinds of lies: lies, damned lies, and statistics.

That famous quote is absolutely applicable for a lot of "research" that I see coming out from tech vendors recently. Personally, unless there's the vendor has solid access of primary resources that pertain to the subject itself, I'd take any report or whitepaper with a grain of salt unless it was done in partnership with an independent source.

Biggest culprit? Surveys. While not inherently bad (the SAINS report was survey driven), I've seen a lot of abuse around survey data then turned into a major call to action that leads toward a vendor's email list for sign up. Beware the survey driven "research papers" out there if its making major pronouncements about what we are supposed to act on or do in our daily lives.

Towards actionable research

Ultimately, vendors can play a very critical and key role in helping shape the direction we should be going in the nonprofit sector. But it is my hope that the following be instituted as rules, not exceptions when industry research is published:

1. Cite the sources and criteria on how the report was created
2. If not pertaining to the immediate subject matter expertise of the vendor, then partner with an independent entity to help perform the research in a meaningful way
3. When appropriate, try to make the data available for independent analysis

That last one is a big one and is one that I personally want to push more forward at Neon One, but there also needs to be protections and policies around that. Luckily, we have been pushing things forward at the sector level with some of the most important data from the Fundraising Effectiveness Project. And Giving USA does a great job making their major reports accessible as well.

I hope this is useful for understanding the processes and approaches that we are taking, though I know that every company does things differently. So I can only speak to what we do, but that's a little bit how our "data sausage" is made!

What do you think we need to help empower us with the right data to act?