Rising boats on a rising tide

Over the past several years, it is no secret that our industry has struggled with data quality. There are a few premises which have caused this outcome.

The first is that, as programmatic buying and selling of sample increased, we all got more and more interconnected. One bad source could inadvertently and instantly supply a bad set into the sample and research ecosystem - and the ecosystem would, programmatically, spread that bad source across.

The other elephant in the room is the nature and overall economics of the internet itself. Advertising led part of this change, and market research followed suit. We simply have not maintained parts of the equation (specifically price) that we should have. The more creative strategies to meet the same demand at a much lower cost per interview is having the adverse effect on data quality.

Let me also put out the disclaimer and say that I am a big fan of it (programmatic, that is, not fraud). Programmatic has helped me personally have a career for over 10 years, and not to mention, I started off my work in MR by ‘launching sample’ on Friday nights for movie trackers on the west coast. Although, I much prefer the APIs doing the job.

In any case, this process and evolution has dragged quality along, and we have been largely reactive.

All that said, I do think that in 2020/1 and going forward, we will see more investment in fighting fraud. Within the ecosystem, we can likely squeeze some more efficiencies which should alleviate the pressure on price to some extent, and there are a few topics which will be relevant to and improve Data Quality in the coming years.

1.     Scoring Consistency

Our definition of fraud must be standardized at some level. Currently, we have many different technologies and internal projects. This causes a lot of wastage and the lack of overlap drives down conversion, and falsely terminates more sample, adding to the pressure we already feel. In simple terms, the suppliers have a rule for what is good/not, the exchanges have another set of rules, and the buyers have their own.

This aspect adds to the failure rates across the spectrum. We have talked about the ‘tragedy of the commons’ ad-nauseum in the context of fraud, but this aspect is yet another piece where a lot of suppliers carry a lot of the burden across the industry. Buyers also carry some cost in terms of reduced feasibility.

If we are to go down this path, we could effectively reduce wastage, and hence increase conversion, alleviate the pressure on price (partially).

2.     Open Source(?)

 Open-source software is a potential option here as well. Data sharing across platforms/ marketplaces/ exchanges/ panel companies / buyers/ fraud detection companies, and ‘opening up’ our fraud scores and scrub out information will help alleviate some of the inconsistency issues.

We (SampleChain) don’t have experience with an open-source project, and structuring this could be tricky, both financially and resource wise, but I do think there are some good outcomes to be had on this path.

Of course, as with any open-source project, the project will have the disadvantages of ownership, support, and continued investment. We are all strained for resources and we will have to find a way to continue generating revenue. Nevertheless, it is a potential path to standardize/ commoditize our definition of fraud.

3.     Lowering Fraud Thresholds

We also recommend lowering the scores and thresholds we have for “quality scoring”. Most companies today (fraud detection and internal software) have some level of thresholds or scores that they use to grade respondents. In our experience, this usually ranges between 0 and 99 – or some variation thereof – and our recommendation is to take the scores down, in theory all the way to “0” or ”1” and only respondents who have no risk whatsoever to be sent into surveys.

Currently, most users and projects managers have control over these settings. In many cases users simply “raise the thresholds” to get more sample – I know because I am guilty of doing it myself.

In most other contexts, a system likely will not allow this, but we help ourselves to this ‘luxury’.

Point 3 is point is the only point within the data set which is recommending us to ‘cut’ more sample. However, I do think that if we all rally behind tighter security, the quality, demand, and other items will follow suit in a positively trending direction.

4.      Price

For this aspect, I think I can speak freely without 'getting into trouble'. As a fraud detection vendor, we are completely un-tied from the prices that the buyer and suppliers engage with. We are Switzerland, as it were (that is usually a good analogy, right?)

What I can say from our angle is that change does have to start at the top. Even though we would love to claim that our technology can ‘solve fraud’, that is simply not true. We need to be aware of the effect that lower costs are having on the industry. It is simply unsustainable to measure the returns / payments to the end respondent in the order tens of cents per minute, likely even less.

Yes, we can talk about the fact that higher prices attract more fraud, and that we can argue that the best research is unsolicited. All of this is true, but not scalable – and besides, those are details within the larger picture of supply and demand. Whether we like it or not, supply/demand rules dominate every single industry (GameStop, anyone?).

As a simple and personal example, I participate in a fully engaged manner when I get phone surveys from GLG or other B2B research companies which pay me a healthy rate, orders of magnitude higher than what the online quantitative is currently at.

The internet likely does not demand the same economics as phone, but it would behoove us to move in the upward direction and rise with the tide.