11/13/2018 | Episode 30
Courtney Bode is a business operations manager turned fraud fighter. At Wanelo, she has built a fraud strategy combining machine learning, rules, and manual order review to catch suspicious behavior across thousands of merchants and millions of users.
Evan: Welcome to Trust & Safety in Numbers presented by Sift Science. I’m your host Evan Ramzipoor. If you’re in a group of people and you want an easy way to figure out which of them works in e-commerce fraud, just mention the holidays. Did one of them look sick to their stomach? You found the fraud analyst. Unfortunately, the most wonderful time of the year is also the fraudiest. Last year, e-commerce fraud shot up by around 22% during the holiday season. That’s why I’m sitting down with Courtney Fahrer, a fraud expert and the former head of marketplace operations at Wanelo. Courtney built Wanelo’s fraud operations from the ground up, so she has some invaluable tips and tricks for surviving the season, whether you finalize your planning for Q4 already or you’ve convinced yourself there’s no time left to plan. But first, let’s warm up with a quick fraud fact. Did you know that the fraudiest day of the year is not during the holiday season? It’s not Black Friday and it’s not Cyber Monday. So what is it? To find out, check out 10 Fraud Myths on the Sift Science blog. Now onto the interview.
So, Courtney, you’ve been a guest on the podcast before, but you’ve been up to a lot since we last chatted. Can you tell me a bit about yourself and your background and what you’ve been up to?
Courtney: I currently have a company called OpsTales and I’m helping other companies to start outsourcing their internal operations. Some of the big areas would be within fraud prevention, starting to build a repeatable chargeback process and then hire an outsourced agent to manage that. Before that, I was the head of marketplace operations at Wanelo and one of the major things I did there was to build their initial fraud strategy.
Evan: During the holidays, most companies see a massive rise in order volume, but how does that impact their fraud? What does fraud look like over the holidays?
Courtney: Well, the holidays are great for business because sales go up, but with the sales increasing, one of the big things that happens are that normal users start to have very changing buying patterns. So that looks like different addresses used to different credit cards, different IP addresses. Even standard statistics like average order value, those can skyrocket. So it’s a problem that all these buying patterns start to shift just during the holidays because when you’re using a purely automated system, it can take a little while for your machine learning model to catch up to these slight changes. In short, fraud prevention becomes significantly murkier and it becomes harder to detect true fraud versus a user that might just be shipping a really expensive watch to their mom across the country as a special present.
Evan: Can you elaborate on the specific changes in user behavior that people should anticipate during the holidays?
Courtney: When you’re going into the holidays, there’s a few things that you want to do. First, you want to work with your sales and business team and start to understand what their projections are for the holidays. The next thing you want to do is start analyzing your historical holiday patterns. You want to look at last year. Did you see the average order value change from $50 to $200? If yes, and your holiday projections are high from your sales team, it’s probably likely that you’ll see a similar pattern this year so you can start making assessments that way and make adjustments to your process. The third thing you should do is start talking to others in the field. While we know companies don’t like to share insider information, fraud is the one space where we really, really need to work together. So do what you can to make connections and share what you’re seeing so that others reciprocate.
Evan: So as many of us know, a lot of companies use a manual review team to parse fraudulent and good orders as they’re coming in, or they might use manual review in conjunction with something like a machine learning solution. Let’s talk about how people can expect their manual review process to change during the holidays. What are some tips you can give people for fine-tuning manual review?
Courtney: So the first part about manual review is that there’s rules that put orders into manual review and that’s what’s so great about Sift Science. The rules that they have are pretty comprehensive and they’re also extremely easy to adjust. So as you go into the holidays and you start finding these expected changes in behavior, the very first thing you need to do is go in and start adjusting your rules. You want to increase the thresholds for flagging orders so that your team isn’t significantly overwhelmed. After you’ve adjusted your rules, the next thing you need to start doing is rethinking how your manual review team process looks like. With the holidays, you’re gonna see a big spike in the gray area users. These are users that are neither obviously good nor obviously bad. So for your manual review team, you need to make it easier for them to figure out what these users are.
You also want to set some thresholds. You want to be careful that your manual review team isn’t declining too many orders, so one really great thing to do is to set a time caps and a max decline rate for agents. An industry standard would be about 2% to 5% of orders rejected.
Another good thing to do is set up an extra form of verification. It needs to work for your specific business and what type of fraud you’re seeing, but an example that I’ve used in the past is Whitepages Pro, that gave us the ability to plug in our addresses and phone number, IP address, email.
Lastly, another good thing to do is set up as a shortlist of key trends for good and bad users. You should maybe have 5 to 10 qualities. If your manual review team sees, say, three to five good qualities, they stop the review right there and accept. Same goes for fraud. This was really important because many analysts are generally rather risk-averse, and so as your fraud queues or your manual review queues start to increase during the holidays, some of these agents will be more inclined to reject these gray area users.
Evan: What about growing your fraud team? Do you necessarily need to grow your team during the holidays to accommodate the increased volume? And if so, what does that look like?
Courtney: My experience, it’s a solid yes. Because your orders are increasing, it’s pretty consistent that your manual review queue will increase. And during the holidays if your orders are in manual review for too long, that pushes things like the shipment dates out too far. And since the holidays are such an important time for consumer buying, too long of a shipping date can cause A, customer unhappiness or orders not being delivered in time. So holistically, if you don’t increase your manual review team as an e-commerce platform, there’s a great chance that you are gonna create a really bad user experience. In terms of to what extent, again, you got to work with your team on projections and you got to see if you can adjust your rules to catch less orders.
The next thing you need to do is figure out what that looks like. Wanelo, we did something a little new and that’s what I’m here to talk about. So we have people in-house and they cost a ton of money and when we got to the holidays and we had two weeks perhaps until Black Friday and we needed to scale quickly, hiring in-house didn’t work for us, period. So we got a little bit creative and we started hiring outsourced agents to do manual review. We were able to keep our team at about two to three agents yearlong and then during the holidays, over two weeks we could grow our team to eight people just until January.
If outsourcing does sound like something you’d be interested in, there’s a few different ways that you can approach it. At Wanelo we went the not so intelligent way of hiring directly off of Craigslist from the beginning until we found a leader who can manage it. And the second option is to work with an agency. That’s the option I would recommend, especially since we’re approaching the holidays so soon. I started OpsTales this year so that I could help companies to outsource operations in this manner.
Evan: Since we’re already so close to the holiday season, some people might think there’s nothing more they could do to prepare for the influx of fraud, but that’s actually not true. Do you have any last minute tips for fraud fighters doing their final holiday prep?
Courtney: As Black Friday rolls around, it’s a really good time to review that piece of data before you move closer to Christmas. So when you see your Black Friday results come in, step aside, take a look at your data and see how your manual review team performed during this Black Friday spike and then see if any chargebacks were immediately triggered. It’s really great time to fine-tune your process as these new trends arise because inevitably the three weeks before Christmas, these trends just continue upwards. So take a step back while you have data, review it, and make last minute changes.
Evan: Thanks for joining me on Trust & Safety in Numbers. Until next time, stay vigilant, fraud fighters.
Learn more about what sets Sift Science’s machine learning apart.
With billions of compromised credentials already in criminals’ hands, how do you protect your users’ accounts, your brand, and your bottom line?