11/14/2017 | Episode 10
Kevin Lee is a Trust & Safety Architect at Sift Science.
A former Marketplace Manager at Wanelo, Courtney Bode has since founded OpsTales.
Evan: Welcome to “Trust and Safety in Numbers,” presented by Sift Science. I’m your host Evan Ramzipoor. Here today with Courtney Bode, Marketplace Operations Manager at Wanelo, and Kevin Lee, Trust and Safety Architect at Sift Science. Thanks for joining me. Today we’ll be talking about chargeback resolution and building up your fraud strategy. But before we get to that, let’s warm up with the fast fraud fact. Did you know that two thirds of millennials don’t password protect their mobile devices and computers? That’s even though most millennials store financial information on their devices. For more information on millennial online trends, check out 10 things you need to know about digital natives on the Sift Science blog. Now on to the interview, so let’s start with this. Tell me a little bit about yourself. What do you do? What does Wanelo do? What’s your story?
Courtney: I am Courtney, I’ve worked for Wanelo for a little bit over two years. I started there to build out their merchant department. Wanelo is a social shopping app. Our demographic are young females looking for fashion and that wanna share it with all of their friends. So when I started they were just starting to build their marketplace and they had me come in to do the merchant side of things, but pretty quickly on I started to see chargebacks. I was like, “Hey, guys, these cost money and we don’t have a process around it. Does anyone know what’s going on here?” And we had some people haphazardly responding here and there, but again, we had a very small knowledge base and so that was when I first started getting into fraud. We’ve operated with a team at most of about 15 people on manual review and chargebacks. Right now, we are a little smaller because we’re trying to optimize.
Evan: Nice. And what about you, Kevin Lee? Most of our listeners probably are familiar with you at this point but who are you? Remind us.
Kevin: Well, hi there. I am the Trust and Safety Architect here at Sift Science. Basically, that means I am the in-house advisor when it comes to anything fraud, risk, trust and safety related.
Evan: Courtney, a lot of people have heard of payment fraud, but some may not know that payment fraud actually comes in two flavors. There’s the straightforward malicious payment fraud that most people are familiar with and then there’s friendly fraud. Can you tell us a little bit about the difference between those two?
Courtney: Happy to. So malicious fraud, you know, most people know about. That’s when someone, for example, steals a credit card and goes on a shopping spree and they’re basically trying to spend as much money as possible. They are generally, I would say, the majority of the fraud industry. Wanelo is a very unique case in that rather than experiencing malicious fraud, mostly we experience friendly fraud. Friendly fraud is when you have a perfectly normal looking user, meaning that all of their behavior is normal, their history is normal, their purchase looked perfectly acceptable, and yet two months later they turn around and dispute the chargeback. This happens a lot on Wanelo because of our demographic. Again, we have primarily teenage females buying fashion items, so a big case for us is when a girl goes and takes her mom’s credit card, spends a couple grand on new clothing, a month later her mom finds out and her only response is to chargeback all of the orders so that she doesn’t have to spend the money. So these are really unique because our entire fraud strategy had to change in order to attack these rather than to fight malicious fraud.
Kevin: So in today’s world, is friendly fraud your biggest thing in terms of fighting that stuff or do you still have to deal with a lot of traditional clean type of fraud?
Courtney: The majority of our fraud is friendly fraud and there’s a big reason for that. We built a very comprehensive system, particularly using Sift Science which means that most of the malicious fraud that comes to our platform is fightable. We catch it before the order can be placed. Whereas friendly fraud you can’t prevent it, you can only attack it on the end by winning the chargeback.
Evan: Walk me through a typical chargeback resolution.
Courtney: So a typical chargeback resolution for us comes through Stripe, our payment processor. We have one girl right now who actually operates out of the Philippines. She receives our chargeback, looks up the user in our system and compiles essentially an on-the-spot analysis. She sits there and says, “Why was this order processed? What does this user look like? Do they have good behavior, bad behavior? What did Sift Science think of them? Can we verify their identity?” So after she does this initial analysis, she then downloads a CSV of all of this customer data, goes back into our chargeback response form through Stripe, types up out of a template a concise response that illustrates why we processed this order and why we think this user deserves to be charged. After that, she compiles all this information into a tracker on an internal system so that we can easily monitor the data feedback loop. Each week she then submits to me a summary of data on these chargebacks and I then use that summary to adjust our system based on new trends.
Kevin: Sorry, did you say the girl is in the Philippines?
Courtney: She is in the Philippines.
Kevin: How did that happen?
Courtney: As a matter of fact, we started our team with contractors in San Francisco and that’s very, very expensive. So over the last two years, we have slowly moved our fraud team over to the Philippines.
Kevin: Obviously, here in San Francisco it’s expensive, but you’re at least right next to me and we can talk and like figure things out. How did you decide, “Hey, we want to put this person halfway around the world,” or why did you decide that?
Courtney: We already had our customer support team operating out of the Philippines, so it wasn’t a huge step to hire people for fraud. Since fraud developed specifically through me at Wanelo, I had a very large knowledge on what it took to train people and what requirements we had for our system. So we very slowly built an extremely thorough training regimen and hired one person and then we went up to two and we very, very slowly and methodically carefully added people to our team. And you know, again, the big takeaway is we moved very slowly and very methodically. As for some of the struggles that came with that, we’ve traditionally held a very high win rate. We see between a 40% to a 65% chargeback win rate at Wanelo, but when we switched our chargeback responder to the Philippines, we saw a drop or to about 30% to 40% win rate for 2 to 3 months while she picked up the pace and really understood our process. So there was a big cost, but now our win rate is back up to our norm and we’re operating at about 10% of the cost now.
Evan: So to build up your fraud strategy, you’ve used an approach that combines machine learning rules and manual order review. Why go with three approaches rather than just sticking with one?
Courtney: It’s a great question. Again, Wanelo faces a pretty unique trend with fraud. We don’t see the same things that everyone else does. When we first built our system, we wanted to outsource completely. So we looked at a few other companies like Signifyd and Chargebacks911 and because we knew our data so well, we were able to give them a breakdown of what specifically they should be able to fight for us. These companies could not target friendly fraud and they could not help with our service related chargebacks. They could only help with malicious fraud. For us that didn’t work because only about 10% of chargebacks fit into that category at that time, so we had to look internally.
So we first built a team to fight chargebacks, that was step one. And then immediately after that, we started working with Sift Science. And Sift was so instrumental in helping us build this multifaceted system because you guys gave us machine learning which gave us an instant rating, so we could immediately pull out the obviously malicious fraud. Then you guys helped us build a manual review queue where we can say, all right, there’s a gray area for users. Some people have, I don’t know, a median risk level, but they do have some other characteristics like the IP address changed. It’s 200 miles away now and their order value is over 200. We want a rule so that we can pull those into a special queue, but we can’t at this point in our company automatically reject those people, we don’t want to get rid of money.
So then once we have machine learning filtering to rules, filtering into a manual review queue, we had to have someone review those orders. So then we have a manual team who goes in and reviews. Besides the high-level data, what does this user actually look like? Are they real? Are they leaving comments? Are they reviewing products? How often do they use our platform? All of those combined have built an extremely comprehensive strategy for us.
Evan: Interesting. So that’s the positive side of using multiple strategies. Kevin, what are some of the challenges that you’ve encountered using multiple approaches to fight fraud?
Kevin: I think one big one would be around making sure that the data is digestible across the entire platform. This is one thing a lot of companies, larger companies, let’s say, can face. Whether it’s you’re building a product from the ground up or maybe you acquire a company and you are integrating them into your system. This happened actually quite a bit between Facebook and Instagram, let’s say, where, for example, Instagram may have a ton of data on bad users, Facebook also has that data, but the systems don’t always talk to each other and are able to communicate and identify the same user that happens to be using both platforms. And so you definitely want to have that later approach where you have human beings, rules, machine learning models in place but making sure that the underlying data is digestible across all parties was pretty key.
Evan: Throughout your careers you’ve probably learned a lot about what to do and what not to do in fighting fraudsters. For all of the other fraud fighters who might be listening, what’s your biggest what not to do? Let’s start with you, Courtney.
Courtney: One of the biggest mistakes I made was when we first created our rules through Sift Science that put our users into our own internal manual review queue. I wasn’t quite as cautious as I probably should have been and within about an hour we had hundreds of users with hundreds of orders in this queue and literally just me and one other person there to review it. And as you can imagine, that was an all hands on deck type of fire but we didn’t have hands to help. So as the very new person having just built this strategy, I had to run to the CEO and be like, “Something happened, I’m gonna fix it.” We paused everything, spent a lot more time thinking thoroughly about the rules and the potential impact, and then two weeks later relaunched.
Kevin: I’d say one thing if you’re just starting out in this field is don’t think that you have to do this by yourself. Oftentimes, if you’re a team of one or a team of two, it can feel really isolating and kind of lonely but there are teams, tools, people out there that do this each and every day. And I’m not talking about just in this room in terms of a podcast but there’s meetups, there are companies that are your competitors, let’s say, in this space, they don’t necessarily have to be your competitors. When it comes to making the web a better place to interact on, it’s not just about Company A versus B versus C. Most of these companies, yes, you want to have your competitive advantages in your things in your products, whatever you want there, but when it comes to fighting fraud, really you should be more collaborative when it comes to fighting that good fight. We know that fraudsters, for example, are very good at collaborating. They have forums and things dedicated to such behavior. On our end it tends to be a little bit tougher sometimes to collaborate because you don’t necessarily want to give up the secret sauce. But the truth of the matter is, it’s not going to get better unless you kind of face that fear and if you do it by yourself, it can be a really long tough road, but if you include not only other people in your company but also across the industry, it can make that fight a lot easier.
Evan: Excellent advice. Share the secret sauce. Courtney Bode and Kevin Lee, thank you both so much for being here.
Courtney: Thank you.
Kevin: Thank you.
Evan: Thank you for joining me on “Trust and Safety in Numbers.” Until next time, stay vigilant, fraud fighters.
Learn more about what sets Sift Science’s machine learning apart.
With billions of compromised credentials already in criminals’ hands, how do you protect your users’ accounts, your brand, and your bottom line?