Drive and Convert

Announcer: [00:00:00] You're listening to Drive and Convert, a podcast about helping online brands to build a better ecommerce growth engine with Jon MacDonald and Ryan Garrow.

Ryan Garrow: All right, Jon, you know me fairly well. And you know that my default is to come up with really cool ideas and move fast and break things and figure it out as I go.

I build the car as I drive it. And it doesn't work all the time. And for most of my ideas, my wife shoots them down from the very beginning. And I don't even get a chance to go work fast and try to break it because she's it's a bad idea. But I've learned if I can quickly bake something up and be like, I think it's a good idea.

Let me, I'm not executing the idea, but I'm just going to do this really quick thing to prove that it's not dumb. I have a much better shot with my wife to get an idea through the system and get approval in the family business ideas. And I thought I was pretty unique in this. And then you are, you've told me about this cool idea you have about or this new thing that I've never heard of about how companies do this on their websites all the time.[00:01:00]

And there's even terms for it. Something about smoking and, but it lets us get, you can do it with less data. You don't have to go through the full A/B testing to prove an idea. It's quicker. I, at the end of the day, it appears that we get to move fast on websites with this cool idea of yours with a lot more confidence.

And I, for me, that is I'm all in on this. And so how do we do this? Tell me what I need to smoke to get this thing working.

Jon MacDonald: Yeah there's a lot of options for what you could smoke for sure, but we'll leave that for another day. Especially in Oregon. Yes, exactly. Look, whether you're building products or marketing campaigns, time and money are just precious commodities.

So I hear you. It makes sense. You want to move fast with confidence, right? There's so much pressure these days to make the right decision the first time. And that is something that we haven't really seen a lot over the years. It's always been something in entrepreneurship and businesses, but lately we're finding that experimentation is important. Nobody wants to try [00:02:00] things and fail for a year until they find something that actually moves them towards their goals. And I don't think this is a generational thing. I hear that quite a bit, but I do think it's because marketing and the technology stacks around that has been moving so quickly that everybody expects to be able to optimize their site quickly now too.

And A/B testing can take some time. It really can to do it right. I think this is where a term I'm calling smoke testing becomes really invaluable. And I say I'm calling, we should talk a little bit about what it is and where it came from. We'll get into that today. But it's not a term that I invented or that The Good invented.

It's a commonly used term in optimization. Actually a lot of different fields, but smoke testing is providing a quick and early feedback on your ideas, the functionality you add to your website, your products. It allows you to validate concepts without really just jumping into full scale development.

So it comes from lean methodologies [00:03:00] to some degree, right? Helps you make informed decisions and reduce the risk of costly mistakes. That's the best way to think about it.

Ryan Garrow: I think that sounds amazing. I would, I probably want to use this in a bunch of other areas, but what does it, what does smoke testing mean when you're speaking about website and DXO?

Jon MacDonald: Yeah, so smoke testing is a process where you would test the basic functionality of an idea or a product before you implement it fully. The goal is to capture valuable feedback. Before you spend time or money or any other resources in developing something for real. Normally we would have done this with A/B testing, right?

But in A/B testing, you often have to develop both options, right? It does require development resources, right? Here, your goal is. To run a smoke test and if it passes, you move forward with confidence. If it fails, you just abandon it without having spent a lot of time and money. Smoke testing also goes by the name of confidence testing or validation testing.

So in short, it's gonna [00:04:00] help you validate your idea And smoke test can even preempt the MVP, or in lean, what you might call minimum viable product concept.

Ryan Garrow: Got it. I don't want to call it confidence testing because that's not nearly as cool as saying, Hey, we're going to go through some smoke tests and figure out what works.

Jon MacDonald: And in reality, if we're getting super technical, which I know we probably don't need to do, but confidence testing is a completely different thing as well.

Ryan Garrow: Got it. Okay. So what, so give me an example, so I can put my brain around this. What is, what If you're going to do smoke tests on a site, what does that walk me through that?

Jon MacDonald: Okay. Let's use maybe hypothetical online media site. I'll just throw that out there. And they want to implement a snooze my subscription feature. So I subscribe to the economist who's one of our clients and I want to snooze my subscription, but the economist doesn't really know if it's going to stop people from canceling, right?

They just say, Hey, this might be a downgrade opportunity, or, Hey, I'm going on vacation for a while. I'm not going to use this, [00:05:00] whatever it might be. So what they could do is instead of building out the full functionality on their site, they could add a button that says, as part of the cancellation process, offer the option instead, I want to snooze my subscription.

And they would measure how many clicks they're getting on that.

Ryan Garrow: He's put an event tag on there like a GA event tag.

Jon MacDonald: Yeah. Whatever tracking you're using, GA, et cetera. Yes. Now there are two ways to look at this. Maybe the button doesn't really do anything but take you to a simple page that says this features.

Okay. Coming soon. Thanks for your interest or in this type of situation, what I would probably want to do is have it connect you to a customer service, right? Something that they can help you implement this, but it's not a fully automated process at this point, right? It might be something like, Oh, great.

We'll pause your subscription. We'll be back in touch to confirm. You can just do something that simple, right? Okay.

Ryan Garrow: Got it.

Jon MacDonald: Where it's very manual, because you're not sure if people want this. Don't build out the functionality. Just do whatever is the [00:06:00] absolute minimum.

Ryan Garrow: I can even imagine that you would just even put like a an email button like snooze at and it would just go to customer support and be like, Hey, this isn't quite ready or to manual process.

And you'd be like, Oh, I can still pause their subscription for three weeks. Exactly.

Jon MacDonald: The idea here though, is not what happens after they click the button. Really it's the number of clips on that button. That's going to help you understand how many people would consider using such a feature. Because there's a whole optimization you can do of that feature later, right?

You could go out and say, Hey what are the different, you could test the different pause lengths, right? You could smoke test a hundred different things after you test that people even care about this. So the goal here is what's the smallest thing you can test that's going to give you the right direction to head in.

Then you're going to continue to run a ton of these smoke tests at each step.

Ryan Garrow: Got it. Cause what in theory you could be. testing this. So snooze subscription. And in reality, 100 percent of them are going to cancel anyway. So why delay this way for [00:07:00] whatever reason?

Jon MacDonald: Yeah. So why even offer it then? You're just like, okay, instead I'm going to focus on smoke testing ways, other ways to get people to not cancel.

Ryan Garrow: Okay. So why do we, I guess before I asked for a why, where did smoke testing come from? Cause it doesn't, he don't smoke. Seems a weird analogy in my head at least.

Jon MacDonald: I hear you. And I agree. The first time I heard it, I was Oh, that's interesting. Okay. I don't know if you've ever heard. I went to, have you ever heard of, there's a website out there called HIGHdeas. It's a hilarious site if you ever just want to kill some time, it's people who basically it's like ideas they came up with while they were high.

Okay. And I was like, so is this just like coming up with random ideas and testing them? And you're just like, Oh, whatever. And it's a fun play on words or whatever. And it turns out, no, it's got much more boring kind of background to it.

Ryan Garrow: Dang it. I'm going to probably go with that as my original idea though.

Jon MacDonald: Yeah. So we don't really know where the term came from. If you do, I did research on this. Natalie on my team has written an article about this. It comes out this week at the [00:08:00] good. com. She did a ton of research on it. We don't know where it came from, but there are two origin stories that popped up more than once.

So one is that it comes from the plumbing industry. This is because they use smoke and to test for leaks and cracks in pipes, so it's a quick way to identify a problem. So another possible origin that came up multiple times was from hardware testing, like hardware, like electronic devices. where they are initially switched on and tested for signs of smoke in their components to make sure that everything's wired together.

Ryan Garrow: Yeah. Smoke is bad. Okay.

Jon MacDonald: Yeah. So it's a quick and easy way to be like, was everything wired? Okay. Oh yeah. Okay. Nothing smoking. We can move it along in QC. But nothing's going to blow up. We've already tested to make sure it's not going to blow up right away. So I would hope that now that I think about that, that every single device that plugs in that I'm using has gone through smoke testing.

Probably not the case, but I would hope that would be the case.

Ryan Garrow: There's probably some things on Amazon that haven't gone through [00:09:00] that.

Jon MacDonald: Yeah. Don't go to Temu or whatever that is. So it's hard to say if either of these are true, but I think they are potential smoke testing origins and they are feasible uses of smoke testing is a larger type of context, but it really has evolved into a widely used term in software development, and I think that's what's cool about this. It's a subset of tests used to assess if a software build is stable enough for the next development stage. It identifies bugs that kind of just block the release of a product pretty quickly.

However, In classic marketing fashion, because marketers like to latch onto everything, growth teams have really adopted and redefined smoke testing as a quick tool to validate ideas. So that's how we're going to talk about it or I'm talking about it.

Announcer: You're listening to Drive and Convert a podcast focused on ecommerce growth.

Your hosts are Jon MacDonald, founder of The Good, a conversion rate optimization agency that works with ecommerce brands to help convert more of their [00:10:00] visitors into buyers, and Ryan Garrow of Logical Position, a digital marketing agency offering pay-per-click management, search engine optimization, and website design services to brands of all sizes.

If you find this podcast helpful, please help us out by leaving a review on Apple Podcasts and sharing it with a friend or colleague. Thank you.

Ryan Garrow: Okay, I get the concept. , but in a basic Shopify store. Why even use it? Just find an app, try something. Is there something Yeah, that makes it more applicable than other things.

Jon MacDonald: Oh, for sure. Look, as I mentioned earlier, the beauty of smoke testing is that it provides quick validation without requiring significant resources. And that's where I think for midsize or smaller e comm sites, it really can make sense, right? So you get to find out if an initiative has genuine interest or demand before you invest anything into that, right?

That means smoke tests, save time and money. But feedback is also another benefit. So if you put out a preliminary [00:11:00] idea, you get to quickly learn what potential users think and feel about it. So that's also really valuable, right? And third, they're just great tools for prioritization of functionality and features.

If your resources are limited, it's important to focus on what's going to bring you the most value. And that's where smoke testing can really come in.

Ryan Garrow: No, I am, I'm a big fan of this. I assume I can't go smoke test five things at once on a site because that would be a lot of buttons that probably don't do what you intend for them to do.

Yeah, you don't want to.

Jon MacDonald: The idea here is to not. have the consumer know that you're smoke testing something. That should be your goal. What is the minimum you can do without making it so obvious that you have, are putting up a facade? That's what I would look at.

Ryan Garrow: Okay. I think that's a great goal for this then.

Okay. And I love it because, we work with a ton of small businesses that don't have traffic. For full A/B tests on a lot of things where the finance is to jump into the deep end with a DXO organization like yourself. So it's in my mind, this is really [00:12:00] helping somebody say, I got to do something, but I don't, and I have some ideas, but I don't know yet what it's going to be.

So I'm going to, I've got these few, like list of smoke tests I want to test rather than try all of them all at once. What are the steps I would take on the first one to say, okay, this is the smoke test I want. How do I know that I've gone through the process to either say it's a good idea or a bad idea?

Jon MacDonald: Yeah fortunately, I think there's simplicity in smoke testing. So setting one up is really straightforward, right? I think what we could do is break this down into a handful of steps because I like to make a process of everything, as you

Ryan Garrow: do, yes.

Jon MacDonald: I think there are five steps that I would do here and look at this.

The first is, of course, start with the end in mind. So establish your acceptance criteria, right? Okay. If you understand what your go, no go decision is, you're going to have clear terms for how the outcomes of your tests will determine what you do next. Okay. So for example, just have a definition of what success is.

So if we generate x dollars in preorders for the new product within the next month

Great [00:13:00] question, right? So I think this is a great way to test if you should even launch a new program, right? Or a new product or anything like that, right? You just basically say, Hey, we're going to put up a product detail page, not have the product, and then just say it's pre order. And how many pre orders can we get before we launch the product?

What do we need? You can always say, We won't charge your card until we ship. And then just email people and be like, Oh, really? Sorry. We canceled that because there just wasn't enough interest. If you are super interested in something similar, here's another product. So the first step is.

What's your acceptance criteria? Then second is design that simple experiment. Keep in mind though, that this should be really simple, right? You might create a landing page for that product and collect pre orders like I just talked about, or build something that's just quick and dirty, but still presentable like the pause my subscription functionality that I mentioned, right?

And I'll use functionality [00:14:00] with air quotes because you're, again, you're not building the functionality. The third step, I would think, should be to drive traffic to the experiment, okay? If you don't have traffic on your site, I would find other ways to put it in front of your audience. And how you do this depends on the nature of your test and, of course, who you're trying to get to interact with it.

But if you're building an online experiment, excuse me, online service or experience of any type, and you don't test that with fake AdWords campaigns ahead of time, I think you're crazy. So you really should be trying to drive traffic and you're going to have to pay for that. But AdWords is a great tool for that.

You'll see how many people click on it and go in. Yeah. I feel you. Speaking my language. All right. And he'll help you set up that fake AdWords campaign and get fake.

Ryan Garrow: And fake means that you're. You're going to buy some keywords and measure interaction rates and see if it has potential to convert and drive revenue, right?

Because this is something that's going to be important. Yeah.

Jon MacDonald: It's not something you're planning on doing for the longterm right up front. It may be an optimized campaign for instance, [00:15:00] right? You may be spending more than what you would want to spend because your quality score is low or whatever. You're just trying to get something quick and dirty done, right?

And then you really are going to want to work to optimize that campaign and make sure you're spending appropriately. Once you say, okay, yeah, I've validated this. So far we have established the criteria, design the experiment, drive traffic to it. Once you're ready to drive traffic, launch it and then track that engagement, right?

Let it run and collect data according to that success criteria we set up in step one. But while you're tracking, you may come up with new ideas. This is almost always what happens. You're going to have that idea, be tracking it and get some data and you're immediately going on, shut it off and say, Oh, I'm going to do this instead.

Resist that temptation. The first thing you should do is document those ideas. You can run additional experiments or adjust it to inform better decisions, but give it a little bit of time. How much time is really up to you. This is a smoke test. After all, you don't want to do. a ton of invest a ton of time, [00:16:00] but don't run it for a couple hours.

Don't run it for a day, right? Give it a, a little bit of time there. Lastly, what you're going to want to do is make your decision pretty cut and dry. But the reality here is you need to understand whether the results warrant you going forward with the idea. And that's really it. So hopefully at this point you'll have as much data as possible to support your decision, but that won't always be the case, right?

Sometimes you just need to make a judgment call and that's okay with smoke testing. If that's the case, you might also decide to do an additional test with a little more effort. But again, don't spend a lot of time making your decisions. This is really all about being fast and decisive.

Ryan Garrow: And I think for, if I'm looking at this, there's a couple of things like it, my decision making would come down to the amount of data I have would be like, would my life, what is that enough data for my wife?

Can she say, we could do this over here in the business world? Like my business partner, the executive team, is this enough data to say, yes, we can invest in this. And I do that a lot, actually, logical position. I'm like, Hey, we got to do something [00:17:00] quick. We can't, we're not going to put a bunch of resources behind something that may or may not work.

So Garrow's going to go do some, thing I'm over here that. Maybe it does something, maybe it doesn't, but it doesn't break anything to do the test. And that's really been one of my overarching goals for a lot of things, but I can say I have really simplified this process into one step, Jon. Ask Jon.

And then it's much easier. You just, I just have one step.

Jon MacDonald: Love that. If I'm a user, you're, the goal here is to get user feedback before implementation, right? So yes, I think before diving into development, it's always smart to gather feedback, whether that's a smoke test or a Jon test or whatever you want to call it.

I think smoke testing, what we talked about today, I think it's practical. I think it's efficient. Let's you test the waters before you really commit any resources. And I think that's just a great ethos for optimization in general, right? Too much stuff with CRO over the years has been flooding the market with things that are [00:18:00] either too simple, like just do this checklist, whether or not you know, it's going to work for you or too complicated.

Like you got to run all these A/B tests are going to take months. and a lot of resources and development and everything else. This, I think, is a great medium. It's a great way to get quick feedback with data backing it and move forward. And it doesn't take building out your ideas to give yourself proof or your wife proof that it will work.

Ryan Garrow: Yeah, this is honestly one of my favorite ideas because it can work across an organization of any size, which tends to be some of the limitations with a lot of the things you and I know is as out there in the market or available to companies, because we work with a lot of large ones that says, man, this is a super cool program, but if you're doing less than 20 million a year, it's just not going to work or not.

You're not gonna be able to afford it. Whereas this is, man, if you can just get a simple thing on a site, which, Upwork for 50 bucks may get you enough development talent to get something done on your site for a smoke test. So I love it.

Jon MacDonald: And with Shopify, a lot of these things can be done in the editor without you really [00:19:00] needing to do anything, hire a developer at all.

So something to consider.

Ryan Garrow: Nope. I love it, Jon. Thanks for telling me about smoke tests. It had nothing to do with what I originally thought about when you said smoke.

Jon MacDonald: Me neither. I was, my hypothesis was quickly shot down by the team as well. Yeah. All right.

Announcer: Thanks for listening to Drive and Convert with Jon MacDonald and Ryan Garrow.

To keep up to date with new episodes, you can subscribe at driveandconvert.com