3 Tips Before You Launch Your Beta
At Achilleion, we’re almost finished with our technical beta and are about to move onto our ‘marketing’ beta (or soft-launch). So I thought I’d share some of the insights I’ve learned over the past 3 months launching and testing software on iOS, Android, Mac, Windows, and on the web.
1. Plan Your Beta
Start with defining your goals clearly and making sure the whole team is on the same page. It sounds straightforward, but even a small team like ours (6 people) can get their signals crossed. Basic things like when a certain group of users will be added, which platforms to start with, and how long each step of the process should last. Beta testing is a critical part of launching any app, and you want to utilize this time to perfect your product as much as possible.
One important part of your plans and goals should be how many people and how long you want to test. Some people will tell you to launch, launch, and launch, but that isn’t a good idea for everyone. You have very few chances to prove to people your product solves their problems, and launching with a buggy platform is a fast way to convince them to look elsewhere- especially now that more products aren’t free (I know it’s crazy). As Joel Spolsky put it, you probably need at least 100 people to test. Also, set a realistic time expectation; I can’t imagine a proper beta test taking less than 2 months unless you’re releasing a very basic or simple app.
2. Pick the Right Testers
Choosing the right testers is a combination of selecting people with the right profile criteria and that you want to test your early-stage app. There are different ways to choose the right criteria, and you ultimately need a good mix of different people. Plus, you’re limited to who you have access to.
I feel like it’s critical to have a good discussion with each beta user before they get access to your app. In our case, we had better outcomes, as defined by more feedback, usage, and engagement in the process, when I personally talked with users about the current software usage, problems, and thoughts about our space. Depending on your goals, you can probably accomplish this with a survey, but I highly recommend a casual conversation first. My pre-onboarding talks with users eventually evolved into a “The Mom Test” type conversation, where I got feedback about the problem and saw if they were a good fit for our beta. At this point, we’ve combined a survey to prioritize perfect fit users, followed by a zoom call before onboarding.
I should point out that we’ve only had 1 bad beta tester, which I had to remove from the test. By bad, I mean they weren’t there for the right reasons and weren’t respecting the process. I knew that the user wasn’t going to be a good fit right from the first call. An existing beta user referred him, but the person really had FOMO of his friend and wanted to see what we had out of curiosity, not because it solved a problem he had or had any interest in providing feedback during the beta. He did his onboarding zoom call in his car while driving with the windows down (I can’t make that up), and as soon as the first error happened on his Android phone, he was out.
If you need users to see if your app crashes when the load gets higher, you don’t need to worry about good vs. bad testers. However, if you want quality feedback from people that will improve future customers’ experience, you need quality beta testers to give appropriate feedback.
3. Collect Feedback & Make It A Team Effort
Towards the beginning of our testing, I was doing all the onboarding and feedback collection. We didn’t use any software or tools other than surveys we made in Microsoft Forms and eventually in Typeform. That felt like the most efficient process, but I quickly realized that even though acting as a filter to the rest of the team or taking feedback and directly pushing bug fixes to the codebase myself was the fastest way to solve issues, it wasn’t the best way to get the most out of beta testing. Besides fixing essential bugs, the biggest thing you get out of beta test is that once a user finds something they really want or really hates, as a team, you get to sit down and think about what you want your product to be. Those constraints are what make a product amazing, and you can’t do that when anyone is filtering feedback. The whole team needs to hear users’ pain points to really be motivated to get things right.
You also need to ignore certain types of feedback, or you’ll make the wrong product decisions. That’s easier to do with the whole team getting involved regularly. In our case, many of the beta users were people we know or close friends; we had to alternate getting feedback to make sure our friends weren’t biased when telling us they liked or disliked things.
Finally, most beta testers try something out when they first get and then lose interest quickly. It’s part of your job in running a successful beta test to keep users engaged. In our case, we didn’t release every feature on every platform at once, even though we had tested them internally. We did this both to make it easier to onboard and collect feedback, but it also helped bring people back to the product for another look.
I hope this helps you in your current or upcoming beta test! Feel free to reach out to me if you want feedback on your app or beta test.
