Create a good data research plan! Step by step guide for online businesses.
--
We went through on many theoretic things so far. What is Conversion Rate Optimization? How to collect data? How can you use the data? Why do you need AB-testing and how to do that right? What is funnel analysis? How to measure retention? How to do website heatmaps? And many-many more things.
It’s time to get it DONE. Go ahead, dig into your data, research everything and understand your users!
BUT! Before you touch any of the tools, you have one extra assignment. Create your data research plan! It’s really-really important.
You are an explorer and you go into the big wild data-jungle. You don’t know, what you will find there, you don’t have a map, you don’t have the GPS and you are the first one, who set foot into the jungle. At least you have the gears (but of course, you can’t be sure, that all your tools work properly), and some gossips by fellow colleagues about the treasure you can find in this data jungle.
The jungle is dark and infinite, so not to get lost, you will need a strategy. You need a Data Research Plan!
I am going to share with you my general data research strategy. It’s not necessarily the best and only way, but I fine-tuned it in the last few years and it works for me very well. You will see that it iterates between two major components: getting a general overview and go really deep into one specific topic. If I stay with the jungle metaphor, first I am trying to sketch up a blurry map about the major areas, then I focusing on one specific area and discover every corner of it.
But let’s break this down step by step.
Step 0: What’s your goal with your research plan?
Remember, you can do 3 basic things with your data: reporting, optimizing, predicting.
I assume, that you are at the phase, where you’ve already set up your main KPIs (aka. Head Metrics). If you have not done it yet, just check this article and do that first.
I also assume you are not yet at the point where you want to do machine learning and predictive analytics stuff.
As most of the startups, e-commerce companies or any other online businesses do, first you want to understand your current visitors’, current users’ and/or current customers’ behaviour. Why they do, what they do and how can you help them? You can call this CRO (conversion-rate-optimization) or retention-optimization, because the final output of this will be a higher conversion/retention rate, hopefully. But let’s try to focus more on the learning — instead of pushing your numbers. This mindset will pay-off on the long-term anyways.
Step 1: Discover the data infrastructure, you will work with!
Most probably someone at your company has set up some tools (at least Google Analytics) already. If not, you are lucky to have a clean start, and you are unlucky not to have any data from the past.
Any cases, try to answer these few questions:
- What tools do you have?
- What kind of data do you have in the tools?
- Do they show accurate data? (Garbage in garbage out.)
- Do you need any other tools?
- And do you have the resources (money, development time) to implement them?
This is also the right time to do some exploratory analysis. While you are experimenting with the tools, you can discover some significant trends, important segments, etc.
Step 2: Meet with the colleagues!
In the first weeks I usually set up 1-on-1s with many people, who could be involved anyhow in the upcoming data driven projects. Marketers, product owners, web developers, team leads, etc. They will have useful insights — as they came up with different ideas, products, campaigns, messaging, etc. based on something. Hopefully they have also interacted with users/customers already. Trust their insights, but also try to separate their gut-feelings from data-backed knowledge. Always double-check everything in your analytics tools quickly.
Step 3: Meet with the users!
At this early part of your researches, you should meet with 2 types of users:
- current active users, who are using your product on a regular basis.
- potential users, who are not using your product yet and maybe they haven’t even heard about that. They should be really similar to your current active users (eg. have similar challenges in their life as your current users.)
Map out these four questions:
- What is that current users love in your product?
- What are their biggest pain-points?
- What are the concerns of your potential users?
- Where are they getting stuck?
On this level, you don’t need to be an expert in usability testing or user interviews. Just follow some basic rules (open ended questions, think-aloud protocol, etc.) not to bias your tests/interviews and I promise that you will discover many things, you’d never think about by yourself. Later on, it’s better if you are doing this on a more professional level — maybe with the help of a UX-researcher. But now let’s just try to get some basic understanding.
Step 4: Set up a hypothetic Funnel
You have understood the strategy and the product of your company and you also met with a few users. Nice!
Let’s setup a hypothetic funnel!
Sometimes it’s not too difficult. If you happened to be an e-commerce business, there’s a 99% chance, that your funnel will look like everyone else’s:
- visitors see your ads
- land on your website
- go through on product lists
- visit a product page
- place one or more products into the basket
- check out
- pay
And sometimes it can be tricky. You might have a more complex product, a more complex goal (eg. retention) or a more complex user journey. No worries! Try to set up your hypothetic funnel — just to have some baseline in your hands. You can (and you will) modify it later, when you have made a few deeper researches and when you have more facts in your hand.
Step 5: Select a well-defined, narrow, but significant segment!
Let’s open your analytics tools again and spend some time discovering the different segments. You can segment the visitors by infinite dimensions, but as a start you need to find only one segment that’s homogeneous enough and not too small. There are some dimensions, that you can and should always use. Break down your users:
- by where they are at your funnel
- by device type
If you want to use further dimensions… well, bad news: there are no further best practices.
The decision, which sub-segment you pick will need your business intelligence. But here are few ideas at least:
- by country
- by activity
- by marketing channel
- by demographics (age/gender)
- etc.
After all a good segment could be for instance: “Mobile users, before registrations in Italy” or “Tablet users, activated, under age 30”. But then again, it should be a significant amount and/or percent of your users.
It’s also a good tactic here to select 2 segments, that are different in only one dimension and compare those to each other. For example it’s reasonable to compare the desktop and the mobile traffic.
You can also find a very basic segmenting example in the Practical Data Dictionary.
Step 6: Select the most important topics to research!
At this point you should have several ideas in your head about what to research. If you don’t know, you did something wrong and you have to go back to one of your previous steps (meet with users, 1-on-1-s, explorative analysis, selecting segments, etc).
Try to write a list of 10–12 topics. You should also consider the requests of your colleagues. As a researcher and analyst your job is supporting their job. But also try to rank by importance. Important are the topics, where you expect:
- the greatest learnings (long-term goal)
- the greatest improvements (short-term goal)
- or quick and easy fixes (aka. low-hanging-fruits)
One short example:
If you are doing analysis around your website’s marketing power, it will probably be an important topic to understand the role of the different landing pages (the “entrances” to your website). Obviously different landings with different traffic-sizes drive different user experience, different conversion, different bounce rate, etc. This research will have a great short-term effect, because if you can see that one of the landings has a great conversion rate, but not too much traffic, you can experiment with bringing a little bit more visitors there. But it will have good long-term effect as well, if you can see a landing with big traffic, but not a very high conversion rate. In this case you know that this is a site for further researches (means understand why conversion is low, then fix it). And if you are “lucky”, you can find low-hanging-fruits like pages with big traffic, but 0% conversion, because of a buggy CTA button. Again: some of your investigations will be more general, some of them will be focused on one specific thing.
Step 7: Create your time-plan and backlog for the research plan!
Okay, now you have a very complex idea about what you will research and analyze. Let’s clean it up and fill it into a time plan. I usually suggest to do weekly or bi-weekly sprints. Each sprint will target one topic from Step 6 (that’s why it was so important to prioritize and make a list). In my opinion the first few sprints (3–6) should focus on one (or top two) particular segment from Step 4. The idea behind that is if you spend a few months understanding one segment in details, then later you will understand every other segments much easier.
Obviously your time plan can contain parallel tasks like doing a research sprint and running an AB-test at the same time. If you are a smaller startup, your research sprints could be shorter and if you are a bigger one, they could be monthly as well.
Step 8: Do your first research sprint!
As I mentioned, you will have many tools and methods for research. You need a framework to organize them. This is an important topic and I’ll write a more detailed article about that (UPDATE: it’s done: here). For now, here’s a short version though.
I usually follow this process and it works for me really well.
- Qualitative research is important. I wrote in step 2 about it. But here I am going to underline it again: even if you are a data analyst, you can get a bunch of great inputs from usability tests and user interviews. You can identify problems, you’d never think about by yourself and it will be much easier to find out, where to start your quantitative research.
- Quantitative research is basically data analysis. Many tools, big amount of data. If qualitative research identifies the problem, quantitative research validates it. Eg. you can see on the usability test, that the 3 out of 5 users are confused by a text of a button. Then you go into your data and check the turn-back rate after clicking the button (or the drop-off rate before clicking the button, or anything, that can validate the issue.) If it’s high, you have validated the issue.
- AB-testing: you have identified a problem and validated it. It’s time to fix it. Creative guys will sit together and find out different solutions. (In our example, different new wordings for the button.) There will be data driven opinions and there will be gut feeling opinions. The good thing, that you can test and compare them with AB-testing very easily.
- Implement: finally when you have the clear winner, you can implement the change and enjoy the great effect of it.
Not so difficult, right?
All you need to do is just do these sprints again and again.
And of course don’t forget to document and communicate the findings of each sprints!
Step 9: Remember, this is an iterative process
Once you have your plan, make sure, that you review and fine-tune it time-to-time! Let’s say monthly.
Remember, how iterative is the funnel (step4) or the segment (step5) part!
Also don’t forget that with every sprint you will learn something new, that you should build-in into future researches.
Sometimes you discover the big picture, sometimes you zoom on the details, but in every case, the learnings will correlate with each other. Step 8 looks like a circle, but that’s a bad point of view. If you look at it from another perspective, it’s more like a spiral. Even if you feel, that you go round and round, with every new sprint, with every new learning you are one level up!
Conclusion
Wow, that was a long blog post! And you’ve read it! Nice job!
Here’s one extra thought for the end. There are no written in stone rules, when it comes to creating a data research plan! Get this as an inspiration and feel free to develop your own! And don’t forget to share it in the comments!
Thanks for reading! Now just go into the data jungle and explore it like no-one else before!
Oh, actually, before that: if you enjoyed the article, please just let me know by clicking the 💚 below. It also helps other people see the story!
Tomi Mester
my blog: data36.com
my Twitter: @data36_com
my Newsletter: Data36 Newsletter
my new free tutorial series: SQL for Data Analysis and Python for Data Science!
