Mozcon 2016: Cara Harshman

DISCLAIMER: This post is written as a live blog from Mozcon. There may be typos and grammar to make my high school English teachers weep. Please excuse those … it’s a fast-paced conference with back-to-back sessions and no time for proofing or even proper writing.

The always-entertaining Cara Harshman is up next with the session Uplevel Your A/B Testing Skills. Regular readers will remember our coverage of her sessions at Mozcon 2015 and CTA Conference. Happy to see her again. So let's get to it. She takes the stage and begins with chatting about Pokemon (a game I know she doesn't play).  She references her New York Experience with hundreds of people playing at one gym.  This is one of the reasons she changes her session from "uplevel" to "evolve". She notes that 70% of marketers say conversion optimization is more of a priority this year than last.  This makes sense - that's where the ROI fro your other marketing efforts comes from. She starts with a critical point ... Any statistically significant outcomes of a test, good or bad, is good.  As long as you get data the test was worth it. In a test for donating to Hillary Clinton's campaign.  To save a card was a high friction process (re-entering info, etc.).  The change automatically created an account for a person when they donated and went straight to allowing the user to save their details with info moved from their initial donation across to the save data form.  Over a 200% lift just from reducing friction. Great A/B tests come from data and analytics, not whiteboard sessions.  Her top 5 reports are:

1. Funnel Report - see where people fall off in funnel
2. Heat map on home and high intent pages
3. Form error submission reports
4. Exit and bounce rate reports
5. High traffic but low converting landing pages

Voice Of The Customer You need to be empathetic.  You know your product but they don't.  Talk to your users.  Check with

1. Customer success team
2. Record user testing sessions
3. Coffee shops.  Just ask.
4. On page surveys (like Hotjar).
5. Email surveys

Voice Of The Human (reading)

1. Lift Mdel by Wider Funner
2. Conversion Equation from Oli Gardener
3. Psychology
4. Design Principles
5. Neuromarketing

Getting Started Team Testing Hackathon - pull in people to site in a group and brainstorm ideas by:

• Give people access to data
• Invite cross-funnel teams
• Give prizes
• Follow up

There's a toolkit for this at https://www.optimizely.com/testing-toolkit/. Next she's talking about moving from Execution to automation ... You need to take a long-term view which needs:

• And emotion-free system to decide what to do.  This means data-based.  Make a chart on ease vs impact of possible changes.  Focus on the easy and high impact ideas first.
• You could also make a score card.  List all your goals and assign points to each goal.  Then list all your possible ideas and the ones with the most points comes first.  This creates a clear direction on what gets tested.  Hotwire uses this.
• The next stage is standardization.  You need to create broadly agreed-upon sets of priorities metrics, criteria, etc.
• Standardize a testing timeframe.  You need to test things that really make a difference.
• Standardize your audience.  Determine traffic segments to test on or to look at specifically for personalization.
• The next stage is moving from teams to advocates.  That is - moving people from doing the work you need to cheerleading it.

5 skills you need on your conversion optimization team:

1. Project manager
2. Developer
3. Designer
4. Analyst
5. Fearlessness

You need to be willing to ask "why" and challenge assumptions and test new things. We also need communication channels.  You can:

1. Opt-in email newsletter
2. Weekly meeting for core group
3. Internal wiki or workflow tool (historical records)
4. Lunch n learns
5. All-hands meetings

The optimal outcome is:

• Learn rate of 100% (won or lose)
• Business growth, higher revenue
• Delighted customers
• Stronger decisions and culture

And now a question: Q - How do you get approval when there's friction? A - Her advice - either use fear of a competitor or just do it and don't ask and show results. Q - What's in your toolkit? A - She like Optimizey (she used to work there).  Heatmaping tool.  A user feedback tool like Hotjar.  A tool like Basecamp (we use Teamwork) - somewhere to stash data.