If you are looking for a new Web optimization vendor, or checking out testing technology for the first time, make sure you are asking all the right questions. Optimizing the entire user experience can be tricky, and you'll want to be sure the technology you are selecting can do everything you need. You'll want to make sure the technology you select can test every part of your site (front-end and back-end) and enable you to run more tests in less time.
If you are in charge of selecting optimization technology, you are also likely responsible for advancing your company's optimization culture and creating measurable wins for your marketers, product managers, and developers. Here are some of the questions our customers have asked us over the years; it's a helpful list to get you started.
1) Can it test all types of content? It's simply not enough to be able to test copy, fonts, layout, and images these days. Not only do you want to be able to test static HTML content, but also dynamic content and mobile content, among other things.
2) Can it test mobile apps? App stores have a defined approval process, and you have little control over the approval timeline once your app is submitted. Companies that test their mobile apps know they are right from the start -- saving a great deal of time and resources.
3) Can it test every part of the conversion funnel on your Web or mobile site - from home page and landing pages to product pages and checkout pages?
4) Can it optimize navigation? For example, how the user experience is sequenced, what each item is labeled (e.g., "My profile" vs. "Profile" vs. "Settings"), how inline text links look, and link style, among other things.
5) Can it test site functionality? For example:
a. On-site search: On-site search is the typically the second-most visited area on a website and our clients have found the following test ideas extremely useful. So you'll want to know if the optimization solution help you optimize on-site search functionality, such as:
i. search results display
ii. search metadata (such as ratings) display
iii. search result filtering options
iv. number of search results displayed
v. search results layout (grid versus list, for example)
vi. how product type impacts search display preference
vii. underlying search algorithms
viii. ancillary search features (such as spelling correction)
b. Checkout process: Can it test how many form fields (and which ones) and number of pages in the checkout process are ideal to maximize conversions on your site? Can it experiment with the checkout process for logged-in versus non-logged in visitors? Can it test plug-ins used during checkout, such as chat?
c. Pricing/Shipping Offers: Can it test optimal free shipping thresholds, for example?
d. New features: There are ways to both mitigate risk and learn valuable insights without negatively impacting your conversion when launching new features. The best practice is to test these new features to a subset of your audience and measure behavior upfront, before a launch to all site visitors/customers. This also offers the ability to quickly measure KPIs while also checking for misplaced code or a poor experience. The size of your test audience for the new feature depends on traffic volume. If your site receives a significant amount of traffic, we recommend rolling out the feature to 1-3 percent of your traffic. This can always be adjusted if you aren't getting the necessary traffic to reach statistical significance. Can the optimization solutions you are considering do all that?
e. Whole site releases: In other words, can it test a new site or new site sections versus the previous site as a control?
6) Can you use it to quantify the impact of your Web development efforts on your KPIs?
7) Does it offer advanced behavioral targeting?You'll want to use this solution to create relevant, personalized experiences for your audience. Can the optimization solution segment your audiences using targeting criteria, such as:
- User behavior, such as adding an item to a cart, bookmarking a favorite, or viewing a product category page
- Location
- History, such as recency, frequency, and monetary value of previous purchases and transactions
- Mobile device and platform
- Browser
- Referral source of visit, such as Google or PPC
- Time of day
- Language
- Externally defined criteria from third-party CRM or SFA databases
8) Can you run additional tests without added expense? Are there any limiting factors to the number of tests you can run?
9) Can it integrate with all major third-party Web analytics solutions? Are you able to view and segment the results of any multivariate test campaign in your Web analytics solution? Can you analyze multivariate tests in the context of virtually any visitor segment, including PPC campaigns and keywords, email promotions, affiliate referrals, multichannel user behavior, and more?
10) Can it be deployed in the cloud or on-premise behind your corporate firewall? Flexible deployment options are important: an on-premise solution can help you meet specific security and privacy requirements, while cloud-based deployment gives you the ability to test and target without installing any additional hardware or software.
11) Can it work without introducing flicker effect or page-loading latency due to JavaScript tags? While optimization platforms provide a multitude of benefits, if they're not carefully designed, they can substantially lengthen page load times by increasing network latency. To minimize this, organizations should review different optimization platforms' architecture and gauge their impact on latency. In this respect, tag-free platforms have the edge over tag-based platforms because they introduce very little additional latency, typically just 4-12 milliseconds.
12) Can you use it to speed up your site? Faster websites not only improve the user experience, but also boost order values and site stickiness. Can your testing solution accelerate the delivery and rendering of both Web and mobile content through a variety of on-the-fly speed optimization techniques?