Predictive Analytics for Identifying
Product Opportunities Through Search
Trends and Unstructured Data
Abel Hernández García
University of Liverpool, United Kingdom.
Dr. Çağatay Iris
Supervisor
MSc. Dissertation
Project
09/2023
Date
This study investigates the potential of internet search trends and unstructured data to discover consumer behaviour patterns and needs that often go unnoticed. Using information from Google Trends and search suggestions, various analytical methods are implemented: from time series forecasting with the Prophet algorithm, to text mining, semantic networks and data visualisation techniques. The study's findings show the effectiveness of these strategies in identifying emerging trends and consumer preferences. This dissertation thus demonstrates that data analysis tools and analytical methods, while not without limitations, offer considerable potential for uncovering insights into consumer behaviour.
The techniques presented here are flexible and could be applied in a variety of settings and for multiple purposes, from product development to market entry strategies. If used intelligently and in combination with other forms of data and analysis, they have the potential to become a useful tool for businesses.
Objective
The primary objective of this study is to develop a comprehensive approach to identify product opportunities by exploiting the power of online search trends and unstructured data analysis. In addition, this project also aims to create a scalable and adaptable model that can serve as a predictive tool for companies looking to expand their product offerings into new markets or launch successful products from one country into another.
To achieve this the project aims to:
-
Understand and evaluate the potential of tools such as Google Trends in predicting
market trends and identifying product launch opportunities across various sectors.
-
Develop and implement machine learning and data mining techniques to analyse unstructured data, with a focus on identifying trends and insights that can inform
product launch strategies.
-
Conduct a comparative analysis of various data sources and methodologies highlighted in the literature, including search query data, and topic modelling from unstructured text, to determine the most effective combination for predicting product opportunities.
-
Develop a model that integrates findings from the above objectives to provide a robust, data-driven framework for identifying product launch opportunities in various sectors.
-
Develop a model that can adapt to different market conditions and incorporate real- time changes in consumer search trends to predict successful product launch opportunities.
-
Identify key indicators in search trends and unstructured data that signal a market's readiness or receptiveness for a new product or a successful product from another country.
A future objective will be to create a user-friendly interface for this model that enables companies to easily input data, interpret results, and make informed decisions about product launches in new markets.
Full document:
Data Collection
Overview
Overcoming Limitations
To address the Google Trends restriction of weekly data for queries beyond 90 days, I applied a technique using the R package trendecon, which allowed me to retrieve daily data over extended periods, overcoming the standard Google Trends limitation.
Granular Daily Data
By disaggregating data to a finer resolution, I could conduct a more detailed analysis of search behaviour, uncovering nuanced insights into weekly seasonality and daily search patterns. This methodological was crucial for the objective to analyse consumer interests with greater precision.
Forecasting data retrieved from Google Trends on RStudio with the prophet algorithm and visualising the results with ggplot2.
Forecasting
Analysing Seasonality
Collection of unstructured data
Consumer Query Analysis
Utilising strategical search queries I gathered predictive insights from Google's autocomplete suggestions, presenting a visual representation of consumer intent and demand.
Keyword Visualization
Text mining facilitated the extraction of prevalent themes from search suggestions, illustrated through word clouds, revealing dominant consumer interests and market trends.
Deciphering Market Signals: Semantic Network Analysis
All Projects
Semantic networks provided an intuitive visual representation of how search terms are interconnected, allowing us to better understand the underlying relationships and patterns that exist between various search queries.
Expanding Market Understanding Through Semantic Clusters
By examining these clusters, we discern the thematic concentrations within consumer searches, which illuminate potential niches ripe for innovative product introductions
Semantic Network: top_related_queries for 'Laptop'. Highlighting gaming's cluster.
Semantic Network: rising_related_queries for 'Laptop'.
Highlighting gaming's cluster.