E-commerce data insights from 2020 reveal huge increase in web scraping

No doubt 2020 had its fair share of challenges, says Gediminas Rickevicius, director of strategic partnerships at Oxylabs. With all of the economic and political events continuously taking place, it’s easy to lose track of what’s happening on the e-commerce landscape. Drawing upon a six-month period of data, generated between April and September last year, I’m going to share some key insights on the latest trends in the e-commerce space and they have been massive. Besides giving us clues on what’s currently taking place, this data can also give insights on where the industry will be headed in 2021 and beyond. Proxies and data collection tools are used by e-commerce businesses of all sizes. These tools help organisations facilitate data extraction for marketing purposes, refine their product catalogue, monitor brand status, detect counterfeits, and much more. The data in this report is varied and comes from several Oxylabs products. Each product is best suited for a specific set of uses, so understanding the use-cases gives more context to the data. There are many uses for web scraping, however the most relevant for e-commerce include:

• Consumer goods: food, electronics & fashion

E-commerce websites in the consumer goods spaces use web scraping to derive product and price-related intelligence. Along with monitoring product demand and popularity through publicly-available comments and reviews public data on the competition can also be collected in an ethical approach. Another rapidly emerging use for web scraping is the use of dynamic pricing. This practice analyses supply and demand-related factors to calculate the most optimal price for a given time period.

• Pricing intelligence platforms

Deriving pricing intelligence for retailers has become a growth industry. Web scraping uses for this sector include product and price monitoring, competitive intelligence (product, pricing and stock listings) and dynamic pricing strategies. In addition, scraping is used to monitor sites for Minimum Advertised Price (MAP) violations in order to prevent resellers from listing products at discounted rates on online marketplaces.

• Advertising agencies/marketing platforms

Millions of third-party retailers sell on e-commerce marketplaces worldwide. This has led to an entire industry of advertising agencies and platforms rising to meet the demand for increased intelligence on these platforms. Use cases include trend monitoring, competitive intelligence and keyword research.

• Price comparison websites

These websites use web scraping to aggregate and track product prices from leading websites in addition to monitoring product demand and popularity through top product listings and reviews.

• E-commerce marketplaces 

E-commerce marketplaces are a separate segment of e-commerce stakeholders that use web scraping to monitor the popularity of products and analyse competitors through pricing and available stock. Data extraction is also used to monitor supply and demand factors for use in dynamic pricing strategies.

E-commerce data insights from Apr-Sept 2020

Now let’s look over the activity across our datacentre proxies, residential proxies and our Real Time Crawler solution. Generally speaking, the six-month period between April and September last year saw a massive increase in web scraping activity overall:

Datacentre proxies

Datacentre proxies are the most commonly used proxies for accessing and extracting content from the web. They are typically used to monitor product demand through top product listings and reviews. At Oxylabs, datacentre proxy traffic per month has gone up 413%, suggesting two shifts in the market. Firstly, it appears that more public information is available for gathering from a larger number of data-rich websites. Secondly, it potentially indicates that the market players are discovering more use cases for the same data, which has been available all along.

Residential proxies

Residential proxies traffic per month at Oxylabs has increased by 50% in the last six months, signifying an increased need to scrape data on complex websites using specific geo-locations. Since residential proxies can be used for almost every use case mentioned above, this increase in activity signifies that more data is being collected and analysed across all e-commerce sectors. Since more data is being collected and used, this could also mean that the competition is increasing as e-commerce players gain new insights and refine their strategies.

Real-time crawler

The Real-Time Crawler is a ready-to-use solution for data collection. Designed for heavy duty data collection, it’s typically used by our customers to monitor pricing and product demand through top product listings and reviews, gather competitor intelligence, pricing violations and keyword research. Average requests per day for the Real-Time Crawler increased 61% over the last 6 months. This demonstrates the increasing popularity of this tool that empowers businesses to focus on insights rather than deal with challenging data gathering procedures.

Scraping is essential for 2021

Web scraping activity in 2020 increased substantially, and this trend is expected to continue. More businesses of all sizes are harnessing the power of web scraping to unlock data for insights that can help them beat the competition. More accessible than ever, web scraping is here and it’s the weapon of choice for businesses in 2021 and beyond. The author is Gediminas Rickevicius, director of strategic partnerships at Oxylabs. Comment on this article below or via Twitter: @VanillaPlus OR @jcvplus