Frequent users of MySwar would have noticed a significant slowdown of MySwar as well as some erratic behaviour over the last few weeks. At the outset, let me apologize for these issues. In this blog post, I’ll talk about the root cause of the issues we’re facing and what we have been doing to address it. (Skip to the last paragraph for the TL;DR version).
Artificial Intelligence (AI) has taken the technology industry by storm in recent years. It is the next big thing, and all major companies investing heavily in it. AI promises to be a humanoid software that knows everything, has the answer to any question one may have and can perform a wide range of digital tasks. The all-knowing AI engines depend on old-fashioned websites like MySwar for knowledge. They scrape websites (among other data sources) and train models that have the intelligence gained from the extracted data encapsulated within them.
Prior to AI bots, search engine bots crawled websites too, but they didn’t extract entire websites. Unlike search engine bots, AI bots aggressively crawl websites and extract them wholesale. From the perspective of website owners, there are two major issues with AI crawling – a) the ethics of extracting all their data without permission, b) their disregard for the detrimental impact crawling has on websites and their users. The first issue is complex, but the second issue is black and white. AI bots are egregiously wrong with their approach of shutting down entire websites with aggressive scraping to satisfy their hunger for data.
This is what MySwar has been dealing with since August 28. We have been bombarded with web page requests with such velocity that our website has slowed down to a near standstill and users have found it very difficult to access MySwar. Given the seriousness of the problem, we decided to use Cloudflare, a Content Delivery Network (CDN), which among other things, helps website ward off aggressive scraping by AI bots and other bad actors.
Using Cloudflare brought relief almost instantly. We could see our server load reducing as Cloudflare started blocking AI bots and other agents. However, things were too good to be true. We started noticing some mobile web pages throwing errors, while those pages loaded without issues on desktop browsers. Advanced Search Results, Similar Songs Listings, certain artist pages, some film pages and a few other pages started throwing errors consistently. These errors disappeared when we turned off Cloudflare, but MySwar was immediately overwhelmed by AI bots when we did so.
This is where we find ourselves now. Between a rock – AI bots making MySwar inaccessible; and a hard place – an issue between Cloudflare and MySwar servers that causes errors in some pages on mobile devices. We have opted to accept the latter for now. As undesirable as it is, enabling Cloudflare at least makes MySwar accessible on desktop and a large portion of the website available on mobile devices. While we work with Cloudflare to resolve the issue between our servers, I ask MySwar users to:
- Bear with us as we work through the issue
- Use MySwar on desktop to the extent possible
- Load the desktop version of the page if it throws an error on the mobile browser
Leave a Reply