JavaScript Crawling Now in Beta
Today marks the launch a new feature we’ve been very excited about for some time — we’re finally opening up the beta for our new JavaScript crawler!
The new crawler is deeply integrated throughout the Dragon Metrics platform in the same way as before, powering tools such as the Site Auditor, Site Explorer, URL X-Ray, and more. But now, the technology behind it runs off of Chromium (the open source foundational core of Chrome, Edge, and other browsers).
This means our crawler will see your site as close to site visitors and search engines like Google and Bing as possible.
Search engine crawling has changed dramatically in the past few years. It wasn’t long ago when there weren’t any search engines that would reliably render on-page JavaScript with their onsite crawlers. The simple reason for this was that rendering JavaScript reliably requires crawling with a headless browser, which uses up to 10x the resources of a simple HTTP request (e.g. crawling without rendering JavaScript). Doing this at the scale that search engines work with requires a truly herculean effort.
Recently, both Google and Bing have done just that — crawling the entire web with the latest version (often referred to as “evergreen”) Chromium. This meant that for the first time, the biggest search engines in the world now view your site’s content essentially like a normal visitor.
While caveats abound, and a fallbacks should be in place for other search engines and visitors that do not render JavaScript, this development has made it much more possible to more safely and reliably use content rendered by JavaScript, including front-end frameworks such as React, Vue, or Angular.
When Google was consistently missing content rendered by JavaScript, it made sense for SEO tool crawlers to report on these issues in the same way. But now is the time for SEO tools to evolve to this change as well — a good SEO crawler today must be able to render JavaScript in the same way Google does.
That’s exactly what today’s announcement is about. With our new crawler, you can be much more confident that content visible to Google and Bing will be visible in your Dragon Metrics site crawls as well.
Our new crawler has been in development and invite-only beta for some time. While we feel it’s been reasonably stable in our testing, we’re still leaving the beta label on it for now due to the unpredictable nature of the web as we continue to round of edge cases.
Our current crawler (based on HTTP requests) isn’t going away anytime soon. Just because Google and Bing can crawl JavaScript, doesn’t mean it’s 100% safe to render content using JS. The majority of other search engines still cannot access content rendered by JavaScript.
Therefore in many cases, it still makes sense to use the most conservative approach when crawling for issues and disabled JavaScript rendering. This will continue to be the default in Dragon Metrics for the foreseeable future. If our crawler cannot access your site properly, however, this may be a good case for enabling JavaScript crawling.
If you’re interested in trying out JavaScript crawling, just get in touch with our Customer Success team via live chat, and they will open up access for your account. Once this happens, you’ll still need to enable it for each campaign individually and kickstart a new crawl.
As with any beta product, your feedback along the way is vital to its development. Please share any questions, comments, or bugs you may encounter with our Customer Success team to help make our crawler the best it can possibly be.
Want more information on exactly how our JS crawler works? get in touch with our Customer Success team who can provide our work-in-progress documentation.