JavaScript is everywhere on the modern web, but it wasn't always this way.

Over time though, websites evolved from pure information to application through the use of JavaScript. However, JavaScript doesn't always just add interactivity to pages. It can also contain information.

Because JavaScript can contain information, search and answer engines often need to render it to fully understand a page. But that doesn't always happen successfully or immediately, and many AI bots don't render JavaScript at all. That means JavaScript-dependent content may be delayed, partially understood, or missed altogether. Rendering can also add seconds of processing time per page, creating crawl-efficiency issues for large sites.

Some organizations have attempted to solve for this issue by implementing server-side or pre-rendering solutions, but those can be incredibly resource-intensive and not always reliable.

That's why we built SpeedWorkers.

• What is SpeedWorkers?
• How does SpeedWorkers work?
• Why optimize for bots?
• Where does SpeedWorkers fit into the Botify suite?
• What does the SpeedWorkers interface look like?
• What results does SpeedWorkers produce for customers?

Rather watch than read? Check the animation on our product page.

What is SpeedWorkers?

SpeedWorkers is a bot & agent optimization and dynamic rendering solution that sends fully-rendered, cached versions of your pages to search and AI bots.

Dynamic rendering sends fully-rendered content to bots and agents while serving human visitors with normal, client-side rendered content. With SpeedWorkers, search bots can access important JavaScript-dependent content without relying on a later rendering pass, while AI bots that don't render JavaScript get access to content that was unavailable for them before.

Deliver pages to search and AI bots as quickly as possible

• Pages are delivered to search and AI bots faster, helping them discover more of your important content efficiently.
• JavaScript-dependent content is delivered fully-rendered, so bots can access the full page without needing to render it themselves.
• You can control which URLs are prioritized, reducing crawl wasted on duplicate, parameterized, or low-value pages.
• Eligible bot requests can be served from Botify instead of your origin infrastructure, helping reduce load and server costs.
• Configurable caching and quality controls help keep the content bots receive fresh and accurate.

Crawl budget is the number of pages on your site that a bot can and is willing to crawl in a given session. Every site's crawl budget is different, and is determined by factors such as how fast your pages load, how many errors your pages serve, the popularity of your pages, and more.

Dynamic rendering is a legitimate, Google-approved solution to these problems that does not constitute cloaking, as Martin reassured us during our dynamic rendering webinar some time ago.

Our promise with SpeedWorkers is to:

• Deliver pages to search and AI bots as quickly as possible
• Cache all your important content, not just a few top pages
• Operate at scale, accommodating even the largest sites
• Keep cached content in sync with live content
• Give customers consistently great performance so they can "set it and forget it"
• Provide quality control measures so you know if and when anything isn't rendering properly
• Be completely compliant with search engine terms & best practices
• Control which URLs bots receive and prioritize, so crawl is focused on the pages that matter most.
• Reduce unnecessary load on your origin infrastructure by serving eligible bot requests from Botify.

How does SpeedWorkers work?

SpeedWorkers operates at the CDN (content delivery network) level.

A content delivery network (CDN) is a network of distributed servers that can help minimize loading delays by reducing the physical distance between the user requesting the page and the server sending the page's contents.

This allows us to read, render, and cache your content on Botify's servers continually, with refresh settings you can customize based on how often different content needs to change.

This equips Botify with an up-to-date collection of all your site's content, fully-rendered, without you or your CDN needing to do anything on a regular basis.

SpeedWorkers can intercept search and AI bot requests and deliver high-speed, bot-optimized content from Botify's servers in response. Because those requests are served by Botify, they can also reduce traffic, load, and cost at your origin infrastructure.

Here is a quick look at how it works:

We went this route with SpeedWorkers for two main reasons. For one, CDNs are extremely close to search and answer engine requests, so we can react very quickly. Second, CDNs are very used to supporting third party applications.  

Implementation depends on your site's delivery setup, but the user experience stays the same: eligible bot and agent traffic is routed through SpeedWorkers while human visitors continue to receive the normal site experience.

This solution means human users are not affected at all. They will continue to receive the normal, client-side rendered content. Their requests for your content never even go to Botify. They're processed normally by your CDN.

Why optimize for bots and agents?

Since SpeedWorkers, like other dynamic rendering solutions, only pre-renders content for bots, you may be wondering why you wouldn't also take the burden of rendering JavaScript off your human visitors as well.

There are a few reasons for this:

1. Humans benefit from usability, personalization, and interactivity, which require JavaScript, so in most cases you'd want to keep client-side rendering your JavaScript.
2. Pre-rendering content for both human users and bots is often cost prohibitive.
3. Solutions like server-side rendering often require more resources to develop and maintain than many engineering teams have time for. Also, if you use a lot of third-party services on your pages (e.g. review feeds), they won't be rendered with server-side rendering.
4. Manually making performance improvements can improve load speeds for both humans and bots, but those projects are often lengthy, taking up to a year or more to see measurable improvements.

There is another reason this matters much more today: AI bots. Many AI bots don't render JavaScript, so this is not only a speed issue. A fully-rendered version can determine whether they see the full page at all. For search bots that can render JavaScript, SpeedWorkers helps get important content to them on the first request instead of relying on a later rendering pass.

We want to make our website experiences enjoyable for people, while also making sure the systems that crawl and understand the web can access the content they need. Search and AI engines rely on bots to do that, which is why the version of your site they receive matters.

Where does SpeedWorkers fit into the Botify suite?

The Botify platform is organized around AI Readiness, AI Visibility, and AI Activation. SpeedWorkers is part of AI Readiness, helping search and AI bots access and understand the content you want them to see.

But even though SpeedWorkers sits within the Botify Activation suite, it integrates and interacts with Botify Analytics and Botify Intelligence as well.

SpeedWorkers also works with Botify's crawl and log data to help you decide which pages to prioritize, how often cached content should refresh, and whether bots are receiving the content you expect.

What does the SpeedWorkers interface look like?

Because SpeedWorkers is a "set it and forget it" solution, the interface is focused on providing you with reporting tools to understand how well the solution is working, rather than giving you data about things you need to act on.

The overview report

The SpeedWorkers overview screen allows you to see two of the most important metrics you'll want to monitor to understand the impact dynamic rendering is having on your site:

1. How much faster your content is being delivered to bots
2. How long it takes before content is updated (re-cached)

Quality checks and controls

We also have a quality checks and controls dashboard. These reports monitor for two types of issues:

1. Issues that prevent pages from being delivered altogether (i.e. pass/fail)
2. Issues with individual scripts

Most pre-rendering solutions don't give users much, if any, insight into how well the dynamic rendering is working, but that's not something you have to worry about with SpeedWorkers. You'll get both high-level and granular insight into how everything is being rendered on a daily basis.

Additionally, these quality controls are completely customizable based on your needs. For example, if something isn't present and isn't working properly, we could fall back to doing a live rendering of the page instead.

Site cache monitoring

If everything looks good on your overview page, you can be reassured that SpeedWorkers is doing your job and you don't need to do any further digging, but if you did want to dig further, there are some additional reports like this Site Cache report, which gives you a breakdown of fails versus warnings, as well as the status codes of all cached content.

Delivery metrics

The delivery page provides stats and metrics about how often your dynamic rendered content is being requested by search engines.

Again, this is just another example of how much information you'll be able to get about how well your dynamic rendering is working. Our solution is robust, and we're confident that you'll get consistently great performance out of it, but we don't just want you to take our word for it. That's why we've provided such robust reporting around SpeedWorkers' performance and impact.

A full list of cached and requested pages

And finally, the SpeedWorkers interface gives you a complete list of every page that's been cached and requested.

You have the ability to sort these pages by any metric, such as delivery time or HTTP code. You can even see what each cached page looks like by clicking the camera icon next to the URL.

What results does SpeedWorkers produce for customers?

Our customers are already seeing great results from SpeedWorkers. Just how great? Well, it's not uncommon for our customers to see 10-30x improvements in speed!

We've seen even higher though -- for example, pages that were taking 10+ seconds to fully render, after SpeedWorkers, served in 300-400ms.

Overall, our customers can expect SpeedWorkers to:

• Improve crawl efficiency so more of your important pages can be discovered and refreshed.
• Make JavaScript-dependent content accessible to search and AI bots.
• Reduce the need for engineering-heavy server-side rendering projects built primarily for bots.
• Reduce origin-server load and cost by serving eligible bot traffic from Botify.
• Keep bot-facing content fresher and more accurate through configurable caching and quality controls.

If you have questions about SpeedWorkers or are interested in learning more about how it could benefit your performance, reach out and let us know! We'd love to hear from you.