Comparing Apples and Grapples

Real browsers vs. virtual browsers. It’s a hot topic among Web performance testing providers and customers.

Which is the better choice? Well, it depends. They’re not intended for use under the same circumstances and requirements, so you aren’t comparing apples to apples, instead, you are comparing apples to grapples. Sure, they both provide the same high-level functionality of putting load on a Web system, but they do so in differing ways that makes one of them more costly, but also more valuable, in most testing scenarios.

Let’s dive into what each option covers, as well as their differences and functional offerings. This should give you a better understanding of each and help you make the right decision when it’s time to choose.

Load Testing 101: Virtual Browsers

In terms of load testing and Web performance, virtual browsers are exactly what they sound like: simulated environments where HTTP traffic may be requested or initiated to a server resource through GET and POST commands, and, in turn, are received back at the virtual browser. In addition, since virtual browsers do not interact with the Document Object Model (DOM) of a Website or any other elements that are presented, they are more capable than real browsers at load testing embedded technologies such as Flash, Silverlight, etc. Why? They are simply sending requests to the server directly, instead of interacting with an object that in turn would send the same request (such as a button).

Virtual browsers are also far less resource-intensive on the host server than real browsers, since they aren’t loading a full graphical user interface and rendering engine, among other things. This means virtual browsers can produce significantly higher traffic loads, into the hundreds of thousands of simultaneous users. Knowing this, virtual browsers are great for load testing requirements where you simply need to send as much traffic as possible to a server resource without regard for how the client-side transaction will interpret the responses. Virtual browsers are also ideal if you need to test an application program interface (API) whose sole purpose is to receive and reply to HTTP requests.

But there are drawbacks to using virtual browsers and some are true deal-breakers. Limitations to using a virtual browser include the inability to directly interact with a Website’s DOM and other source files; the limitation of working with asynchronous data streams; and a more time-intensive and difficult troubleshooting process when failures occur.

Load Testing 102: Real Browsers

Just as we discovered with virtual browsers, real browsers are exactly what you would imagine: a real instance of a browser that is launched and interacted with during a load testing user’s transaction(s). With great accuracy, real browsers imitate the experience a real user would have when visiting your Web services, short of testing across every possible browser in use today.

During a real browser load test, an actual browser (in many cases, Firefox or another release version of a leading browser) is opened on the host environment. A URL is typed into the address bar, and interaction with the subsequent Website occurs exactly how a user would inherently navigate through the site by clicking on buttons, selecting options, filling out form fields or submitting data.

Additionally, real browsers let you take screenshots and video captures of transactions as they occur, saving significant and valuable responses when failures happen and simplifying the troubleshooting process immensely. Because error messages are not always completely understandable, having a screenshot and video of the transaction and failure from a user’s perspective is incredibly helpful in identifying the root cause and other aspects of the failure.

Another benefit of real browsers: the ability to interact with technologies, such as Asynchronous JavaScript and XML (AJAX), which are widely used in Websites today. These technologies allow the user to provide input to a field on the page, and, in turn, an asynchronous data stream sends the input to the server and transmits back the response without the necessity to click on a “submit” button or other element. One example of this functionality occurs when choosing demographic information from a form field, such as a state dropdown list, which in turn provides another dropdown with only the cities or counties belonging to that state. While virtual browsers cannot interact with these technologies in any capacity, real browsers do so very well especially when a skilled and experienced engineer is performing the Web performance scenario scripting.

As browsers become more capable and various tasks are offloaded from the server to the browser for client-side processing, many Websites rely heavily on browsers to display their Web services quickly and correctly. Because today’s browsers can handle technological advances, real ones are often preferred in load testing environments.

Yes, there are drawbacks to real browsers, but they are minimal. The greatest drawback is the resource requirement on the host server for running real browsers. As stated previously, real browsers require far more resources than virtual browsers, and therefore are limited by the hardware and capabilities of the host environment. This makes real browsers more costly to run than virtual browsers, which is why many testing providers either avoid or simply don’t offer them. In fact, some providers offer “real browser load tests” with thousands of concurrent users, however only a few hundred may be real browsers, with the remainder being virtual.

Making the Right Decision

Your choice will come down to the purposes, objectives and requirements of your load testing engagement. Real browser testing is becoming the industry standard, as it allows for the truest measurements and experiences that users will encounter.

Neustar offers both virtual and real browser load tests in their fullest capacity, the latter capable of testing close to 10,000 concurrent users.