Considerations for selecting a Selenium Grid solution in a large enterprise

Test automation is integral to achieving high quality and fast feedback cycles in an organization. It is also one of the main pillars for moving your organization to a CI/CD/DevOps approach.

Selenium is the de-facto standard for automated web testing. Running Selenium tests on different browsers and in parallel enables an organization to achieve high test coverage and fast feedback cycles.

Selenium Grid is the usual approach for a Selenium cross browser testing infrastructure. Selenium Grid allows to run tests on multiple browser / OS combinations in parallel and can help reduce test suite execution times significantly.   

This paper looks at the three main options for a Selenium Grid solution.

  • On-premise homegrown
  • SaaS / Cloud
  • On-premise vendor managed

 

This white paper is intended to give guidance for determining a suitable Selenium Grid solution for your enterprise organization. We will look at the currently available Selenium Grid solutions in the market and compare the pros and cons for each of them.

1.  Background

According to the Gartner Magic Quadrant for software test automation (Gartner, December 2015), Selenium will become the de-facto standard by 2020 for test automation. From our experience,  Selenium already is the standard for automating web and mobile applications. Selenium’s JSON wire protocol is becoming a W3C standard and browser manufacturers like Google, Mozilla and Microsoft are maintaining the respective drivers themselves.

One of the main advantages of Selenium is the ability to run the same test script against multiple Browser / Operating System combinations.

There are two main components to a Selenium setup.

  1. Client side

This is where the test framework resides, where tests are authored and where tests are run from (i.e. developer workstation, continuous integration system).

  1. Browser execution infrastructure (server side)

The browser execution infrastructure (Selenium Grid) receives the Selenium commands and translates them into browser specific actions.

 This white paper focuses on the browser execution infrastructure which is typically resembled by a Selenium Grid.

 

2.  How a traditional Selenium Grid works

Selenium Grid  has 3 main purposes:

  1. Load balancing and routing of tests coming from the client side (i.e. CI system, developer workstation)
  2. Resource management of the browsers
  3. Reliability and browser crash recovery

 

3.  Selenium Grid solutions  

There are 3 main approaches to using Selenium Grid

  • On-premise, homegrown and self-maintained (homegrown*)
  • Cloud / SaaS (cloud*)
  • On-premise vendor managed (vendor managed*)

* term used for this white paper

 

3.1. On-premise, homegrown and self-maintained

This option is the starting point for many organizations. Selenium Grid is integrated into the open source distribution of Selenium. It is fairly simple to get a first demo up and running.

After the initial demo and evaluation, many organizations continue to use their engineering staff to build up the enterprise Selenium Grid environment. Building and maintaining an enterprise grade Selenium Grid requires much more effort than a proof of concept.

Deep Selenium know-how, infrastructure, virtualization and system admin skills are required for developing a secure, scalable, consistently up-to-date and cost efficient Selenium Grid infrastructure. This know-how is often outside a company’s core competencies and internal resources could be more efficiently used for core IT purposes. 

One of the major obstacles with a homegrown solution is maintaining and scaling the infrastructure to growing needs. Such an effort is usually complex and resource intense.

Another major challenge is the maintenance of the various OS / browser combinations. With the frequent releases of new browser versions and the further development of Selenium and drivers, a team of engineers is necessary to keep the Selenium Grid up and running. Even a small mismatch between browser / driver / Selenium can lead to unpredictable functionality of the browser – leading to false positive or negative tests. 

  Setting up an in-house Selenium Grid on your own is complex and usually lies outside of the core competencies of most company’s IT departments

 

3.2. Cloud / SaaS

There are several vendors offering Selenium Grid as a SaaS solution. When using such a cloud service, the provider takes care of all the time consuming  development and maintenance of the Selenium Grid infrastructure. Cloud providers boast a large range of browser / OS / combinations and in some cases support mobile testing.

Depending on the industry that you’re in, cloud providers can be a great fit. They offer various packages from entry level pricing to enterprise packages.

However, the larger the organization grows, the more difficult the use of a cloud provider becomes – first and foremost because of security reasons.

The use of a cloud service requires external access to your enterprise’s infrastructure. You need to access the cloud provider’s infrastructure and their  browsers running in the cloud need to access your application under test.

Enterprises in the banking, insurance and medical field often have strong limitations in giving external access to their systems – making it impossible in many cases to use cloud providers.

Also with cloud based solutions, operations and maintenance lie outside of the organization, resulting in a lack of control over how infrastructure is built, security, performance aspects, resource sharing, monitoring options, etc.

Performance:

In many cases, cloud providers have a central data center. Depending on the location of the cloud provider’s infrastructure and the distance to the customer’s environment (CI, application under test), network latency can have an impact on the performance of tests. Each Selenium command requires a full roundtrip from the client to the server and back. If the customer infrastructure and the servers of the cloud provider are geographically far apart the network latency can drastically increase the test execution time.

Scaling:

The price for cloud solutions often scale linear with the number of tests and the concurrency of tests. Enterprise subscriptions are also offered. However even with enterprise licenses, cloud based solutions often have limitations with regards to scalability and parallel / concurrent test runs.

These limitations should be taken into account and clarified with the vendors, as they can impact the overall goal for fast feedback cycles.

 

3.3.  On-premise / managed Selenium Grid

Up until recently, organizations could choose either the homegrown or the cloud path. However in 2016, Element34 Solutions released an on-premise enterprise Selenium Grid solution  (Selenium Box, https://seleniumbox.com). This solution is fully managed and thereby overcomes all the shortcomings of a homegrown self-maintained solution.

Let’s look at the various aspects of an on-premise Selenium Grid solution to understand why the combination on an on-premise and fully managed solution is in many cases exactly the solution that large and security sensitive enterprises require.

3.3.1. Security:

For organizations in the financial, medical and the security sensitive field a vendor managed Selenium Grid is the most secure option. No external access is required and no data leaves your network.

A vendor managed solution guarantees security and eliminates privacy risks that are associated with cloud based solutions.  

3.3.2. Integration

Due to running in the same network / infrastructure, the setup and integration of vendor managed solution is straight forward. As a result, connecting with the CI, reporting and monitoring systems and the application under test is simple. 

 

3.3.3. Maintenance

A solution like Selenium Box is fully managed and the vendor ensures that the OS / browser / driver / Selenium combinations are compatible with each other. Customers can be sure that the browsers which are released to the system are fully functioning and do not lead to unreliable tests. No further maintenance is required and the customer can focus their engineering staff on the core competencies of the business.

3.3.4. Performance

The proximity of the different systems in a CI pipeline to each other is key to fast and reliable tests. A vendor managed Selenium Grid solution eliminates all the latency issues of cloud solutions. In many cases the execution times of a test suite are 3-5 times faster with an on-premise solution compared to a Selenium Grid in the cloud.

3.3.5. Scalability

Cloud and vendor-managed solutions generally scale well (i.e. to add more concurrent test slots). A vendor managed solution scales by adding more CPU / RAM to the system. With cloud solutions the client needs to upgrade the license which increases the cost.

With vendor managed solution, the client has full control over scaling. With cloud solutions there may be limits in terms of license and/or accessibility of the cloud infrastructure.  

3.3.6. Support / SLA

Compared to a homegrown solution, a vendor managed solution comes with enterprise level support and SLAs for issue resolution. In many cases homegrown solutions are managed “on the side” by development teams and cannot ensure uptime, reliability and consistency.

3.3.7. Value add features

A vendor managed solution comes with many value add features like live view, video recording, monitoring & logging capabilities as well as error recovery. Homegrown solutions lack value added features since they use the out-of-the-box Selenium Grid which is provided on http://seleniumhq.org. Therefore, teams are usually stuck with a barebones Selenium Grid that does not allow for easy troubleshooting, debugging and reporting. The use of a homegrown Selenium Grid creates the risk of a lower adoption of automated testing than with a feature-rich managed Selenium Grid solution. 

 

3.4. Cost

Cost is an important factor in the selection process of a Selenium Grid solution. Let’s be clear: even though Selenium is open source and does not incur any license fees, it still requires an investment to use Selenium.

Building a homegrown Selenium Grid solution typically takes several person years. Also the ongoing maintenance of such a solution usually requires a handful of expert engineers / operations staff.

Cloud solutions do not require in-house staff but incur license fees. For a larger enterprise, fees usually run in the 6-digit space. As well, the integration of the cloud service into the enterprise infrastructure (if possible at all) can be a significant cost factor.

A vendor managed solution incurs a license fee and the hardware cost for running the system.

 

3.5. OTHER FACTORS

3.5.1. Productivity

Engineering resources are scarce these days and enterprises need to focus their staff activities on their core business. Building and maintaining a homegrown solution can be compared with rebuilding Microsoft Office. While it *could* be done, it adds no value to the enterprise especially since there are commercial solutions in the market.

With a managed solution (cloud or vendor managed), teams can fully focus on the task of writing meaningful tests rather than building and maintaining test infrastructure and / or having to working with hard to use and unstable solutions.

 

3.5.2. Business continuity

To implement and maintain a homegrown Selenium Grid a group of experts is required. In case of a change within the team (i.e. reorganization, attrition), it can be difficult to fill these position in a timely manner – potentially slowing down or stopping your test automation and CI/CD efforts all together.

 

CONCLUSION

A secure, reliable and scalable Selenium Grid cross browser infrastructure is crucial for a successful CI/CD and DevOps setup. Setting up and maintaining a mature enterprise grade Selenium Grid infrastructure requires a significant investment and is a complex undertaking.

While a homegrown solution can be a good starting point, for most enterprises they quickly become unmanageable. Homegrown solutions however require a big up-front investment in terms of engineering resources as well as deep Selenium Grid know-how. Maintenance for a homegrown Selenium Grid is time consuming and prone to error due to frequent new release of browsers and the Selenium ecosystem.

Homegrown solutions also lack any comfort functions for easy debugging and troubleshooting.  

When moving away from homegrown solutions, there are two choices:

  • Cloud / SaaS
  • On-premise vendor managed

While a cloud  solution may be a good fit for some organizations, experience shows that with larger enterprises (especially in the financial field), security becomes increasingly important. In many cases, large enterprises cannot use a cloud service. An on-premise solution is often the best option.

For on-premise solutions, organizations can choose between homegrown or vendor managed. Homegrown solutions require a big up-front investment in terms of human resources as well as deep Selenium Grid know-how. Maintenance for a self-built Selenium Grid is time consuming and prone to error.

Vendor managed solutions like the Selenium Box provide maintenance and worry free operation of the Selenium Grid and unlimited scalability options.

A vendor managed on-premise solution is therefore the best solution for all security sensitive organizations or for those who have high scaling needs.

 
 


4.  About Element34 Solutions GmbH

Element34 Solutions GmbH is a highly specialized software development & consulting firm focusing on test automation solutions with Selenium. The company founders are key contributors and thought leaders to open source projects such as Selenium, Selendroid and ios-driver. They have spent over 15 years working in high tech companies around the globe, bringing software development and test engineering to the next level.

The company’s flagship product Selenium Box is geared towards the financial sector and large enterprisesSelenium Box helps organizations to simplify cross browser test automation with Selenium and accelerate their journey to continuous integration and delivery.

More information about the Selenium Box can be found at https://seleniumbox.com

Element34 Solutions GmbH

Teichweg 8

8853 Lachen

Switzerland

 

5.  Selenium Box at a glance

 

What is Selenium Box? 

Selenium Box is a secure, on-premise, maintenance free Selenium Grid solution for running cross browser and mobile Selenium tests in-house.

Who is Selenium Box for? 

The Selenium Box is for organizations who require:

  • Security and data privacy
  • Scalability for running many tests in a short time
  • A maintenance-free Selenium Grid (selenium versions, browsers, drivers)
  • In-house engineering resources to focus on core competencies rather than building a selenium grid infrastructure.
  • Support for all major browsers
  • Full control over the Selenium Grid
  • Seamless integration into common CI systems

Selenium Box highlights

  • Built in Selenium Grid
  • Software or hardware solution
  • Firefox, Chrome, Internet Explorer and Edge
  • Automated updates of Grid hub, nodes and browsers
  • Self-healing nodes, crash recovery
  • Out of the box, customizable reporting: static, video, live view, raw
  • Full integration with common CI systems (Jenkins, Bamboo, TFS, TeamCity etc.)
  • Advanced system monitoring options
  • Use corporate operating system images 
  • Unmodified Selenium / Webdriver API 

Why Selenium Box?

  • On-premise, secure 
  • Automated updates and maintenance 
  • Unlimited scalability
  • No maintenance required
  • Fixed price and simple pricing model 
  • Full control 
  • Cross browser and mobile support 
  • Focus on writing tests instead of building infrastructure 
  • No vendor lock-in