Why buying a commercial Selenium Grid solution is (typically) better than building your own

This is a summary of our whitepaper available at  https://seleniumbox.com/whitepaper/ 

Cost:

  • Building and maintaining your own enterprise grade Grid requires significant resources
  • Building: requires a team of 3-4 full-time senior engineers + supporting functions (project manager / scrum master etc.) for 9-18 months (based on what we have seen with customers)
    • Cost: 450K+ EUR
    • Time lost in recruiting staff and building the solution: min. 1 year 
  • Maintenance and further development: 2+ full-time employees ongoing
    • Cost: 200K+  EUR / year
  • For Mobile these numbers are significantly higher. Appium / Mobile is extremely complex to set up in an enterprise grade fashion especially if you want it to work in a combined Grid for browsers  and mobile. 

Business Continuity:

  • A homegrown Grid requires a highly specialized internal team (with backup) for support, maintenance and further development
  • In case of restructuring , attrition or if key people leave the team the entire test automation infrastructure is jeopardized
  • When buying a commercially supported solution like Selenium Box, there are clear SLAs in place and we also provide 24/7 support 

Skills / Talent:

  • Build and maintaining your own Grid requires a team that has in depth knowledge of Selenium (down to the protocol level) as well as advanced skills in server operations, proxies, firewalls etc. This is typically hard to find and unite in one team.

Productivity:

  • Building and maintaining test automation infrastructure is (most likely) not your core business.
  • With engineering resources being scarce, organizations should focus on tasks and topics that are their core business and lie within their core competencies. For everything else they should use commercial solutions (just like everybody buys Microsoft Office rather than building a text editor on their own). 

Functionality / Features:

  • The open source Selenium Grid is fundamentally not enterprise ready as there are many key features missing. As well the architecture does not allow for easily building these features in. The open source Grid was never meant to be used in enterprises.
  • Here is a high-level overview of the features Selenium Box  in comparison to an open source grid
FeatureOpen Source Selenium GridSelenium Box
On-premises / inside corporate networkYESYES
DESKTOP BROWSERS
ChromeYESYES
FirefoxYESYES
Firefox ESRYESYES
Internet ExlorerYES*YES
EdgeYES*YES
SafariYES*YES
* very high maintenance effort
MOBILE
iOSNOYES
AndroidNOYES
ENTERPRISE FEATURES
Video RecordingNOYES
Live View / Realtime debuggingNOYES
Enterprise Access ControlNOYES
Enterprise Grade Reliability and ScalabilityNOYES
Proof of concept support NOYES
High availability / fault tolerant setupNOYES
Kerberos / SPNEGO Authentication SupportNOYES
Active Directory IntegrationNOYES
Atlassian CROWD IntegrationNOYES
Enterprise proxy supportNOYES
End to End encryption / HTTPSNOYES
Custom SSL supportNOYES
Custom Windows ImagesNOYES
Rights / Permissions ManagementNOYES
Manual / exploratory testing supportNOYES
API accessNOYES
Project ManagementNOYES
Token AccessNOYES
24/7 Enterprise SupportNOYES
Dedicated Engineering ContactNOYES
Bespoke TrainingsNOYES
Professional ServicesNOYES
Customer Success ManagerNOYES
Reporting / MonitoringNOYES
Usage metricsNOYES
Maintenance-freeNOYES
Concurrency control / Quality of ServiceNOYES
Billback capabilities NOYES

Conclusion:

A secure, reliable and scalable Selenium Grid cross browser infrastructure is crucial for a successful continuous testing and DevOps setup. Setting up and maintaining a mature enterprise grade Selenium Grid infrastructure requires a significant investment and is a complex undertaking. While a homegrown solution can be a good starting point, for most enterprises they quickly become unmanageable. Homegrown solutions require a big up-front investment in terms of engineering resources as well as deep Selenium Grid know-how. Maintenance for a homegrown Selenium Grid is time consuming and prone to error due to frequent new release of browsers and the Selenium ecosystem.