Why buying a commercial Selenium Grid solution is (typically) better than building your own
This is a summary of our whitepaper available at https://seleniumbox.com/whitepaper/
Cost:
- Building and maintaining your own enterprise grade Grid requires significant resources
- Building: requires a team of 3-4 full-time senior engineers + supporting functions (project manager / scrum master etc.) for 9-18 months (based on what we have seen with customers)
- Cost: 450K+ EUR
- Time lost in recruiting staff and building the solution: min. 1 year
- Maintenance and further development: 2+ full-time employees ongoing
- Cost: 200K+ EUR / year
- For Mobile these numbers are significantly higher. Appium / Mobile is extremely complex to set up in an enterprise grade fashion especially if you want it to work in a combined Grid for browsers and mobile.
Business Continuity:
- A homegrown Grid requires a highly specialized internal team (with backup) for support, maintenance and further development
- In case of restructuring , attrition or if key people leave the team the entire test automation infrastructure is jeopardized
- When buying a commercially supported solution like Selenium Box, there are clear SLAs in place and we also provide 24/7 support
Skills / Talent:
- Build and maintaining your own Grid requires a team that has in depth knowledge of Selenium (down to the protocol level) as well as advanced skills in server operations, proxies, firewalls etc. This is typically hard to find and unite in one team.
Productivity:
- Building and maintaining test automation infrastructure is (most likely) not your core business.
- With engineering resources being scarce, organizations should focus on tasks and topics that are their core business and lie within their core competencies. For everything else they should use commercial solutions (just like everybody buys Microsoft Office rather than building a text editor on their own).
Functionality / Features:
- The open source Selenium Grid is fundamentally not enterprise ready as there are many key features missing. As well the architecture does not allow for easily building these features in. The open source Grid was never meant to be used in enterprises.
- Here is a high-level overview of the features Selenium Box in comparison to an open source grid
| Feature | Open Source Selenium Grid | Selenium Box |
|---|---|---|
| On-premises / inside corporate network | YES | YES |
| DESKTOP BROWSERS | ||
| Chrome | YES | YES |
| Firefox | YES | YES |
| Firefox ESR | YES | YES |
| Internet Exlorer | YES* | YES |
| Edge | YES* | YES |
| Safari | YES* | YES |
| * very high maintenance effort | ||
| MOBILE | ||
| iOS | NO | YES |
| Android | NO | YES |
| ENTERPRISE FEATURES | ||
| Video Recording | NO | YES |
| Live View / Realtime debugging | NO | YES |
| Enterprise Access Control | NO | YES |
| Enterprise Grade Reliability and Scalability | NO | YES |
| Proof of concept support | NO | YES |
| High availability / fault tolerant setup | NO | YES |
| Kerberos / SPNEGO Authentication Support | NO | YES |
| Active Directory Integration | NO | YES |
| Atlassian CROWD Integration | NO | YES |
| Enterprise proxy support | NO | YES |
| End to End encryption / HTTPS | NO | YES |
| Custom SSL support | NO | YES |
| Custom Windows Images | NO | YES |
| Rights / Permissions Management | NO | YES |
| Manual / exploratory testing support | NO | YES |
| API access | NO | YES |
| Project Management | NO | YES |
| Token Access | NO | YES |
| 24/7 Enterprise Support | NO | YES |
| Dedicated Engineering Contact | NO | YES |
| Bespoke Trainings | NO | YES |
| Professional Services | NO | YES |
| Customer Success Manager | NO | YES |
| Reporting / Monitoring | NO | YES |
| Usage metrics | NO | YES |
| Maintenance-free | NO | YES |
| Concurrency control / Quality of Service | NO | YES |
| Billback capabilities | NO | YES |
Conclusion:
A secure, reliable and scalable Selenium Grid cross browser infrastructure is crucial for a successful continuous testing and DevOps setup. Setting up and maintaining a mature enterprise grade Selenium Grid infrastructure requires a significant investment and is a complex undertaking. While a homegrown solution can be a good starting point, for most enterprises they quickly become unmanageable. Homegrown solutions require a big up-front investment in terms of engineering resources as well as deep Selenium Grid know-how. Maintenance for a homegrown Selenium Grid is time consuming and prone to error due to frequent new release of browsers and the Selenium ecosystem.