De-centralize Data replication and bookkeeping system

By Rishi Shah

Elevator Pitch

The Large Hadron Collider experiment requires multi-PB transfer loads per week across a complex topology of Grid-compliant computing centers. At CERN we are developing distributed, agent-based transferring system to achieve this goal. This talk mainly concerned about why we have used Go to build it.

Description

The CMS experiment at the LHC proton-proton collider developed PhEDEx (Physics Experiment Data Export) service as reliable and scalable data management system to meet experiment requirements in Run I/II. Designed in 2004, and written mainly in Perl. The aim of our new system is to extend basic PhEDEX functionality to address up-coming challenges in exa-byte HL-HLC era via implementation of modern Go programming language.

The motivation for the effort is many fold: - The current system is working well it lacks of support and expertise in perl programming language. We would like to leverage modern language such as Go to enhance concurrency model via native support in a language, and dependency free deployment; - Take advantage of built-in concurrency model of the Go language and explore the scalability boundaries. - The data volume expected in upcoming era will grow significantly to exa-byte level and we need to explore elasticity approach to handle variety of opportunistic resources; We can achieve this by simplicity, Speed and Portability of Go language. - Eliminate central blackboard system and necessity to rely on ORACLE backend via fully distributed nature of the agents, self-discovery and task delegation; - Go-language proves to be an excellent choice for developing web-based de-centralized asynchronous concurrent system due to its native support of those components in the core language.

This talk mainly focused on how Golang helped us to make it completely decentralized and asynchronous system. It will also provide you a deep insight of benchmarking some of the programming language oriented tests.