**Platform Evolution: Automating Updates for a Smooth Business Operation**
**The Importance of a Reliable Platform**
For any large online business, a reliable platform is a foundational piece. eBay, one of the leading online marketplaces, understands the significance of having a stable and reliable platform. eBay’s platform comprises software frameworks and infrastructure in its backend, supporting more than 3,000 site applications that serve various domains, such as payment, selling, marketing tech, and more.
To ensure that these applications run efficiently, eBay requires continual updates to its platform, including framework and infrastructure upgrades. However, completing all application platform evolutions within a short timeframe and ensuring that all dependency systems can work normally after implementing platform changes can be a challenge.
**The Components of the Platform**
The platform is composed of two essential parts: the framework and the infrastructure. The framework provides all fundamental functionalities for the application code, including request context building, metric exposure, and common libraries. The infrastructure, on the other hand, hosts the application, and at eBay, it is the cloud system.
Platform evolution requires updates to both the framework and the infrastructure. These tasks can be prone to errors and require significant manual effort to maintain stability and availability. To address this challenge, an automated solution that covers the entire platform evolution process is necessary. This solution needs to be easily extendable and provide flexibility for different scenarios, such as framework upgrades or infrastructure migrations.
**Different Use Cases for Platform Evolution**
eBay continuously applies platform evolutions due to the size and complexity of its marketplace. There are various use cases for platform evolution, and a few typical ones are highlighted below:
**1. Framework Upgrade**
A framework upgrade involves updating the framework from one version to another. To apply the new version to all applications relying on that framework, an automation pipeline is necessary. This pipeline should be capable of updating the code, testing it, and releasing it without any issues. The framework upgrade pipeline typically includes three flows:
– Update framework version
– Build and test applications
– Release applications
**2. Framework Migration**
Framework migration focuses on moving applications from a legacy platform to a modern one. This migration process requires several flows:
– Collecting metadata
– Setting up new applications and corresponding pools
– Migrating traffic
– Monitoring after migration
– Decommissioning legacy platform pools
**3. Cloud Migration**
Cloud migration involves moving from one cloud infrastructure to another. For example, eBay is moving from a virtual machine-based openstratus to a container-based Kubernetes infrastructure. To successfully migrate, the following flows are required:
– Updating application development lifecycle
– Replacing library-based manifest with image-based manifest
– Replacing VM-based pool with pod-based pool
– Replacing hardware load balance with software load balance
**The Need for an Automation Solution**
To ensure the quality of platform level changes, an overall automation solution is necessary to achieve platform evolution. This solution must offer the following features:
– Support for different workflow types
– Ability to share common tasks
– Workflow and task management and orchestration
– Interaction with the ecosystem to process actions
Based on these requirements, an architecture that includes an orchestrator and a pluggable pipeline is proposed.
**Architecture for Platform Evolution Automation**
The proposed architecture consists of two main parts: the orchestrator and the pluggable pipeline.
**1. Orchestrator**
The orchestrator resolves corresponding flows with predefined tasks based on input. It can split each flow into smaller tasks that can run sequentially. The orchestrator triggers each task defined in the flow, logs task run status, handles exceptions, and ensures that all tasks are completed before moving on to the next flow. An API is provided to invoke the orchestrator.
**2. Pluggable Pipeline**
The pluggable pipeline is designed to be flexible and adaptable for new customized components. Each task in the pipeline can cherry-pick different components to obtain data from the ecosystem and complete specific jobs.
**Flow Execution and Task Context**
To accommodate different application stacks and their specific requirements, platform evolution requires specific flows with different task definitions. Each task runs independently and can be retried based on the context persisted by the previous run. The task context is persistent throughout the flow execution.
**Orchestrator Mechanism**
The orchestrator utilizes the Quartz framework as a task engine to execute tasks. Quartz is selected for its ability to distribute workload, persist job status, and support pausing and resuming jobs. Here is the working mechanism inside the orchestrator:
1. During startup, the flow manager preloads flow definitions and enables task listeners.
2. When a flow is requested, the flow manager stores the flow in the database.
3. The first task of the flow is triggered using the Quartz job engine.
4. Once a task is completed, the Quartz job listener persist its status and context and finds the next task for the stored flow.
5. If there is a next task, the flow manager submits it to the Quartz job engine.
6. The process continues until all tasks in the flow are completed.
By utilizing this architecture, eBay can automate platform evolutions and ensure the stability and availability of its applications.
**Conclusion**
In conclusion, a reliable platform is crucial for the success of any large online business. eBay understands the importance of platform evolution to keep their applications stable and reliable. By implementing an automation solution with an orchestrator and a pluggable pipeline, eBay can streamline the process of updating its framework and infrastructure. This automation solution offers flexibility, scalability, and efficiency, ultimately enhancing eBay’s online business operations.
GIPHY App Key not set. Please check settings