Cloud computing is becoming a key solution to provide the underlying infrastructure to run big data analysis in a pay-per-use basis without up-front investment, especially beneficial for SMEs. These business opportunities for SMEs are strategic for Europe & Brazil.
Data-driven decision processes guide many sectors of our economy. In complex systems that do not lend themselves to intuitive models (e.g, natural sciences, social and engineered systems), data-driven modelling hypothesis generation and high performance computing scenario-based simulations have a key role to understanding system behavior and interactions.
Running effectively big data analysis systems is challenging.The execution time of a complex job is generally unknown in advance. Thus, determining the optimal number of nodes in a cluster, shared among multiple users performing heterogeneous tasks, is an important problem.The incoming workload might change, runtime management policies need react to workload fluctuations and application mutual runtime interferences.
EUBra-BIGSE has developed a framework for ensuring the Quality of Service (e.g., guaranteeing that jobs are completed before a given deadline) of data analytics services on top of cloud computing infrastructures, providing:
- Smart policies for vertical and horizontal elastic adjustment of resources allocated to meet deadlines and dynamic adjustment of workloads at the level of Virtual Machines, containers and physical servers.
- A rich and comprehensive API to enable Virtual Appliances to monitor performance indicators ranging from the low-level to the application-specific, spanning different resources.
- The support of advanced business models (price-based, dynamic re-scheduling of data analyses to exploit the best use of infrastructural resources, minimising costs) for on-premise or public infrastructures.
- The use of standards for the definition of the software appliances.
All the components are platform-agnostic and work on different Cloud Management Platforms (CMPs) avoiding changes to the application code thanks to a vendor-agnostic implementation of the monitoring platform and configuration and adaptation mechanisms. The EUBra-BIGSEA solution includes configuration and contextualisation services and a repository for deployment recipes.
Find out more about the EUBra-BIGSEA Quality of Service solutions: