RISE ICT stories: HOPS, Software-as-a-Service from SICS’s New Datacenter

Hadoop Open Platform-as-a-Service (Hops) is a new distribution of Apache Hadoop that is based on a next-generation, scale-out architecture for HDFS and YARN metadata. Hops is provided as software-as-a-service for researchers and companies in Sweden from the newly started SICS ICE data center in Luleå.

One of the goals of Hops is to make Hadoop easier to use for researchers that may not be data engineers. To this end, we have developed a new user interface to Hops, called HopsWorks, that supports true multi-tenancy in Hadoop. That is, researchers and companies can securely share the same Hadoop cluster resources. This contrasts with existing models for multi-tenancy in Hadoop that limit organizations to running separate Hadoop clusters on virtualized or containerized platforms.

SICS model for multi-tenancy is based around projects. Users in HopsWorks can create projects, manage the quota for projects, control membership of projects, and securely share data between projects. They can make use of data analytics frameworks such as Apache Spark, Apache Flink, and MapReduce. HopsWorks also supports the popular Hadoop notebook Apache Zeppelin for interactive analytics, ElasticSearch for free-text search for files and directories in HDFS, as well as extended metadata.

Hops and HopsWorks are open-source projects and have been developed in collaboration between RISE SICS and KTH.

Download Hops here

Read more

More RISE ICT stories