A Guide to Apache Mesos – Apache Mesos指南

最后修改: 2019年 7月 5日

中文/混合/英文(键盘快捷键:t)

1. Overview

1.概述

We usually deploy various applications on the same cluster of machines. For example, it’s common nowadays to have a distributed processing engine like Apache Spark or Apache Flink with distributed databases like Apache Cassandra in the same cluster.

我们通常在同一个机器集群上部署各种应用程序。例如,现在常见的是在同一个集群中使用分布式处理引擎,如Apache SparkApache Flink与分布式数据库Apache Cassandra

Apache Mesos is a platform that allows effective resource sharing between such applications.

Apache Mesos是一个平台,可以在此类应用程序之间进行有效的资源共享。

In this article, we’ll first discuss a few problems of resource allocation within applications deployed on the same cluster. Later, we’ll see how Apache Mesos provides better resource utilization between applications.

在这篇文章中,我们将首先讨论部署在同一集群上的应用程序内部资源分配的一些问题。之后,我们将看到Apache Mesos如何在应用程序之间提供更好的资源利用。

2. Sharing the Cluster

2.共享群集

Many applications need to share a cluster. By and large, there are two common approaches:

许多应用程序需要共享一个集群。总的来说,有两种常见的方法。

  • Partition the cluster statically and run an application on each partition
  • Allocate a set of machines to an application

Although these approaches allow applications to run independently of each other, it doesn’t achieve high resource utilization.

虽然这些方法允许应用程序相互独立运行,但它并不能实现高资源利用率。

For instance, consider an application that runs only for a short period followed by a period of inactivity. Now, since we have allocated static machines or partitions to this application, we have unutilized resources during the inactive period.

例如,考虑一个应用程序,只运行了很短的时间,然后是一个不活动期。现在,由于我们已经为该应用程序分配了静态机器或分区,我们在不活动期间有未利用的资源。

We can optimize resource utilization by reallocating free resources during the inactive period to other applications.

我们可以通过在不活动期间将空闲资源重新分配给其他应用程序来优化资源利用。

Apache Mesos helps with dynamic resource allocation between applications.

Apache Mesos有助于在应用程序之间进行动态资源分配。

3. Apache Mesos

3. apache mesos

With both cluster sharing approaches we discussed above, applications are only aware of the resources of a particular partition or machine they are running. However, Apache Mesos provides an abstract view of all the resources in the cluster to applications.

在我们上面讨论的两种集群共享方法中,应用程序只知道它们所运行的特定分区或机器的资源。然而,Apache Mesos为应用程序提供了一个集群中所有资源的抽象视图。

As we’ll see shortly, Mesos acts as an interface between machines and applications. It provides applications with the available resources on all machines in the cluster. It frequently updates this information to include resources that are freed up by applications that have reached completion status. This allows applications to make the best decision about which task to execute on which machine.

正如我们很快会看到的,Mesos作为机器和应用程序之间的接口。它为应用程序提供了集群中所有机器上的可用资源。它经常更新这一信息,以包括被已经达到完成状态的应用程序释放的资源。这使得应用程序能够对在哪台机器上执行哪项任务做出最佳决定。

In order to understand how Mesos works, let’s have a look at its architecture:

为了了解Mesos的工作原理,我们来看看它的架构

Mesos arch

This image is part of the official documentation for Mesos (source). Here, Hadoop and MPI are two applications that share the cluster.

这个图片是Mesos官方文档的一部分(source)。这里,HadoopMPI是两个共享集群的应用程序。

We’ll talk about each component shown here in the next few sections.

我们将在接下来的几节中讨论这里显示的每个组件。

3.1. Mesos Master

3.1. 主月

Master is the core component in this setup and stores the current state of resources in the cluster. Additionally, it acts as an orchestrator between the agents and the applications by passing information about things like resources and tasks.

Master是这个设置中的核心组件,它存储集群中资源的当前状态。此外,它作为代理和应用程序之间的协调者,通过传递有关资源和任务等信息。

Since any failure in master results in the loss of state about resources and tasks, we deploy it in high availability configuration. As can be seen in the diagram above, Mesos deploys standby master daemons along with one leader. These daemons rely on Zookeeper for recovering state in case of a failure.

由于主站的任何故障都会导致资源和任务状态的丢失,所以我们将其部署为高可用性配置。从上图中可以看出,Mesos部署了备用的主控程序和一个领导程序。这些守护程序依靠Zookeeper在发生故障时恢复状态。

3.2. Mesos Agents

3.2 Mesos代理

A Mesos cluster must run an agent on every machine. These agents report their resources to the master periodically and in turn, receive tasks that an application has scheduled to run. This cycle repeats after the scheduled task is either complete or lost.

一个Mesos集群必须在每台机器上运行一个代理。这些代理定期向主站报告他们的资源,反过来,接收应用程序已安排运行的任务。这个循环在预定任务完成或丢失后重复进行。

We’ll see how applications schedule and execute tasks on these agents in the following sections.

我们将在以下章节中看到应用程序如何在这些代理上安排和执行任务。

3.3. Mesos Frameworks

3.3.Mesos框架

Mesos allows applications to implement an abstract component that interacts with the Master to receive the available resources in the cluster and moreover make scheduling decisions based on them. These components are known as frameworks.

Mesos允许应用程序实现一个抽象的组件,该组件与Master交互,以接收集群中的可用资源,而且根据这些资源做出调度决定。这些组件被称为框架。

A Mesos framework is composed of two sub-components:

一个Mesos框架由两个子组件组成。

  • Scheduler – Enables applications to schedule tasks based on available resources on all the agents
  • Executor – Runs on all agents and contains all the information necessary to execute any scheduled task on that agent

This entire process is depicted with this flow:

这整个过程用这个流程来描述。

 

Mesos flow

First, the agents report their resources to the master. At this instant, master offers these resources to all registered schedulers. This process is known as a resource offer, and we’ll discuss it in detail in the next section.

首先,代理向主站报告他们的资源。在这一瞬间,主站将这些资源提供给所有注册的调度员。这个过程被称为资源提供,我们将在下一节详细讨论。

The scheduler then picks the best agent and executes various tasks on it through the Master. As soon as the executor completes the assigned task, agents re-publish their resources to the master. Master repeats this process of resource sharing for all frameworks in the cluster.

然后,调度员挑选出最好的代理,并通过主站在其上执行各种任务。一旦执行者完成了指定的任务,代理就会重新将其资源发布给Master。主站对集群中的所有框架重复这个资源共享的过程。

Mesos allows applications to implement their custom scheduler and executor in various programming languages. A Java implementation of scheduler must implement the Scheduler interface:

Mesos允许应用程序用各种编程语言实现他们的自定义调度器和执行器。调度器的Java实现必须实现Scheduler interface

public class HelloWorldScheduler implements Scheduler {
 
    @Override
    public void registered(SchedulerDriver schedulerDriver, Protos.FrameworkID frameworkID, 
      Protos.MasterInfo masterInfo) {
    }
 
    @Override
    public void reregistered(SchedulerDriver schedulerDriver, Protos.MasterInfo masterInfo) {
    }
 
    @Override
    public void resourceOffers(SchedulerDriver schedulerDriver, List<Offer> list) {
    }
 
    @Override
    public void offerRescinded(SchedulerDriver schedulerDriver, OfferID offerID) {
    }
 
    @Override
    public void statusUpdate(SchedulerDriver schedulerDriver, Protos.TaskStatus taskStatus) {
    }
 
    @Override
    public void frameworkMessage(SchedulerDriver schedulerDriver, Protos.ExecutorID executorID, 
      Protos.SlaveID slaveID, byte[] bytes) {
    }
 
    @Override
    public void disconnected(SchedulerDriver schedulerDriver) {
    }
 
    @Override
    public void slaveLost(SchedulerDriver schedulerDriver, Protos.SlaveID slaveID) {
    }
 
    @Override
    public void executorLost(SchedulerDriver schedulerDriver, Protos.ExecutorID executorID, 
      Protos.SlaveID slaveID, int i) {
    }
 
    @Override
    public void error(SchedulerDriver schedulerDriver, String s) {
    }
}

As can be seen, it mostly consists of various callback methods for communication with the master in particular.

可以看出,它主要由各种回调方法组成,特别是与主站的通信

Similarly, the implementation of an executor must implement the Executor interface:

同样地,执行器的实现必须实现执行器接口。

public class HelloWorldExecutor implements Executor {
    @Override
    public void registered(ExecutorDriver driver, Protos.ExecutorInfo executorInfo, 
      Protos.FrameworkInfo frameworkInfo, Protos.SlaveInfo slaveInfo) {
    }
  
    @Override
    public void reregistered(ExecutorDriver driver, Protos.SlaveInfo slaveInfo) {
    }
  
    @Override
    public void disconnected(ExecutorDriver driver) {
    }
  
    @Override
    public void launchTask(ExecutorDriver driver, Protos.TaskInfo task) {
    }
  
    @Override
    public void killTask(ExecutorDriver driver, Protos.TaskID taskId) {
    }
  
    @Override
    public void frameworkMessage(ExecutorDriver driver, byte[] data) {
    }
  
    @Override
    public void shutdown(ExecutorDriver driver) {
    }
}

We’ll see an operational version of the scheduler and executor in a later section.

我们将在后面的章节中看到调度器和执行器的操作版本。

4. Resource Management

4.资源管理

4.1. Resource Offers

4.1.资源报价

As we discussed earlier, agents publish their resource information to the master. In turn, the master offers these resources to the frameworks running in the cluster. This process is known as a resource offer.

正如我们前面所讨论的,代理将他们的资源信息发布给主站。反过来,主站将这些资源提供给集群中运行的框架。这个过程被称为资源提供。

A resource offer consists of two parts – resources and attributes.

一个资源报价由两部分组成–资源和属性。

Resources are used to publish hardware information of the agent machine such as memory, CPU, and disk.

资源用于公布代理机的硬件信息,如内存、CPU和磁盘。

There are five predefined resources for every Agent:

每个代理有五个预定义的资源。

  • cpu
  • gpus
  • mem
  • disk
  • ports

The values for these resources can be defined in one of the three types:

这些资源的价值可以定义为三种类型中的一种。

  • Scalar – Used to represent numerical information using floating point numbers to allow fractional values such as 1.5G of memory
  • Range – Used to represent a range of scalar values – for example, a port range
  • Set – Used to represent multiple text values

By default, Mesos agent tries to detect these resources from the machine.

默认情况下,Mesos代理会尝试从机器上检测这些资源。

However, in some situations, we can configure custom resources on an agent. The values for such custom resources should again be in any one of the types discussed above.

然而,在某些情况下,我们可以在一个代理上配置自定义资源。这种自定义资源的值也应该是上面讨论的任何一种类型。

For instance, we can start our agent with these resources:

例如,我们可以用这些资源启动我们的代理。

--resources='cpus:24;gpus:2;mem:24576;disk:409600;ports:[21000-24000,30000-34000];bugs(debug_role):{a,b,c}'

As can be seen, we’ve configured the agent with few of the predefined resources and one custom resource named bugs which is of set type.

可以看出,我们已经用一些预定义的资源和一个名为bugs的自定义资源配置了代理,该资源属于set类型。

In addition to resources, agents can publish key-value attributes to the master. These attributes act as additional metadata for the agent and help frameworks in scheduling decisions.

除了资源之外,代理可以向主站发布键值属性。这些属性作为代理的额外元数据,帮助框架进行调度决策。

A useful example can be to add agents into different racks or zones and then schedule various tasks on the same rack or zone to achieve data locality:

一个有用的例子是,将代理添加到不同的机架或区域,然后在同一机架或区域上安排各种任务,以实现数据定位。

--attributes='rack:abc;zone:west;os:centos5;level:10;keys:[1000-1500]'

Similar to resources, values for attributes can be either a scalar, a range, or a text type.

与资源类似,属性的值可以是一个标量,一个范围,或者一个文本类型。

4.2. Resource Roles

4.2.资源角色

Many modern-day operating systems support multiple users. Similarly, Mesos also supports multiple users in the same cluster. These users are known as roles. We can consider each role as a resource consumer within a cluster.

许多现代的操作系统都支持多用户。同样地,Mesos也支持同一集群中的多个用户。这些用户被称为roles。我们可以把每个角色看作是集群中的一个资源消费者。

Due to this, Mesos agents can partition the resources under different roles based on different allocation strategies. Furthermore, frameworks can subscribe to these roles within the cluster and have fine-grained control over resources under different roles.

由于这个原因,Mesos代理可以根据不同的分配策略,将资源划分到不同的角色下。此外,框架可以在集群内订阅这些角色,并对不同角色下的资源进行细粒度的控制。

For example, consider a cluster hosting applications which are serving different users in an organization. So by dividing the resources into roles, every application can work in isolation from one another.

例如,考虑一个托管应用程序的集群,该集群为一个组织中的不同用户提供服务。因此,通过将资源划分为角色,每个应用程序都可以在相互隔离的情况下工作

Additionally, frameworks can use these roles to achieve data locality.

此外,框架可以使用这些角色来实现数据定位。

For instance, suppose we have two applications in the cluster named producer and consumer. Here, producer writes data to a persistent volume which consumer can read afterward. We can optimize the consumer application by sharing the volume with the producer.

例如,假设我们在集群中有两个名为producer consumer的应用程序。生产者将数据写入一个持久化卷,消费者可以在之后读取。我们可以通过与生产者共享卷来优化消费者应用程序。

Since Mesos allows multiple applications to subscribe to the same role, we can associate the persistent volume with a resource role. Furthermore, the frameworks for both producer and consumer will both subscribe to the same resource role. Therefore, the consumer application can now launch the data reading task on the same volume as the producer application.

由于Mesos允许多个应用程序订阅同一个角色,我们可以将持久化卷与资源角色相关联。此外,生产者消费者的框架都将订阅同一个资源角色。因此,消费者应用程序现在可以在与生产者应用程序相同的卷上启动数据读取任务

4.3. Resource Reservation

4.3.资源保留

Now the question may arise as to how Mesos allocates cluster resources into different roles. Mesos allocates the resources through reservations.

现在可能出现的问题是,Mesos如何将集群资源分配到不同的角色。Mesos通过预订来分配资源。

There are two types of reservations:

有两种类型的保留。

  • Static Reservation
  • Dynamic Reservation

Static reservation is similar to the resource allocation on agent startup we discussed in the earlier sections:

静态保留类似于我们在前几节讨论的代理启动时的资源分配。

 --resources="cpus:4;mem:2048;cpus(baeldung):8;mem(baeldung):4096"

The only difference here is that now the Mesos agent reserves eight CPUs and 4096m of memory for the role named baeldung.

这里唯一的区别是,现在Mesos代理为名为baeldung的角色保留8个CPU和4096m内存。

Dynamic reservation allows us to reshuffle the resources within roles, unlike the static reservation. Mesos allows frameworks and cluster operators to dynamically change the allocation of resources via framework messages as a response to resource offer or via HTTP endpoints.

动态预留允许我们在角色内重新洗牌资源,与静态预留不同。Mesos允许框架和集群运营商通过框架消息动态地改变资源的分配,作为对资源报价的响应,或者通过HTTP端点

Mesos allocates all resources without any role into a default role named (*). Master offers such resources to all frameworks whether or not they have subscribed to it.

Mesos将所有没有任何角色的资源分配到一个名为(*)的默认角色。Master向所有框架提供这样的资源,无论它们是否订阅了它。

4.4. Resource Weights and Quotas

4.4.资源权重和配额

Generally, the Mesos master offers resources using a fairness strategy. It uses the weighted Dominant Resource Fairness (wDRF) to identify the roles that lack resources. The master then offers more resources to the frameworks that have subscribed to these roles.

一般来说,Mesos主站使用公平策略提供资源。它使用加权主导资源公平性(wDRF)来识别缺乏资源的角色。然后,主站向订阅了这些角色的框架提供更多资源。

Event though fair sharing of resources between applications is an important characteristic of Mesos, its not always necessary. Suppose a cluster hosting applications that have a low resource footprint along with those having a high resource demand. In such deployments, we would want to allocate resources based on the nature of the application.

尽管应用程序之间公平分享资源是Mesos的一个重要特征,但它并不总是必要的。假设一个集群承载着那些资源占用少的应用和那些资源需求大的应用。在这种部署中,我们希望根据应用的性质来分配资源。

Mesos allows frameworks to demand more resources by subscribing to roles and adding a higher value of weight for that role. Therefore, if there are two roles, one of weight 1 and another of weight 2, Mesos will allocate twice the fair share of resources to the second role.

Mesos允许框架通过订阅角色并为该角色添加更高的权重值来要求更多的资源。因此,如果有两个角色,一个权重为1,另一个权重为2,Mesos将为第二个角色分配两倍的公平资源份额。

Similar to resources, we can configure weights via HTTP endpoints.

与资源类似,我们可以通过HTTP端点来配置权重。

Besides ensuring a fair share of resources to a role with weights, Mesos also ensures that the minimum resources for a role are allocated.

除了确保对有权重的角色公平分配资源外,Mesos还确保为角色分配最小的资源。

Mesos allows us to add quotas to the resource roles. A quota specifies the minimum amount of resources that a role is guaranteed to receive.

Mesos允许我们为资源角色添加配额。配额指定了一个角色保证获得的最小资源量

5. Implementing Framework

5.实施框架

As we discussed in an earlier section, Mesos allows applications to provide framework implementations in a language of their choice. In Java, a framework is implemented using the main class – which acts as an entry point for the framework process – and the implementation of Scheduler and Executor discussed earlier.

正如我们在前面的章节中所讨论的,Mesos允许应用程序以他们选择的语言提供框架实现。在Java中,一个框架是通过主类(作为框架进程的入口)和前面讨论的Scheduler Executor 的实现来实现的。

5.1. Framework Main Class

5.1.框架主类

Before we implement a scheduler and an executor, we’ll first implement the entry point for our framework that:

在我们实现调度器和执行器之前,我们首先要实现我们框架的入口点。

  • Registers itself with the master
  • Provides executor runtime information to agents
  • Starts the scheduler

We’ll first add a Maven dependency for Mesos:

我们首先要为Mesos添加一个Maven依赖项

<dependency>
    <groupId>org.apache.mesos</groupId>
    <artifactId>mesos</artifactId>
    <version>0.28.3</version>
</dependency>

Next, we’ll implement the HelloWorldMain for our framework. One of the first things we’ll do is to start the executor process on the Mesos agent:

接下来,我们将为我们的框架实现HelloWorldMain。我们要做的第一件事是在Mesos代理上启动执行器进程。

public static void main(String[] args) {
  
    String path = System.getProperty("user.dir")
      + "/target/libraries2-1.0.0-SNAPSHOT.jar";
  
    CommandInfo.URI uri = CommandInfo.URI.newBuilder().setValue(path).setExtract(false).build();
  
    String helloWorldCommand = "java -cp libraries2-1.0.0-SNAPSHOT.jar com.baeldung.mesos.executors.HelloWorldExecutor";
    CommandInfo commandInfoHelloWorld = CommandInfo.newBuilder()
      .setValue(helloWorldCommand)
      .addUris(uri)
      .build();
  
    ExecutorInfo executorHelloWorld = ExecutorInfo.newBuilder()
      .setExecutorId(Protos.ExecutorID.newBuilder()
      .setValue("HelloWorldExecutor"))
      .setCommand(commandInfoHelloWorld)
      .setName("Hello World (Java)")
      .setSource("java")
      .build();
}

Here, we first configured the executor binary location. Mesos agent would download this binary upon framework registration. Next, the agent would run the given command to start the executor process.

在这里,我们首先配置了执行器二进制的位置。Mesos代理将在框架注册时下载这个二进制文件。接下来,代理将运行给定的命令来启动执行器进程。

Next, we’ll initialize our framework and start the scheduler:

接下来,我们将初始化我们的框架并启动调度器。

FrameworkInfo.Builder frameworkBuilder = FrameworkInfo.newBuilder()
  .setFailoverTimeout(120000)
  .setUser("")
  .setName("Hello World Framework (Java)");
 
frameworkBuilder.setPrincipal("test-framework-java");
 
MesosSchedulerDriver driver = new MesosSchedulerDriver(new HelloWorldScheduler(),
  frameworkBuilder.build(), args[0]);

Finally, we’ll start the MesosSchedulerDriver that registers itself with the Master. For successful registration, we must pass the IP of the Master as a program argument args[0] to this main class:

最后,我们将启动MesosSchedulerDriver,它将自己与主站注册。为了成功注册,我们必须将主站的IP作为程序参数args[0]传递给这个主类:

int status = driver.run() == Protos.Status.DRIVER_STOPPED ? 0 : 1;

driver.stop();

System.exit(status);

In the class shown above, CommandInfo, ExecutorInfo, and FrameworkInfo are all Java representations of protobuf messages between master and frameworks.

在上图所示的类中,CommandInfo、ExecutorInfoFrameworkInfo都是主站和框架之间protobuf消息的Java表示。

5.2. Implementing Scheduler

5.2.实现调度器

Since Mesos 1.0, we can invoke the HTTP endpoint from any Java application to send and receive messages to the Mesos master. Some of these messages include, for example, framework registration, resource offers, and offer rejections.

自Mesos 1.0以来,我们可以从任何Java应用程序中调用HTTP端点,以便向Mesos主站发送和接收消息。其中一些消息包括,例如,框架注册、资源提议和提议拒绝。

For Mesos 0.28 or earlier, we need to implement the Scheduler interface:

对于Mesos 0.28或更早,我们需要实现Scheduler接口

For the most part, we’ll only focus on the resourceOffers method of the Scheduler. Let’s see how a scheduler receives resources and initializes tasks based on them.

在大多数情况下,我们将只关注Scheduler的resourceOffers方法。让我们看看调度器是如何接收资源并根据资源初始化任务的。

First, we’ll see how the scheduler allocates resources for a task:

首先,我们将看到调度器如何为一个任务分配资源。

@Override
public void resourceOffers(SchedulerDriver schedulerDriver, List<Offer> list) {

    for (Offer offer : list) {
        List<TaskInfo> tasks = new ArrayList<TaskInfo>();
        Protos.TaskID taskId = Protos.TaskID.newBuilder()
          .setValue(Integer.toString(launchedTasks++)).build();

        System.out.println("Launching printHelloWorld " + taskId.getValue() + " Hello World Java");

        Protos.Resource.Builder cpus = Protos.Resource.newBuilder()
          .setName("cpus")
          .setType(Protos.Value.Type.SCALAR)
          .setScalar(Protos.Value.Scalar.newBuilder()
            .setValue(1));

        Protos.Resource.Builder mem = Protos.Resource.newBuilder()
          .setName("mem")
          .setType(Protos.Value.Type.SCALAR)
          .setScalar(Protos.Value.Scalar.newBuilder()
            .setValue(128));

Here, we allocated 1 CPU and 128M of memory for our task. Next, we’ll use the SchedulerDriver to launch the task on an agent:

这里,我们为我们的任务分配了1个CPU和128M的内存。接下来,我们将使用SchedulerDriver来启动代理上的任务。

        TaskInfo printHelloWorld = TaskInfo.newBuilder()
          .setName("printHelloWorld " + taskId.getValue())
          .setTaskId(taskId)
          .setSlaveId(offer.getSlaveId())
          .addResources(cpus)
          .addResources(mem)
          .setExecutor(ExecutorInfo.newBuilder(helloWorldExecutor))
          .build();

        List<OfferID> offerIDS = new ArrayList<>();
        offerIDS.add(offer.getId());

        tasks.add(printHelloWorld);

        schedulerDriver.launchTasks(offerIDS, tasks);
    }
}

Alternatively, Scheduler often finds the need to reject resource offers. For example, if the Scheduler cannot launch a task on an agent due to lack of resources, it must immediately decline that offer:

另外,Scheduler经常发现需要拒绝资源提议。例如,如果Scheduler由于缺乏资源而无法在一个代理上启动任务,它必须立即拒绝该提议。

schedulerDriver.declineOffer(offer.getId());

5.3. Implementing Executor

5.3.实现执行器

As we discussed earlier, the executor component of the framework is responsible for executing application tasks on the Mesos agent.

正如我们前面所讨论的,框架的执行器组件负责在Mesos代理上执行应用任务。

We used the HTTP endpoints for implementing Scheduler in Mesos 1.0. Likewise, we can use the HTTP endpoint for the executor.

我们在Mesos 1.0中使用HTTP端点来实现Scheduler。同样地,我们也可以使用HTTP端点来实现执行器。

In an earlier section, we discussed how a framework configures an agent to start the executor process:

在前面一节中,我们讨论了框架如何配置代理来启动执行者进程。

java -cp libraries2-1.0.0-SNAPSHOT.jar com.baeldung.mesos.executors.HelloWorldExecutor

Notably, this command considers HelloWorldExecutor as the main class. We’ll implement this main method to initialize the MesosExecutorDriver that connects with Mesos agents to receive tasks and share other information like task status:

值得注意的是,该命令将HelloWorldExecutor视为主类。我们将实现这个main方法来初始化MesosExecutorDriver,该方法与Mesos代理连接,接收任务并共享其他信息,如任务状态。

public class HelloWorldExecutor implements Executor {
    public static void main(String[] args) {
        MesosExecutorDriver driver = new MesosExecutorDriver(new HelloWorldExecutor());
        System.exit(driver.run() == Protos.Status.DRIVER_STOPPED ? 0 : 1);
    }
}

The last thing to do now is to accept tasks from the framework and launch them on the agent. The information to launch any task is self-contained within the HelloWorldExecutor:

现在要做的最后一件事是接受来自框架的任务并在代理上启动它们。启动任何任务的信息都包含在HelloWorldExecutor中:

public void launchTask(ExecutorDriver driver, TaskInfo task) {
 
    Protos.TaskStatus status = Protos.TaskStatus.newBuilder()
      .setTaskId(task.getTaskId())
      .setState(Protos.TaskState.TASK_RUNNING)
      .build();
    driver.sendStatusUpdate(status);
 
    System.out.println("Execute Task!!!");
 
    status = Protos.TaskStatus.newBuilder()
      .setTaskId(task.getTaskId())
      .setState(Protos.TaskState.TASK_FINISHED)
      .build();
    driver.sendStatusUpdate(status);
}

Of course, this is just a simple implementation, but it explains how an executor shares task status with the master at every stage and then executes the task before sending a completion status.

当然,这只是一个简单的实现,但它解释了一个执行者如何在每个阶段与主站分享任务状态,然后在发送完成状态之前执行任务。

In some cases, executors can also send data back to the scheduler:

在某些情况下,执行者也可以将数据送回给调度器。

String myStatus = "Hello Framework";
driver.sendFrameworkMessage(myStatus.getBytes());

6. Conclusion

6.结论

In this article, we discussed resource sharing between applications running in the same cluster in brief. We also discussed how Apache Mesos helps applications achieve maximum utilization with an abstract view of the cluster resources like CPU and memory.

在这篇文章中,我们简要地讨论了在同一集群中运行的应用程序之间的资源共享。我们还讨论了Apache Mesos如何通过对集群资源(如CPU和内存)的抽象视图帮助应用程序实现最大的利用率。

Later on, we discussed the dynamic allocation of resources between applications based on various fairness policies and roles. Mesos allows applications to make scheduling decisions based on resource offers from Mesos agents in the cluster.

后来,我们讨论了基于各种公平政策和角色的应用程序之间的资源动态分配Mesos允许应用程序根据集群中Mesos代理的资源提议做出调度决定

Finally, we saw an implementation of the Mesos framework in Java.

最后,我们看到了Mesos框架在Java中的实现。

As usual, all examples are available over on GitHub.

像往常一样,所有的例子都可以在GitHub上找到