Container for the parameters to the AddJobFlowSteps operation. AddJobFlowSteps adds new steps to a running job flow. A maximum of 256 steps are allowed in each job flow.

If your job flow is long-running (such as a Hive data warehouse) or complex, you may require more than 256 steps to process your data. You can bypass the 256-step limitation in various ways, including using the SSH shell to connect to the master node and submitting queries directly to the software running on the master node, such as Hive and Hadoop. For more information on how to do this, see Add More than 256 Steps to a Job Flow in the Amazon EMR Developer's Guide.

A step specifies the location of a JAR file stored either on the master node of the job flow or in Amazon S3. Each step is performed by the main function of the main class of the JAR file. The main class can be specified either in the manifest of the JAR or by using the MainFunction parameter of the step.

Amazon EMR executes each step in the order listed. For a step to be considered complete, the main function must exit with a zero exit code and all Hadoop jobs started while the step was running must have completed and run successfully.

You can only add steps to a job flow that is in one of the following states: STARTING, BOOTSTRAPPING, RUNNING, or WAITING.

Inheritance: AmazonElasticMapReduceRequest
コード例 #1
0
 protected override void ProcessRecord()
 {
     AmazonElasticMapReduce client = base.GetClient();
     Amazon.ElasticMapReduce.Model.AddJobFlowStepsRequest request = new Amazon.ElasticMapReduce.Model.AddJobFlowStepsRequest();
     request.JobFlowId = this._JobFlowId;
     Amazon.ElasticMapReduce.Model.AddJobFlowStepsResponse response = client.AddJobFlowSteps(request);
 }
コード例 #2
0
        public object Execute(ExecutorContext context)
        {
            var cmdletContext = context as CmdletContext;
            // create request
            var request = new Amazon.ElasticMapReduce.Model.AddJobFlowStepsRequest();

            if (cmdletContext.JobFlowId != null)
            {
                request.JobFlowId = cmdletContext.JobFlowId;
            }
            if (cmdletContext.Step != null)
            {
                request.Steps = cmdletContext.Step;
            }

            CmdletOutput output;

            // issue call
            var client = Client ?? CreateClient(_CurrentCredentials, _RegionEndpoint);

            try
            {
                var    response       = CallAWSServiceOperation(client, request);
                object pipelineOutput = null;
                pipelineOutput = cmdletContext.Select(response, this);
                output         = new CmdletOutput
                {
                    PipelineOutput  = pipelineOutput,
                    ServiceResponse = response
                };
            }
            catch (Exception e)
            {
                output = new CmdletOutput {
                    ErrorResponse = e
                };
            }

            return(output);
        }
コード例 #3
0
        /// <summary>
        /// Initiates the asynchronous execution of the AddJobFlowSteps operation.
        /// </summary>
        /// 
        /// <param name="request">Container for the necessary parameters to execute the AddJobFlowSteps operation.</param>
        /// <param name="cancellationToken">
        ///     A cancellation token that can be used by other objects or threads to receive notice of cancellation.
        /// </param>
        /// <returns>The task object representing the asynchronous operation.</returns>
        public Task<AddJobFlowStepsResponse> AddJobFlowStepsAsync(AddJobFlowStepsRequest request, System.Threading.CancellationToken cancellationToken = default(CancellationToken))
        {
            var marshaller = new AddJobFlowStepsRequestMarshaller();
            var unmarshaller = AddJobFlowStepsResponseUnmarshaller.Instance;

            return InvokeAsync<AddJobFlowStepsRequest,AddJobFlowStepsResponse>(request, marshaller, 
                unmarshaller, cancellationToken);
        }
コード例 #4
0
        /// <summary>
        /// AddJobFlowSteps adds new steps to a running job flow. A maximum of 256 steps are
        /// allowed in each job flow. 
        /// 
        ///  
        /// <para>
        /// If your job flow is long-running (such as a Hive data warehouse) or complex, you may
        /// require more than 256 steps to process your data. You can bypass the 256-step limitation
        /// in various ways, including using the SSH shell to connect to the master node and submitting
        /// queries directly to the software running on the master node, such as Hive and Hadoop.
        /// For more information on how to do this, go to <a href="http://docs.aws.amazon.com/ElasticMapReduce/latest/DeveloperGuide/AddMoreThan256Steps.html">Add
        /// More than 256 Steps to a Job Flow</a> in the <i>Amazon Elastic MapReduce Developer's
        /// Guide</i>.
        /// </para>
        ///  
        /// <para>
        ///  A step specifies the location of a JAR file stored either on the master node of the
        /// job flow or in Amazon S3. Each step is performed by the main function of the main
        /// class of the JAR file. The main class can be specified either in the manifest of the
        /// JAR or by using the MainFunction parameter of the step. 
        /// </para>
        ///  
        /// <para>
        ///  Elastic MapReduce executes each step in the order listed. For a step to be considered
        /// complete, the main function must exit with a zero exit code and all Hadoop jobs started
        /// while the step was running must have completed and run successfully. 
        /// </para>
        ///  
        /// <para>
        ///  You can only add steps to a job flow that is in one of the following states: STARTING,
        /// BOOTSTRAPPING, RUNNING, or WAITING.
        /// </para>
        /// </summary>
        /// <param name="request">Container for the necessary parameters to execute the AddJobFlowSteps service method.</param>
        /// 
        /// <returns>The response from the AddJobFlowSteps service method, as returned by ElasticMapReduce.</returns>
        /// <exception cref="Amazon.ElasticMapReduce.Model.InternalServerErrorException">
        /// Indicates that an error occurred while processing the request and that the request
        /// was not completed.
        /// </exception>
        public AddJobFlowStepsResponse AddJobFlowSteps(AddJobFlowStepsRequest request)
        {
            var marshaller = new AddJobFlowStepsRequestMarshaller();
            var unmarshaller = AddJobFlowStepsResponseUnmarshaller.Instance;

            return Invoke<AddJobFlowStepsRequest,AddJobFlowStepsResponse>(request, marshaller, unmarshaller);
        }
コード例 #5
0
 /// <summary>
 /// AddJobFlowSteps adds new steps to a running job flow. A maximum of 256 steps are
 /// allowed      in each job flow. 
 /// 
 ///       
 /// <para>
 /// If your job flow is long-running (such as a Hive data warehouse) or complex, you may
 /// require more than 256 steps to process your data. You can bypass the 256-step limitation
 /// in various ways, including using the SSH shell to connect to the master node and submitting
 /// queries directly to the software running on the master node, such as Hive and Hadoop.
 /// For more information on how to do this, go to <a href="http://docs.aws.amazon.com/ElasticMapReduce/latest/DeveloperGuide/AddMoreThan256Steps.html">Add
 /// More than 256 Steps to a Job Flow</a> in the <i>Amazon Elastic MapReduce Developer's
 /// Guide</i>.
 /// </para>
 ///       
 /// <para>
 ///  A step specifies the location of a JAR file stored either on the master node of the
 /// job         flow or in Amazon S3. Each step is performed by the main function of the
 /// main class of the         JAR file. The main class can be specified either in the
 /// manifest of the JAR or by using the         MainFunction parameter of the step. 
 /// </para>
 ///       
 /// <para>
 ///  Elastic MapReduce executes each step in the order listed. For a step to be considered
 ///         complete, the main function must exit with a zero exit code and all Hadoop
 /// jobs started         while the step was running must have completed and run successfully.
 /// 
 /// </para>
 ///       
 /// <para>
 ///  You can only add steps to a job flow that is in one of the following states: STARTING,
 ///      BOOTSTRAPPING, RUNNING, or WAITING.
 /// </para>
 /// </summary>
 /// <param name="request">Container for the necessary parameters to execute the AddJobFlowSteps service method.</param>
 /// 
 /// <returns>The response from the AddJobFlowSteps service method, as returned by ElasticMapReduce.</returns>
 /// <exception cref="InternalServerErrorException">
 /// Indicates that an error occurred while processing the request and that the request
 /// was not         completed.
 /// </exception>
 public AddJobFlowStepsResponse AddJobFlowSteps(AddJobFlowStepsRequest request)
 {
     var task = AddJobFlowStepsAsync(request);
     try
     {
         return task.Result;
     }
     catch(AggregateException e)
     {
         ExceptionDispatchInfo.Capture(e.InnerException).Throw();
         return null;
     }
 }
コード例 #6
0
        /// <summary>
        /// Initiates the asynchronous execution of the AddJobFlowSteps operation.
        /// <seealso cref="Amazon.ElasticMapReduce.IAmazonElasticMapReduce.AddJobFlowSteps"/>
        /// </summary>
        /// 
        /// <param name="request">Container for the necessary parameters to execute the AddJobFlowSteps operation.</param>
        /// <param name="cancellationToken">
        ///     A cancellation token that can be used by other objects or threads to receive notice of cancellation.
        /// </param>
        /// <returns>The task object representing the asynchronous operation.</returns>
		public Task<AddJobFlowStepsResponse> AddJobFlowStepsAsync(AddJobFlowStepsRequest request, CancellationToken cancellationToken = default(CancellationToken))
        {
            var marshaller = new AddJobFlowStepsRequestMarshaller();
            var unmarshaller = AddJobFlowStepsResponseUnmarshaller.GetInstance();
            return Invoke<IRequest, AddJobFlowStepsRequest, AddJobFlowStepsResponse>(request, marshaller, unmarshaller, signer, cancellationToken);
        }
コード例 #7
0
        /// <summary>
        /// <para> AddJobFlowSteps adds new steps to a running job flow. A maximum of 256 steps are allowed in each job flow. </para> <para>If your job
        /// flow is long-running (such as a Hive data warehouse) or complex, you may require more than 256 steps to process your data. You can bypass
        /// the 256-step limitation in various ways, including using the SSH shell to connect to the master node and submitting queries directly to the
        /// software running on the master node, such as Hive and Hadoop. For more information on how to do this, go to <a href="http://docs.aws.amazon.com/ElasticMapReduce/latest/DeveloperGuide/AddMoreThan256Steps.html">Add More than 256 Steps to a Job Flow</a>
        /// in the <i>Amazon Elastic MapReduce Developer's Guide</i> .</para> <para> A step specifies the location of a JAR file stored either on the
        /// master node of the job flow or in Amazon S3. Each step is performed by the main function of the main class of the JAR file. The main class
        /// can be specified either in the manifest of the JAR or by using the MainFunction parameter of the step. </para> <para> Elastic MapReduce
        /// executes each step in the order listed. For a step to be considered complete, the main function must exit with a zero exit code and all
        /// Hadoop jobs started while the step was running must have completed and run successfully. </para> <para> You can only add steps to a job flow
        /// that is in one of the following states: STARTING, BOOTSTRAPPING, RUNNING, or WAITING.</para>
        /// </summary>
        /// 
        /// <param name="request">Container for the necessary parameters to execute the AddJobFlowSteps service method on
        /// AmazonElasticMapReduce.</param>
        /// 
        /// <returns>The response from the AddJobFlowSteps service method, as returned by AmazonElasticMapReduce.</returns>
        /// 
        /// <exception cref="T:Amazon.ElasticMapReduce.Model.InternalServerErrorException" />
		public AddJobFlowStepsResponse AddJobFlowSteps(AddJobFlowStepsRequest request)
        {
            var task = AddJobFlowStepsAsync(request);
            try
            {
                return task.Result;
            }
            catch(AggregateException e)
            {
                throw e.InnerException;
            }
        }
コード例 #8
0
        /// <summary>
        /// Initiates the asynchronous execution of the AddJobFlowSteps operation.
        /// </summary>
        /// 
        /// <param name="request">Container for the necessary parameters to execute the AddJobFlowSteps operation on AmazonElasticMapReduceClient.</param>
        /// <param name="callback">An AsyncCallback delegate that is invoked when the operation completes.</param>
        /// <param name="state">A user-defined state object that is passed to the callback procedure. Retrieve this object from within the callback
        ///          procedure using the AsyncState property.</param>
        /// 
        /// <returns>An IAsyncResult that can be used to poll or wait for results, or both; this value is also needed when invoking EndAddJobFlowSteps
        ///         operation.</returns>
        public IAsyncResult BeginAddJobFlowSteps(AddJobFlowStepsRequest request, AsyncCallback callback, object state)
        {
            var marshaller = new AddJobFlowStepsRequestMarshaller();
            var unmarshaller = AddJobFlowStepsResponseUnmarshaller.Instance;

            return BeginInvoke<AddJobFlowStepsRequest>(request, marshaller, unmarshaller,
                callback, state);
        }
コード例 #9
0
 private Amazon.ElasticMapReduce.Model.AddJobFlowStepsResponse CallAWSServiceOperation(IAmazonElasticMapReduce client, Amazon.ElasticMapReduce.Model.AddJobFlowStepsRequest request)
 {
     Utils.Common.WriteVerboseEndpointMessage(this, client.Config, "Amazon Elastic MapReduce", "AddJobFlowSteps");
     try
     {
         #if DESKTOP
         return(client.AddJobFlowSteps(request));
         #elif CORECLR
         return(client.AddJobFlowStepsAsync(request).GetAwaiter().GetResult());
         #else
                 #error "Unknown build edition"
         #endif
     }
     catch (AmazonServiceException exc)
     {
         var webException = exc.InnerException as System.Net.WebException;
         if (webException != null)
         {
             throw new Exception(Utils.Common.FormatNameResolutionFailureMessage(client.Config, webException.Message), webException);
         }
         throw;
     }
 }
コード例 #10
0
 IAsyncResult invokeAddJobFlowSteps(AddJobFlowStepsRequest addJobFlowStepsRequest, AsyncCallback callback, object state, bool synchronized)
 {
     IRequest irequest = new AddJobFlowStepsRequestMarshaller().Marshall(addJobFlowStepsRequest);
     var unmarshaller = AddJobFlowStepsResponseUnmarshaller.GetInstance();
     AsyncResult result = new AsyncResult(irequest, callback, state, synchronized, signer, unmarshaller);
     Invoke(result);
     return result;
 }
コード例 #11
0
 /// <summary>
 /// Initiates the asynchronous execution of the AddJobFlowSteps operation.
 /// <seealso cref="Amazon.ElasticMapReduce.IAmazonElasticMapReduce.AddJobFlowSteps"/>
 /// </summary>
 /// 
 /// <param name="addJobFlowStepsRequest">Container for the necessary parameters to execute the AddJobFlowSteps operation on
 ///          AmazonElasticMapReduce.</param>
 /// <param name="callback">An AsyncCallback delegate that is invoked when the operation completes.</param>
 /// <param name="state">A user-defined state object that is passed to the callback procedure. Retrieve this object from within the callback
 ///          procedure using the AsyncState property.</param>
 /// 
 /// <returns>An IAsyncResult that can be used to poll or wait for results, or both; this value is also needed when invoking EndAddJobFlowSteps
 ///         operation.</returns>
 public IAsyncResult BeginAddJobFlowSteps(AddJobFlowStepsRequest addJobFlowStepsRequest, AsyncCallback callback, object state)
 {
     return invokeAddJobFlowSteps(addJobFlowStepsRequest, callback, state, false);
 }
コード例 #12
0
 /// <summary>
 /// <para> AddJobFlowSteps adds new steps to a running job flow. A maximum of 256 steps are allowed in each job flow. </para> <para>If your job
 /// flow is long-running (such as a Hive data warehouse) or complex, you may require more than 256 steps to process your data. You can bypass
 /// the 256-step limitation in various ways, including using the SSH shell to connect to the master node and submitting queries directly to the
 /// software running on the master node, such as Hive and Hadoop. For more information on how to do this, go to <a
 /// href="http://docs.aws.amazon.com/ElasticMapReduce/latest/DeveloperGuide/AddMoreThan256Steps.html" >Add More than 256 Steps to a Job Flow</a>
 /// in the <i>Amazon Elastic MapReduce Developer's Guide</i> .</para> <para> A step specifies the location of a JAR file stored either on the
 /// master node of the job flow or in Amazon S3. Each step is performed by the main function of the main class of the JAR file. The main class
 /// can be specified either in the manifest of the JAR or by using the MainFunction parameter of the step. </para> <para> Elastic MapReduce
 /// executes each step in the order listed. For a step to be considered complete, the main function must exit with a zero exit code and all
 /// Hadoop jobs started while the step was running must have completed and run successfully. </para> <para> You can only add steps to a job flow
 /// that is in one of the following states: STARTING, BOOTSTRAPPING, RUNNING, or WAITING.</para>
 /// </summary>
 /// 
 /// <param name="addJobFlowStepsRequest">Container for the necessary parameters to execute the AddJobFlowSteps service method on
 ///          AmazonElasticMapReduce.</param>
 /// 
 /// <returns>The response from the AddJobFlowSteps service method, as returned by AmazonElasticMapReduce.</returns>
 /// 
 /// <exception cref="InternalServerErrorException"/>
 public AddJobFlowStepsResponse AddJobFlowSteps(AddJobFlowStepsRequest addJobFlowStepsRequest)
 {
     IAsyncResult asyncResult = invokeAddJobFlowSteps(addJobFlowStepsRequest, null, null, true);
     return EndAddJobFlowSteps(asyncResult);
 }
コード例 #13
0
 /// <summary>
 /// <para> AddJobFlowSteps adds new steps to a running job flow. A maximum
 /// of 256 steps are allowed in each job flow. </para> <para> A step
 /// specifies the location of a JAR file stored either on the master node
 /// of the job flow or in Amazon S3. Each step is performed by the main
 /// function of the main class of the JAR file. The main class can be
 /// specified either in the manifest of the JAR or by using the
 /// MainFunction parameter of the step. </para> <para> Elastic MapReduce
 /// executes each step in the order listed. For a step to be considered
 /// complete, the main function must exit with a zero exit code and all
 /// Hadoop jobs started while the step was running must have completed and
 /// run successfully. </para> <para> You can only add steps to a job flow
 /// that is in one of the following states: STARTING, BOOTSTAPPING,
 /// RUNNING, or WAITING.</para>
 /// </summary>
 /// 
 /// <param name="addJobFlowStepsRequest">Container for the necessary
 ///           parameters to execute the AddJobFlowSteps service method on
 ///           AmazonElasticMapReduce.</param>
 /// 
 /// <exception cref="InternalServerErrorException"/>
 public AddJobFlowStepsResponse AddJobFlowSteps(AddJobFlowStepsRequest addJobFlowStepsRequest)
 {
     IRequest<AddJobFlowStepsRequest> request = new AddJobFlowStepsRequestMarshaller().Marshall(addJobFlowStepsRequest);
     AddJobFlowStepsResponse response = Invoke<AddJobFlowStepsRequest, AddJobFlowStepsResponse> (request, this.signer, AddJobFlowStepsResponseUnmarshaller.GetInstance());
     return response;
 }
コード例 #14
0
        /// <summary>
        /// <para> AddJobFlowSteps adds new steps to a running job flow. A maximum of 256 steps are allowed in each job flow. </para> <para>If your job
        /// flow is long-running (such as a Hive data warehouse) or complex, you may require more than 256 steps to process your data. You can bypass
        /// the 256-step limitation in various ways, including using the SSH shell to connect to the master node and submitting queries directly to the
        /// software running on the master node, such as Hive and Hadoop. For more information on how to do this, go to <a href="http://docs.aws.amazon.com/ElasticMapReduce/latest/DeveloperGuide/AddMoreThan256Steps.html">Add More than 256 Steps to a Job Flow</a>
        /// in the <i>Amazon Elastic MapReduce Developer's Guide</i> .</para> <para> A step specifies the location of a JAR file stored either on the
        /// master node of the job flow or in Amazon S3. Each step is performed by the main function of the main class of the JAR file. The main class
        /// can be specified either in the manifest of the JAR or by using the MainFunction parameter of the step. </para> <para> Elastic MapReduce
        /// executes each step in the order listed. For a step to be considered complete, the main function must exit with a zero exit code and all
        /// Hadoop jobs started while the step was running must have completed and run successfully. </para> <para> You can only add steps to a job flow
        /// that is in one of the following states: STARTING, BOOTSTRAPPING, RUNNING, or WAITING.</para>
        /// </summary>
        /// 
        /// <param name="addJobFlowStepsRequest">Container for the necessary parameters to execute the AddJobFlowSteps service method on
        /// AmazonElasticMapReduce.</param>
        /// 
        /// <returns>The response from the AddJobFlowSteps service method, as returned by AmazonElasticMapReduce.</returns>
        /// 
        /// <exception cref="T:Amazon.ElasticMapReduce.Model.InternalServerErrorException" />
        /// <param name="cancellationToken">
        ///     A cancellation token that can be used by other objects or threads to receive notice of cancellation.
        /// </param>
		public async Task<AddJobFlowStepsResponse> AddJobFlowStepsAsync(AddJobFlowStepsRequest addJobFlowStepsRequest, CancellationToken cancellationToken = default(CancellationToken))
        {
            var marshaller = new AddJobFlowStepsRequestMarshaller();
            var unmarshaller = AddJobFlowStepsResponseUnmarshaller.GetInstance();
            var response = await Invoke<IRequest, AddJobFlowStepsRequest, AddJobFlowStepsResponse>(addJobFlowStepsRequest, marshaller, unmarshaller, signer, cancellationToken)
                .ConfigureAwait(continueOnCapturedContext: false);
            return response;
        }
コード例 #15
0
        IAsyncResult invokeAddJobFlowSteps(AddJobFlowStepsRequest request, AsyncCallback callback, object state, bool synchronized)
        {
            var marshaller = new AddJobFlowStepsRequestMarshaller();
            var unmarshaller = AddJobFlowStepsResponseUnmarshaller.Instance;

            return Invoke(request, callback, state, synchronized, marshaller, unmarshaller, signer);
        }