JFrog

    Cloud customer?
    Start for Free>
    Upgrade in MyJFrog >
    What's New in Cloud >






    Skip to end of metadata
    Go to start of metadata

    Overview

    The execution of each step in a pipeline may be performed on different nodes. For this reason, you are not guaranteed that changes to the environment that are made within a step will persist to subsequent steps.

    Stateful pipelines remember information generated by steps and make it available to dependent steps or successive runs of the same pipeline. This is a crucial component in order to achieve end-to-end Continuous Delivery.

    Some example use-cases are the following:

    • A step creates information about the commitSha and image/file version that was built, which is then consumed by another step to deploy the correct version into the test environment.
    • A step creates a VPC for the Staging environment. It stores information like VPC info, subnets, security groups, etc. which is required by another step that deploys to the Staging environment.
    • You have a provisioning step that uses Terraform to create the Test environment. At a later stage in your pipeline, you have a deprovisioning step that destroys the test environment. Both these steps need to read and update the Terraform state file so they are aware of the current state of the Test environment.


    Page Contents

    Types of State

    JFrog Pipelines supports three types of state:

    Each of these states is characterized by the scope of information persistence.

    Run State

    A pipeline'srun stateis persistent only between steps within the same run. Information stored in one step is available to subsequent steps in the run of that pipeline. After the run is complete,state may be downloaded when viewing the steps in that run but will not be available in later runs or other pipelines.


    To preserve state across steps, use theutility functions for run state management.


    Pipelines supports two types of run state information that can be preserved between steps.

    Key-Value Pairs

    Using theadd_run_variablesutility function, you can store a key-value pair to the run state. That key-value pair will automatically be available to all subsequent steps in the runas an environment variable.It will not be available to steps that do not have the step that added the variable as an input, either directly or through other steps or resources.

    pipelines: - name: example_run_state_pipeline steps: - name: step_1 type: Bash configuration: inputResources: - name: myAppRepo # Triggering resource execution: onExecute: - add_run_variables first_stepid=$step_id - add_run_variables ${first_stepid}_buildNumber=${run_number} . . . - name: step_5 type: Bash configuration: inputSteps: - name: step_4 execution: onExecute: - echo "Hello world" - echo $first_stepid - echo ${first_stepid}_buildNumber


    Files

    Using theadd_run_filesutility function, a step can store a file in the run state. Any subsequent step can then use therestore_run_filesfunction to retrieve the file from the run state.Files are available to steps in the same run whether or not the step that added the files is an input to the later step. Run state may be downloaded for an individual step, consisting of the files either uploaded or downloaded by that step.

    管道:名字:example_run_file_pipeline steps: - name: step_1 type: Bash configuration: inputResources: - name: myAppRepo # Triggering resource execution: onExecute: echo "Hello world" onComplete: - add_run_files myfile cachefile.txt # save the variable 'myfile' as 'cachefile.txt' . . . - name: step_5 type: Bash configuration: inputSteps: - name: step_4 execution: onStart: - restore_run_files myfile cachefile.txt # save the contents of 'cachefile.txt' to 'myfile' onExecute: echo "Hello world"

    Download Run State

    To download run state from the UI for debugging purposes, set the environment variableJFROG_PIPELINES_RUN_STATE_DEBUG=truein your step's or pipeline's configuration section. For storage efficiency, we recommend removing this configuration option after you are done debugging.

    pipelines: - name: example_run_state_pipeline configuration: environmentVariables: readOnly: JFROG_PIPELINES_RUN_STATE_DEBUG: true steps: - name: step_1 type: Bash configuration: inputResources: - name: myAppRepo # Triggering resource execution: onExecute: - add_run_variables first_stepid=$step_id - add_run_variables ${first_stepid}_buildNumber=${run_number} onComplete: - add_run_files myfile cachefile.txt # save the variable 'myfile' as 'cachefile.txt'

    Pipeline State

    Apipeline stateis persistent for all runs of the same pipeline.ste信息存储p during a pipeline's run is available to subsequent runs of that pipeline.

    To preserve state across pipelines, you may use theutility functions for pipeline state management.

    Pipelines supports two types of run state information that can be preserved between steps.

    Key-Value Pairs

    Using theadd_pipeline_variablesutility function, you can store a key-value pair to the pipeline state. That key-value pair will automatically be available to all subsequent runs as an environment variable.

    pipelines: - name: example_pipeline_state_pipeline steps: - name: step_1 type: Bash configuration: inputResources: - name: myAppRepo # Triggering resource execution: onExecute: - echo "Hello world" - echo $previous_buildNumber . . . - name: final_step type: Bash configuration: inputSteps: - name: prior_step execution: onExecute: # Preserve the build number for the next run of the pipeline - add_pipeline_variables $previous_buildNumber=${run_number}

    Files

    Using theadd_pipeline_filesutility function, a step can store a file to the pipeline state. Any step can then use therestore_pipeline_filesfunction to retrieve the file from the pipeline state.

    pipelines: - name: example_pipeline_file_pipeline steps: - name: step_1 type: Bash configuration: inputResources: - name: myAppRepo # Triggering resource execution: onStart: # Restore the file from the previous run of the pipeline - restore_pipeline_files myfile cachefile.txt # save the contents of 'cachefile.txt' to 'myfile' onExecute: echo "Hello world" . . . - name: final_step type: Bash configuration: inputSteps: - name: prior_step execution: onExecute: echo "Hello world" onComplete: # Preserve file for the next run of the pipeline - add_pipeline_files myfile cachefile.txt # save the variable 'myfile' as 'cachefile.txt'

    Resource-based State

    Using thewrite_outpututility function, key-values can be stored as a property in any output resource. Every step that has the resource as aninput can access the key-value information in its scripts as an environment variable.

    The environment variable for the value is of the formres__.

    Resource-based state information is persistent across pipelines, so it can be used as a mechanism for passing information from one pipeline to the next.

    pipelines: - name: example_resource_state_pipeline steps: - name: step_1 type: Bash configuration: inputResources: - name: myAppRepo # Triggering resource outputResources: - name: myImage # Image resource execution: onExecute: - echo $res_myAppRepo_commitSha - write_output myImage "imageTag=master" "sha=$res_myAppRepo_commitSha" "description=\"hello world\"" - name: step_2 type: Bash configuration: inputResources: - name: myImage execution: onExecute: - echo "Hello world" - echo $res_myImage_imageTag - echo $res_myImage_sha - echo $res_myImage_description
    • No labels
    Copyright © 2023 JFrog Ltd.