//
// Licensed to the Apache Software Foundation (ASF) under one or more
// contributor license agreements.  See the NOTICE file distributed with
// this work for additional information regarding copyright ownership.
// The ASF licenses this file to You under the Apache License, Version 2.0
// (the "License"); you may not use this file except in compliance with
// the License.  You may obtain a copy of the License at
//
//     http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//
= Apache NiFi Toolkit Guide
Apache NiFi Team <dev@nifi.apache.org>
:homepage: http://nifi.apache.org
:linkattrs:

== Overview
The NiFi Toolkit contains several command line utilities to setup and support NiFi in standalone and clustered environments.  The utilities include:

* CLI -- The `cli` tool enables administrators to interact with NiFi and NiFi Registry instances to automate tasks such as deploying versioned flows and managing process groups and cluster nodes.
* Encrypt Config -- The `encrypt-config` tool encrypts the sensitive keys in the _nifi.properties_ file to facilitate the setup of a secure NiFi instance.

The utilities are executed with scripts found in the `bin` folder of your NiFi Toolkit installation.

NOTE: The NiFi Toolkit is downloaded separately from NiFi (see the link:https://nifi.apache.org/download.html[Apache NiFi Downloads^] page).

=== Prerequisites for Running in a Secure Environment
For secured nodes and clusters, two policies should be configured in advance:

* Access the controller – A user that will have access to these utilities should be authorized in NiFi by creating an “access the controller” policy (`/controller`) with both view and modify rights
* Proxy user request – If not previously set, node’s identity (the DN value of the node’s certificate) should be authorized to proxy requests on behalf of a user

When executing either the Notify or Node Manager tools in a secured environment the `proxyDN` flag option should be used in order to properly identify the user that was authorized to execute these commands. In non-secure environments, or if running the status operation on the Node Manager tool, the flag is ignored.

[[nifi_CLI]]
== NiFi CLI
This tool offers a CLI focused on interacting with NiFi and NiFi Registry in order to automate tasks, such as deploying flows from a NIFi Registy to a NiFi instance or managing process groups and cluster nodes.

=== Usage
The CLI toolkit can be executed in standalone mode to execute a single command, or interactive mode to enter an interactive shell.

To execute a single command:

 ./bin/cli.sh <command> <args>

To launch the interactive shell:

 ./bin/cli.sh

To show help:

 ./bin/cli.sh -h

The following are available commands:

 demo quick-import
 nifi current-user
 nifi cluster-summary
 nifi connect-node
 nifi delete-node
 nifi disconnect-node
 nifi get-root-id
 nifi get-node
 nifi get-nodes
 nifi offload-node
 nifi list-reg-clients
 nifi create-reg-client
 nifi update-reg-client
 nifi get-reg-client-id
 nifi pg-import
 nifi pg-connect
 nifi pg-start
 nifi pg-stop
 nifi pg-create
 nifi pg-export
 nifi pg-get-version
 nifi pg-stop-version-control
 nifi pg-change-version
 nifi pg-change-all-versions
 nifi pg-get-all-versions
 nifi pg-list
 nifi pg-status
 nifi pg-get-services
 nifi pg-create-service
 nifi pg-enable-services
 nifi pg-disable-services
 nifi pg-get-param-context
 nifi pg-set-param-context
 nifi pg-replace
 nifi get-services
 nifi get-service
 nifi create-service
 nifi enable-services
 nifi disable-services
 nifi get-reporting-tasks
 nifi get-reporting-task
 nifi create-reporting-task
 nifi delete-reporting-task
 nifi start-reporting-tasks
 nifi stop-reporting-tasks
 nifi export-reporting-tasks
 nifi export-reporting-task
 nifi import-reporting-tasks
 nifi list-users
 nifi create-user
 nifi list-user-groups
 nifi create-user-group
 nifi update-user-group
 nifi get-policy
 nifi update-policy
 nifi list-param-contexts
 nifi get-param-context
 nifi create-param-context
 nifi delete-param-context
 nifi set-inherited-param-contexts
 nifi remove-inherited-param-contexts
 nifi set-param
 nifi delete-param
 nifi export-param-context
 nifi import-param-context
 nifi merge-param-context
 nifi list-param-providers
 nifi get-param-provider
 nifi create-param-provider
 nifi delete-param-provider
 nifi fetch-params
 nifi set-param-provider-property
 nifi get-access-token
 nifi get-access-token-spnego
 nifi logout-access-token
 nifi get-controller-configuration
 nifi update-controller-configuration
 nifi change-version-processor
 registry current-user
 registry list-buckets
 registry create-bucket
 registry delete-bucket
 registry list-flows
 registry create-flow
 registry delete-flow
 registry list-flow-versions
 registry export-flow-version
 registry import-flow-version
 registry sync-flow-versions
 registry transfer-flow-version
 registry diff-flow-versions
 registry upload-bundle
 registry upload-bundles
 registry list-bundle-groups
 registry list-bundle-artifacts
 registry list-bundle-versions
 registry download-bundle
 registry get-bundle-checksum
 registry list-extension-tags
 registry list-extensions
 registry list-users
 registry create-user
 registry update-user
 registry list-user-groups
 registry create-user-group
 registry update-user-group
 registry get-policy
 registry update-policy
 registry update-bucket-policy
 registry get-access-token
 registry get-access-token-spnego
 registry logout-access-token
 registry export-all-flows
 registry import-all-flows
 session keys
 session show
 session get
 session set
 session remove
 session clear
 exit
 help

To display extensive help for a specific command:

 ./bin/cli.sh <command> -h

=== Property/Argument Handling
Most commands will require specifying a baseUrl for the NiFi or NiFi Registry instance.

An example command to list the buckets in a NiFi Registry instance would be the following:

 ./bin/cli.sh registry list-buckets -u http://localhost:18080

In order to avoid specifying the URL (and possibly other optional arguments for TLS) on every command, you can define a properties file containing the repetitive arguments.

An example properties file for a local NiFi Registry instance would look like the following:

[source]
----
 baseUrl=http://localhost:18080
 keystore=
 keystoreType=
 keystorePasswd=
 keyPasswd=
 truststore=
 truststoreType=
 truststorePasswd=
 proxiedEntity=
----

This properties file can then be used on a command by specifying `-p`:

 ./bin/cli.sh registry list-buckets -p /path/to/local-nifi-registry.properties

You could then maintain a properties file for each environment you plan to interact with, such as Dev, QA, and Prod.

In addition to specifying a properties file on each command, you can setup a default properties file to be used in the event that no properties file is specified.

The default properties file is specified using the `session` concept, which persists to the users home directory in a file called _.nifi-cli.config_.

An example of setting the default property files for NiFi would be the following:

 ./bin/cli.sh session set nifi.props /path/to/local-nifi.properties

An example for NiFi Registry would be the following:

 ./bin/cli.sh session set nifi.reg.props /path/to/local-nifi-registry.properties

This will write the above properties into the _.nifi-cli.config_ in the user's home directory and will allow commands to be executed without specifying a URL or properties file:

 ./bin/cli.sh registry list-buckets

The above command will now use the `baseUrl` from _local-nifi-registry.properties_.

The order of resolving an argument is the following:

* A direct argument overrides anything in a properties file or session
* A properties file argument (`-p`) overrides the session
* The session is used when nothing else is specified

=== Security Configuration
If NiFi and NiFi Registry are secured, then commands executed from the CLI will need to make a TLS connection and authenticate as a user with permissions to perform the desired action.

Currently the CLI supports authenticating with a client certificate and an optional proxied-entity. A common scenario would be running the CLI from one of the nodes where NiFi or NiFi Registry is installed, which allows the CLI to use the same keystore and truststore as the NiFi/NiFi Registry instance.

The security configuration can be specified per-command, or in one of the properties files described in the previous section.

The examples below are for NiFi Registry, but the same concept applies for NiFi commands.

==== Example - Secure NiFi Registry without Proxied-Entity
Assuming we have a keystore containing the certificate for "CN=user1, OU=NIFI", an example properties file would be the following:

[source]
----
 baseUrl=https://localhost:18443
 keystore=/path/to/keystore.jks
 keystoreType=JKS
 keystorePasswd=changeme
 keyPasswd=changeme
 truststore=/path/to/truststore.jks
 truststoreType=JKS
 truststorePasswd=changeme
----

In this example, commands will be executed as "CN=user1, OU=NIFI". This user would need to be a user in NiFi Registry, and commands accessing buckets would be restricted to buckets this user has access to.

==== Example - Secure NiFi Registry with Proxied-Entity
Assuming we have access to the keystore of NiFi Registry itself, and that NiFi Registry is also configured to allow Kerberos or LDAP authentication, an example properties file would be the following:

[source]
----
 baseUrl=https://localhost:18443
 keystore=/path/to/keystore.jks
 keystoreType=JKS
 keystorePasswd=changeme
 keyPasswd=changeme
 truststore=/path/to/truststore.jks
 truststoreType=JKS
 truststorePasswd=changeme
 proxiedEntity=user1@NIFI.COM
----

In this example, the certificate in _keystore.jks_ would be for the NiFi Registry server, for example "CN=localhost, OU=NIFI". This identity would need to be defined as a user in NiFi Registry and given permissions to 'Proxy'.

"CN=localhost, OU=NIFI" would be proxying commands to be executed as user1@NIFI.COM.

=== Interactive Usage
In interactive mode the tab key can be used to perform auto-completion.

For example, typing tab at an empty prompt should display possible commands for the first argument:

 #>
 demo       exit       help       nifi       registry   session

Typing "nifi " and then a tab will show the sub-commands for NiFi:

 #> nifi
 change-version-processor   delete-reporting-task           get-policy              merge-param-context     pg-replace                      update-user-group
 cluster-summary            disable-services                get-reg-client-id       offload-node            pg-set-param-context
 connect-node               disconnect-node                 get-reporting-task      pg-change-version       pg-start
 create-param-context       enable-services                 get-reporting-tasks     pg-connect              pg-status
 create-param-provider      export-param-context            get-root-id             pg-create               pg-stop
 create-reg-client          export-reporting-task           get-service             pg-create-service       pg-stop-version-control
 create-reporting-task      export-reporting-tasks          get-services            pg-disable-services     remove-inherited-param-contexts
 create-service             fetch-params                    import-param-context    pg-enable-services      set-inherited-param-contexts
 create-user                get-access-token                import-reporting-tasks  pg-export               set-param
 create-user-group          get-access-token-spnego         list-param-contexts     pg-get-all-versions     set-param-provider-property
 current-user               get-controller-configuration    list-param-providers    pg-get-param-context    start-reporting-tasks
 delete-node                get-node                        list-reg-clients        pg-get-services         stop-reporting-tasks
 delete-param               get-nodes                       list-user-groups        pg-get-version          update-controller-configuration
 delete-param-context       get-param-context               list-users              pg-import               update-policy
 delete-param-provider      get-param-provider              logout-access-token     pg-list                 update-reg-client

Arguments that represent a path to a file, such as `-p` or when setting a properties file in the session, will auto-complete the path being typed:

 #> session set nifi.props /tmp/
 dir1/   dir2/   dir3/

=== Output
Most commands support the ability to specify an `--outputType` argument, or `-ot` for short.

Currently the output type may be simple or json.

The default output type in interactive mode is simple, and the default output type in standalone mode is json.

Example of simple output for `list-buckets`:

 #> registry list-buckets -ot simple
 My Bucket - 3c7b7467-0012-4d8f-a918-6aa42b6b9d39

Example of json output for `list-buckets`:

 #> registry list-buckets -ot json
 [ {
   "identifier" : "3c7b7467-0012-4d8f-a918-6aa42b6b9d39",
   "name" : "My Bucket",
   "createdTimestamp" : 1516718733854,
   "permissions" : {
     "canRead" : true,
     "canWrite" : true,
     "canDelete" : true
   },
   "link" : {
     "params" : {
       "rel" : "self"
     },
     "href" : "buckets/3c7b7467-0012-4d8f-a918-6aa42b6b9d39"
   }
 } ]

=== Back-Referencing
When using the interactive CLI, a common scenario will be using an id from a previous result as the input to the next command. Back-referencing provides a shortcut for referencing a result from the previous command via a positional reference.

NOTE: Not every command produces back-references. To determine if a command supports back-referencing, check the usage.

 #> registry list-buckets help
 Lists the buckets that the current user has access to.
 PRODUCES BACK-REFERENCES

A common scenario for utilizing back-references would be the following:

1. User starts by exploring the available buckets in a registry instance

   #> registry list-buckets
   #   Name           Id                                     Description
   -   ------------   ------------------------------------   -----------
   1   My Bucket      3c7b7467-0012-4d8f-a918-6aa42b6b9d39   (empty)
   2   Other Bucket   175fb557-43a2-4abb-871f-81a354f47bc2   (empty)

2. User then views the flows in one of the buckets using a back-reference to the bucket id from the previous result in position 1

   #> registry list-flows -b &1
   Using a positional back-reference for 'My Bucket'
   #   Name      Id                                     Description
   -   -------   ------------------------------------   ----------------
   1   My Flow   06acb207-d2f1-447f-85ed-9b8672fe6d30   This is my flow.

3. User then views the version of the flow using a back-reference to the flow id from the previous result in position 1

   #> registry list-flow-versions -f &1
   Using a positional back-reference for 'My Flow'
   Ver   Date                         Author                     Message
   ---   --------------------------   ------------------------   -------------------------------------
   1     Tue, Jan 23 2018 09:48 EST   anonymous                  This is the first version of my flow.

4. User deploys version 1 of the flow using back-references to the bucket and flow id from step 2

   #> nifi pg-import -b &1 -f &1 -fv 1
   Using a positional back-reference for 'My Bucket'
   Using a positional back-reference for 'My Flow'
   9bd157d4-0161-1000-b946-c1f9b1832efd

The reason step 4 was able to reference the results from step 2, is because the `list-flow-versions` command in step 3 does not produce back-references, so the results from step 2 are still available.

=== Adding Commands
To add a NiFi command, create a new class that extends `AbstractNiFiCommand`:

[source]
----
public class MyCommand extends AbstractNiFiCommand {

  public MyCommand() {
      super("my-command");
  }

  @Override
  protected void doExecute(NiFiClient client, Properties properties)
          throws NiFiClientException, IOException, MissingOptionException, CommandException {
      // TODO implement
  }

  @Override
  public String getDescription() {
      return "This is my new command";
  }
}
----

Add the new command to `NiFiCommandGroup`:

[source]
----
commands.add(new MyCommand());
----

To add a NiFi Registry command, perform the same steps, but extend from `AbstractNiFiRegistryCommand`, and add the command to `NiFiRegistryCommandGroup`.

[[encrypt_config_tool]]
== Encrypt-Config Tool
The `encrypt-config` command line tool (invoked as `./bin/encrypt-config.sh` or `bin\encrypt-config.bat`) reads from a _nifi.properties_ file with plaintext sensitive configuration values, prompts for a root password or raw hexadecimal key, and encrypts each value. It replaces the plain values with the protected value in the same file, or writes to a new _nifi.properties_ file if specified.

The default encryption algorithm utilized is AES-GCM with 256-bit keys.

=== Usage
To show help:

 ./bin/encrypt-config.sh -h

==== NiFi
The following are available options when targeting NiFi:

* `-h`,`--help`                                   Show usage information (this message)
* `-v`,`--verbose`                                Sets verbose mode (default false)
* `-n`,`--niFiProperties <file>`                  The _nifi.properties_ file containing unprotected config values (will be overwritten unless `-o` is specified)
* `-o`,`--outputNiFiProperties <file>`            The destination _nifi.properties_ file containing protected config values (will not modify input _nifi.properties_)
* `-l`,`--loginIdentityProviders <file>`          The _login-identity-providers.xml_ file containing unprotected config values (will be overwritten unless `-i` is specified)
* `-i`,`--outputLoginIdentityProviders <file>`    The destination _login-identity-providers.xml_ file containing protected config values (will not modify input _login-identity-providers.xml_)
* `-a`,`--authorizers <file>`                     The _authorizers.xml_ file containing unprotected config values (will be overwritten unless `-u` is specified)
* `-u`,`--outputAuthorizers <file>`               The destination _authorizers.xml_ file containing protected config values (will not modify input _authorizers.xml_)
* `-f`,`--flowJson <file>`                        The _flow.json.gz_ file currently protected with old password (will be overwritten unless `-g` is specified)
* `-g`,`--outputFlowJson <file>`                  The destination _flow.json.gz_ file containing protected config values (will not modify input _flow.json.gz_)
* `-b`,`--bootstrapConf <file>`                   The bootstrap.conf file to persist root key and to optionally provide any configuration for the protection scheme.
* `-B`,`--outputBootstrapConf <file>`             The destination _bootstrap.conf_ file to persist root key. If specified, the input _bootstrap.conf_ will not be modified.
* `-S`,`--protectionScheme <protectionScheme>`    Selects the protection scheme for encrypted properties.  Valid values are: [<<AES_GCM>>, <<HASHICORP_VAULT_TRANSIT>>, <<HASHICORP_VAULT_KV>>, <<AWS_KMS>>, <<AWS_SECRETSMANAGER>>, <<AZURE_KEYVAULT_KEY>>, <<AZURE_KEYVAULT_SECRET>>, <<GCP_KMS>>] (default is AES_GCM)
* `-k`,`--key <keyhex>`                           The raw hexadecimal key to use to encrypt the sensitive properties
* `-e`,`--oldKey <keyhex>`                        The old raw hexadecimal key to use during key migration
* `-H`,`--oldProtectionScheme <protectionScheme>` The old protection scheme to use during encryption migration (see --protectionScheme for possible values).  Default is AES_GCM
* `-p`,`--password <password>`                    The password from which to derive the key to use to encrypt the sensitive properties
* `-w`,`--oldPassword <password>`                 The old password from which to derive the key during migration
* `-r`,`--useRawKey`                              If provided, the secure console will prompt for the raw key value in hexadecimal form
* `-m`,`--migrate`                                If provided, the _nifi.properties_ and/or _login-identity-providers.xml_ sensitive properties will be re-encrypted with the new scheme
* `-x`,`--encryptFlowJsonOnly`                    If provided, the properties in _flow.json.gz_ will be re-encrypted with a new key but the _nifi.properties_ and/or _login-identity-providers.xml_ files will not be modified
* `-s`,`--propsKey <password|keyhex>`             The password or key to use to encrypt the sensitive processor properties in _flow.json.gz_
* `-A`,`--newFlowAlgorithm <algorithm>`           The algorithm to use to encrypt the sensitive processor properties in _flow.json.gz_

==== NiFi Registry
The following are available options when targeting NiFi Registry using the `--nifiRegistry` flag:

* `-h`,`--help`                                   Show usage information (this message)
* `-v`,`--verbose`                                Sets verbose mode (default false)
* `-p`,`--password <password>`                    Protect the files using a password-derived key. If an argument is not provided to this flag, interactive mode will be triggered to prompt the user to enter the password.
* `-k`,`--key <keyhex>`                           Protect the files using a raw hexadecimal key. If an argument is not provided to this flag, interactive mode will be triggered to prompt the user to enter the key.
* `-S`,`--protectionScheme <protectionScheme>`    Selects the protection scheme for encrypted properties.  Valid values are: [<<AES_GCM>>, <<HASHICORP_VAULT_TRANSIT>>, <<HASHICORP_VAULT_KV>>, <<AWS_KMS>>, <<AWS_SECRETSMANAGER>>, <<AZURE_KEYVAULT_KEY>>, <<AZURE_KEYVAULT_SECRET>>, <<GCP_KMS>>]  (default is AES_GCM)
* `-w`,`--oldPassword <password>`                 If the input files are already protected using a password-derived key, this specifies the old password so that the files can be unprotected before re-protecting.
* `-e`,`--oldKey <keyhex>`                        If the input files are already protected using a key, this specifies the raw hexadecimal key so that the files can be unprotected before re-protecting.
* `-H`,`--oldProtectionScheme <protectionScheme>` The old protection scheme to use during encryption migration (see --protectionScheme for possible values).  Default is AES_GCM.
* `-b`,`--bootstrapConf <file>`                   The _bootstrap.conf_ file containing no root key or an existing root key, and any other protection scheme configuration properties. If a new password or key is specified (using -p or -k) and no output _bootstrap.conf_ file is specified, then this file will be overwritten to persist the new master key.
* `-B`,`--outputBootstrapConf <file>`             The destination _bootstrap.conf_ file to persist root key. If specified, the input _bootstrap.conf_ will not be modified.
* `-r`,`--nifiRegistryProperties <file>`          The _nifi-registry.properties_ file containing unprotected config values, overwritten if no output file specified.
* `-R`,`--outputNifiRegistryProperties <file>`    The destination _nifi-registry.properties_ file containing protected config values.
* `-a`,`--authorizersXml <file>`                  The _authorizers.xml_ file containing unprotected config values, overwritten if no output file specified.
* `-u`,`--outputAuthorizersXml <file>`            The destination _authorizers.xml_ file containing protected config values.
* `-i`,`--identityProvidersXml <file>`            The _identity-providers.xml_ file containing unprotected config values, overwritten if no output file specified.
* `-I`,`--outputIdentityProvidersXml <file>`      The destination _identity-providers.xml_ file containing protected config values.

=== Protection Schemes
The protection scheme can be selected during encryption using the `--protectionScheme` flag.  During migration, the former protection scheme is specified using the `--oldProtectionScheme` flag.  This distinction allows a set of protected configuration files to be migrated not only to a new key, but to a completely different protection scheme.

==== AES_GCM [[AES_GCM]]
The default protection scheme, `AES-G/CM` simply encrypts sensitive properties and marks their protection as either `aes/gcm/256` or `aes/gcm/256` as appropriate.  This protection is all done within NiFi itself.

==== HASHICORP_VAULT_TRANSIT [[HASHICORP_VAULT_TRANSIT]]
This protection scheme uses https://www.vaultproject.io/docs/secrets/transit[HashiCorp Vault Transit Secrets Engine] to outsource encryption to a configured Vault server. All HashiCorp Vault configuration is stored in the `bootstrap-hashicorp-vault.conf` file, as referenced in the `bootstrap.conf` of a NiFi or NiFi Registry instance.  Therefore, when using the HASHICORP_VAULT_TRANSIT protection scheme, the `nifi(.registry)?.bootstrap.protection.hashicorp.vault.conf` property in the `bootstrap.conf` specified using the `-b` flag must be available to the Encrypt Configuration Tool and must be configured as described in the <<administration-guide.adoc#hashicorp-vault-providers,HashiCorp Vault providers>> section in the link:administration-guide.html[NiFi Administration Guide].

==== HASHICORP_VAULT_KV [[HASHICORP_VAULT_KV]]
This protection scheme uses https://www.vaultproject.io/docs/secrets/kv/kv-v1[HashiCorp Vault Key Value Secrets Engine Version 1] to store sensitive values as Vault Secrets. All HashiCorp Vault configuration is stored in the `bootstrap-hashicorp-vault.conf` file, as referenced in the `bootstrap.conf` of a NiFi or NiFi Registry instance.  Therefore, when using the HASHICORP_VAULT_KV protection scheme, the `nifi(.registry)?.bootstrap.protection.hashicorp.vault.conf` property in the `bootstrap.conf` specified using the `-b` flag must be available to the Encrypt Configuration Tool and must be configured as described in the <<administration-guide.adoc#hashicorp-vault-providers,HashiCorp Vault providers>> section in the link:administration-guide.html[NiFi Administration Guide].

==== AWS_KMS [[AWS_KMS]]
This protection scheme uses https://aws.amazon.com/kms/[AWS Key Management] Service for encryption and decryption. AWS KMS configuration properties can be stored in the `bootstrap-aws.conf` file, as referenced in the `bootstrap.conf` of NiFi or NiFi Registry. If the configuration properties are not specified in `bootstrap-aws.conf`, then the provider will attempt to use the AWS default credentials provider, which checks standard environment variables and system properties.  Therefore, when using the AWS_KMS protection scheme, the `nifi(.registry)?.bootstrap.protection.aws.conf` property in the `bootstrap.conf` specified using the `-b` flag must be available to the Encrypt Configuration Tool and must be configured as described in the <<administration-guide.adoc#aws-kms-provider, AWS KMS provider>> section in the link:administration-guide.html[NiFi Administration Guide].

==== AWS_SECRETSMANAGER [[AWS_SECRETSMANAGER]]
This protection scheme uses https://aws.amazon.com/secrets-manager/[AWS Secrets Manager] Service to store sensitive values as AWS Secrets. AWS Secrets Manager configuration properties can be stored in the `bootstrap-aws.conf` file, as referenced in the `bootstrap.conf` of NiFi or NiFi Registry. If the configuration properties are not specified in `bootstrap-aws.conf`, then the provider will attempt to use the AWS default credentials provider, which checks standard environment variables and system properties.  Therefore, when using the AWS_SECRETS_MANAGER protection scheme, the `nifi(.registry)?.bootstrap.protection.aws.conf` property in the `bootstrap.conf` specified using the `-b` flag must be available to the Encrypt Configuration Tool and must be configured as described in the <<administration-guide.adoc#aws-secrets-manager-provider, AWS Secrets Manager provider>> section in the link:administration-guide.html[NiFi Administration Guide].

==== Microsoft Azure Key Vault Sensitive Property Providers

Azure Key Vault configuration properties can be stored in the `bootstrap-azure.conf` file, as referenced in the
`bootstrap.conf` of NiFi or NiFi Registry.

Azure Key Vault providers will use the
https://docs.microsoft.com/en-us/java/api/com.azure.identity.defaultazurecredential[DefaultAzureCredential]
for authentication.
The https://docs.microsoft.com/en-us/java/api/overview/azure/identity-readme#key-concepts[Azure Identity] client library
describes the process for credentials resolution, which leverages environment variables, system properties, and falls
back to
https://docs.microsoft.com/en-us/java/api/overview/azure/identity-readme#managed-identity-support[Managed Identity]
authentication.

When using Azure Key Vault providers, `bootstrap.conf` must contain the
`nifi.bootstrap.protection.azure.keyvault.conf` property. The `bootstrap.conf` file location must be specified using the
`-b` argument when running the Encrypt Config Tool.

===== AZURE_KEYVAULT_KEY [[AZURE_KEYVAULT_KEY]]

This protection scheme uses keys managed by
https://docs.microsoft.com/en-us/azure/key-vault/keys/about-keys[Azure Key Vault Keys] for encryption and decryption.

See <<administration-guide.adoc#azure-key-vault-key-provider,Azure Key Vault Key Provider>> in the NiFi System
Administrator's Guide for required properties.

===== AZURE_KEYVAULT_SECRET [[AZURE_KEYVAULT_SECRET]]

This protection scheme uses secrets managed by
https://docs.microsoft.com/en-us/azure/key-vault/secrets/about-secrets[Azure Key Vault Secrets] for storing and
retrieving sensitive property values.

See <<administration-guide.adoc#azure-key-vault-secret-provider,Azure Key Vault Secret Provider>> in the NiFi System
Administrator's Guide for required properties.

==== GCP_KMS [[GCP_KMS]]
This protection scheme uses Google Cloud Key Management Service (https://cloud.google.com/security-key-management[Google Cloud Key Management Service]) for encryption and decryption. Google Cloud KMS configuration properties are to be stored in the `bootstrap-gcp.conf` file, as referenced in the `bootstrap.conf` of NiFi or NiFi Registry. Credentials must be configured as per the following documentation: https://cloud.google.com/kms/docs/reference/libraries[Google Cloud KMS documentation]. Therefore, when using the GCP_KMS protection scheme, the `nifi(.registry)?.bootstrap.protection.gcp.kms.conf` property in the `bootstrap.conf` specified using the `-b` flag must be available to the Encrypt Configuration Tool and must be configured as described in the <<administration-guide.adoc#_google_cloud_kms_provider,Google Cloud KMS provider>> section in the link:administration-guide.html[NiFi Administration Guide].

=== Examples

==== NiFi
As an example of how the tool works, assume that you have installed the tool on a machine supporting 256-bit encryption and with the following existing values in the _nifi.properties_ file:

[source]
----
# security properties #
nifi.sensitive.props.key=thisIsABadSensitiveKeyPassword
nifi.sensitive.props.algorithm=NIFI_PBKDF2_AES_GCM_256
nifi.sensitive.props.additional.keys=

nifi.security.keystore=/path/to/keystore.jks
nifi.security.keystoreType=JKS
nifi.security.keystorePasswd=thisIsABadKeystorePassword
nifi.security.keyPasswd=thisIsABadKeyPassword
nifi.security.truststore=
nifi.security.truststoreType=
nifi.security.truststorePasswd=
----

Enter the following arguments when using the tool:

----
encrypt-config.sh \
-b bootstrap.conf \
-k 0123456789ABCDEFFEDCBA98765432100123456789ABCDEFFEDCBA9876543210 \
-n nifi.properties
----

As a result, the _nifi.properties_ file is overwritten with protected properties and sibling encryption identifiers (`aes/gcm/256`, the currently supported algorithm):

[source]
----
# security properties #
nifi.sensitive.props.key=n2z+tTTbHuZ4V4V2||uWhdasyDXD4ZG2lMAes/vqh6u4vaz4xgL4aEbF4Y/dXevqk3ulRcOwf1vc4RDQ==
nifi.sensitive.props.key.protected=aes/gcm/256
nifi.sensitive.props.algorithm=NIFI_PBKDF2_AES_GCM_256
nifi.sensitive.props.additional.keys=

nifi.security.keystore=/path/to/keystore.jks
nifi.security.keystoreType=JKS
nifi.security.keystorePasswd=oBjT92hIGRElIGOh||MZ6uYuWNBrOA6usq/Jt3DaD2e4otNirZDytac/w/KFe0HOkrJR03vcbo
nifi.security.keystorePasswd.protected=aes/gcm/256
nifi.security.keyPasswd=ac/BaE35SL/esLiJ||+ULRvRLYdIDA2VqpE0eQXDEMjaLBMG2kbKOdOwBk/hGebDKlVg==
nifi.security.keyPasswd.protected=aes/gcm/256
nifi.security.truststore=
nifi.security.truststoreType=
nifi.security.truststorePasswd=
----

Additionally, the _bootstrap.conf_ file is updated with the encryption key as follows:

[source]
----
# Root key in hexadecimal format for encrypted sensitive configuration values
nifi.bootstrap.sensitive.key=0123456789ABCDEFFEDCBA98765432100123456789ABCDEFFEDCBA9876543210
----

Sensitive configuration values are encrypted by the tool by default, however you can encrypt any additional properties, if desired.  To encrypt additional properties, specify them as comma-separated values in the `nifi.sensitive.props.additional.keys` property.

If the _nifi.properties_ file already has valid protected values, those property values are not modified by the tool.

When applied to _login-identity-providers.xml_ and _authorizers.xml_, the property elements are updated with an `encryption` attribute:

Example of protected _login-identity-providers.xml_:

[source]
----
   <!-- LDAP Provider -->
   <provider>
       <identifier>ldap-provider</identifier>
       <class>org.apache.nifi.ldap.LdapProvider</class>
       <property name="Authentication Strategy">START_TLS</property>
       <property name="Manager DN">someuser</property>
       <property name="Manager Password" encryption="aes/gcm/128">q4r7WIgN0MaxdAKM||SGgdCTPGSFEcuH4RraMYEdeyVbOx93abdWTVSWvh1w+klA</property>
       <property name="TLS - Keystore"></property>
       <property name="TLS - Keystore Password" encryption="aes/gcm/128">Uah59TWX+Ru5GY5p||B44RT/LJtC08QWA5ehQf01JxIpf0qSJUzug25UwkF5a50g</property>
       <property name="TLS - Keystore Type"></property>
       ...
   </provider>
----

Example of protected _authorizers.xml_:

[source]
----
   <!-- LDAP User Group Provider -->
   <userGroupProvider>
       <identifier>ldap-user-group-provider</identifier>
       <class>org.apache.nifi.ldap.tenants.LdapUserGroupProvider</class>
       <property name="Authentication Strategy">START_TLS</property>
       <property name="Manager DN">someuser</property>
       <property name="Manager Password" encryption="aes/gcm/128">q4r7WIgN0MaxdAKM||SGgdCTPGSFEcuH4RraMYEdeyVbOx93abdWTVSWvh1w+klA</property>
       <property name="TLS - Keystore"></property>
       <property name="TLS - Keystore Password" encryption="aes/gcm/128">Uah59TWX+Ru5GY5p||B44RT/LJtC08QWA5ehQf01JxIpf0qSJUzug25UwkF5a50g</property>
       <property name="TLS - Keystore Type"></property>
       ...
   </userGroupProvider>
----

==== NiFi Registry
As an example of how the tool works, assume that you have installed the tool on a machine supporting 256-bit encryption and with the following existing values in the _nifi-registry.properties_ file:

----
# security properties #
nifi.registry.security.keystore=/path/to/keystore.jks
nifi.registry.security.keystoreType=JKS
nifi.registry.security.keystorePasswd=thisIsABadKeystorePassword
nifi.registry.security.keyPasswd=thisIsABadKeyPassword
nifi.registry.security.truststore=
nifi.registry.security.truststoreType=
nifi.registry.security.truststorePasswd=
----

Enter the following arguments when using the tool:

----
./bin/encrypt-config.sh --nifiRegistry \
-b bootstrap.conf \
-k 0123456789ABCDEFFEDCBA98765432100123456789ABCDEFFEDCBA9876543210 \
-r nifi-registry.properties
----

As a result, the _nifi-registry.properties_ file is overwritten with protected properties and sibling encryption identifiers (`aes/gcm/256`, the currently supported algorithm):

----
# security properties #
nifi.registry.security.keystore=/path/to/keystore.jks
nifi.registry.security.keystoreType=JKS
nifi.registry.security.keystorePasswd=oBjT92hIGRElIGOh||MZ6uYuWNBrOA6usq/Jt3DaD2e4otNirZDytac/w/KFe0HOkrJR03vcbo
nifi.registry.security.keystorePasswd.protected=aes/gcm/256
nifi.registry.security.keyPasswd=ac/BaE35SL/esLiJ||+ULRvRLYdIDA2VqpE0eQXDEMjaLBMG2kbKOdOwBk/hGebDKlVg==
nifi.registry.security.keyPasswd.protected=aes/gcm/256
nifi.registry.security.truststore=
nifi.registry.security.truststoreType=
nifi.registry.security.truststorePasswd=
----

When applied to _identity-providers.xml_ or _authorizers.xml_, the property elements are updated with an `encryption` attribute. For example:

----
<!-- LDAP Provider -->
<provider>
   <identifier>ldap-provider</identifier>
   <class>org.apache.nifi.registry.security.ldap.LdapProvider</class>
   <property name="Authentication Strategy">START_TLS</property>
   <property name="Manager DN">someuser</property>
   <property name="Manager Password" encryption="aes/gcm/128">q4r7WIgN0MaxdAKM||SGgdCTPGSFEcuH4RraMYEdeyVbOx93abdWTVSWvh1w+klA</property>
   <property name="TLS - Keystore">/path/to/keystore.jks</property>
   <property name="TLS - Keystore Password" encryption="aes/gcm/128">Uah59TWX+Ru5GY5p||B44RT/LJtC08QWA5ehQf01JxIpf0qSJUzug25UwkF5a50g</property>
   <property name="TLS - Keystore Type">JKS</property>
   ...
</provider>
----

Additionally, the _bootstrap.conf_ file is updated with the encryption key as follows:

----
# Root key in hexadecimal format for encrypted sensitive configuration values
nifi.registry.bootstrap.sensitive.key=0123456789ABCDEFFEDCBA98765432100123456789ABCDEFFEDCBA9876543210
----

Sensitive configuration values are encrypted by the tool by default, however you can encrypt any additional properties, if desired.
To encrypt additional properties, specify them as comma-separated values in the `nifi.registry.sensitive.props.additional.keys` property.


If the _nifi-registry.properties_ file already has valid protected values and you wish to protect additional values using the
same root key already present in your _bootstrap.conf_, then run the tool without specifying a new key:

----
# bootstrap.conf already contains root key property
# nifi-registy.properties has been updated for nifi.registry.sensitive.props.additional.keys=...

./bin/encrypt-config.sh --nifiRegistry -b bootstrap.conf -r nifi-registry.properties
----

[sensitive_property_key_migration]
=== Sensitive Property Key Migration

In order to change the key used to encrypt the sensitive values, provide the new key or password using the `-k` or `-p` flags as usual,
and provide the existing key or password using `--old-key` or `--old-password` respectively. This will allow the toolkit to decrypt the
existing values and re-encrypt them, and update _bootstrap.conf_ with the new key. Only one of the key or password needs to be specified
for each phase (old vs. new), and any combination is sufficient:

* old key -> new key
* old key -> new password
* old password -> new key
* old password -> new password

In order to change the protection scheme (e.g., migrating from AES encryption to Vault encryption), specify the `--protectionScheme`
and `--oldProtectionScheme` in the migration command.

The following is an example of the commands for protection scheme migration from AES_GCM to AWS_KMS then back. Execute these commands at the `nifi` directory with the `nifi-toolkit` directory as a sibling directory. In addition, make sure to update `bootstrap-aws.conf` with your AWS KMS Key ARN/ID and have your credentials and region configured.


This command encrypts nifi.properties with the AES_GCM protection scheme
----
./../nifi-toolkit-*-SNAPSHOT/bin/encrypt-config.sh \
-b conf/bootstrap.conf \
-n conf/nifi.properties \
-k 0123456789ABCDEFFEDCBA98765432100123456789ABCDEFFEDCBA9876543210 \
-v
----
This command migrates nifi.properties from using AES_GCM to using AWS_KMS protection scheme
----
./../nifi-toolkit-*-SNAPSHOT/bin/encrypt-config.sh \
-b conf/bootstrap.conf \
-n conf/nifi.properties \
-S AWS_KMS \
-H AES_GCM \
-e 0123456789ABCDEFFEDCBA98765432100123456789ABCDEFFEDCBA9876543210 \
-m \
-v
----
This command migrates nifi.properties back from AWS_KMS to AES_GCM protection scheme
----
./../nifi-toolkit-*-SNAPSHOT/bin/encrypt-config.sh \
-b conf/bootstrap.conf \
-n conf/nifi.properties \
-S AES_GCM \
-k 0123456789ABCDEFFEDCBA98765432100123456789ABCDEFFEDCBA9876543210 \
-H AWS_KMS \
-m \
-v
----

[export_import_all_flows]
=== Export All Flows
You can use the `export-all-flows` to perform the following tasks:

* List all the buckets
* For each bucket, list all flows
* For each flow, list all versions
* Export each version into a provided directory

Running the command requires an `--outputDirectory` parameter. The directory must exist and permissions correctly set.

=== Import All Flows
You can use the `import-all-flows` to perform the following tasks:

* List all files, representing a flow version, from a directory created by export-all-flows
* Create all the corresponding buckets
* Create all the corresponding flows
* Import all the corresponding flow versions

Running the command requires 2 parameters:

* `--input` parameter represents a directory to read files from
* `--skipExisting` optional parameter, configuring how to handle existing flow and flow version creation.
If provided the flow and flow version creation will be skipped regardless of there are missing flow versions.
If not provided the missing flow versions will be created.

=== Usage
The input source for an import-all-flows command must be created by an export-all-flows command.
To avoid migration conflicts, no modification should be performed in the NiFi Registry during this activity.
Buckets and flows with the same name are considered equal.

* Export all flow versions:

 ./bin/cli.sh registry export-all-flows -u http://localhost:18080 --outputDirectory "/my_dir/flow_exports"

* Import all flow versions:

 ./bin/cli.sh registry import-all-flows -u http://localhost:18080 --input "/my_dir/flow_exports" --skipExisting

=== Expected behaviour
=== Use case 1: reconfiguring an existing NiFi Registry

NiFi is connecting to NiFi Registry, the NiFi Registry does not change, only its configuration.
All the data will be created.

1. Export versions:

 ./bin/cli.sh registry export-all-flows -u http://localhost:18080 --outputDirectory "/my_dir/flow_exports"

2. Stop registry

3. Switch provider

4. Start registry

5. Import versions

 ./bin/cli.sh registry import-all-flows -u http://localhost:18080 --input "/my_dir/flow_exports" --skipExisting


=== Use case 2: data replication

NiFi_1 is connecting to NiFi Registry_1 and NiFi_2 is connecting to NiFi Registry_2.

For disaster recovery purposes the data from NiFi Registry_1 needs to be periodically replicated to NiFi Registry_2 via a scheduled job.

The initial version of Nifi Registry_2 needs to be created by this tool.

The missing buckets, flows and versions will be created. If bucket and flow exist the missing versions will be created.

1. Export versions:

 ./bin/cli.sh registry export-all-flows -u http://nifi-registry-1:18080 --outputDirectory "/my_dir/flow_exports"

2. Import versions:

 ./bin/cli.sh registry import-all-flows -u http://nifi-registry-2:18080 --input "/my_dir/flow_exports"
