Client

The qalita platform client is used to communicate with the qalita platform. It is also used to run qalita agents. qalita agent run these agents are programs that run packs on sources. They can be likened to workers.

Knowing how the QALITA Platform client works is essential for using the platform. As a data engineer, it will be your responsibility to implement qalita agents on the machines hosting the data sources.

tip

For the most up-to-date command line documentation: Go directly to PyPi.org

Using the command line

QALITA's Command Line Interface (CLI) is a tool designed for use by data engineers configuring agents, sources and assets on the QALITA platform.

It provides easy-to-use commands to help them set up a QALITA platform environment in no time.

• Client
  • Using the command line
  • Quick start
    • Installation
    • Usage
    • Configuration
      • Minimum configuration
      • Connected configuration
      • Create an .env file and export ENV values:
  • qalita agent
    • qalita agent login
    • qalita agent run
      • Job
      • Worker
    • qalita agent joblist
    • qalita agent info
  • qalita pack
    • qalita pack init
    • List of qalita packs
    • qalita pack run
    • qalita pack validate
    • qalita pack push
  • qalita source
    • qalita source add
    • qalita source validate
    • qalita source list
    • qalita source push

Quick start

Installation

As easy as :

pip install qalita

or there is a Docker Image

Usage

For more detailed, context-sensitive help, type

qalita COMMAND -h

Usage: qalita [OPTIONS] COMMAND [ARGS]...



  QALITA command line interface

Configuration

This CLI command communicates with the QALITA platform API.

There are several configuration levels to suit your needs:

Minimum configuration

QALITA_AGENT_NAME="agent_name

The agent will help you identify it in the front-end interface; there are no restrictions on the name.

QALITA_AGENT_MODE=job/worker

Agent mode:

Job: In job mode, when you use the qalita agent run command, it will immediately try to run a job on the current local context.

Worker: In worker mode, when you use the qalita agent run command, it will wait for the backend to give it jobs to run. It's a bit like a scheduler.

Note that the qalita agent run command needs more configuration to work properly, otherwise it will display an error.

Connected configuration

• QALITA_AGENT_ENDPOINT=backend_api_url

Example: http://localhost:3080/api/v1

The agent's url endpoint enables the agent to communicate with the qalita platform's endpoints:

• Listing packs

• Execute jobs

• Publish sources

• Publish packs

• QALITA_AGENT_TOKEN=api_token

The token is provided during the quick-start steps in the front-end application. It is associated with your user and role.

Note that you must have at least the [Data Engineer] role to use the QALITA CLI.

Create an .env file and export ENV values:

You can also create an .env file and export the values to your environment.

.env-local bash QALITA_AGENT_NAME=agent_name``. QALITA_AGENT_MODE=job/worker QALITA_AGENT_ENDPOINT=https://api.company.com/api/v1 QALITA_AGENT_TOKEN=api_token QALITA_PACK_NAME=package_name`

Then export the file values to ENV values with :



```bash
export $(xargs ` .env-local)

qalita agent

The qalita agent command lets you :

• register an agent on the platform
• obtain information about your local agent
• Run a package on a source
• list agent jobs (past and future)

qalita agent login

Parameters:

name: agent name mode : agent mode job/worker. token : the api token you get from the platform url : the url of the platform's backend api.

qalita agent login registers your local agent on the platform, allowing you to run jobs, or create routines (schedules) to run packages programmatically.

You must have configured your agent with :

• QALITA_AGENT_ENDPOINT=backend_api_url
• QALITA_AGENT_TOKEN=api_token.

You can get your token from the frontend or with a call to the OAUTH2 API to the backend endpoint /users/signin.

More information in your frontend documentation, and in the Connected config doc.

qalita agent run

Parameters :

--name : agent name --mode : agent mode job/worker. --token : the api token you get from the platform --url: the url of the platform's backend api.

Specific parameters in job mode:

--source: the source identifier against which you want to run your job --source-version (optional): the version of the source, by default it will be run on the latest version of Soruce. --pack : the pack identifier on which you want to run your job. --pack-version (optional): the version of the pack, by default it will run on the latest version of the pack.

qalita agent run

runs in a different mode:

Job

The agent will run a given configuration

• -p : a pack_id given with the qalita pack list, note that your pack must be pushed on the platform to have an id.
• -s: a source_id given with the qalita source list, note that your source must be pushed to the platform in order to have an id.

Worker

The agent will wait until it receives a command from the frontend, then work in the same way as in job mode.

Note that this mode will work indefinitely.

qalita agent joblist

Parameters:

--name: agent name --mode: agent mode job/worker. --token : the api token you get from the platform --url : the url of the platform's backend api

List jobs from the platform backend.

qalita agent info

Parameters :

--name: agent name --mode : agent mode job/worker. --token : the api token you get from the platform. --url: the url of the platform's backend api.

Get information on configuring your local agent.

qalita pack

The qalita pack command lets you :

• Initialize a new pack
• List all available packs
• Validate the pack
• Run a local pack
• push the version of your pack to the platform

qalita pack init

Parameters:

• --name: the name of the pack

Initializes a new pack, you must have defined a name, this will create a new folder with the name of the pack.

You can define your name by passing a new parameter to the command line or by defining a new environment variable: QALITA_PACK_NAME=my-super-pack``.

This is the tree structure created:

./pack-name_pack/ /run.sh # Entry point file to be executed with the qalita run agent /README.md # Documentation file /properties.yaml # Properties file containing pack properties /main.py # (pack-specific) The main script (you can run your pack in the language of your choice) /config.json # (pack-specific) Your pack's configuration file, which you can use to define any configuration you like. /requirements.txt # (pack-specific) The requirements file that is executed in run.sh

List of qalita packs

Parameters :

• You must be logged in with qalita agent login

Lists all packs accessible with the Qalita platform.

qalita pack run

Parameters:

--name : Pack name

Runs your locally configured pack

qalita pack validate

Parameters:

--name: Pack name

Validate your locally configured pack

qalita pack push

Parameters: --name: Pack name

--name: Pack name

Push your locally configured pack

qalita source

The qalita source command lets you:

• Add a new source to your local configuration List your local sources from your qalita-conf.yml file * Push your local sources from your qalita-conf.yml file * Add a new source to your local configuration Push your local sources from your qalita-conf.yml file * Validate your conf. file.
• Validate your configuration file qalita-conf.yml.

Note, by default the qalita-conf.yml file is stored in ~/.qalita/qalita-conf.yml, set QALITA_HOME env to customize the default path.

qalita source add

This function will help you add a new source to your qalita-conf.yaml configuration file.

This command has no parameters, so you need to follow the command prompts.

• Invite 1 : Source name
• Prompt 2 : Source type
• Prompt 3 : Is conditional, depends on source type.
  • Case : Source Type = file : Source path.
  • Case : Source Type = database : host / port / username / password / database
• Prompt 4 : Source description
• Prompt 5: Is source a reference (bool) (default: false) Prompt 6: Is the source sensitive? (bool) (default: false)
• Prompt 7: Source visibility (private, internal, public) (default: private)

At the end of the prompt, the client will check the accessibility of the source according to the configuration and type. This step is called validate_source.

to complete the process of registering a new source on the platform, you need to push your source with the command: qalita source push.

qalita source validate

Help function to help you add a new source to your configuration file qalita-conf.yaml.

qalita source list

Parameters :

You must have a qalita-conf.yaml file containing the configuration of your sources.

Example:

version : 1
sources :
- config :
    path: /home/user/data_dir
  description : Folder containing csv files
  name: my_csv_files
  owner: user
  reference: false
  visibility: private
  sensitive: false
  type : file

In this example, we have :

General keys

Key	Type	Description
version	int	The configuration version
sources	list	List of sources.

Source keys

| Source key: key, type, description, etc. | | --------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------- | -------------------------------------------- | | Source name - Character string - Source name - Source description - Character string - Source description - Source description - Source description | | description | string | Source description | | owner | string | The owner of the source | | type | string | Source type | config | dict | | config | dict | Source configuration | | visibility | string | Source visibility private/internal/public. | | bool | Is the source a reference source? | | sensitive | bool | Does the source contain sensitive data? |

qalita source push

Registers your sources on the platform

Note: If you want to run a pack on your source, you'll first need to push your source onto the platform. This will give you a source ID with which you can run your pack.