Skip to main content

NoETL Docker Usage Guide

This guide provides detailed instructions for using NoETL with Docker.

Overview​

NoETL can be run in Docker containers, which provides several benefits:

  • Consistent environment across different platforms
  • Easy deployment
  • Isolation from the host system
  • Simplified dependency management

Prerequisites​

  • Docker installed on your system
  • Docker Compose (optional, for multi-container setups)

Quick Start​

Using the Official Docker Image​

The simplest way to run NoETL in Docker is to use the official Docker image:

# Pull the latest NoETL image
docker pull noetl/noetl:latest

# Run the NoETL server
docker run -p 8082:8082 noetl/noetl:latest

This will start the NoETL server and expose it on port 8082 of your host machine.

Building and Running from Source​

If you want to build the Docker image from source:

  1. Clone the repository:

    git clone https://github.com/noetl/noetl.git
    cd noetl

  2. Build the Docker image:

    docker build -t noetl:local .

  3. Run the NoETL server:

    docker run -p 8082:8082 noetl:local

Using Docker Compose​

NoETL provides a Docker Compose configuration that sets up a complete environment with PostgreSQL:

  1. Start the containers:

    docker-compose up

  2. Or build and start the containers:

    docker-compose up --build

  3. To run in detached mode:

    docker-compose up -d

  4. To stop the containers:

    docker-compose down

Using the Makefile​

NoETL provides a Makefile with convenient commands for Docker operations:

# Build the Docker containers
make build

# Start the Docker containers
make up

# Stop the Docker containers
make down

# View logs
make logs

# Run tests in Docker
make test

Accessing the NoETL Server​

Once the NoETL server is running in Docker, you can access it at:

  • Web UI: http://localhost:8082
  • API: http://localhost:8082/api

Running Playbooks in Docker​

Using the CLI​

You can execute NoETL commands inside the Docker container:

# Execute a playbooks directly
docker exec -it noetl noetl agent -f /path/to/playbooks.yaml

# Register a playbooks in the catalog
docker exec -it noetl noetl playbooks --register /path/to/playbooks.yaml

# Execute a playbooks from the catalog
docker exec -it noetl noetl playbooks --execute --path "workflows/example/playbook"

Using the API​

You can also use the NoETL API to execute playbooks:

# Register a playbooks
curl -X POST "http://localhost:8082/catalog/register" \
  -H "Content-Type: application/json" \
  -d '{
    "content_base64": "'"$(base64 -i ./path/to/playbooks.yaml)"'"
  }'

# Execute a playbooks
curl -X POST "http://localhost:8082/playbook/execute" \
  -H "Content-Type: application/json" \
  -d '{
    "path": "workflows/example/playbooks",
    "version": "1.0.0",
    "input_payload": {
      "param1": "value1",
      "param2": "value2"
    }
  }'

Mounting Volumes​

You can mount volumes to persist data and share files with the Docker container:

docker run -p 8082:8082 \
  -v $(pwd)/playbooks:/app/playbooks \
  -v $(pwd)/data:/app/data \
  noetl/noetl:latest

This mounts the local playbooks and data directories to the corresponding directories in the Docker container.

Environment Variables​

You can pass environment variables to the Docker container:

docker run -p 8082:8082 \
  -e POSTGRES_HOST=postgres \
  -e POSTGRES_PORT=5432 \
  -e POSTGRES_USER=noetl \
  -e POSTGRES_PASSWORD=noetl \
  -e POSTGRES_DB=noetl \
  noetl/noetl:latest

Docker Compose Configuration​

The default docker-compose.yaml file includes the following services:

  • noetl: The NoETL server
  • postgres: PostgreSQL database for storing playbook data

You can customize the Docker Compose configuration by editing the docker-compose.yaml file.

Customizing the Docker Image​

If you need to customize the Docker image, you can create your own Dockerfile based on the official NoETL image:

FROM noetl/noetl:latest

# Install additional dependencies
RUN pip install some-package

# Add custom files
COPY ./custom_playbooks /app/playbooks

# Set environment variables
ENV SOME_VARIABLE=some_value

# Set the working directory
WORKDIR /app

# Set the entrypoint
ENTRYPOINT ["noetl", "server"]

Troubleshooting​

Container Fails to Start​

If the container fails to start, check the logs:

docker logs noetl

Cannot Connect to the Server​

If you cannot connect to the NoETL server, check that the port is correctly mapped:

docker ps

Make sure the container is running and the port mapping is correct (e.g., 0.0.0.0:8082->8082/tcp).

Database Connection Issues​

If NoETL cannot connect to the PostgreSQL database, check the environment variables and network configuration:

# Check the network
docker network ls
docker network inspect noetl_default

# Check the PostgreSQL container
docker logs postgres

Next Steps​