> ## Documentation Index
> Fetch the complete documentation index at: https://docs.lancedb.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Dependency Verification

> Diagnose and resolve package version mismatches between local and Ray worker environments.

export const PyCondaClusterInline = "from geneva.cluster.builder import KubeRayClusterBuilder\n\ncluster = (\n    KubeRayClusterBuilder.create(\"my-cluster\")\n    .ray_init_kwargs({\n        \"runtime_env\": {\n            \"conda\": {\n                \"channels\": [\"conda-forge\"],\n                \"dependencies\": [\n                    \"python=3.10\",\n                    \"ffmpeg<8\",\n                    \"torchvision=0.22.1\",\n                ],\n            },\n            \"config\": {\"eager_install\": True},\n        }\n    })\n    .build()\n)\n";

export const PyCondaClusterPath = "from geneva.cluster.builder import KubeRayClusterBuilder\n\ncluster = (\n    KubeRayClusterBuilder.create(\"my-cluster\")\n    .ray_init_kwargs({\n        \"runtime_env\": {\"conda\": \"environment.yml\"}\n    })\n    .build()\n)\n";

export const PyPipManifest = "import geneva\nfrom geneva.manifest.builder import PipManifestBuilder\n\nmanifest = (\n    PipManifestBuilder.create(\"my-manifest\")\n    .pip([\n        \"numpy==1.26.4\",\n        \"torch==2.0.1\",\n        \"attrs==23.2.0\",\n    ])\n    .build()\n)\n\nconn = geneva.connect(\"s3://my-bucket/my-db\")\nconn.define_manifest(\"my-manifest\", manifest)\nwith conn.context(cluster=\"my-cluster\", manifest=\"my-manifest\"):\n    conn.open_table(\"my-table\").backfill(\"my-column\")\n";

export const PyEnvVarsViaCluster = "from geneva.cluster.builder import KubeRayClusterBuilder\nimport os\n\ncluster = (\n    KubeRayClusterBuilder.create(\"my-cluster\")\n    .ray_init_kwargs({\n        \"runtime_env\": {\n            \"env_vars\": {\n                \"AWS_ACCESS_KEY_ID\": os.environ[\"AWS_ACCESS_KEY_ID\"],\n                \"AWS_SECRET_ACCESS_KEY\": os.environ[\"AWS_SECRET_ACCESS_KEY\"],\n            }\n        }\n    })\n    .build()\n)\n";

export const PyQuickFixManifest = "from geneva.manifest.builder import PipManifestBuilder\n\nmanifest = PipManifestBuilder.create(\"fix\").pip([\"numpy==1.26.4\"]).build()\n";

When running Geneva UDFs on Ray, your code is serialized locally and executed on remote workers. If the worker environment differs from your local environment, you may encounter subtle and difficult-to-debug errors.

## Example environment mismatch errors

| Symptom                                                            | Likely Cause                              |
| ------------------------------------------------------------------ | ----------------------------------------- |
| `TypeError: Enum.__new__() missing 1 required positional argument` | `attrs` version mismatch                  |
| `TypeError: Can't instantiate abstract class`                      | Package structure differences             |
| `ArrowInvalid: cannot cast` / serialization errors                 | NumPy 1.x vs 2.x mismatch                 |
| `ModuleNotFoundError` on workers                                   | Package only installed locally            |
| Model loading failures                                             | PyTorch version mismatch                  |
| Permission denied errors                                           | Missing API keys in envrionment variables |

These issues are notoriously difficult to debug because the error messages often don't indicate the root cause.

## The `compare_ray_environments` Tool

Geneva provides a diagnostic tool to compare your local environment against Ray workers.

If you are encountering a hang or exception you can use the following diagnosis worklflow to resolve the problem.

<Steps>
  <Step>
    **Run the diagnostic tool** programatically or via the CLI.
  </Step>

  <Step>
    **Check PACKAGES and ENV VARS output sections for mismatches**.
  </Step>

  <Step>
    **Identify critical packages**: numpy, torch, pyarrow, attrs, pydantic.
  </Step>

  <Step>
    **Identify inconsistent environment variables**: `AWS_*`,  `GOOGLE_APPLICATION_CREDENTIALS`
  </Step>

  <Step>
    **Fix with manifest** for quick testing:

    <CodeGroup>
      <CodeBlock filename="Python" language="python" icon="python">
        {PyQuickFixManifest}
      </CodeBlock>
    </CodeGroup>
  </Step>

  <Step>
    **OPTIONAL: Build custom image** for production (if using KubeRay).
  </Step>
</Steps>

### Programmatic Usage

<CodeGroup>
  ```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
  from geneva.runners.ray.compare_env import compare_ray_environments

  # Compare and print (requires Geneva context to be initialize via `with db.context(..)`)
  result = compare_ray_environments()

  # Compare environments, filtering environment variables with specified prefix
  result = compare_ray_environments(env_prefix="PY")
  ```
</CodeGroup>

### CLI Usage

<CodeGroup>
  ```bash CLI icon="terminal" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
  # Connect to existing Ray cluster
  python -m geneva.runners.ray.compare_env

  # Start new local Ray cluster
  python -m geneva.runners.ray.compare_env --address local

  # Filter env vars by prefix
  python -m geneva.runners.ray.compare_env --env-prefix RAY

  # Show full JSON snapshots
  python -m geneva.runners.ray.compare_env --show-all

  # Skip sys.path comparison
  python -m geneva.runners.ray.compare_env --no-sys-path
  ```
</CodeGroup>

## Understanding the Output

The tool outputs several sections to help you identify mismatches.

### PYTHON / PLATFORM

Shows Python version and OS information for both environments:

```
=== PYTHON / PLATFORM ===
Local:
  Python: 3.11.9 (main, Apr 19 2024, 16:48:06) [GCC 11.2.0]
  Impl  : CPython
  Exec  : /home/user/.venv/bin/python
  OS    : Linux 5.15.0-generic (x86_64)

Remote:
  Python: 3.11.9 (main, Apr 19 2024, 16:48:06) [GCC 11.2.0]
  Impl  : CPython
  Exec  : /home/ray/anaconda3/bin/python
  OS    : Linux 5.4.0-aws (x86_64)
```

<Warning>
  Watch for different Python versions or different OS types (macOS local vs Linux remote).
</Warning>

#### Architecture Mismatch (macOS to Linux)

If you see different OS types (e.g., `Darwin` locally vs `Linux` remotely), compiled extensions may fail with `ModuleNotFoundError` or segfaults.

**Solution**: Run Geneva from the same OS/architecture as your cluster (typically Linux x86\_64). Use a Linux VM, container, or remote development environment.

### Environment Variables

Environment variables present in only one environment:

```
=== ENV VARS: keys only in LOCAL ===
  + CONDA_PREFIX
  + VIRTUAL_ENV

=== ENV VARS: keys only in REMOTE ===
  + RAY_ADDRESS
  + KUBERNETES_SERVICE_HOST
```

<Warning>
  Missing `AWS_*` or `GOOGLE_APPLICATION_CREDENTIALS` can cause storage authentication failures.
</Warning>

#### Passing Environment Variables to Workers

If critical environment variables are missing on workers, you can pass them via the manifest or cluster configuration.

**Option 1: Via Manifest**

<CodeGroup>
  ```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
  from geneva.manifest.builder import PipManifestBuilder
  import os

  manifest = (
      PipManifestBuilder.create("my-manifest")
      .env_vars({
          "AWS_ACCESS_KEY_ID": os.environ["AWS_ACCESS_KEY_ID"],
          "AWS_SECRET_ACCESS_KEY": os.environ["AWS_SECRET_ACCESS_KEY"],
          "MY_API_KEY": os.environ["MY_API_KEY"],
      })
      .build()
  )
  ```
</CodeGroup>

**Option 2: Via Cluster Configuration**

<CodeGroup>
  <CodeBlock filename="Python" language="python" icon="python">
    {PyEnvVarsViaCluster}
  </CodeBlock>
</CodeGroup>

<Warning>
  Avoid hardcoding secrets. Use `os.environ` to pass values from your local environment, or use a secrets manager in production.
</Warning>

### Packages

The tool shows version mismatches and packages only present in one environment:

```
=== PACKAGES: version mismatches ===
  * numpy: local=1.26.4  remote=2.2.6
  * torch: local=2.0.1  remote=2.8.0+cpu
  * attrs: local=23.2.0  remote=24.2.0
  * pyarrow: local=14.0.1  remote=17.0.0

=== PACKAGES: only in LOCAL ===
  + my-custom-package
  + dev-tools

=== PACKAGES: only in REMOTE ===
  + kuberay-client
```

<Warning>
  Watch for major version differences (NumPy 1.x vs 2.x) and PyTorch version mismatches.
</Warning>

#### Common Package Issues

| Issue                | Symptoms                                                             | Fix                             |
| -------------------- | -------------------------------------------------------------------- | ------------------------------- |
| **NumPy 1.x vs 2.x** | `ArrowInvalid`, `ValueError: cannot convert`, serialization failures | Pin `numpy==1.26.4`             |
| **PyTorch mismatch** | Model loading failures, CUDA errors, unexpected inference results    | Pin to matching `torch` version |
| **attrs mismatch**   | `TypeError: Enum.__new__() missing 1 required positional argument`   | Pin `attrs` to local version    |
| **Missing package**  | `ModuleNotFoundError: No module named 'xyz'`                         | Add package to manifest         |

#### Fixing Package Mismatches

**Option 1: Manifest pip Dependencies**

Specify packages in a Geneva manifest for a quick fix:

<CodeGroup>
  <CodeBlock filename="Python" language="python" icon="python">
    {PyPipManifest}
  </CodeBlock>
</CodeGroup>

*Pros*: Quick, reusable across sessions, stored with your database.

*Cons*: Slower startup (downloads packages), may not work for complex dependencies.

**Option 2: Custom Ray Worker Image**

For KubeRay deployments, build a custom worker image:

<CodeGroup>
  ```dockerfile Dockerfile icon="docker" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
  # Dockerfile.ray-worker
  FROM rayproject/ray:2.30.0-py311

  # Install exact versions
  RUN pip install \
      numpy==1.26.4 \
      torch==2.0.1 \
      attrs==23.2.0 \
      geneva==0.8.0

  # Copy any custom packages
  COPY ./my_udfs /app/my_udfs
  ```
</CodeGroup>

Then reference in RayCluster spec:

<CodeGroup>
  ```yaml Kubernetes icon="kubernetes" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
  spec:
    workerGroupSpecs:
      - template:
          spec:
            containers:
              - image: your-registry/ray-worker:latest
  ```
</CodeGroup>

*Pros*: Fastest startup, reproducible.

*Cons*: Requires image build/push workflow.

**Option 3: Conda Environment**

Use a conda environment on workers via the cluster builder:

<CodeGroup>
  <CodeBlock filename="Python" language="python" icon="python">
    {PyCondaClusterPath}
  </CodeBlock>
</CodeGroup>

Or specify conda channels and dependencies inline:

<CodeGroup>
  <CodeBlock filename="Python" language="python" icon="python">
    {PyCondaClusterInline}
  </CodeBlock>
</CodeGroup>

*Pros*: Best for complex dependencies with native libraries (ffmpeg, CUDA).

*Cons*: Slower environment creation, requires conda on workers.

## API Reference

For detailed API documentation on the environment comparison functions, see the [Geneva Diagnostics API Reference](https://lancedb.github.io/geneva/api/diagnostics).