# Integration test for burn collective operations with multiple nodes and devices.

Run `cargo run --bin test_launcher`

There are 3 binaries:

## node.rs

Launches `n` threads each simulating a different device. Currently the backend is NdArray,
so everything is CPU. The program takes a file with configurations and input data.

## global.rs

Runs the global orchestrator, who is responsible for responding to global collective operation
requests. In the case of an all-reduce, the orchestrator responds with a strategy for reducing,
and the node can do the reduction independently.

## test_launcher.rs

Generates input data, calculates the expected results, and launches the nodes each with their
own inputs in a separate file.

The topology is [4, 4, 4, 4]. This means 4 nodes are launched,
each with 4 threads (for each device).

The global orchestrator (`global.rs`) is also launched.

## Output

The outputs and inputs for each node and the orchestrator are written to the `target/test_files` folder

If the nodes or orchestrator stall, there is a timeout.