[COR-4] Cleanup generate_config

[mpoeter]

Scope & Purpose

Previously we had a special add-yaml attribute in the tests/kafka-test-definitions.yml and the generate_config would create a new job based on it. This PR instead adds an explicit kafka-driver-tests with all the containers and has the test-definition refer to that job. This way we avoid the whole magic of creating new jobs with some local yaml snippets.

• 🔨 Refactoring/simplification

---

Note

Replaces add-yaml job derivation with explicit driver/kafka test jobs, refactors CircleCI config and generator, and updates test definitions accordingly.

• CI config (.circleci/base_config.yml):
  • Shared params/commands: Add .shared-test-job-parameters, clone-driver-repo, and run-test for reuse.
  • Jobs:
    • New run-driver-tests job to clone/init driver repos and run suites.
    • New kafka-driver-tests job including Kafka brokers and schema registry containers plus readiness wait.
    • Update run-linux-tests to use shared parameters.
  • Parameters: Add kafkaHostList; introduce docker_image in shared params.
• Generator (.circleci/generate_config.py):
  • Remove add-yaml handling and related plumbing.
  • Allow selecting target job via YAML job field (default run-linux-tests).
  • Simplify readers (read_yaml_suite, read_yaml_multi_bucket_suite, read_definitions) and pass optional init_command.
  • Minor error/validation cleanups.
• Test definitions:
  • tests/arangojs.yml, tests/go.yml, tests/java.yml, tests/spark-datasource.yml: set job: run-driver-tests.
  • tests/kafka.yml: switch to job: kafka-driver-tests; remove previous add-yaml block; use env vars ${KAFKA_HOST_LIST} and ${KAFKA_SCHEMA_HOST} in args.

^{Written by Cursor Bugbot for commit 90ae1b9. This will update automatically on new commits. Configure here.}

[Copilot]

Pull Request Overview

This PR refactors the test configuration system by removing the dynamic job generation mechanism (add-yaml) and replacing it with an explicit kafka-driver-tests job definition. The changes simplify the configuration by eliminating magic job creation and moving Kafka-specific setup from the test definition file into the base CircleCI config.

Key changes:

• Removed add-yaml dynamic job generation logic from generate_config.py
• Added explicit kafka-driver-tests job in base_config.yml with Kafka containers
• Updated test definitions to reference the new job directly and use environment variables

Reviewed Changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.

File	Description
tests/kafka-test-definitions.yml	Removed add-yaml section and inline Kafka container definitions; updated to reference kafka-driver-tests job and use environment variables for hosts
.circleci/generate_config.py	Removed all add-yaml parsing and dynamic job generation logic; simplified function signatures by removing yaml_struct parameter
.circleci/base_config.yml	Added explicit kafka-driver-tests job with Kafka containers; extracted run-test command and refactored run-linux-tests to use it

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.