[refactor] Generalize the design of `tornadovm` package to support multiple new models and types for GPU exec #62

mikepapadim · 2025-11-05T18:29:03Z

No description provided.

This commit introduces new layer components and planners tailored for Qwen3 and TornadoVM environments, including: - LogitsQ8_0Layer class for handling Q8_0 weights - Qwen3FP16FFNLayers for managing FP16 weights in Qwen3 architecture - Qwen3FP16LayerPlanner for planning TornadoVM operations using FP16 layers and weights These additions enhance compatibility and extend functionality for the Qwen3 model.

- Simplified data transfer logic and removed unnecessary comments. - Deprecated `Qwen3TornadoVMLayerPlanner` and related Qwen3 implementation elements. - Renamed and aligned temporary buffers for clarity.

- Remove `GenericLayerPlanner` interface and consolidate layer planner logic. - Standardize `qwen2State` and `qwen3State` usage across layers. - Adjust local work sizes for better efficiency in FP16 and Q8_0 layers. - Cleanup redundant code and comments for improved readability.

Introduce `Phi3FP16LayerPlanner` and `Phi3Q8_0LayerPlanner`, enabling TornadoVM support for the Phi3 model with FP16 and Q8_0 weights. These planners implement Phi3-specific layer components and caching mechanisms for task graphs and schedulers.

Deleted `Phi3TornadoVMLayerPlanner` and `Phi3TornadoVMLayerPlannerQ8_0` classes, consolidating and simplifying planner logic across the project.

- Introduced `WorkerGridFactory` for standardizing RMSNorm worker creation. - Adjusted RMSNorm worker configuration for FP16 and Q8_0 layers with support for conditional weight types. - Removed redundant code and outdated comments for clarity.

- Added utility methods for creating workers: RMSNorm, QKV Matmul, RoPE, Attention, FFN Gate+Up, and FFN Down. - Centralized worker grid logic to improve code readability and maintainability.

- Replaced `TornadoVMGenericLayerPlanner` with `GenericLayerPlanner` for consistency across planners. - Updated `QuantizationPlannerFactory` and related classes to use the new interface. - Added `createSingleWorker` method to `WorkerGridFactory` for standardized single worker creation. - Simplified and cleaned up TornadoVMMasterPlan, removing unused methods and comments.

Copilot

Pull Request Overview

This PR refactors the TornadoVM layer planning architecture from a monolithic, model-specific approach to a clean, extensible factory pattern organized by quantization type and model architecture. The refactoring separates concerns between quantization strategies (FP16, Q8_0) and model types (Llama, Qwen2, Qwen3, Phi3), making the codebase more maintainable and easier to extend for new models or quantization types.

Key Changes:

Introduced a factory-based architecture (QuantizationPlannerFactory) that routes to appropriate planners based on quantization type and model
Created abstract base classes (QuantizedLayerPlanner, FP16LayerPlanner, Q8_0LayerPlanner) to eliminate code duplication
Split layer implementations into separate packages by quantization type (layers/type/fp16/, layers/type/q8_0/)
Reorganized kernel classes into a dedicated kernels package
Fixed package declarations for tokenizer classes to use org.beehive.gpullama3.tokenizer consistently

Reviewed Changes

Copilot reviewed 80 out of 80 changed files in this pull request and generated 43 comments.

Show a summary per file

File	Description
`FloatArrayUtils.java`	Updated package declaration to correct utils path
`Qwen3Q8_0FFNLayers.java`	New Q8_0 quantized FFN layer implementation for Qwen3 with GQA support
`Qwen2Q8_0FFNLayers.java`	New Q8_0 quantized FFN layer implementation for Qwen2 with bias terms
`Phi3Q8_0FFNLayers.java`	New Q8_0 quantized FFN layer implementation for Phi3 with combined QKV
`LogitsQ8_0Layer.java`	New Q8_0 quantized logits layer implementation
`LlamaQ8_0FFNLayers.java`	Refactored Llama Q8_0 FFN layers into dedicated class
`Qwen3FP16FFNLayers.java`	New FP16 FFN layer implementation for Qwen3
`Qwen2FP16FFNLayers.java`	New FP16 FFN layer implementation for Qwen2
`Phi3FP16FFNLayers.java`	New FP16 FFN layer implementation for Phi3
`LogitsFP16Layer.java`	New FP16 logits layer implementation
`LlamaFP16FFNLayers.java`	New FP16 FFN layer implementation for Llama
`SchedulerType.java`	New enum for hardware scheduler detection
`SchedulerDetectionService.java`	New service for detecting appropriate scheduler type
`Activation.java`	New activation layer abstraction
`AbstractLayer.java`	New base class for all layer implementations
`Q8_0LayerPlanner.java`	Abstract base for Q8_0 quantization planners
`FP16LayerPlanner.java`	Abstract base for FP16 quantization planners
Model-specific planners	New planner implementations organized by model and quantization type
`QuantizedLayerPlanner.java`	Abstract base for all quantization-specific planners
`QuantizationPlannerFactory.java`	Factory for creating appropriate planners based on model and quantization
`WorkerGridFactory.java`	Factory for creating worker grid configurations
Kernel files	Moved to `kernels` package with updated package declarations
`TornadoVMMasterPlan.java`	Refactored to use factory pattern instead of model-specific dispatching
`TornadoVMLayerPlanner.java`	Removed - functionality split into specific implementations
`GenericLayerPlanner.java`	Updated interface name and added cache accessor methods
Tokenizer files	Fixed package declarations to remove nested packages
Model files	Updated imports for tokenizer package changes

Comments suppressed due to low confidence (3)

src/main/java/org/beehive/gpullama3/tornadovm/layers/type/q8_0/LlamaQ8_0FFNLayers.java:28

Corrected spelling of 'ffunn' to 'ffn' in variable name.
src/main/java/org/beehive/gpullama3/tornadovm/layers/type/q8_0/LlamaQ8_0FFNLayers.java:82
The parameter 'config' is never used.
src/main/java/org/beehive/gpullama3/tornadovm/layers/type/q8_0/LlamaQ8_0FFNLayers.java:142
This method overrides AbstractLayer.configureLayerDataTransfers; it is advisable to add an Override annotation.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2025-11-05T18:34:52Z

src/main/java/org/beehive/gpullama3/tornadovm/layers/type/fp16/LogitsFP16Layer.java

+        tornadoForwardScheduler.addWorkerGrid("logits.projection", vocabWorker);
+        tornadoForwardScheduler.addWorkerGrid("logits.reductionsOneBlockLogits", logitsRMS);
+        tornadoForwardScheduler.addWorkerGrid("logits.mapContextLogits", logitsRMS);
+        return scheduler;


Method returns uninitialized scheduler field instead of the parameter tornadoForwardScheduler. Should return tornadoForwardScheduler to maintain consistency with the pattern used in other layer classes.

Copilot · 2025-11-05T18:34:52Z

src/main/java/org/beehive/gpullama3/tornadovm/layers/type/fp16/LlamaFP16FFNLayers.java

+public class LlamaFP16FFNLayers extends AbstractLayer {
+
+    String lastTaskGraphID;
+    TaskGraph ffunnLayerTaskGraph;


Corrected spelling of 'ffunn' to 'ffn' in variable name.

Copilot · 2025-11-05T18:34:52Z

src/main/java/org/beehive/gpullama3/tornadovm/layers/type/q8_0/Qwen3Q8_0FFNLayers.java

+                        qkvDim1, kvDim0, LOCAL_WORK_GROUP_SIZE_ALLOC);
+
+        // Qcur: RMS norm with parallel offset for Query
+        Qwen3State qwen3State = (Qwen3State) state;


Redundant cast and variable shadowing. The field qwen3State is already initialized and strongly typed at line 42. This local variable shadows the field and adds unnecessary casting. Remove this line and use the existing field directly.

Copilot · 2025-11-05T18:34:53Z

src/main/java/org/beehive/gpullama3/tornadovm/layers/type/q8_0/Qwen3Q8_0FFNLayers.java

+            unifiedLayer.transferToDevice(DataTransferMode.EVERY_EXECUTION,
+                    qwen3State.positionHolder, qwen3State.temp, qwen3State.tempFFN);
+
+            Qwen3State qwen3State = (Qwen3State) state;


Redundant cast and variable shadowing. The field qwen3State is already initialized and strongly typed at line 42. This local variable shadows the field and adds unnecessary casting. Remove this line and use the existing field directly.

Copilot · 2025-11-05T18:34:53Z

src/main/java/org/beehive/gpullama3/tornadovm/layers/type/q8_0/Qwen3Q8_0FFNLayers.java

+                    qwen3State.wrapKeyCache, qwen3State.wrapValueCache,
+                    qwen3State.wrapAtt, qwen3State.wrapHb, qwen3State.positionHolder);
+
+            Qwen3State qwen3State = (Qwen3State) state;


Redundant cast and variable shadowing. The field qwen3State is already initialized and strongly typed at line 42. This local variable shadows the field and adds unnecessary casting. Remove this line and use the existing field directly.

Copilot · 2025-11-05T18:35:01Z

src/main/java/org/beehive/gpullama3/tornadovm/layers/type/q8_0/Phi3Q8_0FFNLayers.java

+    /**
+     * Configure data transfers for first and subsequent layers
+     */
+    protected TaskGraph configureLayerDataTransfers(TaskGraph unifiedLayer, int layerIndex) {


This method overrides AbstractLayer.configureLayerDataTransfers; it is advisable to add an Override annotation.

Copilot · 2025-11-05T18:35:01Z

src/main/java/org/beehive/gpullama3/tornadovm/layers/type/fp16/Qwen2FP16FFNLayers.java

+    /**
+     * Configure data transfers for first and subsequent layers
+     */
+    protected TaskGraph configureLayerDataTransfers(TaskGraph unifiedLayer, int layerIndex) {


This method overrides AbstractLayer.configureLayerDataTransfers; it is advisable to add an Override annotation.

Copilot · 2025-11-05T18:35:01Z

src/main/java/org/beehive/gpullama3/tornadovm/layers/type/q8_0/Qwen3Q8_0FFNLayers.java

+    /**
+     * Configure data transfers for first and subsequent layers
+     */
+    protected TaskGraph configureLayerDataTransfers(TaskGraph unifiedLayer, int layerIndex) {


This method overrides AbstractLayer.configureLayerDataTransfers; it is advisable to add an Override annotation.

Copilot · 2025-11-05T18:35:01Z

src/main/java/org/beehive/gpullama3/tornadovm/layers/type/q8_0/Qwen2Q8_0FFNLayers.java

+    /**
+     * Configure data transfers for first and subsequent layers
+     */
+    protected TaskGraph configureLayerDataTransfers(TaskGraph unifiedLayer, int layerIndex) {


This method overrides AbstractLayer.configureLayerDataTransfers; it is advisable to add an Override annotation.

Copilot · 2025-11-05T18:35:02Z

src/main/java/org/beehive/gpullama3/tornadovm/layers/type/fp16/Qwen2FP16FFNLayers.java

+
+
+        // Ensure we have Qwen2-specific weights
+        if (!(weights instanceof FP16Weights weights1)) {


There is no need to test whether an instance of Qwen2TornadoWeights is also an instance of FP16Weights - it always is.

- Optimized worker grid setup with `WorkerGridFactory`, introducing `genericWorker` for versatile use cases. - Simplified task graph setup for FP16 and Q8_0 layers using streamlined loops and functional programming. - Removed unused methods and redundant comments from layer implementation classes. - Improved code readability and maintainability across TornadoVMMasterPlan, Logits, and FFN layers.

- Removed `setupLastID` and `getLastTaskGraphID` methods from multiple layer classes, consolidating their functionality into `AbstractLayer`. - Standardized `requireWeightsType` utility for type validation across layers. - Streamlined worker grid setup with `WorkerGridFactory` updates to improve maintainability. - Cleaned up redundant comments and unused methods for better readability.

Copilot

Pull Request Overview

Copilot reviewed 81 out of 81 changed files in this pull request and generated 22 comments.

Comments suppressed due to low confidence (4)

src/main/java/org/beehive/gpullama3/tornadovm/layers/type/q8_0/Qwen3Q8_0FFNLayers.java:262

Variable 'Qwen3State qwen3State' is never read.
src/main/java/org/beehive/gpullama3/tornadovm/layers/type/q8_0/Qwen3Q8_0FFNLayers.java:262
Confusing name: method setupSingleQwen3FFNLayer also refers to field qwen3State (without qualifying it with 'this').
src/main/java/org/beehive/gpullama3/tornadovm/layers/type/q8_0/Qwen3Q8_0FFNLayers.java:339
Confusing name: method configureLayerDataTransfers also refers to field qwen3State (without qualifying it with 'this').
src/main/java/org/beehive/gpullama3/tornadovm/layers/type/q8_0/Qwen3Q8_0FFNLayers.java:356
Confusing name: method configureLayerDataTransfers also refers to field qwen3State (without qualifying it with 'this').

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2025-11-06T09:54:51Z

src/main/java/org/beehive/gpullama3/tornadovm/layers/type/q8_0/Qwen3Q8_0FFNLayers.java

+                        qkvDim1, kvDim0, LOCAL_WORK_GROUP_SIZE_ALLOC);
+
+        // Qcur: RMS norm with parallel offset for Query
+        Qwen3State qwen3State = (Qwen3State) state;


Redundant cast and variable shadowing. The class already has a typed field qwen3State (line 42), so this local variable shadows it unnecessarily. Remove this line and use the existing this.qwen3State field instead.

Copilot · 2025-11-06T09:54:52Z

src/main/java/org/beehive/gpullama3/tornadovm/layers/type/q8_0/Qwen3Q8_0FFNLayers.java

+            unifiedLayer.transferToDevice(DataTransferMode.EVERY_EXECUTION,
+                    qwen3State.positionHolder, qwen3State.temp, qwen3State.tempFFN);
+
+            Qwen3State qwen3State = (Qwen3State) state;


Redundant cast and variable shadowing. This local variable shadows the class field qwen3State (line 42). Use this.qwen3State instead of creating a new local variable.

Copilot · 2025-11-06T09:54:52Z

src/main/java/org/beehive/gpullama3/tornadovm/layers/type/q8_0/Qwen3Q8_0FFNLayers.java

+                    qwen3State.wrapKeyCache, qwen3State.wrapValueCache,
+                    qwen3State.wrapAtt, qwen3State.wrapHb, qwen3State.positionHolder);
+
+            Qwen3State qwen3State = (Qwen3State) state;


Redundant cast and variable shadowing. This local variable shadows the class field qwen3State (line 42). Use this.qwen3State instead of creating a new local variable.

Copilot · 2025-11-06T09:54:52Z

src/main/java/org/beehive/gpullama3/tornadovm/layers/type/fp16/LogitsFP16Layer.java

+        tornadoForwardScheduler.addWorkerGrid("logits.projection", vocabWorker);
+        tornadoForwardScheduler.addWorkerGrid("logits.reductionsOneBlockLogits", logitsRMS);
+        tornadoForwardScheduler.addWorkerGrid("logits.mapContextLogits", logitsRMS);
+        return scheduler;


Incorrect return value in updateGridScheduler. The method should return the updated tornadoForwardScheduler parameter, not the uninitialized instance field scheduler. Change to return tornadoForwardScheduler;.

Copilot · 2025-11-06T09:54:53Z

src/main/java/org/beehive/gpullama3/tornadovm/layers/AbstractLayer.java

+    /** Collected snapshots for scheduling / debugging. */
+    protected final List<ImmutableTaskGraph> taskGraphs = new ArrayList<>();
+
+    protected AbstractLayer(String taskGraphName, State state, Weights weights, Configuration config) {


The parameter 'taskGraphName' is never used.

Copilot · 2025-11-06T09:54:56Z

src/main/java/org/beehive/gpullama3/tornadovm/layers/type/fp16/Qwen3FP16FFNLayers.java

+    /**
+     * Configure data transfers for first and subsequent layers
+     */
+    protected TaskGraph configureLayerDataTransfers(TaskGraph unifiedLayer, int layerIndex) {


This method overrides AbstractLayer.configureLayerDataTransfers; it is advisable to add an Override annotation.

Copilot · 2025-11-06T09:54:56Z

src/main/java/org/beehive/gpullama3/tornadovm/layers/type/q8_0/LlamaQ8_0FFNLayers.java

+            return unifiedLayer;
+    }
+
+    protected TaskGraph configureLayerDataTransfers(TaskGraph unifiedLayer, int layerIndex) {


This method overrides AbstractLayer.configureLayerDataTransfers; it is advisable to add an Override annotation.

Copilot · 2025-11-06T09:54:56Z

src/main/java/org/beehive/gpullama3/tornadovm/layers/type/q8_0/Phi3Q8_0FFNLayers.java

+    /**
+     * Configure data transfers for first and subsequent layers
+     */
+    protected TaskGraph configureLayerDataTransfers(TaskGraph unifiedLayer, int layerIndex) {


This method overrides AbstractLayer.configureLayerDataTransfers; it is advisable to add an Override annotation.

Copilot · 2025-11-06T09:54:56Z

src/main/java/org/beehive/gpullama3/tornadovm/layers/type/q8_0/Qwen2Q8_0FFNLayers.java

+    /**
+     * Configure data transfers for first and subsequent layers
+     */
+    protected TaskGraph configureLayerDataTransfers(TaskGraph unifiedLayer, int layerIndex) {


This method overrides AbstractLayer.configureLayerDataTransfers; it is advisable to add an Override annotation.

Copilot · 2025-11-06T09:54:57Z

src/main/java/org/beehive/gpullama3/tornadovm/layers/type/q8_0/Qwen3Q8_0FFNLayers.java

+    /**
+     * Configure data transfers for first and subsequent layers
+     */
+    protected TaskGraph configureLayerDataTransfers(TaskGraph unifiedLayer, int layerIndex) {


This method overrides AbstractLayer.configureLayerDataTransfers; it is advisable to add an Override annotation.

- Removed unnecessary weight validation code and unused worker grid setup methods across multiple layers. - Consolidated task graph and worker grid logic for better code maintainability. - Cleaned up redundant comments and improved readability.

- Renamed local variables for clarity in FFN layer setup. - Updated `QuantizationPlannerFactory` to include Q4_0 planner skeleton. - Removed unused methods and redundant comments from `TransformerComputeKernelsLayered`. - Cleaned up method parameter formatting for better readability.

- Removed redundant `setupTornadoForwardPlanLayered` and `setupTornadoForwardPlanLayeredNonNvidia` methods across planners. - Replaced custom `RMSNorm` worker configurations with `WorkerGridFactory` utility for consistency. - Cleaned up unused parameters, outdated comments, and redundant code blocks.

…tter readability.

- Moved `createExecutionPlan` method in `TornadoVMMasterPlan` for better logical organization. - Removed unused imports from `GPULLlama3TypeException` and `GenericLayerPlanner`.

- Removed redundant cache setup and forward plan methods from multiple layer planners. - Consolidated cache logic into base planners `FP16LayerPlanner` and `Q8_0LayerPlanner`. - Standardized FFN layers by extending from `AbstractFFNLayers`. - Cleaned up imports, comments, and unused code for improved readability and maintainability.

- Introduced `AbstractFFNLayers` as a base class for FFN layers to ensure type safety and reuse across various models. - Removed unused imports, outdated comments, and redundant code for better readability and maintainability. - Cleaned up and standardized task graph logic and data transfer setup in `AbstractLayer` and its subclasses.

… improved readability.

- Replaced custom WorkerGrid configurations with `WorkerGridFactory` utility across multiple layers for consistency. - Removed unused imports and redundant code blocks for improved readability and maintainability. - Cleaned up comments and optimized method formatting.

- Removed redundant comments and unused code blocks from multiple layers. - Standardized task graph formatting and method calls for improved maintainability. - Simplified worker grid setup by eliminating unnecessary configurations.

…ary blank lines for better readability.

- Removed redundant comments and unused variables to improve readability and maintainability. - Eliminated unnecessary blank lines for more concise method formatting. - Standardized WorkerGrid setup across multiple layers using `WorkerGridFactory`.

- Introduced a Maven profile (`spotless`) in `pom.xml` for automated formatting of Java, XML, Markdown, and properties files. - Configured Google Java Format (AOSP style) for Java files, XML sorting, and trailing whitespace cleanup across multiple file types. - Updated `Makefile` with `lint` and `format` targets for code style validation and automatic formatting.

- Reorganized method definitions across `Vocabulary`, `Qwen3Tokenizer`, `LlamaTokenizer`, and `MistralTokenizer`. - Removed redundant methods and code duplications to enhance maintainability. - Standardized utility functions (`bytesToUnicode`, `merge`, `findAll`) across files for reusability. - Improved formatting for better code readability.

…s for consistency across TornadoVM layers and planners.

…_0` and update references across layers, planners, and model loaders for consistency.

…lace vocab size handling with `vocabulary.size()` for better clarity and maintainability. Remove unused TornadoVM layer planners.

… improved readability.

…or clarity, clean up imports, improve formatting, and annotate deprecated elements with better documentation.

…ve encapsulation.

…tory` with explicit `WorkerGrid1D` and `WorkerGrid2D` configurations across `Qwen2FP16FFNLayers` and `Qwen2Q8_0FFNLayers` for clarity and maintainability. Streamline local and global work size setup logic.

- Simplified weight data transfers with cleaner formatting and streamlined method calls. - Optimized `rmsNormWorker` initialization by incorporating `WorkerGridFactory`. - Improved task graph setup for FFN and attention layers by reducing code duplication and enhancing readability. - Adjusted comments and formatting for consistency across layers.

…ionService` from `layers` to `layerplanner.strategy` for improved modularity and clarity.

Copilot

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

set_paths

src/main/java/org/beehive/gpullama3/tornadovm/layerplanner/quantization/FP16LayerPlanner.java

orionpapadakis · 2025-11-07T08:41:07Z

src/main/java/org/beehive/gpullama3/tornadovm/layerplanner/quantization/FP16LayerPlanner.java

+    protected LogitsFP16Layer logitsLayer;
+
+    protected List<ImmutableTaskGraph> cachedTaskGraphs;
+    protected GridScheduler cachedScheduler;


remove cached from name

orionpapadakis · 2025-11-07T08:41:28Z

src/main/java/org/beehive/gpullama3/tornadovm/layerplanner/quantization/FP16LayerPlanner.java

+     *
+     * Removed from all model-specific planners - centralized here.
+     */
+    public final List<ImmutableTaskGraph> getCachedTaskGraphs() {


remove cached from name

orionpapadakis · 2025-11-07T08:41:38Z

src/main/java/org/beehive/gpullama3/tornadovm/layerplanner/quantization/FP16LayerPlanner.java

+     * Removed from all model-specific planners - centralized here.
+     */
+    @Override
+    public final GridScheduler getCachedGridScheduler() {


remove cached from name

orionpapadakis · 2025-11-07T08:43:15Z

src/main/java/org/beehive/gpullama3/tornadovm/layerplanner/quantization/FP16LayerPlanner.java

+    /**
+     * Clears cache (for strategy optimization or re-initialization).
+     *
+     * Removed from all model-specific planners - centralized here.
+     */
+    public final void clearCache() {
+        this.cachedTaskGraphs = null;
+        this.cachedScheduler = null;
+    }


confusing. as it is unused maybe it can be removed ?

orionpapadakis · 2025-11-07T08:43:31Z

src/main/java/org/beehive/gpullama3/tornadovm/layerplanner/quantization/Q8_0LayerPlanner.java

+    protected List<ImmutableTaskGraph> cachedTaskGraphs;
+    protected GridScheduler cachedScheduler;


remove cached from names

orionpapadakis · 2025-11-07T08:43:45Z

src/main/java/org/beehive/gpullama3/tornadovm/layerplanner/quantization/Q8_0LayerPlanner.java

+     *
+     * Removed from all model-specific planners - centralized here.
+     */
+    public final List<ImmutableTaskGraph> getCachedTaskGraphs() {


remove cached from names

orionpapadakis · 2025-11-07T08:43:56Z

src/main/java/org/beehive/gpullama3/tornadovm/layerplanner/quantization/Q8_0LayerPlanner.java

+     * Removed from all model-specific planners - centralized here.
+     */
+    @Override
+    public final GridScheduler getCachedGridScheduler() {


remove cached from names

orionpapadakis · 2025-11-07T08:44:07Z

src/main/java/org/beehive/gpullama3/tornadovm/layerplanner/quantization/Q8_0LayerPlanner.java

+    public final void clearCache() {
+        this.cachedTaskGraphs = null;
+        this.cachedScheduler = null;
+    }


…d scheduler with immutable versions for improved clarity and consistency across the codebase.

mikepapadim added 10 commits November 1, 2025 10:26

WIP

ba4abdc

[WIP] refactoring

3ead7e4

Refactor Qwen2 and Qwen3 TornadoVM layers:

cec38b3

- Simplified data transfer logic and removed unnecessary comments. - Deprecated `Qwen3TornadoVMLayerPlanner` and related Qwen3 implementation elements. - Renamed and aligned temporary buffers for clarity.

Remove Phi3 TornadoVM layer planners for FP16 and Q8_0 quantizations

3ab251b

Deleted `Phi3TornadoVMLayerPlanner` and `Phi3TornadoVMLayerPlannerQ8_0` classes, consolidating and simplifying planner logic across the project.

Introduce WorkerGridFactory for standardized worker grid creation

72c6619

- Added utility methods for creating workers: RMSNorm, QKV Matmul, RoPE, Attention, FFN Gate+Up, and FFN Down. - Centralized worker grid logic to improve code readability and maintainability.

mikepapadim requested review from Copilot, mairooni, orionpapadakis and stratika November 5, 2025 18:29

mikepapadim self-assigned this Nov 5, 2025

mikepapadim changed the title ~~[refactor] Generalize the design of tornadovm packge to support multiple new models and types for GPU exec~~ [refactor] Generalize the design of tornadovm package to support multiple new models and types for GPU exec Nov 5, 2025

Copilot AI reviewed Nov 5, 2025

View reviewed changes

mikepapadim added 2 commits November 6, 2025 11:28

Copilot AI review requested due to automatic review settings November 6, 2025 09:48

Copilot AI reviewed Nov 6, 2025

View reviewed changes

mikepapadim added 7 commits November 6, 2025 11:55

Refactor TornadoVM layers:

322a442

- Removed unnecessary weight validation code and unused worker grid setup methods across multiple layers. - Consolidated task graph and worker grid logic for better code maintainability. - Cleaned up redundant comments and improved readability.

Clean up Qwen3FP16LayerPlanner: remove redundant blank lines for be…

090da7d

…tter readability.

Refactor TornadoVM classes:

6cdfd2f

- Moved `createExecutionPlan` method in `TornadoVMMasterPlan` for better logical organization. - Removed unused imports from `GPULLlama3TypeException` and `GenericLayerPlanner`.

Clean up FP16LayerPlanner: remove redundant comments for readability.

5f62e93

Copilot AI review requested due to automatic review settings November 6, 2025 12:57

mikepapadim added 17 commits November 6, 2025 15:10

Clean up Qwen3FP16LayerPlanner: remove unused ffnLayers field for…

c253e85

… improved readability.

Clean up TornadoVM layers:

a180a51

- Removed redundant comments and unused code blocks from multiple layers. - Standardized task graph formatting and method calls for improved maintainability. - Simplified worker grid setup by eliminating unnecessary configurations.

Clean up LlamaQ8_0FFNLayers: remove redundant comments and unnecess…

cd96ca5

…ary blank lines for better readability.

Clean up TornadoVM FFN layers:

ff224e4

- Removed redundant comments and unused variables to improve readability and maintainability. - Eliminated unnecessary blank lines for more concise method formatting. - Standardized WorkerGrid setup across multiple layers using `WorkerGridFactory`.

Merge remote-tracking branch 'origin' into refactor/generarilize

b66cf2f

Refactor imports: replace nested package imports with simplified path…

d5f45d8

…s for consistency across TornadoVM layers and planners.

Refactor Q8_0 weights: rename Q8_0Weights to `LlamaTornadoWeightsQ8…

4b0206a

…_0` and update references across layers, planners, and model loaders for consistency.

Refactor Phi3 model loader: streamline configuration creation and rep…

67ecba3

…lace vocab size handling with `vocabulary.size()` for better clarity and maintainability. Remove unused TornadoVM layer planners.

Clean up WorkerGridFactory: remove redundant commented-out code for…

ad7dff8

… improved readability.

Refactor Sampler, Tuple2, and ModelLoader: reorganize methods f…

8d02237

…or clarity, clean up imports, improve formatting, and annotate deprecated elements with better documentation.

Make createSampler package-private to restrict visibility and impro…

fe57e79

…ve encapsulation.

Refactor grid scheduler worker initialization: replace `WorkerGridFac…

155025d

…tory` with explicit `WorkerGrid1D` and `WorkerGrid2D` configurations across `Qwen2FP16FFNLayers` and `Qwen2Q8_0FFNLayers` for clarity and maintainability. Streamline local and global work size setup logic.

Refactor package structure: move SchedulerType and `SchedulerDetect…

14a1529

…ionService` from `layers` to `layerplanner.strategy` for improved modularity and clarity.

Copilot AI reviewed Nov 6, 2025

View reviewed changes

orionpapadakis approved these changes Nov 7, 2025

View reviewed changes

mikepapadim added 3 commits November 7, 2025 10:52

Update tornadovm tip

bda349a

Revert local changes

379cf55

Refactor TornadoVM layer planners: replace cached task graphs and gri…

af0eb5c

…d scheduler with immutable versions for improved clarity and consistency across the codebase.

mikepapadim merged commit c9f00ca into main Nov 7, 2025
1 check passed



		// Ensure we have Qwen2-specific weights
		if (!(weights instanceof FP16Weights weights1)) {

		protected List<ImmutableTaskGraph> cachedTaskGraphs;
		protected GridScheduler cachedScheduler;

[refactor] Generalize the design of tornadovm package to support multiple new models and types for GPU exec #62

[refactor] Generalize the design of tornadovm package to support multiple new models and types for GPU exec #62

Uh oh!

Conversation

mikepapadim commented Nov 5, 2025

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Key Changes:

Reviewed Changes

Uh oh!

Copilot AI Nov 5, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Nov 5, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Nov 5, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Nov 5, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Nov 5, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Nov 5, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Nov 5, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Nov 5, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Nov 5, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Nov 5, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Uh oh!

Copilot AI Nov 6, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Nov 6, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Nov 6, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Nov 6, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Nov 6, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Nov 6, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Nov 6, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Nov 6, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Nov 6, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Nov 6, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

[refactor] Generalize the design of `tornadovm` package to support multiple new models and types for GPU exec #62

[refactor] Generalize the design of `tornadovm` package to support multiple new models and types for GPU exec #62