Architectural bottlenecks in the web API design
The intention was to follow the same concepts as in the CLI and adhere to the DRY (Don't Repeat Yourself) principle. While this goal is desirable, achieving fully shareable code across all layers is not always feasible, especially when performance is a concern.
Currently, there are two major architectural issues affecting the Web API:
1. Excessive I/O overhead on dataset access
In the current setup, the meteorological variable (global horizontal irradiance, direct horizontal irradiance, temperature etc) datasets are opened and closed on every query, leading to significant I/O overhead.
-
Proposed Solution: Refactor the application to lazily load the datasets once during application startup and reuse them across all requests. This will dramatically reduce redundant I/O operations.
2. Redundant timestamp extraction logic
At the moment, a single meteorological variable is selected to generate and share timestamps across the application. This adds unnecessary I/O overhead.
-
Proposed Solution: Reuse already-loaded datasets to extract timestamps directly, eliminating the need for extra reads.