How it Works
- The Expression Translator translates SQL requests into data algebra.
- The Algebraic Optimizer performs adaptive restructuring on the algebraic version of the SQL query by referencing the expression store to identify which parts of the query match stored results, from prior queries, and which parts of a new query data algebra can resolve.
- The Algebraic Optimizer stores these matched results in the Expression Store, effectively allowing queries to become look-ups. It also monitors the contents of the Local Data Store by performing statistical frequency calculations to determine which results are most frequently reused.
- The Expression Translator then passes the query, in SQL form, to Spark SQL and catalyst.
- When catalyst returns the result of the query, the Algebraic Optimizer retrieves its part of the query from the Expression Store, merges it with the data received from Catalyst and passes the result to the application that made the SQL request.
Note: The Local Data Store holds its data in memory and on disk. Its precise contents vary according to the variety of queries dealt with by AQA. Data that turns out to be rarely used is thus gradually replaced by data that is more frequently reused. Upon introduction of new data, AQA takes note and includes the data, when appropriate, in the result store. The process is then repeated and the SQL environment is subsequently optimized for maximum performance.