dbcli/sqlglot
1
0
Fork 0
Code Issues Activity

Merging upstream version 25.29.0.

Browse source 
Signed-off-by: Daniel Baumann <daniel@debian.org>
This commit is contained in:
Daniel Baumann 2025-02-13 21:56:19 +01:00
parent de8c8a17d0
commit 1e53504dfc
Signed by: daniel
GPG key ID: FBB4F0E80A80222F
97 changed files with 64720 additions and 61752 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

115
CHANGELOG.md
View file

@ -1,6 +1,118 @@
Changelog
=========
## [v25.28.0] - 2024-10-25
### :boom: BREAKING CHANGES
- due to [`1691388`](https://github.com/tobymao/sqlglot/commit/16913887f5573f01eb8cd2b9336d4b37b84a449a) - Fix chained exp.SetOperation type annotation *(PR [#4274](https://github.com/tobymao/sqlglot/pull/4274) by [@VaggelisD](https://github.com/VaggelisD))*:
Fix chained exp.SetOperation type annotation (#4274)
- due to [`c3c1997`](https://github.com/tobymao/sqlglot/commit/c3c199714df04edfe3698594680bac06575ca285) - Add support for STRING function *(PR [#4284](https://github.com/tobymao/sqlglot/pull/4284) by [@VaggelisD](https://github.com/VaggelisD))*:
Add support for STRING function (#4284)
### :sparkles: New Features
- [`379f487`](https://github.com/tobymao/sqlglot/commit/379f487080d95ef6e87cbbae8003541cde381ac0) - **bigquery**: transpile EDIT_DISTANCE, closes [#4283](https://github.com/tobymao/sqlglot/pull/4283) *(commit by [@georgesittas](https://github.com/georgesittas))*
- [`c3c1997`](https://github.com/tobymao/sqlglot/commit/c3c199714df04edfe3698594680bac06575ca285) - **bigquery**: Add support for STRING function *(PR [#4284](https://github.com/tobymao/sqlglot/pull/4284) by [@VaggelisD](https://github.com/VaggelisD))*
- [`1a26bff`](https://github.com/tobymao/sqlglot/commit/1a26bff619315a6e9dc3eab4dec07746b4820796) - **snowflake**: Transpile exp.SafeDivide *(PR [#4294](https://github.com/tobymao/sqlglot/pull/4294) by [@VaggelisD](https://github.com/VaggelisD))*
### :bug: Bug Fixes
- [`ac66d2f`](https://github.com/tobymao/sqlglot/commit/ac66d2f4b94e6a984adbf3df01139b6378248158) - **clickhouse**: properly parse CREATE FUNCTION DDLs *(PR [#4282](https://github.com/tobymao/sqlglot/pull/4282) by [@georgesittas](https://github.com/georgesittas))*
- :arrow_lower_right: *fixes issue [#3276](https://github.com/TobikoData/sqlmesh/issues/3276) opened by [@jwhitaker-gridcog](https://github.com/jwhitaker-gridcog)*
- [`1691388`](https://github.com/tobymao/sqlglot/commit/16913887f5573f01eb8cd2b9336d4b37b84a449a) - **optimizer**: Fix chained exp.SetOperation type annotation *(PR [#4274](https://github.com/tobymao/sqlglot/pull/4274) by [@VaggelisD](https://github.com/VaggelisD))*
- :arrow_lower_right: *fixes issue [#4261](https://github.com/tobymao/sqlglot/issues/4261) opened by [@gabrielteotonio](https://github.com/gabrielteotonio)*
- [`559e7bc`](https://github.com/tobymao/sqlglot/commit/559e7bc5bbc77e94dea6de0470659b3c3fa6851f) - **clickhouse**: Wrap subquery if it's LHS of IS NOT NULL *(PR [#4287](https://github.com/tobymao/sqlglot/pull/4287) by [@VaggelisD](https://github.com/VaggelisD))*
- :arrow_lower_right: *fixes issue [#4285](https://github.com/tobymao/sqlglot/issues/4285) opened by [@EugeneTorap](https://github.com/EugeneTorap)*
- [`47bc09a`](https://github.com/tobymao/sqlglot/commit/47bc09a85a3781682f5e58bfde5f453fb1a7c50b) - **sqlite**: Fix UNIQUE parsing *(PR [#4293](https://github.com/tobymao/sqlglot/pull/4293) by [@VaggelisD](https://github.com/VaggelisD))*
- :arrow_lower_right: *fixes issue [#4291](https://github.com/tobymao/sqlglot/issues/4291) opened by [@tshu-w](https://github.com/tshu-w)*
- [`ee266ef`](https://github.com/tobymao/sqlglot/commit/ee266ef8f92fe72252eea36b56e8825715644a4f) - improve support for identifier delimiter escaping *(PR [#4288](https://github.com/tobymao/sqlglot/pull/4288) by [@georgesittas](https://github.com/georgesittas))*
### :wrench: Chores
- [`c6ff7f1`](https://github.com/tobymao/sqlglot/commit/c6ff7f1a0b6e443d80bc0f0ad1086d5c7b13b9f4) - bump sqlglotrs to v0.2.13 *(commit by [@georgesittas](https://github.com/georgesittas))*
## [v25.27.0] - 2024-10-22
### :boom: BREAKING CHANGES
- due to [`4d86499`](https://github.com/tobymao/sqlglot/commit/4d8649940d02ac319f2fec372a52674488f01de5) - include the target node for Move edits *(PR [#4277](https://github.com/tobymao/sqlglot/pull/4277) by [@georgesittas](https://github.com/georgesittas))*:
include the target node for Move edits (#4277)
- due to [`9771965`](https://github.com/tobymao/sqlglot/commit/97719657d1b2074dabfbe54af0e1ea3acd6d4744) - Add support for TIMESTAMP_NTZ_FROM_PARTS *(PR [#4280](https://github.com/tobymao/sqlglot/pull/4280) by [@VaggelisD](https://github.com/VaggelisD))*:
Add support for TIMESTAMP_NTZ_FROM_PARTS (#4280)
- due to [`768adb3`](https://github.com/tobymao/sqlglot/commit/768adb3d85ed88931d761e5ecc8fb4a3a40d0dc5) - time string literals containing fractional seconds *(PR [#4269](https://github.com/tobymao/sqlglot/pull/4269) by [@treysp](https://github.com/treysp))*:
time string literals containing fractional seconds (#4269)
### :sparkles: New Features
- [`9771965`](https://github.com/tobymao/sqlglot/commit/97719657d1b2074dabfbe54af0e1ea3acd6d4744) - **snowflake**: Add support for TIMESTAMP_NTZ_FROM_PARTS *(PR [#4280](https://github.com/tobymao/sqlglot/pull/4280) by [@VaggelisD](https://github.com/VaggelisD))*
- [`9e11654`](https://github.com/tobymao/sqlglot/commit/9e11654c6ebf7451f14d46c006070effe452519a) - **clickhouse**: add geometry types *(PR [#4278](https://github.com/tobymao/sqlglot/pull/4278) by [@jwhitaker-gridcog](https://github.com/jwhitaker-gridcog))*
### :bug: Bug Fixes
- [`c25a9ab`](https://github.com/tobymao/sqlglot/commit/c25a9ab577d7f0a1056e8afab680ca7801c47fff) - **tsql**: Keep CTE's attached to the query when emulating IF NOT EXISTS *(PR [#4279](https://github.com/tobymao/sqlglot/pull/4279) by [@erindru](https://github.com/erindru))*
- [`768adb3`](https://github.com/tobymao/sqlglot/commit/768adb3d85ed88931d761e5ecc8fb4a3a40d0dc5) - **clickhouse**: time string literals containing fractional seconds *(PR [#4269](https://github.com/tobymao/sqlglot/pull/4269) by [@treysp](https://github.com/treysp))*
### :recycle: Refactors
- [`4d86499`](https://github.com/tobymao/sqlglot/commit/4d8649940d02ac319f2fec372a52674488f01de5) - **diff**: include the target node for Move edits *(PR [#4277](https://github.com/tobymao/sqlglot/pull/4277) by [@georgesittas](https://github.com/georgesittas))*
## [v25.26.0] - 2024-10-21
### :boom: BREAKING CHANGES
- due to [`142c3e7`](https://github.com/tobymao/sqlglot/commit/142c3e75b25374ba9259f21b51cd728bbeb280ef) - Support TO_DOUBLE function *(PR [#4255](https://github.com/tobymao/sqlglot/pull/4255) by [@VaggelisD](https://github.com/VaggelisD))*:
Support TO_DOUBLE function (#4255)
- due to [`13d0696`](https://github.com/tobymao/sqlglot/commit/13d06966a2ca9264f35d5a58e1eaff1baa7dc66e) - Support TRY_TO_TIMESTAMP function *(PR [#4257](https://github.com/tobymao/sqlglot/pull/4257) by [@VaggelisD](https://github.com/VaggelisD))*:
Support TRY_TO_TIMESTAMP function (#4257)
- due to [`7fc0055`](https://github.com/tobymao/sqlglot/commit/7fc0055fb04713ba047baa5eda1ce0baf1cc79e2) - dont parse right-hand side operands of ARRAY JOIN as Tables *(PR [#4258](https://github.com/tobymao/sqlglot/pull/4258) by [@georgesittas](https://github.com/georgesittas))*:
dont parse right-hand side operands of ARRAY JOIN as Tables (#4258)
- due to [`222152e`](https://github.com/tobymao/sqlglot/commit/222152e32521dbc6de3384b18ab4c677239c6088) - Add type hints for optimizer rules eliminate & merge subqueries *(PR [#4267](https://github.com/tobymao/sqlglot/pull/4267) by [@VaggelisD](https://github.com/VaggelisD))*:
Add type hints for optimizer rules eliminate & merge subqueries (#4267)
### :sparkles: New Features
- [`6f32e53`](https://github.com/tobymao/sqlglot/commit/6f32e5348d9aeba9c5d51a892023b2e14e072119) - support non-strict qualify_columns *(PR [#4243](https://github.com/tobymao/sqlglot/pull/4243) by [@hsheth2](https://github.com/hsheth2))*
- [`ed97954`](https://github.com/tobymao/sqlglot/commit/ed97954ecd7c2d7d4fe1bbf2ec0ecc000dd02b32) - **duckdb**: Transpile Spark's LATERAL VIEW EXPLODE *(PR [#4252](https://github.com/tobymao/sqlglot/pull/4252) by [@VaggelisD](https://github.com/VaggelisD))*
- :arrow_lower_right: *addresses issue [#4247](https://github.com/tobymao/sqlglot/issues/4247) opened by [@idanyadgar-clutch](https://github.com/idanyadgar-clutch)*
- [`8f5efc7`](https://github.com/tobymao/sqlglot/commit/8f5efc7bc01ba5923584cd6ef38a4d81e763ccae) - **oracle**: parse hints *(PR [#4249](https://github.com/tobymao/sqlglot/pull/4249) by [@mkmoisen](https://github.com/mkmoisen))*
- [`8b7ff5e`](https://github.com/tobymao/sqlglot/commit/8b7ff5ee8713a3ba50c48addd3700927a0240cf5) - **starrocks**: support for ALTER TABLE SWAP WITH *(PR [#4256](https://github.com/tobymao/sqlglot/pull/4256) by [@mrhamburg](https://github.com/mrhamburg))*
- [`1c43348`](https://github.com/tobymao/sqlglot/commit/1c433487a45379298ef27b3688723df2bd740fd1) - **trino**: Support for LISTAGG function *(PR [#4253](https://github.com/tobymao/sqlglot/pull/4253) by [@VaggelisD](https://github.com/VaggelisD))*
- :arrow_lower_right: *addresses issue [#4250](https://github.com/tobymao/sqlglot/issues/4250) opened by [@npochhi](https://github.com/npochhi)*
- [`142c3e7`](https://github.com/tobymao/sqlglot/commit/142c3e75b25374ba9259f21b51cd728bbeb280ef) - **snowflake**: Support TO_DOUBLE function *(PR [#4255](https://github.com/tobymao/sqlglot/pull/4255) by [@VaggelisD](https://github.com/VaggelisD))*
- [`13d0696`](https://github.com/tobymao/sqlglot/commit/13d06966a2ca9264f35d5a58e1eaff1baa7dc66e) - **snowflake**: Support TRY_TO_TIMESTAMP function *(PR [#4257](https://github.com/tobymao/sqlglot/pull/4257) by [@VaggelisD](https://github.com/VaggelisD))*
- [`04dccf3`](https://github.com/tobymao/sqlglot/commit/04dccf3cdaf1c3a0466dda113aba5439f1639ae0) - **tsql**: Support for stored procedure options *(PR [#4260](https://github.com/tobymao/sqlglot/pull/4260) by [@rsanchez-xtillion](https://github.com/rsanchez-xtillion))*
- [`36f6841`](https://github.com/tobymao/sqlglot/commit/36f68416b3dd0d9ac703dd926d1f74bc43566e0d) - **bigquery**: support EDIT_DISTANCE (Levinshtein) function *(PR [#4276](https://github.com/tobymao/sqlglot/pull/4276) by [@esciara](https://github.com/esciara))*
- :arrow_lower_right: *addresses issue [#4275](https://github.com/tobymao/sqlglot/issues/4275) opened by [@esciara](https://github.com/esciara)*
### :bug: Bug Fixes
- [`fcc05c9`](https://github.com/tobymao/sqlglot/commit/fcc05c9daa31c7a51474ec9c72ceafd682359f90) - **bigquery**: Early expand only aliased names in GROUP BY *(PR [#4246](https://github.com/tobymao/sqlglot/pull/4246) by [@VaggelisD](https://github.com/VaggelisD))*
- [`5655cfb`](https://github.com/tobymao/sqlglot/commit/5655cfba7afdf8f95dea53d5ededfde209b77c30) - add support for negative intervals in to_interval *(commit by [@georgesittas](https://github.com/georgesittas))*
- [`51f4d26`](https://github.com/tobymao/sqlglot/commit/51f4d26ed8694365c61fdefd810a420fcfefdeca) - generate single argument ArrayConcat without trailing comma fixes [#4259](https://github.com/tobymao/sqlglot/pull/4259) *(commit by [@georgesittas](https://github.com/georgesittas))*
- [`7fc0055`](https://github.com/tobymao/sqlglot/commit/7fc0055fb04713ba047baa5eda1ce0baf1cc79e2) - **clickhouse**: dont parse right-hand side operands of ARRAY JOIN as Tables *(PR [#4258](https://github.com/tobymao/sqlglot/pull/4258) by [@georgesittas](https://github.com/georgesittas))*
- :arrow_lower_right: *fixes issue [#4254](https://github.com/tobymao/sqlglot/issues/4254) opened by [@xtess16](https://github.com/xtess16)*
- [`8f49ad8`](https://github.com/tobymao/sqlglot/commit/8f49ad87fa795349183d13129110ad59387bfe11) - **clickhouse**: traverse_scope with FINAL modifier *(PR [#4263](https://github.com/tobymao/sqlglot/pull/4263) by [@pkit](https://github.com/pkit))*
- :arrow_lower_right: *fixes issue [#4262](https://github.com/tobymao/sqlglot/issues/4262) opened by [@obazna](https://github.com/obazna)*
- [`83167ea`](https://github.com/tobymao/sqlglot/commit/83167eaa3039195f756c7b1ad95fc9162f19b1b1) - hive dialect hierarchy has no CURRENT_TIME func *(PR [#4264](https://github.com/tobymao/sqlglot/pull/4264) by [@georgesittas](https://github.com/georgesittas))*
- [`7a5c7e0`](https://github.com/tobymao/sqlglot/commit/7a5c7e036fa84eb30bcae75829f3cb94503fa99e) - **presto**: transpile BIT to BOOLEAN *(commit by [@georgesittas](https://github.com/georgesittas))*
- [`48be3d8`](https://github.com/tobymao/sqlglot/commit/48be3d89b1df96c7b8d81536862f53a98e414f11) - make the semantic diffing aware of changes to non-expression leaves *(PR [#4268](https://github.com/tobymao/sqlglot/pull/4268) by [@georgesittas](https://github.com/georgesittas))*
- [`4543fb3`](https://github.com/tobymao/sqlglot/commit/4543fb3cd052dfb20428f5a6254b38def9e756ee) - **optimizer**: Fix merge_subqueries.py::rename_inner_sources() *(PR [#4266](https://github.com/tobymao/sqlglot/pull/4266) by [@VaggelisD](https://github.com/VaggelisD))*
- :arrow_lower_right: *fixes issue [#4245](https://github.com/tobymao/sqlglot/issues/4245) opened by [@daniel769](https://github.com/daniel769)*
- [`222152e`](https://github.com/tobymao/sqlglot/commit/222152e32521dbc6de3384b18ab4c677239c6088) - **optimizer**: Add type hints for optimizer rules eliminate & merge subqueries *(PR [#4267](https://github.com/tobymao/sqlglot/pull/4267) by [@VaggelisD](https://github.com/VaggelisD))*
### :recycle: Refactors
- [`94013a2`](https://github.com/tobymao/sqlglot/commit/94013a21ca69b90da78dc47b16cd86503736597a) - simplify _expression_only_args helper in diff module *(PR [#4251](https://github.com/tobymao/sqlglot/pull/4251) by [@georgesittas](https://github.com/georgesittas))*
- [`41e2eba`](https://github.com/tobymao/sqlglot/commit/41e2eba1a01c1a5b784ad9dc6c5191f3d3bc0d74) - **Oracle**: simplify hint arg formatting *(commit by [@georgesittas](https://github.com/georgesittas))*
- [`cfd692f`](https://github.com/tobymao/sqlglot/commit/cfd692ff28a59f413671aafbc8dcd61eab3558c3) - move SwapTable logic to the base Parser/Generator classes *(commit by [@georgesittas](https://github.com/georgesittas))*
## [v25.25.1] - 2024-10-15
### :bug: Bug Fixes
- [`e6567ae`](https://github.com/tobymao/sqlglot/commit/e6567ae11650834874808a844a19836fbb9ee753) - small overload fix for ensure list taking None *(PR [#4248](https://github.com/tobymao/sqlglot/pull/4248) by [@benfdking](https://github.com/benfdking))*
@ -5038,3 +5150,6 @@ Changelog
[v25.24.5]: https://github.com/tobymao/sqlglot/compare/v25.24.4...v25.24.5
[v25.25.0]: https://github.com/tobymao/sqlglot/compare/v25.24.5...v25.25.0
[v25.25.1]: https://github.com/tobymao/sqlglot/compare/v25.25.0...v25.25.1
[v25.26.0]: https://github.com/tobymao/sqlglot/compare/v25.25.1...v25.26.0
[v25.27.0]: https://github.com/tobymao/sqlglot/compare/v25.26.0...v25.27.0
[v25.28.0]: https://github.com/tobymao/sqlglot/compare/v25.27.0...v25.28.0

4
benchmarks/optimize.py
View file

@ -18,12 +18,12 @@ def gen_condition(n):
BENCHMARKS = {
"tpch": lambda: (
[parse_one(sql) for _, sql, _ in load_sql_fixture_pairs(f"optimizer/tpc-h/tpc-h.sql")],
[parse_one(sql) for _, sql, _ in load_sql_fixture_pairs("optimizer/tpc-h/tpc-h.sql")],
TPCH_SCHEMA,
3,
),
"tpcds": lambda: (
[parse_one(sql) for _, sql, _ in load_sql_fixture_pairs(f"optimizer/tpc-ds/tpc-ds.sql")],
[parse_one(sql) for _, sql, _ in load_sql_fixture_pairs("optimizer/tpc-ds/tpc-ds.sql")],
TPCDS_SCHEMA,
3,
),

2
docs/search.js
View file

File diff suppressed because one or more lines are too long

8
docs/sqlglot/_version.html
View file

@ -76,8 +76,8 @@
</span><span id="L-12"><a href="#L-12"><span class="linenos">12</span></a><span class="n">__version_tuple__</span><span class="p">:</span> <span class="n">VERSION_TUPLE</span>
</span><span id="L-13"><a href="#L-13"><span class="linenos">13</span></a><span class="n">version_tuple</span><span class="p">:</span> <span class="n">VERSION_TUPLE</span>
</span><span id="L-14"><a href="#L-14"><span class="linenos">14</span></a>
</span><span id="L-15"><a href="#L-15"><span class="linenos">15</span></a><span class="n">__version__</span> <span class="o">=</span> <span class="n">version</span> <span class="o">=</span> <span class="s1">&#39;25.25.1&#39;</span>
</span><span id="L-16"><a href="#L-16"><span class="linenos">16</span></a><span class="n">__version_tuple__</span> <span class="o">=</span> <span class="n">version_tuple</span> <span class="o">=</span> <span class="p">(</span><span class="mi">25</span><span class="p">,</span> <span class="mi">25</span><span class="p">,</span> <span class="mi">1</span><span class="p">)</span>
</span><span id="L-15"><a href="#L-15"><span class="linenos">15</span></a><span class="n">__version__</span> <span class="o">=</span> <span class="n">version</span> <span class="o">=</span> <span class="s1">&#39;25.28.0&#39;</span>
</span><span id="L-16"><a href="#L-16"><span class="linenos">16</span></a><span class="n">__version_tuple__</span> <span class="o">=</span> <span class="n">version_tuple</span> <span class="o">=</span> <span class="p">(</span><span class="mi">25</span><span class="p">,</span> <span class="mi">28</span><span class="p">,</span> <span class="mi">0</span><span class="p">)</span>
</span></pre></div>
@ -97,7 +97,7 @@
<section id="version">
<div class="attr variable">
<span class="name">version</span><span class="annotation">: str</span> =
<span class="default_value">&#39;25.25.1&#39;</span>
<span class="default_value">&#39;25.28.0&#39;</span>
</div>
@ -109,7 +109,7 @@
<section id="version_tuple">
<div class="attr variable">
<span class="name">version_tuple</span><span class="annotation">: object</span> =
<span class="default_value">(25, 25, 1)</span>
<span class="default_value">(25, 28, 0)</span>
</div>

17
docs/sqlglot/dialects/athena.html
View file

File diff suppressed because one or more lines are too long

3785
docs/sqlglot/dialects/bigquery.html
View file

File diff suppressed because one or more lines are too long

6769
docs/sqlglot/dialects/clickhouse.html
View file

File diff suppressed because one or more lines are too long

14
docs/sqlglot/dialects/databricks.html
View file

File diff suppressed because one or more lines are too long

4476
docs/sqlglot/dialects/dialect.html
View file

File diff suppressed because one or more lines are too long

12
docs/sqlglot/dialects/doris.html
View file

File diff suppressed because one or more lines are too long

18
docs/sqlglot/dialects/drill.html
View file

File diff suppressed because one or more lines are too long

4874
docs/sqlglot/dialects/duckdb.html
View file

File diff suppressed because one or more lines are too long

2406
docs/sqlglot/dialects/hive.html
View file

File diff suppressed because one or more lines are too long

10
docs/sqlglot/dialects/materialize.html
View file

File diff suppressed because one or more lines are too long

36
docs/sqlglot/dialects/mysql.html
View file

File diff suppressed because one or more lines are too long

2273
docs/sqlglot/dialects/oracle.html
View file

File diff suppressed because one or more lines are too long

20
docs/sqlglot/dialects/postgres.html
View file

File diff suppressed because one or more lines are too long

3507
docs/sqlglot/dialects/presto.html
View file

File diff suppressed because one or more lines are too long

12
docs/sqlglot/dialects/prql.html
View file

File diff suppressed because one or more lines are too long

18
docs/sqlglot/dialects/redshift.html
View file

File diff suppressed because one or more lines are too long

2
docs/sqlglot/dialects/risingwave.html
View file

@ -1104,6 +1104,8 @@ Default: True</li>
<dd id="RisingWave.Generator.grantprincipal_sql" class="function"><a href="../generator.html#Generator.grantprincipal_sql">grantprincipal_sql</a></dd>
<dd id="RisingWave.Generator.columns_sql" class="function"><a href="../generator.html#Generator.columns_sql">columns_sql</a></dd>
<dd id="RisingWave.Generator.overlay_sql" class="function"><a href="../generator.html#Generator.overlay_sql">overlay_sql</a></dd>
<dd id="RisingWave.Generator.todouble_sql" class="function"><a href="../generator.html#Generator.todouble_sql">todouble_sql</a></dd>
<dd id="RisingWave.Generator.string_sql" class="function"><a href="../generator.html#Generator.string_sql">string_sql</a></dd>
</div>
<div><dt><a href="postgres.html#Postgres.Generator">sqlglot.dialects.postgres.Postgres.Generator</a></dt>

5682
docs/sqlglot/dialects/snowflake.html
View file

File diff suppressed because one or more lines are too long

16
docs/sqlglot/dialects/spark.html
View file

File diff suppressed because one or more lines are too long

20
docs/sqlglot/dialects/spark2.html
View file

File diff suppressed because one or more lines are too long

1176
docs/sqlglot/dialects/sqlite.html
View file

File diff suppressed because one or more lines are too long

14
docs/sqlglot/dialects/starrocks.html
View file

File diff suppressed because one or more lines are too long

16
docs/sqlglot/dialects/tableau.html
View file

File diff suppressed because one or more lines are too long

22
docs/sqlglot/dialects/teradata.html
View file

File diff suppressed because one or more lines are too long

582
docs/sqlglot/dialects/trino.html
View file

File diff suppressed because one or more lines are too long

4390
docs/sqlglot/dialects/tsql.html
View file

File diff suppressed because one or more lines are too long

1350
docs/sqlglot/diff.html
View file

File diff suppressed because it is too large Load diff

6
docs/sqlglot/executor/python.html
View file

File diff suppressed because one or more lines are too long

20667
docs/sqlglot/expressions.html
View file

File diff suppressed because one or more lines are too long

22893
docs/sqlglot/generator.html
View file

File diff suppressed because one or more lines are too long

2
docs/sqlglot/helper.html
View file

@ -1874,7 +1874,7 @@ belong to some totally-ordered set.</p>
<section id="DATE_UNITS">
<div class="attr variable">
<span class="name">DATE_UNITS</span> =
<span class="default_value">{&#39;year&#39;, &#39;month&#39;, &#39;week&#39;, &#39;day&#39;, &#39;quarter&#39;, &#39;year_month&#39;}</span>
<span class="default_value">{&#39;year_month&#39;, &#39;quarter&#39;, &#39;week&#39;, &#39;day&#39;, &#39;month&#39;, &#39;year&#39;}</span>
</div>

2
docs/sqlglot/jsonpath.html
View file

@ -586,7 +586,7 @@
<div class="attr variable">
<span class="name">ALL_JSON_PATH_PARTS</span> =
<input id="ALL_JSON_PATH_PARTS-view-value" class="view-value-toggle-state" type="checkbox" aria-hidden="true" tabindex="-1">
<label class="view-value-button pdoc-button" for="ALL_JSON_PATH_PARTS-view-value"></label><span class="default_value">{&lt;class &#39;<a href="expressions.html#JSONPathRecursive">sqlglot.expressions.JSONPathRecursive</a>&#39;&gt;, &lt;class &#39;<a href="expressions.html#JSONPathWildcard">sqlglot.expressions.JSONPathWildcard</a>&#39;&gt;, &lt;class &#39;<a href="expressions.html#JSONPathKey">sqlglot.expressions.JSONPathKey</a>&#39;&gt;, &lt;class &#39;<a href="expressions.html#JSONPathFilter">sqlglot.expressions.JSONPathFilter</a>&#39;&gt;, &lt;class &#39;<a href="expressions.html#JSONPathUnion">sqlglot.expressions.JSONPathUnion</a>&#39;&gt;, &lt;class &#39;<a href="expressions.html#JSONPathSubscript">sqlglot.expressions.JSONPathSubscript</a>&#39;&gt;, &lt;class &#39;<a href="expressions.html#JSONPathSelector">sqlglot.expressions.JSONPathSelector</a>&#39;&gt;, &lt;class &#39;<a href="expressions.html#JSONPathSlice">sqlglot.expressions.JSONPathSlice</a>&#39;&gt;, &lt;class &#39;<a href="expressions.html#JSONPathScript">sqlglot.expressions.JSONPathScript</a>&#39;&gt;, &lt;class &#39;<a href="expressions.html#JSONPathRoot">sqlglot.expressions.JSONPathRoot</a>&#39;&gt;}</span>
<label class="view-value-button pdoc-button" for="ALL_JSON_PATH_PARTS-view-value"></label><span class="default_value">{&lt;class &#39;<a href="expressions.html#JSONPathSubscript">sqlglot.expressions.JSONPathSubscript</a>&#39;&gt;, &lt;class &#39;<a href="expressions.html#JSONPathWildcard">sqlglot.expressions.JSONPathWildcard</a>&#39;&gt;, &lt;class &#39;<a href="expressions.html#JSONPathSelector">sqlglot.expressions.JSONPathSelector</a>&#39;&gt;, &lt;class &#39;<a href="expressions.html#JSONPathSlice">sqlglot.expressions.JSONPathSlice</a>&#39;&gt;, &lt;class &#39;<a href="expressions.html#JSONPathScript">sqlglot.expressions.JSONPathScript</a>&#39;&gt;, &lt;class &#39;<a href="expressions.html#JSONPathRoot">sqlglot.expressions.JSONPathRoot</a>&#39;&gt;, &lt;class &#39;<a href="expressions.html#JSONPathUnion">sqlglot.expressions.JSONPathUnion</a>&#39;&gt;, &lt;class &#39;<a href="expressions.html#JSONPathRecursive">sqlglot.expressions.JSONPathRecursive</a>&#39;&gt;, &lt;class &#39;<a href="expressions.html#JSONPathKey">sqlglot.expressions.JSONPathKey</a>&#39;&gt;, &lt;class &#39;<a href="expressions.html#JSONPathFilter">sqlglot.expressions.JSONPathFilter</a>&#39;&gt;}</span>
</div>

1573
docs/sqlglot/optimizer/annotate_types.html
View file

File diff suppressed because one or more lines are too long

525
docs/sqlglot/optimizer/eliminate_subqueries.html
View file

@ -56,178 +56,195 @@
<label class="view-source-button" for="mod-eliminate_subqueries-view-source"><span>View Source</span></label>
<div class="pdoc-code codehilite"><pre><span></span><span id="L-1"><a href="#L-1"><span class="linenos"> 1</span></a><span class="kn">import</span> <span class="nn">itertools</span>
<div class="pdoc-code codehilite"><pre><span></span><span id="L-1"><a href="#L-1"><span class="linenos"> 1</span></a><span class="kn">from</span> <span class="nn">__future__</span> <span class="kn">import</span> <span class="n">annotations</span>
</span><span id="L-2"><a href="#L-2"><span class="linenos"> 2</span></a>
</span><span id="L-3"><a href="#L-3"><span class="linenos"> 3</span></a><span class="kn">from</span> <span class="nn">sqlglot</span> <span class="kn">import</span> <span class="n">expressions</span> <span class="k">as</span> <span class="n">exp</span>
</span><span id="L-4"><a href="#L-4"><span class="linenos"> 4</span></a><span class="kn">from</span> <span class="nn">sqlglot.helper</span> <span class="kn">import</span> <span class="n">find_new_name</span>
</span><span id="L-5"><a href="#L-5"><span class="linenos"> 5</span></a><span class="kn">from</span> <span class="nn">sqlglot.optimizer.scope</span> <span class="kn">import</span> <span class="n">build_scope</span>
</span><span id="L-6"><a href="#L-6"><span class="linenos"> 6</span></a>
</span><span id="L-7"><a href="#L-7"><span class="linenos"> 7</span></a>
</span><span id="L-8"><a href="#L-8"><span class="linenos"> 8</span></a><span class="k">def</span> <span class="nf">eliminate_subqueries</span><span class="p">(</span><span class="n">expression</span><span class="p">):</span>
</span><span id="L-9"><a href="#L-9"><span class="linenos"> 9</span></a><span class="w"> </span><span class="sd">&quot;&quot;&quot;</span>
</span><span id="L-10"><a href="#L-10"><span class="linenos"> 10</span></a><span class="sd"> Rewrite derived tables as CTES, deduplicating if possible.</span>
</span><span id="L-11"><a href="#L-11"><span class="linenos"> 11</span></a>
</span><span id="L-12"><a href="#L-12"><span class="linenos"> 12</span></a><span class="sd"> Example:</span>
</span><span id="L-13"><a href="#L-13"><span class="linenos"> 13</span></a><span class="sd"> &gt;&gt;&gt; import sqlglot</span>
</span><span id="L-14"><a href="#L-14"><span class="linenos"> 14</span></a><span class="sd"> &gt;&gt;&gt; expression = sqlglot.parse_one(&quot;SELECT a FROM (SELECT * FROM x) AS y&quot;)</span>
</span><span id="L-15"><a href="#L-15"><span class="linenos"> 15</span></a><span class="sd"> &gt;&gt;&gt; eliminate_subqueries(expression).sql()</span>
</span><span id="L-16"><a href="#L-16"><span class="linenos"> 16</span></a><span class="sd"> &#39;WITH y AS (SELECT * FROM x) SELECT a FROM y AS y&#39;</span>
</span><span id="L-17"><a href="#L-17"><span class="linenos"> 17</span></a>
</span><span id="L-18"><a href="#L-18"><span class="linenos"> 18</span></a><span class="sd"> This also deduplicates common subqueries:</span>
</span><span id="L-19"><a href="#L-19"><span class="linenos"> 19</span></a><span class="sd"> &gt;&gt;&gt; expression = sqlglot.parse_one(&quot;SELECT a FROM (SELECT * FROM x) AS y CROSS JOIN (SELECT * FROM x) AS z&quot;)</span>
</span><span id="L-20"><a href="#L-20"><span class="linenos"> 20</span></a><span class="sd"> &gt;&gt;&gt; eliminate_subqueries(expression).sql()</span>
</span><span id="L-21"><a href="#L-21"><span class="linenos"> 21</span></a><span class="sd"> &#39;WITH y AS (SELECT * FROM x) SELECT a FROM y AS y CROSS JOIN y AS z&#39;</span>
</span><span id="L-22"><a href="#L-22"><span class="linenos"> 22</span></a>
</span><span id="L-23"><a href="#L-23"><span class="linenos"> 23</span></a><span class="sd"> Args:</span>
</span><span id="L-24"><a href="#L-24"><span class="linenos"> 24</span></a><span class="sd"> expression (sqlglot.Expression): expression</span>
</span><span id="L-25"><a href="#L-25"><span class="linenos"> 25</span></a><span class="sd"> Returns:</span>
</span><span id="L-26"><a href="#L-26"><span class="linenos"> 26</span></a><span class="sd"> sqlglot.Expression: expression</span>
</span><span id="L-27"><a href="#L-27"><span class="linenos"> 27</span></a><span class="sd"> &quot;&quot;&quot;</span>
</span><span id="L-28"><a href="#L-28"><span class="linenos"> 28</span></a> <span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">expression</span><span class="p">,</span> <span class="n">exp</span><span class="o">.</span><span class="n">Subquery</span><span class="p">):</span>
</span><span id="L-29"><a href="#L-29"><span class="linenos"> 29</span></a> <span class="c1"># It&#39;s possible to have subqueries at the root, e.g. (SELECT * FROM x) LIMIT 1</span>
</span><span id="L-30"><a href="#L-30"><span class="linenos"> 30</span></a> <span class="n">eliminate_subqueries</span><span class="p">(</span><span class="n">expression</span><span class="o">.</span><span class="n">this</span><span class="p">)</span>
</span><span id="L-31"><a href="#L-31"><span class="linenos"> 31</span></a> <span class="k">return</span> <span class="n">expression</span>
</span><span id="L-32"><a href="#L-32"><span class="linenos"> 32</span></a>
</span><span id="L-33"><a href="#L-33"><span class="linenos"> 33</span></a> <span class="n">root</span> <span class="o">=</span> <span class="n">build_scope</span><span class="p">(</span><span class="n">expression</span><span class="p">)</span>
</span><span id="L-34"><a href="#L-34"><span class="linenos"> 34</span></a>
</span><span id="L-35"><a href="#L-35"><span class="linenos"> 35</span></a> <span class="k">if</span> <span class="ow">not</span> <span class="n">root</span><span class="p">:</span>
</span><span id="L-36"><a href="#L-36"><span class="linenos"> 36</span></a> <span class="k">return</span> <span class="n">expression</span>
</span><span id="L-37"><a href="#L-37"><span class="linenos"> 37</span></a>
</span><span id="L-38"><a href="#L-38"><span class="linenos"> 38</span></a> <span class="c1"># Map of alias-&gt;Scope|Table</span>
</span><span id="L-39"><a href="#L-39"><span class="linenos"> 39</span></a> <span class="c1"># These are all aliases that are already used in the expression.</span>
</span><span id="L-40"><a href="#L-40"><span class="linenos"> 40</span></a> <span class="c1"># We don&#39;t want to create new CTEs that conflict with these names.</span>
</span><span id="L-41"><a href="#L-41"><span class="linenos"> 41</span></a> <span class="n">taken</span> <span class="o">=</span> <span class="p">{}</span>
</span><span id="L-42"><a href="#L-42"><span class="linenos"> 42</span></a>
</span><span id="L-43"><a href="#L-43"><span class="linenos"> 43</span></a> <span class="c1"># All CTE aliases in the root scope are taken</span>
</span><span id="L-44"><a href="#L-44"><span class="linenos"> 44</span></a> <span class="k">for</span> <span class="n">scope</span> <span class="ow">in</span> <span class="n">root</span><span class="o">.</span><span class="n">cte_scopes</span><span class="p">:</span>
</span><span id="L-45"><a href="#L-45"><span class="linenos"> 45</span></a> <span class="n">taken</span><span class="p">[</span><span class="n">scope</span><span class="o">.</span><span class="n">expression</span><span class="o">.</span><span class="n">parent</span><span class="o">.</span><span class="n">alias</span><span class="p">]</span> <span class="o">=</span> <span class="n">scope</span>
</span><span id="L-46"><a href="#L-46"><span class="linenos"> 46</span></a>
</span><span id="L-47"><a href="#L-47"><span class="linenos"> 47</span></a> <span class="c1"># All table names are taken</span>
</span><span id="L-48"><a href="#L-48"><span class="linenos"> 48</span></a> <span class="k">for</span> <span class="n">scope</span> <span class="ow">in</span> <span class="n">root</span><span class="o">.</span><span class="n">traverse</span><span class="p">():</span>
</span><span id="L-49"><a href="#L-49"><span class="linenos"> 49</span></a> <span class="n">taken</span><span class="o">.</span><span class="n">update</span><span class="p">(</span>
</span><span id="L-50"><a href="#L-50"><span class="linenos"> 50</span></a> <span class="p">{</span>
</span><span id="L-51"><a href="#L-51"><span class="linenos"> 51</span></a> <span class="n">source</span><span class="o">.</span><span class="n">name</span><span class="p">:</span> <span class="n">source</span>
</span><span id="L-52"><a href="#L-52"><span class="linenos"> 52</span></a> <span class="k">for</span> <span class="n">_</span><span class="p">,</span> <span class="n">source</span> <span class="ow">in</span> <span class="n">scope</span><span class="o">.</span><span class="n">sources</span><span class="o">.</span><span class="n">items</span><span class="p">()</span>
</span><span id="L-53"><a href="#L-53"><span class="linenos"> 53</span></a> <span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">source</span><span class="p">,</span> <span class="n">exp</span><span class="o">.</span><span class="n">Table</span><span class="p">)</span>
</span><span id="L-54"><a href="#L-54"><span class="linenos"> 54</span></a> <span class="p">}</span>
</span><span id="L-55"><a href="#L-55"><span class="linenos"> 55</span></a> <span class="p">)</span>
</span><span id="L-56"><a href="#L-56"><span class="linenos"> 56</span></a>
</span><span id="L-57"><a href="#L-57"><span class="linenos"> 57</span></a> <span class="c1"># Map of Expression-&gt;alias</span>
</span><span id="L-58"><a href="#L-58"><span class="linenos"> 58</span></a> <span class="c1"># Existing CTES in the root expression. We&#39;ll use this for deduplication.</span>
</span><span id="L-59"><a href="#L-59"><span class="linenos"> 59</span></a> <span class="n">existing_ctes</span> <span class="o">=</span> <span class="p">{}</span>
</span><span id="L-60"><a href="#L-60"><span class="linenos"> 60</span></a>
</span><span id="L-61"><a href="#L-61"><span class="linenos"> 61</span></a> <span class="n">with_</span> <span class="o">=</span> <span class="n">root</span><span class="o">.</span><span class="n">expression</span><span class="o">.</span><span class="n">args</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s2">&quot;with&quot;</span><span class="p">)</span>
</span><span id="L-62"><a href="#L-62"><span class="linenos"> 62</span></a> <span class="n">recursive</span> <span class="o">=</span> <span class="kc">False</span>
</span><span id="L-63"><a href="#L-63"><span class="linenos"> 63</span></a> <span class="k">if</span> <span class="n">with_</span><span class="p">:</span>
</span><span id="L-64"><a href="#L-64"><span class="linenos"> 64</span></a> <span class="n">recursive</span> <span class="o">=</span> <span class="n">with_</span><span class="o">.</span><span class="n">args</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s2">&quot;recursive&quot;</span><span class="p">)</span>
</span><span id="L-65"><a href="#L-65"><span class="linenos"> 65</span></a> <span class="k">for</span> <span class="n">cte</span> <span class="ow">in</span> <span class="n">with_</span><span class="o">.</span><span class="n">expressions</span><span class="p">:</span>
</span><span id="L-66"><a href="#L-66"><span class="linenos"> 66</span></a> <span class="n">existing_ctes</span><span class="p">[</span><span class="n">cte</span><span class="o">.</span><span class="n">this</span><span class="p">]</span> <span class="o">=</span> <span class="n">cte</span><span class="o">.</span><span class="n">alias</span>
</span><span id="L-67"><a href="#L-67"><span class="linenos"> 67</span></a> <span class="n">new_ctes</span> <span class="o">=</span> <span class="p">[]</span>
</span><span id="L-68"><a href="#L-68"><span class="linenos"> 68</span></a>
</span><span id="L-69"><a href="#L-69"><span class="linenos"> 69</span></a> <span class="c1"># We&#39;re adding more CTEs, but we want to maintain the DAG order.</span>
</span><span id="L-70"><a href="#L-70"><span class="linenos"> 70</span></a> <span class="c1"># Derived tables within an existing CTE need to come before the existing CTE.</span>
</span><span id="L-71"><a href="#L-71"><span class="linenos"> 71</span></a> <span class="k">for</span> <span class="n">cte_scope</span> <span class="ow">in</span> <span class="n">root</span><span class="o">.</span><span class="n">cte_scopes</span><span class="p">:</span>
</span><span id="L-72"><a href="#L-72"><span class="linenos"> 72</span></a> <span class="c1"># Append all the new CTEs from this existing CTE</span>
</span><span id="L-73"><a href="#L-73"><span class="linenos"> 73</span></a> <span class="k">for</span> <span class="n">scope</span> <span class="ow">in</span> <span class="n">cte_scope</span><span class="o">.</span><span class="n">traverse</span><span class="p">():</span>
</span><span id="L-74"><a href="#L-74"><span class="linenos"> 74</span></a> <span class="k">if</span> <span class="n">scope</span> <span class="ow">is</span> <span class="n">cte_scope</span><span class="p">:</span>
</span><span id="L-75"><a href="#L-75"><span class="linenos"> 75</span></a> <span class="c1"># Don&#39;t try to eliminate this CTE itself</span>
</span><span id="L-76"><a href="#L-76"><span class="linenos"> 76</span></a> <span class="k">continue</span>
</span><span id="L-77"><a href="#L-77"><span class="linenos"> 77</span></a> <span class="n">new_cte</span> <span class="o">=</span> <span class="n">_eliminate</span><span class="p">(</span><span class="n">scope</span><span class="p">,</span> <span class="n">existing_ctes</span><span class="p">,</span> <span class="n">taken</span><span class="p">)</span>
</span><span id="L-78"><a href="#L-78"><span class="linenos"> 78</span></a> <span class="k">if</span> <span class="n">new_cte</span><span class="p">:</span>
</span><span id="L-79"><a href="#L-79"><span class="linenos"> 79</span></a> <span class="n">new_ctes</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">new_cte</span><span class="p">)</span>
</span><span id="L-80"><a href="#L-80"><span class="linenos"> 80</span></a>
</span><span id="L-81"><a href="#L-81"><span class="linenos"> 81</span></a> <span class="c1"># Append the existing CTE itself</span>
</span><span id="L-82"><a href="#L-82"><span class="linenos"> 82</span></a> <span class="n">new_ctes</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">cte_scope</span><span class="o">.</span><span class="n">expression</span><span class="o">.</span><span class="n">parent</span><span class="p">)</span>
</span><span id="L-83"><a href="#L-83"><span class="linenos"> 83</span></a>
</span><span id="L-84"><a href="#L-84"><span class="linenos"> 84</span></a> <span class="c1"># Now append the rest</span>
</span><span id="L-85"><a href="#L-85"><span class="linenos"> 85</span></a> <span class="k">for</span> <span class="n">scope</span> <span class="ow">in</span> <span class="n">itertools</span><span class="o">.</span><span class="n">chain</span><span class="p">(</span><span class="n">root</span><span class="o">.</span><span class="n">union_scopes</span><span class="p">,</span> <span class="n">root</span><span class="o">.</span><span class="n">subquery_scopes</span><span class="p">,</span> <span class="n">root</span><span class="o">.</span><span class="n">table_scopes</span><span class="p">):</span>
</span><span id="L-86"><a href="#L-86"><span class="linenos"> 86</span></a> <span class="k">for</span> <span class="n">child_scope</span> <span class="ow">in</span> <span class="n">scope</span><span class="o">.</span><span class="n">traverse</span><span class="p">():</span>
</span><span id="L-87"><a href="#L-87"><span class="linenos"> 87</span></a> <span class="n">new_cte</span> <span class="o">=</span> <span class="n">_eliminate</span><span class="p">(</span><span class="n">child_scope</span><span class="p">,</span> <span class="n">existing_ctes</span><span class="p">,</span> <span class="n">taken</span><span class="p">)</span>
</span><span id="L-88"><a href="#L-88"><span class="linenos"> 88</span></a> <span class="k">if</span> <span class="n">new_cte</span><span class="p">:</span>
</span><span id="L-89"><a href="#L-89"><span class="linenos"> 89</span></a> <span class="n">new_ctes</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">new_cte</span><span class="p">)</span>
</span><span id="L-3"><a href="#L-3"><span class="linenos"> 3</span></a><span class="kn">import</span> <span class="nn">itertools</span>
</span><span id="L-4"><a href="#L-4"><span class="linenos"> 4</span></a><span class="kn">import</span> <span class="nn">typing</span> <span class="k">as</span> <span class="nn">t</span>
</span><span id="L-5"><a href="#L-5"><span class="linenos"> 5</span></a>
</span><span id="L-6"><a href="#L-6"><span class="linenos"> 6</span></a><span class="kn">from</span> <span class="nn">sqlglot</span> <span class="kn">import</span> <span class="n">expressions</span> <span class="k">as</span> <span class="n">exp</span>
</span><span id="L-7"><a href="#L-7"><span class="linenos"> 7</span></a><span class="kn">from</span> <span class="nn">sqlglot.helper</span> <span class="kn">import</span> <span class="n">find_new_name</span>
</span><span id="L-8"><a href="#L-8"><span class="linenos"> 8</span></a><span class="kn">from</span> <span class="nn">sqlglot.optimizer.scope</span> <span class="kn">import</span> <span class="n">Scope</span><span class="p">,</span> <span class="n">build_scope</span>
</span><span id="L-9"><a href="#L-9"><span class="linenos"> 9</span></a>
</span><span id="L-10"><a href="#L-10"><span class="linenos"> 10</span></a><span class="k">if</span> <span class="n">t</span><span class="o">.</span><span class="n">TYPE_CHECKING</span><span class="p">:</span>
</span><span id="L-11"><a href="#L-11"><span class="linenos"> 11</span></a> <span class="n">ExistingCTEsMapping</span> <span class="o">=</span> <span class="n">t</span><span class="o">.</span><span class="n">Dict</span><span class="p">[</span><span class="n">exp</span><span class="o">.</span><span class="n">Expression</span><span class="p">,</span> <span class="nb">str</span><span class="p">]</span>
</span><span id="L-12"><a href="#L-12"><span class="linenos"> 12</span></a> <span class="n">TakenNameMapping</span> <span class="o">=</span> <span class="n">t</span><span class="o">.</span><span class="n">Dict</span><span class="p">[</span><span class="nb">str</span><span class="p">,</span> <span class="n">t</span><span class="o">.</span><span class="n">Union</span><span class="p">[</span><span class="n">Scope</span><span class="p">,</span> <span class="n">exp</span><span class="o">.</span><span class="n">Expression</span><span class="p">]]</span>
</span><span id="L-13"><a href="#L-13"><span class="linenos"> 13</span></a>
</span><span id="L-14"><a href="#L-14"><span class="linenos"> 14</span></a>
</span><span id="L-15"><a href="#L-15"><span class="linenos"> 15</span></a><span class="k">def</span> <span class="nf">eliminate_subqueries</span><span class="p">(</span><span class="n">expression</span><span class="p">:</span> <span class="n">exp</span><span class="o">.</span><span class="n">Expression</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="n">exp</span><span class="o">.</span><span class="n">Expression</span><span class="p">:</span>
</span><span id="L-16"><a href="#L-16"><span class="linenos"> 16</span></a><span class="w"> </span><span class="sd">&quot;&quot;&quot;</span>
</span><span id="L-17"><a href="#L-17"><span class="linenos"> 17</span></a><span class="sd"> Rewrite derived tables as CTES, deduplicating if possible.</span>
</span><span id="L-18"><a href="#L-18"><span class="linenos"> 18</span></a>
</span><span id="L-19"><a href="#L-19"><span class="linenos"> 19</span></a><span class="sd"> Example:</span>
</span><span id="L-20"><a href="#L-20"><span class="linenos"> 20</span></a><span class="sd"> &gt;&gt;&gt; import sqlglot</span>
</span><span id="L-21"><a href="#L-21"><span class="linenos"> 21</span></a><span class="sd"> &gt;&gt;&gt; expression = sqlglot.parse_one(&quot;SELECT a FROM (SELECT * FROM x) AS y&quot;)</span>
</span><span id="L-22"><a href="#L-22"><span class="linenos"> 22</span></a><span class="sd"> &gt;&gt;&gt; eliminate_subqueries(expression).sql()</span>
</span><span id="L-23"><a href="#L-23"><span class="linenos"> 23</span></a><span class="sd"> &#39;WITH y AS (SELECT * FROM x) SELECT a FROM y AS y&#39;</span>
</span><span id="L-24"><a href="#L-24"><span class="linenos"> 24</span></a>
</span><span id="L-25"><a href="#L-25"><span class="linenos"> 25</span></a><span class="sd"> This also deduplicates common subqueries:</span>
</span><span id="L-26"><a href="#L-26"><span class="linenos"> 26</span></a><span class="sd"> &gt;&gt;&gt; expression = sqlglot.parse_one(&quot;SELECT a FROM (SELECT * FROM x) AS y CROSS JOIN (SELECT * FROM x) AS z&quot;)</span>
</span><span id="L-27"><a href="#L-27"><span class="linenos"> 27</span></a><span class="sd"> &gt;&gt;&gt; eliminate_subqueries(expression).sql()</span>
</span><span id="L-28"><a href="#L-28"><span class="linenos"> 28</span></a><span class="sd"> &#39;WITH y AS (SELECT * FROM x) SELECT a FROM y AS y CROSS JOIN y AS z&#39;</span>
</span><span id="L-29"><a href="#L-29"><span class="linenos"> 29</span></a>
</span><span id="L-30"><a href="#L-30"><span class="linenos"> 30</span></a><span class="sd"> Args:</span>
</span><span id="L-31"><a href="#L-31"><span class="linenos"> 31</span></a><span class="sd"> expression (sqlglot.Expression): expression</span>
</span><span id="L-32"><a href="#L-32"><span class="linenos"> 32</span></a><span class="sd"> Returns:</span>
</span><span id="L-33"><a href="#L-33"><span class="linenos"> 33</span></a><span class="sd"> sqlglot.Expression: expression</span>
</span><span id="L-34"><a href="#L-34"><span class="linenos"> 34</span></a><span class="sd"> &quot;&quot;&quot;</span>
</span><span id="L-35"><a href="#L-35"><span class="linenos"> 35</span></a> <span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">expression</span><span class="p">,</span> <span class="n">exp</span><span class="o">.</span><span class="n">Subquery</span><span class="p">):</span>
</span><span id="L-36"><a href="#L-36"><span class="linenos"> 36</span></a> <span class="c1"># It&#39;s possible to have subqueries at the root, e.g. (SELECT * FROM x) LIMIT 1</span>
</span><span id="L-37"><a href="#L-37"><span class="linenos"> 37</span></a> <span class="n">eliminate_subqueries</span><span class="p">(</span><span class="n">expression</span><span class="o">.</span><span class="n">this</span><span class="p">)</span>
</span><span id="L-38"><a href="#L-38"><span class="linenos"> 38</span></a> <span class="k">return</span> <span class="n">expression</span>
</span><span id="L-39"><a href="#L-39"><span class="linenos"> 39</span></a>
</span><span id="L-40"><a href="#L-40"><span class="linenos"> 40</span></a> <span class="n">root</span> <span class="o">=</span> <span class="n">build_scope</span><span class="p">(</span><span class="n">expression</span><span class="p">)</span>
</span><span id="L-41"><a href="#L-41"><span class="linenos"> 41</span></a>
</span><span id="L-42"><a href="#L-42"><span class="linenos"> 42</span></a> <span class="k">if</span> <span class="ow">not</span> <span class="n">root</span><span class="p">:</span>
</span><span id="L-43"><a href="#L-43"><span class="linenos"> 43</span></a> <span class="k">return</span> <span class="n">expression</span>
</span><span id="L-44"><a href="#L-44"><span class="linenos"> 44</span></a>
</span><span id="L-45"><a href="#L-45"><span class="linenos"> 45</span></a> <span class="c1"># Map of alias-&gt;Scope|Table</span>
</span><span id="L-46"><a href="#L-46"><span class="linenos"> 46</span></a> <span class="c1"># These are all aliases that are already used in the expression.</span>
</span><span id="L-47"><a href="#L-47"><span class="linenos"> 47</span></a> <span class="c1"># We don&#39;t want to create new CTEs that conflict with these names.</span>
</span><span id="L-48"><a href="#L-48"><span class="linenos"> 48</span></a> <span class="n">taken</span><span class="p">:</span> <span class="n">TakenNameMapping</span> <span class="o">=</span> <span class="p">{}</span>
</span><span id="L-49"><a href="#L-49"><span class="linenos"> 49</span></a>
</span><span id="L-50"><a href="#L-50"><span class="linenos"> 50</span></a> <span class="c1"># All CTE aliases in the root scope are taken</span>
</span><span id="L-51"><a href="#L-51"><span class="linenos"> 51</span></a> <span class="k">for</span> <span class="n">scope</span> <span class="ow">in</span> <span class="n">root</span><span class="o">.</span><span class="n">cte_scopes</span><span class="p">:</span>
</span><span id="L-52"><a href="#L-52"><span class="linenos"> 52</span></a> <span class="n">taken</span><span class="p">[</span><span class="n">scope</span><span class="o">.</span><span class="n">expression</span><span class="o">.</span><span class="n">parent</span><span class="o">.</span><span class="n">alias</span><span class="p">]</span> <span class="o">=</span> <span class="n">scope</span>
</span><span id="L-53"><a href="#L-53"><span class="linenos"> 53</span></a>
</span><span id="L-54"><a href="#L-54"><span class="linenos"> 54</span></a> <span class="c1"># All table names are taken</span>
</span><span id="L-55"><a href="#L-55"><span class="linenos"> 55</span></a> <span class="k">for</span> <span class="n">scope</span> <span class="ow">in</span> <span class="n">root</span><span class="o">.</span><span class="n">traverse</span><span class="p">():</span>
</span><span id="L-56"><a href="#L-56"><span class="linenos"> 56</span></a> <span class="n">taken</span><span class="o">.</span><span class="n">update</span><span class="p">(</span>
</span><span id="L-57"><a href="#L-57"><span class="linenos"> 57</span></a> <span class="p">{</span>
</span><span id="L-58"><a href="#L-58"><span class="linenos"> 58</span></a> <span class="n">source</span><span class="o">.</span><span class="n">name</span><span class="p">:</span> <span class="n">source</span>
</span><span id="L-59"><a href="#L-59"><span class="linenos"> 59</span></a> <span class="k">for</span> <span class="n">_</span><span class="p">,</span> <span class="n">source</span> <span class="ow">in</span> <span class="n">scope</span><span class="o">.</span><span class="n">sources</span><span class="o">.</span><span class="n">items</span><span class="p">()</span>
</span><span id="L-60"><a href="#L-60"><span class="linenos"> 60</span></a> <span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">source</span><span class="p">,</span> <span class="n">exp</span><span class="o">.</span><span class="n">Table</span><span class="p">)</span>
</span><span id="L-61"><a href="#L-61"><span class="linenos"> 61</span></a> <span class="p">}</span>
</span><span id="L-62"><a href="#L-62"><span class="linenos"> 62</span></a> <span class="p">)</span>
</span><span id="L-63"><a href="#L-63"><span class="linenos"> 63</span></a>
</span><span id="L-64"><a href="#L-64"><span class="linenos"> 64</span></a> <span class="c1"># Map of Expression-&gt;alias</span>
</span><span id="L-65"><a href="#L-65"><span class="linenos"> 65</span></a> <span class="c1"># Existing CTES in the root expression. We&#39;ll use this for deduplication.</span>
</span><span id="L-66"><a href="#L-66"><span class="linenos"> 66</span></a> <span class="n">existing_ctes</span><span class="p">:</span> <span class="n">ExistingCTEsMapping</span> <span class="o">=</span> <span class="p">{}</span>
</span><span id="L-67"><a href="#L-67"><span class="linenos"> 67</span></a>
</span><span id="L-68"><a href="#L-68"><span class="linenos"> 68</span></a> <span class="n">with_</span> <span class="o">=</span> <span class="n">root</span><span class="o">.</span><span class="n">expression</span><span class="o">.</span><span class="n">args</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s2">&quot;with&quot;</span><span class="p">)</span>
</span><span id="L-69"><a href="#L-69"><span class="linenos"> 69</span></a> <span class="n">recursive</span> <span class="o">=</span> <span class="kc">False</span>
</span><span id="L-70"><a href="#L-70"><span class="linenos"> 70</span></a> <span class="k">if</span> <span class="n">with_</span><span class="p">:</span>
</span><span id="L-71"><a href="#L-71"><span class="linenos"> 71</span></a> <span class="n">recursive</span> <span class="o">=</span> <span class="n">with_</span><span class="o">.</span><span class="n">args</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s2">&quot;recursive&quot;</span><span class="p">)</span>
</span><span id="L-72"><a href="#L-72"><span class="linenos"> 72</span></a> <span class="k">for</span> <span class="n">cte</span> <span class="ow">in</span> <span class="n">with_</span><span class="o">.</span><span class="n">expressions</span><span class="p">:</span>
</span><span id="L-73"><a href="#L-73"><span class="linenos"> 73</span></a> <span class="n">existing_ctes</span><span class="p">[</span><span class="n">cte</span><span class="o">.</span><span class="n">this</span><span class="p">]</span> <span class="o">=</span> <span class="n">cte</span><span class="o">.</span><span class="n">alias</span>
</span><span id="L-74"><a href="#L-74"><span class="linenos"> 74</span></a> <span class="n">new_ctes</span> <span class="o">=</span> <span class="p">[]</span>
</span><span id="L-75"><a href="#L-75"><span class="linenos"> 75</span></a>
</span><span id="L-76"><a href="#L-76"><span class="linenos"> 76</span></a> <span class="c1"># We&#39;re adding more CTEs, but we want to maintain the DAG order.</span>
</span><span id="L-77"><a href="#L-77"><span class="linenos"> 77</span></a> <span class="c1"># Derived tables within an existing CTE need to come before the existing CTE.</span>
</span><span id="L-78"><a href="#L-78"><span class="linenos"> 78</span></a> <span class="k">for</span> <span class="n">cte_scope</span> <span class="ow">in</span> <span class="n">root</span><span class="o">.</span><span class="n">cte_scopes</span><span class="p">:</span>
</span><span id="L-79"><a href="#L-79"><span class="linenos"> 79</span></a> <span class="c1"># Append all the new CTEs from this existing CTE</span>
</span><span id="L-80"><a href="#L-80"><span class="linenos"> 80</span></a> <span class="k">for</span> <span class="n">scope</span> <span class="ow">in</span> <span class="n">cte_scope</span><span class="o">.</span><span class="n">traverse</span><span class="p">():</span>
</span><span id="L-81"><a href="#L-81"><span class="linenos"> 81</span></a> <span class="k">if</span> <span class="n">scope</span> <span class="ow">is</span> <span class="n">cte_scope</span><span class="p">:</span>
</span><span id="L-82"><a href="#L-82"><span class="linenos"> 82</span></a> <span class="c1"># Don&#39;t try to eliminate this CTE itself</span>
</span><span id="L-83"><a href="#L-83"><span class="linenos"> 83</span></a> <span class="k">continue</span>
</span><span id="L-84"><a href="#L-84"><span class="linenos"> 84</span></a> <span class="n">new_cte</span> <span class="o">=</span> <span class="n">_eliminate</span><span class="p">(</span><span class="n">scope</span><span class="p">,</span> <span class="n">existing_ctes</span><span class="p">,</span> <span class="n">taken</span><span class="p">)</span>
</span><span id="L-85"><a href="#L-85"><span class="linenos"> 85</span></a> <span class="k">if</span> <span class="n">new_cte</span><span class="p">:</span>
</span><span id="L-86"><a href="#L-86"><span class="linenos"> 86</span></a> <span class="n">new_ctes</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">new_cte</span><span class="p">)</span>
</span><span id="L-87"><a href="#L-87"><span class="linenos"> 87</span></a>
</span><span id="L-88"><a href="#L-88"><span class="linenos"> 88</span></a> <span class="c1"># Append the existing CTE itself</span>
</span><span id="L-89"><a href="#L-89"><span class="linenos"> 89</span></a> <span class="n">new_ctes</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">cte_scope</span><span class="o">.</span><span class="n">expression</span><span class="o">.</span><span class="n">parent</span><span class="p">)</span>
</span><span id="L-90"><a href="#L-90"><span class="linenos"> 90</span></a>
</span><span id="L-91"><a href="#L-91"><span class="linenos"> 91</span></a> <span class="k">if</span> <span class="n">new_ctes</span><span class="p">:</span>
</span><span id="L-92"><a href="#L-92"><span class="linenos"> 92</span></a> <span class="n">query</span> <span class="o">=</span> <span class="n">expression</span><span class="o">.</span><span class="n">expression</span> <span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">expression</span><span class="p">,</span> <span class="n">exp</span><span class="o">.</span><span class="n">DDL</span><span class="p">)</span> <span class="k">else</span> <span class="n">expression</span>
</span><span id="L-93"><a href="#L-93"><span class="linenos"> 93</span></a> <span class="n">query</span><span class="o">.</span><span class="n">set</span><span class="p">(</span><span class="s2">&quot;with&quot;</span><span class="p">,</span> <span class="n">exp</span><span class="o">.</span><span class="n">With</span><span class="p">(</span><span class="n">expressions</span><span class="o">=</span><span class="n">new_ctes</span><span class="p">,</span> <span class="n">recursive</span><span class="o">=</span><span class="n">recursive</span><span class="p">))</span>
</span><span id="L-94"><a href="#L-94"><span class="linenos"> 94</span></a>
</span><span id="L-95"><a href="#L-95"><span class="linenos"> 95</span></a> <span class="k">return</span> <span class="n">expression</span>
</span><span id="L-96"><a href="#L-96"><span class="linenos"> 96</span></a>
</span><span id="L-91"><a href="#L-91"><span class="linenos"> 91</span></a> <span class="c1"># Now append the rest</span>
</span><span id="L-92"><a href="#L-92"><span class="linenos"> 92</span></a> <span class="k">for</span> <span class="n">scope</span> <span class="ow">in</span> <span class="n">itertools</span><span class="o">.</span><span class="n">chain</span><span class="p">(</span><span class="n">root</span><span class="o">.</span><span class="n">union_scopes</span><span class="p">,</span> <span class="n">root</span><span class="o">.</span><span class="n">subquery_scopes</span><span class="p">,</span> <span class="n">root</span><span class="o">.</span><span class="n">table_scopes</span><span class="p">):</span>
</span><span id="L-93"><a href="#L-93"><span class="linenos"> 93</span></a> <span class="k">for</span> <span class="n">child_scope</span> <span class="ow">in</span> <span class="n">scope</span><span class="o">.</span><span class="n">traverse</span><span class="p">():</span>
</span><span id="L-94"><a href="#L-94"><span class="linenos"> 94</span></a> <span class="n">new_cte</span> <span class="o">=</span> <span class="n">_eliminate</span><span class="p">(</span><span class="n">child_scope</span><span class="p">,</span> <span class="n">existing_ctes</span><span class="p">,</span> <span class="n">taken</span><span class="p">)</span>
</span><span id="L-95"><a href="#L-95"><span class="linenos"> 95</span></a> <span class="k">if</span> <span class="n">new_cte</span><span class="p">:</span>
</span><span id="L-96"><a href="#L-96"><span class="linenos"> 96</span></a> <span class="n">new_ctes</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">new_cte</span><span class="p">)</span>
</span><span id="L-97"><a href="#L-97"><span class="linenos"> 97</span></a>
</span><span id="L-98"><a href="#L-98"><span class="linenos"> 98</span></a><span class="k">def</span> <span class="nf">_eliminate</span><span class="p">(</span><span class="n">scope</span><span class="p">,</span> <span class="n">existing_ctes</span><span class="p">,</span> <span class="n">taken</span><span class="p">):</span>
</span><span id="L-99"><a href="#L-99"><span class="linenos"> 99</span></a> <span class="k">if</span> <span class="n">scope</span><span class="o">.</span><span class="n">is_derived_table</span><span class="p">:</span>
</span><span id="L-100"><a href="#L-100"><span class="linenos">100</span></a> <span class="k">return</span> <span class="n">_eliminate_derived_table</span><span class="p">(</span><span class="n">scope</span><span class="p">,</span> <span class="n">existing_ctes</span><span class="p">,</span> <span class="n">taken</span><span class="p">)</span>
</span><span id="L-98"><a href="#L-98"><span class="linenos"> 98</span></a> <span class="k">if</span> <span class="n">new_ctes</span><span class="p">:</span>
</span><span id="L-99"><a href="#L-99"><span class="linenos"> 99</span></a> <span class="n">query</span> <span class="o">=</span> <span class="n">expression</span><span class="o">.</span><span class="n">expression</span> <span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">expression</span><span class="p">,</span> <span class="n">exp</span><span class="o">.</span><span class="n">DDL</span><span class="p">)</span> <span class="k">else</span> <span class="n">expression</span>
</span><span id="L-100"><a href="#L-100"><span class="linenos">100</span></a> <span class="n">query</span><span class="o">.</span><span class="n">set</span><span class="p">(</span><span class="s2">&quot;with&quot;</span><span class="p">,</span> <span class="n">exp</span><span class="o">.</span><span class="n">With</span><span class="p">(</span><span class="n">expressions</span><span class="o">=</span><span class="n">new_ctes</span><span class="p">,</span> <span class="n">recursive</span><span class="o">=</span><span class="n">recursive</span><span class="p">))</span>
</span><span id="L-101"><a href="#L-101"><span class="linenos">101</span></a>
</span><span id="L-102"><a href="#L-102"><span class="linenos">102</span></a> <span class="k">if</span> <span class="n">scope</span><span class="o">.</span><span class="n">is_cte</span><span class="p">:</span>
</span><span id="L-103"><a href="#L-103"><span class="linenos">103</span></a> <span class="k">return</span> <span class="n">_eliminate_cte</span><span class="p">(</span><span class="n">scope</span><span class="p">,</span> <span class="n">existing_ctes</span><span class="p">,</span> <span class="n">taken</span><span class="p">)</span>
</span><span id="L-102"><a href="#L-102"><span class="linenos">102</span></a> <span class="k">return</span> <span class="n">expression</span>
</span><span id="L-103"><a href="#L-103"><span class="linenos">103</span></a>
</span><span id="L-104"><a href="#L-104"><span class="linenos">104</span></a>
</span><span id="L-105"><a href="#L-105"><span class="linenos">105</span></a>
</span><span id="L-106"><a href="#L-106"><span class="linenos">106</span></a><span class="k">def</span> <span class="nf">_eliminate_derived_table</span><span class="p">(</span><span class="n">scope</span><span class="p">,</span> <span class="n">existing_ctes</span><span class="p">,</span> <span class="n">taken</span><span class="p">):</span>
</span><span id="L-107"><a href="#L-107"><span class="linenos">107</span></a> <span class="c1"># This makes sure that we don&#39;t:</span>
</span><span id="L-108"><a href="#L-108"><span class="linenos">108</span></a> <span class="c1"># - drop the &quot;pivot&quot; arg from a pivoted subquery</span>
</span><span id="L-109"><a href="#L-109"><span class="linenos">109</span></a> <span class="c1"># - eliminate a lateral correlated subquery</span>
</span><span id="L-110"><a href="#L-110"><span class="linenos">110</span></a> <span class="k">if</span> <span class="n">scope</span><span class="o">.</span><span class="n">parent</span><span class="o">.</span><span class="n">pivots</span> <span class="ow">or</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">scope</span><span class="o">.</span><span class="n">parent</span><span class="o">.</span><span class="n">expression</span><span class="p">,</span> <span class="n">exp</span><span class="o">.</span><span class="n">Lateral</span><span class="p">):</span>
</span><span id="L-111"><a href="#L-111"><span class="linenos">111</span></a> <span class="k">return</span> <span class="kc">None</span>
</span><span id="L-112"><a href="#L-112"><span class="linenos">112</span></a>
</span><span id="L-113"><a href="#L-113"><span class="linenos">113</span></a> <span class="c1"># Get rid of redundant exp.Subquery expressions, i.e. those that are just used as wrappers</span>
</span><span id="L-114"><a href="#L-114"><span class="linenos">114</span></a> <span class="n">to_replace</span> <span class="o">=</span> <span class="n">scope</span><span class="o">.</span><span class="n">expression</span><span class="o">.</span><span class="n">parent</span><span class="o">.</span><span class="n">unwrap</span><span class="p">()</span>
</span><span id="L-115"><a href="#L-115"><span class="linenos">115</span></a> <span class="n">name</span><span class="p">,</span> <span class="n">cte</span> <span class="o">=</span> <span class="n">_new_cte</span><span class="p">(</span><span class="n">scope</span><span class="p">,</span> <span class="n">existing_ctes</span><span class="p">,</span> <span class="n">taken</span><span class="p">)</span>
</span><span id="L-116"><a href="#L-116"><span class="linenos">116</span></a> <span class="n">table</span> <span class="o">=</span> <span class="n">exp</span><span class="o">.</span><span class="n">alias_</span><span class="p">(</span><span class="n">exp</span><span class="o">.</span><span class="n">table_</span><span class="p">(</span><span class="n">name</span><span class="p">),</span> <span class="n">alias</span><span class="o">=</span><span class="n">to_replace</span><span class="o">.</span><span class="n">alias</span> <span class="ow">or</span> <span class="n">name</span><span class="p">)</span>
</span><span id="L-117"><a href="#L-117"><span class="linenos">117</span></a> <span class="n">table</span><span class="o">.</span><span class="n">set</span><span class="p">(</span><span class="s2">&quot;joins&quot;</span><span class="p">,</span> <span class="n">to_replace</span><span class="o">.</span><span class="n">args</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s2">&quot;joins&quot;</span><span class="p">))</span>
</span><span id="L-118"><a href="#L-118"><span class="linenos">118</span></a>
</span><span id="L-119"><a href="#L-119"><span class="linenos">119</span></a> <span class="n">to_replace</span><span class="o">.</span><span class="n">replace</span><span class="p">(</span><span class="n">table</span><span class="p">)</span>
</span><span id="L-120"><a href="#L-120"><span class="linenos">120</span></a>
</span><span id="L-121"><a href="#L-121"><span class="linenos">121</span></a> <span class="k">return</span> <span class="n">cte</span>
</span><span id="L-122"><a href="#L-122"><span class="linenos">122</span></a>
</span><span id="L-123"><a href="#L-123"><span class="linenos">123</span></a>
</span><span id="L-124"><a href="#L-124"><span class="linenos">124</span></a><span class="k">def</span> <span class="nf">_eliminate_cte</span><span class="p">(</span><span class="n">scope</span><span class="p">,</span> <span class="n">existing_ctes</span><span class="p">,</span> <span class="n">taken</span><span class="p">):</span>
</span><span id="L-125"><a href="#L-125"><span class="linenos">125</span></a> <span class="n">parent</span> <span class="o">=</span> <span class="n">scope</span><span class="o">.</span><span class="n">expression</span><span class="o">.</span><span class="n">parent</span>
</span><span id="L-126"><a href="#L-126"><span class="linenos">126</span></a> <span class="n">name</span><span class="p">,</span> <span class="n">cte</span> <span class="o">=</span> <span class="n">_new_cte</span><span class="p">(</span><span class="n">scope</span><span class="p">,</span> <span class="n">existing_ctes</span><span class="p">,</span> <span class="n">taken</span><span class="p">)</span>
</span><span id="L-127"><a href="#L-127"><span class="linenos">127</span></a>
</span><span id="L-128"><a href="#L-128"><span class="linenos">128</span></a> <span class="n">with_</span> <span class="o">=</span> <span class="n">parent</span><span class="o">.</span><span class="n">parent</span>
</span><span id="L-129"><a href="#L-129"><span class="linenos">129</span></a> <span class="n">parent</span><span class="o">.</span><span class="n">pop</span><span class="p">()</span>
</span><span id="L-130"><a href="#L-130"><span class="linenos">130</span></a> <span class="k">if</span> <span class="ow">not</span> <span class="n">with_</span><span class="o">.</span><span class="n">expressions</span><span class="p">:</span>
</span><span id="L-131"><a href="#L-131"><span class="linenos">131</span></a> <span class="n">with_</span><span class="o">.</span><span class="n">pop</span><span class="p">()</span>
</span><span id="L-132"><a href="#L-132"><span class="linenos">132</span></a>
</span><span id="L-133"><a href="#L-133"><span class="linenos">133</span></a> <span class="c1"># Rename references to this CTE</span>
</span><span id="L-134"><a href="#L-134"><span class="linenos">134</span></a> <span class="k">for</span> <span class="n">child_scope</span> <span class="ow">in</span> <span class="n">scope</span><span class="o">.</span><span class="n">parent</span><span class="o">.</span><span class="n">traverse</span><span class="p">():</span>
</span><span id="L-135"><a href="#L-135"><span class="linenos">135</span></a> <span class="k">for</span> <span class="n">table</span><span class="p">,</span> <span class="n">source</span> <span class="ow">in</span> <span class="n">child_scope</span><span class="o">.</span><span class="n">selected_sources</span><span class="o">.</span><span class="n">values</span><span class="p">():</span>
</span><span id="L-136"><a href="#L-136"><span class="linenos">136</span></a> <span class="k">if</span> <span class="n">source</span> <span class="ow">is</span> <span class="n">scope</span><span class="p">:</span>
</span><span id="L-137"><a href="#L-137"><span class="linenos">137</span></a> <span class="n">new_table</span> <span class="o">=</span> <span class="n">exp</span><span class="o">.</span><span class="n">alias_</span><span class="p">(</span><span class="n">exp</span><span class="o">.</span><span class="n">table_</span><span class="p">(</span><span class="n">name</span><span class="p">),</span> <span class="n">alias</span><span class="o">=</span><span class="n">table</span><span class="o">.</span><span class="n">alias_or_name</span><span class="p">,</span> <span class="n">copy</span><span class="o">=</span><span class="kc">False</span><span class="p">)</span>
</span><span id="L-138"><a href="#L-138"><span class="linenos">138</span></a> <span class="n">table</span><span class="o">.</span><span class="n">replace</span><span class="p">(</span><span class="n">new_table</span><span class="p">)</span>
</span><span id="L-139"><a href="#L-139"><span class="linenos">139</span></a>
</span><span id="L-140"><a href="#L-140"><span class="linenos">140</span></a> <span class="k">return</span> <span class="n">cte</span>
</span><span id="L-141"><a href="#L-141"><span class="linenos">141</span></a>
</span><span id="L-105"><a href="#L-105"><span class="linenos">105</span></a><span class="k">def</span> <span class="nf">_eliminate</span><span class="p">(</span>
</span><span id="L-106"><a href="#L-106"><span class="linenos">106</span></a> <span class="n">scope</span><span class="p">:</span> <span class="n">Scope</span><span class="p">,</span> <span class="n">existing_ctes</span><span class="p">:</span> <span class="n">ExistingCTEsMapping</span><span class="p">,</span> <span class="n">taken</span><span class="p">:</span> <span class="n">TakenNameMapping</span>
</span><span id="L-107"><a href="#L-107"><span class="linenos">107</span></a><span class="p">)</span> <span class="o">-&gt;</span> <span class="n">t</span><span class="o">.</span><span class="n">Optional</span><span class="p">[</span><span class="n">exp</span><span class="o">.</span><span class="n">Expression</span><span class="p">]:</span>
</span><span id="L-108"><a href="#L-108"><span class="linenos">108</span></a> <span class="k">if</span> <span class="n">scope</span><span class="o">.</span><span class="n">is_derived_table</span><span class="p">:</span>
</span><span id="L-109"><a href="#L-109"><span class="linenos">109</span></a> <span class="k">return</span> <span class="n">_eliminate_derived_table</span><span class="p">(</span><span class="n">scope</span><span class="p">,</span> <span class="n">existing_ctes</span><span class="p">,</span> <span class="n">taken</span><span class="p">)</span>
</span><span id="L-110"><a href="#L-110"><span class="linenos">110</span></a>
</span><span id="L-111"><a href="#L-111"><span class="linenos">111</span></a> <span class="k">if</span> <span class="n">scope</span><span class="o">.</span><span class="n">is_cte</span><span class="p">:</span>
</span><span id="L-112"><a href="#L-112"><span class="linenos">112</span></a> <span class="k">return</span> <span class="n">_eliminate_cte</span><span class="p">(</span><span class="n">scope</span><span class="p">,</span> <span class="n">existing_ctes</span><span class="p">,</span> <span class="n">taken</span><span class="p">)</span>
</span><span id="L-113"><a href="#L-113"><span class="linenos">113</span></a>
</span><span id="L-114"><a href="#L-114"><span class="linenos">114</span></a> <span class="k">return</span> <span class="kc">None</span>
</span><span id="L-115"><a href="#L-115"><span class="linenos">115</span></a>
</span><span id="L-116"><a href="#L-116"><span class="linenos">116</span></a>
</span><span id="L-117"><a href="#L-117"><span class="linenos">117</span></a><span class="k">def</span> <span class="nf">_eliminate_derived_table</span><span class="p">(</span>
</span><span id="L-118"><a href="#L-118"><span class="linenos">118</span></a> <span class="n">scope</span><span class="p">:</span> <span class="n">Scope</span><span class="p">,</span> <span class="n">existing_ctes</span><span class="p">:</span> <span class="n">ExistingCTEsMapping</span><span class="p">,</span> <span class="n">taken</span><span class="p">:</span> <span class="n">TakenNameMapping</span>
</span><span id="L-119"><a href="#L-119"><span class="linenos">119</span></a><span class="p">)</span> <span class="o">-&gt;</span> <span class="n">t</span><span class="o">.</span><span class="n">Optional</span><span class="p">[</span><span class="n">exp</span><span class="o">.</span><span class="n">Expression</span><span class="p">]:</span>
</span><span id="L-120"><a href="#L-120"><span class="linenos">120</span></a> <span class="c1"># This makes sure that we don&#39;t:</span>
</span><span id="L-121"><a href="#L-121"><span class="linenos">121</span></a> <span class="c1"># - drop the &quot;pivot&quot; arg from a pivoted subquery</span>
</span><span id="L-122"><a href="#L-122"><span class="linenos">122</span></a> <span class="c1"># - eliminate a lateral correlated subquery</span>
</span><span id="L-123"><a href="#L-123"><span class="linenos">123</span></a> <span class="k">if</span> <span class="n">scope</span><span class="o">.</span><span class="n">parent</span><span class="o">.</span><span class="n">pivots</span> <span class="ow">or</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">scope</span><span class="o">.</span><span class="n">parent</span><span class="o">.</span><span class="n">expression</span><span class="p">,</span> <span class="n">exp</span><span class="o">.</span><span class="n">Lateral</span><span class="p">):</span>
</span><span id="L-124"><a href="#L-124"><span class="linenos">124</span></a> <span class="k">return</span> <span class="kc">None</span>
</span><span id="L-125"><a href="#L-125"><span class="linenos">125</span></a>
</span><span id="L-126"><a href="#L-126"><span class="linenos">126</span></a> <span class="c1"># Get rid of redundant exp.Subquery expressions, i.e. those that are just used as wrappers</span>
</span><span id="L-127"><a href="#L-127"><span class="linenos">127</span></a> <span class="n">to_replace</span> <span class="o">=</span> <span class="n">scope</span><span class="o">.</span><span class="n">expression</span><span class="o">.</span><span class="n">parent</span><span class="o">.</span><span class="n">unwrap</span><span class="p">()</span>
</span><span id="L-128"><a href="#L-128"><span class="linenos">128</span></a> <span class="n">name</span><span class="p">,</span> <span class="n">cte</span> <span class="o">=</span> <span class="n">_new_cte</span><span class="p">(</span><span class="n">scope</span><span class="p">,</span> <span class="n">existing_ctes</span><span class="p">,</span> <span class="n">taken</span><span class="p">)</span>
</span><span id="L-129"><a href="#L-129"><span class="linenos">129</span></a> <span class="n">table</span> <span class="o">=</span> <span class="n">exp</span><span class="o">.</span><span class="n">alias_</span><span class="p">(</span><span class="n">exp</span><span class="o">.</span><span class="n">table_</span><span class="p">(</span><span class="n">name</span><span class="p">),</span> <span class="n">alias</span><span class="o">=</span><span class="n">to_replace</span><span class="o">.</span><span class="n">alias</span> <span class="ow">or</span> <span class="n">name</span><span class="p">)</span>
</span><span id="L-130"><a href="#L-130"><span class="linenos">130</span></a> <span class="n">table</span><span class="o">.</span><span class="n">set</span><span class="p">(</span><span class="s2">&quot;joins&quot;</span><span class="p">,</span> <span class="n">to_replace</span><span class="o">.</span><span class="n">args</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s2">&quot;joins&quot;</span><span class="p">))</span>
</span><span id="L-131"><a href="#L-131"><span class="linenos">131</span></a>
</span><span id="L-132"><a href="#L-132"><span class="linenos">132</span></a> <span class="n">to_replace</span><span class="o">.</span><span class="n">replace</span><span class="p">(</span><span class="n">table</span><span class="p">)</span>
</span><span id="L-133"><a href="#L-133"><span class="linenos">133</span></a>
</span><span id="L-134"><a href="#L-134"><span class="linenos">134</span></a> <span class="k">return</span> <span class="n">cte</span>
</span><span id="L-135"><a href="#L-135"><span class="linenos">135</span></a>
</span><span id="L-136"><a href="#L-136"><span class="linenos">136</span></a>
</span><span id="L-137"><a href="#L-137"><span class="linenos">137</span></a><span class="k">def</span> <span class="nf">_eliminate_cte</span><span class="p">(</span>
</span><span id="L-138"><a href="#L-138"><span class="linenos">138</span></a> <span class="n">scope</span><span class="p">:</span> <span class="n">Scope</span><span class="p">,</span> <span class="n">existing_ctes</span><span class="p">:</span> <span class="n">ExistingCTEsMapping</span><span class="p">,</span> <span class="n">taken</span><span class="p">:</span> <span class="n">TakenNameMapping</span>
</span><span id="L-139"><a href="#L-139"><span class="linenos">139</span></a><span class="p">)</span> <span class="o">-&gt;</span> <span class="n">t</span><span class="o">.</span><span class="n">Optional</span><span class="p">[</span><span class="n">exp</span><span class="o">.</span><span class="n">Expression</span><span class="p">]:</span>
</span><span id="L-140"><a href="#L-140"><span class="linenos">140</span></a> <span class="n">parent</span> <span class="o">=</span> <span class="n">scope</span><span class="o">.</span><span class="n">expression</span><span class="o">.</span><span class="n">parent</span>
</span><span id="L-141"><a href="#L-141"><span class="linenos">141</span></a> <span class="n">name</span><span class="p">,</span> <span class="n">cte</span> <span class="o">=</span> <span class="n">_new_cte</span><span class="p">(</span><span class="n">scope</span><span class="p">,</span> <span class="n">existing_ctes</span><span class="p">,</span> <span class="n">taken</span><span class="p">)</span>
</span><span id="L-142"><a href="#L-142"><span class="linenos">142</span></a>
</span><span id="L-143"><a href="#L-143"><span class="linenos">143</span></a><span class="k">def</span> <span class="nf">_new_cte</span><span class="p">(</span><span class="n">scope</span><span class="p">,</span> <span class="n">existing_ctes</span><span class="p">,</span> <span class="n">taken</span><span class="p">):</span>
</span><span id="L-144"><a href="#L-144"><span class="linenos">144</span></a><span class="w"> </span><span class="sd">&quot;&quot;&quot;</span>
</span><span id="L-145"><a href="#L-145"><span class="linenos">145</span></a><span class="sd"> Returns:</span>
</span><span id="L-146"><a href="#L-146"><span class="linenos">146</span></a><span class="sd"> tuple of (name, cte)</span>
</span><span id="L-147"><a href="#L-147"><span class="linenos">147</span></a><span class="sd"> where `name` is a new name for this CTE in the root scope and `cte` is a new CTE instance.</span>
</span><span id="L-148"><a href="#L-148"><span class="linenos">148</span></a><span class="sd"> If this CTE duplicates an existing CTE, `cte` will be None.</span>
</span><span id="L-149"><a href="#L-149"><span class="linenos">149</span></a><span class="sd"> &quot;&quot;&quot;</span>
</span><span id="L-150"><a href="#L-150"><span class="linenos">150</span></a> <span class="n">duplicate_cte_alias</span> <span class="o">=</span> <span class="n">existing_ctes</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">scope</span><span class="o">.</span><span class="n">expression</span><span class="p">)</span>
</span><span id="L-151"><a href="#L-151"><span class="linenos">151</span></a> <span class="n">parent</span> <span class="o">=</span> <span class="n">scope</span><span class="o">.</span><span class="n">expression</span><span class="o">.</span><span class="n">parent</span>
</span><span id="L-152"><a href="#L-152"><span class="linenos">152</span></a> <span class="n">name</span> <span class="o">=</span> <span class="n">parent</span><span class="o">.</span><span class="n">alias</span>
</span><span id="L-153"><a href="#L-153"><span class="linenos">153</span></a>
</span><span id="L-154"><a href="#L-154"><span class="linenos">154</span></a> <span class="k">if</span> <span class="ow">not</span> <span class="n">name</span><span class="p">:</span>
</span><span id="L-155"><a href="#L-155"><span class="linenos">155</span></a> <span class="n">name</span> <span class="o">=</span> <span class="n">find_new_name</span><span class="p">(</span><span class="n">taken</span><span class="o">=</span><span class="n">taken</span><span class="p">,</span> <span class="n">base</span><span class="o">=</span><span class="s2">&quot;cte&quot;</span><span class="p">)</span>
</span><span id="L-143"><a href="#L-143"><span class="linenos">143</span></a> <span class="n">with_</span> <span class="o">=</span> <span class="n">parent</span><span class="o">.</span><span class="n">parent</span>
</span><span id="L-144"><a href="#L-144"><span class="linenos">144</span></a> <span class="n">parent</span><span class="o">.</span><span class="n">pop</span><span class="p">()</span>
</span><span id="L-145"><a href="#L-145"><span class="linenos">145</span></a> <span class="k">if</span> <span class="ow">not</span> <span class="n">with_</span><span class="o">.</span><span class="n">expressions</span><span class="p">:</span>
</span><span id="L-146"><a href="#L-146"><span class="linenos">146</span></a> <span class="n">with_</span><span class="o">.</span><span class="n">pop</span><span class="p">()</span>
</span><span id="L-147"><a href="#L-147"><span class="linenos">147</span></a>
</span><span id="L-148"><a href="#L-148"><span class="linenos">148</span></a> <span class="c1"># Rename references to this CTE</span>
</span><span id="L-149"><a href="#L-149"><span class="linenos">149</span></a> <span class="k">for</span> <span class="n">child_scope</span> <span class="ow">in</span> <span class="n">scope</span><span class="o">.</span><span class="n">parent</span><span class="o">.</span><span class="n">traverse</span><span class="p">():</span>
</span><span id="L-150"><a href="#L-150"><span class="linenos">150</span></a> <span class="k">for</span> <span class="n">table</span><span class="p">,</span> <span class="n">source</span> <span class="ow">in</span> <span class="n">child_scope</span><span class="o">.</span><span class="n">selected_sources</span><span class="o">.</span><span class="n">values</span><span class="p">():</span>
</span><span id="L-151"><a href="#L-151"><span class="linenos">151</span></a> <span class="k">if</span> <span class="n">source</span> <span class="ow">is</span> <span class="n">scope</span><span class="p">:</span>
</span><span id="L-152"><a href="#L-152"><span class="linenos">152</span></a> <span class="n">new_table</span> <span class="o">=</span> <span class="n">exp</span><span class="o">.</span><span class="n">alias_</span><span class="p">(</span><span class="n">exp</span><span class="o">.</span><span class="n">table_</span><span class="p">(</span><span class="n">name</span><span class="p">),</span> <span class="n">alias</span><span class="o">=</span><span class="n">table</span><span class="o">.</span><span class="n">alias_or_name</span><span class="p">,</span> <span class="n">copy</span><span class="o">=</span><span class="kc">False</span><span class="p">)</span>
</span><span id="L-153"><a href="#L-153"><span class="linenos">153</span></a> <span class="n">table</span><span class="o">.</span><span class="n">replace</span><span class="p">(</span><span class="n">new_table</span><span class="p">)</span>
</span><span id="L-154"><a href="#L-154"><span class="linenos">154</span></a>
</span><span id="L-155"><a href="#L-155"><span class="linenos">155</span></a> <span class="k">return</span> <span class="n">cte</span>
</span><span id="L-156"><a href="#L-156"><span class="linenos">156</span></a>
</span><span id="L-157"><a href="#L-157"><span class="linenos">157</span></a> <span class="k">if</span> <span class="n">duplicate_cte_alias</span><span class="p">:</span>
</span><span id="L-158"><a href="#L-158"><span class="linenos">158</span></a> <span class="n">name</span> <span class="o">=</span> <span class="n">duplicate_cte_alias</span>
</span><span id="L-159"><a href="#L-159"><span class="linenos">159</span></a> <span class="k">elif</span> <span class="n">taken</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">name</span><span class="p">):</span>
</span><span id="L-160"><a href="#L-160"><span class="linenos">160</span></a> <span class="n">name</span> <span class="o">=</span> <span class="n">find_new_name</span><span class="p">(</span><span class="n">taken</span><span class="o">=</span><span class="n">taken</span><span class="p">,</span> <span class="n">base</span><span class="o">=</span><span class="n">name</span><span class="p">)</span>
</span><span id="L-161"><a href="#L-161"><span class="linenos">161</span></a>
</span><span id="L-162"><a href="#L-162"><span class="linenos">162</span></a> <span class="n">taken</span><span class="p">[</span><span class="n">name</span><span class="p">]</span> <span class="o">=</span> <span class="n">scope</span>
</span><span id="L-163"><a href="#L-163"><span class="linenos">163</span></a>
</span><span id="L-164"><a href="#L-164"><span class="linenos">164</span></a> <span class="k">if</span> <span class="ow">not</span> <span class="n">duplicate_cte_alias</span><span class="p">:</span>
</span><span id="L-165"><a href="#L-165"><span class="linenos">165</span></a> <span class="n">existing_ctes</span><span class="p">[</span><span class="n">scope</span><span class="o">.</span><span class="n">expression</span><span class="p">]</span> <span class="o">=</span> <span class="n">name</span>
</span><span id="L-166"><a href="#L-166"><span class="linenos">166</span></a> <span class="n">cte</span> <span class="o">=</span> <span class="n">exp</span><span class="o">.</span><span class="n">CTE</span><span class="p">(</span>
</span><span id="L-167"><a href="#L-167"><span class="linenos">167</span></a> <span class="n">this</span><span class="o">=</span><span class="n">scope</span><span class="o">.</span><span class="n">expression</span><span class="p">,</span>
</span><span id="L-168"><a href="#L-168"><span class="linenos">168</span></a> <span class="n">alias</span><span class="o">=</span><span class="n">exp</span><span class="o">.</span><span class="n">TableAlias</span><span class="p">(</span><span class="n">this</span><span class="o">=</span><span class="n">exp</span><span class="o">.</span><span class="n">to_identifier</span><span class="p">(</span><span class="n">name</span><span class="p">)),</span>
</span><span id="L-169"><a href="#L-169"><span class="linenos">169</span></a> <span class="p">)</span>
</span><span id="L-170"><a href="#L-170"><span class="linenos">170</span></a> <span class="k">else</span><span class="p">:</span>
</span><span id="L-171"><a href="#L-171"><span class="linenos">171</span></a> <span class="n">cte</span> <span class="o">=</span> <span class="kc">None</span>
</span><span id="L-172"><a href="#L-172"><span class="linenos">172</span></a> <span class="k">return</span> <span class="n">name</span><span class="p">,</span> <span class="n">cte</span>
</span><span id="L-157"><a href="#L-157"><span class="linenos">157</span></a>
</span><span id="L-158"><a href="#L-158"><span class="linenos">158</span></a><span class="k">def</span> <span class="nf">_new_cte</span><span class="p">(</span>
</span><span id="L-159"><a href="#L-159"><span class="linenos">159</span></a> <span class="n">scope</span><span class="p">:</span> <span class="n">Scope</span><span class="p">,</span> <span class="n">existing_ctes</span><span class="p">:</span> <span class="n">ExistingCTEsMapping</span><span class="p">,</span> <span class="n">taken</span><span class="p">:</span> <span class="n">TakenNameMapping</span>
</span><span id="L-160"><a href="#L-160"><span class="linenos">160</span></a><span class="p">)</span> <span class="o">-&gt;</span> <span class="n">t</span><span class="o">.</span><span class="n">Tuple</span><span class="p">[</span><span class="nb">str</span><span class="p">,</span> <span class="n">t</span><span class="o">.</span><span class="n">Optional</span><span class="p">[</span><span class="n">exp</span><span class="o">.</span><span class="n">Expression</span><span class="p">]]:</span>
</span><span id="L-161"><a href="#L-161"><span class="linenos">161</span></a><span class="w"> </span><span class="sd">&quot;&quot;&quot;</span>
</span><span id="L-162"><a href="#L-162"><span class="linenos">162</span></a><span class="sd"> Returns:</span>
</span><span id="L-163"><a href="#L-163"><span class="linenos">163</span></a><span class="sd"> tuple of (name, cte)</span>
</span><span id="L-164"><a href="#L-164"><span class="linenos">164</span></a><span class="sd"> where `name` is a new name for this CTE in the root scope and `cte` is a new CTE instance.</span>
</span><span id="L-165"><a href="#L-165"><span class="linenos">165</span></a><span class="sd"> If this CTE duplicates an existing CTE, `cte` will be None.</span>
</span><span id="L-166"><a href="#L-166"><span class="linenos">166</span></a><span class="sd"> &quot;&quot;&quot;</span>
</span><span id="L-167"><a href="#L-167"><span class="linenos">167</span></a> <span class="n">duplicate_cte_alias</span> <span class="o">=</span> <span class="n">existing_ctes</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">scope</span><span class="o">.</span><span class="n">expression</span><span class="p">)</span>
</span><span id="L-168"><a href="#L-168"><span class="linenos">168</span></a> <span class="n">parent</span> <span class="o">=</span> <span class="n">scope</span><span class="o">.</span><span class="n">expression</span><span class="o">.</span><span class="n">parent</span>
</span><span id="L-169"><a href="#L-169"><span class="linenos">169</span></a> <span class="n">name</span> <span class="o">=</span> <span class="n">parent</span><span class="o">.</span><span class="n">alias</span>
</span><span id="L-170"><a href="#L-170"><span class="linenos">170</span></a>
</span><span id="L-171"><a href="#L-171"><span class="linenos">171</span></a> <span class="k">if</span> <span class="ow">not</span> <span class="n">name</span><span class="p">:</span>
</span><span id="L-172"><a href="#L-172"><span class="linenos">172</span></a> <span class="n">name</span> <span class="o">=</span> <span class="n">find_new_name</span><span class="p">(</span><span class="n">taken</span><span class="o">=</span><span class="n">taken</span><span class="p">,</span> <span class="n">base</span><span class="o">=</span><span class="s2">&quot;cte&quot;</span><span class="p">)</span>
</span><span id="L-173"><a href="#L-173"><span class="linenos">173</span></a>
</span><span id="L-174"><a href="#L-174"><span class="linenos">174</span></a> <span class="k">if</span> <span class="n">duplicate_cte_alias</span><span class="p">:</span>
</span><span id="L-175"><a href="#L-175"><span class="linenos">175</span></a> <span class="n">name</span> <span class="o">=</span> <span class="n">duplicate_cte_alias</span>
</span><span id="L-176"><a href="#L-176"><span class="linenos">176</span></a> <span class="k">elif</span> <span class="n">taken</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">name</span><span class="p">):</span>
</span><span id="L-177"><a href="#L-177"><span class="linenos">177</span></a> <span class="n">name</span> <span class="o">=</span> <span class="n">find_new_name</span><span class="p">(</span><span class="n">taken</span><span class="o">=</span><span class="n">taken</span><span class="p">,</span> <span class="n">base</span><span class="o">=</span><span class="n">name</span><span class="p">)</span>
</span><span id="L-178"><a href="#L-178"><span class="linenos">178</span></a>
</span><span id="L-179"><a href="#L-179"><span class="linenos">179</span></a> <span class="n">taken</span><span class="p">[</span><span class="n">name</span><span class="p">]</span> <span class="o">=</span> <span class="n">scope</span>
</span><span id="L-180"><a href="#L-180"><span class="linenos">180</span></a>
</span><span id="L-181"><a href="#L-181"><span class="linenos">181</span></a> <span class="k">if</span> <span class="ow">not</span> <span class="n">duplicate_cte_alias</span><span class="p">:</span>
</span><span id="L-182"><a href="#L-182"><span class="linenos">182</span></a> <span class="n">existing_ctes</span><span class="p">[</span><span class="n">scope</span><span class="o">.</span><span class="n">expression</span><span class="p">]</span> <span class="o">=</span> <span class="n">name</span>
</span><span id="L-183"><a href="#L-183"><span class="linenos">183</span></a> <span class="n">cte</span> <span class="o">=</span> <span class="n">exp</span><span class="o">.</span><span class="n">CTE</span><span class="p">(</span>
</span><span id="L-184"><a href="#L-184"><span class="linenos">184</span></a> <span class="n">this</span><span class="o">=</span><span class="n">scope</span><span class="o">.</span><span class="n">expression</span><span class="p">,</span>
</span><span id="L-185"><a href="#L-185"><span class="linenos">185</span></a> <span class="n">alias</span><span class="o">=</span><span class="n">exp</span><span class="o">.</span><span class="n">TableAlias</span><span class="p">(</span><span class="n">this</span><span class="o">=</span><span class="n">exp</span><span class="o">.</span><span class="n">to_identifier</span><span class="p">(</span><span class="n">name</span><span class="p">)),</span>
</span><span id="L-186"><a href="#L-186"><span class="linenos">186</span></a> <span class="p">)</span>
</span><span id="L-187"><a href="#L-187"><span class="linenos">187</span></a> <span class="k">else</span><span class="p">:</span>
</span><span id="L-188"><a href="#L-188"><span class="linenos">188</span></a> <span class="n">cte</span> <span class="o">=</span> <span class="kc">None</span>
</span><span id="L-189"><a href="#L-189"><span class="linenos">189</span></a> <span class="k">return</span> <span class="n">name</span><span class="p">,</span> <span class="n">cte</span>
</span></pre></div>
@ -237,100 +254,100 @@
<div class="attr function">
<span class="def">def</span>
<span class="name">eliminate_subqueries</span><span class="signature pdoc-code condensed">(<span class="param"><span class="n">expression</span></span><span class="return-annotation">):</span></span>
<span class="name">eliminate_subqueries</span><span class="signature pdoc-code multiline">(<span class="param"> <span class="n">expression</span><span class="p">:</span> <span class="n"><a href="../expressions.html#Expression">sqlglot.expressions.Expression</a></span></span><span class="return-annotation">) -> <span class="n"><a href="../expressions.html#Expression">sqlglot.expressions.Expression</a></span>:</span></span>
<label class="view-source-button" for="eliminate_subqueries-view-source"><span>View Source</span></label>
</div>
<a class="headerlink" href="#eliminate_subqueries"></a>
<div class="pdoc-code codehilite"><pre><span></span><span id="eliminate_subqueries-9"><a href="#eliminate_subqueries-9"><span class="linenos"> 9</span></a><span class="k">def</span> <span class="nf">eliminate_subqueries</span><span class="p">(</span><span class="n">expression</span><span class="p">):</span>
</span><span id="eliminate_subqueries-10"><a href="#eliminate_subqueries-10"><span class="linenos">10</span></a><span class="w"> </span><span class="sd">&quot;&quot;&quot;</span>
</span><span id="eliminate_subqueries-11"><a href="#eliminate_subqueries-11"><span class="linenos">11</span></a><span class="sd"> Rewrite derived tables as CTES, deduplicating if possible.</span>
</span><span id="eliminate_subqueries-12"><a href="#eliminate_subqueries-12"><span class="linenos">12</span></a>
</span><span id="eliminate_subqueries-13"><a href="#eliminate_subqueries-13"><span class="linenos">13</span></a><span class="sd"> Example:</span>
</span><span id="eliminate_subqueries-14"><a href="#eliminate_subqueries-14"><span class="linenos">14</span></a><span class="sd"> &gt;&gt;&gt; import sqlglot</span>
</span><span id="eliminate_subqueries-15"><a href="#eliminate_subqueries-15"><span class="linenos">15</span></a><span class="sd"> &gt;&gt;&gt; expression = sqlglot.parse_one(&quot;SELECT a FROM (SELECT * FROM x) AS y&quot;)</span>
</span><span id="eliminate_subqueries-16"><a href="#eliminate_subqueries-16"><span class="linenos">16</span></a><span class="sd"> &gt;&gt;&gt; eliminate_subqueries(expression).sql()</span>
</span><span id="eliminate_subqueries-17"><a href="#eliminate_subqueries-17"><span class="linenos">17</span></a><span class="sd"> &#39;WITH y AS (SELECT * FROM x) SELECT a FROM y AS y&#39;</span>
</span><span id="eliminate_subqueries-18"><a href="#eliminate_subqueries-18"><span class="linenos">18</span></a>
</span><span id="eliminate_subqueries-19"><a href="#eliminate_subqueries-19"><span class="linenos">19</span></a><span class="sd"> This also deduplicates common subqueries:</span>
</span><span id="eliminate_subqueries-20"><a href="#eliminate_subqueries-20"><span class="linenos">20</span></a><span class="sd"> &gt;&gt;&gt; expression = sqlglot.parse_one(&quot;SELECT a FROM (SELECT * FROM x) AS y CROSS JOIN (SELECT * FROM x) AS z&quot;)</span>
</span><span id="eliminate_subqueries-21"><a href="#eliminate_subqueries-21"><span class="linenos">21</span></a><span class="sd"> &gt;&gt;&gt; eliminate_subqueries(expression).sql()</span>
</span><span id="eliminate_subqueries-22"><a href="#eliminate_subqueries-22"><span class="linenos">22</span></a><span class="sd"> &#39;WITH y AS (SELECT * FROM x) SELECT a FROM y AS y CROSS JOIN y AS z&#39;</span>
</span><span id="eliminate_subqueries-23"><a href="#eliminate_subqueries-23"><span class="linenos">23</span></a>
</span><span id="eliminate_subqueries-24"><a href="#eliminate_subqueries-24"><span class="linenos">24</span></a><span class="sd"> Args:</span>
</span><span id="eliminate_subqueries-25"><a href="#eliminate_subqueries-25"><span class="linenos">25</span></a><span class="sd"> expression (sqlglot.Expression): expression</span>
</span><span id="eliminate_subqueries-26"><a href="#eliminate_subqueries-26"><span class="linenos">26</span></a><span class="sd"> Returns:</span>
</span><span id="eliminate_subqueries-27"><a href="#eliminate_subqueries-27"><span class="linenos">27</span></a><span class="sd"> sqlglot.Expression: expression</span>
</span><span id="eliminate_subqueries-28"><a href="#eliminate_subqueries-28"><span class="linenos">28</span></a><span class="sd"> &quot;&quot;&quot;</span>
</span><span id="eliminate_subqueries-29"><a href="#eliminate_subqueries-29"><span class="linenos">29</span></a> <span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">expression</span><span class="p">,</span> <span class="n">exp</span><span class="o">.</span><span class="n">Subquery</span><span class="p">):</span>
</span><span id="eliminate_subqueries-30"><a href="#eliminate_subqueries-30"><span class="linenos">30</span></a> <span class="c1"># It&#39;s possible to have subqueries at the root, e.g. (SELECT * FROM x) LIMIT 1</span>
</span><span id="eliminate_subqueries-31"><a href="#eliminate_subqueries-31"><span class="linenos">31</span></a> <span class="n">eliminate_subqueries</span><span class="p">(</span><span class="n">expression</span><span class="o">.</span><span class="n">this</span><span class="p">)</span>
</span><span id="eliminate_subqueries-32"><a href="#eliminate_subqueries-32"><span class="linenos">32</span></a> <span class="k">return</span> <span class="n">expression</span>
</span><span id="eliminate_subqueries-33"><a href="#eliminate_subqueries-33"><span class="linenos">33</span></a>
</span><span id="eliminate_subqueries-34"><a href="#eliminate_subqueries-34"><span class="linenos">34</span></a> <span class="n">root</span> <span class="o">=</span> <span class="n">build_scope</span><span class="p">(</span><span class="n">expression</span><span class="p">)</span>
</span><span id="eliminate_subqueries-35"><a href="#eliminate_subqueries-35"><span class="linenos">35</span></a>
</span><span id="eliminate_subqueries-36"><a href="#eliminate_subqueries-36"><span class="linenos">36</span></a> <span class="k">if</span> <span class="ow">not</span> <span class="n">root</span><span class="p">:</span>
</span><span id="eliminate_subqueries-37"><a href="#eliminate_subqueries-37"><span class="linenos">37</span></a> <span class="k">return</span> <span class="n">expression</span>
</span><span id="eliminate_subqueries-38"><a href="#eliminate_subqueries-38"><span class="linenos">38</span></a>
</span><span id="eliminate_subqueries-39"><a href="#eliminate_subqueries-39"><span class="linenos">39</span></a> <span class="c1"># Map of alias-&gt;Scope|Table</span>
</span><span id="eliminate_subqueries-40"><a href="#eliminate_subqueries-40"><span class="linenos">40</span></a> <span class="c1"># These are all aliases that are already used in the expression.</span>
</span><span id="eliminate_subqueries-41"><a href="#eliminate_subqueries-41"><span class="linenos">41</span></a> <span class="c1"># We don&#39;t want to create new CTEs that conflict with these names.</span>
</span><span id="eliminate_subqueries-42"><a href="#eliminate_subqueries-42"><span class="linenos">42</span></a> <span class="n">taken</span> <span class="o">=</span> <span class="p">{}</span>
</span><span id="eliminate_subqueries-43"><a href="#eliminate_subqueries-43"><span class="linenos">43</span></a>
</span><span id="eliminate_subqueries-44"><a href="#eliminate_subqueries-44"><span class="linenos">44</span></a> <span class="c1"># All CTE aliases in the root scope are taken</span>
</span><span id="eliminate_subqueries-45"><a href="#eliminate_subqueries-45"><span class="linenos">45</span></a> <span class="k">for</span> <span class="n">scope</span> <span class="ow">in</span> <span class="n">root</span><span class="o">.</span><span class="n">cte_scopes</span><span class="p">:</span>
</span><span id="eliminate_subqueries-46"><a href="#eliminate_subqueries-46"><span class="linenos">46</span></a> <span class="n">taken</span><span class="p">[</span><span class="n">scope</span><span class="o">.</span><span class="n">expression</span><span class="o">.</span><span class="n">parent</span><span class="o">.</span><span class="n">alias</span><span class="p">]</span> <span class="o">=</span> <span class="n">scope</span>
</span><span id="eliminate_subqueries-47"><a href="#eliminate_subqueries-47"><span class="linenos">47</span></a>
</span><span id="eliminate_subqueries-48"><a href="#eliminate_subqueries-48"><span class="linenos">48</span></a> <span class="c1"># All table names are taken</span>
</span><span id="eliminate_subqueries-49"><a href="#eliminate_subqueries-49"><span class="linenos">49</span></a> <span class="k">for</span> <span class="n">scope</span> <span class="ow">in</span> <span class="n">root</span><span class="o">.</span><span class="n">traverse</span><span class="p">():</span>
</span><span id="eliminate_subqueries-50"><a href="#eliminate_subqueries-50"><span class="linenos">50</span></a> <span class="n">taken</span><span class="o">.</span><span class="n">update</span><span class="p">(</span>
</span><span id="eliminate_subqueries-51"><a href="#eliminate_subqueries-51"><span class="linenos">51</span></a> <span class="p">{</span>
</span><span id="eliminate_subqueries-52"><a href="#eliminate_subqueries-52"><span class="linenos">52</span></a> <span class="n">source</span><span class="o">.</span><span class="n">name</span><span class="p">:</span> <span class="n">source</span>
</span><span id="eliminate_subqueries-53"><a href="#eliminate_subqueries-53"><span class="linenos">53</span></a> <span class="k">for</span> <span class="n">_</span><span class="p">,</span> <span class="n">source</span> <span class="ow">in</span> <span class="n">scope</span><span class="o">.</span><span class="n">sources</span><span class="o">.</span><span class="n">items</span><span class="p">()</span>
</span><span id="eliminate_subqueries-54"><a href="#eliminate_subqueries-54"><span class="linenos">54</span></a> <span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">source</span><span class="p">,</span> <span class="n">exp</span><span class="o">.</span><span class="n">Table</span><span class="p">)</span>
</span><span id="eliminate_subqueries-55"><a href="#eliminate_subqueries-55"><span class="linenos">55</span></a> <span class="p">}</span>
</span><span id="eliminate_subqueries-56"><a href="#eliminate_subqueries-56"><span class="linenos">56</span></a> <span class="p">)</span>
</span><span id="eliminate_subqueries-57"><a href="#eliminate_subqueries-57"><span class="linenos">57</span></a>
</span><span id="eliminate_subqueries-58"><a href="#eliminate_subqueries-58"><span class="linenos">58</span></a> <span class="c1"># Map of Expression-&gt;alias</span>
</span><span id="eliminate_subqueries-59"><a href="#eliminate_subqueries-59"><span class="linenos">59</span></a> <span class="c1"># Existing CTES in the root expression. We&#39;ll use this for deduplication.</span>
</span><span id="eliminate_subqueries-60"><a href="#eliminate_subqueries-60"><span class="linenos">60</span></a> <span class="n">existing_ctes</span> <span class="o">=</span> <span class="p">{}</span>
</span><span id="eliminate_subqueries-61"><a href="#eliminate_subqueries-61"><span class="linenos">61</span></a>
</span><span id="eliminate_subqueries-62"><a href="#eliminate_subqueries-62"><span class="linenos">62</span></a> <span class="n">with_</span> <span class="o">=</span> <span class="n">root</span><span class="o">.</span><span class="n">expression</span><span class="o">.</span><span class="n">args</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s2">&quot;with&quot;</span><span class="p">)</span>
</span><span id="eliminate_subqueries-63"><a href="#eliminate_subqueries-63"><span class="linenos">63</span></a> <span class="n">recursive</span> <span class="o">=</span> <span class="kc">False</span>
</span><span id="eliminate_subqueries-64"><a href="#eliminate_subqueries-64"><span class="linenos">64</span></a> <span class="k">if</span> <span class="n">with_</span><span class="p">:</span>
</span><span id="eliminate_subqueries-65"><a href="#eliminate_subqueries-65"><span class="linenos">65</span></a> <span class="n">recursive</span> <span class="o">=</span> <span class="n">with_</span><span class="o">.</span><span class="n">args</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s2">&quot;recursive&quot;</span><span class="p">)</span>
</span><span id="eliminate_subqueries-66"><a href="#eliminate_subqueries-66"><span class="linenos">66</span></a> <span class="k">for</span> <span class="n">cte</span> <span class="ow">in</span> <span class="n">with_</span><span class="o">.</span><span class="n">expressions</span><span class="p">:</span>
</span><span id="eliminate_subqueries-67"><a href="#eliminate_subqueries-67"><span class="linenos">67</span></a> <span class="n">existing_ctes</span><span class="p">[</span><span class="n">cte</span><span class="o">.</span><span class="n">this</span><span class="p">]</span> <span class="o">=</span> <span class="n">cte</span><span class="o">.</span><span class="n">alias</span>
</span><span id="eliminate_subqueries-68"><a href="#eliminate_subqueries-68"><span class="linenos">68</span></a> <span class="n">new_ctes</span> <span class="o">=</span> <span class="p">[]</span>
</span><span id="eliminate_subqueries-69"><a href="#eliminate_subqueries-69"><span class="linenos">69</span></a>
</span><span id="eliminate_subqueries-70"><a href="#eliminate_subqueries-70"><span class="linenos">70</span></a> <span class="c1"># We&#39;re adding more CTEs, but we want to maintain the DAG order.</span>
</span><span id="eliminate_subqueries-71"><a href="#eliminate_subqueries-71"><span class="linenos">71</span></a> <span class="c1"># Derived tables within an existing CTE need to come before the existing CTE.</span>
</span><span id="eliminate_subqueries-72"><a href="#eliminate_subqueries-72"><span class="linenos">72</span></a> <span class="k">for</span> <span class="n">cte_scope</span> <span class="ow">in</span> <span class="n">root</span><span class="o">.</span><span class="n">cte_scopes</span><span class="p">:</span>
</span><span id="eliminate_subqueries-73"><a href="#eliminate_subqueries-73"><span class="linenos">73</span></a> <span class="c1"># Append all the new CTEs from this existing CTE</span>
</span><span id="eliminate_subqueries-74"><a href="#eliminate_subqueries-74"><span class="linenos">74</span></a> <span class="k">for</span> <span class="n">scope</span> <span class="ow">in</span> <span class="n">cte_scope</span><span class="o">.</span><span class="n">traverse</span><span class="p">():</span>
</span><span id="eliminate_subqueries-75"><a href="#eliminate_subqueries-75"><span class="linenos">75</span></a> <span class="k">if</span> <span class="n">scope</span> <span class="ow">is</span> <span class="n">cte_scope</span><span class="p">:</span>
</span><span id="eliminate_subqueries-76"><a href="#eliminate_subqueries-76"><span class="linenos">76</span></a> <span class="c1"># Don&#39;t try to eliminate this CTE itself</span>
</span><span id="eliminate_subqueries-77"><a href="#eliminate_subqueries-77"><span class="linenos">77</span></a> <span class="k">continue</span>
</span><span id="eliminate_subqueries-78"><a href="#eliminate_subqueries-78"><span class="linenos">78</span></a> <span class="n">new_cte</span> <span class="o">=</span> <span class="n">_eliminate</span><span class="p">(</span><span class="n">scope</span><span class="p">,</span> <span class="n">existing_ctes</span><span class="p">,</span> <span class="n">taken</span><span class="p">)</span>
</span><span id="eliminate_subqueries-79"><a href="#eliminate_subqueries-79"><span class="linenos">79</span></a> <span class="k">if</span> <span class="n">new_cte</span><span class="p">:</span>
</span><span id="eliminate_subqueries-80"><a href="#eliminate_subqueries-80"><span class="linenos">80</span></a> <span class="n">new_ctes</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">new_cte</span><span class="p">)</span>
</span><span id="eliminate_subqueries-81"><a href="#eliminate_subqueries-81"><span class="linenos">81</span></a>
</span><span id="eliminate_subqueries-82"><a href="#eliminate_subqueries-82"><span class="linenos">82</span></a> <span class="c1"># Append the existing CTE itself</span>
</span><span id="eliminate_subqueries-83"><a href="#eliminate_subqueries-83"><span class="linenos">83</span></a> <span class="n">new_ctes</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">cte_scope</span><span class="o">.</span><span class="n">expression</span><span class="o">.</span><span class="n">parent</span><span class="p">)</span>
</span><span id="eliminate_subqueries-84"><a href="#eliminate_subqueries-84"><span class="linenos">84</span></a>
</span><span id="eliminate_subqueries-85"><a href="#eliminate_subqueries-85"><span class="linenos">85</span></a> <span class="c1"># Now append the rest</span>
</span><span id="eliminate_subqueries-86"><a href="#eliminate_subqueries-86"><span class="linenos">86</span></a> <span class="k">for</span> <span class="n">scope</span> <span class="ow">in</span> <span class="n">itertools</span><span class="o">.</span><span class="n">chain</span><span class="p">(</span><span class="n">root</span><span class="o">.</span><span class="n">union_scopes</span><span class="p">,</span> <span class="n">root</span><span class="o">.</span><span class="n">subquery_scopes</span><span class="p">,</span> <span class="n">root</span><span class="o">.</span><span class="n">table_scopes</span><span class="p">):</span>
</span><span id="eliminate_subqueries-87"><a href="#eliminate_subqueries-87"><span class="linenos">87</span></a> <span class="k">for</span> <span class="n">child_scope</span> <span class="ow">in</span> <span class="n">scope</span><span class="o">.</span><span class="n">traverse</span><span class="p">():</span>
</span><span id="eliminate_subqueries-88"><a href="#eliminate_subqueries-88"><span class="linenos">88</span></a> <span class="n">new_cte</span> <span class="o">=</span> <span class="n">_eliminate</span><span class="p">(</span><span class="n">child_scope</span><span class="p">,</span> <span class="n">existing_ctes</span><span class="p">,</span> <span class="n">taken</span><span class="p">)</span>
</span><span id="eliminate_subqueries-89"><a href="#eliminate_subqueries-89"><span class="linenos">89</span></a> <span class="k">if</span> <span class="n">new_cte</span><span class="p">:</span>
</span><span id="eliminate_subqueries-90"><a href="#eliminate_subqueries-90"><span class="linenos">90</span></a> <span class="n">new_ctes</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">new_cte</span><span class="p">)</span>
</span><span id="eliminate_subqueries-91"><a href="#eliminate_subqueries-91"><span class="linenos">91</span></a>
</span><span id="eliminate_subqueries-92"><a href="#eliminate_subqueries-92"><span class="linenos">92</span></a> <span class="k">if</span> <span class="n">new_ctes</span><span class="p">:</span>
</span><span id="eliminate_subqueries-93"><a href="#eliminate_subqueries-93"><span class="linenos">93</span></a> <span class="n">query</span> <span class="o">=</span> <span class="n">expression</span><span class="o">.</span><span class="n">expression</span> <span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">expression</span><span class="p">,</span> <span class="n">exp</span><span class="o">.</span><span class="n">DDL</span><span class="p">)</span> <span class="k">else</span> <span class="n">expression</span>
</span><span id="eliminate_subqueries-94"><a href="#eliminate_subqueries-94"><span class="linenos">94</span></a> <span class="n">query</span><span class="o">.</span><span class="n">set</span><span class="p">(</span><span class="s2">&quot;with&quot;</span><span class="p">,</span> <span class="n">exp</span><span class="o">.</span><span class="n">With</span><span class="p">(</span><span class="n">expressions</span><span class="o">=</span><span class="n">new_ctes</span><span class="p">,</span> <span class="n">recursive</span><span class="o">=</span><span class="n">recursive</span><span class="p">))</span>
</span><span id="eliminate_subqueries-95"><a href="#eliminate_subqueries-95"><span class="linenos">95</span></a>
</span><span id="eliminate_subqueries-96"><a href="#eliminate_subqueries-96"><span class="linenos">96</span></a> <span class="k">return</span> <span class="n">expression</span>
<div class="pdoc-code codehilite"><pre><span></span><span id="eliminate_subqueries-16"><a href="#eliminate_subqueries-16"><span class="linenos"> 16</span></a><span class="k">def</span> <span class="nf">eliminate_subqueries</span><span class="p">(</span><span class="n">expression</span><span class="p">:</span> <span class="n">exp</span><span class="o">.</span><span class="n">Expression</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="n">exp</span><span class="o">.</span><span class="n">Expression</span><span class="p">:</span>
</span><span id="eliminate_subqueries-17"><a href="#eliminate_subqueries-17"><span class="linenos"> 17</span></a><span class="w"> </span><span class="sd">&quot;&quot;&quot;</span>
</span><span id="eliminate_subqueries-18"><a href="#eliminate_subqueries-18"><span class="linenos"> 18</span></a><span class="sd"> Rewrite derived tables as CTES, deduplicating if possible.</span>
</span><span id="eliminate_subqueries-19"><a href="#eliminate_subqueries-19"><span class="linenos"> 19</span></a>
</span><span id="eliminate_subqueries-20"><a href="#eliminate_subqueries-20"><span class="linenos"> 20</span></a><span class="sd"> Example:</span>
</span><span id="eliminate_subqueries-21"><a href="#eliminate_subqueries-21"><span class="linenos"> 21</span></a><span class="sd"> &gt;&gt;&gt; import sqlglot</span>
</span><span id="eliminate_subqueries-22"><a href="#eliminate_subqueries-22"><span class="linenos"> 22</span></a><span class="sd"> &gt;&gt;&gt; expression = sqlglot.parse_one(&quot;SELECT a FROM (SELECT * FROM x) AS y&quot;)</span>
</span><span id="eliminate_subqueries-23"><a href="#eliminate_subqueries-23"><span class="linenos"> 23</span></a><span class="sd"> &gt;&gt;&gt; eliminate_subqueries(expression).sql()</span>
</span><span id="eliminate_subqueries-24"><a href="#eliminate_subqueries-24"><span class="linenos"> 24</span></a><span class="sd"> &#39;WITH y AS (SELECT * FROM x) SELECT a FROM y AS y&#39;</span>
</span><span id="eliminate_subqueries-25"><a href="#eliminate_subqueries-25"><span class="linenos"> 25</span></a>
</span><span id="eliminate_subqueries-26"><a href="#eliminate_subqueries-26"><span class="linenos"> 26</span></a><span class="sd"> This also deduplicates common subqueries:</span>
</span><span id="eliminate_subqueries-27"><a href="#eliminate_subqueries-27"><span class="linenos"> 27</span></a><span class="sd"> &gt;&gt;&gt; expression = sqlglot.parse_one(&quot;SELECT a FROM (SELECT * FROM x) AS y CROSS JOIN (SELECT * FROM x) AS z&quot;)</span>
</span><span id="eliminate_subqueries-28"><a href="#eliminate_subqueries-28"><span class="linenos"> 28</span></a><span class="sd"> &gt;&gt;&gt; eliminate_subqueries(expression).sql()</span>
</span><span id="eliminate_subqueries-29"><a href="#eliminate_subqueries-29"><span class="linenos"> 29</span></a><span class="sd"> &#39;WITH y AS (SELECT * FROM x) SELECT a FROM y AS y CROSS JOIN y AS z&#39;</span>
</span><span id="eliminate_subqueries-30"><a href="#eliminate_subqueries-30"><span class="linenos"> 30</span></a>
</span><span id="eliminate_subqueries-31"><a href="#eliminate_subqueries-31"><span class="linenos"> 31</span></a><span class="sd"> Args:</span>
</span><span id="eliminate_subqueries-32"><a href="#eliminate_subqueries-32"><span class="linenos"> 32</span></a><span class="sd"> expression (sqlglot.Expression): expression</span>
</span><span id="eliminate_subqueries-33"><a href="#eliminate_subqueries-33"><span class="linenos"> 33</span></a><span class="sd"> Returns:</span>
</span><span id="eliminate_subqueries-34"><a href="#eliminate_subqueries-34"><span class="linenos"> 34</span></a><span class="sd"> sqlglot.Expression: expression</span>
</span><span id="eliminate_subqueries-35"><a href="#eliminate_subqueries-35"><span class="linenos"> 35</span></a><span class="sd"> &quot;&quot;&quot;</span>
</span><span id="eliminate_subqueries-36"><a href="#eliminate_subqueries-36"><span class="linenos"> 36</span></a> <span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">expression</span><span class="p">,</span> <span class="n">exp</span><span class="o">.</span><span class="n">Subquery</span><span class="p">):</span>
</span><span id="eliminate_subqueries-37"><a href="#eliminate_subqueries-37"><span class="linenos"> 37</span></a> <span class="c1"># It&#39;s possible to have subqueries at the root, e.g. (SELECT * FROM x) LIMIT 1</span>
</span><span id="eliminate_subqueries-38"><a href="#eliminate_subqueries-38"><span class="linenos"> 38</span></a> <span class="n">eliminate_subqueries</span><span class="p">(</span><span class="n">expression</span><span class="o">.</span><span class="n">this</span><span class="p">)</span>
</span><span id="eliminate_subqueries-39"><a href="#eliminate_subqueries-39"><span class="linenos"> 39</span></a> <span class="k">return</span> <span class="n">expression</span>
</span><span id="eliminate_subqueries-40"><a href="#eliminate_subqueries-40"><span class="linenos"> 40</span></a>
</span><span id="eliminate_subqueries-41"><a href="#eliminate_subqueries-41"><span class="linenos"> 41</span></a> <span class="n">root</span> <span class="o">=</span> <span class="n">build_scope</span><span class="p">(</span><span class="n">expression</span><span class="p">)</span>
</span><span id="eliminate_subqueries-42"><a href="#eliminate_subqueries-42"><span class="linenos"> 42</span></a>
</span><span id="eliminate_subqueries-43"><a href="#eliminate_subqueries-43"><span class="linenos"> 43</span></a> <span class="k">if</span> <span class="ow">not</span> <span class="n">root</span><span class="p">:</span>
</span><span id="eliminate_subqueries-44"><a href="#eliminate_subqueries-44"><span class="linenos"> 44</span></a> <span class="k">return</span> <span class="n">expression</span>
</span><span id="eliminate_subqueries-45"><a href="#eliminate_subqueries-45"><span class="linenos"> 45</span></a>
</span><span id="eliminate_subqueries-46"><a href="#eliminate_subqueries-46"><span class="linenos"> 46</span></a> <span class="c1"># Map of alias-&gt;Scope|Table</span>
</span><span id="eliminate_subqueries-47"><a href="#eliminate_subqueries-47"><span class="linenos"> 47</span></a> <span class="c1"># These are all aliases that are already used in the expression.</span>
</span><span id="eliminate_subqueries-48"><a href="#eliminate_subqueries-48"><span class="linenos"> 48</span></a> <span class="c1"># We don&#39;t want to create new CTEs that conflict with these names.</span>
</span><span id="eliminate_subqueries-49"><a href="#eliminate_subqueries-49"><span class="linenos"> 49</span></a> <span class="n">taken</span><span class="p">:</span> <span class="n">TakenNameMapping</span> <span class="o">=</span> <span class="p">{}</span>
</span><span id="eliminate_subqueries-50"><a href="#eliminate_subqueries-50"><span class="linenos"> 50</span></a>
</span><span id="eliminate_subqueries-51"><a href="#eliminate_subqueries-51"><span class="linenos"> 51</span></a> <span class="c1"># All CTE aliases in the root scope are taken</span>
</span><span id="eliminate_subqueries-52"><a href="#eliminate_subqueries-52"><span class="linenos"> 52</span></a> <span class="k">for</span> <span class="n">scope</span> <span class="ow">in</span> <span class="n">root</span><span class="o">.</span><span class="n">cte_scopes</span><span class="p">:</span>
</span><span id="eliminate_subqueries-53"><a href="#eliminate_subqueries-53"><span class="linenos"> 53</span></a> <span class="n">taken</span><span class="p">[</span><span class="n">scope</span><span class="o">.</span><span class="n">expression</span><span class="o">.</span><span class="n">parent</span><span class="o">.</span><span class="n">alias</span><span class="p">]</span> <span class="o">=</span> <span class="n">scope</span>
</span><span id="eliminate_subqueries-54"><a href="#eliminate_subqueries-54"><span class="linenos"> 54</span></a>
</span><span id="eliminate_subqueries-55"><a href="#eliminate_subqueries-55"><span class="linenos"> 55</span></a> <span class="c1"># All table names are taken</span>
</span><span id="eliminate_subqueries-56"><a href="#eliminate_subqueries-56"><span class="linenos"> 56</span></a> <span class="k">for</span> <span class="n">scope</span> <span class="ow">in</span> <span class="n">root</span><span class="o">.</span><span class="n">traverse</span><span class="p">():</span>
</span><span id="eliminate_subqueries-57"><a href="#eliminate_subqueries-57"><span class="linenos"> 57</span></a> <span class="n">taken</span><span class="o">.</span><span class="n">update</span><span class="p">(</span>
</span><span id="eliminate_subqueries-58"><a href="#eliminate_subqueries-58"><span class="linenos"> 58</span></a> <span class="p">{</span>
</span><span id="eliminate_subqueries-59"><a href="#eliminate_subqueries-59"><span class="linenos"> 59</span></a> <span class="n">source</span><span class="o">.</span><span class="n">name</span><span class="p">:</span> <span class="n">source</span>
</span><span id="eliminate_subqueries-60"><a href="#eliminate_subqueries-60"><span class="linenos"> 60</span></a> <span class="k">for</span> <span class="n">_</span><span class="p">,</span> <span class="n">source</span> <span class="ow">in</span> <span class="n">scope</span><span class="o">.</span><span class="n">sources</span><span class="o">.</span><span class="n">items</span><span class="p">()</span>
</span><span id="eliminate_subqueries-61"><a href="#eliminate_subqueries-61"><span class="linenos"> 61</span></a> <span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">source</span><span class="p">,</span> <span class="n">exp</span><span class="o">.</span><span class="n">Table</span><span class="p">)</span>
</span><span id="eliminate_subqueries-62"><a href="#eliminate_subqueries-62"><span class="linenos"> 62</span></a> <span class="p">}</span>
</span><span id="eliminate_subqueries-63"><a href="#eliminate_subqueries-63"><span class="linenos"> 63</span></a> <span class="p">)</span>
</span><span id="eliminate_subqueries-64"><a href="#eliminate_subqueries-64"><span class="linenos"> 64</span></a>
</span><span id="eliminate_subqueries-65"><a href="#eliminate_subqueries-65"><span class="linenos"> 65</span></a> <span class="c1"># Map of Expression-&gt;alias</span>
</span><span id="eliminate_subqueries-66"><a href="#eliminate_subqueries-66"><span class="linenos"> 66</span></a> <span class="c1"># Existing CTES in the root expression. We&#39;ll use this for deduplication.</span>
</span><span id="eliminate_subqueries-67"><a href="#eliminate_subqueries-67"><span class="linenos"> 67</span></a> <span class="n">existing_ctes</span><span class="p">:</span> <span class="n">ExistingCTEsMapping</span> <span class="o">=</span> <span class="p">{}</span>
</span><span id="eliminate_subqueries-68"><a href="#eliminate_subqueries-68"><span class="linenos"> 68</span></a>
</span><span id="eliminate_subqueries-69"><a href="#eliminate_subqueries-69"><span class="linenos"> 69</span></a> <span class="n">with_</span> <span class="o">=</span> <span class="n">root</span><span class="o">.</span><span class="n">expression</span><span class="o">.</span><span class="n">args</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s2">&quot;with&quot;</span><span class="p">)</span>
</span><span id="eliminate_subqueries-70"><a href="#eliminate_subqueries-70"><span class="linenos"> 70</span></a> <span class="n">recursive</span> <span class="o">=</span> <span class="kc">False</span>
</span><span id="eliminate_subqueries-71"><a href="#eliminate_subqueries-71"><span class="linenos"> 71</span></a> <span class="k">if</span> <span class="n">with_</span><span class="p">:</span>
</span><span id="eliminate_subqueries-72"><a href="#eliminate_subqueries-72"><span class="linenos"> 72</span></a> <span class="n">recursive</span> <span class="o">=</span> <span class="n">with_</span><span class="o">.</span><span class="n">args</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s2">&quot;recursive&quot;</span><span class="p">)</span>
</span><span id="eliminate_subqueries-73"><a href="#eliminate_subqueries-73"><span class="linenos"> 73</span></a> <span class="k">for</span> <span class="n">cte</span> <span class="ow">in</span> <span class="n">with_</span><span class="o">.</span><span class="n">expressions</span><span class="p">:</span>
</span><span id="eliminate_subqueries-74"><a href="#eliminate_subqueries-74"><span class="linenos"> 74</span></a> <span class="n">existing_ctes</span><span class="p">[</span><span class="n">cte</span><span class="o">.</span><span class="n">this</span><span class="p">]</span> <span class="o">=</span> <span class="n">cte</span><span class="o">.</span><span class="n">alias</span>
</span><span id="eliminate_subqueries-75"><a href="#eliminate_subqueries-75"><span class="linenos"> 75</span></a> <span class="n">new_ctes</span> <span class="o">=</span> <span class="p">[]</span>
</span><span id="eliminate_subqueries-76"><a href="#eliminate_subqueries-76"><span class="linenos"> 76</span></a>
</span><span id="eliminate_subqueries-77"><a href="#eliminate_subqueries-77"><span class="linenos"> 77</span></a> <span class="c1"># We&#39;re adding more CTEs, but we want to maintain the DAG order.</span>
</span><span id="eliminate_subqueries-78"><a href="#eliminate_subqueries-78"><span class="linenos"> 78</span></a> <span class="c1"># Derived tables within an existing CTE need to come before the existing CTE.</span>
</span><span id="eliminate_subqueries-79"><a href="#eliminate_subqueries-79"><span class="linenos"> 79</span></a> <span class="k">for</span> <span class="n">cte_scope</span> <span class="ow">in</span> <span class="n">root</span><span class="o">.</span><span class="n">cte_scopes</span><span class="p">:</span>
</span><span id="eliminate_subqueries-80"><a href="#eliminate_subqueries-80"><span class="linenos"> 80</span></a> <span class="c1"># Append all the new CTEs from this existing CTE</span>
</span><span id="eliminate_subqueries-81"><a href="#eliminate_subqueries-81"><span class="linenos"> 81</span></a> <span class="k">for</span> <span class="n">scope</span> <span class="ow">in</span> <span class="n">cte_scope</span><span class="o">.</span><span class="n">traverse</span><span class="p">():</span>
</span><span id="eliminate_subqueries-82"><a href="#eliminate_subqueries-82"><span class="linenos"> 82</span></a> <span class="k">if</span> <span class="n">scope</span> <span class="ow">is</span> <span class="n">cte_scope</span><span class="p">:</span>
</span><span id="eliminate_subqueries-83"><a href="#eliminate_subqueries-83"><span class="linenos"> 83</span></a> <span class="c1"># Don&#39;t try to eliminate this CTE itself</span>
</span><span id="eliminate_subqueries-84"><a href="#eliminate_subqueries-84"><span class="linenos"> 84</span></a> <span class="k">continue</span>
</span><span id="eliminate_subqueries-85"><a href="#eliminate_subqueries-85"><span class="linenos"> 85</span></a> <span class="n">new_cte</span> <span class="o">=</span> <span class="n">_eliminate</span><span class="p">(</span><span class="n">scope</span><span class="p">,</span> <span class="n">existing_ctes</span><span class="p">,</span> <span class="n">taken</span><span class="p">)</span>
</span><span id="eliminate_subqueries-86"><a href="#eliminate_subqueries-86"><span class="linenos"> 86</span></a> <span class="k">if</span> <span class="n">new_cte</span><span class="p">:</span>
</span><span id="eliminate_subqueries-87"><a href="#eliminate_subqueries-87"><span class="linenos"> 87</span></a> <span class="n">new_ctes</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">new_cte</span><span class="p">)</span>
</span><span id="eliminate_subqueries-88"><a href="#eliminate_subqueries-88"><span class="linenos"> 88</span></a>
</span><span id="eliminate_subqueries-89"><a href="#eliminate_subqueries-89"><span class="linenos"> 89</span></a> <span class="c1"># Append the existing CTE itself</span>
</span><span id="eliminate_subqueries-90"><a href="#eliminate_subqueries-90"><span class="linenos"> 90</span></a> <span class="n">new_ctes</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">cte_scope</span><span class="o">.</span><span class="n">expression</span><span class="o">.</span><span class="n">parent</span><span class="p">)</span>
</span><span id="eliminate_subqueries-91"><a href="#eliminate_subqueries-91"><span class="linenos"> 91</span></a>
</span><span id="eliminate_subqueries-92"><a href="#eliminate_subqueries-92"><span class="linenos"> 92</span></a> <span class="c1"># Now append the rest</span>
</span><span id="eliminate_subqueries-93"><a href="#eliminate_subqueries-93"><span class="linenos"> 93</span></a> <span class="k">for</span> <span class="n">scope</span> <span class="ow">in</span> <span class="n">itertools</span><span class="o">.</span><span class="n">chain</span><span class="p">(</span><span class="n">root</span><span class="o">.</span><span class="n">union_scopes</span><span class="p">,</span> <span class="n">root</span><span class="o">.</span><span class="n">subquery_scopes</span><span class="p">,</span> <span class="n">root</span><span class="o">.</span><span class="n">table_scopes</span><span class="p">):</span>
</span><span id="eliminate_subqueries-94"><a href="#eliminate_subqueries-94"><span class="linenos"> 94</span></a> <span class="k">for</span> <span class="n">child_scope</span> <span class="ow">in</span> <span class="n">scope</span><span class="o">.</span><span class="n">traverse</span><span class="p">():</span>
</span><span id="eliminate_subqueries-95"><a href="#eliminate_subqueries-95"><span class="linenos"> 95</span></a> <span class="n">new_cte</span> <span class="o">=</span> <span class="n">_eliminate</span><span class="p">(</span><span class="n">child_scope</span><span class="p">,</span> <span class="n">existing_ctes</span><span class="p">,</span> <span class="n">taken</span><span class="p">)</span>
</span><span id="eliminate_subqueries-96"><a href="#eliminate_subqueries-96"><span class="linenos"> 96</span></a> <span class="k">if</span> <span class="n">new_cte</span><span class="p">:</span>
</span><span id="eliminate_subqueries-97"><a href="#eliminate_subqueries-97"><span class="linenos"> 97</span></a> <span class="n">new_ctes</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">new_cte</span><span class="p">)</span>
</span><span id="eliminate_subqueries-98"><a href="#eliminate_subqueries-98"><span class="linenos"> 98</span></a>
</span><span id="eliminate_subqueries-99"><a href="#eliminate_subqueries-99"><span class="linenos"> 99</span></a> <span class="k">if</span> <span class="n">new_ctes</span><span class="p">:</span>
</span><span id="eliminate_subqueries-100"><a href="#eliminate_subqueries-100"><span class="linenos">100</span></a> <span class="n">query</span> <span class="o">=</span> <span class="n">expression</span><span class="o">.</span><span class="n">expression</span> <span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">expression</span><span class="p">,</span> <span class="n">exp</span><span class="o">.</span><span class="n">DDL</span><span class="p">)</span> <span class="k">else</span> <span class="n">expression</span>
</span><span id="eliminate_subqueries-101"><a href="#eliminate_subqueries-101"><span class="linenos">101</span></a> <span class="n">query</span><span class="o">.</span><span class="n">set</span><span class="p">(</span><span class="s2">&quot;with&quot;</span><span class="p">,</span> <span class="n">exp</span><span class="o">.</span><span class="n">With</span><span class="p">(</span><span class="n">expressions</span><span class="o">=</span><span class="n">new_ctes</span><span class="p">,</span> <span class="n">recursive</span><span class="o">=</span><span class="n">recursive</span><span class="p">))</span>
</span><span id="eliminate_subqueries-102"><a href="#eliminate_subqueries-102"><span class="linenos">102</span></a>
</span><span id="eliminate_subqueries-103"><a href="#eliminate_subqueries-103"><span class="linenos">103</span></a> <span class="k">return</span> <span class="n">expression</span>
</span></pre></div>

943
docs/sqlglot/optimizer/merge_subqueries.html
View file

File diff suppressed because it is too large Load diff

289
docs/sqlglot/optimizer/qualify.html
View file

@ -85,78 +85,81 @@
</span><span id="L-27"><a href="#L-27"><span class="linenos"> 27</span></a> <span class="n">infer_schema</span><span class="p">:</span> <span class="n">t</span><span class="o">.</span><span class="n">Optional</span><span class="p">[</span><span class="nb">bool</span><span class="p">]</span> <span class="o">=</span> <span class="kc">None</span><span class="p">,</span>
</span><span id="L-28"><a href="#L-28"><span class="linenos"> 28</span></a> <span class="n">isolate_tables</span><span class="p">:</span> <span class="nb">bool</span> <span class="o">=</span> <span class="kc">False</span><span class="p">,</span>
</span><span id="L-29"><a href="#L-29"><span class="linenos"> 29</span></a> <span class="n">qualify_columns</span><span class="p">:</span> <span class="nb">bool</span> <span class="o">=</span> <span class="kc">True</span><span class="p">,</span>
</span><span id="L-30"><a href="#L-30"><span class="linenos"> 30</span></a> <span class="n">validate_qualify_columns</span><span class="p">:</span> <span class="nb">bool</span> <span class="o">=</span> <span class="kc">True</span><span class="p">,</span>
</span><span id="L-31"><a href="#L-31"><span class="linenos"> 31</span></a> <span class="n">quote_identifiers</span><span class="p">:</span> <span class="nb">bool</span> <span class="o">=</span> <span class="kc">True</span><span class="p">,</span>
</span><span id="L-32"><a href="#L-32"><span class="linenos"> 32</span></a> <span class="n">identify</span><span class="p">:</span> <span class="nb">bool</span> <span class="o">=</span> <span class="kc">True</span><span class="p">,</span>
</span><span id="L-33"><a href="#L-33"><span class="linenos"> 33</span></a> <span class="n">infer_csv_schemas</span><span class="p">:</span> <span class="nb">bool</span> <span class="o">=</span> <span class="kc">False</span><span class="p">,</span>
</span><span id="L-34"><a href="#L-34"><span class="linenos"> 34</span></a><span class="p">)</span> <span class="o">-&gt;</span> <span class="n">exp</span><span class="o">.</span><span class="n">Expression</span><span class="p">:</span>
</span><span id="L-35"><a href="#L-35"><span class="linenos"> 35</span></a><span class="w"> </span><span class="sd">&quot;&quot;&quot;</span>
</span><span id="L-36"><a href="#L-36"><span class="linenos"> 36</span></a><span class="sd"> Rewrite sqlglot AST to have normalized and qualified tables and columns.</span>
</span><span id="L-37"><a href="#L-37"><span class="linenos"> 37</span></a>
</span><span id="L-38"><a href="#L-38"><span class="linenos"> 38</span></a><span class="sd"> This step is necessary for all further SQLGlot optimizations.</span>
</span><span id="L-39"><a href="#L-39"><span class="linenos"> 39</span></a>
</span><span id="L-40"><a href="#L-40"><span class="linenos"> 40</span></a><span class="sd"> Example:</span>
</span><span id="L-41"><a href="#L-41"><span class="linenos"> 41</span></a><span class="sd"> &gt;&gt;&gt; import sqlglot</span>
</span><span id="L-42"><a href="#L-42"><span class="linenos"> 42</span></a><span class="sd"> &gt;&gt;&gt; schema = {&quot;tbl&quot;: {&quot;col&quot;: &quot;INT&quot;}}</span>
</span><span id="L-43"><a href="#L-43"><span class="linenos"> 43</span></a><span class="sd"> &gt;&gt;&gt; expression = sqlglot.parse_one(&quot;SELECT col FROM tbl&quot;)</span>
</span><span id="L-44"><a href="#L-44"><span class="linenos"> 44</span></a><span class="sd"> &gt;&gt;&gt; qualify(expression, schema=schema).sql()</span>
</span><span id="L-45"><a href="#L-45"><span class="linenos"> 45</span></a><span class="sd"> &#39;SELECT &quot;tbl&quot;.&quot;col&quot; AS &quot;col&quot; FROM &quot;tbl&quot; AS &quot;tbl&quot;&#39;</span>
</span><span id="L-46"><a href="#L-46"><span class="linenos"> 46</span></a>
</span><span id="L-47"><a href="#L-47"><span class="linenos"> 47</span></a><span class="sd"> Args:</span>
</span><span id="L-48"><a href="#L-48"><span class="linenos"> 48</span></a><span class="sd"> expression: Expression to qualify.</span>
</span><span id="L-49"><a href="#L-49"><span class="linenos"> 49</span></a><span class="sd"> db: Default database name for tables.</span>
</span><span id="L-50"><a href="#L-50"><span class="linenos"> 50</span></a><span class="sd"> catalog: Default catalog name for tables.</span>
</span><span id="L-51"><a href="#L-51"><span class="linenos"> 51</span></a><span class="sd"> schema: Schema to infer column names and types.</span>
</span><span id="L-52"><a href="#L-52"><span class="linenos"> 52</span></a><span class="sd"> expand_alias_refs: Whether to expand references to aliases.</span>
</span><span id="L-53"><a href="#L-53"><span class="linenos"> 53</span></a><span class="sd"> expand_stars: Whether to expand star queries. This is a necessary step</span>
</span><span id="L-54"><a href="#L-54"><span class="linenos"> 54</span></a><span class="sd"> for most of the optimizer&#39;s rules to work; do not set to False unless you</span>
</span><span id="L-55"><a href="#L-55"><span class="linenos"> 55</span></a><span class="sd"> know what you&#39;re doing!</span>
</span><span id="L-56"><a href="#L-56"><span class="linenos"> 56</span></a><span class="sd"> infer_schema: Whether to infer the schema if missing.</span>
</span><span id="L-57"><a href="#L-57"><span class="linenos"> 57</span></a><span class="sd"> isolate_tables: Whether to isolate table selects.</span>
</span><span id="L-58"><a href="#L-58"><span class="linenos"> 58</span></a><span class="sd"> qualify_columns: Whether to qualify columns.</span>
</span><span id="L-59"><a href="#L-59"><span class="linenos"> 59</span></a><span class="sd"> validate_qualify_columns: Whether to validate columns.</span>
</span><span id="L-60"><a href="#L-60"><span class="linenos"> 60</span></a><span class="sd"> quote_identifiers: Whether to run the quote_identifiers step.</span>
</span><span id="L-61"><a href="#L-61"><span class="linenos"> 61</span></a><span class="sd"> This step is necessary to ensure correctness for case sensitive queries.</span>
</span><span id="L-62"><a href="#L-62"><span class="linenos"> 62</span></a><span class="sd"> But this flag is provided in case this step is performed at a later time.</span>
</span><span id="L-63"><a href="#L-63"><span class="linenos"> 63</span></a><span class="sd"> identify: If True, quote all identifiers, else only necessary ones.</span>
</span><span id="L-64"><a href="#L-64"><span class="linenos"> 64</span></a><span class="sd"> infer_csv_schemas: Whether to scan READ_CSV calls in order to infer the CSVs&#39; schemas.</span>
</span><span id="L-65"><a href="#L-65"><span class="linenos"> 65</span></a>
</span><span id="L-66"><a href="#L-66"><span class="linenos"> 66</span></a><span class="sd"> Returns:</span>
</span><span id="L-67"><a href="#L-67"><span class="linenos"> 67</span></a><span class="sd"> The qualified expression.</span>
</span><span id="L-68"><a href="#L-68"><span class="linenos"> 68</span></a><span class="sd"> &quot;&quot;&quot;</span>
</span><span id="L-69"><a href="#L-69"><span class="linenos"> 69</span></a> <span class="n">schema</span> <span class="o">=</span> <span class="n">ensure_schema</span><span class="p">(</span><span class="n">schema</span><span class="p">,</span> <span class="n">dialect</span><span class="o">=</span><span class="n">dialect</span><span class="p">)</span>
</span><span id="L-70"><a href="#L-70"><span class="linenos"> 70</span></a> <span class="n">expression</span> <span class="o">=</span> <span class="n">normalize_identifiers</span><span class="p">(</span><span class="n">expression</span><span class="p">,</span> <span class="n">dialect</span><span class="o">=</span><span class="n">dialect</span><span class="p">)</span>
</span><span id="L-71"><a href="#L-71"><span class="linenos"> 71</span></a> <span class="n">expression</span> <span class="o">=</span> <span class="n">qualify_tables</span><span class="p">(</span>
</span><span id="L-72"><a href="#L-72"><span class="linenos"> 72</span></a> <span class="n">expression</span><span class="p">,</span>
</span><span id="L-73"><a href="#L-73"><span class="linenos"> 73</span></a> <span class="n">db</span><span class="o">=</span><span class="n">db</span><span class="p">,</span>
</span><span id="L-74"><a href="#L-74"><span class="linenos"> 74</span></a> <span class="n">catalog</span><span class="o">=</span><span class="n">catalog</span><span class="p">,</span>
</span><span id="L-75"><a href="#L-75"><span class="linenos"> 75</span></a> <span class="n">schema</span><span class="o">=</span><span class="n">schema</span><span class="p">,</span>
</span><span id="L-76"><a href="#L-76"><span class="linenos"> 76</span></a> <span class="n">dialect</span><span class="o">=</span><span class="n">dialect</span><span class="p">,</span>
</span><span id="L-77"><a href="#L-77"><span class="linenos"> 77</span></a> <span class="n">infer_csv_schemas</span><span class="o">=</span><span class="n">infer_csv_schemas</span><span class="p">,</span>
</span><span id="L-78"><a href="#L-78"><span class="linenos"> 78</span></a> <span class="p">)</span>
</span><span id="L-79"><a href="#L-79"><span class="linenos"> 79</span></a>
</span><span id="L-80"><a href="#L-80"><span class="linenos"> 80</span></a> <span class="k">if</span> <span class="n">isolate_tables</span><span class="p">:</span>
</span><span id="L-81"><a href="#L-81"><span class="linenos"> 81</span></a> <span class="n">expression</span> <span class="o">=</span> <span class="n">isolate_table_selects</span><span class="p">(</span><span class="n">expression</span><span class="p">,</span> <span class="n">schema</span><span class="o">=</span><span class="n">schema</span><span class="p">)</span>
</span><span id="L-82"><a href="#L-82"><span class="linenos"> 82</span></a>
</span><span id="L-83"><a href="#L-83"><span class="linenos"> 83</span></a> <span class="k">if</span> <span class="n">Dialect</span><span class="o">.</span><span class="n">get_or_raise</span><span class="p">(</span><span class="n">dialect</span><span class="p">)</span><span class="o">.</span><span class="n">PREFER_CTE_ALIAS_COLUMN</span><span class="p">:</span>
</span><span id="L-84"><a href="#L-84"><span class="linenos"> 84</span></a> <span class="n">expression</span> <span class="o">=</span> <span class="n">pushdown_cte_alias_columns_func</span><span class="p">(</span><span class="n">expression</span><span class="p">)</span>
</span><span id="L-85"><a href="#L-85"><span class="linenos"> 85</span></a>
</span><span id="L-86"><a href="#L-86"><span class="linenos"> 86</span></a> <span class="k">if</span> <span class="n">qualify_columns</span><span class="p">:</span>
</span><span id="L-87"><a href="#L-87"><span class="linenos"> 87</span></a> <span class="n">expression</span> <span class="o">=</span> <span class="n">qualify_columns_func</span><span class="p">(</span>
</span><span id="L-88"><a href="#L-88"><span class="linenos"> 88</span></a> <span class="n">expression</span><span class="p">,</span>
</span><span id="L-89"><a href="#L-89"><span class="linenos"> 89</span></a> <span class="n">schema</span><span class="p">,</span>
</span><span id="L-90"><a href="#L-90"><span class="linenos"> 90</span></a> <span class="n">expand_alias_refs</span><span class="o">=</span><span class="n">expand_alias_refs</span><span class="p">,</span>
</span><span id="L-91"><a href="#L-91"><span class="linenos"> 91</span></a> <span class="n">expand_stars</span><span class="o">=</span><span class="n">expand_stars</span><span class="p">,</span>
</span><span id="L-92"><a href="#L-92"><span class="linenos"> 92</span></a> <span class="n">infer_schema</span><span class="o">=</span><span class="n">infer_schema</span><span class="p">,</span>
</span><span id="L-93"><a href="#L-93"><span class="linenos"> 93</span></a> <span class="p">)</span>
</span><span id="L-94"><a href="#L-94"><span class="linenos"> 94</span></a>
</span><span id="L-95"><a href="#L-95"><span class="linenos"> 95</span></a> <span class="k">if</span> <span class="n">quote_identifiers</span><span class="p">:</span>
</span><span id="L-96"><a href="#L-96"><span class="linenos"> 96</span></a> <span class="n">expression</span> <span class="o">=</span> <span class="n">quote_identifiers_func</span><span class="p">(</span><span class="n">expression</span><span class="p">,</span> <span class="n">dialect</span><span class="o">=</span><span class="n">dialect</span><span class="p">,</span> <span class="n">identify</span><span class="o">=</span><span class="n">identify</span><span class="p">)</span>
</span><span id="L-30"><a href="#L-30"><span class="linenos"> 30</span></a> <span class="n">allow_partial_qualification</span><span class="p">:</span> <span class="nb">bool</span> <span class="o">=</span> <span class="kc">False</span><span class="p">,</span>
</span><span id="L-31"><a href="#L-31"><span class="linenos"> 31</span></a> <span class="n">validate_qualify_columns</span><span class="p">:</span> <span class="nb">bool</span> <span class="o">=</span> <span class="kc">True</span><span class="p">,</span>
</span><span id="L-32"><a href="#L-32"><span class="linenos"> 32</span></a> <span class="n">quote_identifiers</span><span class="p">:</span> <span class="nb">bool</span> <span class="o">=</span> <span class="kc">True</span><span class="p">,</span>
</span><span id="L-33"><a href="#L-33"><span class="linenos"> 33</span></a> <span class="n">identify</span><span class="p">:</span> <span class="nb">bool</span> <span class="o">=</span> <span class="kc">True</span><span class="p">,</span>
</span><span id="L-34"><a href="#L-34"><span class="linenos"> 34</span></a> <span class="n">infer_csv_schemas</span><span class="p">:</span> <span class="nb">bool</span> <span class="o">=</span> <span class="kc">False</span><span class="p">,</span>
</span><span id="L-35"><a href="#L-35"><span class="linenos"> 35</span></a><span class="p">)</span> <span class="o">-&gt;</span> <span class="n">exp</span><span class="o">.</span><span class="n">Expression</span><span class="p">:</span>
</span><span id="L-36"><a href="#L-36"><span class="linenos"> 36</span></a><span class="w"> </span><span class="sd">&quot;&quot;&quot;</span>
</span><span id="L-37"><a href="#L-37"><span class="linenos"> 37</span></a><span class="sd"> Rewrite sqlglot AST to have normalized and qualified tables and columns.</span>
</span><span id="L-38"><a href="#L-38"><span class="linenos"> 38</span></a>
</span><span id="L-39"><a href="#L-39"><span class="linenos"> 39</span></a><span class="sd"> This step is necessary for all further SQLGlot optimizations.</span>
</span><span id="L-40"><a href="#L-40"><span class="linenos"> 40</span></a>
</span><span id="L-41"><a href="#L-41"><span class="linenos"> 41</span></a><span class="sd"> Example:</span>
</span><span id="L-42"><a href="#L-42"><span class="linenos"> 42</span></a><span class="sd"> &gt;&gt;&gt; import sqlglot</span>
</span><span id="L-43"><a href="#L-43"><span class="linenos"> 43</span></a><span class="sd"> &gt;&gt;&gt; schema = {&quot;tbl&quot;: {&quot;col&quot;: &quot;INT&quot;}}</span>
</span><span id="L-44"><a href="#L-44"><span class="linenos"> 44</span></a><span class="sd"> &gt;&gt;&gt; expression = sqlglot.parse_one(&quot;SELECT col FROM tbl&quot;)</span>
</span><span id="L-45"><a href="#L-45"><span class="linenos"> 45</span></a><span class="sd"> &gt;&gt;&gt; qualify(expression, schema=schema).sql()</span>
</span><span id="L-46"><a href="#L-46"><span class="linenos"> 46</span></a><span class="sd"> &#39;SELECT &quot;tbl&quot;.&quot;col&quot; AS &quot;col&quot; FROM &quot;tbl&quot; AS &quot;tbl&quot;&#39;</span>
</span><span id="L-47"><a href="#L-47"><span class="linenos"> 47</span></a>
</span><span id="L-48"><a href="#L-48"><span class="linenos"> 48</span></a><span class="sd"> Args:</span>
</span><span id="L-49"><a href="#L-49"><span class="linenos"> 49</span></a><span class="sd"> expression: Expression to qualify.</span>
</span><span id="L-50"><a href="#L-50"><span class="linenos"> 50</span></a><span class="sd"> db: Default database name for tables.</span>
</span><span id="L-51"><a href="#L-51"><span class="linenos"> 51</span></a><span class="sd"> catalog: Default catalog name for tables.</span>
</span><span id="L-52"><a href="#L-52"><span class="linenos"> 52</span></a><span class="sd"> schema: Schema to infer column names and types.</span>
</span><span id="L-53"><a href="#L-53"><span class="linenos"> 53</span></a><span class="sd"> expand_alias_refs: Whether to expand references to aliases.</span>
</span><span id="L-54"><a href="#L-54"><span class="linenos"> 54</span></a><span class="sd"> expand_stars: Whether to expand star queries. This is a necessary step</span>
</span><span id="L-55"><a href="#L-55"><span class="linenos"> 55</span></a><span class="sd"> for most of the optimizer&#39;s rules to work; do not set to False unless you</span>
</span><span id="L-56"><a href="#L-56"><span class="linenos"> 56</span></a><span class="sd"> know what you&#39;re doing!</span>
</span><span id="L-57"><a href="#L-57"><span class="linenos"> 57</span></a><span class="sd"> infer_schema: Whether to infer the schema if missing.</span>
</span><span id="L-58"><a href="#L-58"><span class="linenos"> 58</span></a><span class="sd"> isolate_tables: Whether to isolate table selects.</span>
</span><span id="L-59"><a href="#L-59"><span class="linenos"> 59</span></a><span class="sd"> qualify_columns: Whether to qualify columns.</span>
</span><span id="L-60"><a href="#L-60"><span class="linenos"> 60</span></a><span class="sd"> allow_partial_qualification: Whether to allow partial qualification.</span>
</span><span id="L-61"><a href="#L-61"><span class="linenos"> 61</span></a><span class="sd"> validate_qualify_columns: Whether to validate columns.</span>
</span><span id="L-62"><a href="#L-62"><span class="linenos"> 62</span></a><span class="sd"> quote_identifiers: Whether to run the quote_identifiers step.</span>
</span><span id="L-63"><a href="#L-63"><span class="linenos"> 63</span></a><span class="sd"> This step is necessary to ensure correctness for case sensitive queries.</span>
</span><span id="L-64"><a href="#L-64"><span class="linenos"> 64</span></a><span class="sd"> But this flag is provided in case this step is performed at a later time.</span>
</span><span id="L-65"><a href="#L-65"><span class="linenos"> 65</span></a><span class="sd"> identify: If True, quote all identifiers, else only necessary ones.</span>
</span><span id="L-66"><a href="#L-66"><span class="linenos"> 66</span></a><span class="sd"> infer_csv_schemas: Whether to scan READ_CSV calls in order to infer the CSVs&#39; schemas.</span>
</span><span id="L-67"><a href="#L-67"><span class="linenos"> 67</span></a>
</span><span id="L-68"><a href="#L-68"><span class="linenos"> 68</span></a><span class="sd"> Returns:</span>
</span><span id="L-69"><a href="#L-69"><span class="linenos"> 69</span></a><span class="sd"> The qualified expression.</span>
</span><span id="L-70"><a href="#L-70"><span class="linenos"> 70</span></a><span class="sd"> &quot;&quot;&quot;</span>
</span><span id="L-71"><a href="#L-71"><span class="linenos"> 71</span></a> <span class="n">schema</span> <span class="o">=</span> <span class="n">ensure_schema</span><span class="p">(</span><span class="n">schema</span><span class="p">,</span> <span class="n">dialect</span><span class="o">=</span><span class="n">dialect</span><span class="p">)</span>
</span><span id="L-72"><a href="#L-72"><span class="linenos"> 72</span></a> <span class="n">expression</span> <span class="o">=</span> <span class="n">normalize_identifiers</span><span class="p">(</span><span class="n">expression</span><span class="p">,</span> <span class="n">dialect</span><span class="o">=</span><span class="n">dialect</span><span class="p">)</span>
</span><span id="L-73"><a href="#L-73"><span class="linenos"> 73</span></a> <span class="n">expression</span> <span class="o">=</span> <span class="n">qualify_tables</span><span class="p">(</span>
</span><span id="L-74"><a href="#L-74"><span class="linenos"> 74</span></a> <span class="n">expression</span><span class="p">,</span>
</span><span id="L-75"><a href="#L-75"><span class="linenos"> 75</span></a> <span class="n">db</span><span class="o">=</span><span class="n">db</span><span class="p">,</span>
</span><span id="L-76"><a href="#L-76"><span class="linenos"> 76</span></a> <span class="n">catalog</span><span class="o">=</span><span class="n">catalog</span><span class="p">,</span>
</span><span id="L-77"><a href="#L-77"><span class="linenos"> 77</span></a> <span class="n">schema</span><span class="o">=</span><span class="n">schema</span><span class="p">,</span>
</span><span id="L-78"><a href="#L-78"><span class="linenos"> 78</span></a> <span class="n">dialect</span><span class="o">=</span><span class="n">dialect</span><span class="p">,</span>
</span><span id="L-79"><a href="#L-79"><span class="linenos"> 79</span></a> <span class="n">infer_csv_schemas</span><span class="o">=</span><span class="n">infer_csv_schemas</span><span class="p">,</span>
</span><span id="L-80"><a href="#L-80"><span class="linenos"> 80</span></a> <span class="p">)</span>
</span><span id="L-81"><a href="#L-81"><span class="linenos"> 81</span></a>
</span><span id="L-82"><a href="#L-82"><span class="linenos"> 82</span></a> <span class="k">if</span> <span class="n">isolate_tables</span><span class="p">:</span>
</span><span id="L-83"><a href="#L-83"><span class="linenos"> 83</span></a> <span class="n">expression</span> <span class="o">=</span> <span class="n">isolate_table_selects</span><span class="p">(</span><span class="n">expression</span><span class="p">,</span> <span class="n">schema</span><span class="o">=</span><span class="n">schema</span><span class="p">)</span>
</span><span id="L-84"><a href="#L-84"><span class="linenos"> 84</span></a>
</span><span id="L-85"><a href="#L-85"><span class="linenos"> 85</span></a> <span class="k">if</span> <span class="n">Dialect</span><span class="o">.</span><span class="n">get_or_raise</span><span class="p">(</span><span class="n">dialect</span><span class="p">)</span><span class="o">.</span><span class="n">PREFER_CTE_ALIAS_COLUMN</span><span class="p">:</span>
</span><span id="L-86"><a href="#L-86"><span class="linenos"> 86</span></a> <span class="n">expression</span> <span class="o">=</span> <span class="n">pushdown_cte_alias_columns_func</span><span class="p">(</span><span class="n">expression</span><span class="p">)</span>
</span><span id="L-87"><a href="#L-87"><span class="linenos"> 87</span></a>
</span><span id="L-88"><a href="#L-88"><span class="linenos"> 88</span></a> <span class="k">if</span> <span class="n">qualify_columns</span><span class="p">:</span>
</span><span id="L-89"><a href="#L-89"><span class="linenos"> 89</span></a> <span class="n">expression</span> <span class="o">=</span> <span class="n">qualify_columns_func</span><span class="p">(</span>
</span><span id="L-90"><a href="#L-90"><span class="linenos"> 90</span></a> <span class="n">expression</span><span class="p">,</span>
</span><span id="L-91"><a href="#L-91"><span class="linenos"> 91</span></a> <span class="n">schema</span><span class="p">,</span>
</span><span id="L-92"><a href="#L-92"><span class="linenos"> 92</span></a> <span class="n">expand_alias_refs</span><span class="o">=</span><span class="n">expand_alias_refs</span><span class="p">,</span>
</span><span id="L-93"><a href="#L-93"><span class="linenos"> 93</span></a> <span class="n">expand_stars</span><span class="o">=</span><span class="n">expand_stars</span><span class="p">,</span>
</span><span id="L-94"><a href="#L-94"><span class="linenos"> 94</span></a> <span class="n">infer_schema</span><span class="o">=</span><span class="n">infer_schema</span><span class="p">,</span>
</span><span id="L-95"><a href="#L-95"><span class="linenos"> 95</span></a> <span class="n">allow_partial_qualification</span><span class="o">=</span><span class="n">allow_partial_qualification</span><span class="p">,</span>
</span><span id="L-96"><a href="#L-96"><span class="linenos"> 96</span></a> <span class="p">)</span>
</span><span id="L-97"><a href="#L-97"><span class="linenos"> 97</span></a>
</span><span id="L-98"><a href="#L-98"><span class="linenos"> 98</span></a> <span class="k">if</span> <span class="n">validate_qualify_columns</span><span class="p">:</span>
</span><span id="L-99"><a href="#L-99"><span class="linenos"> 99</span></a> <span class="n">validate_qualify_columns_func</span><span class="p">(</span><span class="n">expression</span><span class="p">)</span>
</span><span id="L-98"><a href="#L-98"><span class="linenos"> 98</span></a> <span class="k">if</span> <span class="n">quote_identifiers</span><span class="p">:</span>
</span><span id="L-99"><a href="#L-99"><span class="linenos"> 99</span></a> <span class="n">expression</span> <span class="o">=</span> <span class="n">quote_identifiers_func</span><span class="p">(</span><span class="n">expression</span><span class="p">,</span> <span class="n">dialect</span><span class="o">=</span><span class="n">dialect</span><span class="p">,</span> <span class="n">identify</span><span class="o">=</span><span class="n">identify</span><span class="p">)</span>
</span><span id="L-100"><a href="#L-100"><span class="linenos">100</span></a>
</span><span id="L-101"><a href="#L-101"><span class="linenos">101</span></a> <span class="k">return</span> <span class="n">expression</span>
</span><span id="L-101"><a href="#L-101"><span class="linenos">101</span></a> <span class="k">if</span> <span class="n">validate_qualify_columns</span><span class="p">:</span>
</span><span id="L-102"><a href="#L-102"><span class="linenos">102</span></a> <span class="n">validate_qualify_columns_func</span><span class="p">(</span><span class="n">expression</span><span class="p">)</span>
</span><span id="L-103"><a href="#L-103"><span class="linenos">103</span></a>
</span><span id="L-104"><a href="#L-104"><span class="linenos">104</span></a> <span class="k">return</span> <span class="n">expression</span>
</span></pre></div>
@ -166,7 +169,7 @@
<div class="attr function">
<span class="def">def</span>
<span class="name">qualify</span><span class="signature pdoc-code multiline">(<span class="param"> <span class="n">expression</span><span class="p">:</span> <span class="n"><a href="../expressions.html#Expression">sqlglot.expressions.Expression</a></span>,</span><span class="param"> <span class="n">dialect</span><span class="p">:</span> <span class="n">Union</span><span class="p">[</span><span class="nb">str</span><span class="p">,</span> <span class="n"><a href="../dialects/dialect.html#Dialect">sqlglot.dialects.dialect.Dialect</a></span><span class="p">,</span> <span class="n">Type</span><span class="p">[</span><span class="n"><a href="../dialects/dialect.html#Dialect">sqlglot.dialects.dialect.Dialect</a></span><span class="p">],</span> <span class="n">NoneType</span><span class="p">]</span> <span class="o">=</span> <span class="kc">None</span>,</span><span class="param"> <span class="n">db</span><span class="p">:</span> <span class="n">Optional</span><span class="p">[</span><span class="nb">str</span><span class="p">]</span> <span class="o">=</span> <span class="kc">None</span>,</span><span class="param"> <span class="n">catalog</span><span class="p">:</span> <span class="n">Optional</span><span class="p">[</span><span class="nb">str</span><span class="p">]</span> <span class="o">=</span> <span class="kc">None</span>,</span><span class="param"> <span class="n">schema</span><span class="p">:</span> <span class="n">Union</span><span class="p">[</span><span class="nb">dict</span><span class="p">,</span> <span class="n"><a href="../schema.html#Schema">sqlglot.schema.Schema</a></span><span class="p">,</span> <span class="n">NoneType</span><span class="p">]</span> <span class="o">=</span> <span class="kc">None</span>,</span><span class="param"> <span class="n">expand_alias_refs</span><span class="p">:</span> <span class="nb">bool</span> <span class="o">=</span> <span class="kc">True</span>,</span><span class="param"> <span class="n">expand_stars</span><span class="p">:</span> <span class="nb">bool</span> <span class="o">=</span> <span class="kc">True</span>,</span><span class="param"> <span class="n">infer_schema</span><span class="p">:</span> <span class="n">Optional</span><span class="p">[</span><span class="nb">bool</span><span class="p">]</span> <span class="o">=</span> <span class="kc">None</span>,</span><span class="param"> <span class="n">isolate_tables</span><span class="p">:</span> <span class="nb">bool</span> <span class="o">=</span> <span class="kc">False</span>,</span><span class="param"> <span class="n">qualify_columns</span><span class="p">:</span> <span class="nb">bool</span> <span class="o">=</span> <span class="kc">True</span>,</span><span class="param"> <span class="n">validate_qualify_columns</span><span class="p">:</span> <span class="nb">bool</span> <span class="o">=</span> <span class="kc">True</span>,</span><span class="param"> <span class="n">quote_identifiers</span><span class="p">:</span> <span class="nb">bool</span> <span class="o">=</span> <span class="kc">True</span>,</span><span class="param"> <span class="n">identify</span><span class="p">:</span> <span class="nb">bool</span> <span class="o">=</span> <span class="kc">True</span>,</span><span class="param"> <span class="n">infer_csv_schemas</span><span class="p">:</span> <span class="nb">bool</span> <span class="o">=</span> <span class="kc">False</span></span><span class="return-annotation">) -> <span class="n"><a href="../expressions.html#Expression">sqlglot.expressions.Expression</a></span>:</span></span>
<span class="name">qualify</span><span class="signature pdoc-code multiline">(<span class="param"> <span class="n">expression</span><span class="p">:</span> <span class="n"><a href="../expressions.html#Expression">sqlglot.expressions.Expression</a></span>,</span><span class="param"> <span class="n">dialect</span><span class="p">:</span> <span class="n">Union</span><span class="p">[</span><span class="nb">str</span><span class="p">,</span> <span class="n"><a href="../dialects/dialect.html#Dialect">sqlglot.dialects.dialect.Dialect</a></span><span class="p">,</span> <span class="n">Type</span><span class="p">[</span><span class="n"><a href="../dialects/dialect.html#Dialect">sqlglot.dialects.dialect.Dialect</a></span><span class="p">],</span> <span class="n">NoneType</span><span class="p">]</span> <span class="o">=</span> <span class="kc">None</span>,</span><span class="param"> <span class="n">db</span><span class="p">:</span> <span class="n">Optional</span><span class="p">[</span><span class="nb">str</span><span class="p">]</span> <span class="o">=</span> <span class="kc">None</span>,</span><span class="param"> <span class="n">catalog</span><span class="p">:</span> <span class="n">Optional</span><span class="p">[</span><span class="nb">str</span><span class="p">]</span> <span class="o">=</span> <span class="kc">None</span>,</span><span class="param"> <span class="n">schema</span><span class="p">:</span> <span class="n">Union</span><span class="p">[</span><span class="nb">dict</span><span class="p">,</span> <span class="n"><a href="../schema.html#Schema">sqlglot.schema.Schema</a></span><span class="p">,</span> <span class="n">NoneType</span><span class="p">]</span> <span class="o">=</span> <span class="kc">None</span>,</span><span class="param"> <span class="n">expand_alias_refs</span><span class="p">:</span> <span class="nb">bool</span> <span class="o">=</span> <span class="kc">True</span>,</span><span class="param"> <span class="n">expand_stars</span><span class="p">:</span> <span class="nb">bool</span> <span class="o">=</span> <span class="kc">True</span>,</span><span class="param"> <span class="n">infer_schema</span><span class="p">:</span> <span class="n">Optional</span><span class="p">[</span><span class="nb">bool</span><span class="p">]</span> <span class="o">=</span> <span class="kc">None</span>,</span><span class="param"> <span class="n">isolate_tables</span><span class="p">:</span> <span class="nb">bool</span> <span class="o">=</span> <span class="kc">False</span>,</span><span class="param"> <span class="n">qualify_columns</span><span class="p">:</span> <span class="nb">bool</span> <span class="o">=</span> <span class="kc">True</span>,</span><span class="param"> <span class="n">allow_partial_qualification</span><span class="p">:</span> <span class="nb">bool</span> <span class="o">=</span> <span class="kc">False</span>,</span><span class="param"> <span class="n">validate_qualify_columns</span><span class="p">:</span> <span class="nb">bool</span> <span class="o">=</span> <span class="kc">True</span>,</span><span class="param"> <span class="n">quote_identifiers</span><span class="p">:</span> <span class="nb">bool</span> <span class="o">=</span> <span class="kc">True</span>,</span><span class="param"> <span class="n">identify</span><span class="p">:</span> <span class="nb">bool</span> <span class="o">=</span> <span class="kc">True</span>,</span><span class="param"> <span class="n">infer_csv_schemas</span><span class="p">:</span> <span class="nb">bool</span> <span class="o">=</span> <span class="kc">False</span></span><span class="return-annotation">) -> <span class="n"><a href="../expressions.html#Expression">sqlglot.expressions.Expression</a></span>:</span></span>
<label class="view-source-button" for="qualify-view-source"><span>View Source</span></label>
@ -183,78 +186,81 @@
</span><span id="qualify-28"><a href="#qualify-28"><span class="linenos"> 28</span></a> <span class="n">infer_schema</span><span class="p">:</span> <span class="n">t</span><span class="o">.</span><span class="n">Optional</span><span class="p">[</span><span class="nb">bool</span><span class="p">]</span> <span class="o">=</span> <span class="kc">None</span><span class="p">,</span>
</span><span id="qualify-29"><a href="#qualify-29"><span class="linenos"> 29</span></a> <span class="n">isolate_tables</span><span class="p">:</span> <span class="nb">bool</span> <span class="o">=</span> <span class="kc">False</span><span class="p">,</span>
</span><span id="qualify-30"><a href="#qualify-30"><span class="linenos"> 30</span></a> <span class="n">qualify_columns</span><span class="p">:</span> <span class="nb">bool</span> <span class="o">=</span> <span class="kc">True</span><span class="p">,</span>
</span><span id="qualify-31"><a href="#qualify-31"><span class="linenos"> 31</span></a> <span class="n">validate_qualify_columns</span><span class="p">:</span> <span class="nb">bool</span> <span class="o">=</span> <span class="kc">True</span><span class="p">,</span>
</span><span id="qualify-32"><a href="#qualify-32"><span class="linenos"> 32</span></a> <span class="n">quote_identifiers</span><span class="p">:</span> <span class="nb">bool</span> <span class="o">=</span> <span class="kc">True</span><span class="p">,</span>
</span><span id="qualify-33"><a href="#qualify-33"><span class="linenos"> 33</span></a> <span class="n">identify</span><span class="p">:</span> <span class="nb">bool</span> <span class="o">=</span> <span class="kc">True</span><span class="p">,</span>
</span><span id="qualify-34"><a href="#qualify-34"><span class="linenos"> 34</span></a> <span class="n">infer_csv_schemas</span><span class="p">:</span> <span class="nb">bool</span> <span class="o">=</span> <span class="kc">False</span><span class="p">,</span>
</span><span id="qualify-35"><a href="#qualify-35"><span class="linenos"> 35</span></a><span class="p">)</span> <span class="o">-&gt;</span> <span class="n">exp</span><span class="o">.</span><span class="n">Expression</span><span class="p">:</span>
</span><span id="qualify-36"><a href="#qualify-36"><span class="linenos"> 36</span></a><span class="w"> </span><span class="sd">&quot;&quot;&quot;</span>
</span><span id="qualify-37"><a href="#qualify-37"><span class="linenos"> 37</span></a><span class="sd"> Rewrite sqlglot AST to have normalized and qualified tables and columns.</span>
</span><span id="qualify-38"><a href="#qualify-38"><span class="linenos"> 38</span></a>
</span><span id="qualify-39"><a href="#qualify-39"><span class="linenos"> 39</span></a><span class="sd"> This step is necessary for all further SQLGlot optimizations.</span>
</span><span id="qualify-40"><a href="#qualify-40"><span class="linenos"> 40</span></a>
</span><span id="qualify-41"><a href="#qualify-41"><span class="linenos"> 41</span></a><span class="sd"> Example:</span>
</span><span id="qualify-42"><a href="#qualify-42"><span class="linenos"> 42</span></a><span class="sd"> &gt;&gt;&gt; import sqlglot</span>
</span><span id="qualify-43"><a href="#qualify-43"><span class="linenos"> 43</span></a><span class="sd"> &gt;&gt;&gt; schema = {&quot;tbl&quot;: {&quot;col&quot;: &quot;INT&quot;}}</span>
</span><span id="qualify-44"><a href="#qualify-44"><span class="linenos"> 44</span></a><span class="sd"> &gt;&gt;&gt; expression = sqlglot.parse_one(&quot;SELECT col FROM tbl&quot;)</span>
</span><span id="qualify-45"><a href="#qualify-45"><span class="linenos"> 45</span></a><span class="sd"> &gt;&gt;&gt; qualify(expression, schema=schema).sql()</span>
</span><span id="qualify-46"><a href="#qualify-46"><span class="linenos"> 46</span></a><span class="sd"> &#39;SELECT &quot;tbl&quot;.&quot;col&quot; AS &quot;col&quot; FROM &quot;tbl&quot; AS &quot;tbl&quot;&#39;</span>
</span><span id="qualify-47"><a href="#qualify-47"><span class="linenos"> 47</span></a>
</span><span id="qualify-48"><a href="#qualify-48"><span class="linenos"> 48</span></a><span class="sd"> Args:</span>
</span><span id="qualify-49"><a href="#qualify-49"><span class="linenos"> 49</span></a><span class="sd"> expression: Expression to qualify.</span>
</span><span id="qualify-50"><a href="#qualify-50"><span class="linenos"> 50</span></a><span class="sd"> db: Default database name for tables.</span>
</span><span id="qualify-51"><a href="#qualify-51"><span class="linenos"> 51</span></a><span class="sd"> catalog: Default catalog name for tables.</span>
</span><span id="qualify-52"><a href="#qualify-52"><span class="linenos"> 52</span></a><span class="sd"> schema: Schema to infer column names and types.</span>
</span><span id="qualify-53"><a href="#qualify-53"><span class="linenos"> 53</span></a><span class="sd"> expand_alias_refs: Whether to expand references to aliases.</span>
</span><span id="qualify-54"><a href="#qualify-54"><span class="linenos"> 54</span></a><span class="sd"> expand_stars: Whether to expand star queries. This is a necessary step</span>
</span><span id="qualify-55"><a href="#qualify-55"><span class="linenos"> 55</span></a><span class="sd"> for most of the optimizer&#39;s rules to work; do not set to False unless you</span>
</span><span id="qualify-56"><a href="#qualify-56"><span class="linenos"> 56</span></a><span class="sd"> know what you&#39;re doing!</span>
</span><span id="qualify-57"><a href="#qualify-57"><span class="linenos"> 57</span></a><span class="sd"> infer_schema: Whether to infer the schema if missing.</span>
</span><span id="qualify-58"><a href="#qualify-58"><span class="linenos"> 58</span></a><span class="sd"> isolate_tables: Whether to isolate table selects.</span>
</span><span id="qualify-59"><a href="#qualify-59"><span class="linenos"> 59</span></a><span class="sd"> qualify_columns: Whether to qualify columns.</span>
</span><span id="qualify-60"><a href="#qualify-60"><span class="linenos"> 60</span></a><span class="sd"> validate_qualify_columns: Whether to validate columns.</span>
</span><span id="qualify-61"><a href="#qualify-61"><span class="linenos"> 61</span></a><span class="sd"> quote_identifiers: Whether to run the quote_identifiers step.</span>
</span><span id="qualify-62"><a href="#qualify-62"><span class="linenos"> 62</span></a><span class="sd"> This step is necessary to ensure correctness for case sensitive queries.</span>
</span><span id="qualify-63"><a href="#qualify-63"><span class="linenos"> 63</span></a><span class="sd"> But this flag is provided in case this step is performed at a later time.</span>
</span><span id="qualify-64"><a href="#qualify-64"><span class="linenos"> 64</span></a><span class="sd"> identify: If True, quote all identifiers, else only necessary ones.</span>
</span><span id="qualify-65"><a href="#qualify-65"><span class="linenos"> 65</span></a><span class="sd"> infer_csv_schemas: Whether to scan READ_CSV calls in order to infer the CSVs&#39; schemas.</span>
</span><span id="qualify-66"><a href="#qualify-66"><span class="linenos"> 66</span></a>
</span><span id="qualify-67"><a href="#qualify-67"><span class="linenos"> 67</span></a><span class="sd"> Returns:</span>
</span><span id="qualify-68"><a href="#qualify-68"><span class="linenos"> 68</span></a><span class="sd"> The qualified expression.</span>
</span><span id="qualify-69"><a href="#qualify-69"><span class="linenos"> 69</span></a><span class="sd"> &quot;&quot;&quot;</span>
</span><span id="qualify-70"><a href="#qualify-70"><span class="linenos"> 70</span></a> <span class="n">schema</span> <span class="o">=</span> <span class="n">ensure_schema</span><span class="p">(</span><span class="n">schema</span><span class="p">,</span> <span class="n">dialect</span><span class="o">=</span><span class="n">dialect</span><span class="p">)</span>
</span><span id="qualify-71"><a href="#qualify-71"><span class="linenos"> 71</span></a> <span class="n">expression</span> <span class="o">=</span> <span class="n">normalize_identifiers</span><span class="p">(</span><span class="n">expression</span><span class="p">,</span> <span class="n">dialect</span><span class="o">=</span><span class="n">dialect</span><span class="p">)</span>
</span><span id="qualify-72"><a href="#qualify-72"><span class="linenos"> 72</span></a> <span class="n">expression</span> <span class="o">=</span> <span class="n">qualify_tables</span><span class="p">(</span>
</span><span id="qualify-73"><a href="#qualify-73"><span class="linenos"> 73</span></a> <span class="n">expression</span><span class="p">,</span>
</span><span id="qualify-74"><a href="#qualify-74"><span class="linenos"> 74</span></a> <span class="n">db</span><span class="o">=</span><span class="n">db</span><span class="p">,</span>
</span><span id="qualify-75"><a href="#qualify-75"><span class="linenos"> 75</span></a> <span class="n">catalog</span><span class="o">=</span><span class="n">catalog</span><span class="p">,</span>
</span><span id="qualify-76"><a href="#qualify-76"><span class="linenos"> 76</span></a> <span class="n">schema</span><span class="o">=</span><span class="n">schema</span><span class="p">,</span>
</span><span id="qualify-77"><a href="#qualify-77"><span class="linenos"> 77</span></a> <span class="n">dialect</span><span class="o">=</span><span class="n">dialect</span><span class="p">,</span>
</span><span id="qualify-78"><a href="#qualify-78"><span class="linenos"> 78</span></a> <span class="n">infer_csv_schemas</span><span class="o">=</span><span class="n">infer_csv_schemas</span><span class="p">,</span>
</span><span id="qualify-79"><a href="#qualify-79"><span class="linenos"> 79</span></a> <span class="p">)</span>
</span><span id="qualify-80"><a href="#qualify-80"><span class="linenos"> 80</span></a>
</span><span id="qualify-81"><a href="#qualify-81"><span class="linenos"> 81</span></a> <span class="k">if</span> <span class="n">isolate_tables</span><span class="p">:</span>
</span><span id="qualify-82"><a href="#qualify-82"><span class="linenos"> 82</span></a> <span class="n">expression</span> <span class="o">=</span> <span class="n">isolate_table_selects</span><span class="p">(</span><span class="n">expression</span><span class="p">,</span> <span class="n">schema</span><span class="o">=</span><span class="n">schema</span><span class="p">)</span>
</span><span id="qualify-83"><a href="#qualify-83"><span class="linenos"> 83</span></a>
</span><span id="qualify-84"><a href="#qualify-84"><span class="linenos"> 84</span></a> <span class="k">if</span> <span class="n">Dialect</span><span class="o">.</span><span class="n">get_or_raise</span><span class="p">(</span><span class="n">dialect</span><span class="p">)</span><span class="o">.</span><span class="n">PREFER_CTE_ALIAS_COLUMN</span><span class="p">:</span>
</span><span id="qualify-85"><a href="#qualify-85"><span class="linenos"> 85</span></a> <span class="n">expression</span> <span class="o">=</span> <span class="n">pushdown_cte_alias_columns_func</span><span class="p">(</span><span class="n">expression</span><span class="p">)</span>
</span><span id="qualify-86"><a href="#qualify-86"><span class="linenos"> 86</span></a>
</span><span id="qualify-87"><a href="#qualify-87"><span class="linenos"> 87</span></a> <span class="k">if</span> <span class="n">qualify_columns</span><span class="p">:</span>
</span><span id="qualify-88"><a href="#qualify-88"><span class="linenos"> 88</span></a> <span class="n">expression</span> <span class="o">=</span> <span class="n">qualify_columns_func</span><span class="p">(</span>
</span><span id="qualify-89"><a href="#qualify-89"><span class="linenos"> 89</span></a> <span class="n">expression</span><span class="p">,</span>
</span><span id="qualify-90"><a href="#qualify-90"><span class="linenos"> 90</span></a> <span class="n">schema</span><span class="p">,</span>
</span><span id="qualify-91"><a href="#qualify-91"><span class="linenos"> 91</span></a> <span class="n">expand_alias_refs</span><span class="o">=</span><span class="n">expand_alias_refs</span><span class="p">,</span>
</span><span id="qualify-92"><a href="#qualify-92"><span class="linenos"> 92</span></a> <span class="n">expand_stars</span><span class="o">=</span><span class="n">expand_stars</span><span class="p">,</span>
</span><span id="qualify-93"><a href="#qualify-93"><span class="linenos"> 93</span></a> <span class="n">infer_schema</span><span class="o">=</span><span class="n">infer_schema</span><span class="p">,</span>
</span><span id="qualify-94"><a href="#qualify-94"><span class="linenos"> 94</span></a> <span class="p">)</span>
</span><span id="qualify-95"><a href="#qualify-95"><span class="linenos"> 95</span></a>
</span><span id="qualify-96"><a href="#qualify-96"><span class="linenos"> 96</span></a> <span class="k">if</span> <span class="n">quote_identifiers</span><span class="p">:</span>
</span><span id="qualify-97"><a href="#qualify-97"><span class="linenos"> 97</span></a> <span class="n">expression</span> <span class="o">=</span> <span class="n">quote_identifiers_func</span><span class="p">(</span><span class="n">expression</span><span class="p">,</span> <span class="n">dialect</span><span class="o">=</span><span class="n">dialect</span><span class="p">,</span> <span class="n">identify</span><span class="o">=</span><span class="n">identify</span><span class="p">)</span>
</span><span id="qualify-31"><a href="#qualify-31"><span class="linenos"> 31</span></a> <span class="n">allow_partial_qualification</span><span class="p">:</span> <span class="nb">bool</span> <span class="o">=</span> <span class="kc">False</span><span class="p">,</span>
</span><span id="qualify-32"><a href="#qualify-32"><span class="linenos"> 32</span></a> <span class="n">validate_qualify_columns</span><span class="p">:</span> <span class="nb">bool</span> <span class="o">=</span> <span class="kc">True</span><span class="p">,</span>
</span><span id="qualify-33"><a href="#qualify-33"><span class="linenos"> 33</span></a> <span class="n">quote_identifiers</span><span class="p">:</span> <span class="nb">bool</span> <span class="o">=</span> <span class="kc">True</span><span class="p">,</span>
</span><span id="qualify-34"><a href="#qualify-34"><span class="linenos"> 34</span></a> <span class="n">identify</span><span class="p">:</span> <span class="nb">bool</span> <span class="o">=</span> <span class="kc">True</span><span class="p">,</span>
</span><span id="qualify-35"><a href="#qualify-35"><span class="linenos"> 35</span></a> <span class="n">infer_csv_schemas</span><span class="p">:</span> <span class="nb">bool</span> <span class="o">=</span> <span class="kc">False</span><span class="p">,</span>
</span><span id="qualify-36"><a href="#qualify-36"><span class="linenos"> 36</span></a><span class="p">)</span> <span class="o">-&gt;</span> <span class="n">exp</span><span class="o">.</span><span class="n">Expression</span><span class="p">:</span>
</span><span id="qualify-37"><a href="#qualify-37"><span class="linenos"> 37</span></a><span class="w"> </span><span class="sd">&quot;&quot;&quot;</span>
</span><span id="qualify-38"><a href="#qualify-38"><span class="linenos"> 38</span></a><span class="sd"> Rewrite sqlglot AST to have normalized and qualified tables and columns.</span>
</span><span id="qualify-39"><a href="#qualify-39"><span class="linenos"> 39</span></a>
</span><span id="qualify-40"><a href="#qualify-40"><span class="linenos"> 40</span></a><span class="sd"> This step is necessary for all further SQLGlot optimizations.</span>
</span><span id="qualify-41"><a href="#qualify-41"><span class="linenos"> 41</span></a>
</span><span id="qualify-42"><a href="#qualify-42"><span class="linenos"> 42</span></a><span class="sd"> Example:</span>
</span><span id="qualify-43"><a href="#qualify-43"><span class="linenos"> 43</span></a><span class="sd"> &gt;&gt;&gt; import sqlglot</span>
</span><span id="qualify-44"><a href="#qualify-44"><span class="linenos"> 44</span></a><span class="sd"> &gt;&gt;&gt; schema = {&quot;tbl&quot;: {&quot;col&quot;: &quot;INT&quot;}}</span>
</span><span id="qualify-45"><a href="#qualify-45"><span class="linenos"> 45</span></a><span class="sd"> &gt;&gt;&gt; expression = sqlglot.parse_one(&quot;SELECT col FROM tbl&quot;)</span>
</span><span id="qualify-46"><a href="#qualify-46"><span class="linenos"> 46</span></a><span class="sd"> &gt;&gt;&gt; qualify(expression, schema=schema).sql()</span>
</span><span id="qualify-47"><a href="#qualify-47"><span class="linenos"> 47</span></a><span class="sd"> &#39;SELECT &quot;tbl&quot;.&quot;col&quot; AS &quot;col&quot; FROM &quot;tbl&quot; AS &quot;tbl&quot;&#39;</span>
</span><span id="qualify-48"><a href="#qualify-48"><span class="linenos"> 48</span></a>
</span><span id="qualify-49"><a href="#qualify-49"><span class="linenos"> 49</span></a><span class="sd"> Args:</span>
</span><span id="qualify-50"><a href="#qualify-50"><span class="linenos"> 50</span></a><span class="sd"> expression: Expression to qualify.</span>
</span><span id="qualify-51"><a href="#qualify-51"><span class="linenos"> 51</span></a><span class="sd"> db: Default database name for tables.</span>
</span><span id="qualify-52"><a href="#qualify-52"><span class="linenos"> 52</span></a><span class="sd"> catalog: Default catalog name for tables.</span>
</span><span id="qualify-53"><a href="#qualify-53"><span class="linenos"> 53</span></a><span class="sd"> schema: Schema to infer column names and types.</span>
</span><span id="qualify-54"><a href="#qualify-54"><span class="linenos"> 54</span></a><span class="sd"> expand_alias_refs: Whether to expand references to aliases.</span>
</span><span id="qualify-55"><a href="#qualify-55"><span class="linenos"> 55</span></a><span class="sd"> expand_stars: Whether to expand star queries. This is a necessary step</span>
</span><span id="qualify-56"><a href="#qualify-56"><span class="linenos"> 56</span></a><span class="sd"> for most of the optimizer&#39;s rules to work; do not set to False unless you</span>
</span><span id="qualify-57"><a href="#qualify-57"><span class="linenos"> 57</span></a><span class="sd"> know what you&#39;re doing!</span>
</span><span id="qualify-58"><a href="#qualify-58"><span class="linenos"> 58</span></a><span class="sd"> infer_schema: Whether to infer the schema if missing.</span>
</span><span id="qualify-59"><a href="#qualify-59"><span class="linenos"> 59</span></a><span class="sd"> isolate_tables: Whether to isolate table selects.</span>
</span><span id="qualify-60"><a href="#qualify-60"><span class="linenos"> 60</span></a><span class="sd"> qualify_columns: Whether to qualify columns.</span>
</span><span id="qualify-61"><a href="#qualify-61"><span class="linenos"> 61</span></a><span class="sd"> allow_partial_qualification: Whether to allow partial qualification.</span>
</span><span id="qualify-62"><a href="#qualify-62"><span class="linenos"> 62</span></a><span class="sd"> validate_qualify_columns: Whether to validate columns.</span>
</span><span id="qualify-63"><a href="#qualify-63"><span class="linenos"> 63</span></a><span class="sd"> quote_identifiers: Whether to run the quote_identifiers step.</span>
</span><span id="qualify-64"><a href="#qualify-64"><span class="linenos"> 64</span></a><span class="sd"> This step is necessary to ensure correctness for case sensitive queries.</span>
</span><span id="qualify-65"><a href="#qualify-65"><span class="linenos"> 65</span></a><span class="sd"> But this flag is provided in case this step is performed at a later time.</span>
</span><span id="qualify-66"><a href="#qualify-66"><span class="linenos"> 66</span></a><span class="sd"> identify: If True, quote all identifiers, else only necessary ones.</span>
</span><span id="qualify-67"><a href="#qualify-67"><span class="linenos"> 67</span></a><span class="sd"> infer_csv_schemas: Whether to scan READ_CSV calls in order to infer the CSVs&#39; schemas.</span>
</span><span id="qualify-68"><a href="#qualify-68"><span class="linenos"> 68</span></a>
</span><span id="qualify-69"><a href="#qualify-69"><span class="linenos"> 69</span></a><span class="sd"> Returns:</span>
</span><span id="qualify-70"><a href="#qualify-70"><span class="linenos"> 70</span></a><span class="sd"> The qualified expression.</span>
</span><span id="qualify-71"><a href="#qualify-71"><span class="linenos"> 71</span></a><span class="sd"> &quot;&quot;&quot;</span>
</span><span id="qualify-72"><a href="#qualify-72"><span class="linenos"> 72</span></a> <span class="n">schema</span> <span class="o">=</span> <span class="n">ensure_schema</span><span class="p">(</span><span class="n">schema</span><span class="p">,</span> <span class="n">dialect</span><span class="o">=</span><span class="n">dialect</span><span class="p">)</span>
</span><span id="qualify-73"><a href="#qualify-73"><span class="linenos"> 73</span></a> <span class="n">expression</span> <span class="o">=</span> <span class="n">normalize_identifiers</span><span class="p">(</span><span class="n">expression</span><span class="p">,</span> <span class="n">dialect</span><span class="o">=</span><span class="n">dialect</span><span class="p">)</span>
</span><span id="qualify-74"><a href="#qualify-74"><span class="linenos"> 74</span></a> <span class="n">expression</span> <span class="o">=</span> <span class="n">qualify_tables</span><span class="p">(</span>
</span><span id="qualify-75"><a href="#qualify-75"><span class="linenos"> 75</span></a> <span class="n">expression</span><span class="p">,</span>
</span><span id="qualify-76"><a href="#qualify-76"><span class="linenos"> 76</span></a> <span class="n">db</span><span class="o">=</span><span class="n">db</span><span class="p">,</span>
</span><span id="qualify-77"><a href="#qualify-77"><span class="linenos"> 77</span></a> <span class="n">catalog</span><span class="o">=</span><span class="n">catalog</span><span class="p">,</span>
</span><span id="qualify-78"><a href="#qualify-78"><span class="linenos"> 78</span></a> <span class="n">schema</span><span class="o">=</span><span class="n">schema</span><span class="p">,</span>
</span><span id="qualify-79"><a href="#qualify-79"><span class="linenos"> 79</span></a> <span class="n">dialect</span><span class="o">=</span><span class="n">dialect</span><span class="p">,</span>
</span><span id="qualify-80"><a href="#qualify-80"><span class="linenos"> 80</span></a> <span class="n">infer_csv_schemas</span><span class="o">=</span><span class="n">infer_csv_schemas</span><span class="p">,</span>
</span><span id="qualify-81"><a href="#qualify-81"><span class="linenos"> 81</span></a> <span class="p">)</span>
</span><span id="qualify-82"><a href="#qualify-82"><span class="linenos"> 82</span></a>
</span><span id="qualify-83"><a href="#qualify-83"><span class="linenos"> 83</span></a> <span class="k">if</span> <span class="n">isolate_tables</span><span class="p">:</span>
</span><span id="qualify-84"><a href="#qualify-84"><span class="linenos"> 84</span></a> <span class="n">expression</span> <span class="o">=</span> <span class="n">isolate_table_selects</span><span class="p">(</span><span class="n">expression</span><span class="p">,</span> <span class="n">schema</span><span class="o">=</span><span class="n">schema</span><span class="p">)</span>
</span><span id="qualify-85"><a href="#qualify-85"><span class="linenos"> 85</span></a>
</span><span id="qualify-86"><a href="#qualify-86"><span class="linenos"> 86</span></a> <span class="k">if</span> <span class="n">Dialect</span><span class="o">.</span><span class="n">get_or_raise</span><span class="p">(</span><span class="n">dialect</span><span class="p">)</span><span class="o">.</span><span class="n">PREFER_CTE_ALIAS_COLUMN</span><span class="p">:</span>
</span><span id="qualify-87"><a href="#qualify-87"><span class="linenos"> 87</span></a> <span class="n">expression</span> <span class="o">=</span> <span class="n">pushdown_cte_alias_columns_func</span><span class="p">(</span><span class="n">expression</span><span class="p">)</span>
</span><span id="qualify-88"><a href="#qualify-88"><span class="linenos"> 88</span></a>
</span><span id="qualify-89"><a href="#qualify-89"><span class="linenos"> 89</span></a> <span class="k">if</span> <span class="n">qualify_columns</span><span class="p">:</span>
</span><span id="qualify-90"><a href="#qualify-90"><span class="linenos"> 90</span></a> <span class="n">expression</span> <span class="o">=</span> <span class="n">qualify_columns_func</span><span class="p">(</span>
</span><span id="qualify-91"><a href="#qualify-91"><span class="linenos"> 91</span></a> <span class="n">expression</span><span class="p">,</span>
</span><span id="qualify-92"><a href="#qualify-92"><span class="linenos"> 92</span></a> <span class="n">schema</span><span class="p">,</span>
</span><span id="qualify-93"><a href="#qualify-93"><span class="linenos"> 93</span></a> <span class="n">expand_alias_refs</span><span class="o">=</span><span class="n">expand_alias_refs</span><span class="p">,</span>
</span><span id="qualify-94"><a href="#qualify-94"><span class="linenos"> 94</span></a> <span class="n">expand_stars</span><span class="o">=</span><span class="n">expand_stars</span><span class="p">,</span>
</span><span id="qualify-95"><a href="#qualify-95"><span class="linenos"> 95</span></a> <span class="n">infer_schema</span><span class="o">=</span><span class="n">infer_schema</span><span class="p">,</span>
</span><span id="qualify-96"><a href="#qualify-96"><span class="linenos"> 96</span></a> <span class="n">allow_partial_qualification</span><span class="o">=</span><span class="n">allow_partial_qualification</span><span class="p">,</span>
</span><span id="qualify-97"><a href="#qualify-97"><span class="linenos"> 97</span></a> <span class="p">)</span>
</span><span id="qualify-98"><a href="#qualify-98"><span class="linenos"> 98</span></a>
</span><span id="qualify-99"><a href="#qualify-99"><span class="linenos"> 99</span></a> <span class="k">if</span> <span class="n">validate_qualify_columns</span><span class="p">:</span>
</span><span id="qualify-100"><a href="#qualify-100"><span class="linenos">100</span></a> <span class="n">validate_qualify_columns_func</span><span class="p">(</span><span class="n">expression</span><span class="p">)</span>
</span><span id="qualify-99"><a href="#qualify-99"><span class="linenos"> 99</span></a> <span class="k">if</span> <span class="n">quote_identifiers</span><span class="p">:</span>
</span><span id="qualify-100"><a href="#qualify-100"><span class="linenos">100</span></a> <span class="n">expression</span> <span class="o">=</span> <span class="n">quote_identifiers_func</span><span class="p">(</span><span class="n">expression</span><span class="p">,</span> <span class="n">dialect</span><span class="o">=</span><span class="n">dialect</span><span class="p">,</span> <span class="n">identify</span><span class="o">=</span><span class="n">identify</span><span class="p">)</span>
</span><span id="qualify-101"><a href="#qualify-101"><span class="linenos">101</span></a>
</span><span id="qualify-102"><a href="#qualify-102"><span class="linenos">102</span></a> <span class="k">return</span> <span class="n">expression</span>
</span><span id="qualify-102"><a href="#qualify-102"><span class="linenos">102</span></a> <span class="k">if</span> <span class="n">validate_qualify_columns</span><span class="p">:</span>
</span><span id="qualify-103"><a href="#qualify-103"><span class="linenos">103</span></a> <span class="n">validate_qualify_columns_func</span><span class="p">(</span><span class="n">expression</span><span class="p">)</span>
</span><span id="qualify-104"><a href="#qualify-104"><span class="linenos">104</span></a>
</span><span id="qualify-105"><a href="#qualify-105"><span class="linenos">105</span></a> <span class="k">return</span> <span class="n">expression</span>
</span></pre></div>
@ -289,6 +295,7 @@ know what you're doing!</li>
<li><strong>infer_schema:</strong> Whether to infer the schema if missing.</li>
<li><strong>isolate_tables:</strong> Whether to isolate table selects.</li>
<li><strong>qualify_columns:</strong> Whether to qualify columns.</li>
<li><strong>allow_partial_qualification:</strong> Whether to allow partial qualification.</li>
<li><strong>validate_qualify_columns:</strong> Whether to validate columns.</li>
<li><strong>quote_identifiers:</strong> Whether to run the quote_identifiers step.
This step is necessary to ensure correctness for case sensitive queries.

2513
docs/sqlglot/optimizer/qualify_columns.html
View file

File diff suppressed because it is too large Load diff

500
docs/sqlglot/optimizer/scope.html
View file

@ -926,188 +926,190 @@
</span><span id="L-679"><a href="#L-679"><span class="linenos">679</span></a> <span class="n">expressions</span><span class="o">.</span><span class="n">extend</span><span class="p">(</span><span class="n">scope</span><span class="o">.</span><span class="n">expression</span><span class="o">.</span><span class="n">args</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s2">&quot;laterals&quot;</span><span class="p">)</span> <span class="ow">or</span> <span class="p">[])</span>
</span><span id="L-680"><a href="#L-680"><span class="linenos">680</span></a>
</span><span id="L-681"><a href="#L-681"><span class="linenos">681</span></a> <span class="k">for</span> <span class="n">expression</span> <span class="ow">in</span> <span class="n">expressions</span><span class="p">:</span>
</span><span id="L-682"><a href="#L-682"><span class="linenos">682</span></a> <span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">expression</span><span class="p">,</span> <span class="n">exp</span><span class="o">.</span><span class="n">Table</span><span class="p">):</span>
</span><span id="L-683"><a href="#L-683"><span class="linenos">683</span></a> <span class="n">table_name</span> <span class="o">=</span> <span class="n">expression</span><span class="o">.</span><span class="n">name</span>
</span><span id="L-684"><a href="#L-684"><span class="linenos">684</span></a> <span class="n">source_name</span> <span class="o">=</span> <span class="n">expression</span><span class="o">.</span><span class="n">alias_or_name</span>
</span><span id="L-685"><a href="#L-685"><span class="linenos">685</span></a>
</span><span id="L-686"><a href="#L-686"><span class="linenos">686</span></a> <span class="k">if</span> <span class="n">table_name</span> <span class="ow">in</span> <span class="n">scope</span><span class="o">.</span><span class="n">sources</span> <span class="ow">and</span> <span class="ow">not</span> <span class="n">expression</span><span class="o">.</span><span class="n">db</span><span class="p">:</span>
</span><span id="L-687"><a href="#L-687"><span class="linenos">687</span></a> <span class="c1"># This is a reference to a parent source (e.g. a CTE), not an actual table, unless</span>
</span><span id="L-688"><a href="#L-688"><span class="linenos">688</span></a> <span class="c1"># it is pivoted, because then we get back a new table and hence a new source.</span>
</span><span id="L-689"><a href="#L-689"><span class="linenos">689</span></a> <span class="n">pivots</span> <span class="o">=</span> <span class="n">expression</span><span class="o">.</span><span class="n">args</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s2">&quot;pivots&quot;</span><span class="p">)</span>
</span><span id="L-690"><a href="#L-690"><span class="linenos">690</span></a> <span class="k">if</span> <span class="n">pivots</span><span class="p">:</span>
</span><span id="L-691"><a href="#L-691"><span class="linenos">691</span></a> <span class="n">sources</span><span class="p">[</span><span class="n">pivots</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span><span class="o">.</span><span class="n">alias</span><span class="p">]</span> <span class="o">=</span> <span class="n">expression</span>
</span><span id="L-692"><a href="#L-692"><span class="linenos">692</span></a> <span class="k">else</span><span class="p">:</span>
</span><span id="L-693"><a href="#L-693"><span class="linenos">693</span></a> <span class="n">sources</span><span class="p">[</span><span class="n">source_name</span><span class="p">]</span> <span class="o">=</span> <span class="n">scope</span><span class="o">.</span><span class="n">sources</span><span class="p">[</span><span class="n">table_name</span><span class="p">]</span>
</span><span id="L-694"><a href="#L-694"><span class="linenos">694</span></a> <span class="k">elif</span> <span class="n">source_name</span> <span class="ow">in</span> <span class="n">sources</span><span class="p">:</span>
</span><span id="L-695"><a href="#L-695"><span class="linenos">695</span></a> <span class="n">sources</span><span class="p">[</span><span class="n">find_new_name</span><span class="p">(</span><span class="n">sources</span><span class="p">,</span> <span class="n">table_name</span><span class="p">)]</span> <span class="o">=</span> <span class="n">expression</span>
</span><span id="L-696"><a href="#L-696"><span class="linenos">696</span></a> <span class="k">else</span><span class="p">:</span>
</span><span id="L-697"><a href="#L-697"><span class="linenos">697</span></a> <span class="n">sources</span><span class="p">[</span><span class="n">source_name</span><span class="p">]</span> <span class="o">=</span> <span class="n">expression</span>
</span><span id="L-698"><a href="#L-698"><span class="linenos">698</span></a>
</span><span id="L-699"><a href="#L-699"><span class="linenos">699</span></a> <span class="c1"># Make sure to not include the joins twice</span>
</span><span id="L-700"><a href="#L-700"><span class="linenos">700</span></a> <span class="k">if</span> <span class="n">expression</span> <span class="ow">is</span> <span class="ow">not</span> <span class="n">scope</span><span class="o">.</span><span class="n">expression</span><span class="p">:</span>
</span><span id="L-701"><a href="#L-701"><span class="linenos">701</span></a> <span class="n">expressions</span><span class="o">.</span><span class="n">extend</span><span class="p">(</span><span class="n">join</span><span class="o">.</span><span class="n">this</span> <span class="k">for</span> <span class="n">join</span> <span class="ow">in</span> <span class="n">expression</span><span class="o">.</span><span class="n">args</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s2">&quot;joins&quot;</span><span class="p">)</span> <span class="ow">or</span> <span class="p">[])</span>
</span><span id="L-702"><a href="#L-702"><span class="linenos">702</span></a>
</span><span id="L-703"><a href="#L-703"><span class="linenos">703</span></a> <span class="k">continue</span>
</span><span id="L-682"><a href="#L-682"><span class="linenos">682</span></a> <span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">expression</span><span class="p">,</span> <span class="n">exp</span><span class="o">.</span><span class="n">Final</span><span class="p">):</span>
</span><span id="L-683"><a href="#L-683"><span class="linenos">683</span></a> <span class="n">expression</span> <span class="o">=</span> <span class="n">expression</span><span class="o">.</span><span class="n">this</span>
</span><span id="L-684"><a href="#L-684"><span class="linenos">684</span></a> <span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">expression</span><span class="p">,</span> <span class="n">exp</span><span class="o">.</span><span class="n">Table</span><span class="p">):</span>
</span><span id="L-685"><a href="#L-685"><span class="linenos">685</span></a> <span class="n">table_name</span> <span class="o">=</span> <span class="n">expression</span><span class="o">.</span><span class="n">name</span>
</span><span id="L-686"><a href="#L-686"><span class="linenos">686</span></a> <span class="n">source_name</span> <span class="o">=</span> <span class="n">expression</span><span class="o">.</span><span class="n">alias_or_name</span>
</span><span id="L-687"><a href="#L-687"><span class="linenos">687</span></a>
</span><span id="L-688"><a href="#L-688"><span class="linenos">688</span></a> <span class="k">if</span> <span class="n">table_name</span> <span class="ow">in</span> <span class="n">scope</span><span class="o">.</span><span class="n">sources</span> <span class="ow">and</span> <span class="ow">not</span> <span class="n">expression</span><span class="o">.</span><span class="n">db</span><span class="p">:</span>
</span><span id="L-689"><a href="#L-689"><span class="linenos">689</span></a> <span class="c1"># This is a reference to a parent source (e.g. a CTE), not an actual table, unless</span>
</span><span id="L-690"><a href="#L-690"><span class="linenos">690</span></a> <span class="c1"># it is pivoted, because then we get back a new table and hence a new source.</span>
</span><span id="L-691"><a href="#L-691"><span class="linenos">691</span></a> <span class="n">pivots</span> <span class="o">=</span> <span class="n">expression</span><span class="o">.</span><span class="n">args</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s2">&quot;pivots&quot;</span><span class="p">)</span>
</span><span id="L-692"><a href="#L-692"><span class="linenos">692</span></a> <span class="k">if</span> <span class="n">pivots</span><span class="p">:</span>
</span><span id="L-693"><a href="#L-693"><span class="linenos">693</span></a> <span class="n">sources</span><span class="p">[</span><span class="n">pivots</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span><span class="o">.</span><span class="n">alias</span><span class="p">]</span> <span class="o">=</span> <span class="n">expression</span>
</span><span id="L-694"><a href="#L-694"><span class="linenos">694</span></a> <span class="k">else</span><span class="p">:</span>
</span><span id="L-695"><a href="#L-695"><span class="linenos">695</span></a> <span class="n">sources</span><span class="p">[</span><span class="n">source_name</span><span class="p">]</span> <span class="o">=</span> <span class="n">scope</span><span class="o">.</span><span class="n">sources</span><span class="p">[</span><span class="n">table_name</span><span class="p">]</span>
</span><span id="L-696"><a href="#L-696"><span class="linenos">696</span></a> <span class="k">elif</span> <span class="n">source_name</span> <span class="ow">in</span> <span class="n">sources</span><span class="p">:</span>
</span><span id="L-697"><a href="#L-697"><span class="linenos">697</span></a> <span class="n">sources</span><span class="p">[</span><span class="n">find_new_name</span><span class="p">(</span><span class="n">sources</span><span class="p">,</span> <span class="n">table_name</span><span class="p">)]</span> <span class="o">=</span> <span class="n">expression</span>
</span><span id="L-698"><a href="#L-698"><span class="linenos">698</span></a> <span class="k">else</span><span class="p">:</span>
</span><span id="L-699"><a href="#L-699"><span class="linenos">699</span></a> <span class="n">sources</span><span class="p">[</span><span class="n">source_name</span><span class="p">]</span> <span class="o">=</span> <span class="n">expression</span>
</span><span id="L-700"><a href="#L-700"><span class="linenos">700</span></a>
</span><span id="L-701"><a href="#L-701"><span class="linenos">701</span></a> <span class="c1"># Make sure to not include the joins twice</span>
</span><span id="L-702"><a href="#L-702"><span class="linenos">702</span></a> <span class="k">if</span> <span class="n">expression</span> <span class="ow">is</span> <span class="ow">not</span> <span class="n">scope</span><span class="o">.</span><span class="n">expression</span><span class="p">:</span>
</span><span id="L-703"><a href="#L-703"><span class="linenos">703</span></a> <span class="n">expressions</span><span class="o">.</span><span class="n">extend</span><span class="p">(</span><span class="n">join</span><span class="o">.</span><span class="n">this</span> <span class="k">for</span> <span class="n">join</span> <span class="ow">in</span> <span class="n">expression</span><span class="o">.</span><span class="n">args</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s2">&quot;joins&quot;</span><span class="p">)</span> <span class="ow">or</span> <span class="p">[])</span>
</span><span id="L-704"><a href="#L-704"><span class="linenos">704</span></a>
</span><span id="L-705"><a href="#L-705"><span class="linenos">705</span></a> <span class="k">if</span> <span class="ow">not</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">expression</span><span class="p">,</span> <span class="n">exp</span><span class="o">.</span><span class="n">DerivedTable</span><span class="p">):</span>
</span><span id="L-706"><a href="#L-706"><span class="linenos">706</span></a> <span class="k">continue</span>
</span><span id="L-707"><a href="#L-707"><span class="linenos">707</span></a>
</span><span id="L-708"><a href="#L-708"><span class="linenos">708</span></a> <span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">expression</span><span class="p">,</span> <span class="n">exp</span><span class="o">.</span><span class="n">UDTF</span><span class="p">):</span>
</span><span id="L-709"><a href="#L-709"><span class="linenos">709</span></a> <span class="n">lateral_sources</span> <span class="o">=</span> <span class="n">sources</span>
</span><span id="L-710"><a href="#L-710"><span class="linenos">710</span></a> <span class="n">scope_type</span> <span class="o">=</span> <span class="n">ScopeType</span><span class="o">.</span><span class="n">UDTF</span>
</span><span id="L-711"><a href="#L-711"><span class="linenos">711</span></a> <span class="n">scopes</span> <span class="o">=</span> <span class="n">scope</span><span class="o">.</span><span class="n">udtf_scopes</span>
</span><span id="L-712"><a href="#L-712"><span class="linenos">712</span></a> <span class="k">elif</span> <span class="n">_is_derived_table</span><span class="p">(</span><span class="n">expression</span><span class="p">):</span>
</span><span id="L-713"><a href="#L-713"><span class="linenos">713</span></a> <span class="n">lateral_sources</span> <span class="o">=</span> <span class="kc">None</span>
</span><span id="L-714"><a href="#L-714"><span class="linenos">714</span></a> <span class="n">scope_type</span> <span class="o">=</span> <span class="n">ScopeType</span><span class="o">.</span><span class="n">DERIVED_TABLE</span>
</span><span id="L-715"><a href="#L-715"><span class="linenos">715</span></a> <span class="n">scopes</span> <span class="o">=</span> <span class="n">scope</span><span class="o">.</span><span class="n">derived_table_scopes</span>
</span><span id="L-716"><a href="#L-716"><span class="linenos">716</span></a> <span class="n">expressions</span><span class="o">.</span><span class="n">extend</span><span class="p">(</span><span class="n">join</span><span class="o">.</span><span class="n">this</span> <span class="k">for</span> <span class="n">join</span> <span class="ow">in</span> <span class="n">expression</span><span class="o">.</span><span class="n">args</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s2">&quot;joins&quot;</span><span class="p">)</span> <span class="ow">or</span> <span class="p">[])</span>
</span><span id="L-717"><a href="#L-717"><span class="linenos">717</span></a> <span class="k">else</span><span class="p">:</span>
</span><span id="L-718"><a href="#L-718"><span class="linenos">718</span></a> <span class="c1"># Makes sure we check for possible sources in nested table constructs</span>
</span><span id="L-719"><a href="#L-719"><span class="linenos">719</span></a> <span class="n">expressions</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">expression</span><span class="o">.</span><span class="n">this</span><span class="p">)</span>
</span><span id="L-720"><a href="#L-720"><span class="linenos">720</span></a> <span class="n">expressions</span><span class="o">.</span><span class="n">extend</span><span class="p">(</span><span class="n">join</span><span class="o">.</span><span class="n">this</span> <span class="k">for</span> <span class="n">join</span> <span class="ow">in</span> <span class="n">expression</span><span class="o">.</span><span class="n">args</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s2">&quot;joins&quot;</span><span class="p">)</span> <span class="ow">or</span> <span class="p">[])</span>
</span><span id="L-721"><a href="#L-721"><span class="linenos">721</span></a> <span class="k">continue</span>
</span><span id="L-722"><a href="#L-722"><span class="linenos">722</span></a>
</span><span id="L-723"><a href="#L-723"><span class="linenos">723</span></a> <span class="k">for</span> <span class="n">child_scope</span> <span class="ow">in</span> <span class="n">_traverse_scope</span><span class="p">(</span>
</span><span id="L-724"><a href="#L-724"><span class="linenos">724</span></a> <span class="n">scope</span><span class="o">.</span><span class="n">branch</span><span class="p">(</span>
</span><span id="L-725"><a href="#L-725"><span class="linenos">725</span></a> <span class="n">expression</span><span class="p">,</span>
</span><span id="L-726"><a href="#L-726"><span class="linenos">726</span></a> <span class="n">lateral_sources</span><span class="o">=</span><span class="n">lateral_sources</span><span class="p">,</span>
</span><span id="L-727"><a href="#L-727"><span class="linenos">727</span></a> <span class="n">outer_columns</span><span class="o">=</span><span class="n">expression</span><span class="o">.</span><span class="n">alias_column_names</span><span class="p">,</span>
</span><span id="L-728"><a href="#L-728"><span class="linenos">728</span></a> <span class="n">scope_type</span><span class="o">=</span><span class="n">scope_type</span><span class="p">,</span>
</span><span id="L-729"><a href="#L-729"><span class="linenos">729</span></a> <span class="p">)</span>
</span><span id="L-730"><a href="#L-730"><span class="linenos">730</span></a> <span class="p">):</span>
</span><span id="L-731"><a href="#L-731"><span class="linenos">731</span></a> <span class="k">yield</span> <span class="n">child_scope</span>
</span><span id="L-732"><a href="#L-732"><span class="linenos">732</span></a>
</span><span id="L-733"><a href="#L-733"><span class="linenos">733</span></a> <span class="c1"># Tables without aliases will be set as &quot;&quot;</span>
</span><span id="L-734"><a href="#L-734"><span class="linenos">734</span></a> <span class="c1"># This shouldn&#39;t be a problem once qualify_columns runs, as it adds aliases on everything.</span>
</span><span id="L-735"><a href="#L-735"><span class="linenos">735</span></a> <span class="c1"># Until then, this means that only a single, unaliased derived table is allowed (rather,</span>
</span><span id="L-736"><a href="#L-736"><span class="linenos">736</span></a> <span class="c1"># the latest one wins.</span>
</span><span id="L-737"><a href="#L-737"><span class="linenos">737</span></a> <span class="n">sources</span><span class="p">[</span><span class="n">expression</span><span class="o">.</span><span class="n">alias</span><span class="p">]</span> <span class="o">=</span> <span class="n">child_scope</span>
</span><span id="L-738"><a href="#L-738"><span class="linenos">738</span></a>
</span><span id="L-739"><a href="#L-739"><span class="linenos">739</span></a> <span class="c1"># append the final child_scope yielded</span>
</span><span id="L-740"><a href="#L-740"><span class="linenos">740</span></a> <span class="n">scopes</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">child_scope</span><span class="p">)</span>
</span><span id="L-741"><a href="#L-741"><span class="linenos">741</span></a> <span class="n">scope</span><span class="o">.</span><span class="n">table_scopes</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">child_scope</span><span class="p">)</span>
</span><span id="L-742"><a href="#L-742"><span class="linenos">742</span></a>
</span><span id="L-743"><a href="#L-743"><span class="linenos">743</span></a> <span class="n">scope</span><span class="o">.</span><span class="n">sources</span><span class="o">.</span><span class="n">update</span><span class="p">(</span><span class="n">sources</span><span class="p">)</span>
</span><span id="L-705"><a href="#L-705"><span class="linenos">705</span></a> <span class="k">continue</span>
</span><span id="L-706"><a href="#L-706"><span class="linenos">706</span></a>
</span><span id="L-707"><a href="#L-707"><span class="linenos">707</span></a> <span class="k">if</span> <span class="ow">not</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">expression</span><span class="p">,</span> <span class="n">exp</span><span class="o">.</span><span class="n">DerivedTable</span><span class="p">):</span>
</span><span id="L-708"><a href="#L-708"><span class="linenos">708</span></a> <span class="k">continue</span>
</span><span id="L-709"><a href="#L-709"><span class="linenos">709</span></a>
</span><span id="L-710"><a href="#L-710"><span class="linenos">710</span></a> <span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">expression</span><span class="p">,</span> <span class="n">exp</span><span class="o">.</span><span class="n">UDTF</span><span class="p">):</span>
</span><span id="L-711"><a href="#L-711"><span class="linenos">711</span></a> <span class="n">lateral_sources</span> <span class="o">=</span> <span class="n">sources</span>
</span><span id="L-712"><a href="#L-712"><span class="linenos">712</span></a> <span class="n">scope_type</span> <span class="o">=</span> <span class="n">ScopeType</span><span class="o">.</span><span class="n">UDTF</span>
</span><span id="L-713"><a href="#L-713"><span class="linenos">713</span></a> <span class="n">scopes</span> <span class="o">=</span> <span class="n">scope</span><span class="o">.</span><span class="n">udtf_scopes</span>
</span><span id="L-714"><a href="#L-714"><span class="linenos">714</span></a> <span class="k">elif</span> <span class="n">_is_derived_table</span><span class="p">(</span><span class="n">expression</span><span class="p">):</span>
</span><span id="L-715"><a href="#L-715"><span class="linenos">715</span></a> <span class="n">lateral_sources</span> <span class="o">=</span> <span class="kc">None</span>
</span><span id="L-716"><a href="#L-716"><span class="linenos">716</span></a> <span class="n">scope_type</span> <span class="o">=</span> <span class="n">ScopeType</span><span class="o">.</span><span class="n">DERIVED_TABLE</span>
</span><span id="L-717"><a href="#L-717"><span class="linenos">717</span></a> <span class="n">scopes</span> <span class="o">=</span> <span class="n">scope</span><span class="o">.</span><span class="n">derived_table_scopes</span>
</span><span id="L-718"><a href="#L-718"><span class="linenos">718</span></a> <span class="n">expressions</span><span class="o">.</span><span class="n">extend</span><span class="p">(</span><span class="n">join</span><span class="o">.</span><span class="n">this</span> <span class="k">for</span> <span class="n">join</span> <span class="ow">in</span> <span class="n">expression</span><span class="o">.</span><span class="n">args</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s2">&quot;joins&quot;</span><span class="p">)</span> <span class="ow">or</span> <span class="p">[])</span>
</span><span id="L-719"><a href="#L-719"><span class="linenos">719</span></a> <span class="k">else</span><span class="p">:</span>
</span><span id="L-720"><a href="#L-720"><span class="linenos">720</span></a> <span class="c1"># Makes sure we check for possible sources in nested table constructs</span>
</span><span id="L-721"><a href="#L-721"><span class="linenos">721</span></a> <span class="n">expressions</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">expression</span><span class="o">.</span><span class="n">this</span><span class="p">)</span>
</span><span id="L-722"><a href="#L-722"><span class="linenos">722</span></a> <span class="n">expressions</span><span class="o">.</span><span class="n">extend</span><span class="p">(</span><span class="n">join</span><span class="o">.</span><span class="n">this</span> <span class="k">for</span> <span class="n">join</span> <span class="ow">in</span> <span class="n">expression</span><span class="o">.</span><span class="n">args</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s2">&quot;joins&quot;</span><span class="p">)</span> <span class="ow">or</span> <span class="p">[])</span>
</span><span id="L-723"><a href="#L-723"><span class="linenos">723</span></a> <span class="k">continue</span>
</span><span id="L-724"><a href="#L-724"><span class="linenos">724</span></a>
</span><span id="L-725"><a href="#L-725"><span class="linenos">725</span></a> <span class="k">for</span> <span class="n">child_scope</span> <span class="ow">in</span> <span class="n">_traverse_scope</span><span class="p">(</span>
</span><span id="L-726"><a href="#L-726"><span class="linenos">726</span></a> <span class="n">scope</span><span class="o">.</span><span class="n">branch</span><span class="p">(</span>
</span><span id="L-727"><a href="#L-727"><span class="linenos">727</span></a> <span class="n">expression</span><span class="p">,</span>
</span><span id="L-728"><a href="#L-728"><span class="linenos">728</span></a> <span class="n">lateral_sources</span><span class="o">=</span><span class="n">lateral_sources</span><span class="p">,</span>
</span><span id="L-729"><a href="#L-729"><span class="linenos">729</span></a> <span class="n">outer_columns</span><span class="o">=</span><span class="n">expression</span><span class="o">.</span><span class="n">alias_column_names</span><span class="p">,</span>
</span><span id="L-730"><a href="#L-730"><span class="linenos">730</span></a> <span class="n">scope_type</span><span class="o">=</span><span class="n">scope_type</span><span class="p">,</span>
</span><span id="L-731"><a href="#L-731"><span class="linenos">731</span></a> <span class="p">)</span>
</span><span id="L-732"><a href="#L-732"><span class="linenos">732</span></a> <span class="p">):</span>
</span><span id="L-733"><a href="#L-733"><span class="linenos">733</span></a> <span class="k">yield</span> <span class="n">child_scope</span>
</span><span id="L-734"><a href="#L-734"><span class="linenos">734</span></a>
</span><span id="L-735"><a href="#L-735"><span class="linenos">735</span></a> <span class="c1"># Tables without aliases will be set as &quot;&quot;</span>
</span><span id="L-736"><a href="#L-736"><span class="linenos">736</span></a> <span class="c1"># This shouldn&#39;t be a problem once qualify_columns runs, as it adds aliases on everything.</span>
</span><span id="L-737"><a href="#L-737"><span class="linenos">737</span></a> <span class="c1"># Until then, this means that only a single, unaliased derived table is allowed (rather,</span>
</span><span id="L-738"><a href="#L-738"><span class="linenos">738</span></a> <span class="c1"># the latest one wins.</span>
</span><span id="L-739"><a href="#L-739"><span class="linenos">739</span></a> <span class="n">sources</span><span class="p">[</span><span class="n">expression</span><span class="o">.</span><span class="n">alias</span><span class="p">]</span> <span class="o">=</span> <span class="n">child_scope</span>
</span><span id="L-740"><a href="#L-740"><span class="linenos">740</span></a>
</span><span id="L-741"><a href="#L-741"><span class="linenos">741</span></a> <span class="c1"># append the final child_scope yielded</span>
</span><span id="L-742"><a href="#L-742"><span class="linenos">742</span></a> <span class="n">scopes</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">child_scope</span><span class="p">)</span>
</span><span id="L-743"><a href="#L-743"><span class="linenos">743</span></a> <span class="n">scope</span><span class="o">.</span><span class="n">table_scopes</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">child_scope</span><span class="p">)</span>
</span><span id="L-744"><a href="#L-744"><span class="linenos">744</span></a>
</span><span id="L-745"><a href="#L-745"><span class="linenos">745</span></a>
</span><span id="L-746"><a href="#L-746"><span class="linenos">746</span></a><span class="k">def</span> <span class="nf">_traverse_subqueries</span><span class="p">(</span><span class="n">scope</span><span class="p">):</span>
</span><span id="L-747"><a href="#L-747"><span class="linenos">747</span></a> <span class="k">for</span> <span class="n">subquery</span> <span class="ow">in</span> <span class="n">scope</span><span class="o">.</span><span class="n">subqueries</span><span class="p">:</span>
</span><span id="L-748"><a href="#L-748"><span class="linenos">748</span></a> <span class="n">top</span> <span class="o">=</span> <span class="kc">None</span>
</span><span id="L-749"><a href="#L-749"><span class="linenos">749</span></a> <span class="k">for</span> <span class="n">child_scope</span> <span class="ow">in</span> <span class="n">_traverse_scope</span><span class="p">(</span><span class="n">scope</span><span class="o">.</span><span class="n">branch</span><span class="p">(</span><span class="n">subquery</span><span class="p">,</span> <span class="n">scope_type</span><span class="o">=</span><span class="n">ScopeType</span><span class="o">.</span><span class="n">SUBQUERY</span><span class="p">)):</span>
</span><span id="L-750"><a href="#L-750"><span class="linenos">750</span></a> <span class="k">yield</span> <span class="n">child_scope</span>
</span><span id="L-751"><a href="#L-751"><span class="linenos">751</span></a> <span class="n">top</span> <span class="o">=</span> <span class="n">child_scope</span>
</span><span id="L-752"><a href="#L-752"><span class="linenos">752</span></a> <span class="n">scope</span><span class="o">.</span><span class="n">subquery_scopes</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">top</span><span class="p">)</span>
</span><span id="L-753"><a href="#L-753"><span class="linenos">753</span></a>
</span><span id="L-754"><a href="#L-754"><span class="linenos">754</span></a>
</span><span id="L-755"><a href="#L-755"><span class="linenos">755</span></a><span class="k">def</span> <span class="nf">_traverse_udtfs</span><span class="p">(</span><span class="n">scope</span><span class="p">):</span>
</span><span id="L-756"><a href="#L-756"><span class="linenos">756</span></a> <span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">scope</span><span class="o">.</span><span class="n">expression</span><span class="p">,</span> <span class="n">exp</span><span class="o">.</span><span class="n">Unnest</span><span class="p">):</span>
</span><span id="L-757"><a href="#L-757"><span class="linenos">757</span></a> <span class="n">expressions</span> <span class="o">=</span> <span class="n">scope</span><span class="o">.</span><span class="n">expression</span><span class="o">.</span><span class="n">expressions</span>
</span><span id="L-758"><a href="#L-758"><span class="linenos">758</span></a> <span class="k">elif</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">scope</span><span class="o">.</span><span class="n">expression</span><span class="p">,</span> <span class="n">exp</span><span class="o">.</span><span class="n">Lateral</span><span class="p">):</span>
</span><span id="L-759"><a href="#L-759"><span class="linenos">759</span></a> <span class="n">expressions</span> <span class="o">=</span> <span class="p">[</span><span class="n">scope</span><span class="o">.</span><span class="n">expression</span><span class="o">.</span><span class="n">this</span><span class="p">]</span>
</span><span id="L-760"><a href="#L-760"><span class="linenos">760</span></a> <span class="k">else</span><span class="p">:</span>
</span><span id="L-761"><a href="#L-761"><span class="linenos">761</span></a> <span class="n">expressions</span> <span class="o">=</span> <span class="p">[]</span>
</span><span id="L-762"><a href="#L-762"><span class="linenos">762</span></a>
</span><span id="L-763"><a href="#L-763"><span class="linenos">763</span></a> <span class="n">sources</span> <span class="o">=</span> <span class="p">{}</span>
</span><span id="L-764"><a href="#L-764"><span class="linenos">764</span></a> <span class="k">for</span> <span class="n">expression</span> <span class="ow">in</span> <span class="n">expressions</span><span class="p">:</span>
</span><span id="L-765"><a href="#L-765"><span class="linenos">765</span></a> <span class="k">if</span> <span class="n">_is_derived_table</span><span class="p">(</span><span class="n">expression</span><span class="p">):</span>
</span><span id="L-766"><a href="#L-766"><span class="linenos">766</span></a> <span class="n">top</span> <span class="o">=</span> <span class="kc">None</span>
</span><span id="L-767"><a href="#L-767"><span class="linenos">767</span></a> <span class="k">for</span> <span class="n">child_scope</span> <span class="ow">in</span> <span class="n">_traverse_scope</span><span class="p">(</span>
</span><span id="L-768"><a href="#L-768"><span class="linenos">768</span></a> <span class="n">scope</span><span class="o">.</span><span class="n">branch</span><span class="p">(</span>
</span><span id="L-769"><a href="#L-769"><span class="linenos">769</span></a> <span class="n">expression</span><span class="p">,</span>
</span><span id="L-770"><a href="#L-770"><span class="linenos">770</span></a> <span class="n">scope_type</span><span class="o">=</span><span class="n">ScopeType</span><span class="o">.</span><span class="n">SUBQUERY</span><span class="p">,</span>
</span><span id="L-771"><a href="#L-771"><span class="linenos">771</span></a> <span class="n">outer_columns</span><span class="o">=</span><span class="n">expression</span><span class="o">.</span><span class="n">alias_column_names</span><span class="p">,</span>
</span><span id="L-772"><a href="#L-772"><span class="linenos">772</span></a> <span class="p">)</span>
</span><span id="L-773"><a href="#L-773"><span class="linenos">773</span></a> <span class="p">):</span>
</span><span id="L-774"><a href="#L-774"><span class="linenos">774</span></a> <span class="k">yield</span> <span class="n">child_scope</span>
</span><span id="L-775"><a href="#L-775"><span class="linenos">775</span></a> <span class="n">top</span> <span class="o">=</span> <span class="n">child_scope</span>
</span><span id="L-776"><a href="#L-776"><span class="linenos">776</span></a> <span class="n">sources</span><span class="p">[</span><span class="n">expression</span><span class="o">.</span><span class="n">alias</span><span class="p">]</span> <span class="o">=</span> <span class="n">child_scope</span>
</span><span id="L-777"><a href="#L-777"><span class="linenos">777</span></a>
</span><span id="L-778"><a href="#L-778"><span class="linenos">778</span></a> <span class="n">scope</span><span class="o">.</span><span class="n">subquery_scopes</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">top</span><span class="p">)</span>
</span><span id="L-745"><a href="#L-745"><span class="linenos">745</span></a> <span class="n">scope</span><span class="o">.</span><span class="n">sources</span><span class="o">.</span><span class="n">update</span><span class="p">(</span><span class="n">sources</span><span class="p">)</span>
</span><span id="L-746"><a href="#L-746"><span class="linenos">746</span></a>
</span><span id="L-747"><a href="#L-747"><span class="linenos">747</span></a>
</span><span id="L-748"><a href="#L-748"><span class="linenos">748</span></a><span class="k">def</span> <span class="nf">_traverse_subqueries</span><span class="p">(</span><span class="n">scope</span><span class="p">):</span>
</span><span id="L-749"><a href="#L-749"><span class="linenos">749</span></a> <span class="k">for</span> <span class="n">subquery</span> <span class="ow">in</span> <span class="n">scope</span><span class="o">.</span><span class="n">subqueries</span><span class="p">:</span>
</span><span id="L-750"><a href="#L-750"><span class="linenos">750</span></a> <span class="n">top</span> <span class="o">=</span> <span class="kc">None</span>
</span><span id="L-751"><a href="#L-751"><span class="linenos">751</span></a> <span class="k">for</span> <span class="n">child_scope</span> <span class="ow">in</span> <span class="n">_traverse_scope</span><span class="p">(</span><span class="n">scope</span><span class="o">.</span><span class="n">branch</span><span class="p">(</span><span class="n">subquery</span><span class="p">,</span> <span class="n">scope_type</span><span class="o">=</span><span class="n">ScopeType</span><span class="o">.</span><span class="n">SUBQUERY</span><span class="p">)):</span>
</span><span id="L-752"><a href="#L-752"><span class="linenos">752</span></a> <span class="k">yield</span> <span class="n">child_scope</span>
</span><span id="L-753"><a href="#L-753"><span class="linenos">753</span></a> <span class="n">top</span> <span class="o">=</span> <span class="n">child_scope</span>
</span><span id="L-754"><a href="#L-754"><span class="linenos">754</span></a> <span class="n">scope</span><span class="o">.</span><span class="n">subquery_scopes</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">top</span><span class="p">)</span>
</span><span id="L-755"><a href="#L-755"><span class="linenos">755</span></a>
</span><span id="L-756"><a href="#L-756"><span class="linenos">756</span></a>
</span><span id="L-757"><a href="#L-757"><span class="linenos">757</span></a><span class="k">def</span> <span class="nf">_traverse_udtfs</span><span class="p">(</span><span class="n">scope</span><span class="p">):</span>
</span><span id="L-758"><a href="#L-758"><span class="linenos">758</span></a> <span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">scope</span><span class="o">.</span><span class="n">expression</span><span class="p">,</span> <span class="n">exp</span><span class="o">.</span><span class="n">Unnest</span><span class="p">):</span>
</span><span id="L-759"><a href="#L-759"><span class="linenos">759</span></a> <span class="n">expressions</span> <span class="o">=</span> <span class="n">scope</span><span class="o">.</span><span class="n">expression</span><span class="o">.</span><span class="n">expressions</span>
</span><span id="L-760"><a href="#L-760"><span class="linenos">760</span></a> <span class="k">elif</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">scope</span><span class="o">.</span><span class="n">expression</span><span class="p">,</span> <span class="n">exp</span><span class="o">.</span><span class="n">Lateral</span><span class="p">):</span>
</span><span id="L-761"><a href="#L-761"><span class="linenos">761</span></a> <span class="n">expressions</span> <span class="o">=</span> <span class="p">[</span><span class="n">scope</span><span class="o">.</span><span class="n">expression</span><span class="o">.</span><span class="n">this</span><span class="p">]</span>
</span><span id="L-762"><a href="#L-762"><span class="linenos">762</span></a> <span class="k">else</span><span class="p">:</span>
</span><span id="L-763"><a href="#L-763"><span class="linenos">763</span></a> <span class="n">expressions</span> <span class="o">=</span> <span class="p">[]</span>
</span><span id="L-764"><a href="#L-764"><span class="linenos">764</span></a>
</span><span id="L-765"><a href="#L-765"><span class="linenos">765</span></a> <span class="n">sources</span> <span class="o">=</span> <span class="p">{}</span>
</span><span id="L-766"><a href="#L-766"><span class="linenos">766</span></a> <span class="k">for</span> <span class="n">expression</span> <span class="ow">in</span> <span class="n">expressions</span><span class="p">:</span>
</span><span id="L-767"><a href="#L-767"><span class="linenos">767</span></a> <span class="k">if</span> <span class="n">_is_derived_table</span><span class="p">(</span><span class="n">expression</span><span class="p">):</span>
</span><span id="L-768"><a href="#L-768"><span class="linenos">768</span></a> <span class="n">top</span> <span class="o">=</span> <span class="kc">None</span>
</span><span id="L-769"><a href="#L-769"><span class="linenos">769</span></a> <span class="k">for</span> <span class="n">child_scope</span> <span class="ow">in</span> <span class="n">_traverse_scope</span><span class="p">(</span>
</span><span id="L-770"><a href="#L-770"><span class="linenos">770</span></a> <span class="n">scope</span><span class="o">.</span><span class="n">branch</span><span class="p">(</span>
</span><span id="L-771"><a href="#L-771"><span class="linenos">771</span></a> <span class="n">expression</span><span class="p">,</span>
</span><span id="L-772"><a href="#L-772"><span class="linenos">772</span></a> <span class="n">scope_type</span><span class="o">=</span><span class="n">ScopeType</span><span class="o">.</span><span class="n">SUBQUERY</span><span class="p">,</span>
</span><span id="L-773"><a href="#L-773"><span class="linenos">773</span></a> <span class="n">outer_columns</span><span class="o">=</span><span class="n">expression</span><span class="o">.</span><span class="n">alias_column_names</span><span class="p">,</span>
</span><span id="L-774"><a href="#L-774"><span class="linenos">774</span></a> <span class="p">)</span>
</span><span id="L-775"><a href="#L-775"><span class="linenos">775</span></a> <span class="p">):</span>
</span><span id="L-776"><a href="#L-776"><span class="linenos">776</span></a> <span class="k">yield</span> <span class="n">child_scope</span>
</span><span id="L-777"><a href="#L-777"><span class="linenos">777</span></a> <span class="n">top</span> <span class="o">=</span> <span class="n">child_scope</span>
</span><span id="L-778"><a href="#L-778"><span class="linenos">778</span></a> <span class="n">sources</span><span class="p">[</span><span class="n">expression</span><span class="o">.</span><span class="n">alias</span><span class="p">]</span> <span class="o">=</span> <span class="n">child_scope</span>
</span><span id="L-779"><a href="#L-779"><span class="linenos">779</span></a>
</span><span id="L-780"><a href="#L-780"><span class="linenos">780</span></a> <span class="n">scope</span><span class="o">.</span><span class="n">sources</span><span class="o">.</span><span class="n">update</span><span class="p">(</span><span class="n">sources</span><span class="p">)</span>
</span><span id="L-780"><a href="#L-780"><span class="linenos">780</span></a> <span class="n">scope</span><span class="o">.</span><span class="n">subquery_scopes</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">top</span><span class="p">)</span>
</span><span id="L-781"><a href="#L-781"><span class="linenos">781</span></a>
</span><span id="L-782"><a href="#L-782"><span class="linenos">782</span></a>
</span><span id="L-783"><a href="#L-783"><span class="linenos">783</span></a><span class="k">def</span> <span class="nf">walk_in_scope</span><span class="p">(</span><span class="n">expression</span><span class="p">,</span> <span class="n">bfs</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span> <span class="n">prune</span><span class="o">=</span><span class="kc">None</span><span class="p">):</span>
</span><span id="L-784"><a href="#L-784"><span class="linenos">784</span></a><span class="w"> </span><span class="sd">&quot;&quot;&quot;</span>
</span><span id="L-785"><a href="#L-785"><span class="linenos">785</span></a><span class="sd"> Returns a generator object which visits all nodes in the syntrax tree, stopping at</span>
</span><span id="L-786"><a href="#L-786"><span class="linenos">786</span></a><span class="sd"> nodes that start child scopes.</span>
</span><span id="L-787"><a href="#L-787"><span class="linenos">787</span></a>
</span><span id="L-788"><a href="#L-788"><span class="linenos">788</span></a><span class="sd"> Args:</span>
</span><span id="L-789"><a href="#L-789"><span class="linenos">789</span></a><span class="sd"> expression (exp.Expression):</span>
</span><span id="L-790"><a href="#L-790"><span class="linenos">790</span></a><span class="sd"> bfs (bool): if set to True the BFS traversal order will be applied,</span>
</span><span id="L-791"><a href="#L-791"><span class="linenos">791</span></a><span class="sd"> otherwise the DFS traversal will be used instead.</span>
</span><span id="L-792"><a href="#L-792"><span class="linenos">792</span></a><span class="sd"> prune ((node, parent, arg_key) -&gt; bool): callable that returns True if</span>
</span><span id="L-793"><a href="#L-793"><span class="linenos">793</span></a><span class="sd"> the generator should stop traversing this branch of the tree.</span>
</span><span id="L-794"><a href="#L-794"><span class="linenos">794</span></a>
</span><span id="L-795"><a href="#L-795"><span class="linenos">795</span></a><span class="sd"> Yields:</span>
</span><span id="L-796"><a href="#L-796"><span class="linenos">796</span></a><span class="sd"> tuple[exp.Expression, Optional[exp.Expression], str]: node, parent, arg key</span>
</span><span id="L-797"><a href="#L-797"><span class="linenos">797</span></a><span class="sd"> &quot;&quot;&quot;</span>
</span><span id="L-798"><a href="#L-798"><span class="linenos">798</span></a> <span class="c1"># We&#39;ll use this variable to pass state into the dfs generator.</span>
</span><span id="L-799"><a href="#L-799"><span class="linenos">799</span></a> <span class="c1"># Whenever we set it to True, we exclude a subtree from traversal.</span>
</span><span id="L-800"><a href="#L-800"><span class="linenos">800</span></a> <span class="n">crossed_scope_boundary</span> <span class="o">=</span> <span class="kc">False</span>
</span><span id="L-801"><a href="#L-801"><span class="linenos">801</span></a>
</span><span id="L-802"><a href="#L-802"><span class="linenos">802</span></a> <span class="k">for</span> <span class="n">node</span> <span class="ow">in</span> <span class="n">expression</span><span class="o">.</span><span class="n">walk</span><span class="p">(</span>
</span><span id="L-803"><a href="#L-803"><span class="linenos">803</span></a> <span class="n">bfs</span><span class="o">=</span><span class="n">bfs</span><span class="p">,</span> <span class="n">prune</span><span class="o">=</span><span class="k">lambda</span> <span class="n">n</span><span class="p">:</span> <span class="n">crossed_scope_boundary</span> <span class="ow">or</span> <span class="p">(</span><span class="n">prune</span> <span class="ow">and</span> <span class="n">prune</span><span class="p">(</span><span class="n">n</span><span class="p">))</span>
</span><span id="L-804"><a href="#L-804"><span class="linenos">804</span></a> <span class="p">):</span>
</span><span id="L-805"><a href="#L-805"><span class="linenos">805</span></a> <span class="n">crossed_scope_boundary</span> <span class="o">=</span> <span class="kc">False</span>
</span><span id="L-806"><a href="#L-806"><span class="linenos">806</span></a>
</span><span id="L-807"><a href="#L-807"><span class="linenos">807</span></a> <span class="k">yield</span> <span class="n">node</span>
</span><span id="L-782"><a href="#L-782"><span class="linenos">782</span></a> <span class="n">scope</span><span class="o">.</span><span class="n">sources</span><span class="o">.</span><span class="n">update</span><span class="p">(</span><span class="n">sources</span><span class="p">)</span>
</span><span id="L-783"><a href="#L-783"><span class="linenos">783</span></a>
</span><span id="L-784"><a href="#L-784"><span class="linenos">784</span></a>
</span><span id="L-785"><a href="#L-785"><span class="linenos">785</span></a><span class="k">def</span> <span class="nf">walk_in_scope</span><span class="p">(</span><span class="n">expression</span><span class="p">,</span> <span class="n">bfs</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span> <span class="n">prune</span><span class="o">=</span><span class="kc">None</span><span class="p">):</span>
</span><span id="L-786"><a href="#L-786"><span class="linenos">786</span></a><span class="w"> </span><span class="sd">&quot;&quot;&quot;</span>
</span><span id="L-787"><a href="#L-787"><span class="linenos">787</span></a><span class="sd"> Returns a generator object which visits all nodes in the syntrax tree, stopping at</span>
</span><span id="L-788"><a href="#L-788"><span class="linenos">788</span></a><span class="sd"> nodes that start child scopes.</span>
</span><span id="L-789"><a href="#L-789"><span class="linenos">789</span></a>
</span><span id="L-790"><a href="#L-790"><span class="linenos">790</span></a><span class="sd"> Args:</span>
</span><span id="L-791"><a href="#L-791"><span class="linenos">791</span></a><span class="sd"> expression (exp.Expression):</span>
</span><span id="L-792"><a href="#L-792"><span class="linenos">792</span></a><span class="sd"> bfs (bool): if set to True the BFS traversal order will be applied,</span>
</span><span id="L-793"><a href="#L-793"><span class="linenos">793</span></a><span class="sd"> otherwise the DFS traversal will be used instead.</span>
</span><span id="L-794"><a href="#L-794"><span class="linenos">794</span></a><span class="sd"> prune ((node, parent, arg_key) -&gt; bool): callable that returns True if</span>
</span><span id="L-795"><a href="#L-795"><span class="linenos">795</span></a><span class="sd"> the generator should stop traversing this branch of the tree.</span>
</span><span id="L-796"><a href="#L-796"><span class="linenos">796</span></a>
</span><span id="L-797"><a href="#L-797"><span class="linenos">797</span></a><span class="sd"> Yields:</span>
</span><span id="L-798"><a href="#L-798"><span class="linenos">798</span></a><span class="sd"> tuple[exp.Expression, Optional[exp.Expression], str]: node, parent, arg key</span>
</span><span id="L-799"><a href="#L-799"><span class="linenos">799</span></a><span class="sd"> &quot;&quot;&quot;</span>
</span><span id="L-800"><a href="#L-800"><span class="linenos">800</span></a> <span class="c1"># We&#39;ll use this variable to pass state into the dfs generator.</span>
</span><span id="L-801"><a href="#L-801"><span class="linenos">801</span></a> <span class="c1"># Whenever we set it to True, we exclude a subtree from traversal.</span>
</span><span id="L-802"><a href="#L-802"><span class="linenos">802</span></a> <span class="n">crossed_scope_boundary</span> <span class="o">=</span> <span class="kc">False</span>
</span><span id="L-803"><a href="#L-803"><span class="linenos">803</span></a>
</span><span id="L-804"><a href="#L-804"><span class="linenos">804</span></a> <span class="k">for</span> <span class="n">node</span> <span class="ow">in</span> <span class="n">expression</span><span class="o">.</span><span class="n">walk</span><span class="p">(</span>
</span><span id="L-805"><a href="#L-805"><span class="linenos">805</span></a> <span class="n">bfs</span><span class="o">=</span><span class="n">bfs</span><span class="p">,</span> <span class="n">prune</span><span class="o">=</span><span class="k">lambda</span> <span class="n">n</span><span class="p">:</span> <span class="n">crossed_scope_boundary</span> <span class="ow">or</span> <span class="p">(</span><span class="n">prune</span> <span class="ow">and</span> <span class="n">prune</span><span class="p">(</span><span class="n">n</span><span class="p">))</span>
</span><span id="L-806"><a href="#L-806"><span class="linenos">806</span></a> <span class="p">):</span>
</span><span id="L-807"><a href="#L-807"><span class="linenos">807</span></a> <span class="n">crossed_scope_boundary</span> <span class="o">=</span> <span class="kc">False</span>
</span><span id="L-808"><a href="#L-808"><span class="linenos">808</span></a>
</span><span id="L-809"><a href="#L-809"><span class="linenos">809</span></a> <span class="k">if</span> <span class="n">node</span> <span class="ow">is</span> <span class="n">expression</span><span class="p">:</span>
</span><span id="L-810"><a href="#L-810"><span class="linenos">810</span></a> <span class="k">continue</span>
</span><span id="L-811"><a href="#L-811"><span class="linenos">811</span></a> <span class="k">if</span> <span class="p">(</span>
</span><span id="L-812"><a href="#L-812"><span class="linenos">812</span></a> <span class="nb">isinstance</span><span class="p">(</span><span class="n">node</span><span class="p">,</span> <span class="n">exp</span><span class="o">.</span><span class="n">CTE</span><span class="p">)</span>
</span><span id="L-813"><a href="#L-813"><span class="linenos">813</span></a> <span class="ow">or</span> <span class="p">(</span>
</span><span id="L-814"><a href="#L-814"><span class="linenos">814</span></a> <span class="nb">isinstance</span><span class="p">(</span><span class="n">node</span><span class="o">.</span><span class="n">parent</span><span class="p">,</span> <span class="p">(</span><span class="n">exp</span><span class="o">.</span><span class="n">From</span><span class="p">,</span> <span class="n">exp</span><span class="o">.</span><span class="n">Join</span><span class="p">,</span> <span class="n">exp</span><span class="o">.</span><span class="n">Subquery</span><span class="p">))</span>
</span><span id="L-815"><a href="#L-815"><span class="linenos">815</span></a> <span class="ow">and</span> <span class="p">(</span><span class="n">_is_derived_table</span><span class="p">(</span><span class="n">node</span><span class="p">)</span> <span class="ow">or</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">node</span><span class="p">,</span> <span class="n">exp</span><span class="o">.</span><span class="n">UDTF</span><span class="p">))</span>
</span><span id="L-816"><a href="#L-816"><span class="linenos">816</span></a> <span class="p">)</span>
</span><span id="L-817"><a href="#L-817"><span class="linenos">817</span></a> <span class="ow">or</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">node</span><span class="p">,</span> <span class="n">exp</span><span class="o">.</span><span class="n">UNWRAPPED_QUERIES</span><span class="p">)</span>
</span><span id="L-818"><a href="#L-818"><span class="linenos">818</span></a> <span class="p">):</span>
</span><span id="L-819"><a href="#L-819"><span class="linenos">819</span></a> <span class="n">crossed_scope_boundary</span> <span class="o">=</span> <span class="kc">True</span>
</span><span id="L-820"><a href="#L-820"><span class="linenos">820</span></a>
</span><span id="L-821"><a href="#L-821"><span class="linenos">821</span></a> <span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">node</span><span class="p">,</span> <span class="p">(</span><span class="n">exp</span><span class="o">.</span><span class="n">Subquery</span><span class="p">,</span> <span class="n">exp</span><span class="o">.</span><span class="n">UDTF</span><span class="p">)):</span>
</span><span id="L-822"><a href="#L-822"><span class="linenos">822</span></a> <span class="c1"># The following args are not actually in the inner scope, so we should visit them</span>
</span><span id="L-823"><a href="#L-823"><span class="linenos">823</span></a> <span class="k">for</span> <span class="n">key</span> <span class="ow">in</span> <span class="p">(</span><span class="s2">&quot;joins&quot;</span><span class="p">,</span> <span class="s2">&quot;laterals&quot;</span><span class="p">,</span> <span class="s2">&quot;pivots&quot;</span><span class="p">):</span>
</span><span id="L-824"><a href="#L-824"><span class="linenos">824</span></a> <span class="k">for</span> <span class="n">arg</span> <span class="ow">in</span> <span class="n">node</span><span class="o">.</span><span class="n">args</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">key</span><span class="p">)</span> <span class="ow">or</span> <span class="p">[]:</span>
</span><span id="L-825"><a href="#L-825"><span class="linenos">825</span></a> <span class="k">yield from</span> <span class="n">walk_in_scope</span><span class="p">(</span><span class="n">arg</span><span class="p">,</span> <span class="n">bfs</span><span class="o">=</span><span class="n">bfs</span><span class="p">)</span>
</span><span id="L-826"><a href="#L-826"><span class="linenos">826</span></a>
</span><span id="L-827"><a href="#L-827"><span class="linenos">827</span></a>
</span><span id="L-828"><a href="#L-828"><span class="linenos">828</span></a><span class="k">def</span> <span class="nf">find_all_in_scope</span><span class="p">(</span><span class="n">expression</span><span class="p">,</span> <span class="n">expression_types</span><span class="p">,</span> <span class="n">bfs</span><span class="o">=</span><span class="kc">True</span><span class="p">):</span>
</span><span id="L-829"><a href="#L-829"><span class="linenos">829</span></a><span class="w"> </span><span class="sd">&quot;&quot;&quot;</span>
</span><span id="L-830"><a href="#L-830"><span class="linenos">830</span></a><span class="sd"> Returns a generator object which visits all nodes in this scope and only yields those that</span>
</span><span id="L-831"><a href="#L-831"><span class="linenos">831</span></a><span class="sd"> match at least one of the specified expression types.</span>
</span><span id="L-832"><a href="#L-832"><span class="linenos">832</span></a>
</span><span id="L-833"><a href="#L-833"><span class="linenos">833</span></a><span class="sd"> This does NOT traverse into subscopes.</span>
</span><span id="L-809"><a href="#L-809"><span class="linenos">809</span></a> <span class="k">yield</span> <span class="n">node</span>
</span><span id="L-810"><a href="#L-810"><span class="linenos">810</span></a>
</span><span id="L-811"><a href="#L-811"><span class="linenos">811</span></a> <span class="k">if</span> <span class="n">node</span> <span class="ow">is</span> <span class="n">expression</span><span class="p">:</span>
</span><span id="L-812"><a href="#L-812"><span class="linenos">812</span></a> <span class="k">continue</span>
</span><span id="L-813"><a href="#L-813"><span class="linenos">813</span></a> <span class="k">if</span> <span class="p">(</span>
</span><span id="L-814"><a href="#L-814"><span class="linenos">814</span></a> <span class="nb">isinstance</span><span class="p">(</span><span class="n">node</span><span class="p">,</span> <span class="n">exp</span><span class="o">.</span><span class="n">CTE</span><span class="p">)</span>
</span><span id="L-815"><a href="#L-815"><span class="linenos">815</span></a> <span class="ow">or</span> <span class="p">(</span>
</span><span id="L-816"><a href="#L-816"><span class="linenos">816</span></a> <span class="nb">isinstance</span><span class="p">(</span><span class="n">node</span><span class="o">.</span><span class="n">parent</span><span class="p">,</span> <span class="p">(</span><span class="n">exp</span><span class="o">.</span><span class="n">From</span><span class="p">,</span> <span class="n">exp</span><span class="o">.</span><span class="n">Join</span><span class="p">,</span> <span class="n">exp</span><span class="o">.</span><span class="n">Subquery</span><span class="p">))</span>
</span><span id="L-817"><a href="#L-817"><span class="linenos">817</span></a> <span class="ow">and</span> <span class="p">(</span><span class="n">_is_derived_table</span><span class="p">(</span><span class="n">node</span><span class="p">)</span> <span class="ow">or</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">node</span><span class="p">,</span> <span class="n">exp</span><span class="o">.</span><span class="n">UDTF</span><span class="p">))</span>
</span><span id="L-818"><a href="#L-818"><span class="linenos">818</span></a> <span class="p">)</span>
</span><span id="L-819"><a href="#L-819"><span class="linenos">819</span></a> <span class="ow">or</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">node</span><span class="p">,</span> <span class="n">exp</span><span class="o">.</span><span class="n">UNWRAPPED_QUERIES</span><span class="p">)</span>
</span><span id="L-820"><a href="#L-820"><span class="linenos">820</span></a> <span class="p">):</span>
</span><span id="L-821"><a href="#L-821"><span class="linenos">821</span></a> <span class="n">crossed_scope_boundary</span> <span class="o">=</span> <span class="kc">True</span>
</span><span id="L-822"><a href="#L-822"><span class="linenos">822</span></a>
</span><span id="L-823"><a href="#L-823"><span class="linenos">823</span></a> <span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">node</span><span class="p">,</span> <span class="p">(</span><span class="n">exp</span><span class="o">.</span><span class="n">Subquery</span><span class="p">,</span> <span class="n">exp</span><span class="o">.</span><span class="n">UDTF</span><span class="p">)):</span>
</span><span id="L-824"><a href="#L-824"><span class="linenos">824</span></a> <span class="c1"># The following args are not actually in the inner scope, so we should visit them</span>
</span><span id="L-825"><a href="#L-825"><span class="linenos">825</span></a> <span class="k">for</span> <span class="n">key</span> <span class="ow">in</span> <span class="p">(</span><span class="s2">&quot;joins&quot;</span><span class="p">,</span> <span class="s2">&quot;laterals&quot;</span><span class="p">,</span> <span class="s2">&quot;pivots&quot;</span><span class="p">):</span>
</span><span id="L-826"><a href="#L-826"><span class="linenos">826</span></a> <span class="k">for</span> <span class="n">arg</span> <span class="ow">in</span> <span class="n">node</span><span class="o">.</span><span class="n">args</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">key</span><span class="p">)</span> <span class="ow">or</span> <span class="p">[]:</span>
</span><span id="L-827"><a href="#L-827"><span class="linenos">827</span></a> <span class="k">yield from</span> <span class="n">walk_in_scope</span><span class="p">(</span><span class="n">arg</span><span class="p">,</span> <span class="n">bfs</span><span class="o">=</span><span class="n">bfs</span><span class="p">)</span>
</span><span id="L-828"><a href="#L-828"><span class="linenos">828</span></a>
</span><span id="L-829"><a href="#L-829"><span class="linenos">829</span></a>
</span><span id="L-830"><a href="#L-830"><span class="linenos">830</span></a><span class="k">def</span> <span class="nf">find_all_in_scope</span><span class="p">(</span><span class="n">expression</span><span class="p">,</span> <span class="n">expression_types</span><span class="p">,</span> <span class="n">bfs</span><span class="o">=</span><span class="kc">True</span><span class="p">):</span>
</span><span id="L-831"><a href="#L-831"><span class="linenos">831</span></a><span class="w"> </span><span class="sd">&quot;&quot;&quot;</span>
</span><span id="L-832"><a href="#L-832"><span class="linenos">832</span></a><span class="sd"> Returns a generator object which visits all nodes in this scope and only yields those that</span>
</span><span id="L-833"><a href="#L-833"><span class="linenos">833</span></a><span class="sd"> match at least one of the specified expression types.</span>
</span><span id="L-834"><a href="#L-834"><span class="linenos">834</span></a>
</span><span id="L-835"><a href="#L-835"><span class="linenos">835</span></a><span class="sd"> Args:</span>
</span><span id="L-836"><a href="#L-836"><span class="linenos">836</span></a><span class="sd"> expression (exp.Expression):</span>
</span><span id="L-837"><a href="#L-837"><span class="linenos">837</span></a><span class="sd"> expression_types (tuple[type]|type): the expression type(s) to match.</span>
</span><span id="L-838"><a href="#L-838"><span class="linenos">838</span></a><span class="sd"> bfs (bool): True to use breadth-first search, False to use depth-first.</span>
</span><span id="L-839"><a href="#L-839"><span class="linenos">839</span></a>
</span><span id="L-840"><a href="#L-840"><span class="linenos">840</span></a><span class="sd"> Yields:</span>
</span><span id="L-841"><a href="#L-841"><span class="linenos">841</span></a><span class="sd"> exp.Expression: nodes</span>
</span><span id="L-842"><a href="#L-842"><span class="linenos">842</span></a><span class="sd"> &quot;&quot;&quot;</span>
</span><span id="L-843"><a href="#L-843"><span class="linenos">843</span></a> <span class="k">for</span> <span class="n">expression</span> <span class="ow">in</span> <span class="n">walk_in_scope</span><span class="p">(</span><span class="n">expression</span><span class="p">,</span> <span class="n">bfs</span><span class="o">=</span><span class="n">bfs</span><span class="p">):</span>
</span><span id="L-844"><a href="#L-844"><span class="linenos">844</span></a> <span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">expression</span><span class="p">,</span> <span class="nb">tuple</span><span class="p">(</span><span class="n">ensure_collection</span><span class="p">(</span><span class="n">expression_types</span><span class="p">))):</span>
</span><span id="L-845"><a href="#L-845"><span class="linenos">845</span></a> <span class="k">yield</span> <span class="n">expression</span>
</span><span id="L-846"><a href="#L-846"><span class="linenos">846</span></a>
</span><span id="L-847"><a href="#L-847"><span class="linenos">847</span></a>
</span><span id="L-848"><a href="#L-848"><span class="linenos">848</span></a><span class="k">def</span> <span class="nf">find_in_scope</span><span class="p">(</span><span class="n">expression</span><span class="p">,</span> <span class="n">expression_types</span><span class="p">,</span> <span class="n">bfs</span><span class="o">=</span><span class="kc">True</span><span class="p">):</span>
</span><span id="L-849"><a href="#L-849"><span class="linenos">849</span></a><span class="w"> </span><span class="sd">&quot;&quot;&quot;</span>
</span><span id="L-850"><a href="#L-850"><span class="linenos">850</span></a><span class="sd"> Returns the first node in this scope which matches at least one of the specified types.</span>
</span><span id="L-851"><a href="#L-851"><span class="linenos">851</span></a>
</span><span id="L-852"><a href="#L-852"><span class="linenos">852</span></a><span class="sd"> This does NOT traverse into subscopes.</span>
</span><span id="L-835"><a href="#L-835"><span class="linenos">835</span></a><span class="sd"> This does NOT traverse into subscopes.</span>
</span><span id="L-836"><a href="#L-836"><span class="linenos">836</span></a>
</span><span id="L-837"><a href="#L-837"><span class="linenos">837</span></a><span class="sd"> Args:</span>
</span><span id="L-838"><a href="#L-838"><span class="linenos">838</span></a><span class="sd"> expression (exp.Expression):</span>
</span><span id="L-839"><a href="#L-839"><span class="linenos">839</span></a><span class="sd"> expression_types (tuple[type]|type): the expression type(s) to match.</span>
</span><span id="L-840"><a href="#L-840"><span class="linenos">840</span></a><span class="sd"> bfs (bool): True to use breadth-first search, False to use depth-first.</span>
</span><span id="L-841"><a href="#L-841"><span class="linenos">841</span></a>
</span><span id="L-842"><a href="#L-842"><span class="linenos">842</span></a><span class="sd"> Yields:</span>
</span><span id="L-843"><a href="#L-843"><span class="linenos">843</span></a><span class="sd"> exp.Expression: nodes</span>
</span><span id="L-844"><a href="#L-844"><span class="linenos">844</span></a><span class="sd"> &quot;&quot;&quot;</span>
</span><span id="L-845"><a href="#L-845"><span class="linenos">845</span></a> <span class="k">for</span> <span class="n">expression</span> <span class="ow">in</span> <span class="n">walk_in_scope</span><span class="p">(</span><span class="n">expression</span><span class="p">,</span> <span class="n">bfs</span><span class="o">=</span><span class="n">bfs</span><span class="p">):</span>
</span><span id="L-846"><a href="#L-846"><span class="linenos">846</span></a> <span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">expression</span><span class="p">,</span> <span class="nb">tuple</span><span class="p">(</span><span class="n">ensure_collection</span><span class="p">(</span><span class="n">expression_types</span><span class="p">))):</span>
</span><span id="L-847"><a href="#L-847"><span class="linenos">847</span></a> <span class="k">yield</span> <span class="n">expression</span>
</span><span id="L-848"><a href="#L-848"><span class="linenos">848</span></a>
</span><span id="L-849"><a href="#L-849"><span class="linenos">849</span></a>
</span><span id="L-850"><a href="#L-850"><span class="linenos">850</span></a><span class="k">def</span> <span class="nf">find_in_scope</span><span class="p">(</span><span class="n">expression</span><span class="p">,</span> <span class="n">expression_types</span><span class="p">,</span> <span class="n">bfs</span><span class="o">=</span><span class="kc">True</span><span class="p">):</span>
</span><span id="L-851"><a href="#L-851"><span class="linenos">851</span></a><span class="w"> </span><span class="sd">&quot;&quot;&quot;</span>
</span><span id="L-852"><a href="#L-852"><span class="linenos">852</span></a><span class="sd"> Returns the first node in this scope which matches at least one of the specified types.</span>
</span><span id="L-853"><a href="#L-853"><span class="linenos">853</span></a>
</span><span id="L-854"><a href="#L-854"><span class="linenos">854</span></a><span class="sd"> Args:</span>
</span><span id="L-855"><a href="#L-855"><span class="linenos">855</span></a><span class="sd"> expression (exp.Expression):</span>
</span><span id="L-856"><a href="#L-856"><span class="linenos">856</span></a><span class="sd"> expression_types (tuple[type]|type): the expression type(s) to match.</span>
</span><span id="L-857"><a href="#L-857"><span class="linenos">857</span></a><span class="sd"> bfs (bool): True to use breadth-first search, False to use depth-first.</span>
</span><span id="L-858"><a href="#L-858"><span class="linenos">858</span></a>
</span><span id="L-859"><a href="#L-859"><span class="linenos">859</span></a><span class="sd"> Returns:</span>
</span><span id="L-860"><a href="#L-860"><span class="linenos">860</span></a><span class="sd"> exp.Expression: the node which matches the criteria or None if no node matching</span>
</span><span id="L-861"><a href="#L-861"><span class="linenos">861</span></a><span class="sd"> the criteria was found.</span>
</span><span id="L-862"><a href="#L-862"><span class="linenos">862</span></a><span class="sd"> &quot;&quot;&quot;</span>
</span><span id="L-863"><a href="#L-863"><span class="linenos">863</span></a> <span class="k">return</span> <span class="nb">next</span><span class="p">(</span><span class="n">find_all_in_scope</span><span class="p">(</span><span class="n">expression</span><span class="p">,</span> <span class="n">expression_types</span><span class="p">,</span> <span class="n">bfs</span><span class="o">=</span><span class="n">bfs</span><span class="p">),</span> <span class="kc">None</span><span class="p">)</span>
</span><span id="L-854"><a href="#L-854"><span class="linenos">854</span></a><span class="sd"> This does NOT traverse into subscopes.</span>
</span><span id="L-855"><a href="#L-855"><span class="linenos">855</span></a>
</span><span id="L-856"><a href="#L-856"><span class="linenos">856</span></a><span class="sd"> Args:</span>
</span><span id="L-857"><a href="#L-857"><span class="linenos">857</span></a><span class="sd"> expression (exp.Expression):</span>
</span><span id="L-858"><a href="#L-858"><span class="linenos">858</span></a><span class="sd"> expression_types (tuple[type]|type): the expression type(s) to match.</span>
</span><span id="L-859"><a href="#L-859"><span class="linenos">859</span></a><span class="sd"> bfs (bool): True to use breadth-first search, False to use depth-first.</span>
</span><span id="L-860"><a href="#L-860"><span class="linenos">860</span></a>
</span><span id="L-861"><a href="#L-861"><span class="linenos">861</span></a><span class="sd"> Returns:</span>
</span><span id="L-862"><a href="#L-862"><span class="linenos">862</span></a><span class="sd"> exp.Expression: the node which matches the criteria or None if no node matching</span>
</span><span id="L-863"><a href="#L-863"><span class="linenos">863</span></a><span class="sd"> the criteria was found.</span>
</span><span id="L-864"><a href="#L-864"><span class="linenos">864</span></a><span class="sd"> &quot;&quot;&quot;</span>
</span><span id="L-865"><a href="#L-865"><span class="linenos">865</span></a> <span class="k">return</span> <span class="nb">next</span><span class="p">(</span><span class="n">find_all_in_scope</span><span class="p">(</span><span class="n">expression</span><span class="p">,</span> <span class="n">expression_types</span><span class="p">,</span> <span class="n">bfs</span><span class="o">=</span><span class="n">bfs</span><span class="p">),</span> <span class="kc">None</span><span class="p">)</span>
</span></pre></div>
@ -3078,49 +3080,49 @@ incomplete properties which is confusing.</p>
</div>
<a class="headerlink" href="#walk_in_scope"></a>
<div class="pdoc-code codehilite"><pre><span></span><span id="walk_in_scope-784"><a href="#walk_in_scope-784"><span class="linenos">784</span></a><span class="k">def</span> <span class="nf">walk_in_scope</span><span class="p">(</span><span class="n">expression</span><span class="p">,</span> <span class="n">bfs</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span> <span class="n">prune</span><span class="o">=</span><span class="kc">None</span><span class="p">):</span>
</span><span id="walk_in_scope-785"><a href="#walk_in_scope-785"><span class="linenos">785</span></a><span class="w"> </span><span class="sd">&quot;&quot;&quot;</span>
</span><span id="walk_in_scope-786"><a href="#walk_in_scope-786"><span class="linenos">786</span></a><span class="sd"> Returns a generator object which visits all nodes in the syntrax tree, stopping at</span>
</span><span id="walk_in_scope-787"><a href="#walk_in_scope-787"><span class="linenos">787</span></a><span class="sd"> nodes that start child scopes.</span>
</span><span id="walk_in_scope-788"><a href="#walk_in_scope-788"><span class="linenos">788</span></a>
</span><span id="walk_in_scope-789"><a href="#walk_in_scope-789"><span class="linenos">789</span></a><span class="sd"> Args:</span>
</span><span id="walk_in_scope-790"><a href="#walk_in_scope-790"><span class="linenos">790</span></a><span class="sd"> expression (exp.Expression):</span>
</span><span id="walk_in_scope-791"><a href="#walk_in_scope-791"><span class="linenos">791</span></a><span class="sd"> bfs (bool): if set to True the BFS traversal order will be applied,</span>
</span><span id="walk_in_scope-792"><a href="#walk_in_scope-792"><span class="linenos">792</span></a><span class="sd"> otherwise the DFS traversal will be used instead.</span>
</span><span id="walk_in_scope-793"><a href="#walk_in_scope-793"><span class="linenos">793</span></a><span class="sd"> prune ((node, parent, arg_key) -&gt; bool): callable that returns True if</span>
</span><span id="walk_in_scope-794"><a href="#walk_in_scope-794"><span class="linenos">794</span></a><span class="sd"> the generator should stop traversing this branch of the tree.</span>
</span><span id="walk_in_scope-795"><a href="#walk_in_scope-795"><span class="linenos">795</span></a>
</span><span id="walk_in_scope-796"><a href="#walk_in_scope-796"><span class="linenos">796</span></a><span class="sd"> Yields:</span>
</span><span id="walk_in_scope-797"><a href="#walk_in_scope-797"><span class="linenos">797</span></a><span class="sd"> tuple[exp.Expression, Optional[exp.Expression], str]: node, parent, arg key</span>
</span><span id="walk_in_scope-798"><a href="#walk_in_scope-798"><span class="linenos">798</span></a><span class="sd"> &quot;&quot;&quot;</span>
</span><span id="walk_in_scope-799"><a href="#walk_in_scope-799"><span class="linenos">799</span></a> <span class="c1"># We&#39;ll use this variable to pass state into the dfs generator.</span>
</span><span id="walk_in_scope-800"><a href="#walk_in_scope-800"><span class="linenos">800</span></a> <span class="c1"># Whenever we set it to True, we exclude a subtree from traversal.</span>
</span><span id="walk_in_scope-801"><a href="#walk_in_scope-801"><span class="linenos">801</span></a> <span class="n">crossed_scope_boundary</span> <span class="o">=</span> <span class="kc">False</span>
</span><span id="walk_in_scope-802"><a href="#walk_in_scope-802"><span class="linenos">802</span></a>
</span><span id="walk_in_scope-803"><a href="#walk_in_scope-803"><span class="linenos">803</span></a> <span class="k">for</span> <span class="n">node</span> <span class="ow">in</span> <span class="n">expression</span><span class="o">.</span><span class="n">walk</span><span class="p">(</span>
</span><span id="walk_in_scope-804"><a href="#walk_in_scope-804"><span class="linenos">804</span></a> <span class="n">bfs</span><span class="o">=</span><span class="n">bfs</span><span class="p">,</span> <span class="n">prune</span><span class="o">=</span><span class="k">lambda</span> <span class="n">n</span><span class="p">:</span> <span class="n">crossed_scope_boundary</span> <span class="ow">or</span> <span class="p">(</span><span class="n">prune</span> <span class="ow">and</span> <span class="n">prune</span><span class="p">(</span><span class="n">n</span><span class="p">))</span>
</span><span id="walk_in_scope-805"><a href="#walk_in_scope-805"><span class="linenos">805</span></a> <span class="p">):</span>
</span><span id="walk_in_scope-806"><a href="#walk_in_scope-806"><span class="linenos">806</span></a> <span class="n">crossed_scope_boundary</span> <span class="o">=</span> <span class="kc">False</span>
</span><span id="walk_in_scope-807"><a href="#walk_in_scope-807"><span class="linenos">807</span></a>
</span><span id="walk_in_scope-808"><a href="#walk_in_scope-808"><span class="linenos">808</span></a> <span class="k">yield</span> <span class="n">node</span>
<div class="pdoc-code codehilite"><pre><span></span><span id="walk_in_scope-786"><a href="#walk_in_scope-786"><span class="linenos">786</span></a><span class="k">def</span> <span class="nf">walk_in_scope</span><span class="p">(</span><span class="n">expression</span><span class="p">,</span> <span class="n">bfs</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span> <span class="n">prune</span><span class="o">=</span><span class="kc">None</span><span class="p">):</span>
</span><span id="walk_in_scope-787"><a href="#walk_in_scope-787"><span class="linenos">787</span></a><span class="w"> </span><span class="sd">&quot;&quot;&quot;</span>
</span><span id="walk_in_scope-788"><a href="#walk_in_scope-788"><span class="linenos">788</span></a><span class="sd"> Returns a generator object which visits all nodes in the syntrax tree, stopping at</span>
</span><span id="walk_in_scope-789"><a href="#walk_in_scope-789"><span class="linenos">789</span></a><span class="sd"> nodes that start child scopes.</span>
</span><span id="walk_in_scope-790"><a href="#walk_in_scope-790"><span class="linenos">790</span></a>
</span><span id="walk_in_scope-791"><a href="#walk_in_scope-791"><span class="linenos">791</span></a><span class="sd"> Args:</span>
</span><span id="walk_in_scope-792"><a href="#walk_in_scope-792"><span class="linenos">792</span></a><span class="sd"> expression (exp.Expression):</span>
</span><span id="walk_in_scope-793"><a href="#walk_in_scope-793"><span class="linenos">793</span></a><span class="sd"> bfs (bool): if set to True the BFS traversal order will be applied,</span>
</span><span id="walk_in_scope-794"><a href="#walk_in_scope-794"><span class="linenos">794</span></a><span class="sd"> otherwise the DFS traversal will be used instead.</span>
</span><span id="walk_in_scope-795"><a href="#walk_in_scope-795"><span class="linenos">795</span></a><span class="sd"> prune ((node, parent, arg_key) -&gt; bool): callable that returns True if</span>
</span><span id="walk_in_scope-796"><a href="#walk_in_scope-796"><span class="linenos">796</span></a><span class="sd"> the generator should stop traversing this branch of the tree.</span>
</span><span id="walk_in_scope-797"><a href="#walk_in_scope-797"><span class="linenos">797</span></a>
</span><span id="walk_in_scope-798"><a href="#walk_in_scope-798"><span class="linenos">798</span></a><span class="sd"> Yields:</span>
</span><span id="walk_in_scope-799"><a href="#walk_in_scope-799"><span class="linenos">799</span></a><span class="sd"> tuple[exp.Expression, Optional[exp.Expression], str]: node, parent, arg key</span>
</span><span id="walk_in_scope-800"><a href="#walk_in_scope-800"><span class="linenos">800</span></a><span class="sd"> &quot;&quot;&quot;</span>
</span><span id="walk_in_scope-801"><a href="#walk_in_scope-801"><span class="linenos">801</span></a> <span class="c1"># We&#39;ll use this variable to pass state into the dfs generator.</span>
</span><span id="walk_in_scope-802"><a href="#walk_in_scope-802"><span class="linenos">802</span></a> <span class="c1"># Whenever we set it to True, we exclude a subtree from traversal.</span>
</span><span id="walk_in_scope-803"><a href="#walk_in_scope-803"><span class="linenos">803</span></a> <span class="n">crossed_scope_boundary</span> <span class="o">=</span> <span class="kc">False</span>
</span><span id="walk_in_scope-804"><a href="#walk_in_scope-804"><span class="linenos">804</span></a>
</span><span id="walk_in_scope-805"><a href="#walk_in_scope-805"><span class="linenos">805</span></a> <span class="k">for</span> <span class="n">node</span> <span class="ow">in</span> <span class="n">expression</span><span class="o">.</span><span class="n">walk</span><span class="p">(</span>
</span><span id="walk_in_scope-806"><a href="#walk_in_scope-806"><span class="linenos">806</span></a> <span class="n">bfs</span><span class="o">=</span><span class="n">bfs</span><span class="p">,</span> <span class="n">prune</span><span class="o">=</span><span class="k">lambda</span> <span class="n">n</span><span class="p">:</span> <span class="n">crossed_scope_boundary</span> <span class="ow">or</span> <span class="p">(</span><span class="n">prune</span> <span class="ow">and</span> <span class="n">prune</span><span class="p">(</span><span class="n">n</span><span class="p">))</span>
</span><span id="walk_in_scope-807"><a href="#walk_in_scope-807"><span class="linenos">807</span></a> <span class="p">):</span>
</span><span id="walk_in_scope-808"><a href="#walk_in_scope-808"><span class="linenos">808</span></a> <span class="n">crossed_scope_boundary</span> <span class="o">=</span> <span class="kc">False</span>
</span><span id="walk_in_scope-809"><a href="#walk_in_scope-809"><span class="linenos">809</span></a>
</span><span id="walk_in_scope-810"><a href="#walk_in_scope-810"><span class="linenos">810</span></a> <span class="k">if</span> <span class="n">node</span> <span class="ow">is</span> <span class="n">expression</span><span class="p">:</span>
</span><span id="walk_in_scope-811"><a href="#walk_in_scope-811"><span class="linenos">811</span></a> <span class="k">continue</span>
</span><span id="walk_in_scope-812"><a href="#walk_in_scope-812"><span class="linenos">812</span></a> <span class="k">if</span> <span class="p">(</span>
</span><span id="walk_in_scope-813"><a href="#walk_in_scope-813"><span class="linenos">813</span></a> <span class="nb">isinstance</span><span class="p">(</span><span class="n">node</span><span class="p">,</span> <span class="n">exp</span><span class="o">.</span><span class="n">CTE</span><span class="p">)</span>
</span><span id="walk_in_scope-814"><a href="#walk_in_scope-814"><span class="linenos">814</span></a> <span class="ow">or</span> <span class="p">(</span>
</span><span id="walk_in_scope-815"><a href="#walk_in_scope-815"><span class="linenos">815</span></a> <span class="nb">isinstance</span><span class="p">(</span><span class="n">node</span><span class="o">.</span><span class="n">parent</span><span class="p">,</span> <span class="p">(</span><span class="n">exp</span><span class="o">.</span><span class="n">From</span><span class="p">,</span> <span class="n">exp</span><span class="o">.</span><span class="n">Join</span><span class="p">,</span> <span class="n">exp</span><span class="o">.</span><span class="n">Subquery</span><span class="p">))</span>
</span><span id="walk_in_scope-816"><a href="#walk_in_scope-816"><span class="linenos">816</span></a> <span class="ow">and</span> <span class="p">(</span><span class="n">_is_derived_table</span><span class="p">(</span><span class="n">node</span><span class="p">)</span> <span class="ow">or</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">node</span><span class="p">,</span> <span class="n">exp</span><span class="o">.</span><span class="n">UDTF</span><span class="p">))</span>
</span><span id="walk_in_scope-817"><a href="#walk_in_scope-817"><span class="linenos">817</span></a> <span class="p">)</span>
</span><span id="walk_in_scope-818"><a href="#walk_in_scope-818"><span class="linenos">818</span></a> <span class="ow">or</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">node</span><span class="p">,</span> <span class="n">exp</span><span class="o">.</span><span class="n">UNWRAPPED_QUERIES</span><span class="p">)</span>
</span><span id="walk_in_scope-819"><a href="#walk_in_scope-819"><span class="linenos">819</span></a> <span class="p">):</span>
</span><span id="walk_in_scope-820"><a href="#walk_in_scope-820"><span class="linenos">820</span></a> <span class="n">crossed_scope_boundary</span> <span class="o">=</span> <span class="kc">True</span>
</span><span id="walk_in_scope-821"><a href="#walk_in_scope-821"><span class="linenos">821</span></a>
</span><span id="walk_in_scope-822"><a href="#walk_in_scope-822"><span class="linenos">822</span></a> <span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">node</span><span class="p">,</span> <span class="p">(</span><span class="n">exp</span><span class="o">.</span><span class="n">Subquery</span><span class="p">,</span> <span class="n">exp</span><span class="o">.</span><span class="n">UDTF</span><span class="p">)):</span>
</span><span id="walk_in_scope-823"><a href="#walk_in_scope-823"><span class="linenos">823</span></a> <span class="c1"># The following args are not actually in the inner scope, so we should visit them</span>
</span><span id="walk_in_scope-824"><a href="#walk_in_scope-824"><span class="linenos">824</span></a> <span class="k">for</span> <span class="n">key</span> <span class="ow">in</span> <span class="p">(</span><span class="s2">&quot;joins&quot;</span><span class="p">,</span> <span class="s2">&quot;laterals&quot;</span><span class="p">,</span> <span class="s2">&quot;pivots&quot;</span><span class="p">):</span>
</span><span id="walk_in_scope-825"><a href="#walk_in_scope-825"><span class="linenos">825</span></a> <span class="k">for</span> <span class="n">arg</span> <span class="ow">in</span> <span class="n">node</span><span class="o">.</span><span class="n">args</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">key</span><span class="p">)</span> <span class="ow">or</span> <span class="p">[]:</span>
</span><span id="walk_in_scope-826"><a href="#walk_in_scope-826"><span class="linenos">826</span></a> <span class="k">yield from</span> <span class="n">walk_in_scope</span><span class="p">(</span><span class="n">arg</span><span class="p">,</span> <span class="n">bfs</span><span class="o">=</span><span class="n">bfs</span><span class="p">)</span>
</span><span id="walk_in_scope-810"><a href="#walk_in_scope-810"><span class="linenos">810</span></a> <span class="k">yield</span> <span class="n">node</span>
</span><span id="walk_in_scope-811"><a href="#walk_in_scope-811"><span class="linenos">811</span></a>
</span><span id="walk_in_scope-812"><a href="#walk_in_scope-812"><span class="linenos">812</span></a> <span class="k">if</span> <span class="n">node</span> <span class="ow">is</span> <span class="n">expression</span><span class="p">:</span>
</span><span id="walk_in_scope-813"><a href="#walk_in_scope-813"><span class="linenos">813</span></a> <span class="k">continue</span>
</span><span id="walk_in_scope-814"><a href="#walk_in_scope-814"><span class="linenos">814</span></a> <span class="k">if</span> <span class="p">(</span>
</span><span id="walk_in_scope-815"><a href="#walk_in_scope-815"><span class="linenos">815</span></a> <span class="nb">isinstance</span><span class="p">(</span><span class="n">node</span><span class="p">,</span> <span class="n">exp</span><span class="o">.</span><span class="n">CTE</span><span class="p">)</span>
</span><span id="walk_in_scope-816"><a href="#walk_in_scope-816"><span class="linenos">816</span></a> <span class="ow">or</span> <span class="p">(</span>
</span><span id="walk_in_scope-817"><a href="#walk_in_scope-817"><span class="linenos">817</span></a> <span class="nb">isinstance</span><span class="p">(</span><span class="n">node</span><span class="o">.</span><span class="n">parent</span><span class="p">,</span> <span class="p">(</span><span class="n">exp</span><span class="o">.</span><span class="n">From</span><span class="p">,</span> <span class="n">exp</span><span class="o">.</span><span class="n">Join</span><span class="p">,</span> <span class="n">exp</span><span class="o">.</span><span class="n">Subquery</span><span class="p">))</span>
</span><span id="walk_in_scope-818"><a href="#walk_in_scope-818"><span class="linenos">818</span></a> <span class="ow">and</span> <span class="p">(</span><span class="n">_is_derived_table</span><span class="p">(</span><span class="n">node</span><span class="p">)</span> <span class="ow">or</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">node</span><span class="p">,</span> <span class="n">exp</span><span class="o">.</span><span class="n">UDTF</span><span class="p">))</span>
</span><span id="walk_in_scope-819"><a href="#walk_in_scope-819"><span class="linenos">819</span></a> <span class="p">)</span>
</span><span id="walk_in_scope-820"><a href="#walk_in_scope-820"><span class="linenos">820</span></a> <span class="ow">or</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">node</span><span class="p">,</span> <span class="n">exp</span><span class="o">.</span><span class="n">UNWRAPPED_QUERIES</span><span class="p">)</span>
</span><span id="walk_in_scope-821"><a href="#walk_in_scope-821"><span class="linenos">821</span></a> <span class="p">):</span>
</span><span id="walk_in_scope-822"><a href="#walk_in_scope-822"><span class="linenos">822</span></a> <span class="n">crossed_scope_boundary</span> <span class="o">=</span> <span class="kc">True</span>
</span><span id="walk_in_scope-823"><a href="#walk_in_scope-823"><span class="linenos">823</span></a>
</span><span id="walk_in_scope-824"><a href="#walk_in_scope-824"><span class="linenos">824</span></a> <span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">node</span><span class="p">,</span> <span class="p">(</span><span class="n">exp</span><span class="o">.</span><span class="n">Subquery</span><span class="p">,</span> <span class="n">exp</span><span class="o">.</span><span class="n">UDTF</span><span class="p">)):</span>
</span><span id="walk_in_scope-825"><a href="#walk_in_scope-825"><span class="linenos">825</span></a> <span class="c1"># The following args are not actually in the inner scope, so we should visit them</span>
</span><span id="walk_in_scope-826"><a href="#walk_in_scope-826"><span class="linenos">826</span></a> <span class="k">for</span> <span class="n">key</span> <span class="ow">in</span> <span class="p">(</span><span class="s2">&quot;joins&quot;</span><span class="p">,</span> <span class="s2">&quot;laterals&quot;</span><span class="p">,</span> <span class="s2">&quot;pivots&quot;</span><span class="p">):</span>
</span><span id="walk_in_scope-827"><a href="#walk_in_scope-827"><span class="linenos">827</span></a> <span class="k">for</span> <span class="n">arg</span> <span class="ow">in</span> <span class="n">node</span><span class="o">.</span><span class="n">args</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">key</span><span class="p">)</span> <span class="ow">or</span> <span class="p">[]:</span>
</span><span id="walk_in_scope-828"><a href="#walk_in_scope-828"><span class="linenos">828</span></a> <span class="k">yield from</span> <span class="n">walk_in_scope</span><span class="p">(</span><span class="n">arg</span><span class="p">,</span> <span class="n">bfs</span><span class="o">=</span><span class="n">bfs</span><span class="p">)</span>
</span></pre></div>
@ -3157,24 +3159,24 @@ the generator should stop traversing this branch of the tree.</li>
</div>
<a class="headerlink" href="#find_all_in_scope"></a>
<div class="pdoc-code codehilite"><pre><span></span><span id="find_all_in_scope-829"><a href="#find_all_in_scope-829"><span class="linenos">829</span></a><span class="k">def</span> <span class="nf">find_all_in_scope</span><span class="p">(</span><span class="n">expression</span><span class="p">,</span> <span class="n">expression_types</span><span class="p">,</span> <span class="n">bfs</span><span class="o">=</span><span class="kc">True</span><span class="p">):</span>
</span><span id="find_all_in_scope-830"><a href="#find_all_in_scope-830"><span class="linenos">830</span></a><span class="w"> </span><span class="sd">&quot;&quot;&quot;</span>
</span><span id="find_all_in_scope-831"><a href="#find_all_in_scope-831"><span class="linenos">831</span></a><span class="sd"> Returns a generator object which visits all nodes in this scope and only yields those that</span>
</span><span id="find_all_in_scope-832"><a href="#find_all_in_scope-832"><span class="linenos">832</span></a><span class="sd"> match at least one of the specified expression types.</span>
</span><span id="find_all_in_scope-833"><a href="#find_all_in_scope-833"><span class="linenos">833</span></a>
</span><span id="find_all_in_scope-834"><a href="#find_all_in_scope-834"><span class="linenos">834</span></a><span class="sd"> This does NOT traverse into subscopes.</span>
<div class="pdoc-code codehilite"><pre><span></span><span id="find_all_in_scope-831"><a href="#find_all_in_scope-831"><span class="linenos">831</span></a><span class="k">def</span> <span class="nf">find_all_in_scope</span><span class="p">(</span><span class="n">expression</span><span class="p">,</span> <span class="n">expression_types</span><span class="p">,</span> <span class="n">bfs</span><span class="o">=</span><span class="kc">True</span><span class="p">):</span>
</span><span id="find_all_in_scope-832"><a href="#find_all_in_scope-832"><span class="linenos">832</span></a><span class="w"> </span><span class="sd">&quot;&quot;&quot;</span>
</span><span id="find_all_in_scope-833"><a href="#find_all_in_scope-833"><span class="linenos">833</span></a><span class="sd"> Returns a generator object which visits all nodes in this scope and only yields those that</span>
</span><span id="find_all_in_scope-834"><a href="#find_all_in_scope-834"><span class="linenos">834</span></a><span class="sd"> match at least one of the specified expression types.</span>
</span><span id="find_all_in_scope-835"><a href="#find_all_in_scope-835"><span class="linenos">835</span></a>
</span><span id="find_all_in_scope-836"><a href="#find_all_in_scope-836"><span class="linenos">836</span></a><span class="sd"> Args:</span>
</span><span id="find_all_in_scope-837"><a href="#find_all_in_scope-837"><span class="linenos">837</span></a><span class="sd"> expression (exp.Expression):</span>
</span><span id="find_all_in_scope-838"><a href="#find_all_in_scope-838"><span class="linenos">838</span></a><span class="sd"> expression_types (tuple[type]|type): the expression type(s) to match.</span>
</span><span id="find_all_in_scope-839"><a href="#find_all_in_scope-839"><span class="linenos">839</span></a><span class="sd"> bfs (bool): True to use breadth-first search, False to use depth-first.</span>
</span><span id="find_all_in_scope-840"><a href="#find_all_in_scope-840"><span class="linenos">840</span></a>
</span><span id="find_all_in_scope-841"><a href="#find_all_in_scope-841"><span class="linenos">841</span></a><span class="sd"> Yields:</span>
</span><span id="find_all_in_scope-842"><a href="#find_all_in_scope-842"><span class="linenos">842</span></a><span class="sd"> exp.Expression: nodes</span>
</span><span id="find_all_in_scope-843"><a href="#find_all_in_scope-843"><span class="linenos">843</span></a><span class="sd"> &quot;&quot;&quot;</span>
</span><span id="find_all_in_scope-844"><a href="#find_all_in_scope-844"><span class="linenos">844</span></a> <span class="k">for</span> <span class="n">expression</span> <span class="ow">in</span> <span class="n">walk_in_scope</span><span class="p">(</span><span class="n">expression</span><span class="p">,</span> <span class="n">bfs</span><span class="o">=</span><span class="n">bfs</span><span class="p">):</span>
</span><span id="find_all_in_scope-845"><a href="#find_all_in_scope-845"><span class="linenos">845</span></a> <span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">expression</span><span class="p">,</span> <span class="nb">tuple</span><span class="p">(</span><span class="n">ensure_collection</span><span class="p">(</span><span class="n">expression_types</span><span class="p">))):</span>
</span><span id="find_all_in_scope-846"><a href="#find_all_in_scope-846"><span class="linenos">846</span></a> <span class="k">yield</span> <span class="n">expression</span>
</span><span id="find_all_in_scope-836"><a href="#find_all_in_scope-836"><span class="linenos">836</span></a><span class="sd"> This does NOT traverse into subscopes.</span>
</span><span id="find_all_in_scope-837"><a href="#find_all_in_scope-837"><span class="linenos">837</span></a>
</span><span id="find_all_in_scope-838"><a href="#find_all_in_scope-838"><span class="linenos">838</span></a><span class="sd"> Args:</span>
</span><span id="find_all_in_scope-839"><a href="#find_all_in_scope-839"><span class="linenos">839</span></a><span class="sd"> expression (exp.Expression):</span>
</span><span id="find_all_in_scope-840"><a href="#find_all_in_scope-840"><span class="linenos">840</span></a><span class="sd"> expression_types (tuple[type]|type): the expression type(s) to match.</span>
</span><span id="find_all_in_scope-841"><a href="#find_all_in_scope-841"><span class="linenos">841</span></a><span class="sd"> bfs (bool): True to use breadth-first search, False to use depth-first.</span>
</span><span id="find_all_in_scope-842"><a href="#find_all_in_scope-842"><span class="linenos">842</span></a>
</span><span id="find_all_in_scope-843"><a href="#find_all_in_scope-843"><span class="linenos">843</span></a><span class="sd"> Yields:</span>
</span><span id="find_all_in_scope-844"><a href="#find_all_in_scope-844"><span class="linenos">844</span></a><span class="sd"> exp.Expression: nodes</span>
</span><span id="find_all_in_scope-845"><a href="#find_all_in_scope-845"><span class="linenos">845</span></a><span class="sd"> &quot;&quot;&quot;</span>
</span><span id="find_all_in_scope-846"><a href="#find_all_in_scope-846"><span class="linenos">846</span></a> <span class="k">for</span> <span class="n">expression</span> <span class="ow">in</span> <span class="n">walk_in_scope</span><span class="p">(</span><span class="n">expression</span><span class="p">,</span> <span class="n">bfs</span><span class="o">=</span><span class="n">bfs</span><span class="p">):</span>
</span><span id="find_all_in_scope-847"><a href="#find_all_in_scope-847"><span class="linenos">847</span></a> <span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">expression</span><span class="p">,</span> <span class="nb">tuple</span><span class="p">(</span><span class="n">ensure_collection</span><span class="p">(</span><span class="n">expression_types</span><span class="p">))):</span>
</span><span id="find_all_in_scope-848"><a href="#find_all_in_scope-848"><span class="linenos">848</span></a> <span class="k">yield</span> <span class="n">expression</span>
</span></pre></div>
@ -3211,22 +3213,22 @@ match at least one of the specified expression types.</p>
</div>
<a class="headerlink" href="#find_in_scope"></a>
<div class="pdoc-code codehilite"><pre><span></span><span id="find_in_scope-849"><a href="#find_in_scope-849"><span class="linenos">849</span></a><span class="k">def</span> <span class="nf">find_in_scope</span><span class="p">(</span><span class="n">expression</span><span class="p">,</span> <span class="n">expression_types</span><span class="p">,</span> <span class="n">bfs</span><span class="o">=</span><span class="kc">True</span><span class="p">):</span>
</span><span id="find_in_scope-850"><a href="#find_in_scope-850"><span class="linenos">850</span></a><span class="w"> </span><span class="sd">&quot;&quot;&quot;</span>
</span><span id="find_in_scope-851"><a href="#find_in_scope-851"><span class="linenos">851</span></a><span class="sd"> Returns the first node in this scope which matches at least one of the specified types.</span>
</span><span id="find_in_scope-852"><a href="#find_in_scope-852"><span class="linenos">852</span></a>
</span><span id="find_in_scope-853"><a href="#find_in_scope-853"><span class="linenos">853</span></a><span class="sd"> This does NOT traverse into subscopes.</span>
<div class="pdoc-code codehilite"><pre><span></span><span id="find_in_scope-851"><a href="#find_in_scope-851"><span class="linenos">851</span></a><span class="k">def</span> <span class="nf">find_in_scope</span><span class="p">(</span><span class="n">expression</span><span class="p">,</span> <span class="n">expression_types</span><span class="p">,</span> <span class="n">bfs</span><span class="o">=</span><span class="kc">True</span><span class="p">):</span>
</span><span id="find_in_scope-852"><a href="#find_in_scope-852"><span class="linenos">852</span></a><span class="w"> </span><span class="sd">&quot;&quot;&quot;</span>
</span><span id="find_in_scope-853"><a href="#find_in_scope-853"><span class="linenos">853</span></a><span class="sd"> Returns the first node in this scope which matches at least one of the specified types.</span>
</span><span id="find_in_scope-854"><a href="#find_in_scope-854"><span class="linenos">854</span></a>
</span><span id="find_in_scope-855"><a href="#find_in_scope-855"><span class="linenos">855</span></a><span class="sd"> Args:</span>
</span><span id="find_in_scope-856"><a href="#find_in_scope-856"><span class="linenos">856</span></a><span class="sd"> expression (exp.Expression):</span>
</span><span id="find_in_scope-857"><a href="#find_in_scope-857"><span class="linenos">857</span></a><span class="sd"> expression_types (tuple[type]|type): the expression type(s) to match.</span>
</span><span id="find_in_scope-858"><a href="#find_in_scope-858"><span class="linenos">858</span></a><span class="sd"> bfs (bool): True to use breadth-first search, False to use depth-first.</span>
</span><span id="find_in_scope-859"><a href="#find_in_scope-859"><span class="linenos">859</span></a>
</span><span id="find_in_scope-860"><a href="#find_in_scope-860"><span class="linenos">860</span></a><span class="sd"> Returns:</span>
</span><span id="find_in_scope-861"><a href="#find_in_scope-861"><span class="linenos">861</span></a><span class="sd"> exp.Expression: the node which matches the criteria or None if no node matching</span>
</span><span id="find_in_scope-862"><a href="#find_in_scope-862"><span class="linenos">862</span></a><span class="sd"> the criteria was found.</span>
</span><span id="find_in_scope-863"><a href="#find_in_scope-863"><span class="linenos">863</span></a><span class="sd"> &quot;&quot;&quot;</span>
</span><span id="find_in_scope-864"><a href="#find_in_scope-864"><span class="linenos">864</span></a> <span class="k">return</span> <span class="nb">next</span><span class="p">(</span><span class="n">find_all_in_scope</span><span class="p">(</span><span class="n">expression</span><span class="p">,</span> <span class="n">expression_types</span><span class="p">,</span> <span class="n">bfs</span><span class="o">=</span><span class="n">bfs</span><span class="p">),</span> <span class="kc">None</span><span class="p">)</span>
</span><span id="find_in_scope-855"><a href="#find_in_scope-855"><span class="linenos">855</span></a><span class="sd"> This does NOT traverse into subscopes.</span>
</span><span id="find_in_scope-856"><a href="#find_in_scope-856"><span class="linenos">856</span></a>
</span><span id="find_in_scope-857"><a href="#find_in_scope-857"><span class="linenos">857</span></a><span class="sd"> Args:</span>
</span><span id="find_in_scope-858"><a href="#find_in_scope-858"><span class="linenos">858</span></a><span class="sd"> expression (exp.Expression):</span>
</span><span id="find_in_scope-859"><a href="#find_in_scope-859"><span class="linenos">859</span></a><span class="sd"> expression_types (tuple[type]|type): the expression type(s) to match.</span>
</span><span id="find_in_scope-860"><a href="#find_in_scope-860"><span class="linenos">860</span></a><span class="sd"> bfs (bool): True to use breadth-first search, False to use depth-first.</span>
</span><span id="find_in_scope-861"><a href="#find_in_scope-861"><span class="linenos">861</span></a>
</span><span id="find_in_scope-862"><a href="#find_in_scope-862"><span class="linenos">862</span></a><span class="sd"> Returns:</span>
</span><span id="find_in_scope-863"><a href="#find_in_scope-863"><span class="linenos">863</span></a><span class="sd"> exp.Expression: the node which matches the criteria or None if no node matching</span>
</span><span id="find_in_scope-864"><a href="#find_in_scope-864"><span class="linenos">864</span></a><span class="sd"> the criteria was found.</span>
</span><span id="find_in_scope-865"><a href="#find_in_scope-865"><span class="linenos">865</span></a><span class="sd"> &quot;&quot;&quot;</span>
</span><span id="find_in_scope-866"><a href="#find_in_scope-866"><span class="linenos">866</span></a> <span class="k">return</span> <span class="nb">next</span><span class="p">(</span><span class="n">find_all_in_scope</span><span class="p">(</span><span class="n">expression</span><span class="p">,</span> <span class="n">expression_types</span><span class="p">,</span> <span class="n">bfs</span><span class="o">=</span><span class="n">bfs</span><span class="p">),</span> <span class="kc">None</span><span class="p">)</span>
</span></pre></div>

4
docs/sqlglot/optimizer/simplify.html
View file

@ -3231,7 +3231,7 @@ prefix are statically known.</p>
<div class="attr variable">
<span class="name">DATETRUNC_COMPARISONS</span> =
<input id="DATETRUNC_COMPARISONS-view-value" class="view-value-toggle-state" type="checkbox" aria-hidden="true" tabindex="-1">
<label class="view-value-button pdoc-button" for="DATETRUNC_COMPARISONS-view-value"></label><span class="default_value">{&lt;class &#39;<a href="../expressions.html#LT">sqlglot.expressions.LT</a>&#39;&gt;, &lt;class &#39;<a href="../expressions.html#EQ">sqlglot.expressions.EQ</a>&#39;&gt;, &lt;class &#39;<a href="../expressions.html#LTE">sqlglot.expressions.LTE</a>&#39;&gt;, &lt;class &#39;<a href="../expressions.html#GTE">sqlglot.expressions.GTE</a>&#39;&gt;, &lt;class &#39;<a href="../expressions.html#NEQ">sqlglot.expressions.NEQ</a>&#39;&gt;, &lt;class &#39;<a href="../expressions.html#In">sqlglot.expressions.In</a>&#39;&gt;, &lt;class &#39;<a href="../expressions.html#GT">sqlglot.expressions.GT</a>&#39;&gt;}</span>
<label class="view-value-button pdoc-button" for="DATETRUNC_COMPARISONS-view-value"></label><span class="default_value">{&lt;class &#39;<a href="../expressions.html#GTE">sqlglot.expressions.GTE</a>&#39;&gt;, &lt;class &#39;<a href="../expressions.html#LT">sqlglot.expressions.LT</a>&#39;&gt;, &lt;class &#39;<a href="../expressions.html#LTE">sqlglot.expressions.LTE</a>&#39;&gt;, &lt;class &#39;<a href="../expressions.html#EQ">sqlglot.expressions.EQ</a>&#39;&gt;, &lt;class &#39;<a href="../expressions.html#In">sqlglot.expressions.In</a>&#39;&gt;, &lt;class &#39;<a href="../expressions.html#NEQ">sqlglot.expressions.NEQ</a>&#39;&gt;, &lt;class &#39;<a href="../expressions.html#GT">sqlglot.expressions.GT</a>&#39;&gt;}</span>
</div>
@ -3315,7 +3315,7 @@ prefix are statically known.</p>
<section id="JOINS">
<div class="attr variable">
<span class="name">JOINS</span> =
<span class="default_value">{(&#39;RIGHT&#39;, &#39;&#39;), (&#39;&#39;, &#39;&#39;), (&#39;&#39;, &#39;INNER&#39;), (&#39;RIGHT&#39;, &#39;OUTER&#39;)}</span>
<span class="default_value">{(&#39;&#39;, &#39;&#39;), (&#39;RIGHT&#39;, &#39;&#39;), (&#39;RIGHT&#39;, &#39;OUTER&#39;), (&#39;&#39;, &#39;INNER&#39;)}</span>
</div>

27994
docs/sqlglot/parser.html
View file

File diff suppressed because one or more lines are too long

2
docs/sqlglot/time.html
View file

File diff suppressed because one or more lines are too long

5339
docs/sqlglot/tokens.html
View file

File diff suppressed because one or more lines are too long

2
setup.py
View file

@ -37,7 +37,7 @@ setup(
"pytz",
"pdoc",
"pre-commit",
"ruff==0.4.3",
"ruff==0.7.2",
"types-python-dateutil",
"types-pytz",
"typing_extensions",

53
sqlglot/dialects/bigquery.py
View file

@ -231,6 +231,13 @@ def _build_regexp_extract(args: t.List) -> exp.RegexpExtract:
)
def _build_json_extract_scalar(args: t.List, dialect: Dialect) -> exp.JSONExtractScalar:
if len(args) == 1:
# The default value for the JSONPath is '$' i.e all of the data
args.append(exp.Literal.string("$"))
return parser.build_extract_json_with_path(exp.JSONExtractScalar)(args, dialect)
def _str_to_datetime_sql(
self: BigQuery.Generator, expression: exp.StrToDate | exp.StrToTime
) -> str:
@ -420,18 +427,22 @@ class BigQuery(Dialect):
"DATE_TRUNC": lambda args: exp.DateTrunc(
unit=exp.Literal.string(str(seq_get(args, 1))),
this=seq_get(args, 0),
zone=seq_get(args, 2),
),
"DATETIME": _build_datetime,
"DATETIME_ADD": build_date_delta_with_interval(exp.DatetimeAdd),
"DATETIME_SUB": build_date_delta_with_interval(exp.DatetimeSub),
"DIV": binary_from_function(exp.IntDiv),
"EDIT_DISTANCE": lambda args: exp.Levenshtein(
this=seq_get(args, 0), expression=seq_get(args, 1), max_dist=seq_get(args, 2)
),
"FORMAT_DATE": lambda args: exp.TimeToStr(
this=exp.TsOrDsToDate(this=seq_get(args, 1)), format=seq_get(args, 0)
),
"GENERATE_ARRAY": exp.GenerateSeries.from_arg_list,
"JSON_EXTRACT_SCALAR": lambda args: exp.JSONExtractScalar(
this=seq_get(args, 0), expression=seq_get(args, 1) or exp.Literal.string("$")
),
"JSON_EXTRACT_SCALAR": _build_json_extract_scalar,
"JSON_QUERY": parser.build_extract_json_with_path(exp.JSONExtract),
"JSON_VALUE": _build_json_extract_scalar,
"LENGTH": lambda args: exp.Length(this=seq_get(args, 0), binary=True),
"MD5": exp.MD5Digest.from_arg_list,
"TO_HEX": _build_to_hex,
@ -565,6 +576,7 @@ class BigQuery(Dialect):
table.set("this", exp.Identifier(this=parts[1]))
if isinstance(table.this, exp.Identifier) and any("." in p.name for p in table.parts):
alias = table.this
catalog, db, this, *rest = (
exp.to_identifier(p, quoted=True)
for p in split_num_words(".".join(p.name for p in table.parts), ".", 3)
@ -577,6 +589,36 @@ class BigQuery(Dialect):
this=this, db=db, catalog=catalog, pivots=table.args.get("pivots")
)
table.meta["quoted_table"] = True
else:
alias = None
# The `INFORMATION_SCHEMA` views in BigQuery need to be qualified by a region or
# dataset, so if the project identifier is omitted we need to fix the ast so that
# the `INFORMATION_SCHEMA.X` bit is represented as a single (quoted) Identifier.
# Otherwise, we wouldn't correctly qualify a `Table` node that references these
# views, because it would seem like the "catalog" part is set, when it'd actually
# be the region/dataset. Merging the two identifiers into a single one is done to
# avoid producing a 4-part Table reference, which would cause issues in the schema
# module, when there are 3-part table names mixed with information schema views.
#
# See: https://cloud.google.com/bigquery/docs/information-schema-intro#syntax
table_parts = table.parts
if len(table_parts) > 1 and table_parts[-2].name.upper() == "INFORMATION_SCHEMA":
# We need to alias the table here to avoid breaking existing qualified columns.
# This is expected to be safe, because if there's an actual alias coming up in
# the token stream, it will overwrite this one. If there isn't one, we are only
# exposing the name that can be used to reference the view explicitly (a no-op).
exp.alias_(
table,
t.cast(exp.Identifier, alias or table_parts[-1]),
table=True,
copy=False,
)
info_schema_view = f"{table_parts[-2].name}.{table_parts[-1].name}"
table.set("this", exp.Identifier(this=info_schema_view, quoted=True))
table.set("db", seq_get(table_parts, -3))
table.set("catalog", seq_get(table_parts, -4))
return table
@ -722,7 +764,9 @@ class BigQuery(Dialect):
exp.DateSub: date_add_interval_sql("DATE", "SUB"),
exp.DatetimeAdd: date_add_interval_sql("DATETIME", "ADD"),
exp.DatetimeSub: date_add_interval_sql("DATETIME", "SUB"),
exp.DateTrunc: lambda self, e: self.func("DATE_TRUNC", e.this, e.text("unit")),
exp.DateTrunc: lambda self, e: self.func(
"DATE_TRUNC", e.this, e.text("unit"), e.args.get("zone")
),
exp.FromTimeZone: lambda self, e: self.func(
"DATETIME", self.func("TIMESTAMP", e.this, e.args.get("zone")), "'UTC'"
),
@ -764,6 +808,7 @@ class BigQuery(Dialect):
exp.StabilityProperty: lambda self, e: (
"DETERMINISTIC" if e.name == "IMMUTABLE" else "NOT DETERMINISTIC"
),
exp.String: rename_func("STRING"),
exp.StrToDate: _str_to_datetime_sql,
exp.StrToTime: _str_to_datetime_sql,
exp.TimeAdd: date_add_interval_sql("TIME", "ADD"),

92
sqlglot/dialects/clickhouse.py
View file

@ -1,8 +1,6 @@
from __future__ import annotations
import typing as t
import datetime
from sqlglot import exp, generator, parser, tokens
from sqlglot.dialects.dialect import (
Dialect,
@ -108,24 +106,54 @@ def _datetime_delta_sql(name: str) -> t.Callable[[Generator, DATEΤΙΜΕ_DELTA]
def _timestrtotime_sql(self: ClickHouse.Generator, expression: exp.TimeStrToTime):
tz = expression.args.get("zone")
datatype = exp.DataType.build(exp.DataType.Type.TIMESTAMP)
ts = expression.this
if tz:
# build a datatype that encodes the timezone as a type parameter, eg DateTime('America/Los_Angeles')
datatype = exp.DataType.build(
exp.DataType.Type.TIMESTAMPTZ, # Type.TIMESTAMPTZ maps to DateTime
expressions=[exp.DataTypeParam(this=tz)],
tz = expression.args.get("zone")
if tz and isinstance(ts, exp.Literal):
# Clickhouse will not accept timestamps that include a UTC offset, so we must remove them.
# The first step to removing is parsing the string with `datetime.datetime.fromisoformat`.
#
# In python <3.11, `fromisoformat()` can only parse timestamps of millisecond (3 digit)
# or microsecond (6 digit) precision. It will error if passed any other number of fractional
# digits, so we extract the fractional seconds and pad to 6 digits before parsing.
ts_string = ts.name.strip()
# separate [date and time] from [fractional seconds and UTC offset]
ts_parts = ts_string.split(".")
if len(ts_parts) == 2:
# separate fractional seconds and UTC offset
offset_sep = "+" if "+" in ts_parts[1] else "-"
ts_frac_parts = ts_parts[1].split(offset_sep)
num_frac_parts = len(ts_frac_parts)
# pad to 6 digits if fractional seconds present
ts_frac_parts[0] = ts_frac_parts[0].ljust(6, "0")
ts_string = "".join(
[
ts_parts[0], # date and time
".",
ts_frac_parts[0], # fractional seconds
offset_sep if num_frac_parts > 1 else "",
ts_frac_parts[1] if num_frac_parts > 1 else "", # utc offset (if present)
]
)
if isinstance(ts, exp.Literal):
# strip the timezone out of the literal, eg turn '2020-01-01 12:13:14-08:00' into '2020-01-01 12:13:14'
# this is because Clickhouse encodes the timezone as a data type parameter and throws an error if it's part of the timestamp string
# return literal with no timezone, eg turn '2020-01-01 12:13:14-08:00' into '2020-01-01 12:13:14'
# this is because Clickhouse encodes the timezone as a data type parameter and throws an error if
# it's part of the timestamp string
ts_without_tz = (
datetime.datetime.fromisoformat(ts.name).replace(tzinfo=None).isoformat(sep=" ")
datetime.datetime.fromisoformat(ts_string).replace(tzinfo=None).isoformat(sep=" ")
)
ts = exp.Literal.string(ts_without_tz)
# Non-nullable DateTime64 with microsecond precision
expressions = [exp.DataTypeParam(this=tz)] if tz else []
datatype = exp.DataType.build(
exp.DataType.Type.DATETIME64,
expressions=[exp.DataTypeParam(this=exp.Literal.number(6)), *expressions],
nullable=False,
)
return self.sql(exp.cast(ts, datatype, dialect=self.dialect))
@ -155,6 +183,7 @@ class ClickHouse(Dialect):
class Tokenizer(tokens.Tokenizer):
COMMENTS = ["--", "#", "#!", ("/*", "*/")]
IDENTIFIERS = ['"', "`"]
IDENTIFIER_ESCAPES = ["\\"]
STRING_ESCAPES = ["'", "\\"]
BIT_STRINGS = [("0b", "")]
HEX_STRINGS = [("0x", ""), ("0X", "")]
@ -187,6 +216,12 @@ class ClickHouse(Dialect):
"UINT8": TokenType.UTINYINT,
"IPV4": TokenType.IPV4,
"IPV6": TokenType.IPV6,
"POINT": TokenType.POINT,
"RING": TokenType.RING,
"LINESTRING": TokenType.LINESTRING,
"MULTILINESTRING": TokenType.MULTILINESTRING,
"POLYGON": TokenType.POLYGON,
"MULTIPOLYGON": TokenType.MULTIPOLYGON,
"AGGREGATEFUNCTION": TokenType.AGGREGATEFUNCTION,
"SIMPLEAGGREGATEFUNCTION": TokenType.SIMPLEAGGREGATEFUNCTION,
"SYSTEM": TokenType.COMMAND,
@ -396,6 +431,7 @@ class ClickHouse(Dialect):
**parser.Parser.FUNCTION_PARSERS,
"ARRAYJOIN": lambda self: self.expression(exp.Explode, this=self._parse_expression()),
"QUANTILE": lambda self: self._parse_quantile(),
"MEDIAN": lambda self: self._parse_quantile(),
"COLUMNS": lambda self: self._parse_columns(),
}
@ -469,6 +505,10 @@ class ClickHouse(Dialect):
TokenType.L_BRACE: lambda self: self._parse_query_parameter(),
}
# https://clickhouse.com/docs/en/sql-reference/statements/create/function
def _parse_user_defined_function_expression(self) -> t.Optional[exp.Expression]:
return self._parse_lambda()
def _parse_types(
self, check_func: bool = False, schema: bool = False, allow_identifiers: bool = True
) -> t.Optional[exp.Expression]:
@ -839,6 +879,11 @@ class ClickHouse(Dialect):
exp.DataType.Type.DATE32: "Date32",
exp.DataType.Type.DATETIME: "DateTime",
exp.DataType.Type.DATETIME64: "DateTime64",
exp.DataType.Type.DECIMAL: "Decimal",
exp.DataType.Type.DECIMAL32: "Decimal32",
exp.DataType.Type.DECIMAL64: "Decimal64",
exp.DataType.Type.DECIMAL128: "Decimal128",
exp.DataType.Type.DECIMAL256: "Decimal256",
exp.DataType.Type.TIMESTAMP: "DateTime",
exp.DataType.Type.TIMESTAMPTZ: "DateTime",
exp.DataType.Type.DOUBLE: "Float64",
@ -865,6 +910,12 @@ class ClickHouse(Dialect):
exp.DataType.Type.UTINYINT: "UInt8",
exp.DataType.Type.IPV4: "IPv4",
exp.DataType.Type.IPV6: "IPv6",
exp.DataType.Type.POINT: "Point",
exp.DataType.Type.RING: "Ring",
exp.DataType.Type.LINESTRING: "LineString",
exp.DataType.Type.MULTILINESTRING: "MultiLineString",
exp.DataType.Type.POLYGON: "Polygon",
exp.DataType.Type.MULTIPOLYGON: "MultiPolygon",
exp.DataType.Type.AGGREGATEFUNCTION: "AggregateFunction",
exp.DataType.Type.SIMPLEAGGREGATEFUNCTION: "SimpleAggregateFunction",
}
@ -961,6 +1012,12 @@ class ClickHouse(Dialect):
exp.DataType.Type.ARRAY,
exp.DataType.Type.MAP,
exp.DataType.Type.STRUCT,
exp.DataType.Type.POINT,
exp.DataType.Type.RING,
exp.DataType.Type.LINESTRING,
exp.DataType.Type.MULTILINESTRING,
exp.DataType.Type.POLYGON,
exp.DataType.Type.MULTIPOLYGON,
}
def strtodate_sql(self, expression: exp.StrToDate) -> str:
@ -1162,3 +1219,12 @@ class ClickHouse(Dialect):
def projectiondef_sql(self, expression: exp.ProjectionDef) -> str:
return f"PROJECTION {self.sql(expression.this)} {self.wrap(expression.expression)}"
def is_sql(self, expression: exp.Is) -> str:
is_sql = super().is_sql(expression)
if isinstance(expression.parent, exp.Not):
# value IS NOT NULL -> NOT (value IS NULL)
is_sql = self.wrap(is_sql)
return is_sql

12
sqlglot/dialects/databricks.py
View file

@ -7,6 +7,7 @@ from sqlglot.dialects.dialect import (
date_delta_sql,
build_date_delta,
timestamptrunc_sql,
timestampdiff_sql,
)
from sqlglot.dialects.spark import Spark
from sqlglot.tokens import TokenType
@ -19,12 +20,6 @@ def _build_json_extract(args: t.List) -> exp.JSONExtract:
return exp.JSONExtract(this=this, expression=path)
def _timestamp_diff(
self: Databricks.Generator, expression: exp.DatetimeDiff | exp.TimestampDiff
) -> str:
return self.func("TIMESTAMPDIFF", expression.unit, expression.expression, expression.this)
def _jsonextract_sql(
self: Databricks.Generator, expression: exp.JSONExtract | exp.JSONExtractScalar
) -> str:
@ -80,13 +75,14 @@ class Databricks(Spark):
exp.Mul(this=e.expression, expression=exp.Literal.number(-1)),
e.this,
),
exp.DatetimeDiff: _timestamp_diff,
exp.TimestampDiff: _timestamp_diff,
exp.DatetimeDiff: timestampdiff_sql,
exp.TimestampDiff: timestampdiff_sql,
exp.DatetimeTrunc: timestamptrunc_sql(),
exp.Select: transforms.preprocess(
[
transforms.eliminate_distinct_on,
transforms.unnest_to_explode,
transforms.any_to_exists,
]
),
exp.JSONExtract: _jsonextract_sql,

10
sqlglot/dialects/dialect.py
View file

@ -637,6 +637,7 @@ class Dialect(metaclass=_Dialect):
exp.Initcap,
exp.Lower,
exp.Substring,
exp.String,
exp.TimeToStr,
exp.TimeToTimeStr,
exp.Trim,
@ -1016,10 +1017,10 @@ def no_recursive_cte_sql(self: Generator, expression: exp.With) -> str:
return self.with_sql(expression)
def no_safe_divide_sql(self: Generator, expression: exp.SafeDivide) -> str:
def no_safe_divide_sql(self: Generator, expression: exp.SafeDivide, if_sql: str = "IF") -> str:
n = self.sql(expression, "this")
d = self.sql(expression, "expression")
return f"IF(({d}) <> 0, ({n}) / ({d}), NULL)"
return f"{if_sql}(({d}) <> 0, ({n}) / ({d}), NULL)"
def no_tablesample_sql(self: Generator, expression: exp.TableSample) -> str:
@ -1697,6 +1698,7 @@ def build_regexp_extract(args: t.List, dialect: Dialect) -> exp.RegexpExtract:
this=seq_get(args, 0),
expression=seq_get(args, 1),
group=seq_get(args, 2) or exp.Literal.number(dialect.REGEXP_EXTRACT_DEFAULT_GROUP),
parameters=seq_get(args, 3),
)
@ -1713,3 +1715,7 @@ def explode_to_unnest_sql(self: Generator, expression: exp.Lateral) -> str:
)
)
return self.lateral_sql(expression)
def timestampdiff_sql(self: Generator, expression: exp.DatetimeDiff | exp.TimestampDiff) -> str:
return self.func("TIMESTAMPDIFF", expression.unit, expression.expression, expression.this)

26
sqlglot/dialects/duckdb.py
View file

@ -25,7 +25,6 @@ from sqlglot.dialects.dialect import (
no_time_sql,
no_timestamp_sql,
pivot_column_names,
regexp_extract_sql,
rename_func,
str_position_sql,
str_to_time_sql,
@ -37,6 +36,7 @@ from sqlglot.dialects.dialect import (
build_regexp_extract,
explode_to_unnest_sql,
)
from sqlglot.generator import unsupported_args
from sqlglot.helper import seq_get
from sqlglot.tokens import TokenType
from sqlglot.parser import binary_range_parser
@ -104,7 +104,7 @@ def _timediff_sql(self: DuckDB.Generator, expression: exp.TimeDiff) -> str:
return self.func("DATE_DIFF", unit_to_str(expression), expr, this)
@generator.unsupported_args(("expression", "DuckDB's ARRAY_SORT does not support a comparator."))
@unsupported_args(("expression", "DuckDB's ARRAY_SORT does not support a comparator."))
def _array_sort_sql(self: DuckDB.Generator, expression: exp.ArraySort) -> str:
return self.func("ARRAY_SORT", expression.this)
@ -374,9 +374,6 @@ class DuckDB(Dialect):
"LIST_VALUE": lambda args: exp.Array(expressions=args),
"MAKE_TIME": exp.TimeFromParts.from_arg_list,
"MAKE_TIMESTAMP": _build_make_timestamp,
"MEDIAN": lambda args: exp.PercentileCont(
this=seq_get(args, 0), expression=exp.Literal.number(0.5)
),
"QUANTILE_CONT": exp.PercentileCont.from_arg_list,
"QUANTILE_DISC": exp.PercentileDisc.from_arg_list,
"REGEXP_EXTRACT": build_regexp_extract,
@ -536,6 +533,7 @@ class DuckDB(Dialect):
exp.IntDiv: lambda self, e: self.binary(e, "//"),
exp.IsInf: rename_func("ISINF"),
exp.IsNan: rename_func("ISNAN"),
exp.JSONBExists: rename_func("JSON_EXISTS"),
exp.JSONExtract: _arrow_json_extract_sql,
exp.JSONExtractScalar: _arrow_json_extract_sql,
exp.JSONFormat: _json_format_sql,
@ -554,7 +552,6 @@ class DuckDB(Dialect):
# DuckDB doesn't allow qualified columns inside of PIVOT expressions.
# See: https://github.com/duckdb/duckdb/blob/671faf92411182f81dce42ac43de8bfb05d9909e/src/planner/binder/tableref/bind_pivot.cpp#L61-L62
exp.Pivot: transforms.preprocess([transforms.unqualify_columns]),
exp.RegexpExtract: regexp_extract_sql,
exp.RegexpReplace: lambda self, e: self.func(
"REGEXP_REPLACE",
e.this,
@ -956,3 +953,20 @@ class DuckDB(Dialect):
this = f"LIST_TRANSFORM({this}, x -> COALESCE(x, {null_text}))"
return self.func("ARRAY_TO_STRING", this, expression.expression)
@unsupported_args("position", "occurrence")
def regexpextract_sql(self, expression: exp.RegexpExtract) -> str:
group = expression.args.get("group")
params = expression.args.get("parameters")
# Do not render group if there is no following argument,
# and it's the default value for this dialect
if (
not params
and group
and group.name == str(self.dialect.REGEXP_EXTRACT_DEFAULT_GROUP)
):
group = None
return self.func(
"REGEXP_EXTRACT", expression.this, expression.expression, group, params
)

8
sqlglot/dialects/hive.py
View file

@ -460,6 +460,7 @@ class Hive(Dialect):
WITH_PROPERTIES_PREFIX = "TBLPROPERTIES"
PARSE_JSON_NAME = None
PAD_FILL_PATTERN_IS_REQUIRED = True
SUPPORTS_MEDIAN = False
EXPRESSIONS_WITHOUT_NESTED_CTES = {
exp.Insert,
@ -556,6 +557,7 @@ class Hive(Dialect):
transforms.eliminate_qualify,
transforms.eliminate_distinct_on,
partial(transforms.unnest_to_explode, unnest_using_arrays_zip=False),
transforms.any_to_exists,
]
),
exp.StrPosition: strposition_to_locate_sql,
@ -709,3 +711,9 @@ class Hive(Dialect):
exprs = self.expressions(expression, flat=True)
return f"{prefix}SERDEPROPERTIES ({exprs})"
def exists_sql(self, expression: exp.Exists):
if expression.expression:
return self.function_fallback_sql(expression)
return super().exists_sql(expression)

1
sqlglot/dialects/mysql.py
View file

@ -704,6 +704,7 @@ class MySQL(Dialect):
PAD_FILL_PATTERN_IS_REQUIRED = True
WRAP_DERIVED_VALUES = False
VARCHAR_REQUIRES_SIZE = True
SUPPORTS_MEDIAN = False
TRANSFORMS = {
**generator.Generator.TRANSFORMS,

4
sqlglot/dialects/oracle.py
View file

@ -121,7 +121,9 @@ class Oracle(Dialect):
"TO_TIMESTAMP": build_formatted_time(exp.StrToTime, "oracle"),
"TO_DATE": build_formatted_time(exp.StrToDate, "oracle"),
"TRUNC": lambda args: exp.DateTrunc(
unit=seq_get(args, 1) or exp.Literal.string("DD"), this=seq_get(args, 0)
unit=seq_get(args, 1) or exp.Literal.string("DD"),
this=seq_get(args, 0),
unabbreviate=False,
),
}

10
sqlglot/dialects/postgres.py
View file

@ -370,6 +370,7 @@ class Postgres(Dialect):
FUNCTION_PARSERS = {
**parser.Parser.FUNCTION_PARSERS,
"DATE_PART": lambda self: self._parse_date_part(),
"JSONB_EXISTS": lambda self: self._parse_jsonb_exists(),
}
BITWISE = {
@ -443,6 +444,14 @@ class Postgres(Dialect):
def _parse_unique_key(self) -> t.Optional[exp.Expression]:
return None
def _parse_jsonb_exists(self) -> exp.JSONBExists:
return self.expression(
exp.JSONBExists,
this=self._parse_bitwise(),
path=self._match(TokenType.COMMA)
and self.dialect.to_json_path(self._parse_bitwise()),
)
class Generator(generator.Generator):
SINGLE_STRING_INTERVAL = True
RENAME_TABLE_WITH_DB = False
@ -462,6 +471,7 @@ class Postgres(Dialect):
CAN_IMPLEMENT_ARRAY_ANY = True
COPY_HAS_INTO_KEYWORD = False
ARRAY_CONCAT_IS_VAR_LEN = False
SUPPORTS_MEDIAN = False
SUPPORTED_JSON_PATH_PARTS = {
exp.JSONPathKey,

3
sqlglot/dialects/presto.py
View file

@ -317,6 +317,7 @@ class Presto(Dialect):
PARSE_JSON_NAME = "JSON_PARSE"
PAD_FILL_PATTERN_IS_REQUIRED = True
EXCEPT_INTERSECT_SUPPORT_ALL_CLAUSE = False
SUPPORTS_MEDIAN = False
PROPERTIES_LOCATION = {
**generator.Generator.PROPERTIES_LOCATION,
@ -336,6 +337,7 @@ class Presto(Dialect):
exp.DataType.Type.STRUCT: "ROW",
exp.DataType.Type.TEXT: "VARCHAR",
exp.DataType.Type.TIMESTAMPTZ: "TIMESTAMP",
exp.DataType.Type.TIMESTAMPNTZ: "TIMESTAMP",
exp.DataType.Type.TIMETZ: "TIME",
}
@ -364,6 +366,7 @@ class Presto(Dialect):
),
exp.BitwiseXor: lambda self, e: self.func("BITWISE_XOR", e.this, e.expression),
exp.Cast: transforms.preprocess([transforms.epoch_cast_to_ts]),
exp.CurrentTime: lambda *_: "CURRENT_TIME",
exp.CurrentTimestamp: lambda *_: "CURRENT_TIMESTAMP",
exp.DateAdd: _date_delta_sql("DATE_ADD"),
exp.DateDiff: lambda self, e: self.func(

1
sqlglot/dialects/redshift.py
View file

@ -157,6 +157,7 @@ class Redshift(Postgres):
ARRAY_CONCAT_IS_VAR_LEN = False
SUPPORTS_CONVERT_TIMEZONE = True
EXCEPT_INTERSECT_SUPPORT_ALL_CLAUSE = False
SUPPORTS_MEDIAN = True
# Redshift doesn't have `WITH` as part of their with_properties so we remove it
WITH_PROPERTIES_PREFIX = " "

38
sqlglot/dialects/snowflake.py
View file

@ -22,7 +22,11 @@ from sqlglot.dialects.dialect import (
timestrtotime_sql,
var_map_sql,
map_date_part,
no_safe_divide_sql,
no_timestamp_sql,
timestampdiff_sql,
)
from sqlglot.generator import unsupported_args
from sqlglot.helper import flatten, is_float, is_int, seq_get
from sqlglot.tokens import TokenType
@ -329,9 +333,6 @@ class Snowflake(Dialect):
"LEN": lambda args: exp.Length(this=seq_get(args, 0), binary=True),
"LENGTH": lambda args: exp.Length(this=seq_get(args, 0), binary=True),
"LISTAGG": exp.GroupConcat.from_arg_list,
"MEDIAN": lambda args: exp.PercentileCont(
this=seq_get(args, 0), expression=exp.Literal.number(0.5)
),
"NULLIFZERO": _build_if_from_nullifzero,
"OBJECT_CONSTRUCT": _build_object_construct,
"REGEXP_REPLACE": _build_regexp_replace,
@ -351,6 +352,8 @@ class Snowflake(Dialect):
"TIMESTAMPDIFF": _build_datediff,
"TIMESTAMPFROMPARTS": build_timestamp_from_parts,
"TIMESTAMP_FROM_PARTS": build_timestamp_from_parts,
"TIMESTAMPNTZFROMPARTS": build_timestamp_from_parts,
"TIMESTAMP_NTZ_FROM_PARTS": build_timestamp_from_parts,
"TRY_PARSE_JSON": lambda args: exp.ParseJSON(this=seq_get(args, 0), safe=True),
"TRY_TO_DATE": _build_datetime("TRY_TO_DATE", exp.DataType.Type.DATE, safe=True),
"TRY_TO_TIMESTAMP": _build_datetime(
@ -766,6 +769,7 @@ class Snowflake(Dialect):
ARRAY_CONCAT_IS_VAR_LEN = False
SUPPORTS_CONVERT_TIMEZONE = True
EXCEPT_INTERSECT_SUPPORT_ALL_CLAUSE = False
SUPPORTS_MEDIAN = True
TRANSFORMS = {
**generator.Generator.TRANSFORMS,
@ -782,6 +786,8 @@ class Snowflake(Dialect):
exp.Create: transforms.preprocess([_flatten_structured_types_unless_iceberg]),
exp.DateAdd: date_delta_sql("DATEADD"),
exp.DateDiff: date_delta_sql("DATEDIFF"),
exp.DatetimeAdd: date_delta_sql("TIMESTAMPADD"),
exp.DatetimeDiff: timestampdiff_sql,
exp.DateStrToDate: datestrtodate_sql,
exp.DayOfMonth: rename_func("DAYOFMONTH"),
exp.DayOfWeek: rename_func("DAYOFWEEK"),
@ -796,7 +802,6 @@ class Snowflake(Dialect):
),
exp.GroupConcat: rename_func("LISTAGG"),
exp.If: if_sql(name="IFF", false_value="NULL"),
exp.JSONExtract: lambda self, e: self.func("GET_PATH", e.this, e.expression),
exp.JSONExtractScalar: lambda self, e: self.func(
"JSON_EXTRACT_PATH_TEXT", e.this, e.expression
),
@ -828,14 +833,18 @@ class Snowflake(Dialect):
_unnest_generate_date_array,
]
),
exp.SafeDivide: lambda self, e: no_safe_divide_sql(self, e, "IFF"),
exp.SHA: rename_func("SHA1"),
exp.StarMap: rename_func("OBJECT_CONSTRUCT"),
exp.StartsWith: rename_func("STARTSWITH"),
exp.StrPosition: lambda self, e: self.func(
"POSITION", e.args.get("substr"), e.this, e.args.get("position")
),
exp.StrToDate: lambda self, e: self.func("DATE", e.this, self.format_time(e)),
exp.Stuff: rename_func("INSERT"),
exp.TimeAdd: date_delta_sql("TIMEADD"),
exp.Timestamp: no_timestamp_sql,
exp.TimestampAdd: date_delta_sql("TIMESTAMPADD"),
exp.TimestampDiff: lambda self, e: self.func(
"TIMESTAMPDIFF", e.unit, e.expression, e.this
),
@ -1061,7 +1070,7 @@ class Snowflake(Dialect):
return self.func("OBJECT_CONSTRUCT", *flatten(zip(keys, values)))
@generator.unsupported_args("weight", "accuracy")
@unsupported_args("weight", "accuracy")
def approxquantile_sql(self, expression: exp.ApproxQuantile) -> str:
return self.func("APPROX_PERCENTILE", expression.this, expression.args.get("quantile"))
@ -1082,3 +1091,22 @@ class Snowflake(Dialect):
return self.func(
f"{safe_prefix}TO_TIMESTAMP", expression.this, self.format_time(expression)
)
def timestampsub_sql(self, expression: exp.TimestampSub):
return self.sql(
exp.TimestampAdd(
this=expression.this,
expression=expression.expression * -1,
unit=expression.unit,
)
)
def jsonextract_sql(self, expression: exp.JSONExtract):
this = expression.this
# JSON strings are valid coming from other dialects such as BQ
return self.func(
"GET_PATH",
exp.ParseJSON(this=this) if this.is_string else this,
expression.expression,
)

1
sqlglot/dialects/spark.py
View file

@ -136,6 +136,7 @@ class Spark(Spark2):
SUPPORTS_TO_NUMBER = True
PAD_FILL_PATTERN_IS_REQUIRED = False
SUPPORTS_CONVERT_TIMEZONE = True
SUPPORTS_MEDIAN = True
TYPE_MAPPING = {
**Spark2.Generator.TYPE_MAPPING,

1
sqlglot/dialects/spark2.py
View file

@ -285,6 +285,7 @@ class Spark2(Hive):
transforms.eliminate_qualify,
transforms.eliminate_distinct_on,
transforms.unnest_to_explode,
transforms.any_to_exists,
]
),
exp.StrToDate: _str_to_date,

9
sqlglot/dialects/sqlite.py
View file

@ -120,6 +120,14 @@ class SQLite(Dialect):
}
STRING_ALIASES = True
def _parse_unique(self) -> exp.UniqueColumnConstraint:
# Do not consume more tokens if UNIQUE is used as a standalone constraint, e.g:
# CREATE TABLE foo (bar TEXT UNIQUE REFERENCES baz ...)
if self._curr.text.upper() in self.CONSTRAINT_PARSERS:
return self.expression(exp.UniqueColumnConstraint)
return super()._parse_unique()
class Generator(generator.Generator):
JOIN_HINTS = False
TABLE_HINTS = False
@ -130,6 +138,7 @@ class SQLite(Dialect):
SUPPORTS_TABLE_ALIAS_COLUMNS = False
SUPPORTS_TO_NUMBER = False
EXCEPT_INTERSECT_SUPPORT_ALL_CLAUSE = False
SUPPORTS_MEDIAN = False
SUPPORTED_JSON_PATH_PARTS = {
exp.JSONPathKey,

3
sqlglot/dialects/teradata.py
View file

@ -71,6 +71,9 @@ class Teradata(Dialect):
}
class Tokenizer(tokens.Tokenizer):
# Tested each of these and they work, although there is no
# Teradata documentation explicitly mentioning them.
HEX_STRINGS = [("X'", "'"), ("x'", "'"), ("0x", "")]
# https://docs.teradata.com/r/Teradata-Database-SQL-Functions-Operators-Expressions-and-Predicates/March-2017/Comparison-Operators-and-Functions/Comparison-Operators/ANSI-Compliance
# https://docs.teradata.com/r/SQL-Functions-Operators-Expressions-and-Predicates/June-2017/Arithmetic-Trigonometric-Hyperbolic-Operators/Functions
KEYWORDS = {

4
sqlglot/dialects/trino.py
View file

@ -77,6 +77,8 @@ class Trino(Presto):
if this.this:
this = this.this.pop()
return f"LISTAGG({self.format_args(this, separator)}) WITHIN GROUP ({self.sql(expression.this).lstrip()})"
on_overflow = self.sql(expression, "on_overflow")
on_overflow = f" ON OVERFLOW {on_overflow}" if on_overflow else ""
return f"LISTAGG({self.format_args(this, separator)}{on_overflow}) WITHIN GROUP ({self.sql(expression.this).lstrip()})"
return super().groupconcat_sql(expression)

10
sqlglot/dialects/tsql.py
View file

@ -901,6 +901,7 @@ class TSQL(Dialect):
exp.JSONExtract: _json_extract_sql,
exp.JSONExtractScalar: _json_extract_sql,
exp.LastDay: lambda self, e: self.func("EOMONTH", e.this),
exp.Ln: rename_func("LOG"),
exp.Max: max_or_greatest,
exp.MD5: lambda self, e: self.func("HASHBYTES", exp.Literal.string("MD5"), e.this),
exp.Min: min_or_least,
@ -1103,9 +1104,10 @@ class TSQL(Dialect):
if exists:
identifier = self.sql(exp.Literal.string(exp.table_name(table) if table else ""))
sql = self.sql(exp.Literal.string(sql))
sql_with_ctes = self.prepend_ctes(expression, sql)
sql_literal = self.sql(exp.Literal.string(sql_with_ctes))
if kind == "SCHEMA":
sql = f"""IF NOT EXISTS (SELECT * FROM information_schema.schemata WHERE schema_name = {identifier}) EXEC({sql})"""
return f"""IF NOT EXISTS (SELECT * FROM information_schema.schemata WHERE schema_name = {identifier}) EXEC({sql_literal})"""
elif kind == "TABLE":
assert table
where = exp.and_(
@ -1113,10 +1115,10 @@ class TSQL(Dialect):
exp.column("table_schema").eq(table.db) if table.db else None,
exp.column("table_catalog").eq(table.catalog) if table.catalog else None,
)
sql = f"""IF NOT EXISTS (SELECT * FROM information_schema.tables WHERE {where}) EXEC({sql})"""
return f"""IF NOT EXISTS (SELECT * FROM information_schema.tables WHERE {where}) EXEC({sql_literal})"""
elif kind == "INDEX":
index = self.sql(exp.Literal.string(expression.this.text("this")))
sql = f"""IF NOT EXISTS (SELECT * FROM sys.indexes WHERE object_id = object_id({identifier}) AND name = {index}) EXEC({sql})"""
return f"""IF NOT EXISTS (SELECT * FROM sys.indexes WHERE object_id = object_id({identifier}) AND name = {index}) EXEC({sql_literal})"""
elif expression.args.get("replace"):
sql = sql.replace("CREATE OR REPLACE ", "CREATE OR ALTER ", 1)

74
sqlglot/diff.py
View file

@ -10,6 +10,7 @@ import typing as t
from collections import defaultdict
from dataclasses import dataclass
from heapq import heappop, heappush
from itertools import chain
from sqlglot import Dialect, expressions as exp
from sqlglot.helper import seq_get
@ -36,7 +37,8 @@ class Remove:
class Move:
"""Indicates that an existing node's position within the tree has changed"""
expression: exp.Expression
source: exp.Expression
target: exp.Expression
@dataclass(frozen=True)
@ -93,11 +95,11 @@ def diff(
matchings: the list of pre-matched node pairs which is used to help the algorithm's
heuristics produce better results for subtrees that are known by a caller to be matching.
Note: expression references in this list must refer to the same node objects that are
referenced in source / target trees.
referenced in the source / target trees.
delta_only: excludes all `Keep` nodes from the diff.
copy: whether to copy the input expressions.
Note: if this is set to false, the caller must ensure that there are no shared references
in the two ASTs, otherwise the diffing algorithm may produce unexpected behavior.
in the two trees, otherwise the diffing algorithm may produce unexpected behavior.
kwargs: additional arguments to pass to the ChangeDistiller instance.
Returns:
@ -111,11 +113,19 @@ def diff(
def compute_node_mappings(
original: exp.Expression, copy: exp.Expression
) -> t.Dict[int, exp.Expression]:
return {
id(old_node): new_node
for old_node, new_node in zip(original.walk(), copy.walk())
if id(old_node) in matching_ids
}
node_mapping = {}
for old_node, new_node in zip(
reversed(tuple(original.walk())), reversed(tuple(copy.walk()))
):
# We cache the hash of each new node here to speed up equality comparisons. If the input
# trees aren't copied, these hashes will be evicted before returning the edit script.
new_node._hash = hash(new_node)
old_node_id = id(old_node)
if old_node_id in matching_ids:
node_mapping[old_node_id] = new_node
return node_mapping
source_copy = source.copy() if copy else source
target_copy = target.copy() if copy else target
@ -126,13 +136,19 @@ def diff(
}
matchings_copy = [(node_mappings[id(s)], node_mappings[id(t)]) for s, t in matchings]
return ChangeDistiller(**kwargs).diff(
edit_script = ChangeDistiller(**kwargs).diff(
source_copy,
target_copy,
matchings=matchings_copy,
delta_only=delta_only,
)
if not copy:
for node in chain(source.walk(), target.walk()):
node._hash = None
return edit_script
# The expression types for which Update edits are allowed.
UPDATABLE_EXPRESSION_TYPES = (
@ -186,28 +202,38 @@ class ChangeDistiller:
self._bigram_histo_cache: t.Dict[int, t.DefaultDict[str, int]] = {}
matching_set = self._compute_matching_set() | set(pre_matched_nodes.items())
return self._generate_edit_script(matching_set, delta_only)
return self._generate_edit_script(dict(matching_set), delta_only)
def _generate_edit_script(
self,
matching_set: t.Set[t.Tuple[int, int]],
delta_only: bool,
) -> t.List[Edit]:
def _generate_edit_script(self, matchings: t.Dict[int, int], delta_only: bool) -> t.List[Edit]:
edit_script: t.List[Edit] = []
for removed_node_id in self._unmatched_source_nodes:
edit_script.append(Remove(self._source_index[removed_node_id]))
for inserted_node_id in self._unmatched_target_nodes:
edit_script.append(Insert(self._target_index[inserted_node_id]))
for kept_source_node_id, kept_target_node_id in matching_set:
for kept_source_node_id, kept_target_node_id in matchings.items():
source_node = self._source_index[kept_source_node_id]
target_node = self._target_index[kept_target_node_id]
identical_nodes = source_node == target_node
if not isinstance(source_node, UPDATABLE_EXPRESSION_TYPES) or identical_nodes:
if identical_nodes:
source_parent = source_node.parent
target_parent = target_node.parent
if (
not isinstance(source_node, UPDATABLE_EXPRESSION_TYPES)
or source_node == target_node
(source_parent and not target_parent)
or (not source_parent and target_parent)
or (
source_parent
and target_parent
and matchings.get(id(source_parent)) != id(target_parent)
)
):
edit_script.append(Move(source=source_node, target=target_node))
else:
edit_script.extend(
self._generate_move_edits(source_node, target_node, matching_set)
self._generate_move_edits(source_node, target_node, matchings)
)
source_non_expression_leaves = dict(_get_non_expression_leaves(source_node))
@ -223,17 +249,21 @@ class ChangeDistiller:
return edit_script
def _generate_move_edits(
self, source: exp.Expression, target: exp.Expression, matching_set: t.Set[t.Tuple[int, int]]
self, source: exp.Expression, target: exp.Expression, matchings: t.Dict[int, int]
) -> t.List[Move]:
source_args = [id(e) for e in _expression_only_args(source)]
target_args = [id(e) for e in _expression_only_args(target)]
args_lcs = set(_lcs(source_args, target_args, lambda l, r: (l, r) in matching_set))
args_lcs = set(
_lcs(source_args, target_args, lambda l, r: matchings.get(t.cast(int, l)) == r)
)
move_edits = []
for a in source_args:
if a not in args_lcs and a not in self._unmatched_source_nodes:
move_edits.append(Move(self._source_index[a]))
move_edits.append(
Move(source=self._source_index[a], target=self._target_index[matchings[a]])
)
return move_edits

61
sqlglot/expressions.py
View file

@ -4312,6 +4312,7 @@ class DataType(Expression):
DECIMAL32 = auto()
DECIMAL64 = auto()
DECIMAL128 = auto()
DECIMAL256 = auto()
DOUBLE = auto()
ENUM = auto()
ENUM8 = auto()
@ -4320,6 +4321,12 @@ class DataType(Expression):
FLOAT = auto()
GEOGRAPHY = auto()
GEOMETRY = auto()
POINT = auto()
RING = auto()
LINESTRING = auto()
MULTILINESTRING = auto()
POLYGON = auto()
MULTIPOLYGON = auto()
HLLSKETCH = auto()
HSTORE = auto()
IMAGE = auto()
@ -4467,6 +4474,7 @@ class DataType(Expression):
Type.DECIMAL32,
Type.DECIMAL64,
Type.DECIMAL128,
Type.DECIMAL256,
Type.MONEY,
Type.SMALLMONEY,
Type.UDECIMAL,
@ -4597,10 +4605,6 @@ class Any(SubqueryPredicate):
pass
class Exists(SubqueryPredicate):
pass
# Commands to interact with the databases or engines. For most of the command
# expressions we parse whatever comes after the command's name as a string.
class Command(Expression):
@ -5276,6 +5280,11 @@ class ArrayToString(Func):
_sql_names = ["ARRAY_TO_STRING", "ARRAY_JOIN"]
# https://cloud.google.com/bigquery/docs/reference/standard-sql/timestamp_functions#string
class String(Func):
arg_types = {"this": True, "zone": False}
class StringToArray(Func):
arg_types = {"this": True, "expression": True, "null": False}
_sql_names = ["STRING_TO_ARRAY", "SPLIT_BY_STRING"]
@ -5494,11 +5503,17 @@ class DateTrunc(Func):
arg_types = {"unit": True, "this": True, "zone": False}
def __init__(self, **args):
# Across most dialects it's safe to unabbreviate the unit (e.g. 'Q' -> 'QUARTER') except Oracle
# https://docs.oracle.com/en/database/oracle/oracle-database/21/sqlrf/ROUND-and-TRUNC-Date-Functions.html
unabbreviate = args.pop("unabbreviate", True)
unit = args.get("unit")
if isinstance(unit, TimeUnit.VAR_LIKE):
args["unit"] = Literal.string(
(TimeUnit.UNABBREVIATED_UNIT_NAME.get(unit.name) or unit.name).upper()
)
unit_name = unit.name.upper()
if unabbreviate and unit_name in TimeUnit.UNABBREVIATED_UNIT_NAME:
unit_name = TimeUnit.UNABBREVIATED_UNIT_NAME[unit_name]
args["unit"] = Literal.string(unit_name)
elif isinstance(unit, Week):
unit.set("this", Literal.string(unit.this.name.upper()))
@ -5570,6 +5585,10 @@ class Extract(Func):
arg_types = {"this": True, "expression": True}
class Exists(Func, SubqueryPredicate):
arg_types = {"this": True, "expression": False}
class Timestamp(Func):
arg_types = {"this": False, "zone": False, "with_tz": False}
@ -5746,8 +5765,14 @@ class Greatest(Func):
is_var_len_args = True
# Trino's `ON OVERFLOW TRUNCATE [filler_string] {WITH | WITHOUT} COUNT`
# https://trino.io/docs/current/functions/aggregate.html#listagg
class OverflowTruncateBehavior(Expression):
arg_types = {"this": False, "with_count": True}
class GroupConcat(AggFunc):
arg_types = {"this": True, "separator": False}
arg_types = {"this": True, "separator": False, "on_overflow": False}
class Hex(Func):
@ -5947,6 +5972,11 @@ class JSONBContains(Binary, Func):
_sql_names = ["JSONB_CONTAINS"]
class JSONBExists(Func):
arg_types = {"this": True, "path": True}
_sql_names = ["JSONB_EXISTS"]
class JSONExtract(Binary, Func):
arg_types = {
"this": True,
@ -6025,6 +6055,7 @@ class Levenshtein(Func):
"ins_cost": False,
"del_cost": False,
"sub_cost": False,
"max_dist": False,
}
@ -6116,6 +6147,10 @@ class MD5Digest(Func):
_sql_names = ["MD5_DIGEST"]
class Median(AggFunc):
pass
class Min(AggFunc):
arg_types = {"this": True, "expressions": False}
is_var_len_args = True
@ -6198,11 +6233,11 @@ class Reduce(Func):
class RegexpExtract(Func):
arg_types = {
"this": True,
"expression": True,
"position": False,
"occurrence": False,
"parameters": False,
"group": False,
"expression": True, # The pattern
"position": False, # Only start searching the string from this index
"occurrence": False, # Skip the first `occurence-1` matches
"parameters": False, # Flags, eg "i" for case-insensitive
"group": False, # Which group to return
}

44
sqlglot/generator.py
View file

@ -433,6 +433,9 @@ class Generator(metaclass=_Generator):
# Whether CONVERT_TIMEZONE() is supported; if not, it will be generated as exp.AtTimeZone
SUPPORTS_CONVERT_TIMEZONE = False
# Whether MEDIAN(expr) is supported; if not, it will be generated as PERCENTILE_CONT(expr, 0.5)
SUPPORTS_MEDIAN = True
# The name to generate for the JSONPath expression. If `None`, only `this` will be generated
PARSE_JSON_NAME: t.Optional[str] = "PARSE_JSON"
@ -2314,8 +2317,10 @@ class Generator(metaclass=_Generator):
step_sql = self.sql(expression, "step")
step_sql = f" STEP {step_sql}" if step_sql else ""
interpolated_values = [
f"{self.sql(named_expression, 'alias')} AS {self.sql(named_expression, 'this')}"
for named_expression in expression.args.get("interpolate") or []
f"{self.sql(e, 'alias')} AS {self.sql(e, 'this')}"
if isinstance(e, exp.Alias)
else self.sql(e, "this")
for e in expression.args.get("interpolate") or []
]
interpolate = (
f" INTERPOLATE ({', '.join(interpolated_values)})" if interpolated_values else ""
@ -4362,19 +4367,24 @@ class Generator(metaclass=_Generator):
def arrayagg_sql(self, expression: exp.ArrayAgg) -> str:
array_agg = self.function_fallback_sql(expression)
# Add a NULL FILTER on the column to mimic the results going from a dialect that excludes nulls
# on ARRAY_AGG (e.g Spark) to one that doesn't (e.g. DuckDB)
if self.dialect.ARRAY_AGG_INCLUDES_NULLS and expression.args.get("nulls_excluded"):
parent = expression.parent
if isinstance(parent, exp.Filter):
parent_cond = parent.expression.this
parent_cond.replace(parent_cond.and_(expression.this.is_(exp.null()).not_()))
else:
# DISTINCT is already present in the agg function, do not propagate it to FILTER as well
this = expression.this
# Do not add the filter if the input is not a column (e.g. literal, struct etc)
if this.find(exp.Column):
# DISTINCT is already present in the agg function, do not propagate it to FILTER as well
this_sql = (
self.expressions(this)
if isinstance(this, exp.Distinct)
else self.sql(expression, "this")
)
array_agg = f"{array_agg} FILTER(WHERE {this_sql} IS NOT NULL)"
return array_agg
@ -4434,3 +4444,31 @@ class Generator(metaclass=_Generator):
@unsupported_args("format")
def todouble_sql(self, expression: exp.ToDouble) -> str:
return self.sql(exp.cast(expression.this, exp.DataType.Type.DOUBLE))
def string_sql(self, expression: exp.String) -> str:
this = expression.this
zone = expression.args.get("zone")
if zone:
# This is a BigQuery specific argument for STRING(<timestamp_expr>, <time_zone>)
# BigQuery stores timestamps internally as UTC, so ConvertTimezone is used with UTC
# set for source_tz to transpile the time conversion before the STRING cast
this = exp.ConvertTimezone(
source_tz=exp.Literal.string("UTC"), target_tz=zone, timestamp=this
)
return self.sql(exp.cast(this, exp.DataType.Type.VARCHAR))
def median_sql(self, expression: exp.Median):
if not self.SUPPORTS_MEDIAN:
return self.sql(
exp.PercentileCont(this=expression.this, expression=exp.Literal.number(0.5))
)
return self.function_fallback_sql(expression)
def overflowtruncatebehavior_sql(self, expression: exp.OverflowTruncateBehavior) -> str:
filler = self.sql(expression, "this")
filler = f" {filler}" if filler else ""
with_count = "WITH COUNT" if expression.args.get("with_count") else "WITHOUT COUNT"
return f"TRUNCATE{filler} {with_count}"

39
sqlglot/optimizer/annotate_types.py
View file

@ -192,6 +192,11 @@ class TypeAnnotator(metaclass=_TypeAnnotator):
# Caches the ids of annotated sub-Expressions, to ensure we only visit them once
self._visited: t.Set[int] = set()
# Maps an exp.SetOperation's id (e.g. UNION) to its projection types. This is computed if the
# exp.SetOperation is the expression of a scope source, as selecting from it multiple times
# would reprocess the entire subtree to coerce the types of its operands' projections
self._setop_column_types: t.Dict[int, t.Dict[str, exp.DataType.Type]] = {}
def _set_type(
self, expression: exp.Expression, target_type: t.Optional[exp.DataType | exp.DataType.Type]
) -> None:
@ -231,22 +236,42 @@ class TypeAnnotator(metaclass=_TypeAnnotator):
elif isinstance(expression, exp.SetOperation) and len(expression.left.selects) == len(
expression.right.selects
):
if expression.args.get("by_name"):
r_type_by_select = {s.alias_or_name: s.type for s in expression.right.selects}
selects[name] = {
selects[name] = col_types = self._setop_column_types.setdefault(id(expression), {})
if not col_types:
# Process a chain / sub-tree of set operations
for set_op in expression.walk(
prune=lambda n: not isinstance(n, (exp.SetOperation, exp.Subquery))
):
if not isinstance(set_op, exp.SetOperation):
continue
if set_op.args.get("by_name"):
r_type_by_select = {
s.alias_or_name: s.type for s in set_op.right.selects
}
setop_cols = {
s.alias_or_name: self._maybe_coerce(
t.cast(exp.DataType, s.type),
r_type_by_select.get(s.alias_or_name) or exp.DataType.Type.UNKNOWN,
r_type_by_select.get(s.alias_or_name)
or exp.DataType.Type.UNKNOWN,
)
for s in expression.left.selects
for s in set_op.left.selects
}
else:
selects[name] = {
setop_cols = {
ls.alias_or_name: self._maybe_coerce(
t.cast(exp.DataType, ls.type), t.cast(exp.DataType, rs.type)
)
for ls, rs in zip(expression.left.selects, expression.right.selects)
for ls, rs in zip(set_op.left.selects, set_op.right.selects)
}
# Coerce intermediate results with the previously registered types, if they exist
for col_name, col_type in setop_cols.items():
col_types[col_name] = self._maybe_coerce(
col_type, col_types.get(col_name, exp.DataType.Type.NULL)
)
else:
selects[name] = {s.alias_or_name: s.type for s in expression.selects}

60
sqlglot/parser.py
View file

@ -351,11 +351,18 @@ class Parser(metaclass=_Parser):
TokenType.DECIMAL32,
TokenType.DECIMAL64,
TokenType.DECIMAL128,
TokenType.DECIMAL256,
TokenType.UDECIMAL,
TokenType.BIGDECIMAL,
TokenType.UUID,
TokenType.GEOGRAPHY,
TokenType.GEOMETRY,
TokenType.POINT,
TokenType.RING,
TokenType.LINESTRING,
TokenType.MULTILINESTRING,
TokenType.POLYGON,
TokenType.MULTIPOLYGON,
TokenType.HLLSKETCH,
TokenType.HSTORE,
TokenType.PSEUDO_TYPE,
@ -1838,7 +1845,7 @@ class Parser(metaclass=_Parser):
expression = self._parse_string()
extend_props(self._parse_properties())
else:
expression = self._parse_statement()
expression = self._parse_user_defined_function_expression()
end = self._match_text_seq("END")
@ -3049,7 +3056,11 @@ class Parser(metaclass=_Parser):
elif self._match(TokenType.DESCRIBE):
this = self._parse_describe()
elif self._match_text_seq("STREAM"):
this = self.expression(exp.Stream, this=self._parse_function())
this = self._parse_function()
if this:
this = self.expression(exp.Stream, this=this)
else:
self._retreat(self._index - 1)
else:
this = None
@ -4159,12 +4170,11 @@ class Parser(metaclass=_Parser):
return self.expression(exp.Connect, start=start, connect=connect, nocycle=nocycle)
def _parse_name_as_expression(self) -> exp.Alias:
return self.expression(
exp.Alias,
alias=self._parse_id_var(any_token=True),
this=self._match(TokenType.ALIAS) and self._parse_assignment(),
)
def _parse_name_as_expression(self) -> t.Optional[exp.Expression]:
this = self._parse_id_var(any_token=True)
if self._match(TokenType.ALIAS):
this = self.expression(exp.Alias, alias=this, this=self._parse_assignment())
return this
def _parse_interpolate(self) -> t.Optional[t.List[exp.Expression]]:
if self._match_text_seq("INTERPOLATE"):
@ -5335,6 +5345,9 @@ class Parser(metaclass=_Parser):
return transformed
def _parse_user_defined_function_expression(self) -> t.Optional[exp.Expression]:
return self._parse_statement()
def _parse_function_parameter(self) -> t.Optional[exp.Expression]:
return self._parse_column_def(self._parse_id_var())
@ -5941,7 +5954,7 @@ class Parser(metaclass=_Parser):
action=self._parse_var_from_options(self.CAST_ACTIONS, raise_unmatched=False),
)
def _parse_string_agg(self) -> exp.Expression:
def _parse_string_agg(self) -> exp.GroupConcat:
if self._match(TokenType.DISTINCT):
args: t.List[t.Optional[exp.Expression]] = [
self.expression(exp.Distinct, expressions=[self._parse_assignment()])
@ -5951,6 +5964,23 @@ class Parser(metaclass=_Parser):
else:
args = self._parse_csv(self._parse_assignment) # type: ignore
if self._match_text_seq("ON", "OVERFLOW"):
# trino: LISTAGG(expression [, separator] [ON OVERFLOW overflow_behavior])
if self._match_text_seq("ERROR"):
on_overflow: t.Optional[exp.Expression] = exp.var("ERROR")
else:
self._match_text_seq("TRUNCATE")
on_overflow = self.expression(
exp.OverflowTruncateBehavior,
this=self._parse_string(),
with_count=(
self._match_text_seq("WITH", "COUNT")
or not self._match_text_seq("WITHOUT", "COUNT")
),
)
else:
on_overflow = None
index = self._index
if not self._match(TokenType.R_PAREN) and args:
# postgres: STRING_AGG([DISTINCT] expression, separator [ORDER BY expression1 {ASC | DESC} [, ...]])
@ -5965,9 +5995,15 @@ class Parser(metaclass=_Parser):
self._retreat(index)
return self.validate_expression(exp.GroupConcat.from_arg_list(args), args)
self._match_l_paren() # The corresponding match_r_paren will be called in parse_function (caller)
order = self._parse_order(this=seq_get(args, 0))
return self.expression(exp.GroupConcat, this=order, separator=seq_get(args, 1))
# The corresponding match_r_paren will be called in parse_function (caller)
self._match_l_paren()
return self.expression(
exp.GroupConcat,
this=self._parse_order(this=seq_get(args, 0)),
separator=seq_get(args, 1),
on_overflow=on_overflow,
)
def _parse_convert(
self, strict: bool, safe: t.Optional[bool] = None

14
sqlglot/schema.py
View file

@ -151,9 +151,7 @@ class AbstractMappingSchema:
return self._supported_table_args
def table_parts(self, table: exp.Table) -> t.List[str]:
if isinstance(table.this, exp.ReadCSV):
return [table.this.name]
return [table.text(part) for part in exp.TABLE_PARTS if table.text(part)]
return [part.name for part in reversed(table.parts)]
def find(
self, table: exp.Table, raise_on_missing: bool = True, ensure_data_types: bool = False
@ -417,12 +415,10 @@ class MappingSchema(AbstractMappingSchema, Schema):
normalized_table = exp.maybe_parse(table, into=exp.Table, dialect=dialect, copy=normalize)
if normalize:
for arg in exp.TABLE_PARTS:
value = normalized_table.args.get(arg)
if isinstance(value, exp.Identifier):
normalized_table.set(
arg,
normalize_name(value, dialect=dialect, is_table=True, normalize=normalize),
for part in normalized_table.parts:
if isinstance(part, exp.Identifier):
part.replace(
normalize_name(part, dialect=dialect, is_table=True, normalize=normalize)
)
return normalized_table

18
sqlglot/tokens.py
View file

@ -127,6 +127,7 @@ class TokenType(AutoName):
DECIMAL32 = auto()
DECIMAL64 = auto()
DECIMAL128 = auto()
DECIMAL256 = auto()
UDECIMAL = auto()
BIGDECIMAL = auto()
CHAR = auto()
@ -175,6 +176,12 @@ class TokenType(AutoName):
GEOGRAPHY = auto()
NULLABLE = auto()
GEOMETRY = auto()
POINT = auto()
RING = auto()
LINESTRING = auto()
MULTILINESTRING = auto()
POLYGON = auto()
MULTIPOLYGON = auto()
HLLSKETCH = auto()
HSTORE = auto()
SUPER = auto()
@ -602,11 +609,15 @@ class Tokenizer(metaclass=_Tokenizer):
HEREDOC_STRINGS: t.List[str | t.Tuple[str, str]] = []
UNICODE_STRINGS: t.List[str | t.Tuple[str, str]] = []
IDENTIFIERS: t.List[str | t.Tuple[str, str]] = ['"']
IDENTIFIER_ESCAPES = ['"']
QUOTES: t.List[t.Tuple[str, str] | str] = ["'"]
STRING_ESCAPES = ["'"]
VAR_SINGLE_TOKENS: t.Set[str] = set()
# The strings in this list can always be used as escapes, regardless of the surrounding
# identifier delimiters. By default, the closing delimiter is assumed to also act as an
# identifier escape, e.g. if we use double-quotes, then they also act as escapes: "x"""
IDENTIFIER_ESCAPES: t.List[str] = []
# Whether the heredoc tags follow the same lexical rules as unquoted identifiers
HEREDOC_TAG_IS_IDENTIFIER = False
@ -840,6 +851,7 @@ class Tokenizer(metaclass=_Tokenizer):
"DECIMAL32": TokenType.DECIMAL32,
"DECIMAL64": TokenType.DECIMAL64,
"DECIMAL128": TokenType.DECIMAL128,
"DECIMAL256": TokenType.DECIMAL256,
"BIGDECIMAL": TokenType.BIGDECIMAL,
"BIGNUMERIC": TokenType.BIGDECIMAL,
"LIST": TokenType.LIST,
@ -1355,7 +1367,9 @@ class Tokenizer(metaclass=_Tokenizer):
def _scan_identifier(self, identifier_end: str) -> None:
self._advance()
text = self._extract_string(identifier_end, escapes=self._IDENTIFIER_ESCAPES)
text = self._extract_string(
identifier_end, escapes=self._IDENTIFIER_ESCAPES | {identifier_end}
)
self._add(TokenType.IDENTIFIER, text)
def _scan_var(self) -> None:

27
sqlglot/transforms.py
View file

@ -914,3 +914,30 @@ def eliminate_join_marks(expression: exp.Expression) -> exp.Expression:
where.pop()
return expression
def any_to_exists(expression: exp.Expression) -> exp.Expression:
"""
Transform ANY operator to Spark's EXISTS
For example,
- Postgres: SELECT * FROM tbl WHERE 5 > ANY(tbl.col)
- Spark: SELECT * FROM tbl WHERE EXISTS(tbl.col, x -> x < 5)
Both ANY and EXISTS accept queries but currently only array expressions are supported for this
transformation
"""
if isinstance(expression, exp.Select):
for any in expression.find_all(exp.Any):
this = any.this
if isinstance(this, exp.Query):
continue
binop = any.parent
if isinstance(binop, exp.Binary):
lambda_arg = exp.to_identifier("x")
any.replace(lambda_arg)
lambda_expr = exp.Lambda(this=binop.copy(), expressions=[lambda_arg])
binop.replace(exp.Exists(this=this.unnest(), expression=lambda_expr))
return expression

2
sqlglotrs/Cargo.lock generated
View file

@ -188,7 +188,7 @@ checksum = "4dccd0940a2dcdf68d092b8cbab7dc0ad8fa938bf95787e1b916b0e3d0e8e970"
[[package]]
name = "sqlglotrs"
version = "0.2.12"
version = "0.2.13"
dependencies = [
"pyo3",
]

2
sqlglotrs/Cargo.toml
View file

@ -1,6 +1,6 @@
[package]
name = "sqlglotrs"
version = "0.2.12"
version = "0.2.13"
edition = "2021"
license = "MIT"

5
sqlglotrs/src/tokenizer.rs
View file

@ -609,8 +609,11 @@ impl<'a> TokenizerState<'a> {
let mut text = String::from("");
loop {
let mut new_identifier_escapes;
let escapes = if use_identifier_escapes {
&self.settings.identifier_escapes
new_identifier_escapes = self.settings.identifier_escapes.clone();
new_identifier_escapes.extend(delimiter.chars());
&new_identifier_escapes
} else {
&self.settings.string_escapes
};

136
tests/dialects/test_bigquery.py
View file

@ -9,7 +9,6 @@ from sqlglot import (
UnsupportedError,
exp,
parse,
parse_one,
transpile,
)
from sqlglot.helper import logger as helper_logger
@ -85,12 +84,21 @@ LANGUAGE js AS
"PARSE_TIMESTAMP('%Y-%m-%dT%H:%M:%E*S%z', x)",
)
table = parse_one("x-0._y.z", dialect="bigquery", into=exp.Table)
for prefix in ("c.db.", "db.", ""):
with self.subTest(f"Parsing {prefix}INFORMATION_SCHEMA.X into a Table"):
table = self.parse_one(f"`{prefix}INFORMATION_SCHEMA.X`", into=exp.Table)
this = table.this
self.assertIsInstance(this, exp.Identifier)
self.assertTrue(this.quoted)
self.assertEqual(this.name, "INFORMATION_SCHEMA.X")
table = self.parse_one("x-0._y.z", into=exp.Table)
self.assertEqual(table.catalog, "x-0")
self.assertEqual(table.db, "_y")
self.assertEqual(table.name, "z")
table = parse_one("x-0._y", dialect="bigquery", into=exp.Table)
table = self.parse_one("x-0._y", into=exp.Table)
self.assertEqual(table.db, "x-0")
self.assertEqual(table.name, "_y")
@ -165,6 +173,7 @@ LANGUAGE js AS
self.validate_identity("SELECT * FROM foo.bar.25ab c", "SELECT * FROM foo.bar.`25ab` AS c")
self.validate_identity("x <> ''")
self.validate_identity("DATE_TRUNC(col, WEEK(MONDAY))")
self.validate_identity("DATE_TRUNC(col, MONTH, 'UTC+8')")
self.validate_identity("SELECT b'abc'")
self.validate_identity("SELECT AS STRUCT 1 AS a, 2 AS b")
self.validate_identity("SELECT DISTINCT AS STRUCT 1 AS a, 2 AS b")
@ -182,7 +191,6 @@ LANGUAGE js AS
self.validate_identity("SELECT y + 1 FROM x GROUP BY y + 1 ORDER BY 1")
self.validate_identity("SELECT TIMESTAMP_SECONDS(2) AS t")
self.validate_identity("SELECT TIMESTAMP_MILLIS(2) AS t")
self.validate_identity("""SELECT JSON_EXTRACT_SCALAR('{"a": 5}', '$.a')""")
self.validate_identity("UPDATE x SET y = NULL")
self.validate_identity("LOG(n, b)")
self.validate_identity("SELECT COUNT(x RESPECT NULLS)")
@ -194,11 +202,11 @@ LANGUAGE js AS
self.validate_identity("CAST(x AS NVARCHAR)", "CAST(x AS STRING)")
self.validate_identity("CAST(x AS TIMESTAMPTZ)", "CAST(x AS TIMESTAMP)")
self.validate_identity("CAST(x AS RECORD)", "CAST(x AS STRUCT)")
self.validate_identity(
"MERGE INTO dataset.NewArrivals USING (SELECT * FROM UNNEST([('microwave', 10, 'warehouse #1'), ('dryer', 30, 'warehouse #1'), ('oven', 20, 'warehouse #2')])) ON FALSE WHEN NOT MATCHED THEN INSERT ROW WHEN NOT MATCHED BY SOURCE THEN DELETE"
self.validate_identity("EDIT_DISTANCE('a', 'a', max_distance => 2)").assert_is(
exp.Levenshtein
)
self.validate_identity(
"SELECT * FROM `SOME_PROJECT_ID.SOME_DATASET_ID.INFORMATION_SCHEMA.SOME_VIEW`"
"MERGE INTO dataset.NewArrivals USING (SELECT * FROM UNNEST([('microwave', 10, 'warehouse #1'), ('dryer', 30, 'warehouse #1'), ('oven', 20, 'warehouse #2')])) ON FALSE WHEN NOT MATCHED THEN INSERT ROW WHEN NOT MATCHED BY SOURCE THEN DELETE"
)
self.validate_identity(
"SELECT * FROM test QUALIFY a IS DISTINCT FROM b WINDOW c AS (PARTITION BY d)"
@ -228,10 +236,23 @@ LANGUAGE js AS
"SELECT LAST_VALUE(a IGNORE NULLS) OVER y FROM x WINDOW y AS (PARTITION BY CATEGORY)",
)
self.validate_identity(
"""SELECT JSON_EXTRACT_SCALAR('5')""", """SELECT JSON_EXTRACT_SCALAR('5', '$')"""
"CREATE OR REPLACE VIEW test (tenant_id OPTIONS (description='Test description on table creation')) AS SELECT 1 AS tenant_id, 1 AS customer_id",
)
self.validate_identity(
"CREATE OR REPLACE VIEW test (tenant_id OPTIONS (description='Test description on table creation')) AS SELECT 1 AS tenant_id, 1 AS customer_id",
"SELECT * FROM `proj.dataset.INFORMATION_SCHEMA.SOME_VIEW`",
"SELECT * FROM `proj.dataset.INFORMATION_SCHEMA.SOME_VIEW` AS `proj.dataset.INFORMATION_SCHEMA.SOME_VIEW`",
)
self.validate_identity(
"SELECT * FROM region_or_dataset.INFORMATION_SCHEMA.TABLES",
"SELECT * FROM region_or_dataset.`INFORMATION_SCHEMA.TABLES` AS TABLES",
)
self.validate_identity(
"SELECT * FROM region_or_dataset.INFORMATION_SCHEMA.TABLES AS some_name",
"SELECT * FROM region_or_dataset.`INFORMATION_SCHEMA.TABLES` AS some_name",
)
self.validate_identity(
"SELECT * FROM proj.region_or_dataset.INFORMATION_SCHEMA.TABLES",
"SELECT * FROM proj.region_or_dataset.`INFORMATION_SCHEMA.TABLES` AS TABLES",
)
self.validate_identity(
"CREATE VIEW `d.v` OPTIONS (expiration_timestamp=TIMESTAMP '2020-01-02T04:05:06.007Z') AS SELECT 1 AS c",
@ -302,6 +323,13 @@ LANGUAGE js AS
"SELECT CAST(1 AS INT64)",
)
self.validate_all(
"EDIT_DISTANCE(a, b)",
write={
"bigquery": "EDIT_DISTANCE(a, b)",
"duckdb": "LEVENSHTEIN(a, b)",
},
)
self.validate_all(
"SAFE_CAST(some_date AS DATE FORMAT 'DD MONTH YYYY')",
write={
@ -361,9 +389,18 @@ LANGUAGE js AS
write={
"bigquery": "TIMESTAMP(x)",
"duckdb": "CAST(x AS TIMESTAMPTZ)",
"snowflake": "CAST(x AS TIMESTAMPTZ)",
"presto": "CAST(x AS TIMESTAMP WITH TIME ZONE)",
},
)
self.validate_all(
"SELECT TIMESTAMP('2008-12-25 15:30:00', 'America/Los_Angeles')",
write={
"bigquery": "SELECT TIMESTAMP('2008-12-25 15:30:00', 'America/Los_Angeles')",
"duckdb": "SELECT CAST('2008-12-25 15:30:00' AS TIMESTAMP) AT TIME ZONE 'America/Los_Angeles'",
"snowflake": "SELECT CONVERT_TIMEZONE('America/Los_Angeles', CAST('2008-12-25 15:30:00' AS TIMESTAMP))",
},
)
self.validate_all(
"SELECT SUM(x IGNORE NULLS) AS x",
read={
@ -629,6 +666,7 @@ LANGUAGE js AS
write={
"bigquery": "SELECT DATETIME_DIFF('2023-01-01T00:00:00', '2023-01-01T05:00:00', MILLISECOND)",
"databricks": "SELECT TIMESTAMPDIFF(MILLISECOND, '2023-01-01T05:00:00', '2023-01-01T00:00:00')",
"snowflake": "SELECT TIMESTAMPDIFF(MILLISECOND, '2023-01-01T05:00:00', '2023-01-01T00:00:00')",
},
),
)
@ -639,6 +677,7 @@ LANGUAGE js AS
"bigquery": "SELECT DATETIME_ADD('2023-01-01T00:00:00', INTERVAL '1' MILLISECOND)",
"databricks": "SELECT TIMESTAMPADD(MILLISECOND, '1', '2023-01-01T00:00:00')",
"duckdb": "SELECT CAST('2023-01-01T00:00:00' AS DATETIME) + INTERVAL '1' MILLISECOND",
"snowflake": "SELECT TIMESTAMPADD(MILLISECOND, '1', '2023-01-01T00:00:00')",
},
),
)
@ -670,6 +709,7 @@ LANGUAGE js AS
"databricks": "SELECT DATE_ADD(MINUTE, '10', CAST('2008-12-25 15:30:00+00' AS TIMESTAMP))",
"mysql": "SELECT DATE_ADD(TIMESTAMP('2008-12-25 15:30:00+00'), INTERVAL '10' MINUTE)",
"spark": "SELECT DATE_ADD(MINUTE, '10', CAST('2008-12-25 15:30:00+00' AS TIMESTAMP))",
"snowflake": "SELECT TIMESTAMPADD(MINUTE, '10', CAST('2008-12-25 15:30:00+00' AS TIMESTAMPTZ))",
},
)
self.validate_all(
@ -677,6 +717,14 @@ LANGUAGE js AS
write={
"bigquery": "SELECT TIMESTAMP_SUB(CAST('2008-12-25 15:30:00+00' AS TIMESTAMP), INTERVAL '10' MINUTE)",
"mysql": "SELECT DATE_SUB(TIMESTAMP('2008-12-25 15:30:00+00'), INTERVAL '10' MINUTE)",
"snowflake": "SELECT TIMESTAMPADD(MINUTE, '10' * -1, CAST('2008-12-25 15:30:00+00' AS TIMESTAMPTZ))",
},
)
self.validate_all(
'SELECT TIMESTAMP_SUB(TIMESTAMP "2008-12-25 15:30:00+00", INTERVAL col MINUTE)',
write={
"bigquery": "SELECT TIMESTAMP_SUB(CAST('2008-12-25 15:30:00+00' AS TIMESTAMP), INTERVAL col MINUTE)",
"snowflake": "SELECT TIMESTAMPADD(MINUTE, col * -1, CAST('2008-12-25 15:30:00+00' AS TIMESTAMPTZ))",
},
)
self.validate_all(
@ -1113,7 +1161,8 @@ LANGUAGE js AS
write={
"bigquery": "CURRENT_TIME()",
"duckdb": "CURRENT_TIME",
"presto": "CURRENT_TIME()",
"presto": "CURRENT_TIME",
"trino": "CURRENT_TIME",
"hive": "CURRENT_TIME()",
"spark": "CURRENT_TIME()",
},
@ -1490,6 +1539,14 @@ WHERE
"duckdb": "SELECT CAST(STRPTIME('Thursday Dec 25 2008', '%A %b %-d %Y') AS DATE)",
},
)
self.validate_all(
"SELECT PARSE_DATE('%Y%m%d', '20081225')",
write={
"bigquery": "SELECT PARSE_DATE('%Y%m%d', '20081225')",
"duckdb": "SELECT CAST(STRPTIME('20081225', '%Y%m%d') AS DATE)",
"snowflake": "SELECT DATE('20081225', 'yyyymmDD')",
},
)
self.validate_all(
"SELECT ARRAY_TO_STRING(['cake', 'pie', NULL], '--') AS text",
write={
@ -1504,9 +1561,48 @@ WHERE
"duckdb": "SELECT ARRAY_TO_STRING(LIST_TRANSFORM(['cake', 'pie', NULL], x -> COALESCE(x, 'MISSING')), '--') AS text",
},
)
self.validate_all(
"STRING(a)",
write={
"bigquery": "STRING(a)",
"snowflake": "CAST(a AS VARCHAR)",
"duckdb": "CAST(a AS TEXT)",
},
)
self.validate_all(
"STRING('2008-12-25 15:30:00', 'America/New_York')",
write={
"bigquery": "STRING('2008-12-25 15:30:00', 'America/New_York')",
"snowflake": "CAST(CONVERT_TIMEZONE('UTC', 'America/New_York', '2008-12-25 15:30:00') AS VARCHAR)",
"duckdb": "CAST(CAST('2008-12-25 15:30:00' AS TIMESTAMP) AT TIME ZONE 'UTC' AT TIME ZONE 'America/New_York' AS TEXT)",
},
)
self.validate_identity("SELECT * FROM a-b c", "SELECT * FROM a-b AS c")
self.validate_all(
"SAFE_DIVIDE(x, y)",
write={
"bigquery": "SAFE_DIVIDE(x, y)",
"duckdb": "IF((y) <> 0, (x) / (y), NULL)",
"presto": "IF((y) <> 0, (x) / (y), NULL)",
"trino": "IF((y) <> 0, (x) / (y), NULL)",
"hive": "IF((y) <> 0, (x) / (y), NULL)",
"spark2": "IF((y) <> 0, (x) / (y), NULL)",
"spark": "IF((y) <> 0, (x) / (y), NULL)",
"databricks": "IF((y) <> 0, (x) / (y), NULL)",
"snowflake": "IFF((y) <> 0, (x) / (y), NULL)",
},
)
self.validate_all(
"""SELECT JSON_QUERY('{"class": {"students": []}}', '$.class')""",
write={
"bigquery": """SELECT JSON_QUERY('{"class": {"students": []}}', '$.class')""",
"duckdb": """SELECT '{"class": {"students": []}}' -> '$.class'""",
"snowflake": """SELECT GET_PATH(PARSE_JSON('{"class": {"students": []}}'), 'class')""",
},
)
def test_errors(self):
with self.assertRaises(TokenError):
transpile("'\\'", read="bigquery")
@ -2000,3 +2096,23 @@ OPTIONS (
"bigquery": f"SELECT color, ARRAY_AGG(id ORDER BY id {sort_order}) AS ids FROM colors GROUP BY 1",
},
)
def test_json_extract_scalar(self):
for func in ("JSON_EXTRACT_SCALAR", "JSON_VALUE"):
with self.subTest(f"Testing BigQuery's {func}"):
self.validate_all(
f"SELECT {func}('5')",
write={
"bigquery": f"SELECT {func}('5', '$')",
"duckdb": """SELECT '5' ->> '$'""",
},
)
self.validate_all(
f"""SELECT {func}('{{"name": "Jakob", "age": "6"}}', '$.age')""",
write={
"bigquery": f"""SELECT {func}('{{"name": "Jakob", "age": "6"}}', '$.age')""",
"duckdb": """SELECT '{"name": "Jakob", "age": "6"}' ->> '$.age'""",
"snowflake": """SELECT JSON_EXTRACT_PATH_TEXT('{"name": "Jakob", "age": "6"}', 'age')""",
},
)

120
tests/dialects/test_clickhouse.py
View file

@ -1,4 +1,4 @@
from datetime import date
from datetime import date, datetime, timezone
from sqlglot import exp, parse_one
from sqlglot.dialects import ClickHouse
from sqlglot.expressions import convert
@ -88,6 +88,7 @@ class TestClickhouse(Validator):
self.validate_identity("CAST(x AS DATETIME)", "CAST(x AS DateTime)")
self.validate_identity("CAST(x AS TIMESTAMPTZ)", "CAST(x AS DateTime)")
self.validate_identity("CAST(x as MEDIUMINT)", "CAST(x AS Int32)")
self.validate_identity("CAST(x AS DECIMAL(38, 2))", "CAST(x AS Decimal(38, 2))")
self.validate_identity("SELECT arrayJoin([1, 2, 3] AS src) AS dst, 'Hello', src")
self.validate_identity("""SELECT JSONExtractString('{"x": {"y": 1}}', 'x', 'y')""")
self.validate_identity("SELECT * FROM table LIMIT 1 BY a, b")
@ -95,6 +96,9 @@ class TestClickhouse(Validator):
self.validate_identity("TRUNCATE TABLE t1 ON CLUSTER test_cluster")
self.validate_identity("TRUNCATE DATABASE db")
self.validate_identity("TRUNCATE DATABASE db ON CLUSTER test_cluster")
self.validate_identity(
"SELECT CAST(1730098800 AS DateTime64) AS DATETIME, 'test' AS interp ORDER BY DATETIME WITH FILL FROM toDateTime64(1730098800, 3) - INTERVAL '7' HOUR TO toDateTime64(1730185140, 3) - INTERVAL '7' HOUR STEP toIntervalSecond(900) INTERPOLATE (interp)"
)
self.validate_identity(
"SELECT number, COUNT() OVER (PARTITION BY number % 3) AS partition_count FROM numbers(10) WINDOW window_name AS (PARTITION BY number) QUALIFY partition_count = 4 ORDER BY number"
)
@ -149,6 +153,10 @@ class TestClickhouse(Validator):
self.validate_identity(
"CREATE TABLE t (foo String CODEC(LZ4HC(9), ZSTD, DELTA), size String ALIAS formatReadableSize(size_bytes), INDEX idx1 a TYPE bloom_filter(0.001) GRANULARITY 1, INDEX idx2 a TYPE set(100) GRANULARITY 2, INDEX idx3 a TYPE minmax GRANULARITY 3)"
)
self.validate_identity(
"SELECT (toUInt8('1') + toUInt8('2')) IS NOT NULL",
"SELECT NOT ((toUInt8('1') + toUInt8('2')) IS NULL)",
)
self.validate_identity(
"SELECT $1$foo$1$",
"SELECT 'foo'",
@ -424,8 +432,13 @@ class TestClickhouse(Validator):
)
self.validate_all(
"SELECT quantile(0.5)(a)",
read={"duckdb": "SELECT quantile(a, 0.5)"},
write={"clickhouse": "SELECT quantile(0.5)(a)"},
read={
"duckdb": "SELECT quantile(a, 0.5)",
"clickhouse": "SELECT median(a)",
},
write={
"clickhouse": "SELECT quantile(0.5)(a)",
},
)
self.validate_all(
"SELECT quantiles(0.5, 0.4)(a)",
@ -526,6 +539,10 @@ class TestClickhouse(Validator):
"SELECT * FROM ABC WHERE hasAny(COLUMNS('.*field') APPLY(toUInt64) APPLY(to), (SELECT groupUniqArray(toUInt64(field))))"
)
self.validate_identity("SELECT col apply", "SELECT col AS apply")
self.validate_identity(
"SELECT name FROM data WHERE (SELECT DISTINCT name FROM data) IS NOT NULL",
"SELECT name FROM data WHERE NOT ((SELECT DISTINCT name FROM data) IS NULL)",
)
def test_clickhouse_values(self):
values = exp.select("*").from_(
@ -645,6 +662,12 @@ class TestClickhouse(Validator):
write={"clickhouse": f"CAST(pow(2, 32) AS {data_type})"},
)
def test_geom_types(self):
data_types = ["Point", "Ring", "LineString", "MultiLineString", "Polygon", "MultiPolygon"]
for data_type in data_types:
with self.subTest(f"Casting to ClickHouse {data_type}"):
self.validate_identity(f"SELECT CAST(val AS {data_type})")
def test_ddl(self):
db_table_expr = exp.Table(this=None, db=exp.to_identifier("foo"), catalog=None)
create_with_cluster = exp.Create(
@ -678,6 +701,7 @@ class TestClickhouse(Validator):
"CREATE TABLE foo ENGINE=Memory AS (SELECT * FROM db.other_table) COMMENT 'foo'",
)
self.validate_identity("CREATE FUNCTION linear_equation AS (x, k, b) -> k * x + b")
self.validate_identity("CREATE MATERIALIZED VIEW a.b TO a.c (c Int32) AS SELECT * FROM a.d")
self.validate_identity("""CREATE TABLE ip_data (ip4 IPv4, ip6 IPv6) ENGINE=TinyLog()""")
self.validate_identity("""CREATE TABLE dates (dt1 Date32) ENGINE=TinyLog()""")
@ -701,6 +725,10 @@ class TestClickhouse(Validator):
self.validate_identity(
"CREATE TABLE foo (x UInt32) TTL time_column + INTERVAL '1' MONTH DELETE WHERE column = 'value'"
)
self.validate_identity(
"CREATE FUNCTION parity_str AS (n) -> IF(n % 2, 'odd', 'even')",
"CREATE FUNCTION parity_str AS n -> CASE WHEN n % 2 THEN 'odd' ELSE 'even' END",
)
self.validate_identity(
"CREATE TABLE a ENGINE=Memory AS SELECT 1 AS c COMMENT 'foo'",
"CREATE TABLE a ENGINE=Memory AS (SELECT 1 AS c) COMMENT 'foo'",
@ -1094,6 +1122,92 @@ LIFETIME(MIN 0 MAX 0)""",
convert(date(2020, 1, 1)).sql(dialect=self.dialect), "toDate('2020-01-01')"
)
# no fractional seconds
self.assertEqual(
convert(datetime(2020, 1, 1, 0, 0, 1)).sql(dialect=self.dialect),
"CAST('2020-01-01 00:00:01' AS DateTime64(6))",
)
self.assertEqual(
convert(datetime(2020, 1, 1, 0, 0, 1, tzinfo=timezone.utc)).sql(dialect=self.dialect),
"CAST('2020-01-01 00:00:01' AS DateTime64(6, 'UTC'))",
)
# with fractional seconds
self.assertEqual(
convert(datetime(2020, 1, 1, 0, 0, 1, 1)).sql(dialect=self.dialect),
"CAST('2020-01-01 00:00:01.000001' AS DateTime64(6))",
)
self.assertEqual(
convert(datetime(2020, 1, 1, 0, 0, 1, 1, tzinfo=timezone.utc)).sql(
dialect=self.dialect
),
"CAST('2020-01-01 00:00:01.000001' AS DateTime64(6, 'UTC'))",
)
def test_timestr_to_time(self):
# no fractional seconds
time_strings = [
"2020-01-01 00:00:01",
"2020-01-01 00:00:01+01:00",
" 2020-01-01 00:00:01-01:00 ",
"2020-01-01T00:00:01+01:00",
]
for time_string in time_strings:
with self.subTest(f"'{time_string}'"):
self.assertEqual(
exp.TimeStrToTime(this=exp.Literal.string(time_string)).sql(
dialect=self.dialect
),
f"CAST('{time_string}' AS DateTime64(6))",
)
time_strings_no_utc = ["2020-01-01 00:00:01" for i in range(4)]
for utc, no_utc in zip(time_strings, time_strings_no_utc):
with self.subTest(f"'{time_string}' with UTC timezone"):
self.assertEqual(
exp.TimeStrToTime(
this=exp.Literal.string(utc), zone=exp.Literal.string("UTC")
).sql(dialect=self.dialect),
f"CAST('{no_utc}' AS DateTime64(6, 'UTC'))",
)
# with fractional seconds
time_strings = [
"2020-01-01 00:00:01.001",
"2020-01-01 00:00:01.000001",
"2020-01-01 00:00:01.001+00:00",
"2020-01-01 00:00:01.000001-00:00",
"2020-01-01 00:00:01.0001",
"2020-01-01 00:00:01.1+00:00",
]
for time_string in time_strings:
with self.subTest(f"'{time_string}'"):
self.assertEqual(
exp.TimeStrToTime(this=exp.Literal.string(time_string[0])).sql(
dialect=self.dialect
),
f"CAST('{time_string[0]}' AS DateTime64(6))",
)
time_strings_no_utc = [
"2020-01-01 00:00:01.001000",
"2020-01-01 00:00:01.000001",
"2020-01-01 00:00:01.001000",
"2020-01-01 00:00:01.000001",
"2020-01-01 00:00:01.000100",
"2020-01-01 00:00:01.100000",
]
for utc, no_utc in zip(time_strings, time_strings_no_utc):
with self.subTest(f"'{time_string}' with UTC timezone"):
self.assertEqual(
exp.TimeStrToTime(
this=exp.Literal.string(utc), zone=exp.Literal.string("UTC")
).sql(dialect=self.dialect),
f"CAST('{no_utc}' AS DateTime64(6, 'UTC'))",
)
def test_grant(self):
self.validate_identity("GRANT SELECT(x, y) ON db.table TO john WITH GRANT OPTION")
self.validate_identity("GRANT INSERT(x, y) ON db.table TO john")

12
tests/dialects/test_databricks.py
View file

@ -7,6 +7,7 @@ class TestDatabricks(Validator):
dialect = "databricks"
def test_databricks(self):
self.validate_identity("SELECT * FROM stream")
self.validate_identity("SELECT t.current_time FROM t")
self.validate_identity("ALTER TABLE labels ADD COLUMN label_score FLOAT")
self.validate_identity("DESCRIBE HISTORY a.b")
@ -116,6 +117,17 @@ class TestDatabricks(Validator):
},
)
self.validate_all(
"SELECT ANY(col) FROM VALUES (TRUE), (FALSE) AS tab(col)",
read={
"databricks": "SELECT ANY(col) FROM VALUES (TRUE), (FALSE) AS tab(col)",
"spark": "SELECT ANY(col) FROM VALUES (TRUE), (FALSE) AS tab(col)",
},
write={
"spark": "SELECT ANY(col) FROM VALUES (TRUE), (FALSE) AS tab(col)",
},
)
# https://docs.databricks.com/sql/language-manual/functions/colonsign.html
def test_json(self):
self.validate_identity("SELECT c1:price, c1:price.foo, c1:price.bar[1]")

128
tests/dialects/test_dialect.py
View file

@ -526,7 +526,7 @@ class TestDialect(Validator):
write={
"": "SELECT NVL2(a, b, c)",
"bigquery": "SELECT CASE WHEN NOT a IS NULL THEN b ELSE c END",
"clickhouse": "SELECT CASE WHEN NOT a IS NULL THEN b ELSE c END",
"clickhouse": "SELECT CASE WHEN NOT (a IS NULL) THEN b ELSE c END",
"databricks": "SELECT NVL2(a, b, c)",
"doris": "SELECT CASE WHEN NOT a IS NULL THEN b ELSE c END",
"drill": "SELECT CASE WHEN NOT a IS NULL THEN b ELSE c END",
@ -552,7 +552,7 @@ class TestDialect(Validator):
write={
"": "SELECT NVL2(a, b)",
"bigquery": "SELECT CASE WHEN NOT a IS NULL THEN b END",
"clickhouse": "SELECT CASE WHEN NOT a IS NULL THEN b END",
"clickhouse": "SELECT CASE WHEN NOT (a IS NULL) THEN b END",
"databricks": "SELECT NVL2(a, b)",
"doris": "SELECT CASE WHEN NOT a IS NULL THEN b END",
"drill": "SELECT CASE WHEN NOT a IS NULL THEN b END",
@ -651,7 +651,7 @@ class TestDialect(Validator):
"snowflake": "CAST('2020-01-01' AS TIMESTAMP)",
"spark": "CAST('2020-01-01' AS TIMESTAMP)",
"trino": "CAST('2020-01-01' AS TIMESTAMP)",
"clickhouse": "CAST('2020-01-01' AS Nullable(DateTime))",
"clickhouse": "CAST('2020-01-01' AS DateTime64(6))",
"drill": "CAST('2020-01-01' AS TIMESTAMP)",
"hive": "CAST('2020-01-01' AS TIMESTAMP)",
"presto": "CAST('2020-01-01' AS TIMESTAMP)",
@ -688,7 +688,7 @@ class TestDialect(Validator):
"snowflake": "CAST('2020-01-01 12:13:14-08:00' AS TIMESTAMPTZ)",
"spark": "CAST('2020-01-01 12:13:14-08:00' AS TIMESTAMP)",
"trino": "CAST('2020-01-01 12:13:14-08:00' AS TIMESTAMP WITH TIME ZONE)",
"clickhouse": "CAST('2020-01-01 12:13:14' AS Nullable(DateTime('America/Los_Angeles')))",
"clickhouse": "CAST('2020-01-01 12:13:14' AS DateTime64(6, 'America/Los_Angeles'))",
"drill": "CAST('2020-01-01 12:13:14-08:00' AS TIMESTAMP)",
"hive": "CAST('2020-01-01 12:13:14-08:00' AS TIMESTAMP)",
"presto": "CAST('2020-01-01 12:13:14-08:00' AS TIMESTAMP WITH TIME ZONE)",
@ -709,7 +709,7 @@ class TestDialect(Validator):
"snowflake": "CAST(col AS TIMESTAMPTZ)",
"spark": "CAST(col AS TIMESTAMP)",
"trino": "CAST(col AS TIMESTAMP WITH TIME ZONE)",
"clickhouse": "CAST(col AS Nullable(DateTime('America/Los_Angeles')))",
"clickhouse": "CAST(col AS DateTime64(6, 'America/Los_Angeles'))",
"drill": "CAST(col AS TIMESTAMP)",
"hive": "CAST(col AS TIMESTAMP)",
"presto": "CAST(col AS TIMESTAMP WITH TIME ZONE)",
@ -2893,3 +2893,121 @@ FROM subquery2""",
"snowflake": "UUID_STRING()",
},
)
def test_escaped_identifier_delimiter(self):
for dialect in ("databricks", "hive", "mysql", "spark2", "spark"):
with self.subTest(f"Testing escaped backtick in identifier name for {dialect}"):
self.validate_all(
'SELECT 1 AS "x`"',
read={
dialect: "SELECT 1 AS `x```",
},
write={
dialect: "SELECT 1 AS `x```",
},
)
for dialect in (
"",
"clickhouse",
"duckdb",
"postgres",
"presto",
"trino",
"redshift",
"snowflake",
"sqlite",
):
with self.subTest(f"Testing escaped double-quote in identifier name for {dialect}"):
self.validate_all(
'SELECT 1 AS "x"""',
read={
dialect: 'SELECT 1 AS "x"""',
},
write={
dialect: 'SELECT 1 AS "x"""',
},
)
for dialect in ("clickhouse", "sqlite"):
with self.subTest(f"Testing escaped backtick in identifier name for {dialect}"):
self.validate_all(
'SELECT 1 AS "x`"',
read={
dialect: "SELECT 1 AS `x```",
},
write={
dialect: 'SELECT 1 AS "x`"',
},
)
self.validate_all(
'SELECT 1 AS "x`"',
read={
"clickhouse": "SELECT 1 AS `x\\``",
},
write={
"clickhouse": 'SELECT 1 AS "x`"',
},
)
for name in ('"x\\""', '`x"`'):
with self.subTest(f"Testing ClickHouse delimiter escaping: {name}"):
self.validate_all(
'SELECT 1 AS "x"""',
read={
"clickhouse": f"SELECT 1 AS {name}",
},
write={
"clickhouse": 'SELECT 1 AS "x"""',
},
)
for name in ("[[x]]]", '"[x]"'):
with self.subTest(f"Testing T-SQL delimiter escaping: {name}"):
self.validate_all(
'SELECT 1 AS "[x]"',
read={
"tsql": f"SELECT 1 AS {name}",
},
write={
"tsql": "SELECT 1 AS [[x]]]",
},
)
for name in ('[x"]', '"x"""'):
with self.subTest(f"Testing T-SQL delimiter escaping: {name}"):
self.validate_all(
'SELECT 1 AS "x"""',
read={
"tsql": f"SELECT 1 AS {name}",
},
write={
"tsql": 'SELECT 1 AS [x"]',
},
)
def test_median(self):
for suffix in (
"",
" OVER ()",
):
self.validate_all(
f"MEDIAN(x){suffix}",
read={
"snowflake": f"MEDIAN(x){suffix}",
"duckdb": f"MEDIAN(x){suffix}",
"spark": f"MEDIAN(x){suffix}",
"databricks": f"MEDIAN(x){suffix}",
"redshift": f"MEDIAN(x){suffix}",
"oracle": f"MEDIAN(x){suffix}",
},
write={
"snowflake": f"MEDIAN(x){suffix}",
"duckdb": f"MEDIAN(x){suffix}",
"spark": f"MEDIAN(x){suffix}",
"databricks": f"MEDIAN(x){suffix}",
"redshift": f"MEDIAN(x){suffix}",
"oracle": f"MEDIAN(x){suffix}",
"clickhouse": f"MEDIAN(x){suffix}",
"postgres": f"PERCENTILE_CONT(0.5) WITHIN GROUP (ORDER BY x){suffix}",
},
)

45
tests/dialects/test_duckdb.py
View file

@ -619,12 +619,6 @@ class TestDuckDB(Validator):
"spark": "ARRAY_SUM(ARRAY(1, 2))",
},
)
self.validate_all(
"IF((y) <> 0, (x) / (y), NULL)",
read={
"bigquery": "SAFE_DIVIDE(x, y)",
},
)
self.validate_all(
"STRUCT_PACK(x := 1, y := '2')",
write={
@ -758,16 +752,9 @@ class TestDuckDB(Validator):
"snowflake": "SELECT PERCENTILE_DISC(q) WITHIN GROUP (ORDER BY x) FROM t",
},
)
self.validate_all(
"SELECT MEDIAN(x) FROM t",
write={
"duckdb": "SELECT QUANTILE_CONT(x, 0.5) FROM t",
"postgres": "SELECT PERCENTILE_CONT(0.5) WITHIN GROUP (ORDER BY x) FROM t",
"snowflake": "SELECT PERCENTILE_CONT(0.5) WITHIN GROUP (ORDER BY x) FROM t",
},
)
with self.assertRaises(UnsupportedError):
# bq has the position arg, but duckdb doesn't
transpile(
"SELECT REGEXP_EXTRACT(a, 'pattern', 1) from table",
read="bigquery",
@ -775,6 +762,36 @@ class TestDuckDB(Validator):
unsupported_level=ErrorLevel.IMMEDIATE,
)
self.validate_all(
"SELECT REGEXP_EXTRACT(a, 'pattern') FROM t",
read={
"duckdb": "SELECT REGEXP_EXTRACT(a, 'pattern') FROM t",
"bigquery": "SELECT REGEXP_EXTRACT(a, 'pattern') FROM t",
"snowflake": "SELECT REGEXP_SUBSTR(a, 'pattern') FROM t",
},
write={
"duckdb": "SELECT REGEXP_EXTRACT(a, 'pattern') FROM t",
"bigquery": "SELECT REGEXP_EXTRACT(a, 'pattern') FROM t",
"snowflake": "SELECT REGEXP_SUBSTR(a, 'pattern') FROM t",
},
)
self.validate_all(
"SELECT REGEXP_EXTRACT(a, 'pattern', 2, 'i') FROM t",
read={
"snowflake": "SELECT REGEXP_SUBSTR(a, 'pattern', 1, 1, 'i', 2) FROM t",
},
write={
"duckdb": "SELECT REGEXP_EXTRACT(a, 'pattern', 2, 'i') FROM t",
"snowflake": "SELECT REGEXP_SUBSTR(a, 'pattern', 1, 1, 'i', 2) FROM t",
},
)
self.validate_identity(
"SELECT REGEXP_EXTRACT(a, 'pattern', 0)",
"SELECT REGEXP_EXTRACT(a, 'pattern')",
)
self.validate_identity("SELECT REGEXP_EXTRACT(a, 'pattern', 0, 'i')")
self.validate_identity("SELECT REGEXP_EXTRACT(a, 'pattern', 1, 'i')")
self.validate_identity("SELECT ISNAN(x)")
self.validate_all(

19
tests/dialects/test_hive.py
View file

@ -1,5 +1,7 @@
from tests.dialects.test_dialect import Validator
from sqlglot import exp
class TestHive(Validator):
dialect = "hive"
@ -787,6 +789,23 @@ class TestHive(Validator):
},
)
self.validate_identity("EXISTS(col, x -> x % 2 = 0)").assert_is(exp.Exists)
self.validate_all(
"SELECT EXISTS(ARRAY(2, 3), x -> x % 2 = 0)",
read={
"hive": "SELECT EXISTS(ARRAY(2, 3), x -> x % 2 = 0)",
"spark2": "SELECT EXISTS(ARRAY(2, 3), x -> x % 2 = 0)",
"spark": "SELECT EXISTS(ARRAY(2, 3), x -> x % 2 = 0)",
"databricks": "SELECT EXISTS(ARRAY(2, 3), x -> x % 2 = 0)",
},
write={
"spark2": "SELECT EXISTS(ARRAY(2, 3), x -> x % 2 = 0)",
"spark": "SELECT EXISTS(ARRAY(2, 3), x -> x % 2 = 0)",
"databricks": "SELECT EXISTS(ARRAY(2, 3), x -> x % 2 = 0)",
},
)
def test_escapes(self) -> None:
self.validate_identity("'\n'", "'\\n'")
self.validate_identity("'\\n'")

4
tests/dialects/test_mysql.py
View file

@ -388,7 +388,7 @@ class TestMySQL(Validator):
"sqlite": "SELECT x'CC'",
"starrocks": "SELECT x'CC'",
"tableau": "SELECT 204",
"teradata": "SELECT 204",
"teradata": "SELECT X'CC'",
"trino": "SELECT X'CC'",
"tsql": "SELECT 0xCC",
}
@ -409,7 +409,7 @@ class TestMySQL(Validator):
"sqlite": "SELECT x'0000CC'",
"starrocks": "SELECT x'0000CC'",
"tableau": "SELECT 204",
"teradata": "SELECT 204",
"teradata": "SELECT X'0000CC'",
"trino": "SELECT X'0000CC'",
"tsql": "SELECT 0x0000CC",
}

24
tests/dialects/test_oracle.py
View file

@ -119,13 +119,6 @@ class TestOracle(Validator):
"tsql": UnsupportedError,
},
)
self.validate_all(
"TRUNC(SYSDATE, 'YEAR')",
write={
"clickhouse": "DATE_TRUNC('YEAR', CURRENT_TIMESTAMP())",
"oracle": "TRUNC(SYSDATE, 'YEAR')",
},
)
self.validate_all(
"SELECT * FROM test WHERE MOD(col1, 4) = 3",
read={
@ -632,3 +625,20 @@ WHERE
self.validate_identity("GRANT UPDATE, TRIGGER ON TABLE t TO anita, zhi")
self.validate_identity("GRANT EXECUTE ON PROCEDURE p TO george")
self.validate_identity("GRANT USAGE ON SEQUENCE order_id TO sales_role")
def test_datetrunc(self):
self.validate_all(
"TRUNC(SYSDATE, 'YEAR')",
write={
"clickhouse": "DATE_TRUNC('YEAR', CURRENT_TIMESTAMP())",
"oracle": "TRUNC(SYSDATE, 'YEAR')",
},
)
# Make sure units are not normalized e.g 'Q' -> 'QUARTER' and 'W' -> 'WEEK'
# https://docs.oracle.com/en/database/oracle/oracle-database/21/sqlrf/ROUND-and-TRUNC-Date-Functions.html
for unit in (
"'Q'",
"'W'",
):
self.validate_identity(f"TRUNC(x, {unit})")

18
tests/dialects/test_postgres.py
View file

@ -797,6 +797,24 @@ class TestPostgres(Validator):
self.validate_identity("SELECT OVERLAY(a PLACING b FROM 1 FOR 1)")
self.validate_identity("ARRAY[1, 2, 3] && ARRAY[1, 2]").assert_is(exp.ArrayOverlaps)
self.validate_all(
"""SELECT JSONB_EXISTS('{"a": [1,2,3]}', 'a')""",
write={
"postgres": """SELECT JSONB_EXISTS('{"a": [1,2,3]}', 'a')""",
"duckdb": """SELECT JSON_EXISTS('{"a": [1,2,3]}', '$.a')""",
},
)
self.validate_all(
"WITH t AS (SELECT ARRAY[1, 2, 3] AS col) SELECT * FROM t WHERE 1 <= ANY(col) AND 2 = ANY(col)",
write={
"postgres": "WITH t AS (SELECT ARRAY[1, 2, 3] AS col) SELECT * FROM t WHERE 1 <= ANY(col) AND 2 = ANY(col)",
"hive": "WITH t AS (SELECT ARRAY(1, 2, 3) AS col) SELECT * FROM t WHERE EXISTS(col, x -> 1 <= x) AND EXISTS(col, x -> 2 = x)",
"spark2": "WITH t AS (SELECT ARRAY(1, 2, 3) AS col) SELECT * FROM t WHERE EXISTS(col, x -> 1 <= x) AND EXISTS(col, x -> 2 = x)",
"spark": "WITH t AS (SELECT ARRAY(1, 2, 3) AS col) SELECT * FROM t WHERE EXISTS(col, x -> 1 <= x) AND EXISTS(col, x -> 2 = x)",
"databricks": "WITH t AS (SELECT ARRAY(1, 2, 3) AS col) SELECT * FROM t WHERE EXISTS(col, x -> 1 <= x) AND EXISTS(col, x -> 2 = x)",
},
)
def test_ddl(self):
# Checks that user-defined types are parsed into DataType instead of Identifier
self.parse_one("CREATE TABLE t (a udt)").this.expressions[0].args["kind"].assert_is(

7
tests/dialects/test_presto.py
View file

@ -414,13 +414,6 @@ class TestPresto(Validator):
"CAST(x AS TIMESTAMP)",
read={"mysql": "TIMESTAMP(x)"},
)
self.validate_all(
"TIMESTAMP(x, 'America/Los_Angeles')",
write={
"duckdb": "CAST(x AS TIMESTAMP) AT TIME ZONE 'America/Los_Angeles'",
"presto": "AT_TIMEZONE(CAST(x AS TIMESTAMP), 'America/Los_Angeles')",
},
)
# this case isn't really correct, but it's a fall back for mysql's version
self.validate_all(
"TIMESTAMP(x, '12:00:00')",

16
tests/dialects/test_snowflake.py
View file

@ -331,10 +331,15 @@ WHERE
"snowflake": "SELECT TIME_FROM_PARTS(12, 34, 56, 987654321)",
},
)
self.validate_identity(
"SELECT TIMESTAMPNTZFROMPARTS(2013, 4, 5, 12, 00, 00)",
"SELECT TIMESTAMP_FROM_PARTS(2013, 4, 5, 12, 00, 00)",
)
self.validate_all(
"SELECT TIMESTAMP_FROM_PARTS(2013, 4, 5, 12, 00, 00)",
read={
"duckdb": "SELECT MAKE_TIMESTAMP(2013, 4, 5, 12, 00, 00)",
"snowflake": "SELECT TIMESTAMP_NTZ_FROM_PARTS(2013, 4, 5, 12, 00, 00)",
},
write={
"duckdb": "SELECT MAKE_TIMESTAMP(2013, 4, 5, 12, 00, 00)",
@ -519,7 +524,6 @@ WHERE
self.validate_all(
f"SELECT PERCENTILE_CONT(0.5) WITHIN GROUP (ORDER BY x){suffix}",
read={
"snowflake": f"SELECT MEDIAN(x){suffix}",
"postgres": f"SELECT PERCENTILE_CONT(0.5) WITHIN GROUP (ORDER BY x){suffix}",
},
write={
@ -529,15 +533,6 @@ WHERE
"snowflake": f"SELECT PERCENTILE_CONT(0.5) WITHIN GROUP (ORDER BY x){suffix}",
},
)
self.validate_all(
f"SELECT MEDIAN(x){suffix}",
write={
"": f"SELECT PERCENTILE_CONT(x, 0.5){suffix}",
"duckdb": f"SELECT QUANTILE_CONT(x, 0.5){suffix}",
"postgres": f"SELECT PERCENTILE_CONT(0.5) WITHIN GROUP (ORDER BY x){suffix}",
"snowflake": f"SELECT PERCENTILE_CONT(0.5) WITHIN GROUP (ORDER BY x){suffix}",
},
)
for func in (
"CORR",
"COVAR_POP",
@ -1768,7 +1763,6 @@ FROM persons AS p, LATERAL FLATTEN(input => p.c, path => 'contact') AS _flattene
"REGEXP_SUBSTR(subject, pattern)",
read={
"bigquery": "REGEXP_EXTRACT(subject, pattern)",
"snowflake": "REGEXP_EXTRACT(subject, pattern)",
},
write={
"bigquery": "REGEXP_EXTRACT(subject, pattern)",

20
tests/dialects/test_spark.py
View file

@ -314,6 +314,20 @@ TBLPROPERTIES (
"spark": "SELECT COLLECT_LIST(x) FILTER(WHERE x = 5) FROM (SELECT 1 UNION ALL SELECT NULL) AS t(x)",
},
)
self.validate_all(
"SELECT ARRAY_AGG(1)",
write={
"duckdb": "SELECT ARRAY_AGG(1)",
"spark": "SELECT COLLECT_LIST(1)",
},
)
self.validate_all(
"SELECT ARRAY_AGG(DISTINCT STRUCT('a'))",
write={
"duckdb": "SELECT ARRAY_AGG(DISTINCT {'col1': 'a'})",
"spark": "SELECT COLLECT_LIST(DISTINCT STRUCT('a' AS col1))",
},
)
self.validate_all(
"SELECT DATE_FORMAT(DATE '2020-01-01', 'EEEE') AS weekday",
write={
@ -875,3 +889,9 @@ TBLPROPERTIES (
"databricks": "SELECT * FROM db.table1 EXCEPT SELECT * FROM db.table2",
},
)
def test_string(self):
for dialect in ("hive", "spark2", "spark", "databricks"):
with self.subTest(f"Testing STRING() for {dialect}"):
query = parse_one("STRING(a)", dialect=dialect)
self.assertEqual(query.sql(dialect), "CAST(a AS STRING)")

4
tests/dialects/test_sqlite.py
View file

@ -222,3 +222,7 @@ class TestSQLite(Validator):
"mysql": "CREATE TABLE `x` (`Name` VARCHAR(200) NOT NULL)",
},
)
self.validate_identity(
"CREATE TABLE store (store_id INTEGER PRIMARY KEY AUTOINCREMENT, mgr_id INTEGER NOT NULL UNIQUE REFERENCES staff ON UPDATE CASCADE)"
)

4
tests/dialects/test_teradata.py
View file

@ -32,6 +32,10 @@ class TestTeradata(Validator):
},
)
self.validate_identity("SELECT 0x1d", "SELECT X'1d'")
self.validate_identity("SELECT X'1D'", "SELECT X'1D'")
self.validate_identity("SELECT x'1d'", "SELECT X'1d'")
self.validate_identity(
"RENAME TABLE emp TO employee", check_command_warning=True
).assert_is(exp.Command)

21
tests/dialects/test_trino.py
View file

@ -9,9 +9,30 @@ class TestTrino(Validator):
self.validate_identity("JSON_QUERY(content, 'lax $.HY.*')")
self.validate_identity("JSON_QUERY(content, 'strict $.HY.*' WITH UNCONDITIONAL WRAPPER)")
self.validate_identity("JSON_QUERY(content, 'strict $.HY.*' WITHOUT CONDITIONAL WRAPPER)")
def test_listagg(self):
self.validate_identity(
"SELECT LISTAGG(DISTINCT col, ',') WITHIN GROUP (ORDER BY col ASC) FROM tbl"
)
self.validate_identity(
"SELECT LISTAGG(col, '; ' ON OVERFLOW ERROR) WITHIN GROUP (ORDER BY col ASC) FROM tbl"
)
self.validate_identity(
"SELECT LISTAGG(col, '; ' ON OVERFLOW TRUNCATE WITH COUNT) WITHIN GROUP (ORDER BY col ASC) FROM tbl"
)
self.validate_identity(
"SELECT LISTAGG(col, '; ' ON OVERFLOW TRUNCATE WITHOUT COUNT) WITHIN GROUP (ORDER BY col ASC) FROM tbl"
)
self.validate_identity(
"SELECT LISTAGG(col, '; ' ON OVERFLOW TRUNCATE '...' WITH COUNT) WITHIN GROUP (ORDER BY col ASC) FROM tbl"
)
self.validate_identity(
"SELECT LISTAGG(col, '; ' ON OVERFLOW TRUNCATE '...' WITHOUT COUNT) WITHIN GROUP (ORDER BY col ASC) FROM tbl"
)
self.validate_identity(
"SELECT LISTAGG(col) WITHIN GROUP (ORDER BY col DESC) FROM tbl",
"SELECT LISTAGG(col, ',') WITHIN GROUP (ORDER BY col DESC) FROM tbl",
)
def test_trim(self):
self.validate_identity("SELECT TRIM('!' FROM '!foo!')")

9
tests/dialects/test_tsql.py
View file

@ -411,6 +411,7 @@ class TestTSQL(Validator):
},
)
self.validate_identity("HASHBYTES('MD2', 'x')")
self.validate_identity("LOG(n)")
self.validate_identity("LOG(n, b)")
self.validate_all(
@ -920,6 +921,12 @@ class TestTSQL(Validator):
"": "CREATE TABLE IF NOT EXISTS foo.bar.baz AS SELECT '2020' AS z FROM a.b.c",
},
)
self.validate_all(
"IF NOT EXISTS (SELECT * FROM information_schema.tables WHERE table_name = 'baz' AND table_schema = 'bar' AND table_catalog = 'foo') EXEC('WITH cte1 AS (SELECT 1 AS col_a), cte2 AS (SELECT 1 AS col_b) SELECT * INTO foo.bar.baz FROM (SELECT col_a FROM cte1 UNION ALL SELECT col_b FROM cte2) AS temp')",
read={
"": "CREATE TABLE IF NOT EXISTS foo.bar.baz AS WITH cte1 AS (SELECT 1 AS col_a), cte2 AS (SELECT 1 AS col_b) SELECT col_a FROM cte1 UNION ALL SELECT col_b FROM cte2"
},
)
self.validate_all(
"CREATE OR ALTER VIEW a.b AS SELECT 1",
read={
@ -1567,7 +1574,7 @@ WHERE
"SELECT DATEDIFF(DAY, CAST(a AS DATETIME2), CAST(b AS DATETIME2)) AS x FROM foo",
write={
"tsql": "SELECT DATEDIFF(DAY, CAST(a AS DATETIME2), CAST(b AS DATETIME2)) AS x FROM foo",
"clickhouse": "SELECT DATE_DIFF(DAY, CAST(a AS Nullable(DateTime)), CAST(b AS Nullable(DateTime))) AS x FROM foo",
"clickhouse": "SELECT DATE_DIFF(DAY, CAST(CAST(a AS Nullable(DateTime)) AS DateTime64(6)), CAST(CAST(b AS Nullable(DateTime)) AS DateTime64(6))) AS x FROM foo",
},
)

1
tests/fixtures/identity.sql vendored
View file

@ -250,7 +250,6 @@ SELECT LEAD(a, 1) OVER (PARTITION BY a ORDER BY a) AS x
SELECT LEAD(a, 1, b) OVER (PARTITION BY a ORDER BY a) AS x
SELECT X((a, b) -> a + b, z -> z) AS x
SELECT X(a -> a + ("z" - 1))
SELECT EXISTS(ARRAY(2, 3), x -> x % 2 = 0)
SELECT test.* FROM test
SELECT a AS b FROM test
SELECT "a"."b" FROM "a"

8
tests/fixtures/optimizer/annotate_functions.sql vendored
View file

@ -307,3 +307,11 @@ ARRAY<STRING>;
# dialect: bigquery
SPLIT(tbl.bin_col, delim);
ARRAY<BINARY>;
# dialect: bigquery
STRING(json_expr);
STRING;
# dialect: bigquery
STRING(timestamp_expr, timezone);
STRING;

20
tests/fixtures/optimizer/qualify_tables.sql vendored
View file

@ -14,6 +14,26 @@ SELECT 1 FROM x.y.z AS z;
SELECT 1 FROM x.y.z AS z;
SELECT 1 FROM x.y.z AS z;
# title: only information schema
# dialect: bigquery
SELECT * FROM information_schema.tables;
SELECT * FROM c.db.`information_schema.tables` AS tables;
# title: information schema with db
# dialect: bigquery
SELECT * FROM y.information_schema.tables;
SELECT * FROM c.y.`information_schema.tables` AS tables;
# title: information schema with db, catalog
# dialect: bigquery
SELECT * FROM x.y.information_schema.tables;
SELECT * FROM x.y.`information_schema.tables` AS tables;
# title: information schema with db, catalog, alias
# dialect: bigquery
SELECT * FROM x.y.information_schema.tables AS z;
SELECT * FROM x.y.`information_schema.tables` AS z;
# title: redshift unnest syntax, z.a should be a column, not a table
# dialect: redshift
SELECT 1 FROM y.z AS z, z.a;

139
tests/test_diff.py
View file

@ -2,7 +2,6 @@ import unittest
from sqlglot import exp, parse_one
from sqlglot.diff import Insert, Move, Remove, Update, diff
from sqlglot.expressions import Join, to_table
def diff_delta_only(source, target, matchings=None, **kwargs):
@ -14,22 +13,24 @@ class TestDiff(unittest.TestCase):
self._validate_delta_only(
diff_delta_only(parse_one("SELECT a + b"), parse_one("SELECT a - b")),
[
Remove(parse_one("a + b")), # the Add node
Insert(parse_one("a - b")), # the Sub node
Remove(expression=parse_one("a + b")), # the Add node
Insert(expression=parse_one("a - b")), # the Sub node
Move(source=parse_one("a"), target=parse_one("a")), # the `a` Column node
Move(source=parse_one("b"), target=parse_one("b")), # the `b` Column node
],
)
self._validate_delta_only(
diff_delta_only(parse_one("SELECT a, b, c"), parse_one("SELECT a, c")),
[
Remove(parse_one("b")), # the Column node
Remove(expression=parse_one("b")), # the Column node
],
)
self._validate_delta_only(
diff_delta_only(parse_one("SELECT a, b"), parse_one("SELECT a, b, c")),
[
Insert(parse_one("c")), # the Column node
Insert(expression=parse_one("c")), # the Column node
],
)
@ -40,8 +41,8 @@ class TestDiff(unittest.TestCase):
),
[
Update(
to_table("table_one", quoted=False),
to_table("table_two", quoted=False),
source=exp.to_table("table_one", quoted=False),
target=exp.to_table("table_two", quoted=False),
), # the Table node
],
)
@ -53,8 +54,12 @@ class TestDiff(unittest.TestCase):
),
[
Update(
exp.Lambda(this=exp.to_identifier("a"), expressions=[exp.to_identifier("a")]),
exp.Lambda(this=exp.to_identifier("b"), expressions=[exp.to_identifier("b")]),
source=exp.Lambda(
this=exp.to_identifier("a"), expressions=[exp.to_identifier("a")]
),
target=exp.Lambda(
this=exp.to_identifier("b"), expressions=[exp.to_identifier("b")]
),
),
],
)
@ -65,8 +70,8 @@ class TestDiff(unittest.TestCase):
parse_one('SELECT a, b, "my.udf1"()'), parse_one('SELECT a, b, "my.udf2"()')
),
[
Insert(parse_one('"my.udf2"()')),
Remove(parse_one('"my.udf1"()')),
Insert(expression=parse_one('"my.udf2"()')),
Remove(expression=parse_one('"my.udf1"()')),
],
)
self._validate_delta_only(
@ -75,41 +80,73 @@ class TestDiff(unittest.TestCase):
parse_one('SELECT a, b, "my.udf"(x, y, w)'),
),
[
Insert(exp.column("w")),
Remove(exp.column("z")),
Insert(expression=exp.column("w")),
Remove(expression=exp.column("z")),
],
)
def test_node_position_changed(self):
expr_src = parse_one("SELECT a, b, c")
expr_tgt = parse_one("SELECT c, a, b")
self._validate_delta_only(
diff_delta_only(parse_one("SELECT a, b, c"), parse_one("SELECT c, a, b")),
diff_delta_only(expr_src, expr_tgt),
[
Move(parse_one("c")), # the Column node
Move(source=expr_src.selects[2], target=expr_tgt.selects[0]),
],
)
expr_src = parse_one("SELECT a + b")
expr_tgt = parse_one("SELECT b + a")
self._validate_delta_only(
diff_delta_only(parse_one("SELECT a + b"), parse_one("SELECT b + a")),
diff_delta_only(expr_src, expr_tgt),
[
Move(parse_one("a")), # the Column node
Move(source=expr_src.selects[0].left, target=expr_tgt.selects[0].right),
],
)
expr_src = parse_one("SELECT aaaa AND bbbb")
expr_tgt = parse_one("SELECT bbbb AND aaaa")
self._validate_delta_only(
diff_delta_only(parse_one("SELECT aaaa AND bbbb"), parse_one("SELECT bbbb AND aaaa")),
diff_delta_only(expr_src, expr_tgt),
[
Move(parse_one("aaaa")), # the Column node
Move(source=expr_src.selects[0].left, target=expr_tgt.selects[0].right),
],
)
expr_src = parse_one("SELECT aaaa OR bbbb OR cccc")
expr_tgt = parse_one("SELECT cccc OR bbbb OR aaaa")
self._validate_delta_only(
diff_delta_only(
parse_one("SELECT aaaa OR bbbb OR cccc"),
parse_one("SELECT cccc OR bbbb OR aaaa"),
),
diff_delta_only(expr_src, expr_tgt),
[
Move(parse_one("aaaa")), # the Column node
Move(parse_one("cccc")), # the Column node
Move(source=expr_src.selects[0].left.left, target=expr_tgt.selects[0].right),
Move(source=expr_src.selects[0].right, target=expr_tgt.selects[0].left.left),
],
)
expr_src = parse_one("SELECT a, b FROM t WHERE CONCAT('a', 'b') = 'ab'")
expr_tgt = parse_one("SELECT a FROM t WHERE CONCAT('a', 'b', b) = 'ab'")
self._validate_delta_only(
diff_delta_only(expr_src, expr_tgt),
[
Move(source=expr_src.selects[1], target=expr_tgt.find(exp.Concat).expressions[-1]),
],
)
expr_src = parse_one("SELECT a as a, b as b FROM t WHERE CONCAT('a', 'b') = 'ab'")
expr_tgt = parse_one("SELECT a as a FROM t WHERE CONCAT('a', 'b', b) = 'ab'")
b_alias = expr_src.selects[1]
self._validate_delta_only(
diff_delta_only(expr_src, expr_tgt),
[
Remove(expression=b_alias),
Move(source=b_alias.this, target=expr_tgt.find(exp.Concat).expressions[-1]),
],
)
@ -130,23 +167,30 @@ class TestDiff(unittest.TestCase):
self._validate_delta_only(
diff_delta_only(parse_one(expr_src), parse_one(expr_tgt)),
[
Remove(parse_one("LOWER(c) AS c")), # the Alias node
Remove(parse_one("LOWER(c)")), # the Lower node
Remove(parse_one("'filter'")), # the Literal node
Insert(parse_one("'different_filter'")), # the Literal node
Remove(expression=parse_one("LOWER(c) AS c")), # the Alias node
Remove(expression=parse_one("LOWER(c)")), # the Lower node
Remove(expression=parse_one("'filter'")), # the Literal node
Insert(expression=parse_one("'different_filter'")), # the Literal node
Move(source=parse_one("c"), target=parse_one("c")), # the new Column c
],
)
def test_join(self):
expr_src = "SELECT a, b FROM t1 LEFT JOIN t2 ON t1.key = t2.key"
expr_tgt = "SELECT a, b FROM t1 RIGHT JOIN t2 ON t1.key = t2.key"
expr_src = parse_one("SELECT a, b FROM t1 LEFT JOIN t2 ON t1.key = t2.key")
expr_tgt = parse_one("SELECT a, b FROM t1 RIGHT JOIN t2 ON t1.key = t2.key")
changes = diff_delta_only(parse_one(expr_src), parse_one(expr_tgt))
src_join = expr_src.find(exp.Join)
tgt_join = expr_tgt.find(exp.Join)
self.assertEqual(len(changes), 2)
self.assertTrue(isinstance(changes[0], Remove))
self.assertTrue(isinstance(changes[1], Insert))
self.assertTrue(all(isinstance(c.expression, Join) for c in changes))
self._validate_delta_only(
diff_delta_only(expr_src, expr_tgt),
[
Remove(expression=src_join),
Insert(expression=tgt_join),
Move(source=exp.to_table("t2"), target=exp.to_table("t2")),
Move(source=src_join.args["on"], target=tgt_join.args["on"]),
],
)
def test_window_functions(self):
expr_src = parse_one("SELECT ROW_NUMBER() OVER (PARTITION BY a ORDER BY b)")
@ -157,8 +201,8 @@ class TestDiff(unittest.TestCase):
self._validate_delta_only(
diff_delta_only(expr_src, expr_tgt),
[
Remove(parse_one("ROW_NUMBER()")),
Insert(parse_one("RANK()")),
Remove(expression=parse_one("ROW_NUMBER()")),
Insert(expression=parse_one("RANK()")),
Update(source=expr_src.selects[0], target=expr_tgt.selects[0]),
],
)
@ -178,20 +222,21 @@ class TestDiff(unittest.TestCase):
self._validate_delta_only(
diff_delta_only(expr_src, expr_tgt),
[
Remove(expr_src),
Insert(expr_tgt),
Insert(exp.Literal.number(2)),
Insert(exp.Literal.number(3)),
Insert(exp.Literal.number(4)),
Remove(expression=expr_src),
Insert(expression=expr_tgt),
Insert(expression=exp.Literal.number(2)),
Insert(expression=exp.Literal.number(3)),
Insert(expression=exp.Literal.number(4)),
Move(source=exp.Literal.number(1), target=exp.Literal.number(1)),
],
)
self._validate_delta_only(
diff_delta_only(expr_src, expr_tgt, matchings=[(expr_src, expr_tgt)]),
[
Insert(exp.Literal.number(2)),
Insert(exp.Literal.number(3)),
Insert(exp.Literal.number(4)),
Insert(expression=exp.Literal.number(2)),
Insert(expression=exp.Literal.number(3)),
Insert(expression=exp.Literal.number(4)),
],
)
@ -274,7 +319,7 @@ class TestDiff(unittest.TestCase):
source=expr_src.find(exp.Order).expressions[0],
target=expr_tgt.find(exp.Order).expressions[0],
),
Move(parse_one("a")),
Move(source=expr_src.selects[0], target=expr_tgt.selects[1]),
],
)

43
tests/test_optimizer.py
View file

@ -315,7 +315,7 @@ class TestOptimizer(unittest.TestCase):
),
dialect="bigquery",
).sql(),
'WITH "x" AS (SELECT "y"."a" AS "a" FROM "DB"."y" AS "y" CROSS JOIN "a"."b"."INFORMATION_SCHEMA"."COLUMNS" AS "COLUMNS") SELECT "x"."a" AS "a" FROM "x" AS "x"',
'WITH "x" AS (SELECT "y"."a" AS "a" FROM "DB"."y" AS "y" CROSS JOIN "a"."b"."INFORMATION_SCHEMA.COLUMNS" AS "columns") SELECT "x"."a" AS "a" FROM "x" AS "x"',
)
self.assertEqual(
@ -1337,6 +1337,47 @@ FROM READ_CSV('tests/fixtures/optimizer/tpc-h/nation.csv.gz', 'delimiter', '|')
self.assertEqual(union_by_name.selects[0].type.this, exp.DataType.Type.BIGINT)
self.assertEqual(union_by_name.selects[1].type.this, exp.DataType.Type.DOUBLE)
# Test chained UNIONs
sql = """
WITH t AS
(
SELECT NULL AS col
UNION
SELECT NULL AS col
UNION
SELECT 'a' AS col
UNION
SELECT NULL AS col
UNION
SELECT NULL AS col
)
SELECT col FROM t;
"""
self.assertEqual(optimizer.optimize(sql).selects[0].type.this, exp.DataType.Type.VARCHAR)
# Test UNIONs with nested subqueries
sql = """
WITH t AS
(
SELECT NULL AS col
UNION
(SELECT NULL AS col UNION ALL SELECT 'a' AS col)
)
SELECT col FROM t;
"""
self.assertEqual(optimizer.optimize(sql).selects[0].type.this, exp.DataType.Type.VARCHAR)
sql = """
WITH t AS
(
(SELECT NULL AS col UNION ALL SELECT 'a' AS col)
UNION
SELECT NULL AS col
)
SELECT col FROM t;
"""
self.assertEqual(optimizer.optimize(sql).selects[0].type.this, exp.DataType.Type.VARCHAR)
def test_recursive_cte(self):
query = parse_one(
"""